[FR] ability to specify a merge strategy for dependencies

[imz]

I have a workflow where I must "git merge -s ours topic1 topic2 ..." into master (before releasing a package in ALTLinux (in Russian) which includes topic1, topic2, etc. as patches in the package. (Then the commit is tagged as the release, so that the patches can always be generated if you have Git history up to the tagged commit.)

I'd like to automate this workflow with TopGit, so that tg update would know that these dependencies should be merged with -s ours.

tg create should be able to save this additional information about the desired merge strategy in .topdeps, and tg update should follow this specification.

[greenrd]

The link you gave is in Russian, and I don't speak Russian. Can you explain the desired use case in more detail, please? Because, as I understand it, "-s ours" is kind of like a fake merge that doesn't really merge, so it seems like a strange thing to want to do in this context.

[imz]

(Sorry for unreadable Russian, but since there was not much in English about this, I simply put a reference for my issue of using TopGit together with Gear (in English), to have some references. I'll try to explain this concisely.)

One of the things that Gear does is similar to tg export --quilt: it generates patches from marked Git branches, and puts them in an .src.rpm, which is to be built for a distro. (About this scenario (in English).) A Git commit (passed as a tag corresponding to the release of the package) containing the meta-information for a built package is the only argument that the Gear-based builder robot at git.alt receives. So it operates on the commit and must get all needed objects (sources) from the Git history which is pulled when you pull that commit.

(First, the tag is pulled, then the Gear-based builder is run in an isolated environment, so that it can't--by design--fetch additional commits. Roughly, the SHA-ids of the objects that should be used to generate the sources and patches are stored in the pulled .spec branch in a special file, .gear/tags.)

Therefore it is a common practice among ALT developers to make the branches with upstream sources and the branches with patches (that will be converted into an .src.rpm by Gear and built) "fake" ancestors of the commit in the branch with the meta-information (.spec) of the package. It is done by "git merge -s ours upstream patch1 patch2 ...", not to pollute the branch which holds the meta-information. (This trick is not however documented in the "scenario" I gave a link to above.)

BTW, this has a nice property: then, from the Git history, it is clear which commits of the sources were used in every tagged release of the package (because there are explicit merges in the history).

Here, from the point of view of TopGit, the .spec branch (holding the meta-information for a distro, say, Sisyphus) should be created with

tg create sisyphus upstream patch1 patch2 ...

and updated with tg update whenever there are updates in upstream or patch1 etc.

The only missing bit is that the update must be a fake merge (-s ours).

[greenrd]

If you used tg export, you would be able to have a real branch or a real Quilt patch series (no need for fake branches), and you would be able to manually fix any merge conflicts as soon as they occurred, when doing tg update. What is the advantage of your approach, compared to that?

[imz]

A TopGit branch has all the information needed for creating a patch (the base and the tip), so there is no need for an extra step like tg export if the package maintainers are collaborating on TopGit branches.

The changes used to build a package release are saved forever in the Gear repo (a Git repo). If we did tg export afresh for every release, the Git history would become a terrible meaningless mess, because new exported branches would appear in the history on every package release, whereas the idea is to track the step-by-step development of the package (and step-by-step development of the patches in the branches).

The automatic building system "git.alt" also does an inheritance check: it is required that the new commit corresponding to a new release has the old release as an ancestor. This would require--in the case of branches exported every time--to have merges of apparently unrelated branches on every release (whereas this is simply another export of the same TopGit branch).

But all this is not so important. The situation is like this:

1. The build automation tool "Gear" does generate patches from Git branches and puts them into .src.rpm. We do not need or want to influence this.
2. The build automation system "Gear" and "git.alt" require that the branches (their history) that the patches are generated from are fetched by Git as a part of the history of the release tag. This boils down to a workflow where you merge all used branches in their up-to-date state into the package meta-info branch before every release.
3. If we have decided to create an .src.rpm with patches (the "traditional old good .src.rpm"), there is no need to make a real merge and pollute the meta-info branch with sources from other branches. (The source archive and patches will be automatically generated from the corresponding commits anyway.) We simply do 'git merge -s ours' to symbolize that the merged commits (as it appears in the Git history) were used to generate this release of the .src.rpm and then to build the binary packages. The release tag (sitting on the meta-info branch) is signed, so this information about the used sources is unforgeable because Git gives the guarantees.

So, before every release, we need to update the meta-info branch by merging all patch branches (and the upstream branch). This is a thing TopGit is good at (i.e., updating by merging). Another thing TopGit is good at is doing development on the inter-dependent patch branches, and package maintainers would collaborate on TopGit branches.

The only missing piece is that ALT package maintainers need simply a fake "-s ours" merge into the meta-info branch (whhich depends an all patch branches). It would be convenient if tg update was able to do this.

[imz]

I seem to have coded up a simple solution for this FR, I'll show it in a while.

[imz]

and you would be able to manually fix any merge conflicts as soon as they occurred, when doing tg update. What is the advantage of your approach, compared to that?

As for the advantages, I've just told about them above: not polluting the meta-info branch (used to release packages) with sources.

But you make a very important observation concerning the conflicts. I totally agree that it (resolving the conflicts) would be a convenient consequence of using a real merge instead of "-s ours". Thanks a lot for this idea, I haven't thought about it when adapting TopGit to the task of maintaining ALT packages.

Well, since the meta-info branch (I usually call it "sisyphus" after the package repository name for which the package is prepared) has no sources in common with other branches it depends upon ("upstream" and the patch branches), meaningful conflicts and conflict resolutions would happen only when merging the depended branches into "top-bases/sisyphus" (the base for the meta-info branch).

Interesting observation. It leads to the idea that the base should be done with a real merge, but merging the base into the meta-info branch could be with -s ours.

I see an open problem in this workflow which I haven't thought over yet: the generated patches should take the conflict resolutions done in "top-bases/sisyphus" into account, otherwise building the .src.rpm would simply fail (because of conflicting patches)!

Of course, teaching Gear to invoke tg export as an option for generating patches is one solution. Its downside is that Gear was thought of as a tool independent of any special workflows which the maintainers like (and independent of any Git-based tools the package maintainers would like to use in their work).

[greenrd]

I see an open problem in this workflow which I haven't thought over yet: the generated patches should take the conflict resolutions done in "top-bases/sisyphus" into account, otherwise building the .src.rpm would simply fail (because of conflicting patches)!

This would still be a problem even if you used tg export to linearize the patch series, actually. I can see two possible solutions:

1. Enable git rerere in your git config - this will be used automatically by tg export. This might work satisfactorily - I'm not sure, I haven't tried it.
2. Create a fixed linear dependency ordering for your topic branches, like tg import does. You might want this anyway, in order to prevent tg export confusingly choosing a different order of patches each time. When the time comes to drop a patch, TopGit does not yet provide an automated way to delete any dependency - let alone a way to delete a dependency from a linear chain of topic branches, which is more complicated. But that's OK, you can just tg annihilate it, and then tg export should treat it as if it's not there.

[greenrd]

Actually, forget the first idea, because git rerere is not distributed at all - your team members wouldn't be able to use conflict resolutions you have done, even though they are recorded in commits!

[imz]

Ok, I see I'll have to invent something if I want to resolve conflicts...

BTW, is such a situation somehow treated in TopGit?.. You want to export several branches (they are not declared as depending on each other), and in fact they have conflicts with each other. (Not semantic conflicts, but some "trivial", resolvable conflicts which arise because of the need for a linear order of application.)

I thought, perhaps, the linearizing modes of tg export would probably ask to resolve the conflicts when linearizing, but your words seem not to support this:

This would still be a problem even if you used tg export to linearize the patch series

Ok, I'll think about this more.

[greenrd]

Yes, they do ask to resolve the conflicts, but there is no memory of the conflict resolutions unless you use git rerere, which isn't distributed.

[imz]

I've described my (successful) experience of using this at http://en.altlinux.org/Gear_with_topgit. With my small patch.

My case didn't involve any conflicts between patches, so this problem didn't arise.