If tip commit of local hg repo is stripped, git-remote-hg does not automatically recover

[fingolfin]

Suppose you have a local hg repository, and a git clone of that, both are fully in sync. Now the tip commit of the remote hg repository is stripped. If one does git pull, then origin/master is correctly updated.

I expected to be able to push the old commit (stripped from hg, but still present on the git side) out again, but not so, instead I get into this "infinite cycle" and no commit appears in the hg repo:

$ git push
To hg::/path/to/test.hg
   18b66f9..4b5fcb1  master -> master
$ git pull
From hg::/path/to/test.hg
 + 4b5fcb1...18b66f9 master     -> origin/master  (forced update)
Already up-to-date.
$ git pull
Already up-to-date.
$ git push
To hg::/path/to/test.hg
   18b66f9..4b5fcb1  master -> master
$ git pull
From hg::/path/to/test.hg
 + 4b5fcb1...18b66f9 master     -> origin/master  (forced update)
Already up-to-date.
$ git pull
Already up-to-date.

I guess this is because the marks file still references the commit, but the commit is gone in the repository. And since for local hg repositories no clone is made, it cannot recover from it.

Note that everything works fine if I set the GIT_REMOTE_HG_TEST_REMOTE env to force the creation of a clone even for local repositories.

So one possible fix would be to simply remove the special case for local repositories.

[fgb]

@fingolfin: So you're saying that stripping the tip commit from hg removes the commit from git? For me, doing that is the surest way to cause a broken pipe issue that cripples git-remote-hg until you fix it via the steps on #4 (comment).

[fingolfin]

@fgb I am saying what I saying :). I think I even made a test script to setup hg and git repos like I described, to replicate the issue. No "broken pipe" for me. Might be due to differences in our setup (Mercurial & git version used, version of git-remote-hg used), but of course also with the repository (I only tested this with toy repositories, and at the test time, only the one commit was removed -- your situation might be different, e.g. a commit removed and others added, etc.). It's been a long time I looked at this, so my memoy is a bit fuzzy, but I am pretty sure I run these tests specifically to try and see whether I could reproduce your issues and some others I experienced myself in the past. Unfortunately, they tend to be difficult to reproduce consistently (ideally: with a script)... :-(

I'll try to dig up my complete test script for this and add it here.

[mnauw]

As commented above, there are quite some things that can go wrong when doing a strip on a remote hg repo, though git-remote-hg is not really to blame. That is, strip is a pretty invasive and tricky operation and such fiddling with tips will confuse things as recorded in marks-hg file. One could have git-remote-hg resort to some guessing but that's no safe way to recover. A safer way is to bring marks-hg back into a consistent state using git-marks-check (as referenced in #4 (comment)) or a helper script added in mnauw/git-remote-hg@38741e0 (which also resets the tips in marks-hg so that does not have to be done manually).

[mnauw]

Actually, I stand self-corrected; git-remote-hg would not have to resort to guessing (tips status). With some more thorough modifications of how to-be-imported revisions are determined, it can always accomplish that task just fine in a safe and robust way. In particular, the approach of above commit no longer relies on any tip tracking, so any manual or otherwise script fiddling with that is no longer necessary.

Note that the counterpart of an hg strip is a git gc, and performing that might need some maintenance to recover from then (pruned) invalid git commit ids in the marks-git file, e.g. using the aforementioned helper script's gc subcommand (whose cleanup actions might alternatively be auto-performed by git-remote-hg at startup).

[fgb]

@mnauw Have you checked out #56? I'd rather have a solution rolled into git-remote-hg instead of relying on updates to git's upstream, but @felipec does have a point about git fast-import needing a patch.

[mnauw]

Indeed, fixes and solutions should preferably be within git-remote-hg. As such, @fgb, #58 and its fixes might help in that regard. In the same way commits such as mnauw/git-remote-hg@b3fccdd here eliminate tips metadata tracking, those commits eliminate last_note metadata tracking. In fact, the only relevant metadata that is then left is the mapping from hg revision to mark number (in the same way the git side only relates commit sha1 to mark). After all, if that suffices for the Git DAG side, it also should for the Mercurial DAG side (and indeed it does). Less (superfluous) metadata tracking means less that can go stale and that can break things (and indeed, git fast-import could do with a patch in the mentioned case, but that does not absolve git-remote-hg from its own responsibilities and robust metadata management in the first place).

[fgb]

@mnauw I agree. Thanks for the linked commits. I'll see if I have better luck integrating #58. The patch in #56 hasn't solved all failure modes for me at this point, but breakage has become rarer.

[novalis]

@mnauw I tried to use your fork to solve the issue with re-pushing stripped commits, but it doesn't seem to work for me. Can you tell me what I'm doing wrong? (I would have filed this as an issue on your fork, but you don't seem to have issues open).

I've attached my demo script (sorry it's a .txt -- github won't let me upload a .sh under that name).
demo.txt

Or is that just not supported yet? I saw "recovering from a strip on remote repo" in your README, so I thought I might get lucky.

Thanks.

[mnauw]

@novalis oops, seems to have missed enabling issues, that should be ok now.

At the time of writing that in the README, I mainly had handling fetching following a strip in mind. To recover in a way for a push to succeed as well, the helper script's gc subcommand (mentioned above) could be used.

However, alternatively, I have had some more changes pending for some time. So, at present, upgrading to the latest master that I just pushed also handles pushing after a strip. In particular, running the demo.txt script with latest master results in seeing the expected. Note that some minor settings change (and "conversion") is required to have similar results on existing repos (see documentation), or cloning afresh will also do.

[novalis]

@mnauw It does indeed seem to work. Thanks!