tar_hash mismatch: need to remove bad release from repo

[walt-jones]

We had a bad release (1.29) that was added to the repo which we subsequently rebuilt and added again to the repo (still 1.29) to replace the bad version. Now we hit a "tar_hash mismatch" trying to update from versions <1.29 to the current version >1.30.

We tried removing the patch for 1.29, but then updates fail with a 404 not found on the 1.29 patch and we remain stuck on versions <1.29. We also tried removing 1.29 from targets.json but that gives us the same "tar_hash mismatch" error as if 1.29 was still in there.

How can we remove 1.29 from the repo and skip over it for updates?

2024/29/10 04:13:32+0000 518ms : DEBUG : client : _apply_updates : update failed: target renamed to C:\Windows\system32\config\systemprofile\AppData\Local\notactualltheapp\update_cache\targets\notactualltheapp-1.28.patch.failed 2024/29/10 04:13:32+0000 519ms : DEBUG : client : _apply_updates : update failed: target renamed to C:\Windows\system32\config\systemprofile\AppData\Local\notactualltheapp\update_cache\targets\notactualltheapp-1.29.patch.failed 2024/29/10 04:13:32+0000 519ms : DEBUG : client : _apply_updates : update failed: target renamed to C:\Windows\system32\config\systemprofile\AppData\Local\notactualltheapp\update_cache\targets\notactualltheapp-1.30.patch.failed 2024/29/10 04:13:32+0000 520ms : DEBUG : client : _apply_updates : update failed: target renamed to C:\Windows\system32\config\systemprofile\AppData\Local\notactualltheapp\update_cache\targets\notactualltheapp-1.31.patch.failed 2024/29/10 04:13:32+0000 521ms : DEBUG : client : _apply_updates : update failed: target renamed to C:\Windows\system32\config\systemprofile\AppData\Local\notactualltheapp\update_cache\targets\notactualltheapp-1.32.patch.failed 2024/29/10 04:13:32+0000 521ms : ERROR : client : _apply_updates : update aborted: verification failed: tar_hash mismatch

[walt-jones]

It's worth mentioning as well that deployments of >=1.30 all update themselves fine without tar_hash mismatch errors. It's getting around the bad 1.29 that seems impossible.

[walt-jones]

We also tried removing 1.28 from targets.json since the first patch error originates with that version, but then we get a "targets was signed by 0/1 keys" error with updates not possible:

2024/29/10 05:31:53+0000 049ms : DEBUG : fetcher : download_file : Downloaded 7149 out of 5000000 bytes 2024/29/10 05:31:53+0000 050ms : DEBUG : trusted_metadata_set : update_delegated_targets : Updating targets delegated by root 2024/29/10 05:31:53+0000 051ms : INFO : _payload : get_verification_result : Key 92df308145efeb97bd22dc1e4af7b923a332417ec9fec0ae1e6353c71e5e1f80 failed to verify targets 2024/29/10 05:31:53+0000 051ms : WARNING : client : check_for_updates : Cannot refresh metadata: targets was signed by 0/1 keys 2024/29/10 05:31:53+0000 051ms : DEBUG : __init__ : update : No updates!

[dennisvang]

@walt-jones Could you post the content of targets.json? (please make sure to redact any sensitive information where necessary)

[walt-jones]

Sure @dennisvang:
targets.json

[walt-jones]

@dennisvang any luck finding a solution to this?

[dennisvang]

@walt-jones sorry for the delay. Although I cannot be certain without more info, it looks like you've removed things in a way that is not supported.

[...] which we subsequently rebuilt and added again to the repo (still 1.29) [...]

How exactly did you do that? Did you use the commands provided by tufup, or did you edit the metadata manually?

Also note that tufup only supports rollback of the latest release, using e.g. tufup targets remove-latest. If you need to replace an earlier release, you'll need to roll back one by one, and subsequently re-release multiple times to get back to the current version. This is especially important in the case of patches, because they represent interdependent links in a chain.

See #117 for more information.

[walt-jones]

Thanks @dennisvang

[...] which we subsequently rebuilt and added again to the repo (still 1.29) [...]

How exactly did you do that? Did you use the commands provided by tufup, or did you edit the metadata manually?

We used the .bat to build a new .exe and then ran repo_add_bundle.py which prompted y/n to overwrite the version and we said "y". We did not otherwise modify the metadata or make any changes by hand... until we discovered the target verification errors and then tried to remove that version from all records in hopes we could somehow jump over it.

Should we have used tufup targets remove-latest instead of saying "y" to repo_add_bundle.py's prompt to overwrite the version?

[dennisvang]

[...] and then ran repo_add_bundle.py which prompted y/n to overwrite the version [...]

Hi @walt-jones, just to be clear, this is based on the tufup-example script repo_add_bundle.py?

I wonder: Did you do this after 1.30 was already published, or was 1.29 the latest version at that time?

Unfortunately this specific case seems to have slipped through the cracks, although I would have expected the condition in add_bundle() to cover it:

tufup/src/tufup/repo/__init__.py

Lines 790 to 791 in 8dad64b

	if not latest_archive or latest_archive.version < new_archive.version:
	# register new archive

Should we have used tufup targets remove-latest instead of saying "y" to repo_add_bundle.py's prompt to overwrite the version?

Yes, that would have been the best approach.

In any case, looks like we need to improve both our tests and the docs, in this regard.

[walt-jones]

@dennisvang Yes, it's a completely unmodified tufup-example script repo_add_bundle.py. We ran it twice for 1.29 which at the time was the latest version. We never decrement the version or try and retroactively patch.

Anecdotally I can say that we've hit this issue before and since on another repository - our workflow now is never ever run repo_add_bundle.py more than once on the same version. It always seems to screw things up.

Going forward we'll try tufup targets remove-latest.