Explore the idea of a Reverse Manifest Download of an Object/Version

[terrywbrady]

For object/version retrievals, would it be possible for Merritt to deliver a reverse manifest object containing links to individual file downloads? Once a reverse manifest has been downloaded to a user's desktop, could software be written to download and assemble the component files directly from the user's desktop?

As Merritt introduces signed URL downloads for files, the creation of a reverse manifest would eliminate the need to assemble (and duplicate) content in order to serve up an object or version download.

The reverse manifest download tool would behave like Box/Google Drive/Dropbox synchronization of a folder.

Have any other preservation systems implemented a feature like this?

• Ask the OCFL community if this is under consideration
• Ask APTrust if they have implemented something like this
• Ask John K if he knows of any efforts in this area

[terrywbrady]

Conversation with John K:...

If I understand correctly, it may already be solved by anyone with a library that implements bag (BagIt) fetching via the fetch.txt file. Merritt would ship a skimpy "holey" bag, metaphorically filled with "holes", that the recipient would fill by running through the received fetch.txt file (the reverse manifest). (edited)

A post about Wellcome Collection, which describes their approach to building a storage service layer on top of Amazon S3 with BagIt.

• https://stacks.wellcomecollection.org/building-wellcome-collections-new-archival-storage-service-3f68ff21927e

One specific thing that caught my eye was their use of the fetch.txt to do versioning of bags:

• https://github.com/wellcomecollection/docs/tree/master/rfcs/002-archival_storage#updating-an-existing-bag

[terrywbrady]

A new article

• https://stacks.wellcomecollection.org/how-we-store-multiple-versions-of-bagit-bags-e68499815184

[terrywbrady]

Doing a quick scan, I have not found client software to resolve fetch.txt entries.

• Download files in fetch.txt LibraryOfCongress/bagit-python#118

Under what conditions could we ask an end user to use client software to resolve fetch.txt files?

• Make this mandatory for any object/version retrieval
• Make this mandatory for any object/version retrieval above a size threshold

What would make this useful for an end user?

• For certain types of objects/versions, the processing of the fetch.txt file might be easier to throttle on the client side rather than performing one really large download
• Recovery/retries would be easier to perform

[terrywbrady]

The python bagit code might have what we need.

https://github.com/LibraryOfCongress/bagit-python/blob/a3a8702842d4ccd02486e5a9cf85ca8ad2b02995/test.py#L1007