Ingesting tarballs

CernVM-FS allows you to ingest the contents of a tarball to a repository, without having to extract and ingest the files manually. This is useful for adding large tarballs with many/large software installations.

Making the tarball

When making the tarball, make sure that you don't include the top directory, i.e. the "YYYY.MM" directory in case of the pilots. If you do include, the tarball has to be ingested to the root of the repo, which is currently not possible with CVMFS (due to this issue).

So, if you want to make a tarball with all newly installed software from some overlay's upper directory (here $OVERLAY_UPPER), use something like the following to only include the software directory and everything below it:

tar -C $OVERLAY_UPPER/cvmfs/pilot.eessi-hpc.org/2020.10 -czf EESSI_2020.10_software_intel_broadwell.tar.gz software

Ingesting the tarball

Currently, tarballs can only be ingested on the Stratum 0 server. Store your tarball somewhere where Bob can download it, or store it in /software/eessi on the Stratum 0 after asking Bob for access. In either case, send him a message when the tarball is available, after which the tarball can be ingested.

Ingestion of an uncompressed tarball with /cvmfs/pilot.eessi-hpc.org/2020.10 as base directory for the extraction can be done using:

sudo cvmfs_server ingest -b 2020.10 -t somearchive.tar

For ingesting a compressed tarball, use the appropriate uncompress command and extract the tarball to stdout, e.g.:

gunzip -c somearchive.tar.gz | sudo cvmfs_server ingest -b 2020.10 -t - pilot.eessi-hpc.org
xzcat somearchive.tar.xz | sudo cvmfs_server ingest -b 2020.10 -t - pilot.eessi-hpc.org

Note that in all cases the base directory depends on how the tarball was created.