Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support ingesting of combined tarballs for software layer #142

Open
boegel opened this issue Jan 6, 2023 · 1 comment
Open

support ingesting of combined tarballs for software layer #142

boegel opened this issue Jan 6, 2023 · 1 comment

Comments

@boegel
Copy link
Contributor

boegel commented Jan 6, 2023

It would be interesting if the ingest-tarball.sh script would support ingesting a combined tarball of software installations for the software layer, in which a small number of installations across all (or at least multiple) CPU targets are included.

For OSU-Micro-Benchmarks (see PR #201), we had to create and ingest 10 separate small tarballs, which only adds overhead (since for each tarball, a PR to the staging repo is created, and needs to be reviewed/merged).
The bot could combine all tarballs into a single one when it's instructed to do the deploy.

The ingest script currently makes a couple of assumptions that are incorrect for such a combined tarball, including that the tarball only includes software installations for a single CPU target, and that tar tf ${tarball} | head -1 allows to determine whether it's an ingesting for software/compat/init .

@boegel
Copy link
Contributor Author

boegel commented Jun 26, 2024

A problem arose today when trying to ingest a tarball that included updates hooks (2023.06/init/easybuild/eb_hooks.py), since that messed up the way in which the CPU architecture is being checked, which is based on this:

tar_first_file=$(tar tf "${tar_file}" | head -n 1)

@bedroge Do you think it's OK if we add grep -v '/init/' in there, before the head?
I guess we should also make sure that $tar_first_file is not empty then...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant