Garbage collection for Spack Environments #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…m node dictionaries
# Conflicts: # lib/spack/spack/cmd/uninstall.py
|
[Ooops... I thought this was going on the main Spack site. @scheibelp it's here because your PR branch is here, not because I want you to merge it into that branch. I will reference this PR from the main Spack.] |
scheibelp
force-pushed
the
features/environments
branch
2 times, most recently
from
0ddecbe
to
41cc369
Compare
tgamblin
force-pushed
the
features/environments
branch
2 times, most recently
from
30fc088
to
acdcc29
Compare
scheibelp
force-pushed
the
features/environments
branch
2 times, most recently
from
7439b78
to
3529325
Compare
tgamblin
force-pushed
the
features/environments
branch
3 times, most recently
from
0af3375
to
438737b
Compare
24 tasks
scheibelp
pushed a commit
that referenced
this pull request
Nov 27, 2018
Verification build [email protected]:pr.ucx.spack $ spack install ucx % gcc @ 4.8.5 ==> Installing libsigsegv ==> Installing m4 ==> Installing pkgconf ==> Installing ncurses ==> Installing readline ==> Installing gdbm ==> Installing perl ==> Installing autoconf ==> Installing automake ==> Installing libtool ==> Installing numactl ==> Installing zlib ==> Installing openssl ==> Installing diffutils ==> Installing bzip2 ==> Installing xz ==> Installing libxml2 ==> Installing tar ==> Installing gettext ==> Installing help2man ==> Installing bison ==> Installing flex ==> Installing libnl ==> Installing rdma-core ==> Installing ucx ==> Successfully installed ucx Fetch: 1.17s. Build: 40.59s. Total: 41.76s. [+] /scratch/users/dantopa/new-spack/pr.ucx.spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/ucx-1.4.0-ilfonxj2f5my4ur65li3sr7kuvfg3kim [email protected]:pr.ucx.spack $ spack spec ucx Input spec -------------------------------- ucx Concretized -------------------------------- [email protected]%[email protected] arch=linux-centos7-x86_64 ^[email protected]%[email protected] patches=592f30f7f5f757dfc239ad0ffd39a9a048487ad803c26b419e0f96b8cda08c1a arch=linux-centos7-x86_64 ^[email protected]%[email protected] arch=linux-centos7-x86_64 ^[email protected]%[email protected] patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-centos7-x86_64 ^[email protected]%[email protected] arch=linux-centos7-x86_64 ^[email protected]%[email protected]+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=linux-centos7-x86_64 ^[email protected]%[email protected] arch=linux-centos7-x86_64 ^[email protected]%[email protected] arch=linux-centos7-x86_64 ^[email protected]%[email protected]~symlinks~termlib arch=linux-centos7-x86_64 ^[email protected]%[email protected] arch=linux-centos7-x86_64 ^[email protected]%[email protected] arch=linux-centos7-x86_64 ^[email protected]%[email protected] arch=linux-centos7-x86_64 ^rdma-core@20%[email protected] build_type=RelWithDebInfo arch=linux-centos7-x86_64 ^[email protected]%[email protected]~doc+ncurses+openssl+ownlibs patches=dd3a40d4d92f6b2158b87d6fb354c277947c776424aa03f6dc8096cf3135f5d0 ~qt arch=linux-centos7-x86_64 ^[email protected]%[email protected]+systemcerts arch=linux-centos7-x86_64 ^[email protected]%[email protected]+optimize+pic+shared arch=linux-centos7-x86_64 ^[email protected]%[email protected] arch=linux-centos7-x86_64 ^[email protected]%[email protected] arch=linux-centos7-x86_64 ^[email protected]%[email protected] arch=linux-centos7-x86_64 ^[email protected]%[email protected] arch=linux-centos7-x86_64 ^[email protected]%[email protected]+bzip2+curses+git~libunistring+libxml2 patches=9acdb4e73f67c241b5ef32505c9ddf7cf6884ca8ea661692f21dca28483b04b8 +tar+xz arch=linux-centos7-x86_64 ^[email protected]%[email protected]+shared arch=linux-centos7-x86_64 ^[email protected]%[email protected]~python arch=linux-centos7-x86_64 ^[email protected]%[email protected] arch=linux-centos7-x86_64 ^[email protected]%[email protected] arch=linux-centos7-x86_64 ^[email protected]%[email protected]+lex arch=linux-centos7-x86_64 Environment: [email protected]:pr.ucx.spack $ echo $HOSTNAME darwin-fe2.lanl.gov [email protected]:pr.ucx.spack $ grep -i 'model name' /proc/cpuinfo | sort | uniq model name : Intel(R) Xeon(R) CPU E5-2695 v3 @ 2.30GHz [email protected]:pr.ucx.spack $ lsb_release -d -bash: lsb_release: command not found [email protected]:pr.ucx.spack $ uname -a Linux darwin-fe2.lanl.gov 4.18.8-1.el7.elrepo.x86_64 #1 SMP Sat Sep 15 10:10:09 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux [email protected]:pr.ucx.spack $ date Mon Nov 19 10:47:47 MST 2018 [email protected]:pr.ucx.spack $ pwd /scratch/users/dantopa/new-spack/pr.ucx.spack Signed-off-by: Daniel Topa <[email protected]>
scheibelp
pushed a commit
that referenced
this pull request
Jun 24, 2020
Whenever attempting to use any ncurses functionality within cscope, a
page fault would result within the ncurses library.
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7fad3cf in termattrs_sp () from .../lib/libncursesw.so.6
(gdb) bt
#0 0x00007ffff7fad3cf in termattrs_sp () from .../lib/libncursesw.so.6
#1 0x00007ffff7faa794 in _nc_setupscreen_sp () from .../lib/libncursesw.so.6
#2 0x00007ffff7fa614c in newterm_sp () from .../lib/libncursesw.so.6
#3 0x00007ffff7fa65b9 in newterm () from .../lib/libncursesw.so.6
#4 0x00007ffff7fa2970 in initscr () from .../lib/libncursesw.so.6
#5 0x0000000000403dc2 in main (argc=<optimized out>, argv=0x7fffffffcea8) at main.c:574
This is due to a conflict between libtinfo.so and libtinfow.so. Both are
linked into cscope:
$ ldd $(which cscope)
/bin/bash: .../lib/libtinfo.so.6: no version information available (required by /bin/bash)
linux-vdso.so.1 (0x00007fff5dbcb000)
libncursesw.so.6 => .../lib/libncursesw.so.6 (0x00007f435cc69000)
libtinfo.so.6 => .../lib/libtinfo.so.6 (0x00007f435cc2c000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f435ca29000)
libtinfow.so.6 => .../lib/libtinfow.so.6 (0x00007f435c9e8000)
/lib64/ld-linux-x86-64.so.2 (0x00007f435cca7000)
Specifically linking libtinfow.so instead of libtinfo.so resolves the
issue.
All instances of '...' above represent the path to the installed ncurses
for Spack.
scheibelp
pushed a commit
that referenced
this pull request
Oct 24, 2020
The r-devtools package was not installable due to a few issues. - The rstudioapi spec was for 0.11.0 but the rstudioapi version is actually 0.11. This caused an error during concretization. - Set r-usethis to depend on [email protected]: rather than [email protected]. - Set r-usethis to depend on [email protected]: rather than [email protected]. - Added version r-gh-1.1.0 as it is not currently present in spack.
scheibelp
pushed a commit
that referenced
this pull request
Jun 1, 2021
Before: ``` $ hyperfine '~/spack/bin/spack -e . build-env rocfft' Benchmark #1: ~/spack/bin/spack -e . build-env rocfft Time (mean ± σ): 1.593 s ± 0.016 s [User: 1.468 s, System: 0.126 s] Range (min … max): 1.575 s … 1.628 s 10 runs ``` After: ``` $ hyperfine '~/spack/bin/spack -e . build-env rocfft' Benchmark #1: ~/spack/bin/spack -e . build-env rocfft Time (mean ± σ): 1.407 s ± 0.020 s [User: 1.280 s, System: 0.127 s] Range (min … max): 1.393 s … 1.455 s 10 runs ```
scheibelp
pushed a commit
that referenced
this pull request
Oct 15, 2021
Git 2.24 introduced a feature flag for repositories with many files, see: https://github.blog/2019-11-03-highlights-from-git-2-24/#feature-macros Since Spack's Git repository contains roughly 8,500 files, it can be worthwhile to enable this, especially on slow file systems such as NFS: ``` $ hyperfine --warmup 3 'cd spack-default; git status' 'cd spack-manyfiles; git status' Benchmark #1: cd spack-default; git status Time (mean ± σ): 3.388 s ± 0.095 s [User: 256.2 ms, System: 625.8 ms] Range (min … max): 3.168 s … 3.535 s 10 runs Benchmark #2: cd spack-manyfiles; git status Time (mean ± σ): 168.7 ms ± 10.9 ms [User: 98.6 ms, System: 126.1 ms] Range (min … max): 144.8 ms … 188.0 ms 19 runs Summary 'cd spack-manyfiles; git status' ran 20.09 ± 1.42 times faster than 'cd spack-default; git status' ```
scheibelp
pushed a commit
that referenced
this pull request
May 8, 2023
1. support version 3.1.3, which now depends on sundials@6 2. support version 3.1.2:, which broke the two patch files and therefore the two patch files have been replaced by more flexible filter_file() commands inside a patch() function. 3. rename the variant for python extension from using the package name "+pyuqtk" to the more standard "+python" 4. add maintainers @omsai and the upstream developer @bjdebus who offered to help with the spack packaging. 5. swig should only be a build-time dependency. swig is only necessary until @:3.1.0 6. confirmed python dependencies are correct by inspecting imports, subset python dependencies type to build, run, and confirmed all 31 build-time tests pass including the 9 python tests: ```console $ spack env create uqtk-dev $ spack add [email protected] $ spack install --test root && cat $(spack location -i uqtk)/.spack/install-time-test-log.txt ==> Testing package uqtk-3.1.3-nok6fut ==> [2023-04-19-14:56:25.005361] Running build-time tests ==> [2023-04-19-14:56:25.005536] RUN-TESTS: build-time tests [check] ==> [2023-04-19-14:56:25.009543] '/home/omsai/src/spack/opt/spack/linux-pureos10-skylake/gcc-10.2.1/gmake-4.4.1-b6g4apmfvxz3bn4eabh37dehcrg65fj7/bin/make' '-j4' '-n' 'test' ==> [2023-04-19-14:56:25.014903] '/home/omsai/src/spack/opt/spack/linux-pureos10-skylake/gcc-10.2.1/gmake-4.4.1-b6g4apmfvxz3bn4eabh37dehcrg65fj7/bin/make' '-j4' 'test' Running tests... /home/omsai/src/spack/opt/spack/linux-pureos10-skylake/gcc-10.2.1/cmake-3.26.3-zjmsfz23j5l4ytniz26uzvxonlu5qebr/bin/ctest --force-new-ctest-process Test project /tmp/omsai/spack-stage/spack-stage-uqtk-3.1.3-nok6fut47h42cnaau7wkoohgqy5f2qqa/spack-build-nok6fut Start 1: ArrayReadAndWrite Start 2: ArrayDelColumn Start 3: Array1DMiscTest Start 4: Array2DMiscTest 1/31 Test #1: ArrayReadAndWrite ................ Passed 0.01 sec Start 5: ArraySortTest 2/31 Test #2: ArrayDelColumn ................... Passed 0.01 sec Start 6: MultiIndexTest 3/31 Test #3: Array1DMiscTest .................. Passed 0.01 sec Start 7: CorrTest 4/31 Test #4: Array2DMiscTest .................. Passed 0.01 sec Start 8: QuadLUTest 5/31 Test #5: ArraySortTest .................... Passed 0.02 sec Start 9: MCMC2dTest 6/31 Test #6: MultiIndexTest ................... Passed 0.01 sec Start 10: MCMCRandomTest 7/31 Test #8: QuadLUTest ....................... Passed 0.02 sec Start 11: MCMCNestedTest 8/31 Test #10: MCMCRandomTest ................... Passed 0.02 sec Start 12: Deriv1dTest 9/31 Test #12: Deriv1dTest ...................... Passed 0.01 sec Start 13: SecondDeriv1dTest 10/31 Test #13: SecondDeriv1dTest ................ Passed 0.01 sec Start 14: GradHessianTest 11/31 Test #11: MCMCNestedTest ................... Passed 0.03 sec Start 15: GradientPCETest 12/31 Test #14: GradHessianTest .................. Passed 0.01 sec Start 16: PCE1dTest 13/31 Test #15: GradientPCETest .................. Passed 0.01 sec Start 17: PCEImplTest 14/31 Test #16: PCE1dTest ........................ Passed 0.01 sec Start 18: PCELogTest 15/31 Test #18: PCELogTest ....................... Passed 0.01 sec Start 19: Hessian2dTest 16/31 Test #19: Hessian2dTest .................... Passed 0.01 sec Start 20: BCS1dTest 17/31 Test #20: BCS1dTest ........................ Passed 0.01 sec Start 21: BCS2dTest 18/31 Test #21: BCS2dTest ........................ Passed 0.01 sec Start 22: LowRankRegrTest 19/31 Test #22: LowRankRegrTest .................. Passed 0.01 sec Start 23: PyModTest 20/31 Test #17: PCEImplTest ...................... Passed 0.07 sec Start 24: PyArrayTest 21/31 Test #23: PyModTest ........................ Passed 0.08 sec Start 25: PyArrayTest2 22/31 Test #25: PyArrayTest2 ..................... Passed 0.30 sec Start 26: PyQuadTest 23/31 Test #24: PyArrayTest ...................... Passed 1.44 sec Start 27: PyBCSTest1D 24/31 Test #26: PyQuadTest ....................... Passed 1.68 sec Start 28: PyBCSTest2D 25/31 Test #27: PyBCSTest1D ...................... Passed 1.66 sec Start 29: PyBADPTest 26/31 Test #7: CorrTest ......................... Passed 3.43 sec Start 30: PyRegressionTest 27/31 Test #28: PyBCSTest2D ...................... Passed 1.50 sec Start 31: PyGalerkinTest 28/31 Test #9: MCMC2dTest ....................... Passed 3.90 sec 29/31 Test #29: PyBADPTest ....................... Passed 1.66 sec 30/31 Test #30: PyRegressionTest ................. Passed 1.72 sec 31/31 Test #31: PyGalerkinTest ................... Passed 1.63 sec 100% tests passed, 0 tests failed out of 31 Total Test time (real) = 5.35 sec ==> [2023-04-19-14:56:30.382797] '/home/omsai/src/spack/opt/spack/linux-pureos10-skylake/gcc-10.2.1/gmake-4.4.1-b6g4apmfvxz3bn4eabh37dehcrg65fj7/bin/make' '-j4' '-n' 'check' ==> [2023-04-19-14:56:30.385983] Target 'check' not found in Makefile ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
@scheibelp
I ran out of inodes on our supercomputer, so I needed a way to uninstall obsolete Spack packages. Luckily, EVERYTHING I do is in a Spack Environment. So obsolete packages can be determined by looking for any packages NOT in one of my Spack Environments. The user interface is:
At that point, it works like any other
spack uninstallcommand. Mechanical garbage collection is one (more) advantage of using Spack Environments.I'm putting this as a separate PR because I'm really really really hoping Spack Environments will be merged Real Soon Now. I don't want to delay it any further with new features.