enhance archdetect to support detection of NVIDIA GPUs + using that in EESSI init script #767

boegel · 2024-09-30T16:31:02Z

Example output:

$ . init/bash
Found EESSI repo @ /cvmfs/software.eessi.io/versions/2023.06!
archdetect says x86_64/amd/zen3
archdetect found accelerator: accel/nvidia/cc80
Using x86_64/amd/zen3 as software subdirectory.
Found Lmod configuration file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/.lmod/lmodrc.lua
Found Lmod SitePackage.lua file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/.lmod/SitePackage.lua
Using /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/modules/all as the directory to be added to MODULEPATH.
Using /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen3/modules/all as the site extension directory to be added to MODULEPATH.
Using /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/modules/all as additional directory (for accelerators) to be added to MODULEPATH.
Found libcurl CAs file at RHEL location, setting CURL_CA_BUNDLE
Initializing Lmod...
Prepending /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/modules/all to $MODULEPATH...
Prepending site path /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen3/modules/all to $MODULEPATH...
Prepending /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/modules/all to $MODULEPATH...
Environment set up to use EESSI (2023.06), have fun!

{EESSI 2023.06} $ echo $MODULEPATH
/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/modules/all:/cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen3/modules/all:/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/modules/all

{EESSI 2023.06} $ ls -ld $(echo $MODULEPATH | cut -f1 -d:)
drwxr-xr-x 10 cvmfs cvmfs 4096 Sep 25 13:21 /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/modules/all

{EESSI 2023.06} $ module avail LAMMPS

---------- /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/modules/all ----------
   LAMMPS/2Aug2023_update2-foss-2023a-kokkos-CUDA-12.1.1 (g)

---------- /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/modules/all ----------
   LAMMPS/2Aug2023_update2-foss-2023a-kokkos    LAMMPS/29Aug2024-foss-2023b-kokkos (D)

…n EESSI init script

eessi-bot · 2024-09-30T16:31:06Z

Instance eessi-bot-mc-aws is configured to build for:

architectures: x86_64/generic, x86_64/intel/haswell, x86_64/intel/skylake_avx512, x86_64/amd/zen2, x86_64/amd/zen3, aarch64/generic, aarch64/neoverse_n1, aarch64/neoverse_v1
repositories: eessi.io-2023.06-compat, eessi-hpc.org-2023.06-software, eessi-hpc.org-2023.06-compat, eessi.io-2023.06-software

eessi-build-deploy-bot-deucalion · 2024-09-30T16:31:07Z

Instance boegel-bot-deucalion is configured to build for:

architectures: aarch64/a64fx
repositories: eessi.io-2023.06-software

eessi-bot · 2024-09-30T16:31:07Z

Instance eessi-bot-mc-azure is configured to build for:

architectures: x86_64/amd/zen4
repositories: eessi-hpc.org-2023.06-software, eessi-hpc.org-2023.06-compat, eessi.io-2023.06-software, eessi.io-2023.06-compat

init/eessi_environment_variables

ocaisa

I think we can give people the override capabilities to allow mix and match CPU/GPU stacks (in terms of the CPU architectures)

init/eessi_archdetect.sh

init/eessi_environment_variables

…GPU_SOFTWARE_SUBDIR_OVERRIDE Co-authored-by: ocaisa <[email protected]>

init/eessi_archdetect.sh

…ailed to run + take that into account in EESSI init script + allow overriding software subdirectory for accel/* via $EESSI_ACCEL_SOFTWARE_SUBDIR_OVERRIDE

…t, must be 'accel/nvidia/cc[0-9][0-9]'

…with 'No devices were found' if no GPUs are available in Slurm job

boegel · 2024-10-03T18:18:41Z

@ocaisa Don't merge this just yet (although it's ready for re-review + testing).

We should go all the way here, and also set up some CI for this, by using fake nvidia-smi scripts that produce known output & exit code + checking the result of both archdetect and init/bash...

boegel · 2024-10-03T19:45:17Z

Tested extensively:

CPU-only system (zen2), no `nvidia-smi`

$ init/eessi_archdetect.sh accelpath
$ echo $?
2

$ init/eessi_archdetect.sh -d accelpath
2024-10-03 20:00:33 [DEBUG] accelpath: Override variable set as ''
2024-10-03 20:00:33 [DEBUG] accelpath: nvidia-smi command not found
$ echo $?
2

$ . init/bash
Found EESSI repo @ /cvmfs/software.eessi.io/versions/2023.06!
archdetect says x86_64/amd/zen2
archdetect could not detect any accelerators
Using x86_64/amd/zen2 as software subdirectory.
Found Lmod configuration file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/.lmod/lmodrc.lua
Found Lmod SitePackage.lua file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/.lmod/SitePackage.lua
Using /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/modules/all as the directory to be added to MODULEPATH.
Using /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen2/modules/all as the site extension directory to be added to MODULEPATH.
Found libcurl CAs file at RHEL location, setting CURL_CA_BUNDLE
Initializing Lmod...
Prepending /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/modules/all to $MODULEPATH...
Prepending site path /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen2/modules/all to $MODULEPATH...
Environment set up to use EESSI (2023.06), have fun!

A100 GPU system (zen3), 1 GPU available

Without overrides:

$ init/eessi_archdetect.sh accelpath
accel/nvidia/cc80
$ echo $?
0

$ init/eessi_archdetect.sh -d accelpath
2024-10-03 21:28:05 [DEBUG] accelpath: Override variable set as ''
2024-10-03 21:28:05 [DEBUG] accelpath: nvidia-smi command found @ /usr/bin/nvidia-smi
2024-10-03 21:28:05 [DEBUG] accelpath: CUDA compute capability '80' derived from nvidia-smi output 'NVIDIA A100-SXM4-80GB, 1, 545.23.08, 8.0'
2024-10-03 21:28:05 [DEBUG] accelpath: result: accel/nvidia/cc80
accel/nvidia/cc80
$ echo $?
0

$ . init/bash
Found EESSI repo @ /cvmfs/software.eessi.io/versions/2023.06!
archdetect says x86_64/amd/zen3
archdetect found supported accelerator for CPU target x86_64/amd/zen3: accel/nvidia/cc80
Using x86_64/amd/zen3 as software subdirectory.
Found Lmod configuration file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/.lmod/lmodrc.lua
Found Lmod SitePackage.lua file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/.lmod/SitePackage.lua
Using /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/modules/all as the directory to be added to MODULEPATH.
Using /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen3/modules/all as the site extension directory to be added to MODULEPATH.
Using /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/modules/all as additional directory (for accelerators) to be added to MODULEPATH.
Found libcurl CAs file at RHEL location, setting CURL_CA_BUNDLE
Initializing Lmod...
Prepending /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/modules/all to $MODULEPATH...
Prepending site path /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen3/modules/all to $MODULEPATH...
Prepending /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/modules/all to $MODULEPATH...
Environment set up to use EESSI (2023.06), have fun!

With $EESSI_ACCELERATOR_TARGET_OVERRIDE set (unsupported GPU target):

$ export EESSI_ACCELERATOR_TARGET_OVERRIDE=accel/nvidia/cc12

$ init/eessi_archdetect.sh -d accelpath
2024-10-03 21:41:17 [DEBUG] accelpath: Override variable set as 'accel/nvidia/cc12'
accel/nvidia/cc12
$ echo $?
0

$ init/eessi_archdetect.sh accelpath
accel/nvidia/cc12
$ echo $?
0

$ export EESSI_ACCELERATOR_TARGET_OVERRIDE=accel/nvidia/cc12
$ . init/bash
Found EESSI repo @ /cvmfs/software.eessi.io/versions/2023.06!
archdetect says x86_64/amd/zen3
No matching path found in x86_64/amd/zen3 for accelerator detected by archdetect (accel/nvidia/cc12)
Using x86_64/amd/zen3 as software subdirectory.
Found Lmod configuration file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/.lmod/lmodrc.lua
Found Lmod SitePackage.lua file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/.lmod/SitePackage.lua
Using /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/modules/all as the directory to be added to MODULEPATH.
Using /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen3/modules/all as the site extension directory to be added to MODULEPATH.
Found libcurl CAs file at RHEL location, setting CURL_CA_BUNDLE
Initializing Lmod...
Prepending /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/modules/all to $MODULEPATH...
Prepending site path /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen3/modules/all to $MODULEPATH...
Environment set up to use EESSI (2023.06), have fun!

With $EESSI_ACCELERATOR_TARGET_OVERRIDE set (incorrectly):

$ export EESSI_ACCELERATOR_TARGET_OVERRIDE=accel/nvidia/cc123

$ init/eessi_archdetect.sh accelpath
2024-10-03 21:43:26 [ERROR] Value of $EESSI_ACCELERATOR_TARGET_OVERRIDE should match 'accel/nvidia/cc[0-9[0-9]', but it does not: 'accel/nvidia/cc123'
$ echo $?
1

$ init/eessi_archdetect.sh -d accelpath
2024-10-03 21:43:43 [DEBUG] accelpath: Override variable set as 'accel/nvidia/cc123'
2024-10-03 21:43:43 [ERROR] Value of $EESSI_ACCELERATOR_TARGET_OVERRIDE should match 'accel/nvidia/cc[0-9[0-9]', but it does not: 'accel/nvidia/cc123'
$ echo $?
1

$ . init/bash
Found EESSI repo @ /cvmfs/software.eessi.io/versions/2023.06!
archdetect says x86_64/amd/zen3
2024-10-03 21:44:15 [ERROR] Value of $EESSI_ACCELERATOR_TARGET_OVERRIDE should match 'accel/nvidia/cc[0-9[0-9]', but it does not: 'accel/nvidia/cc123'
archdetect could not detect any accelerators
Using x86_64/amd/zen3 as software subdirectory.
Found Lmod configuration file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/.lmod/lmodrc.lua
Found Lmod SitePackage.lua file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/.lmod/SitePackage.lua
Using /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/modules/all as the directory to be added to MODULEPATH.
Using /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen3/modules/all as the site extension directory to be added to MODULEPATH.
Found libcurl CAs file at RHEL location, setting CURL_CA_BUNDLE
Initializing Lmod...
Prepending /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/modules/all to $MODULEPATH...
Prepending site path /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen3/modules/all to $MODULEPATH...
Environment set up to use EESSI (2023.06), have fun!

With $EESSI_ACCEL_SOFTWARE_SUBDIR_OVERRIDE set:

export EESSI_ACCEL_SOFTWARE_SUBDIR_OVERRIDE=x86_64/amd/zen2
$ . init/bash
Found EESSI repo @ /cvmfs/software.eessi.io/versions/2023.06!
archdetect says x86_64/amd/zen3
archdetect found supported accelerator for CPU target x86_64/amd/zen2: accel/nvidia/cc80
Using x86_64/amd/zen3 as software subdirectory.
Found Lmod configuration file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/.lmod/lmodrc.lua
Found Lmod SitePackage.lua file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/.lmod/SitePackage.lua
Using /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/modules/all as the directory to be added to MODULEPATH.
Using /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen3/modules/all as the site extension directory to be added to MODULEPATH.
Using /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/modules/all as additional directory (for accelerators) to be added to MODULEPATH.
Found libcurl CAs file at RHEL location, setting CURL_CA_BUNDLE
Initializing Lmod...
Prepending /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/modules/all to $MODULEPATH...
Prepending site path /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen3/modules/all to $MODULEPATH...
Prepending /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/modules/all to $MODULEPATH...
Environment set up to use EESSI (2023.06), have fun!

A2 GPU system (Intel Cascade Lake), 1 GPU available

$ init/eessi_archdetect.sh accelpath
accel/nvidia/cc86
$ echo $?
0

$ init/eessi_archdetect.sh -d accelpath
2024-10-03 21:29:20 [DEBUG] accelpath: Override variable set as ''
2024-10-03 21:29:20 [DEBUG] accelpath: nvidia-smi command found @ /usr/bin/nvidia-smi
2024-10-03 21:29:20 [DEBUG] accelpath: CUDA compute capability '86' derived from nvidia-smi output 'NVIDIA A2, 1, 545.23.08, 8.6'
2024-10-03 21:29:20 [DEBUG] accelpath: result: accel/nvidia/cc86
accel/nvidia/cc86
$ echo $?
0

$ . init/bash
Found EESSI repo @ /cvmfs/software.eessi.io/versions/2023.06!
archdetect says x86_64/intel/skylake_avx512
No matching path found in x86_64/intel/skylake_avx512 for accelerator detected by archdetect (accel/nvidia/cc86)
Using x86_64/intel/skylake_avx512 as software subdirectory.
Found Lmod configuration file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/.lmod/lmodrc.lua
Found Lmod SitePackage.lua file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/.lmod/SitePackage.lua
Using /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/modules/all as the directory to be added to MODULEPATH.
Using /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/intel/skylake_avx512/modules/all as the site extension directory to be added to MODULEPATH.
Found libcurl CAs file at RHEL location, setting CURL_CA_BUNDLE
Initializing Lmod...
Prepending /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/modules/all to $MODULEPATH...
Prepending site path /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/intel/skylake_avx512/modules/all to $MODULEPATH...
Environment set up to use EESSI (2023.06), have fun!

V100 GPU system (Intel Cascade Lake), `nvidia-smi` available, but no GPU allocated in Slurm job

$ init/eessi_archdetect.sh accelpath
$ echo $?
3

$ init/eessi_archdetect.sh -d accelpath
2024-10-03 21:30:38 [DEBUG] accelpath: Override variable set as ''
2024-10-03 21:30:38 [DEBUG] accelpath: nvidia-smi command found @ /usr/bin/nvidia-smi
2024-10-03 21:30:38 [DEBUG] accelpath: nvidia-smi command failed, see output in /tmp/nvidia_smi_out.wRvfw
$ echo $?
3

$ cat /tmp/nvidia_smi_out.wRvfw
No devices were found

$ . init/bash
Found EESSI repo @ /cvmfs/software.eessi.io/versions/2023.06!
archdetect says x86_64/intel/skylake_avx512
archdetect could not detect any accelerators
Using x86_64/intel/skylake_avx512 as software subdirectory.
Found Lmod configuration file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/.lmod/lmodrc.lua
Found Lmod SitePackage.lua file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/.lmod/SitePackage.lua
Using /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/modules/all as the directory to be added to MODULEPATH.
Using /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/intel/skylake_avx512/modules/all as the site extension directory to be added to MODULEPATH.
Found libcurl CAs file at RHEL location, setting CURL_CA_BUNDLE
Initializing Lmod...
Prepending /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/modules/all to $MODULEPATH...
Prepending site path /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/intel/skylake_avx512/modules/all to $MODULEPATH...
Environment set up to use EESSI (2023.06), have fun!

boegel · 2024-10-06T11:53:37Z

@ocaisa I've added an extensive GitHub Actions workflow for verifying the NVIDIA GPU accelerator detection implemented in this PR, see 24f0620.

There's one issue though: implementing these tests revealed that the EESSI init script now "chokes" when both:

the errexit shell option is enabled (set -e), which it is by default in GitHub Actions and non uncommon in interactive shell session or job scripts;
no accelerator was detected by archdetect, which is a very common case (it's all that we supported up until very recently in EESSI);

For me, this is for enough reason to re-consider your (currently implemented) suggestion to let the accelpath function in archdetect produce a non-zero exit code when no accelerator is detected, and revert back to making it produce no output in that case...

Thoughts?

ocaisa · 2024-10-06T12:56:08Z

I would temporarily disable set -e when making the call to archdetect since we know it may return an error

ocaisa · 2024-10-06T12:57:07Z

Or give archdetect an environment variable or option that allows overriding returning an error code

…ct to detect accelerator

boegel · 2024-10-07T12:31:30Z

I would temporarily disable set -e when making the call to archdetect since we know it may return an error

@ocaisa done in 3bf3b12

.github/workflows/tests_archdetect_nvidia_gpu.yml

init/bash

init/eessi_environment_variables

boegel · 2024-10-09T07:32:39Z

@ocaisa Anything blocking this now?

ocaisa

LGTM

ocaisa · 2024-10-09T09:34:19Z

bot: build repo:eessi.io-2023.06-software arch:zen2

eessi-bot · 2024-10-09T09:34:22Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:zen2 from ocaisa
- expanded format: build repository:eessi.io-2023.06-software architecture:zen2
handling command build repository:eessi.io-2023.06-software architecture:zen2 resulted in:
- submitted job 22289, for details & status see enhance archdetect to support detection of NVIDIA GPUs + using that in EESSI init script #767 (comment)

eessi-bot · 2024-10-09T09:34:22Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build repo:eessi.io-2023.06-software arch:zen2 from ocaisa
- expanded format: build repository:eessi.io-2023.06-software architecture:zen2
handling command build repository:eessi.io-2023.06-software architecture:zen2 resulted in:
- no jobs were submitted

eessi-bot · 2024-10-09T09:34:26Z

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen2 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.10/pr_767/22289

date	job status	comment
Oct 09 09:34:25 UTC 2024	submitted	job id `22289` awaits release by job manager
Oct 09 09:34:32 UTC 2024	released	job awaits launch by Slurm scheduler
Oct 09 09:35:34 UTC 2024	running	job `22289` is running
Oct 09 09:41:41 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-22289.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1728466503.tar.gz` size: 0 MiB (5818 bytes) entries: 3 modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/amd/zen2/software no software packages in tarball other under 2023.06/software/linux/x86_64/amd/zen2 `2023.06/init/bash` `2023.06/init/eessi_archdetect.sh` `2023.06/init/eessi_environment_variables`
Oct 09 09:41:41 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 10/10 test case Details ✅ job output file `slurm-22289.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`
Oct 09 10:09:54 UTC 2024	uploaded	transfer of `eessi-2023.06-software-linux-x86_64-amd-zen2-1728466503.tar.gz` to S3 bucket succeeded

ocaisa · 2024-10-09T13:32:42Z

Staging PR merged, good to go!

eessi-bot · 2024-10-09T13:32:49Z

PR merged! Moved ['/project/def-users/SHARED/jobs/2024.10/pr_767/22289'] to /project/def-users/SHARED/trash_bin/EESSI/software-layer/2024.10.09

eessi-bot · 2024-10-09T13:32:49Z

PR merged! Moved [] to /project/def-users/SHARED/trash_bin/EESSI/software-layer/2024.10.09

enhance archdetect to support detection of NVIDIA GPUs + using that i…

67bed25

…n EESSI init script

boegel added 2023.06-software.eessi.io 2023.06 version of software.eessi.io accel:nvidia labels Sep 30, 2024

boegel requested a review from ocaisa September 30, 2024 16:31

ocaisa reviewed Sep 30, 2024

View reviewed changes

init/eessi_environment_variables Outdated Show resolved Hide resolved

ocaisa requested changes Oct 1, 2024

View reviewed changes

init/eessi_archdetect.sh Show resolved Hide resolved

init/eessi_archdetect.sh Outdated Show resolved Hide resolved

init/eessi_environment_variables Outdated Show resolved Hide resolved

init/eessi_environment_variables Outdated Show resolved Hide resolved

introduce $EESSI_GPU_SOFTWARE_PATH which can be overriden via $EESSI_…

6571035

…GPU_SOFTWARE_SUBDIR_OVERRIDE Co-authored-by: ocaisa <[email protected]>

ocaisa reviewed Oct 3, 2024

View reviewed changes

init/eessi_archdetect.sh Show resolved Hide resolved

boegel added 4 commits October 3, 2024 19:29

use non-zero exit code in archdetect if nvidia-smi was not found or f…

228eaa2

…ailed to run + take that into account in EESSI init script + allow overriding software subdirectory for accel/* via $EESSI_ACCEL_SOFTWARE_SUBDIR_OVERRIDE

sanity check value of $EESSI_ACCELERATOR_TARGET_OVERRIDE, if it is se…

f823375

…t, must be 'accel/nvidia/cc[0-9][0-9]'

take into account that nvidia-smi command may be available, but fail …

b04cc29

…with 'No devices were found' if no GPUs are available in Slurm job

only set $EESSI_MODULEPATH_ACCEL if directory exists

d9bca16

add CI workflow for testing NVIDIA GPU accelerator detection

24f0620

temporarily disable errexit shell option when calling out to archdete…

3bf3b12

…ct to detect accelerator

ocaisa reviewed Oct 7, 2024

View reviewed changes

.github/workflows/tests_archdetect_nvidia_gpu.yml Show resolved Hide resolved

init/bash Show resolved Hide resolved

init/eessi_environment_variables Show resolved Hide resolved

ocaisa reviewed Oct 7, 2024

View reviewed changes

init/eessi_environment_variables Show resolved Hide resolved

ocaisa approved these changes Oct 9, 2024

View reviewed changes

ocaisa added the bot:deploy Ask bot to deploy missing software installations to EESSI label Oct 9, 2024

ocaisa merged commit 37226d0 into EESSI:2023.06-software.eessi.io Oct 9, 2024
42 checks passed

boegel deleted the 2023.06-software.eessi.io_archdetect_nvidia branch October 9, 2024 16:29

boegel mentioned this pull request Oct 9, 2024

{2023.06}[foss/2023a] waLBerla 6.1 #779

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enhance archdetect to support detection of NVIDIA GPUs + using that in EESSI init script #767

enhance archdetect to support detection of NVIDIA GPUs + using that in EESSI init script #767

boegel commented Sep 30, 2024

eessi-bot bot commented Sep 30, 2024

eessi-build-deploy-bot-deucalion bot commented Sep 30, 2024

eessi-bot bot commented Sep 30, 2024

ocaisa left a comment

boegel commented Oct 3, 2024

boegel commented Oct 3, 2024

boegel commented Oct 6, 2024

ocaisa commented Oct 6, 2024

ocaisa commented Oct 6, 2024

boegel commented Oct 7, 2024

boegel commented Oct 9, 2024

ocaisa left a comment

ocaisa commented Oct 9, 2024

eessi-bot bot commented Oct 9, 2024 •

edited

Loading

eessi-bot bot commented Oct 9, 2024 •

edited

Loading

eessi-bot bot commented Oct 9, 2024 •

edited

Loading

ocaisa commented Oct 9, 2024

eessi-bot bot commented Oct 9, 2024

eessi-bot bot commented Oct 9, 2024

enhance archdetect to support detection of NVIDIA GPUs + using that in EESSI init script #767

enhance archdetect to support detection of NVIDIA GPUs + using that in EESSI init script #767

Conversation

boegel commented Sep 30, 2024

eessi-bot bot commented Sep 30, 2024

eessi-build-deploy-bot-deucalion bot commented Sep 30, 2024

eessi-bot bot commented Sep 30, 2024

ocaisa left a comment

Choose a reason for hiding this comment

boegel commented Oct 3, 2024

boegel commented Oct 3, 2024

CPU-only system (zen2), no nvidia-smi

A100 GPU system (zen3), 1 GPU available

A2 GPU system (Intel Cascade Lake), 1 GPU available

V100 GPU system (Intel Cascade Lake), nvidia-smi available, but no GPU allocated in Slurm job

boegel commented Oct 6, 2024

ocaisa commented Oct 6, 2024

ocaisa commented Oct 6, 2024

boegel commented Oct 7, 2024

boegel commented Oct 9, 2024

ocaisa left a comment

Choose a reason for hiding this comment

ocaisa commented Oct 9, 2024

eessi-bot bot commented Oct 9, 2024 • edited Loading

eessi-bot bot commented Oct 9, 2024 • edited Loading

eessi-bot bot commented Oct 9, 2024 • edited Loading

ocaisa commented Oct 9, 2024

eessi-bot bot commented Oct 9, 2024

eessi-bot bot commented Oct 9, 2024

CPU-only system (zen2), no `nvidia-smi`

V100 GPU system (Intel Cascade Lake), `nvidia-smi` available, but no GPU allocated in Slurm job

eessi-bot bot commented Oct 9, 2024 •

edited

Loading

eessi-bot bot commented Oct 9, 2024 •

edited

Loading

eessi-bot bot commented Oct 9, 2024 •

edited

Loading