{2023.06}[system] cuTENSOR v2.0.1.2 with CUDA/12.1.1 #367

trz42 · 2024-05-22T18:28:39Z

This PR attempts to add cuTENSOR/2.0.1.2 to NESSI and includes several improvements:

adds a generic script to install CUDA and cu* libraries under host_injections using the NESSI-extend module and an easystack file
creates a single Lmod hook for all CUDA and cu* libraries modules
uses a generic function that replaces non-distributable files with symlinks (used in post_sanitycheck_* EB hooks)
moves cuDNN from easystack file for foss/2023a to the one for system

This PR attempts to add cuTENSOR/2.0.1.2 to NESSI and includes several improvements: - adds a generic script to install CUDA and cu* libraries under host_injections using the NESSI-extend module and an easystack file - creates a single Lmod hook for all CUDA and cu* libraries modules - uses a generic function that replaces non-distributable files with symlinks (used in post_sanitycheck_* EB hooks) - moves cuDNN from easystack file for foss/2023a to the one for system

nessi-bot · 2024-05-22T18:28:43Z

Instance AWS-MC-NESSI is configured to build for:

architectures: x86_64/generic, x86_64/intel/skylake_avx512, x86_64/amd/zen2, aarch64/generic
repositories: nessi-2023.06-swl-deb11, nessi-2023.06-cl, nessi-2023.06-swl-deb10

nessi-bot · 2024-05-22T18:28:43Z

Instance Saga-NESSI is configured to build for:

architectures: x86_64/intel/skylake_avx512, x86_64/intel/broadwell, x86_64/generic
repositories: nessi-2023.06-cl, nessi-2023.06-swl-deb11, nessi-2023.06-swl-deb10

nessi-bot · 2024-05-22T18:28:43Z

Instance Fram-NESSI is configured to build for:

architectures: x86_64/generic, x86_64/intel/broadwell
repositories: nessi-2023.06-swl-deb10, nessi-2023.06-cl, nessi-2023.06-swl-deb11

nessi-bot · 2024-05-22T18:28:43Z

Instance eX3-NESSI is configured to build for:

architectures: x86_64/amd/zen2, aarch64/generic
repositories: nessi-2023.06-cl, nessi-2023.06-swl-deb11, nessi-2023.06-swl-deb10

trz42 · 2024-05-22T18:29:42Z

Just a first trial build...

bot: build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2

nessi-bot · 2024-05-22T18:29:44Z

Updates by the bot instance AWS-MC-NESSI (click for details)

received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2

nessi-bot · 2024-05-22T18:29:44Z

Updates by the bot instance eX3-NESSI (click for details)

received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2 resulted in:
- no jobs were submitted

nessi-bot · 2024-05-22T18:29:44Z

Updates by the bot instance Fram-NESSI (click for details)

received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2 resulted in:
- no jobs were submitted

nessi-bot · 2024-05-22T18:29:45Z

Updates by the bot instance Saga-NESSI (click for details)

received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2 resulted in:
- no jobs were submitted

nessi-bot · 2024-05-22T18:29:48Z

error: patch failed: easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.9.1-2023a.yml:34 error: easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.9.1-2023a.yml: patch does not apply Unable to download or merge changes between the source branch and the destination branch.Tip: This can usually be resolved by syncing your branch and resolving any merge conflicts.

…into nessi-2023.06-cuTENSOR-2.0.1.2-CUDA-12.1.1

trz42 · 2024-05-22T18:37:04Z

Just a first trial build (after fixing merge conflict)...

bot: build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2

nessi-bot · 2024-05-22T18:37:07Z

Updates by the bot instance eX3-NESSI (click for details)

received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2 resulted in:
- no jobs were submitted

nessi-bot · 2024-05-22T18:37:07Z

Updates by the bot instance Fram-NESSI (click for details)

received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2 resulted in:
- no jobs were submitted

nessi-bot · 2024-05-22T18:37:07Z

Updates by the bot instance AWS-MC-NESSI (click for details)

received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2 resulted in:
- submitted job 11222, for details & status see {2023.06}[system] cuTENSOR v2.0.1.2 with CUDA/12.1.1 #367 (comment)

nessi-bot · 2024-05-22T18:37:07Z

Updates by the bot instance Saga-NESSI (click for details)

received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2 resulted in:
- no jobs were submitted

nessi-bot · 2024-05-22T18:37:11Z

New job on instance AWS-MC-NESSI for architecture x86_64-amd-zen2 for repository nessi-2023.06-swl-deb11 in job dir /project/def-nessi/SHARED/jobs/2024.05/pr_367/11222

date	job status	comment
May 22 18:37:10 UTC 2024	submitted	job id `11222` awaits release by job manager
May 22 18:37:18 UTC 2024	released	job awaits launch by Slurm scheduler
May 22 18:42:25 UTC 2024	running	job `11222` is running
May 22 19:14:27 UTC 2024	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-11222.out` ❌ found message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1716403956.tar.gz` size: 1601 MiB (1678953570 bytes) entries: 54 modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all `cuTENSOR/2.0.1.2-CUDA-12.1.1.lua` software under 2023.06/software/linux/x86_64/amd/zen2/software `cuTENSOR/2.0.1.2-CUDA-12.1.1` other under 2023.06/software/linux/x86_64/amd/zen2 `2023.06/init/easybuild/eb_hooks.py` `2023.06/scripts/gpu_support/nvidia/eessi-2023.06-cuda-and-libraries.yml` `2023.06/scripts/gpu_support/nvidia/install_cuda_and_libraries.sh` `.lmod/SitePackage.lua`
May 22 19:14:27 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 7/7 test case(s) from 7 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-11222.out` ❌ found message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

trz42 · 2024-05-22T19:40:21Z

Next build should remove files we cannot distribute (plus it should not run into the problem creating a temp directory when installing CU* libraries under host_injections)...

bot: build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2

nessi-bot · 2024-05-22T19:40:24Z

Updates by the bot instance eX3-NESSI (click for details)

received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2 resulted in:
- no jobs were submitted

nessi-bot · 2024-05-22T19:40:24Z

Updates by the bot instance Fram-NESSI (click for details)

received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2 resulted in:
- no jobs were submitted

nessi-bot · 2024-05-22T19:40:24Z

Updates by the bot instance AWS-MC-NESSI (click for details)

received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2 resulted in:
- submitted job 11225, for details & status see {2023.06}[system] cuTENSOR v2.0.1.2 with CUDA/12.1.1 #367 (comment)

nessi-bot · 2024-05-22T19:40:24Z

Updates by the bot instance Saga-NESSI (click for details)

received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2 resulted in:
- no jobs were submitted

nessi-bot · 2024-05-22T19:40:32Z

New job on instance AWS-MC-NESSI for architecture x86_64-amd-zen2 for repository nessi-2023.06-swl-deb11 in job dir /project/def-nessi/SHARED/jobs/2024.05/pr_367/11225

date	job status	comment
May 22 19:40:31 UTC 2024	submitted	job id `11225` awaits release by job manager
May 22 19:40:56 UTC 2024	released	job awaits launch by Slurm scheduler
May 22 19:42:00 UTC 2024	running	job `11225` is running
May 22 20:16:22 UTC 2024	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-11225.out` ❌ found message matching `ERROR:` ✅ no message matching `FAILED:` ❌ found message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts No artefacts were created or found.
May 22 20:16:22 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 7/7 test case(s) from 7 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-11225.out` ❌ found message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

trz42 · 2024-05-22T20:35:58Z

Next build should remove files we cannot distribute (should fix the argument issue for the replace... function)...

bot: build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2

nessi-bot · 2024-05-22T20:36:01Z

Updates by the bot instance AWS-MC-NESSI (click for details)

received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2 resulted in:
- submitted job 11226, for details & status see {2023.06}[system] cuTENSOR v2.0.1.2 with CUDA/12.1.1 #367 (comment)

nessi-bot · 2024-05-22T20:36:01Z

Updates by the bot instance eX3-NESSI (click for details)

received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2 resulted in:
- no jobs were submitted

nessi-bot · 2024-05-22T20:36:01Z

Updates by the bot instance Fram-NESSI (click for details)

received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2 resulted in:
- no jobs were submitted

nessi-bot · 2024-05-22T20:36:01Z

Updates by the bot instance Saga-NESSI (click for details)

received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2 resulted in:
- no jobs were submitted

nessi-bot · 2024-05-22T20:36:05Z

New job on instance AWS-MC-NESSI for architecture x86_64-amd-zen2 for repository nessi-2023.06-swl-deb11 in job dir /project/def-nessi/SHARED/jobs/2024.05/pr_367/11226

date	job status	comment
May 22 20:36:04 UTC 2024	submitted	job id `11226` awaits release by job manager
May 22 20:36:45 UTC 2024	released	job awaits launch by Slurm scheduler
May 22 20:37:47 UTC 2024	running	job `11226` is running
May 22 21:11:21 UTC 2024	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-11226.out` ❌ found message matching `ERROR:` ❌ found message matching `FAILED:` ❌ found message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts No artefacts were created or found.
May 22 21:11:21 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 7/7 test case(s) from 7 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-11226.out` ❌ found message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

trz42 · 2024-05-22T20:53:06Z

Next build should remove files we cannot distribute (should fix silly logic mistake in the replace... function)...

bot: build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2

nessi-bot · 2024-05-22T20:53:08Z

Updates by the bot instance AWS-MC-NESSI (click for details)

received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2 resulted in:
- submitted job 11227, for details & status see {2023.06}[system] cuTENSOR v2.0.1.2 with CUDA/12.1.1 #367 (comment)

nessi-bot · 2024-05-22T20:53:08Z

Updates by the bot instance eX3-NESSI (click for details)

received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2 resulted in:
- no jobs were submitted

nessi-bot · 2024-05-22T20:53:09Z

Updates by the bot instance Fram-NESSI (click for details)

received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2 resulted in:
- no jobs were submitted

nessi-bot · 2024-05-22T20:53:09Z

Updates by the bot instance Saga-NESSI (click for details)

received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2 resulted in:
- no jobs were submitted

nessi-bot · 2024-05-22T20:53:12Z

New job on instance AWS-MC-NESSI for architecture x86_64-amd-zen2 for repository nessi-2023.06-swl-deb11 in job dir /project/def-nessi/SHARED/jobs/2024.05/pr_367/11227

date	job status	comment
May 22 20:53:12 UTC 2024	submitted	job id `11227` awaits release by job manager
May 22 20:53:20 UTC 2024	released	job awaits launch by Slurm scheduler
May 22 20:54:25 UTC 2024	running	job `11227` is running
May 22 21:31:11 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-11227.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1716412115.tar.gz` size: 1175 MiB (1233032053 bytes) entries: 54 modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all `cuTENSOR/2.0.1.2-CUDA-12.1.1.lua` software under 2023.06/software/linux/x86_64/amd/zen2/software `cuTENSOR/2.0.1.2-CUDA-12.1.1` other under 2023.06/software/linux/x86_64/amd/zen2 `2023.06/init/easybuild/eb_hooks.py` `2023.06/scripts/gpu_support/nvidia/eessi-2023.06-cuda-and-libraries.yml` `2023.06/scripts/gpu_support/nvidia/install_cuda_and_libraries.sh` `.lmod/SitePackage.lua`
May 22 21:31:11 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 7/7 test case(s) from 7 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-11227.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`
May 23 03:49:55 UTC 2024	uploaded	transfer of `eessi-2023.06-software-linux-x86_64-amd-zen2-1716412115.tar.gz` to S3 bucket succeeded
May 23 03:59:09 AM UTC 2024	staged	tarball `eessi-2023.06-software-linux-x86_64-amd-zen2-1716412115.tar.gz` downloaded to Stratum-0
May 23 04:00:34 AM UTC 2024	pr_opened	merge PR https://github.com/NorESSI/2024-staging/pull/352 to approve ingest
May 23 04:11:59 AM UTC 2024	approved	👍 tarball `eessi-2023.06-software-linux-x86_64-amd-zen2-1716412115.tar.gz` approved, see PR https://github.com/NorESSI/2024-staging/pull/352
May 23 04:20:13 AM UTC 2024	ingested	🎉 tarball `eessi-2023.06-software-linux-x86_64-amd-zen2-1716412115.tar.gz` successfully ingested at `2023.06/`

trz42 · 2024-05-22T21:10:06Z

Looks good. Let's build the other guys...

bot: build inst:Saga-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/broadwell
bot: build inst:Saga-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/skylake
bot: build inst:eX3-NESSI repo:nessi-2023.06-swl-deb10 arch:aarch64/generic
bot: build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/generic

nessi-bot · 2024-05-22T21:10:10Z

Updates by the bot instance eX3-NESSI (click for details)

received bot command build inst:Saga-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/broadwell from trz42
- expanded format: build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell
received bot command build inst:Saga-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/skylake from trz42
- expanded format: build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/skylake
received bot command build inst:eX3-NESSI repo:nessi-2023.06-swl-deb10 arch:aarch64/generic from trz42
- expanded format: build instance:eX3-NESSI repository:nessi-2023.06-swl-deb10 architecture:aarch64/generic
received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/generic from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/generic
handling command build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell resulted in:
- no jobs were submitted
handling command build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/skylake resulted in:
- no jobs were submitted
handling command build instance:eX3-NESSI repository:nessi-2023.06-swl-deb10 architecture:aarch64/generic resulted in:
- submitted job 203985, for details & status see {2023.06}[system] cuTENSOR v2.0.1.2 with CUDA/12.1.1 #367 (comment)
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/generic resulted in:
- no jobs were submitted

nessi-bot · 2024-05-22T21:10:10Z

Updates by the bot instance Saga-NESSI (click for details)

received bot command build inst:Saga-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/broadwell from trz42
- expanded format: build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell
received bot command build inst:Saga-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/skylake from trz42
- expanded format: build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/skylake
received bot command build inst:eX3-NESSI repo:nessi-2023.06-swl-deb10 arch:aarch64/generic from trz42
- expanded format: build instance:eX3-NESSI repository:nessi-2023.06-swl-deb10 architecture:aarch64/generic
received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/generic from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/generic
handling command build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell resulted in:
- submitted job 11641053, for details & status see {2023.06}[system] cuTENSOR v2.0.1.2 with CUDA/12.1.1 #367 (comment)
handling command build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/skylake resulted in:
- submitted job 11641054, for details & status see {2023.06}[system] cuTENSOR v2.0.1.2 with CUDA/12.1.1 #367 (comment)
handling command build instance:eX3-NESSI repository:nessi-2023.06-swl-deb10 architecture:aarch64/generic resulted in:
- no jobs were submitted
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/generic resulted in:
- no jobs were submitted

nessi-bot · 2024-05-22T21:10:10Z

Updates by the bot instance Fram-NESSI (click for details)

received bot command build inst:Saga-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/broadwell from trz42
- expanded format: build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell
received bot command build inst:Saga-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/skylake from trz42
- expanded format: build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/skylake
received bot command build inst:eX3-NESSI repo:nessi-2023.06-swl-deb10 arch:aarch64/generic from trz42
- expanded format: build instance:eX3-NESSI repository:nessi-2023.06-swl-deb10 architecture:aarch64/generic
received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/generic from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/generic
handling command build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell resulted in:
- no jobs were submitted
handling command build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/skylake resulted in:
- no jobs were submitted
handling command build instance:eX3-NESSI repository:nessi-2023.06-swl-deb10 architecture:aarch64/generic resulted in:
- no jobs were submitted
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/generic resulted in:
- no jobs were submitted

nessi-bot · 2024-05-22T21:10:10Z

Updates by the bot instance AWS-MC-NESSI (click for details)

received bot command build inst:Saga-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/broadwell from trz42
- expanded format: build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell
received bot command build inst:Saga-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/skylake from trz42
- expanded format: build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/skylake
received bot command build inst:eX3-NESSI repo:nessi-2023.06-swl-deb10 arch:aarch64/generic from trz42
- expanded format: build instance:eX3-NESSI repository:nessi-2023.06-swl-deb10 architecture:aarch64/generic
received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/generic from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/generic
handling command build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell resulted in:
- no jobs were submitted
handling command build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/skylake resulted in:
- no jobs were submitted
handling command build instance:eX3-NESSI repository:nessi-2023.06-swl-deb10 architecture:aarch64/generic resulted in:
- no jobs were submitted
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/generic resulted in:
- submitted job 11228, for details & status see {2023.06}[system] cuTENSOR v2.0.1.2 with CUDA/12.1.1 #367 (comment)

nessi-bot · 2024-05-22T21:10:15Z

New job on instance eX3-NESSI for architecture aarch64-generic for repository nessi-2023.06-swl-deb10 in job dir /home/thomarob/pilot.nessi.no/jobs/2024.05/pr_367/203985

date	job status	comment
May 22 09:10:14 PM UTC 2024	submitted	job id `203985` awaits release by job manager
May 22 09:10:23 PM UTC 2024	released	job awaits launch by Slurm scheduler
May 22 09:11:26 PM UTC 2024	running	job `203985` is running
May 22 10:00:39 PM UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-203985.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-aarch64-generic-1716413108.tar.gz` size: 1175 MiB (1233033527 bytes) entries: 54 modules under 2023.06/software/linux/aarch64/generic/modules/all `cuTENSOR/2.0.1.2-CUDA-12.1.1.lua` software under 2023.06/software/linux/aarch64/generic/software `cuTENSOR/2.0.1.2-CUDA-12.1.1` other under 2023.06/software/linux/aarch64/generic `2023.06/init/easybuild/eb_hooks.py` `2023.06/scripts/gpu_support/nvidia/eessi-2023.06-cuda-and-libraries.yml` `2023.06/scripts/gpu_support/nvidia/install_cuda_and_libraries.sh` `.lmod/SitePackage.lua`
May 22 10:00:39 PM UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 7/7 test case(s) from 7 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-203985.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`
May 23 03:49:51 AM UTC 2024	uploaded	transfer of `eessi-2023.06-software-linux-aarch64-generic-1716413108.tar.gz` to S3 bucket succeeded
May 23 03:59:01 AM UTC 2024	staged	tarball `eessi-2023.06-software-linux-aarch64-generic-1716413108.tar.gz` downloaded to Stratum-0
May 23 04:00:05 AM UTC 2024	pr_opened	merge PR https://github.com/NorESSI/2024-staging/pull/351 to approve ingest
May 23 04:11:56 AM UTC 2024	approved	👍 tarball `eessi-2023.06-software-linux-aarch64-generic-1716413108.tar.gz` approved, see PR https://github.com/NorESSI/2024-staging/pull/351
May 23 04:16:06 AM UTC 2024	ingested	🎉 tarball `eessi-2023.06-software-linux-aarch64-generic-1716413108.tar.gz` successfully ingested at `2023.06/`

nessi-bot · 2024-05-22T21:10:22Z

New job on instance Saga-NESSI for architecture x86_64-intel-broadwell for repository nessi-2023.06-swl-deb11 in job dir /cluster/projects/nn9992k/pilot.nessi.no/jobs/2024.05/pr_367/11641053

date	job status	comment
May 22 09:10:21 PM UTC 2024	submitted	job id `11641053` awaits release by job manager
May 22 09:10:31 PM UTC 2024	released	job awaits launch by Slurm scheduler
May 22 09:11:38 PM UTC 2024	running	job `11641053` is running
May 22 10:29:49 PM UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-11641053.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-intel-broadwell-1716413500.tar.gz` size: 1175 MiB (1233033835 bytes) entries: 54 modules under 2023.06/software/linux/x86_64/intel/broadwell/modules/all `cuTENSOR/2.0.1.2-CUDA-12.1.1.lua` software under 2023.06/software/linux/x86_64/intel/broadwell/software `cuTENSOR/2.0.1.2-CUDA-12.1.1` other under 2023.06/software/linux/x86_64/intel/broadwell `2023.06/init/easybuild/eb_hooks.py` `2023.06/scripts/gpu_support/nvidia/eessi-2023.06-cuda-and-libraries.yml` `2023.06/scripts/gpu_support/nvidia/install_cuda_and_libraries.sh` `.lmod/SitePackage.lua`
May 22 10:29:49 PM UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 7/7 test case(s) from 7 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-11641053.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`
May 23 03:49:59 AM UTC 2024	uploaded	transfer of `eessi-2023.06-software-linux-x86_64-intel-broadwell-1716413500.tar.gz` to S3 bucket succeeded
May 23 03:59:27 AM UTC 2024	staged	tarball `eessi-2023.06-software-linux-x86_64-intel-broadwell-1716413500.tar.gz` downloaded to Stratum-0
May 23 04:01:39 AM UTC 2024	pr_opened	merge PR https://github.com/NorESSI/2024-staging/pull/354 to approve ingest
May 23 04:12:05 AM UTC 2024	approved	👍 tarball `eessi-2023.06-software-linux-x86_64-intel-broadwell-1716413500.tar.gz` approved, see PR https://github.com/NorESSI/2024-staging/pull/354
May 23 04:28:58 AM UTC 2024	ingested	🎉 tarball `eessi-2023.06-software-linux-x86_64-intel-broadwell-1716413500.tar.gz` successfully ingested at `2023.06/`

nessi-bot · 2024-05-22T21:10:23Z

New job on instance AWS-MC-NESSI for architecture x86_64-generic for repository nessi-2023.06-swl-deb11 in job dir /project/def-nessi/SHARED/jobs/2024.05/pr_367/11228

date	job status	comment
May 22 21:10:22 UTC 2024	submitted	job id `11228` awaits release by job manager
May 22 21:11:19 UTC 2024	released	job awaits launch by Slurm scheduler
May 22 21:12:26 UTC 2024	running	job `11228` is running
May 22 21:54:43 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-11228.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-generic-1716413238.tar.gz` size: 1175 MiB (1233033503 bytes) entries: 54 modules under 2023.06/software/linux/x86_64/generic/modules/all `cuTENSOR/2.0.1.2-CUDA-12.1.1.lua` software under 2023.06/software/linux/x86_64/generic/software `cuTENSOR/2.0.1.2-CUDA-12.1.1` other under 2023.06/software/linux/x86_64/generic `2023.06/init/easybuild/eb_hooks.py` `2023.06/scripts/gpu_support/nvidia/eessi-2023.06-cuda-and-libraries.yml` `2023.06/scripts/gpu_support/nvidia/install_cuda_and_libraries.sh` `.lmod/SitePackage.lua`
May 22 21:54:43 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 7/7 test case(s) from 7 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-11228.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`
May 23 03:50:48 UTC 2024	uploaded	transfer of `eessi-2023.06-software-linux-x86_64-generic-1716413238.tar.gz` to S3 bucket succeeded
May 23 03:59:18 AM UTC 2024	staged	tarball `eessi-2023.06-software-linux-x86_64-generic-1716413238.tar.gz` downloaded to Stratum-0
May 23 04:01:04 AM UTC 2024	pr_opened	merge PR https://github.com/NorESSI/2024-staging/pull/353 to approve ingest
May 23 04:12:02 AM UTC 2024	approved	👍 tarball `eessi-2023.06-software-linux-x86_64-generic-1716413238.tar.gz` approved, see PR https://github.com/NorESSI/2024-staging/pull/353
May 23 04:24:58 AM UTC 2024	ingested	🎉 tarball `eessi-2023.06-software-linux-x86_64-generic-1716413238.tar.gz` successfully ingested at `2023.06/`

nessi-bot · 2024-05-22T21:10:33Z

New job on instance Saga-NESSI for architecture x86_64-intel-skylake_avx512 for repository nessi-2023.06-swl-deb11 in job dir /cluster/projects/nn9992k/pilot.nessi.no/jobs/2024.05/pr_367/11641054

date	job status	comment
May 22 09:10:32 PM UTC 2024	submitted	job id `11641054` awaits release by job manager
May 22 09:11:35 PM UTC 2024	released	job awaits launch by Slurm scheduler
May 22 09:12:44 PM UTC 2024	running	job `11641054` is running
May 22 09:56:56 PM UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-11641054.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-intel-skylake_avx512-1716413263.tar.gz` size: 1175 MiB (1233033142 bytes) entries: 54 modules under 2023.06/software/linux/x86_64/intel/skylake_avx512/modules/all `cuTENSOR/2.0.1.2-CUDA-12.1.1.lua` software under 2023.06/software/linux/x86_64/intel/skylake_avx512/software `cuTENSOR/2.0.1.2-CUDA-12.1.1` other under 2023.06/software/linux/x86_64/intel/skylake_avx512 `2023.06/init/easybuild/eb_hooks.py` `2023.06/scripts/gpu_support/nvidia/eessi-2023.06-cuda-and-libraries.yml` `2023.06/scripts/gpu_support/nvidia/install_cuda_and_libraries.sh` `.lmod/SitePackage.lua`
May 22 09:56:56 PM UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 7/7 test case(s) from 7 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-11641054.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`
May 23 03:50:55 AM UTC 2024	uploaded	transfer of `eessi-2023.06-software-linux-x86_64-intel-skylake_avx512-1716413263.tar.gz` to S3 bucket succeeded
May 23 03:59:35 AM UTC 2024	staged	tarball `eessi-2023.06-software-linux-x86_64-intel-skylake_avx512-1716413263.tar.gz` downloaded to Stratum-0
May 23 04:02:10 AM UTC 2024	pr_opened	merge PR https://github.com/NorESSI/2024-staging/pull/355 to approve ingest
May 23 04:12:08 AM UTC 2024	approved	👍 tarball `eessi-2023.06-software-linux-x86_64-intel-skylake_avx512-1716413263.tar.gz` approved, see PR https://github.com/NorESSI/2024-staging/pull/355
May 23 04:33:09 AM UTC 2024	ingested	🎉 tarball `eessi-2023.06-software-linux-x86_64-intel-skylake_avx512-1716413263.tar.gz` successfully ingested at `2023.06/`

trz42 · 2024-05-23T03:48:32Z

Checklist before starting deployment (setting bot:deploy label):

Check if the SPDX license identifier is provided
- NVIDIA license imposing restrictions on what can be distributed... (see updated post_sanitycheck_cutensor hook)
Check whether builds for all required architectures succeed (SUCCESS message + reasonably sized tarball)
Check if the PR is up-to-date with the target branch nessi.no-2023.06 in the repository (if not what are the differences)
Assess if all requested changes are sound (checking files changed on GitHub.com)
Verify that all easyconfig/s being built are included with the EB version used (if not why not)
Review changes (if any) needed to get the build(s) succeed (common changes for all architectures, changes for a single architecture, changes because of build environment specifics, etc.)

trz42 · 2024-05-23T03:50:21Z

Target architectures

x86_64-intel-broadwell
x86_64-intel-skylake_avx512
x86_64-amd-zen2
x86_64-generic
aarch64-generic

Checklist for deployment/ingestion

at least one tarball for each architecture has been uploaded
at least one tarball for each architecture has been staged
for each staged tarball was a PR for approval/rejection opened
for each architecture one and only one tarball has been approved
for each architecture one and only one tarball was ingested
all tests in the CI have succeeded (rerun failed tests ~ 10 mins after last ingest)
has the package become available via CernVM-FS for all architectures ingested (for lmod cache updates pay attention to timestamp of files)

command & log

command

BASE_DIR=/cvmfs/pilot.nessi.no/versions/2023.06/software/linux \
ARCHS=() \
ARCHS+=("aarch64/generic") ; \
ARCHS+=("x86_64/generic") ; \
ARCHS+=("x86_64/amd/zen2") ; \
ARCHS+=("x86_64/intel/broadwell") ; \
ARCHS+=("x86_64/intel/skylake_avx512") ; \
for arch in "${ARCHS[@]}"; do \
ls -lR \
${BASE_DIR}/${arch}/{software,modules/all}/cuTENSOR/2.0.1.2-CUDA-12.1.1* \
${BASE_DIR}/${arch}/.lmod/SitePackage.lua ; \
done ; \
ls -l \
${BASE_DIR}/../../init/easybuild/eb_hooks.py \
${BASE_DIR}/../../scripts/gpu_support/nvidia/eessi-2023.06-cuda-and-libraries.yml \
${BASE_DIR}/../../scripts/gpu_support/nvidia/install_cuda_and_libraries.sh

log - BEFORE ingestion

ls: cannot access '/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/aarch64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1*': No such file or directory
ls: cannot access '/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/aarch64/generic/modules/all/cuTENSOR/2.0.1.2-CUDA-12.1.1*': No such file or directory
-rw-rw-r-- 1 cvmfs cvmfs 10429 May 16 23:25  /cvmfs/pilot.nessi.no/versions/2023.06/software/linux/aarch64/generic/.lmod/SitePackage.lua
ls: cannot access '/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1*': No such file or directory
ls: cannot access '/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/generic/modules/all/cuTENSOR/2.0.1.2-CUDA-12.1.1*': No such file or directory
-rw-rw-r-- 1 cvmfs cvmfs 10429 May 16 20:48  /cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/generic/.lmod/SitePackage.lua
ls: cannot access '/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/amd/zen2/software/cuTENSOR/2.0.1.2-CUDA-12.1.1*': No such file or directory
ls: cannot access '/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/amd/zen2/modules/all/cuTENSOR/2.0.1.2-CUDA-12.1.1*': No such file or directory
-rw-rw-r-- 1 cvmfs cvmfs 10429 May 16 20:04  /cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/amd/zen2/.lmod/SitePackage.lua
ls: cannot access '/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/broadwell/software/cuTENSOR/2.0.1.2-CUDA-12.1.1*': No such file or directory
ls: cannot access '/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/broadwell/modules/all/cuTENSOR/2.0.1.2-CUDA-12.1.1*': No such file or directory
-rw-rw-r-- 1 cvmfs cvmfs 10429 May 17 01:08  /cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/broadwell/.lmod/SitePackage.lua
ls: cannot access '/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/cuTENSOR/2.0.1.2-CUDA-12.1.1*': No such file or directory
ls: cannot access '/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/modules/all/cuTENSOR/2.0.1.2-CUDA-12.1.1*': No such file or directory
-rw-rw-r-- 1 cvmfs cvmfs 10429 May 16 20:50  /cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/.lmod/SitePackage.lua
ls: cannot access '/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/../../scripts/gpu_support/nvidia/eessi-2023.06-cuda-and-libraries.yml': No such file or directory
ls: cannot access '/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/../../scripts/gpu_support/nvidia/install_cuda_and_libraries.sh': No such file or directory
-rw-rw-r-- 1 cvmfs cvmfs 42654 May 21 23:28 /cvmfs/pilot.nessi.no/versions/2023.06/software/linux/../../init/easybuild/eb_hooks.py

log - AFTER ingestion

-rw-rw-r--. 1 cvmfs cvmfs 8836 May 22 21:25 /cvmfs/pilot.nessi.no/versions/2023.06/software/linux/aarch64/generic/.lmod/SitePackage.lua
-rw-r--r--. 1 cvmfs cvmfs 1179 May 22 21:21 /cvmfs/pilot.nessi.no/versions/2023.06/software/linux/aarch64/generic/modules/all/cuTENSOR/2.0.1.2-CUDA-12.1.1.lua

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/aarch64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1:
total 32
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May 22 21:21 easybuild
dr-xr-xr-x. 3 cvmfs cvmfs  4096 Jan 19 13:34 include
dr-xr-xr-x. 5 cvmfs cvmfs  4096 Jan 19 13:34 lib
lrwxrwxrwx. 1 cvmfs cvmfs     3 May 22 21:21 lib64 -> lib
-r--r--r--. 1 cvmfs cvmfs 17789 Jan 19 12:39 LICENSE

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/aarch64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/easybuild:
total 55
-r--r--r--. 1 cvmfs cvmfs  6018 May 22 21:21 cuTENSOR-2.0.1.2-CUDA-12.1.1-easybuild-devel
-r--r--r--. 1 cvmfs cvmfs  1088 Apr 10 20:16 cuTENSOR-2.0.1.2-CUDA-12.1.1.eb
-r--r--r--. 1 cvmfs cvmfs 25187 May 22 21:21 easybuild-cuTENSOR-2.0.1.2-20240522.232116.log.bz2
-rw-r--r--. 1 cvmfs cvmfs 18201 May 22 21:21 easybuild-cuTENSOR-2.0.1.2-20240522.232116_test_report.md
dr-xr-xr-x. 4 cvmfs cvmfs  4096 May 22 21:21 reprod

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/aarch64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/easybuild/reprod:
total 11
-r--r--r--. 1 cvmfs cvmfs 1044 May 22 21:20 cuTENSOR-2.0.1.2-CUDA-12.1.1.eb
-r--r--r--. 1 cvmfs cvmfs  276 May 22 21:21 cuTENSOR-2.0.1.2-CUDA-12.1.1.env
dr-xr-xr-x. 2 cvmfs cvmfs 4096 May 22 21:20 easyblocks
dr-xr-xr-x. 2 cvmfs cvmfs 4096 May 22 21:20 hooks

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/aarch64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/easybuild/reprod/easyblocks:
total 6
-r--r--r--. 1 cvmfs cvmfs 5655 Apr 10 20:15 tarball.py

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/aarch64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/easybuild/reprod/hooks:
total 43
-r--r--r--. 1 cvmfs cvmfs 43825 May 22 21:10 eb_hooks.py

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/aarch64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/include:
total 103
dr-xr-xr-x. 2 cvmfs cvmfs  4096 Jan 19 13:25 cutensor
-r--r--r--. 1 cvmfs cvmfs 69855 Jan 19 12:39 cutensor.h
-r--r--r--. 1 cvmfs cvmfs 30584 Jan 19 12:39 cutensorMg.h

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/aarch64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/include/cutensor:
total 15
-r--r--r--. 1 cvmfs cvmfs 15283 Jan 19 13:21 types.h

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/aarch64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib:
total 14
dr-xr-xr-x. 2 cvmfs cvmfs 4096 May 22 21:21 11
dr-xr-xr-x. 2 cvmfs cvmfs 4096 May 22 21:21 11.0
dr-xr-xr-x. 2 cvmfs cvmfs 4096 May 22 21:21 12

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/aarch64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11:
total 655992
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:02 libcutensorMg.so
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:02 libcutensorMg.so.2
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:02 libcutensorMg.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       144 May 22 21:21 libcutensorMg_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/aarch64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11/libcutensorMg_static.a
-r-xr-xr-x. 1 cvmfs cvmfs 216839664 Jan 19 13:02 libcutensor.so
-r-xr-xr-x. 1 cvmfs cvmfs 216839664 Jan 19 13:02 libcutensor.so.2
-r-xr-xr-x. 1 cvmfs cvmfs 216839664 Jan 19 13:02 libcutensor.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       142 May 22 21:21 libcutensor_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/aarch64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11/libcutensor_static.a

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/aarch64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11.0:
total 515209
-r-xr-xr-x. 1 cvmfs cvmfs   6887248 Jan 19 12:48 libcutensorMg.so
-r-xr-xr-x. 1 cvmfs cvmfs   6887248 Jan 19 12:48 libcutensorMg.so.2
-r-xr-xr-x. 1 cvmfs cvmfs   6887248 Jan 19 12:48 libcutensorMg.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       146 May 22 21:21 libcutensorMg_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/aarch64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11.0/libcutensorMg_static.a
-r-xr-xr-x. 1 cvmfs cvmfs 168969928 Jan 19 12:48 libcutensor.so
-r-xr-xr-x. 1 cvmfs cvmfs 168969928 Jan 19 12:48 libcutensor.so.2
-r-xr-xr-x. 1 cvmfs cvmfs 168969928 Jan 19 12:48 libcutensor.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       144 May 22 21:21 libcutensor_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/aarch64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11.0/libcutensor_static.a

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/aarch64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/12:
total 708514
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:23 libcutensorMg.so
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:23 libcutensorMg.so.2
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:23 libcutensorMg.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       144 May 22 21:21 libcutensorMg_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/aarch64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/12/libcutensorMg_static.a
-r-xr-xr-x. 1 cvmfs cvmfs 234767248 Jan 19 13:23 libcutensor.so
-r-xr-xr-x. 1 cvmfs cvmfs 234767248 Jan 19 13:23 libcutensor.so.2
-r-xr-xr-x. 1 cvmfs cvmfs 234767248 Jan 19 13:23 libcutensor.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       142 May 22 21:21 libcutensor_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/aarch64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/12/libcutensor_static.a
-rw-rw-r--. 1 cvmfs cvmfs 8836 May 22 21:26 /cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/generic/.lmod/SitePackage.lua
-rw-rw-r--. 1 cvmfs cvmfs 1178 May 22 21:22 /cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/generic/modules/all/cuTENSOR/2.0.1.2-CUDA-12.1.1.lua

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1:
total 32
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May 22 21:23 easybuild
dr-xr-xr-x. 3 cvmfs cvmfs  4096 Jan 19 13:34 include
dr-xr-xr-x. 5 cvmfs cvmfs  4096 Jan 19 13:34 lib
lrwxrwxrwx. 1 cvmfs cvmfs     3 May 22 21:22 lib64 -> lib
-r--r--r--. 1 cvmfs cvmfs 17789 Jan 19 12:39 LICENSE

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/easybuild:
total 56
-r--r--r--. 1 cvmfs cvmfs  5972 May 22 21:22 cuTENSOR-2.0.1.2-CUDA-12.1.1-easybuild-devel
-r--r--r--. 1 cvmfs cvmfs  1088 Apr 10 19:41 cuTENSOR-2.0.1.2-CUDA-12.1.1.eb
-r--r--r--. 1 cvmfs cvmfs 25110 May 22 21:23 easybuild-cuTENSOR-2.0.1.2-20240522.212331.log.bz2
-rw-rw-r--. 1 cvmfs cvmfs 18874 May 22 21:23 easybuild-cuTENSOR-2.0.1.2-20240522.212331_test_report.md
dr-xr-xr-x. 4 cvmfs cvmfs  4096 May 22 21:22 reprod

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/easybuild/reprod:
total 11
-r--r--r--. 1 cvmfs cvmfs 1048 May 22 21:21 cuTENSOR-2.0.1.2-CUDA-12.1.1.eb
-r--r--r--. 1 cvmfs cvmfs  276 May 22 21:22 cuTENSOR-2.0.1.2-CUDA-12.1.1.env
dr-xr-xr-x. 2 cvmfs cvmfs 4096 May 22 21:21 easyblocks
dr-xr-xr-x. 2 cvmfs cvmfs 4096 May 22 21:21 hooks

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/easybuild/reprod/easyblocks:
total 6
-r--r--r--. 1 cvmfs cvmfs 5655 Apr 10 19:41 tarball.py

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/easybuild/reprod/hooks:
total 43
-r--r--r--. 1 cvmfs cvmfs 43825 May 22 21:10 eb_hooks.py

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/include:
total 103
dr-xr-xr-x. 2 cvmfs cvmfs  4096 Jan 19 13:25 cutensor
-r--r--r--. 1 cvmfs cvmfs 69855 Jan 19 12:39 cutensor.h
-r--r--r--. 1 cvmfs cvmfs 30584 Jan 19 12:39 cutensorMg.h

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/include/cutensor:
total 15
-r--r--r--. 1 cvmfs cvmfs 15283 Jan 19 13:21 types.h

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib:
total 14
dr-xr-xr-x. 2 cvmfs cvmfs 4096 May 22 21:22 11
dr-xr-xr-x. 2 cvmfs cvmfs 4096 May 22 21:22 11.0
dr-xr-xr-x. 2 cvmfs cvmfs 4096 May 22 21:22 12

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11:
total 655992
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:02 libcutensorMg.so
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:02 libcutensorMg.so.2
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:02 libcutensorMg.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       143 May 22 21:22 libcutensorMg_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11/libcutensorMg_static.a
-r-xr-xr-x. 1 cvmfs cvmfs 216839664 Jan 19 13:02 libcutensor.so
-r-xr-xr-x. 1 cvmfs cvmfs 216839664 Jan 19 13:02 libcutensor.so.2
-r-xr-xr-x. 1 cvmfs cvmfs 216839664 Jan 19 13:02 libcutensor.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       141 May 22 21:22 libcutensor_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11/libcutensor_static.a

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11.0:
total 515209
-r-xr-xr-x. 1 cvmfs cvmfs   6887248 Jan 19 12:48 libcutensorMg.so
-r-xr-xr-x. 1 cvmfs cvmfs   6887248 Jan 19 12:48 libcutensorMg.so.2
-r-xr-xr-x. 1 cvmfs cvmfs   6887248 Jan 19 12:48 libcutensorMg.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       145 May 22 21:22 libcutensorMg_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11.0/libcutensorMg_static.a
-r-xr-xr-x. 1 cvmfs cvmfs 168969928 Jan 19 12:48 libcutensor.so
-r-xr-xr-x. 1 cvmfs cvmfs 168969928 Jan 19 12:48 libcutensor.so.2
-r-xr-xr-x. 1 cvmfs cvmfs 168969928 Jan 19 12:48 libcutensor.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       143 May 22 21:22 libcutensor_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11.0/libcutensor_static.a

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/12:
total 708514
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:23 libcutensorMg.so
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:23 libcutensorMg.so.2
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:23 libcutensorMg.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       143 May 22 21:22 libcutensorMg_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/12/libcutensorMg_static.a
-r-xr-xr-x. 1 cvmfs cvmfs 234767248 Jan 19 13:23 libcutensor.so
-r-xr-xr-x. 1 cvmfs cvmfs 234767248 Jan 19 13:23 libcutensor.so.2
-r-xr-xr-x. 1 cvmfs cvmfs 234767248 Jan 19 13:23 libcutensor.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       141 May 22 21:22 libcutensor_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/generic/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/12/libcutensor_static.a
-rw-rw-r--. 1 cvmfs cvmfs 8836 May 22 21:08 /cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/amd/zen2/.lmod/SitePackage.lua
-rw-rw-r--. 1 cvmfs cvmfs 1179 May 22 21:03 /cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/amd/zen2/modules/all/cuTENSOR/2.0.1.2-CUDA-12.1.1.lua

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/amd/zen2/software/cuTENSOR/2.0.1.2-CUDA-12.1.1:
total 32
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May 22 21:04 easybuild
dr-xr-xr-x. 3 cvmfs cvmfs  4096 Jan 19 13:34 include
dr-xr-xr-x. 5 cvmfs cvmfs  4096 Jan 19 13:34 lib
lrwxrwxrwx. 1 cvmfs cvmfs     3 May 22 21:03 lib64 -> lib
-r--r--r--. 1 cvmfs cvmfs 17789 Jan 19 12:39 LICENSE

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/amd/zen2/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/easybuild:
total 56
-r--r--r--. 1 cvmfs cvmfs  5998 May 22 21:03 cuTENSOR-2.0.1.2-CUDA-12.1.1-easybuild-devel
-r--r--r--. 1 cvmfs cvmfs  1088 Apr 10 19:41 cuTENSOR-2.0.1.2-CUDA-12.1.1.eb
-r--r--r--. 1 cvmfs cvmfs 25295 May 22 21:04 easybuild-cuTENSOR-2.0.1.2-20240522.210435.log.bz2
-rw-rw-r--. 1 cvmfs cvmfs 18830 May 22 21:04 easybuild-cuTENSOR-2.0.1.2-20240522.210435_test_report.md
dr-xr-xr-x. 4 cvmfs cvmfs  4096 May 22 21:03 reprod

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/amd/zen2/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/easybuild/reprod:
total 11
-r--r--r--. 1 cvmfs cvmfs 1048 May 22 21:03 cuTENSOR-2.0.1.2-CUDA-12.1.1.eb
-r--r--r--. 1 cvmfs cvmfs  276 May 22 21:03 cuTENSOR-2.0.1.2-CUDA-12.1.1.env
dr-xr-xr-x. 2 cvmfs cvmfs 4096 May 22 21:03 easyblocks
dr-xr-xr-x. 2 cvmfs cvmfs 4096 May 22 21:03 hooks

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/amd/zen2/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/easybuild/reprod/easyblocks:
total 6
-r--r--r--. 1 cvmfs cvmfs 5655 Apr 10 19:41 tarball.py

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/amd/zen2/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/easybuild/reprod/hooks:
total 43
-r--r--r--. 1 cvmfs cvmfs 43825 May 22 20:53 eb_hooks.py

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/amd/zen2/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/include:
total 103
dr-xr-xr-x. 2 cvmfs cvmfs  4096 Jan 19 13:25 cutensor
-r--r--r--. 1 cvmfs cvmfs 69855 Jan 19 12:39 cutensor.h
-r--r--r--. 1 cvmfs cvmfs 30584 Jan 19 12:39 cutensorMg.h

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/amd/zen2/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/include/cutensor:
total 15
-r--r--r--. 1 cvmfs cvmfs 15283 Jan 19 13:21 types.h

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/amd/zen2/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib:
total 14
dr-xr-xr-x. 2 cvmfs cvmfs 4096 May 22 21:03 11
dr-xr-xr-x. 2 cvmfs cvmfs 4096 May 22 21:03 11.0
dr-xr-xr-x. 2 cvmfs cvmfs 4096 May 22 21:03 12

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/amd/zen2/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11:
total 655992
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:02 libcutensorMg.so
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:02 libcutensorMg.so.2
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:02 libcutensorMg.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       144 May 22 21:03 libcutensorMg_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/amd/zen2/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11/libcutensorMg_static.a
-r-xr-xr-x. 1 cvmfs cvmfs 216839664 Jan 19 13:02 libcutensor.so
-r-xr-xr-x. 1 cvmfs cvmfs 216839664 Jan 19 13:02 libcutensor.so.2
-r-xr-xr-x. 1 cvmfs cvmfs 216839664 Jan 19 13:02 libcutensor.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       142 May 22 21:03 libcutensor_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/amd/zen2/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11/libcutensor_static.a

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/amd/zen2/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11.0:
total 515209
-r-xr-xr-x. 1 cvmfs cvmfs   6887248 Jan 19 12:48 libcutensorMg.so
-r-xr-xr-x. 1 cvmfs cvmfs   6887248 Jan 19 12:48 libcutensorMg.so.2
-r-xr-xr-x. 1 cvmfs cvmfs   6887248 Jan 19 12:48 libcutensorMg.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       146 May 22 21:03 libcutensorMg_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/amd/zen2/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11.0/libcutensorMg_static.a
-r-xr-xr-x. 1 cvmfs cvmfs 168969928 Jan 19 12:48 libcutensor.so
-r-xr-xr-x. 1 cvmfs cvmfs 168969928 Jan 19 12:48 libcutensor.so.2
-r-xr-xr-x. 1 cvmfs cvmfs 168969928 Jan 19 12:48 libcutensor.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       144 May 22 21:03 libcutensor_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/amd/zen2/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11.0/libcutensor_static.a

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/amd/zen2/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/12:
total 708514
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:23 libcutensorMg.so
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:23 libcutensorMg.so.2
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:23 libcutensorMg.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       144 May 22 21:03 libcutensorMg_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/amd/zen2/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/12/libcutensorMg_static.a
-r-xr-xr-x. 1 cvmfs cvmfs 234767248 Jan 19 13:23 libcutensor.so
-r-xr-xr-x. 1 cvmfs cvmfs 234767248 Jan 19 13:23 libcutensor.so.2
-r-xr-xr-x. 1 cvmfs cvmfs 234767248 Jan 19 13:23 libcutensor.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       142 May 22 21:03 libcutensor_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/amd/zen2/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/12/libcutensor_static.a
-rw-rw-r--. 1 cvmfs cvmfs 8836 May 22 21:31 /cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/broadwell/.lmod/SitePackage.lua
-rw-rw-r--. 1 cvmfs cvmfs 1186 May 22 21:26 /cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/broadwell/modules/all/cuTENSOR/2.0.1.2-CUDA-12.1.1.lua

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/broadwell/software/cuTENSOR/2.0.1.2-CUDA-12.1.1:
total 32
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May 22 21:27 easybuild
dr-xr-xr-x. 3 cvmfs cvmfs  4096 Jan 19 13:34 include
dr-xr-xr-x. 5 cvmfs cvmfs  4096 Jan 19 13:34 lib
lrwxrwxrwx. 1 cvmfs cvmfs     3 May 22 21:26 lib64 -> lib
-r--r--r--. 1 cvmfs cvmfs 17789 Jan 19 12:39 LICENSE

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/broadwell/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/easybuild:
total 56
-r--r--r--. 1 cvmfs cvmfs  6200 May 22 21:26 cuTENSOR-2.0.1.2-CUDA-12.1.1-easybuild-devel
-r--r--r--. 1 cvmfs cvmfs  1088 Apr 10 19:40 cuTENSOR-2.0.1.2-CUDA-12.1.1.eb
-r--r--r--. 1 cvmfs cvmfs 25182 May 22 21:27 easybuild-cuTENSOR-2.0.1.2-20240522.232729.log.bz2
-rw-rw-r--. 1 cvmfs cvmfs 18423 May 22 21:27 easybuild-cuTENSOR-2.0.1.2-20240522.232729_test_report.md
dr-xr-xr-x. 4 cvmfs cvmfs  4096 May 22 21:26 reprod

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/broadwell/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/easybuild/reprod:
total 11
-r--r--r--. 1 cvmfs cvmfs 1048 May 22 21:24 cuTENSOR-2.0.1.2-CUDA-12.1.1.eb
-r--r--r--. 1 cvmfs cvmfs  276 May 22 21:26 cuTENSOR-2.0.1.2-CUDA-12.1.1.env
dr-xr-xr-x. 2 cvmfs cvmfs 4096 May 22 21:24 easyblocks
dr-xr-xr-x. 2 cvmfs cvmfs 4096 May 22 21:24 hooks

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/broadwell/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/easybuild/reprod/easyblocks:
total 6
-r--r--r--. 1 cvmfs cvmfs 5655 Apr 10 19:40 tarball.py

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/broadwell/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/easybuild/reprod/hooks:
total 43
-r--r--r--. 1 cvmfs cvmfs 43825 May 22 21:10 eb_hooks.py

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/broadwell/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/include:
total 103
dr-xr-xr-x. 2 cvmfs cvmfs  4096 Jan 19 13:25 cutensor
-r--r--r--. 1 cvmfs cvmfs 69855 Jan 19 12:39 cutensor.h
-r--r--r--. 1 cvmfs cvmfs 30584 Jan 19 12:39 cutensorMg.h

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/broadwell/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/include/cutensor:
total 15
-r--r--r--. 1 cvmfs cvmfs 15283 Jan 19 13:21 types.h

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/broadwell/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib:
total 14
dr-xr-xr-x. 2 cvmfs cvmfs 4096 May 22 21:26 11
dr-xr-xr-x. 2 cvmfs cvmfs 4096 May 22 21:26 11.0
dr-xr-xr-x. 2 cvmfs cvmfs 4096 May 22 21:26 12

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/broadwell/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11:
total 655992
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:02 libcutensorMg.so
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:02 libcutensorMg.so.2
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:02 libcutensorMg.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       151 May 22 21:26 libcutensorMg_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/intel/broadwell/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11/libcutensorMg_static.a
-r-xr-xr-x. 1 cvmfs cvmfs 216839664 Jan 19 13:02 libcutensor.so
-r-xr-xr-x. 1 cvmfs cvmfs 216839664 Jan 19 13:02 libcutensor.so.2
-r-xr-xr-x. 1 cvmfs cvmfs 216839664 Jan 19 13:02 libcutensor.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       149 May 22 21:26 libcutensor_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/intel/broadwell/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11/libcutensor_static.a

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/broadwell/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11.0:
total 515209
-r-xr-xr-x. 1 cvmfs cvmfs   6887248 Jan 19 12:48 libcutensorMg.so
-r-xr-xr-x. 1 cvmfs cvmfs   6887248 Jan 19 12:48 libcutensorMg.so.2
-r-xr-xr-x. 1 cvmfs cvmfs   6887248 Jan 19 12:48 libcutensorMg.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       153 May 22 21:26 libcutensorMg_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/intel/broadwell/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11.0/libcutensorMg_static.a
-r-xr-xr-x. 1 cvmfs cvmfs 168969928 Jan 19 12:48 libcutensor.so
-r-xr-xr-x. 1 cvmfs cvmfs 168969928 Jan 19 12:48 libcutensor.so.2
-r-xr-xr-x. 1 cvmfs cvmfs 168969928 Jan 19 12:48 libcutensor.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       151 May 22 21:26 libcutensor_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/intel/broadwell/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11.0/libcutensor_static.a

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/broadwell/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/12:
total 708514
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:23 libcutensorMg.so
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:23 libcutensorMg.so.2
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:23 libcutensorMg.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       151 May 22 21:26 libcutensorMg_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/intel/broadwell/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/12/libcutensorMg_static.a
-r-xr-xr-x. 1 cvmfs cvmfs 234767248 Jan 19 13:23 libcutensor.so
-r-xr-xr-x. 1 cvmfs cvmfs 234767248 Jan 19 13:23 libcutensor.so.2
-r-xr-xr-x. 1 cvmfs cvmfs 234767248 Jan 19 13:23 libcutensor.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       149 May 22 21:26 libcutensor_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/intel/broadwell/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/12/libcutensor_static.a
-rw-rw-r--. 1 cvmfs cvmfs 8836 May 22 21:27 /cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/.lmod/SitePackage.lua
-rw-rw-r--. 1 cvmfs cvmfs 1191 May 22 21:23 /cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/modules/all/cuTENSOR/2.0.1.2-CUDA-12.1.1.lua

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/cuTENSOR/2.0.1.2-CUDA-12.1.1:
total 32
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May 22 21:24 easybuild
dr-xr-xr-x. 3 cvmfs cvmfs  4096 Jan 19 13:34 include
dr-xr-xr-x. 5 cvmfs cvmfs  4096 Jan 19 13:34 lib
lrwxrwxrwx. 1 cvmfs cvmfs     3 May 22 21:23 lib64 -> lib
-r--r--r--. 1 cvmfs cvmfs 17789 Jan 19 12:39 LICENSE

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/easybuild:
total 56
-r--r--r--. 1 cvmfs cvmfs  6374 May 22 21:23 cuTENSOR-2.0.1.2-CUDA-12.1.1-easybuild-devel
-r--r--r--. 1 cvmfs cvmfs  1088 Apr 10 19:40 cuTENSOR-2.0.1.2-CUDA-12.1.1.eb
-r--r--r--. 1 cvmfs cvmfs 25566 May 22 21:24 easybuild-cuTENSOR-2.0.1.2-20240522.232407.log.bz2
-rw-rw-r--. 1 cvmfs cvmfs 18537 May 22 21:24 easybuild-cuTENSOR-2.0.1.2-20240522.232407_test_report.md
dr-xr-xr-x. 4 cvmfs cvmfs  4096 May 22 21:23 reprod

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/easybuild/reprod:
total 11
-r--r--r--. 1 cvmfs cvmfs 1048 May 22 21:22 cuTENSOR-2.0.1.2-CUDA-12.1.1.eb
-r--r--r--. 1 cvmfs cvmfs  276 May 22 21:23 cuTENSOR-2.0.1.2-CUDA-12.1.1.env
dr-xr-xr-x. 2 cvmfs cvmfs 4096 May 22 21:22 easyblocks
dr-xr-xr-x. 2 cvmfs cvmfs 4096 May 22 21:22 hooks

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/easybuild/reprod/easyblocks:
total 6
-r--r--r--. 1 cvmfs cvmfs 5655 Apr 10 19:40 tarball.py

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/easybuild/reprod/hooks:
total 43
-r--r--r--. 1 cvmfs cvmfs 43825 May 22 21:10 eb_hooks.py

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/include:
total 103
dr-xr-xr-x. 2 cvmfs cvmfs  4096 Jan 19 13:25 cutensor
-r--r--r--. 1 cvmfs cvmfs 69855 Jan 19 12:39 cutensor.h
-r--r--r--. 1 cvmfs cvmfs 30584 Jan 19 12:39 cutensorMg.h

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/include/cutensor:
total 15
-r--r--r--. 1 cvmfs cvmfs 15283 Jan 19 13:21 types.h

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib:
total 14
dr-xr-xr-x. 2 cvmfs cvmfs 4096 May 22 21:23 11
dr-xr-xr-x. 2 cvmfs cvmfs 4096 May 22 21:23 11.0
dr-xr-xr-x. 2 cvmfs cvmfs 4096 May 22 21:23 12

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11:
total 655992
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:02 libcutensorMg.so
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:02 libcutensorMg.so.2
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:02 libcutensorMg.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       156 May 22 21:23 libcutensorMg_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/intel/skylake_avx512/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11/libcutensorMg_static.a
-r-xr-xr-x. 1 cvmfs cvmfs 216839664 Jan 19 13:02 libcutensor.so
-r-xr-xr-x. 1 cvmfs cvmfs 216839664 Jan 19 13:02 libcutensor.so.2
-r-xr-xr-x. 1 cvmfs cvmfs 216839664 Jan 19 13:02 libcutensor.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       154 May 22 21:23 libcutensor_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/intel/skylake_avx512/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11/libcutensor_static.a

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11.0:
total 515209
-r-xr-xr-x. 1 cvmfs cvmfs   6887248 Jan 19 12:48 libcutensorMg.so
-r-xr-xr-x. 1 cvmfs cvmfs   6887248 Jan 19 12:48 libcutensorMg.so.2
-r-xr-xr-x. 1 cvmfs cvmfs   6887248 Jan 19 12:48 libcutensorMg.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       158 May 22 21:23 libcutensorMg_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/intel/skylake_avx512/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11.0/libcutensorMg_static.a
-r-xr-xr-x. 1 cvmfs cvmfs 168969928 Jan 19 12:48 libcutensor.so
-r-xr-xr-x. 1 cvmfs cvmfs 168969928 Jan 19 12:48 libcutensor.so.2
-r-xr-xr-x. 1 cvmfs cvmfs 168969928 Jan 19 12:48 libcutensor.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       156 May 22 21:23 libcutensor_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/intel/skylake_avx512/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/11.0/libcutensor_static.a

/cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/12:
total 708514
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:23 libcutensorMg.so
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:23 libcutensorMg.so.2
-r-xr-xr-x. 1 cvmfs cvmfs   7071568 Jan 19 13:23 libcutensorMg.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       156 May 22 21:23 libcutensorMg_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/intel/skylake_avx512/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/12/libcutensorMg_static.a
-r-xr-xr-x. 1 cvmfs cvmfs 234767248 Jan 19 13:23 libcutensor.so
-r-xr-xr-x. 1 cvmfs cvmfs 234767248 Jan 19 13:23 libcutensor.so.2
-r-xr-xr-x. 1 cvmfs cvmfs 234767248 Jan 19 13:23 libcutensor.so.2.0.1
lrwxrwxrwx. 1 cvmfs cvmfs       154 May 22 21:23 libcutensor_static.a -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/intel/skylake_avx512/software/cuTENSOR/2.0.1.2-CUDA-12.1.1/lib/12/libcutensor_static.a
-rw-rw-r--. 1 cvmfs cvmfs 43825 May 22 21:12 /cvmfs/pilot.nessi.no/versions/2023.06/software/linux/../../init/easybuild/eb_hooks.py
-rw-rw-r--. 1 cvmfs cvmfs   102 May 22 21:12 /cvmfs/pilot.nessi.no/versions/2023.06/software/linux/../../scripts/gpu_support/nvidia/eessi-2023.06-cuda-and-libraries.yml
-rwxrwxr-x. 1 cvmfs cvmfs  8428 May 22 21:12 /cvmfs/pilot.nessi.no/versions/2023.06/software/linux/../../scripts/gpu_support/nvidia/install_cuda_and_libraries.sh

poksumdo

One step closer to GPU-enabled software.

truib added 2 commits May 22, 2024 20:18

adding unstaged changes

433d588

trz42 added NESSI/2023.06 GPU labels May 22, 2024

trz42 added the DONT_INGEST label May 22, 2024

Merge branch 'nessi.no-2023.06' of github-trz:NorESSI/software-layer …

7c231ef

…into nessi-2023.06-cuTENSOR-2.0.1.2-CUDA-12.1.1

truib added 2 commits May 22, 2024 21:00

make sure that base directory for tmp dir exists

8a9f586

add post_sanitycheck_cutensor

8a9c913

need to pass more args to replace func

23406aa

fix silly mistake in replace function

b3a6215

trz42 added bot:deploy and removed DONT_INGEST labels May 23, 2024

poksumdo approved these changes May 23, 2024

View reviewed changes

poksumdo merged commit da4971d into NorESSI:nessi.no-2023.06 May 23, 2024
25 of 26 checks passed

trz42 added the ingested label May 23, 2024

{2023.06}[system] cuTENSOR v2.0.1.2 with CUDA/12.1.1 #367

{2023.06}[system] cuTENSOR v2.0.1.2 with CUDA/12.1.1 #367

Conversation

trz42 commented May 22, 2024

nessi-bot bot commented May 22, 2024

nessi-bot bot commented May 22, 2024

nessi-bot bot commented May 22, 2024

nessi-bot bot commented May 22, 2024

trz42 commented May 22, 2024

nessi-bot bot commented May 22, 2024

nessi-bot bot commented May 22, 2024 • edited Loading

nessi-bot bot commented May 22, 2024 • edited Loading

nessi-bot bot commented May 22, 2024 • edited Loading

nessi-bot bot commented May 22, 2024

trz42 commented May 22, 2024

nessi-bot bot commented May 22, 2024 • edited Loading

nessi-bot bot commented May 22, 2024 • edited Loading

nessi-bot bot commented May 22, 2024 • edited Loading

nessi-bot bot commented May 22, 2024 • edited Loading

nessi-bot bot commented May 22, 2024 • edited Loading

trz42 commented May 22, 2024

nessi-bot bot commented May 22, 2024 • edited Loading

nessi-bot bot commented May 22, 2024 • edited Loading

nessi-bot bot commented May 22, 2024 • edited Loading

nessi-bot bot commented May 22, 2024 • edited Loading

nessi-bot bot commented May 22, 2024 • edited Loading

trz42 commented May 22, 2024

nessi-bot bot commented May 22, 2024 • edited Loading

nessi-bot bot commented May 22, 2024 • edited Loading

nessi-bot bot commented May 22, 2024 • edited Loading

nessi-bot bot commented May 22, 2024 • edited Loading

nessi-bot bot commented May 22, 2024 • edited Loading

trz42 commented May 22, 2024

nessi-bot bot commented May 22, 2024 • edited Loading

nessi-bot bot commented May 22, 2024 • edited Loading

nessi-bot bot commented May 22, 2024 • edited Loading

nessi-bot bot commented May 22, 2024 • edited Loading

nessi-bot bot commented May 22, 2024 • edited by trz42 Loading

trz42 commented May 22, 2024

nessi-bot bot commented May 22, 2024 • edited Loading

nessi-bot bot commented May 22, 2024 • edited Loading

nessi-bot bot commented May 22, 2024 • edited Loading

nessi-bot bot commented May 22, 2024 • edited Loading

nessi-bot bot commented May 22, 2024 • edited by trz42 Loading

nessi-bot bot commented May 22, 2024 • edited by trz42 Loading

nessi-bot bot commented May 22, 2024 • edited by trz42 Loading

nessi-bot bot commented May 22, 2024 • edited by trz42 Loading

trz42 commented May 23, 2024

trz42 commented May 23, 2024 • edited Loading

poksumdo left a comment

Choose a reason for hiding this comment

nessi-bot bot commented May 22, 2024 •

edited

Loading

nessi-bot bot commented May 22, 2024 •

edited

Loading

nessi-bot bot commented May 22, 2024 •

edited

Loading

nessi-bot bot commented May 22, 2024 •

edited

Loading

nessi-bot bot commented May 22, 2024 •

edited

Loading

nessi-bot bot commented May 22, 2024 •

edited

Loading

nessi-bot bot commented May 22, 2024 •

edited

Loading

nessi-bot bot commented May 22, 2024 •

edited

Loading

nessi-bot bot commented May 22, 2024 •

edited

Loading

nessi-bot bot commented May 22, 2024 •

edited

Loading

nessi-bot bot commented May 22, 2024 •

edited

Loading

nessi-bot bot commented May 22, 2024 •

edited

Loading

nessi-bot bot commented May 22, 2024 •

edited

Loading

nessi-bot bot commented May 22, 2024 •

edited

Loading

nessi-bot bot commented May 22, 2024 •

edited

Loading

nessi-bot bot commented May 22, 2024 •

edited

Loading

nessi-bot bot commented May 22, 2024 •

edited

Loading

nessi-bot bot commented May 22, 2024 •

edited

Loading

nessi-bot bot commented May 22, 2024 •

edited

Loading

nessi-bot bot commented May 22, 2024 •

edited

Loading

nessi-bot bot commented May 22, 2024 •

edited

Loading

nessi-bot bot commented May 22, 2024 •

edited

Loading

nessi-bot bot commented May 22, 2024 •

edited by trz42

Loading

nessi-bot bot commented May 22, 2024 •

edited

Loading

nessi-bot bot commented May 22, 2024 •

edited

Loading

nessi-bot bot commented May 22, 2024 •

edited

Loading

nessi-bot bot commented May 22, 2024 •

edited

Loading

nessi-bot bot commented May 22, 2024 •

edited by trz42

Loading

nessi-bot bot commented May 22, 2024 •

edited by trz42

Loading

nessi-bot bot commented May 22, 2024 •

edited by trz42

Loading

nessi-bot bot commented May 22, 2024 •

edited by trz42

Loading

trz42 commented May 23, 2024 •

edited

Loading