Add new init scripts for new initialization module #668

MaKaNu · 2024-08-12T12:49:48Z

This is a followup PR for #667

After successful merge of #667 I have some test cases prepared, to test against the different shell.

For the moment, I struggle with an implementation for csh. It only seems to happen when I try to load the module inside csh. Error response of csh is not the brightest I've seen.

- zsh - fish

variable setting and calling

- ksh

- csh

eessi-bot · 2024-08-12T12:49:53Z

Instance eessi-bot-mc-aws is configured to build for:

architectures: x86_64/generic, x86_64/intel/haswell, x86_64/intel/skylake_avx512, x86_64/amd/zen2, x86_64/amd/zen3, aarch64/generic, aarch64/neoverse_n1, aarch64/neoverse_v1
repositories: eessi.io-2023.06-compat, eessi-hpc.org-2023.06-software, eessi-hpc.org-2023.06-compat, eessi.io-2023.06-software

eessi-build-deploy-bot-deucalion · 2024-08-12T12:49:53Z

Instance boegel-bot-deucalion is configured to build for:

architectures: aarch64/a64fx
repositories: eessi.io-2023.06-software

eessi-bot · 2024-08-12T12:49:54Z

Instance eessi-bot-mc-azure is configured to build for:

architectures: x86_64/amd/zen4
repositories: eessi.io-2023.06-compat, eessi-hpc.org-2023.06-compat, eessi-hpc.org-2023.06-software, eessi.io-2023.06-software

ocaisa · 2024-08-12T12:57:00Z

init/bash

+export MODULEPATH=/cvmfs/software.eessi.io/versions/"$EESSI_VERSION"/init/modules
+. /cvmfs/software.eessi.io/versions/"$EESSI_VERSION"/compat/linux/$(uname -m)/usr/share/Lmod/init/bash
+
+module load "$LMOD_SYSTEM_DEFAULT_MODULES"


The module load should not be necessary, they should be loaded when Lmod is initialised

I'm not 100% though...https://lmod.readthedocs.io/en/latest/070_standard_modules.html#providing-a-standard-set-of-modules-for-all-users

The module load should not be necessary, they should be loaded when Lmod is initialised

In my test cases the EESSI/2023.06 module was not loaded automatically, that why I put it in.

Just setting StdEnv.lua doesn't do the trick also.

Was there already an Lmod loaded when you sourced it?

module load $LMOD_SYSTEM_DEFAULT_MODULES is a little brittle as this won't work if there is more than module in the variable (I think). It's probably best to take the official route as per the Lmod docs (shell dependent):

if [ -z "$__Init_Default_Modules" ]; then export __Init_Default_Modules=1; ## ability to predefine elsewhere the default list LMOD_SYSTEM_DEFAULT_MODULES=${LMOD_SYSTEM_DEFAULT_MODULES:-"StdEnv"} export LMOD_SYSTEM_DEFAULT_MODULES module --initial_load --no_redirect restore else module refresh fi

Was there already an Lmod loaded when you sourced it?

No, Lmod was not loaded at that moment. Only loaded with the script.

MaKaNu · 2024-08-12T14:08:13Z

Ok it expanded the PS1 inside the git commit message 🥲

MaKaNu · 2024-08-13T21:20:08Z

module load $LMOD_SYSTEM_DEFAULT_MODULES is a little brittle as this won't work if there is more than module in the variable (I think). It's probably best to take the official route as per the Lmod docs (shell dependent):

Implemented in last commit. Works as expected, with exclusion for csh.

- add test script - add github actions

not sure what the output for generic would be. But different arches should be already tested on module

- Fix Format

MaKaNu · 2024-08-16T15:03:44Z

Pushed tests are expected to fail for now (getting interesting after merge of #667)

Pull Last merge from Upstream

…eessi.io

init/bash

ocaisa · 2024-09-05T13:45:29Z

init/bash

+   export __Init_Default_Modules=1;
+
+   ## ability to predefine elsewhere the default list
+   LMOD_SYSTEM_DEFAULT_MODULES=${LMOD_SYSTEM_DEFAULT_MODULES:-"StdEnv"}


Suggested change

LMOD_SYSTEM_DEFAULT_MODULES=${LMOD_SYSTEM_DEFAULT_MODULES:-"StdEnv"}

LMOD_SYSTEM_DEFAULT_MODULES=${LMOD_SYSTEM_DEFAULT_MODULES:-"EESSI"}

You'll need this everywhere

Some sites also force full name matching so you probably can't leave it versionless

This is now implemented with Version number. It doesn't matter to me but like to understand the difference why StdEnv.lua is not the correct solution.

There is no existing StdEnv.lua when Lmod is configured (we don't ship one)

ocaisa · 2024-09-05T13:52:08Z

We never included PS1 in #667 so your tests will fail for that currently. If you are keen to have that, it could be part of this PR or a follow-up after this is merged (this PR has higher priority in my opinion)

- move new scripts - copy bash

restore original content of bash init

csh is not as feature rich as other shells I have implemented. Maybe someday I learn how to approach it!

init/lmod/csh

.github/workflows/scripts/test_init_scripts.sh

before we run any tests we check if the shell provided is actually testable. If a shell is not tested yet only means we don't know how to test the shell or the shell is not added to the TEST_SHELLS Array.

We actually don't know how to csh for now. But if we figure out we don't need to update here.

remove leftover comment Co-authored-by: ocaisa <[email protected]>

ocaisa

LGTM

ocaisa · 2024-09-12T16:50:32Z

bot: build repo:eessi.io-2023.06-software arch:x86_64/generic

eessi-bot · 2024-09-12T16:50:35Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/generic from ocaisa
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/generic
handling command build repository:eessi.io-2023.06-software architecture:x86_64/generic resulted in:
- submitted job 18438, for details & status see Add new init scripts for new initialization module #668 (comment)

eessi-bot · 2024-09-12T16:50:35Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/generic from ocaisa
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/generic
handling command build repository:eessi.io-2023.06-software architecture:x86_64/generic resulted in:
- no jobs were submitted

eessi-bot · 2024-09-12T16:50:39Z

New job on instance eessi-bot-mc-aws for architecture x86_64-generic for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.09/pr_668/18438

date	job status	comment
Sep 12 16:50:38 UTC 2024	submitted	job id `18438` awaits release by job manager
Sep 12 16:51:12 UTC 2024	released	job awaits launch by Slurm scheduler
Sep 12 16:57:14 UTC 2024	running	job `18438` is running
Sep 12 17:16:35 UTC 2024	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-18438.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts No artefacts were created or found.
Sep 12 17:16:35 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 18/18 test case(s) from 18 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-18438.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

eessi-build-deploy-bot-deucalion · 2024-09-12T16:50:42Z

Updates by the bot instance boegel-bot-deucalion (click for details)

account ocaisa has NO permission to send commands to the bot

ocaisa · 2024-09-12T17:57:09Z

You need something similar to https://github.com/EESSI/software-layer/blob/2023.06-software.eessi.io/install_scripts.sh#L94-L97 so that the new scripts are actually deployed

Add copy of init lmod scripts

ocaisa · 2024-09-12T20:48:05Z

bot: build repo:eessi.io-2023.06-software arch:x86_64/generic

eessi-bot · 2024-09-12T20:48:09Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/generic from ocaisa
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/generic
handling command build repository:eessi.io-2023.06-software architecture:x86_64/generic resulted in:
- submitted job 18439, for details & status see Add new init scripts for new initialization module #668 (comment)

eessi-bot · 2024-09-12T20:48:09Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/generic from ocaisa
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/generic
handling command build repository:eessi.io-2023.06-software architecture:x86_64/generic resulted in:
- no jobs were submitted

eessi-build-deploy-bot-deucalion · 2024-09-12T20:48:10Z

Updates by the bot instance boegel-bot-deucalion (click for details)

account ocaisa has NO permission to send commands to the bot

eessi-bot · 2024-09-12T20:48:13Z

New job on instance eessi-bot-mc-aws for architecture x86_64-generic for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.09/pr_668/18439

date	job status	comment
Sep 12 20:48:12 UTC 2024	submitted	job id `18439` awaits release by job manager
Sep 12 20:48:54 UTC 2024	released	job awaits launch by Slurm scheduler
Sep 12 20:54:57 UTC 2024	running	job `18439` is running
Sep 12 21:14:17 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-18439.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-generic-1726174458.tar.gz` size: 0 MiB (756 bytes) entries: 5 modules under 2023.06/software/linux/x86_64/generic/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/generic/software no software packages in tarball other under 2023.06/software/linux/x86_64/generic `2023.06/init/lmod/bash` `2023.06/init/lmod/csh` `2023.06/init/lmod/fish` `2023.06/init/lmod/ksh` `2023.06/init/lmod/zsh`
Sep 12 21:14:17 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 18/18 test case(s) from 18 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-18439.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`
Sep 12 22:31:36 UTC 2024	uploaded	transfer of `eessi-2023.06-software-linux-x86_64-generic-1726174458.tar.gz` to S3 bucket succeeded

MaKaNu · 2024-09-12T21:21:38Z

bot: build repo:eessi.io-2023.06-software arch:x86_64/generic

eessi-bot · 2024-09-12T21:21:41Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

account MaKaNu has NO permission to send commands to the bot

eessi-bot · 2024-09-12T21:21:41Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

account MaKaNu has NO permission to send commands to the bot

eessi-build-deploy-bot-deucalion · 2024-09-12T21:21:42Z

Updates by the bot instance boegel-bot-deucalion (click for details)

account MaKaNu has NO permission to send commands to the bot

ocaisa

LGTM, thanks for steering this over the line!

eessi-build-deploy-bot-deucalion · 2024-09-12T22:31:19Z

Label bot:deploy has been set by user ocaisa, but this person does not have permission to trigger deployments

ocaisa · 2024-09-12T22:51:01Z

Staging PR merged, this is now in the wild!

MaKaNu added 5 commits August 12, 2024 13:37

Add init scripts

d512675

- zsh - fish

Fix fish init script

7f71c28

variable setting and calling

Add init script

1e2f154

- ksh

convert bash init script

a0c4377

Add init script

68aea9f

- csh

ocaisa reviewed Aug 12, 2024

View reviewed changes

Export ${debian_chroot:+($debian_chroot)}\u@\h:\w\$ for all init scripts

788375d

Implement module load following doc

b14cef8

MaKaNu added 7 commits August 16, 2024 13:58

Add init scripts test

1aa8cb8

- add test script - add github actions

Cleanup script and FIX init module path

0b6bc54

Use haswell for testing

4aa7d00

not sure what the output for generic would be. But different arches should be already tested on module

Fix Pipeline

71543e3

- Fix Format

Add StdEnv.lua module

ea32ffe

Install Shells on github actions

bb3c8bd

Fix requirements install on github action

385aee0

MaKaNu and others added 2 commits August 19, 2024 08:34

Fix Fish command call issue

461c941

Merge pull request #1 from EESSI/2023.06-software.eessi.io

aee306c

Pull Last merge from Upstream

TopRichard mentioned this pull request Aug 26, 2024

EESSI bash initialization to module file #667

Merged

Merge branch 'EESSI:2023.06-software.eessi.io' into 2023.06-software.…

6d7fde5

…eessi.io

ocaisa reviewed Sep 5, 2024

View reviewed changes

init/bash Outdated Show resolved Hide resolved

ocaisa reviewed Sep 5, 2024

View reviewed changes

MaKaNu added 2 commits September 6, 2024 06:25

move init scripts into init/lmod

fcf4937

- move new scripts - copy bash

Revert a0c4377

032cdae

restore original content of bash init

boegel mentioned this pull request Sep 12, 2024

Support for zsh #702

Closed

Add Exception for CSH

462f180

csh is not as feature rich as other shells I have implemented. Maybe someday I learn how to approach it!

ocaisa reviewed Sep 12, 2024

View reviewed changes

init/lmod/csh Outdated Show resolved Hide resolved

ocaisa reviewed Sep 12, 2024

View reviewed changes

.github/workflows/scripts/test_init_scripts.sh Outdated Show resolved Hide resolved

MaKaNu and others added 3 commits September 12, 2024 15:47

Add Check for not testable shells

68ad3af

before we run any tests we check if the shell provided is actually testable. If a shell is not tested yet only means we don't know how to test the shell or the shell is not added to the TEST_SHELLS Array.

Add csh to test

bccd55a

We actually don't know how to csh for now. But if we figure out we don't need to update here.

Update init/lmod/csh

7ad306a

remove leftover comment Co-authored-by: ocaisa <[email protected]>

ocaisa previously approved these changes Sep 12, 2024

View reviewed changes

Update install_scripts.sh

1b38cec

Add copy of init lmod scripts

MaKaNu dismissed ocaisa’s stale review via 1b38cec September 12, 2024 20:41

ocaisa approved these changes Sep 12, 2024

View reviewed changes

ocaisa added the bot:deploy Ask bot to deploy missing software installations to EESSI label Sep 12, 2024

ocaisa merged commit 4b5bd66 into EESSI:2023.06-software.eessi.io Sep 12, 2024
35 checks passed

	LMOD_SYSTEM_DEFAULT_MODULES=${LMOD_SYSTEM_DEFAULT_MODULES:-"StdEnv"}
	LMOD_SYSTEM_DEFAULT_MODULES=${LMOD_SYSTEM_DEFAULT_MODULES:-"EESSI"}

Add new init scripts for new initialization module #668

Add new init scripts for new initialization module #668

Conversation

MaKaNu commented Aug 12, 2024

eessi-bot bot commented Aug 12, 2024

eessi-build-deploy-bot-deucalion bot commented Aug 12, 2024

eessi-bot bot commented Aug 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MaKaNu commented Aug 12, 2024

MaKaNu commented Aug 13, 2024

MaKaNu commented Aug 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ocaisa commented Sep 5, 2024

ocaisa left a comment

Choose a reason for hiding this comment

ocaisa commented Sep 12, 2024

eessi-bot bot commented Sep 12, 2024 • edited Loading

eessi-bot bot commented Sep 12, 2024 • edited Loading

eessi-bot bot commented Sep 12, 2024 • edited Loading

eessi-build-deploy-bot-deucalion bot commented Sep 12, 2024

ocaisa commented Sep 12, 2024

ocaisa commented Sep 12, 2024

eessi-bot bot commented Sep 12, 2024 • edited Loading

eessi-bot bot commented Sep 12, 2024 • edited Loading

eessi-build-deploy-bot-deucalion bot commented Sep 12, 2024

eessi-bot bot commented Sep 12, 2024 • edited Loading

MaKaNu commented Sep 12, 2024

eessi-bot bot commented Sep 12, 2024

eessi-bot bot commented Sep 12, 2024

eessi-build-deploy-bot-deucalion bot commented Sep 12, 2024

ocaisa left a comment

Choose a reason for hiding this comment

eessi-build-deploy-bot-deucalion bot commented Sep 12, 2024

ocaisa commented Sep 12, 2024

eessi-bot bot commented Sep 12, 2024 •

edited

Loading

eessi-bot bot commented Sep 12, 2024 •

edited

Loading

eessi-bot bot commented Sep 12, 2024 •

edited

Loading

eessi-bot bot commented Sep 12, 2024 •

edited

Loading

eessi-bot bot commented Sep 12, 2024 •

edited

Loading

eessi-bot bot commented Sep 12, 2024 •

edited

Loading