Skip to content

Conference call notes 20241023

Jump to bottom
Bart Oldeman edited this page Oct 23, 2024 · 10 revisions

(back to Conference calls)

Notes on the 257th EasyBuild conference call, Wednesday 23 Oct 2024 (15:00 UTC)

Attendees

List of attendees (18):

  • Simon Branford (Univ. of Birmingham, UK)
  • Bibek Chapagain (Barcelona Supercomputing Centre, Spain)
  • Alex Domingo (Vrije Universiteit Brussel, Belgium)
  • Davide Grassano (CECAM, Switzerland)
  • Jasper Grimm (University of York, UK)
  • Jorge Guerra (Universidad Politécnica de Madrid, Spain)
  • Alexander Grund (ZIH, Dresden, Germany)
  • Leonardo Honfi Camilo (Univ. of Wageningen, Netherlands)
  • Kenneth Hoste (HPC-UGent, Belgium)
  • Adam Huffman (Big Data Institute, Oxford, UK)
  • Sébastien Moretti (SIB, Switzerland)
  • Alan O'Cais (CECAM, Univ. of Barcelona)
  • Jan Reuter (JSC, Germany)
  • Jörg Saßmannshausen (Imperial College London, UK)
  • Åke Sandgren (Umeå University, Sweden)
  • Alexandre Strube (JSC, Germany)
  • Alain van Hoof (TU Eindhoven, Netherlands)
  • Cintia Willemyns (Vrije Universiteit Brussel, Belgium)
  • David Coster (Max Planck Institute for Plasma Physics, Germany)
  • Em Dragowsky (Case Western Reserve University, Ohio, US)
  • Chia-Jung Hsu (Chalmers University of Technology, Sweden)
  • Georgios Kafanas (University of Luxembourg)
  • Kurt Lust (UAntwerpen, Belgium + LUMI User Support Team)
  • Sam Moors (Vrije Universiteit Brussel, Belgium)
  • Mikael Öhman (Chalmers University of Technology, Sweden)

Agenda

  • overview of recent developments
  • outlook to EasyBuild 5.0 release
  • 2024b update of common toolchains
  • Q&A

Recent developments

  • latest EasyBuild release: 4.9.4 (22 Sept 2024)
  • next EasyBuild release:
    • very likely EasyBuild v5.0.0 🔥
    • additional EasyBuild 4.9.x versions could still be done via 4.9.x branches, but none are planned currently
  • EasyBuild v5.0.0
    • project board: https://github.com/orgs/easybuilders/projects/18/views/2
    • detailed notes on latest developments in https://hackmd.io/tZtz6vMmRfGL6N9Rf__OEA
    • to test development version of EasyBuild 5.0: 
      # set up Python virtual environment, and jump into it
python3 -m venv eb5
source eb5/bin/activate

# install EasyBuild 5.0 development version into it
pip install https://github.com/easybuilders/easybuild-framework/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyblocks/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyconfigs/archive/5.0.x.tar.gz

# go!
eb --version
  • collapsing of 5.0.x branches in develop post-poned
    • impact on people using EasyBuild 4.x considered too big, so we want to limit the time window in which problems will arise as much as possible
    • impact:
      • --from-pr will be broken with EasyBuild 4.x due to major version mismatch
        • --from-commit can be used as a workaround
      • open easyconfig PRs should be synced with develop branch before they get merged
        • due to changes in easyconfigs test suite implemented in 5.0.x
        • for example use_pip = True should no longer be used
        • same goes for framework & easyblocks

Merged PRs

  • docs (merged PRs)

    • ...

  • framework (merged PRs)

    • bug fixes
      • [5.0.x] Use more robust mechanism for unsetting environment variables (PR #4685)
    • enhancements
      • [5.0.x] disable strict RPATH sanity check by default, allow re-enabling it via --strict-rpath-sanity-check configuration option (PR #4475)
      • [5.0.x] don't raise error when required extensions are not found when installing extensions in parallel (PR #4671)
      • [5.0.x] include path to cmd.sh script in output generated by run_shell_cmd when a command fails + use colors: red for ERROR line, yellow for path to output files + cmd.sh script (PR #4666)
      • [develop] add support for %(rpath_enabled)s template value (PR #4670)
    • changes
      • ...
    • code cleanup
      • ...
    • tests
      • ...

  • easyblocks (merged PRs)

    • bug fixes
      • [5.0.x] take into account that 'pic' toolchain option may not be defined in CMakeMake easyblock (PR #3470)
      • [5.0.x] Make MesonNinja respect the toolchainopts with buildtype as well as --debug and --optimization flags (PR #3454)
      • [5.0.x] use *DESCRIPTION rather than */DESCRIPTION in RPackage.requires_deps (PR #3490)
      • [5.0.x] add missing return for customised install_extension_async methods in Rserve and Rmpi easyblocks (PR #3474)
      • [develop] disable CPU-specific optimizations for generic builds (by using a custom processor_arch) in LAMMPS easyblock (PR #3484)
      • [develop] update Kokkos CPU mapping for LAMMPS >= 2Aug2023 to use icx on Intel Icelake systems (PR #3321)
      • [develop] use DYNAMIC_ARCH=1 when building OpenBLAS with --optarch=GENERIC (PR #3492)
      • [develop] also add path to libtorch.so & co to $LIBRARY_PATH in generated module file for PyTorch (PR #3488)
    • enhancements
      • [5.0.x] enhance CMakeMake easyblock to check whether correct Python installation was picked up by CMake (PR #3399)
      • [develop] enhance Cargo easyblock to printmessage for generating Cargo.lock if it's missing (PR #3491)
      • [develop] Enhance AOCC EasyBlock to correctly pass GCC toolchain and compiler driver (PR #3480)
      • [develop] let CMakeMake easyblock also set Python_EXECUTABLE option, as well as Python3_EXECUTABLE and Python2_EXECUTABLE derivatives (when appropriate) (PR #3463)
      • [develop] enhance cuDNN easyblock to verify that EULA is accepted before installing it ((PR #3473)
    • updates
      • ...
    • changes
      • ...
    • new easyblocks
      • ...
    • code cleanup
      • [5.0.x] refactor search for python_cmd (PR #3475)

  • easyconfigs (merged PRs)

    • ~65 easyconfig PRs were merged since last conf call
    • bug fixes/reports
      • [develop] fix the zero division bug of networkx-3.1 (PR #21702)
      • [develop] Remove redundant sanity check from jedi easyconfigs (PR #21686)
      • [develop] Fix alignment bug in patchelf 0.18.0 by adding patch (PR #21674)
      • [5.0.x] fix failing RPATH sanity check for recent dorado easyconfigs using foss/2023a toolchain (PR #21673)
      • [develop] demote hatchling to build-only dependency in easyconfigs for anndata 0.10.5.post1, Cassiopeia 2.0.0, phonemizer 3.2.1, scanpy 1.9.8, wandb 0.16.1 (PR #21657)
      • [develop] stop using --buildtype=release in easyconfigs using MesonNinja easyblock (PR #21619)
      • [develop] fix installation of NextPolish v1.4.1 when RPATH linking is used (+ move to GCC toolchain) (PR #21588)
    • enhancements
      • [develop] Add trimmomatic wrapper script for recent Trimmomatic 0.39 easyconfigs (PR #21704)
    • (noteworthy) new software
    • noteworthy software updates
      • ...
    • cleanup
      • [5.0.x] stop using modextrapaths to update $PYTHONPATH with standard path to installed Python packages (lib/python%(pyshortver)s/site-packages) (PR #20960)
    • changes
      • [5.0.x] use pycodestyle for code style check + stop using l in list comprehensions (PR #21502)
    • EasyBuild 5.0
      • [5.0.x] sync with develop (20240930) (PR #21497)

Open (active) PRs

  • docs (open PRs + issues)

    • ...

  • framework (open PRs + issues)

    • bug fixes
      • [5.0.x] Reverse order for parsing files in $XDG_CONFIG_DIRS (PR #4630)
      • [5.0.x] improve portability of reproducible tarballs by replacing external tar command with tarfile module (PR #4660)
        • to fix issue #4657: reproducible tarballs have unexpected checksums on mac
        • tarfile module in Python allows creating tarballs in a reproducible way across systems (Linux + macOS)
          • but there's a breaking change in Python 3.9, so we get different tarballs across different Python versions :man-facepalming:
          • two options to deal with this:
            • only support creating reproducible tarballs with Python 3.9+
              • people using Python < 3.9 would have to use --ignore-checksums
              • EasyBuild 5.0 could print a warning when using Python < 3.9 that checksums can not be verified?
              • this seems like the best way forward?
              • we should probably consider deprecating support for Python < 3.9 in EasyBuild 5.0
            • always include two checksums for source tarballs created via git_config: one for Python < 3.9, one for Python >= 3.9
          • binary patching generated tarballs with older Python versions would be very difficult
      • get_os_name should be updated to include Rocky and Alma (PR #4663)
    • enhancements
      • [5.0.x] disable strict RPATH sanity check by default, allow re-enabling it via --strict-rpath-sanity-check configuration option (PR #4475)
      • [develop] Copy build log and artifacts to a permanent location after failures (WIP PR #4601)
      • [develop] allow extensions to add text to the module file (PR #4652)
      • Problem using $CPATH in modulefiles overwriting system paths (issue #3331)
        • [5.0.x] add --search-path-cpp-headers configuration option to control how EasyBuild sets paths to headers at build time (PR #4645)
        • [5.0.x] refactor generation of required environment variables in module files (PR #4653)
          • test was added to verify that making ModuleLoadEnvironment a singleton causes trouble
          • needs more love, additional test is broken
          • easyblocks will need to be updated accordingly to not use make_module_req_guess anymore
        • [5.0.x] add module-search-path-headers configuration option to control how modules set search paths to header files (PR #4655)
      • [develop] add support for injecting checksums for cargo crates (PR #4661)
      • [develop] add support for %(rpath)s template value (PR #4670)
        • this makes it even more important that we merge the PR that ensures that templates are resolved
    • changes
      • [5.0.x] Rename SOURCE_STEP to EXTRACT_STEP (PR #4629)
      • [5.0.x] Deprecate use of parallel easyconfig parameter and fix updating the template value (PR #4580)
      • [5.0.x] Deprecate post_install_step, rename it to post_processing_step (issue #4656)
      • [5.0.x] proposed changest to add a timeout to run_shell_cmd (PR #4665)
      • [5.0.x] include path to cmd.sh script in output generated by run_shell_cmd when a command fails + use colors: red for ERROR line, yellow for path to output files + cmd.sh script (PR #4666)
      • [develop] Added possibility to call amend/try-amend multiple times from easystack (PR #4667)
      • [5.0.x] Let jobs retweak easyconfigs themselves (draft PR #4669)
      • [5.0.x] don't raise error when required extensions are not found when installing extensions in parallel (draft PR #4671)
      • [5.0.x] mark support for installing extensions in parallel as being mature, since it's no longer experimental (PR #4672)
      • [5.0.x] mark easystack support as being mature, since it's no longer experimental (PR #4673)

  • easyblocks (open PRs + issues)

    • bug fixes
      • [develop] update Kokkos CPU mapping for LAMMPS >= 2Aug2023 to use icx on Intel Icelake systems #3321 (PR #3321)
      • [5.0.x] run pip check only once for PythonBundle (PR #3432)
      • [develop] Avoid failure in Python sanity check when PIP_REQUIRE_VIRTUALENV is set (PR #3460)
      • [develop] let CMakeMake easyblock also set Python_EXECUTABLE option, as well as Python3_EXECUTABLE and Python2_EXECUTABLE derivatives (when appropriate) (PR #3463)
      • [develop] take into account that f90wrap_version may be None in MetalWalls easyblock (PR #3468)
      • [develop] fix setting of CMAKE_OPTIONS for SuiteSparse versions between 5.1.2 and 6.0.0 (PR #3471)
    • enhancements
      • [develop] Set CMake hints for path to Python installation if it's direct or indirect dependency (when using CMake >= 3.12) (PR #3282)
      • [develop -> 5.0.x] enhance LLVM easyblock for compilation of clang/flang + other llvm-projects (PR #3373)
      • [develop] ignore user python packages by using -s option when running Python code via "python -c" (PR #3440)
      • [develop] enhance generic Bundle EasyBlock to transfer module requirements of components (PR #3472)
    • updates
      • [develop] update custom easyblock for CP2K >= v2024 (PR #3433)
    • changes
      • ...
    • new
      • [develop] custom easyblock for DeepSpeed (PR #3450)

  • easyconfigs (open PRs + issues)

    • bug fixes/reports
      • [develop -> 5.0.x] switch to CargoPythonBundle easyblock for orjson 3.9.15 (PR #21392)
        • requires being able to produce reproducible source tarballs from Git repository (WIP in 5.0.x branch of EasyBuild framework)
      • [5.0.x] promote pybind11 to runtime dependency for CPPE (to fix pip check failure) (PR #21564)
      • [develop] add patch to SciPy-bundle 2024.05 that fixes test failure on aarch64 (PR #21559)
    • enhancements
      • [develop] ignore user-installed packages when running python in sanitycheck (PR #21362)
      • [develop] also install w90spn2spn.x + w90chk2chk.x for Wannier90 3.1.0 (PR #21383)
    • (noteworthy) new software
      • ...
    • software updates
    • changes
      • [5.0.x] Replace parallel by maxparallel (PR #19375)
      • [develop] GTDB-Tk v2.4.0 should use skani and not FastANI (PR #21518)
        • (Åke) make sure that CPU boost is working on your systems if people are using this...

2024b common toolchains

  • GCC 14.2 as a base (see easyconfigs PR #21114)
  • we should define candidate toolchains using latest version of all components
    • are newer versions of Intel components compared to intel/2024a already available?
  • should we consider replacing intel with Clang-based toolchain as 2nd common toolchain next to foss?
    • Fortran frontend in Clang is becoming mature enough?

Q&A / others

  • NVHPC
    • missing library path for QC libraries, should be added to NVHPC easyblock
  • trouble with building Wannier90 with NVHPC (Fortran code)
  • "funny" case: infinite loop in spaln configure script (see https://github.com/ogotoh/spaln/issues/74)
  • Implement sanity check in EasyBuild for CUDA compute capabilities (see EESSI support issue #92)
    • should probably be implemented in EasyBuild, but as opt-in?
    • needs to be taken into account in easyblocks like LAMMPS that pick one of the specified CUDA compute capabilities
      • easyblock should "register" which CUDA compute capability it's actually using
      • which one is picked is inconsistent across easyblocks
  • (Alexandre) why are we still using Bazel 6.x (not 7.x)
    • there are several open PRs for Bazel 7.x
    • TensorFlow & friends may require specific (older) Bazel versions
  • (Jan) PR to implement make_module_req_guess in Bundle (see PR #3472)
    • should try and avoid re-creating easyblock instances
    • required for CodePlay SYCL, see easyconfigs PR #21582
  • (Alexander) upcoming PR for Cargo easyblock to change how it deal with git_config, would be good to test it throroughly
Clone this wiki locally