Skip to content

Conference call notes 20210915

Jump to bottom
Kenneth Hoste edited this page Sep 15, 2021 · 5 revisions

(back to Conference calls)

Notes on the 181st EasyBuild conference call, Wednesday Sept 15th 2021 (15:00 UTC)

Attendees

Alphabetical list of attendees (12):

  • Fotis Georgatos (SDSC, Switzerland)
  • Jasper Grimm (University of York, UK)
  • Alexander Grund (TU Dresden, Germany
  • Kenneth Hoste (HPC-UGent, Belgium)
  • Adam Huffman (Big Data Institute, Oxford, UK)
  • Daniel Lopez (CESGA, Spain)
  • Kurt Lust (Univ. of Antwerp, Belgium + LUMI User Support Team)
  • Mikael Öhman (Chalmers University of Technology, Sweden)
  • Bart Oldeman (Compute Canada)
  • Jörg Saßmannshausen (NIHR Biomedical Research Centre, UK)
  • Åke Sandgren (Umeå University, Sweden)
  • Davide Vanzo (Microsoft Azure)

Agenda

  • overview of recent developments
  • out-of-the-box performance of TensorFlow installations with EasyBuild (see easyblocks issue #2577)
  • Q&A

Recent developments

  • release timeline
    • latest release: EasyBuild v4.4.2 (Sept 7th 2021)
    • next release
  • recent changes
    • framework
      • bug fixes
        • make sure git clone with a tag argument actually downloads a tag (PR #3795)
        • correctly resolve templates for patches in extensions when uploading to GitHub (PR #3805)
        • ensure that path configuration options have absolute path values (PR #3832)
      • enhancements
        • Use Rich (if available) to show progress bar when installing easyconfigs (PR #3823 + PR #3826)
          • can be disabled via --disable-show-progress-bar or --output-style=basic
        • add --output-style configuration option, which can be used to disable use of Rich or type of any colored output (PR #3833)
        • add support for checking required/optional EasyBuild dependencies via 'eb --check-eb-deps' (PR #3829)
      • changes
        • ...
    • easyblocks
      • bug fixes
        • explicitly use only OpenBLAS for PyTorch if MKL is not used (PR #2448)
        • reset runtest to None in CMakePythonPackage (PR #2536)
        • adjust qa list in WIEN2k easyblock to handle new questions + fix sanity check for v21.1 (PR #2572)
      • enhancements
        • make sure AOCC commands like clang/flang pick up GCCcore as GCC toolchain (PR #2538)
        • update ABAQUS easyblock to add support for also installing Tosca component in ABAQUS (PR #2560) + support installing of version 2021 with hot fixes (PR #2562 + PR #2569)
        • update sanity check in VTune easyblock for 2021.x versions (PR #2563 + PR #2576)
        • let GROMACS runtime logs show EasyBuild was used (PR #2571)
      • new easyblocks
        • (none)
      • changes
        • (none)
    • easyconfigs
      • over 100 easyconfig PRs merged since last conf call!
      • bug fixes
        • fix ITK easyconfigs (PR #13619)
        • fix LAMMPS 3Mar2020 easyconfigs using intel toolchain on AMD CPUs by patching out hardcoded -xHost (PR #11577)
        • update to more recent Meson build dependency for GCCcore/8.3.0 easyconfigs to fix failing RPATH sanity check (PR #13910)
        • add patch for OpenBLAS 0.3.15 (for foss/2020b) to fix failing numpy tests on aarch64 (PR #13923
        • add UnZip build dependency to recent TensorFlow easyconfigs (PR #13958, PR #13959, PR #13960)
        • add additional valid checksum for lhs 1.1.1 extension in recent R 4.0.x and 4.1.0 easyconfigs (PR #13969)
        • add missing Brotli dependency to freetype 2.10.x (PR #13915)
      • enhancements
        • (nothing major)
      • new software
      • noteworthy software updates
      • changes
        • update to libjxl 0.5 + disable building of man pages (PR #13829)
        • rename CUDAcore to CUDA for v11.3.1 and v11.4.1 after merging foss/fosscuda, to ensure that get_software_root('CUDA') used in easyblocks works (PR #13874)
        • update CMake build dependency for Eigen 3.3.4 + 3.3.5 using system toolchain (PR #13889)
        • use GTK2 and GTK3 as dependencies rather than GTK+ for easyconfigs using GCC/10.3.0 toolchain (PR #13900)
        • remove superflous -DCMAKE_BUILD_TYPE=Release configuration option in various easyconfigs (PR #13615, #13614, PR #13621)
  • to merge/fix/tackle soon
    • framework
      • reported bugs / bug fixes
        • add optimal optimization flags for Intel compilers on AMD CPUs (issue #3793)
          • -xHost results in only using SSE2 on AMD systems...
          • see also (PR #3797
      • enhancements
        • avoid using a priority in prepend_module_path (Lmod) to avoid costly module calls (PR #3636)
        • add support for installing extensions in parallel (WIP) (PR #3667)
          • needs more test + a dedicated unit test
          • should be marked experimental at first?
        • finding modules with multiple modulepaths and HMNS (issue #3703)
        • filter out duplicate paths added to module files (PR #3770)
      • changes
        • fully drop support code for Python < 2.7 (PR #3788)
        • test suite improvements (PR #3790)
        • change copy_file function to raise an error when trying to copy non-existing file (PR #3836)
    • easyblocks
      • reported bugs / bug fixes
        • Prevent OpenMPI from failing on machines with less than 8 cores (PR #2570)
          • caused by sanity check that was added to OpenMPI easyblock recently (see PR #2444)
      • enhancements
        • add support for installing R extensions in parallel (WIP) (PR #2408)
        • enhance GCC easyblock to add support for AMD GPU offloading (PR #2578)
        • reduce the number of command line args for CMake configure (PR #2514)
        • enhance TensorFlow easyblock to take into account provided OpenSSL dependency (PR #2575)
        • enable MKL-DNN/oneDNN usage for aarch64 CPUs in TensorFlow 2.5+ (PR #2574)
      • changes
        • (nothing major?)
      • new software
        • (nothing major?)
    • easyconfigs

Common toolchains

2021b (WIP!)

  • for now: foss/2021.07 and intel/2021.07 (candidates for 2021b after testing confirms they work well)
    • foss/2021.07: included with EasyBuild v4.4.2 release
    • intel/2021.07: WIP at PR #13639
  • support for GCC 11.2 is expected in upcoming intel-compilers 2021.4 release...

TensorFlow performance

  • see easyblocks issue #2577
  • performance of TensorFlow is significantly slower compared to a pip installation (on quick benchmark runs)
    • boils down to difference in threading mode: OpenMP vs Eigen
    • workaround to remedy the issue: export OMP_NUM_THREADS=1 to avoid massive oversubscription of cores due to too many threads
    • too big a hammer to let TensorFlow modules set $OMP_NUM_THREADS to 1...
    • something to raise to Tensorflow SIG Build mailing list/meeting?

Q&A

  • Fotis: text user interface for EasyBuild?
    • s-lang (see Midnight Commander) or dialog
    • also see (new) Textual Python library
    • any interest in this?
  • Jörg: working on medaka (which requires TensorFlow v2.2.3) (see PR #...)
    • dependency for arctic, longread_umi, etc.
  • Bart: add support to FlexiBLAS easyblock to also support switching to MKL as backend
    • why was BLIS not added as a dependency for FlexiBLAS?
      • broken tests (see opened issue), and you can let FlexiBLAS use BLIS as a backend anyway by letting it pick up libblas.so
  • Mikael: NVHPC toolchain as proper toolchain?
    • should indicate CUDA version as a versionsuffix?
      • or not, and silently use a particular CUDA we control
    • toolchain = doesn't allow specifying a versionsuffix for the toolchain yet
    • Åke: NVHPC can also be installed without a CUDA component
      • and then CUDA can be used as a proper dependency
  • Mikael: building less documentation to avoid having to add (build) dependencies to Perl
    • we also break man in some OSs because of a huge $MANPATH (there's a hard limit of like 500 chars?)
      • limitation of 128 entries in $MANPATH
    • is adding documentation to our installations really useful?
    • Kurt: not installing man pages could mean that users see outdated man page if tool is also installed in the OS (e.g. man python)
    • Åke: stop adding man pages for stuff built with system toolchain?
    • do we need to add an EasyBuild configuration option to control installing man pages?
    • a setup where there's no perl in the OS is pretty weird
    • disabling building docs saves quite a bit of build time when building from scratch (no Perl build dep requires for flex, etc.)
    • EasyBuild could park build dependencies in a separate install directory, so they can be removed easily if desired
      • just need to make sure that stuff that's also a runtime dependency is installed in the right place (e.g. binutils)
Clone this wiki locally