Conference call notes 20191113

(back to Conference calls)

Notes on the 136th EasyBuild conference call, Wednesday Nov 13th 2019 (17:00 - 18:00 CET)

Attendees

Alphabetical list of attendees (7):

• Fotis Georgatos (SDSC, Switzerland)
• Kenneth Hoste (HPC-UGent, Belgium)
• Adam Huffman (Big Data Institute, Oxford)
• Eirini Koutsaniti (CSCS, Switzerland)
• Mikael Öhman (Chalmers University of Technology, Sweden)
• Davide Vanzo (UT Southwestern)
• Victor Holanda Rusu (CSCS, Switzerland)

Agenda

• update on recent developments for next release (EasyBuild v4.1.0)
• pros & cons of GitHub Actions vs Travis CI for running test suite
• EasyBuild @ SC19
• EUM'20
• Q&A

Update on recent developments

• next release (EasyBuild v4.1.0) planned for end of Nov'19

• intention is to let Miguel take care of the actual release (with support from @boegel)

• significant changes in framework (https://github.com/easybuilders/easybuild-framework/milestone/73)

  • significant speedup in easyconfig test suite

    • thanks to https://github.com/easybuilders/easybuild-framework/pull/3074 + https://github.com/easybuilders/easybuild-framework/pull/3073 (latter only relevant for Python 3.x)
    • test suite now runs in 7-8 mins, used to be ~1h with Python 3.x...

  • deprecating use of:

    • Lmod 6.x (https://github.com/easybuilders/easybuild-framework/pull/3077)
    • Python 2.6 (https://github.com/easybuilders/easybuild-framework/pull/3076)
    • mostly motivated by wanting to trim test configurations in Travis/GitHub a bit

  • fixed minor compatibility issues with using EasyBuild on top of Python 3.8

    • https://github.com/easybuilders/easybuild-framework/pull/3078

  • better integration with modules provided via packages in OpenHPC

    • first step: https://github.com/easybuilders/easybuild-framework/pull/3075
      • OpenHPC maintainers are planning to include a metadata files per RPM which EasyBuild can pick up on
    • more minor changes needed in order to use OpenHPC-provided modules as toolchain components

• significant changes in easyblocks (https://github.com/easybuilders/easybuild-easyblocks/milestone/62)

  • also run easyconfigs test suite in GitHub Actions
    • https://github.com/easybuilders/easybuild-easyblocks/pull/1844

• significant changes in easyconfigs (https://github.com/easybuilders/easybuild-easyconfigs/milestone/65)

  • fixed definition of fosscuda/2019b for use in hierarchical module naming scheme

    • w.r.t. location of CUDA in module hierarchy (from Core/ to Compiler/*) + correct subtoolchain (not GCCcore)

  • also run easyconfigs test suite in GitHub Actions

    • https://github.com/easybuilders/easybuild-easyconfigs/pull/9231 + https://github.com/easybuilders/easybuild-easyconfigs/pull/9255

GitHub Actions vs Travis CI for running test suite

• Travis

  • pros
    • has worked well for a long time (in combination with @boegelbot)
    • automatic comments by @boegelbot to indicate broken tests in PR
  • cons
    • flaky test failures (usually due to connectivity issues)
      • mostly fixed by letting @boegelbot re-trigger these jobs in Travis
    • slow, only 5-6 test runs in parallel (across all easybuilders repositories)

• GitHub Actions

  • pros
    • 20 test runs in parallel, so results come in a lot faster
    • native in GitHub
    • easily allows contributors to get test suite to pass first in their own fork before opening a PR
  • cons
    • (still in beta, but going stable mid Nov'19)
    • minor problem: closing/re-opening triggers a new batch of tests, old batch of (broken) tests in GitHub CI stays in place
      • can be dealt with by implementing support for eb --sync-pr-with-develop?

• self-hosted runners for GitHub Actions?

  • supported (see https://github.blog/2019-11-05-self-hosted-runners-for-github-actions-is-now-in-beta/), but not recommended for repositories with forks for security reasons...
  • workaround could be to implement own webhook which can be triggered by events on GitHub

EUM

• Jan 29th-31st 2020 in Barcelona (Spain)
• https://github.com/easybuilders/easybuild/wiki/5th-EasyBuild-User-Meeting

EasyBuild @ SC19

• "Getting Scientific Software Installed" BoF session hosted by Davide Vanzo (@vanzod) & Robert McLay (Lmod/XALT)
  • short talk by Maxime (ComputeCanada) on their CVMFS+Nix+EasyBuild setup (PEARC'19 paper)
  • interactive survey on tools related to software installation
• get in touch with EasyBuilders attending SC19 via #sc19 channel in EasyBuild Slack
• Davide will put together a social event (@ arcade bar)

Q&A

• duplicate CUDA in fosscuda vs intelcuda
  • CUDAcore with system toolchain (hidden from users in ComputeCanada)
  • (empty) CUDA wrapper which loads CUDAcore & extends $MODULEPATH (in HMNS)
  • this didn't go happen for fosscuda/2019b
    • Kenneth was reluctant to go through with it, not a big fan of the *core naming scheme (confuses users)
  • Bart had another suggestion (which doesn't use CUDAcore)?
• Victor & Eirini: aim at CSCS is to fully automate software deployment
  • upgrading of software version + toolchain + dependencies
  • Python scripts to automate this
    • files_to_copy sometimes disappears when dumping easyconfig files
      • see open issue: https://github.com/easybuilders/easybuild-framework/issues/3049
    • comments move around to weird places (open issue?)
  • anyone else working on this?
  • any progress on integration easy_update in EasyBuild framework?
    • see https://github.com/fizwit/easy_update
    • no progress there due to lack of time...
  • how can we contribute back this functionality?
  • Fotis: release monitoring in Fedora could be useful
• Victor: how are test reports for PRs triggered?
  • currently done manually (via eb --from-pr 12345 -fr --upload-test-report, see https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html)
  • this could be automated, but needs some work (keeping security concerns in mind)
    • PRs that pass CI tests & are approved by a maintainer
    • VM could be spin op on demand for this
  • PRs that depend on other PRs could be re-visited automatically via scripting in GitHub Actions?