make sure Python dependency included for ESPResSo is actually used by specifying `-DPYTHON_EXECUTABLE` #18963

maxim-masterov · 2023-10-10T09:03:14Z

Specify python in ESPResSo. See comment here.

boegel · 2023-10-10T10:11:04Z

Worth clarifying: this is required in EESSI because there we have a very recent "system" Python (3.11) in the compat layer, which CMake happily prefers over the specific Python dependency we're providing.

We saw the same problem with VTK (cfr. #16741), which suggests we should be automatically setting -DPython3_EXECUTABLE in the CMakeMake easyblock when Python is included as a dependency...

boegel · 2023-10-10T10:11:09Z

@boegelbot please test @ generoso

boegelbot · 2023-10-10T10:15:04Z

@boegel: Request for testing this PR well received on login1

PR test command 'EB_PR=18963 EB_ARGS= EB_CONTAINER= EB_REPO=easybuild-easyconfigs /opt/software/slurm/bin/sbatch --job-name test_PR_18963 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 11909

Test results coming soon (I hope)...

- notification for comment with ID 1754879789 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2023-10-10T11:59:50Z

Test report by @boegelbot
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
cns1 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/e3a1c3eb2799ec889ecfec4528c6639c for a full test report.

smoors · 2023-10-10T13:27:41Z

@boegelbot please test @ jsc-zen2

boegelbot · 2023-10-10T13:30:09Z

@smoors: Request for testing this PR well received on jsczen2l1.int.jsc-zen2.easybuild-test.cluster

PR test command 'EB_PR=18963 EB_ARGS= EB_REPO=easybuild-easyconfigs /opt/software/slurm/bin/sbatch --mem-per-cpu=4000M --job-name test_PR_18963 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen2.sh' executed!

exit code: 0
output:

Submitted batch job 3527

Test results coming soon (I hope)...

- notification for comment with ID 1755429652 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegel · 2023-10-10T16:09:37Z

The latest build failure in EESSI/software-layer#331 (comment) shows that only adding -DPython3_EXECUTABLE is not sufficient, CMake is still preferring Python 3.11 from compat layer...

So we need extra CMake options to make it behave.

boegel · 2023-10-10T16:26:56Z

From CMake output, which shows that Python 3.11 from compat layer is picked up, instead of the provided dependency...

-- Found PythonInterp: /cvmfs/pilot.eessi-hpc.org/versions/2023.06/compat/linux/aarch64/usr/bin/python3 (found suitable version "3.11.4", minimum required is "3.8")
-- Found Cython: /cvmfs/pilot.eessi-hpc.org/versions/2023.06/compat/linux/aarch64/usr/bin/python3;-m;cython (found suitable version "0.29.35", minimum required is "0.29.14")
-- Found PythonHeaders: /cvmfs/pilot.eessi-hpc.org/versions/2023.06/compat/linux/aarch64/usr/include/python3.11
Traceback (most recent call last):
  File "/cvmfs/pilot.eessi-hpc.org/versions/2023.06/software/linux/aarch64/neoverse_v1/software/SciPy-bundle/2022.05-foss-2022a/lib/python3.10/site-packages/numpy/core/__init__.py", line 23, in <module>
    from . import multiarray
  File "/cvmfs/pilot.eessi-hpc.org/versions/2023.06/software/linux/aarch64/neoverse_v1/software/SciPy-bundle/2022.05-foss-2022a/lib/python3.10/site-packages/numpy/core/multiarray.py", line 10, in <module>
    from . import overrides
  File "/cvmfs/pilot.eessi-hpc.org/versions/2023.06/software/linux/aarch64/neoverse_v1/software/SciPy-bundle/2022.05-foss-2022a/lib/python3.10/site-packages/numpy/core/overrides.py", line 6, in <module>
    from numpy.core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

Looks like we also need to use -DPython3_FIND_STRATEGY=LOCATION (the default is VERSION, which means preferring the most recent Python version that can be found), or by using -DCMAKE_POLICY_DEFAULT_CMP_0094=NEW (cfr. https://cmake.org/cmake/help/latest/policy/CMP0094.html).

It probably doesn't hurt to also specify -DPython3_ROOT_DIR=$EBROOTPYTHON, along with -DPython3_EXECUTABLE (I think it depends on the CMakeLists.txt with Python3_EXECUTABLE is actually honored at all).

See also https://cmake.org/cmake/help/latest/module/FindPython3.html

boegel · 2023-10-10T17:15:10Z

Since ESPResSo 4.2.1 uses find_package(PythonInterp 3.8 REQUIRED) (see here), apparently the magic keyword is -DPython_EXECUTABLE=... (cfr. https://cmake.org/cmake/help/latest/module/FindPython.html), which is not the same as -DPython3_EXECUTABLE=....

Yaay CMake!

boegel · 2023-10-10T17:18:53Z

Wait, -DPython_EXECUTABLE=... actually doesn't work, but -DPYTHON_EXECUTABLE=$EBROOTPYTHON/bin/python does work.

What...

…thon version used by CMake when building ESPResSo

boegel · 2023-10-10T17:51:47Z

Switched to -DPYTHON_EXECUTABLE in 66cf178 ...

branfosj · 2023-10-10T18:11:35Z

Wait, -DPython_EXECUTABLE=... actually doesn't work, but -DPYTHON_EXECUTABLE=$EBROOTPYTHON/bin/python does work.

What...

CMake find_package(PackageName) looks for a Find<PackageName>.cmake (first inside CMAKE_MODULE_PATH and then in the CMake install).

ESPResSo 4.2 uses find_package(PythonInterp 3.8 REQUIRED). So that means FindPythonInterp and PYTHON_EXECUTABLE.

The latest ESPResSO code uses find_package(Python 3.9 REQUIRED COMPONENTS Interpreter Development NumPy). So that means FindPython and Python_EXECUTABLE.

boegel · 2023-10-10T19:04:24Z

@branfosj Very little room for confusion there... Thanks for clearing that up! puts 🥃 away

boegel · 2023-10-11T06:45:14Z

@boegelbot please test @ generoso

boegel

tests in EESSI/software-layer#331 show that fix is working as intended (failure for aarch64/neoverse_v1 is due to known flaky tests in ESPResSo test suite)

boegelbot · 2023-10-11T06:55:04Z

@boegel: Request for testing this PR well received on login1

PR test command 'EB_PR=18963 EB_ARGS= EB_CONTAINER= EB_REPO=easybuild-easyconfigs /opt/software/slurm/bin/sbatch --job-name test_PR_18963 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 11918

Test results coming soon (I hope)...

- notification for comment with ID 1756944034 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2023-10-11T08:37:59Z

Test report by @boegelbot
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
cns1 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/c147c38da9d9a84117a48d224f119e09 for a full test report.

boegel · 2023-10-11T08:52:52Z

Test report by @boegel
FAILED
Build succeeded for 3 out of 4 (4 easyconfigs in total)
node3301.joltik.os - Linux RHEL 8.8, x86_64, Intel(R) Xeon(R) Gold 6242 CPU @ 2.80GHz (cascadelake), 1 x NVIDIA Tesla V100-SXM2-32GB, 535.104.12, Python 3.6.8
See https://gist.github.com/boegel/558e77e2b8f86d070df2a714dc540ef0 for a full test report.

maxim-masterov · 2023-10-11T09:09:29Z

The last build failed due to existing lock:

/user/gent/400/vsc40023/eb_scratch/RHEL8/cascadelake-volta-ib/software/.locks/_user_gent_400_vsc40023_eb_scratch_RHEL8_cascadelake-volta-ib_software_ESPResSo_4.2.1-foss-2022a.lock already exists, aborting!

boegel · 2023-10-11T10:10:36Z

Test report by @boegel
SUCCESS
Build succeeded for 1 out of 1 (4 easyconfigs in total)
node3301.joltik.os - Linux RHEL 8.8, x86_64, Intel(R) Xeon(R) Gold 6242 CPU @ 2.80GHz (cascadelake), 1 x NVIDIA Tesla V100-SXM2-32GB, 535.104.12, Python 3.6.8
See https://gist.github.com/boegel/e4f5d339cd22c96f3c0d398e24a77ee9 for a full test report.

boegel · 2023-10-11T18:15:25Z

Going in, thanks @maxim-masterov!

Specify python in ESPResSo

7ba0007

maxim-masterov added update EESSI Related to EESSI project labels Oct 10, 2023

This was referenced Oct 10, 2023

{2023.06}[foss/2022a] ESPResSo v4.2.1 EESSI/software-layer#331

Merged

{2023.06}[foss/2021a] ESPResSo v4.2.1 EESSI/software-layer#332

Closed

boegel changed the title ~~{chem}[foss/2021a, foss/2022a] ESPResSo v4.2.1, UPD~~ make sure Python dependency included for ESPResSo is actually used by specifying -DPython3_EXECUTABLE Oct 10, 2023

boegel added bug fix and removed update labels Oct 10, 2023

boegel added this to the next release (4.8.2?) milestone Oct 10, 2023

use -DPYTHON_EXECUTABLE instead of -DPython3_EXECUTABLE to control Py…

66cf178

…thon version used by CMake when building ESPResSo

boegel changed the title ~~make sure Python dependency included for ESPResSo is actually used by specifying -DPython3_EXECUTABLE~~ make sure Python dependency included for ESPResSo is actually used by specifying -DPYTHON_EXECUTABLE Oct 11, 2023

boegel approved these changes Oct 11, 2023

View reviewed changes

boegel merged commit e5df6cd into easybuilders:develop Oct 11, 2023
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

make sure Python dependency included for ESPResSo is actually used by specifying `-DPYTHON_EXECUTABLE` #18963

make sure Python dependency included for ESPResSo is actually used by specifying `-DPYTHON_EXECUTABLE` #18963

maxim-masterov commented Oct 10, 2023

boegel commented Oct 10, 2023

boegel commented Oct 10, 2023

boegelbot commented Oct 10, 2023

boegelbot commented Oct 10, 2023

smoors commented Oct 10, 2023

boegelbot commented Oct 10, 2023

boegel commented Oct 10, 2023

boegel commented Oct 10, 2023

boegel commented Oct 10, 2023

boegel commented Oct 10, 2023

boegel commented Oct 10, 2023

branfosj commented Oct 10, 2023 •

edited

Loading

boegel commented Oct 10, 2023

boegel commented Oct 11, 2023

boegel left a comment

boegelbot commented Oct 11, 2023

boegelbot commented Oct 11, 2023

boegel commented Oct 11, 2023

maxim-masterov commented Oct 11, 2023 •

edited

Loading

boegel commented Oct 11, 2023

boegel commented Oct 11, 2023

make sure Python dependency included for ESPResSo is actually used by specifying -DPYTHON_EXECUTABLE #18963

make sure Python dependency included for ESPResSo is actually used by specifying -DPYTHON_EXECUTABLE #18963

Conversation

maxim-masterov commented Oct 10, 2023

boegel commented Oct 10, 2023

boegel commented Oct 10, 2023

boegelbot commented Oct 10, 2023

boegelbot commented Oct 10, 2023

smoors commented Oct 10, 2023

boegelbot commented Oct 10, 2023

boegel commented Oct 10, 2023

boegel commented Oct 10, 2023

boegel commented Oct 10, 2023

boegel commented Oct 10, 2023

boegel commented Oct 10, 2023

branfosj commented Oct 10, 2023 • edited Loading

boegel commented Oct 10, 2023

boegel commented Oct 11, 2023

boegel left a comment

Choose a reason for hiding this comment

boegelbot commented Oct 11, 2023

boegelbot commented Oct 11, 2023

boegel commented Oct 11, 2023

maxim-masterov commented Oct 11, 2023 • edited Loading

boegel commented Oct 11, 2023

boegel commented Oct 11, 2023

make sure Python dependency included for ESPResSo is actually used by specifying `-DPYTHON_EXECUTABLE` #18963

make sure Python dependency included for ESPResSo is actually used by specifying `-DPYTHON_EXECUTABLE` #18963

branfosj commented Oct 10, 2023 •

edited

Loading

maxim-masterov commented Oct 11, 2023 •

edited

Loading