Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate incompatibility of MUSIC with mpi4py version 4 #3095

Closed
JCGoran opened this issue Sep 25, 2024 · 2 comments · Fixed by #3251
Closed

Investigate incompatibility of MUSIC with mpi4py version 4 #3095

JCGoran opened this issue Sep 25, 2024 · 2 comments · Fixed by #3251
Labels
cython parallel Parallelisation related issues including MPI python

Comments

@JCGoran
Copy link
Collaborator

JCGoran commented Sep 25, 2024

In #3017 we pinned the version of mpi4py to <4 since MUSIC does not build properly under version 4 and above, review what can be done to make it compatible.
@JCGoran JCGoran added python parallel Parallelisation related issues including MPI cython labels Sep 25, 2024
@JCGoran
Copy link
Collaborator Author

JCGoran commented Oct 3, 2024

The traceback seems to be (taken from https://github.com/neuronsimulator/nrn/actions/runs/11139748696/job/30957040687):

cd ../pymusic; /home/runner/work/_temp/music-venv/bin/python -c \
"from Cython.Build import cythonize; \
cythonize('pymusic.pyx', verbose=1)"
cd ../pymusic; /home/runner/work/_temp/music-venv/bin/python -c \
"from Cython.Build import cythonize; \
cythonize('pybuffer.pyx', verbose=1)"
/home/runner/work/_temp/music-venv/lib/python3.13/site-packages/Cython/Compiler/Main.py:381: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /home/runner/work/_temp/MUSIC/pymusic/pybuffer.pxd
  tree = Parsing.p_module(s, pxd, full_module_name)
warning: pybuffer.pxd:8:0: The 'IF' statement is deprecated and will be removed in a future Cython version. Consider using runtime conditions or C macros instead. See https://github.com/cython/cython/issues/4310
/home/runner/work/_temp/music-venv/lib/python3.13/site-packages/Cython/Compiler/Main.py:381: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /home/runner/work/_temp/MUSIC/pymusic/pymusic.pxd
  tree = Parsing.p_module(s, pxd, full_module_name)
warning: pymusic.pxd:8:0: The 'IF' statement is deprecated and will be removed in a future Cython version. Consider using runtime conditions or C macros instead. See https://github.com/cython/cython/issues/4310
Compiling pybuffer.pyx because it changed.
[1/1] Cythonizing pybuffer.pyx
warning: pymusic.pxd:140:37: Declarations should not be declared inline.
warning: pymusic.pxd:141:37: Declarations should not be declared inline.
warning: pymusic.pxd:142:25: Declarations should not be declared inline.
warning: pymusic.pxd:143:25: Declarations should not be declared inline.
warning: pymusic.pxd:144:26: Declarations should not be declared inline.
warning: pymusic.pxd:145:51: Declarations should not be declared inline.
warning: pymusic.pxd:147:59: Declarations should not be declared inline.
warning: pymusic.pxd:149:59: Declarations should not be declared inline.
warning: pymusic.pxd:151:59: Declarations should not be declared inline.
warning: pymusic.pxd:154:59: Declarations should not be declared inline.
warning: pymusic.pxd:156:75: Declarations should not be declared inline.
warning: pymusic.pxd:158:59: Declarations should not be declared inline.
warning: pymusic.pxd:160:59: Declarations should not be declared inline.
warning: pymusic.pxd:162:71: Declarations should not be declared inline.
warning: music/pybuffer.pxd:8:0: The 'IF' statement is deprecated and will be removed in a future Cython version. Consider using runtime conditions or C macros instead. See https://github.com/cython/cython/issues/4310
Compiling pymusic.pyx because it changed.
[1/1] Cythonizing pymusic.pyx
make  install-recursive
make[2]: Entering directory '/home/runner/work/_temp/MUSIC/pymusic'
Making install in examples
make[3]: Entering directory '/home/runner/work/_temp/MUSIC/pymusic/examples'
make[4]: Entering directory '/home/runner/work/_temp/MUSIC/pymusic/examples'
make[4]: Nothing to be done for 'install-exec-am'.
 /usr/bin/mkdir -p '/opt/MUSIC/share/music-1.2.1/examples/python'
 /usr/bin/install -c -m 644 eventlogger.py event.music eventsource.py helloworld.music messagelogger.py message.music message.py messagesource.py receivers.py senders.py test.music testsink.py testsource.py '/opt/MUSIC/share/music-1.2.1/examples/python'
make[4]: Leaving directory '/home/runner/work/_temp/MUSIC/pymusic/examples'
make[3]: Leaving directory '/home/runner/work/_temp/MUSIC/pymusic/examples'
make[3]: Entering directory '/home/runner/work/_temp/MUSIC/pymusic'
/bin/bash ../libtool  --tag=CXX   --mode=compile mpicxx -DHAVE_CONFIG_H -I. -I..   -I/opt/hostedtoolcache/Python/3.13.0-rc.2/x64/include/python3.13 -I../src -I../src  -fno-strict-aliasing -g -O3 -g -O3 -std=c++11 -MT pymusic_la-pymusic.lo -MD -MP -MF .deps/pymusic_la-pymusic.Tpo -c -o pymusic_la-pymusic.lo `test -f 'pymusic.cpp' || echo './'`pymusic.cpp
/bin/bash ../libtool  --tag=CXX   --mode=compile mpicxx -DHAVE_CONFIG_H -I. -I..   -I/opt/hostedtoolcache/Python/3.13.0-rc.2/x64/include/python3.13 -I../src  -fno-strict-aliasing -g -O3 -g -O3 -std=c++11 -MT pybuffer_la-pybuffer.lo -MD -MP -MF .deps/pybuffer_la-pybuffer.Tpo -c -o pybuffer_la-pybuffer.lo `test -f 'pybuffer.cpp' || echo './'`pybuffer.cpp
libtool: compile:  mpicxx -DHAVE_CONFIG_H -I. -I.. -I/opt/hostedtoolcache/Python/3.13.0-rc.2/x64/include/python3.13 -I../src -I../src -fno-strict-aliasing -g -O3 -g -O3 -std=c++11 -MT pymusic_la-pymusic.lo -MD -MP -MF .deps/pymusic_la-pymusic.Tpo -c pymusic.cpp  -fPIC -DPIC -o .libs/pymusic_la-pymusic.o
libtool: compile:  mpicxx -DHAVE_CONFIG_H -I. -I.. -I/opt/hostedtoolcache/Python/3.13.0-rc.2/x64/include/python3.13 -I../src -fno-strict-aliasing -g -O3 -g -O3 -std=c++11 -MT pybuffer_la-pybuffer.lo -MD -MP -MF .deps/pybuffer_la-pybuffer.Tpo -c pybuffer.cpp  -fPIC -DPIC -o .libs/pybuffer_la-pybuffer.o
pybuffer.cpp:1709:3: error: ‘MPI_Session’ does not name a type
 1709 |   MPI_Session ob_mpi;
      |   ^~~~~~~~~~~
make[3]: *** [Makefile:564: pybuffer_la-pybuffer.lo] Error 1
make[3]: *** Waiting for unfinished jobs....
pymusic.cpp:1818:3: error: ‘MPI_Session’ does not name a type
 1818 |   MPI_Session ob_mpi;
      |   ^~~~~~~~~~~

Note that mpi4py 3.1 does not build properly OOTB on Python 3.13, however, thanks to this comment, it can be built manually with the following:

  • clone mpi4py branch maint-3.1
  • run ./conf/cython.sh from the top level dir
  • run pip install .

Unfortunately, this cannot be converted automatically to a simple pip install since there's no way to run a pre/post command with it, but it may be possible to modify the CI to do it (though the installation instructions would then need to be modified as well).

This was referenced Oct 3, 2024
Closed
Closed
@dalcinl
Copy link

dalcinl commented Oct 3, 2024

Folks, the build issue seems to be related to the MPI_Session type.
You are using an MPI implementation/version that does not support MPI 4, thus no MPI_Session
For a relatively easy workaround, see mpi4py/mpi4py#525 (comment).

@JCGoran JCGoran mentioned this issue Oct 8, 2024
Closed
@JCGoran JCGoran mentioned this issue Nov 26, 2024
Merged
@1uc 1uc closed this as completed in #3251 Dec 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
cython parallel Parallelisation related issues including MPI python
Projects
None yet
Milestone
No milestone
2 participants
@dalcinl @JCGoran