Installing Firedrake on HPC

[lindsayad]

I have repeatedly gotten this error during the h5py step of the install

  Traceback (most recent call last):
    File "/ibmstor/scratch/lindad/firedrake/lib/python3.10/site-packages/numpy/core/__init__.py", line 23, in <module>
      from . import multiarray
    File "/ibmstor/scratch/lindad/firedrake/lib/python3.10/site-packages/numpy/core/multiarray.py", line 10, in <module>
      from . import overrides
    File "/ibmstor/scratch/lindad/firedrake/lib/python3.10/site-packages/numpy/core/overrides.py", line 6, in <module>
      from numpy.core._multiarray_umath import (
  ImportError: libmpi.so.12: cannot open shared object file: No such file or directory

At first I tried using my HPC's installation of openmpi and passing --mpicc mpicc etc. to the the firedrake installation script and got this error. I then unloaded the openmpi module, such that which mpicc is empty, and then attempted the firedrake install again without passing --mpicc etc. (I think this leads to petsc downloading mpich) and got the same error. Any suggestions? Logs from the latter attempt are attached

configure.log
make.log
firedrake-install.log

[connorjward]

@lindsayad can you share the result of running ldd $VIRTUAL_ENV/lib/python3.10/site-packages/numpy/*.so (I may have gotten the path slightly wrong - we just need to see what libraries the compiled numpy objects are linking to).

@JDBetteridge do you have any ideas?

[JDBetteridge]

If you're running on HPC you almost certainly don't want to let PETSc build MPI. You want to load your preferred MPI module at installation time and whenever you activate the virtual environment.

As could you try running the command Connor mentioned above except you need to add one more bit to the end of the path:

ldd $VIRTUAL_ENV/lib/python3.10/site-packages/numpy/core/*.so

I'm not 100% sure why your numpy is linking against MPI in the first place, perhaps the BLAS/LAPACK on the system is linked against MPI.

[lindsayad]

Here is the requested output

lindad@sawtooth2:/scratch/lindad/firedrake/lib/python3.10/site-packages/numpy/core$ ldd *.so
_multiarray_tests.cpython-310-x86_64-linux-gnu.so:
        linux-vdso.so.1 (0x00007ffc5bdd9000)
        libmpi.so.12 => not found
        libc.so.6 => /lib64/libc.so.6 (0x00007fd3e0198000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fd3e055d000)
_multiarray_umath.cpython-310-x86_64-linux-gnu.so:
        linux-vdso.so.1 (0x00007ffe8538e000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f1ab5175000)
        libmpi.so.12 => not found
        libc.so.6 => /lib64/libc.so.6 (0x00007f1ab4db0000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f1ab591e000)
_operand_flag_tests.cpython-310-x86_64-linux-gnu.so:
        linux-vdso.so.1 (0x00007ffc4c7d4000)
        libmpi.so.12 => not found
        libc.so.6 => /lib64/libc.so.6 (0x00007f05518bb000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f0551c80000)
_rational_tests.cpython-310-x86_64-linux-gnu.so:
        linux-vdso.so.1 (0x00007fffb7387000)
        libmpi.so.12 => not found
        libc.so.6 => /lib64/libc.so.6 (0x00007f5570f8a000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f557134f000)
_simd.cpython-310-x86_64-linux-gnu.so:
        linux-vdso.so.1 (0x00007ffcac997000)
        libmpi.so.12 => not found
        libc.so.6 => /lib64/libc.so.6 (0x00007ff4bb3e6000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ff4bb7ab000)
_struct_ufunc_tests.cpython-310-x86_64-linux-gnu.so:
        linux-vdso.so.1 (0x00007ffd1dbe9000)
        libmpi.so.12 => not found
        libc.so.6 => /lib64/libc.so.6 (0x00007f91766e7000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f9176aac000)
_umath_tests.cpython-310-x86_64-linux-gnu.so:
        linux-vdso.so.1 (0x00007ffda136b000)
        libmpi.so.12 => not found
        libc.so.6 => /lib64/libc.so.6 (0x00007eff2c79a000)
        /lib64/ld-linux-x86-64.so.2 (0x00007eff2cb5f000)

If you're running on HPC you almost certainly don't want to let PETSc build MPI. You want to load your preferred MPI module at installation time

Indeed. I tried that first and hit the same error message. I will re-attempt with my HPC mpi modules and attach the resulting logs

[lindsayad]

with my openmpi3 module loaded

/scratch/lindad/firedrake/lib/python3.10/site-packages/numpy/core/_multiarray_tests.cpython-310-x86_64-linux-gnu.so:
        linux-vdso.so.1 (0x00007f0db74ea000)
        libmpi.so.12 => not found
        libc.so.6 => /lib64/libc.so.6 (0x00007f0db6ef9000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f0db72be000)
/scratch/lindad/firedrake/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-x86_64-linux-gnu.so:
        linux-vdso.so.1 (0x00007ffd99df2000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f648573c000)
        libmpi.so.12 => not found
        libc.so.6 => /lib64/libc.so.6 (0x00007f6485377000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f6485ee5000)
/scratch/lindad/firedrake/lib/python3.10/site-packages/numpy/core/_operand_flag_tests.cpython-310-x86_64-linux-gnu.so:
        linux-vdso.so.1 (0x00007ffedef38000)
        libmpi.so.12 => not found
        libc.so.6 => /lib64/libc.so.6 (0x00007f700bac4000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f700be89000)
/scratch/lindad/firedrake/lib/python3.10/site-packages/numpy/core/_rational_tests.cpython-310-x86_64-linux-gnu.so:
        linux-vdso.so.1 (0x00007ffc5391c000)
        libmpi.so.12 => not found
        libc.so.6 => /lib64/libc.so.6 (0x00007fdbf16fc000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fdbf1ac1000)
/scratch/lindad/firedrake/lib/python3.10/site-packages/numpy/core/_simd.cpython-310-x86_64-linux-gnu.so:
        linux-vdso.so.1 (0x00007ffccaf0c000)
        libmpi.so.12 => not found
        libc.so.6 => /lib64/libc.so.6 (0x00007f898af66000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f898b32b000)
/scratch/lindad/firedrake/lib/python3.10/site-packages/numpy/core/_struct_ufunc_tests.cpython-310-x86_64-linux-gnu.so:
        linux-vdso.so.1 (0x00007fffa2fc6000)
        libmpi.so.12 => not found
        libc.so.6 => /lib64/libc.so.6 (0x00007f38fc2c4000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f38fc689000)
/scratch/lindad/firedrake/lib/python3.10/site-packages/numpy/core/_umath_tests.cpython-310-x86_64-linux-gnu.so:
        linux-vdso.so.1 (0x00007ffd88df2000)
        libmpi.so.12 => not found
        libc.so.6 => /lib64/libc.so.6 (0x00007f037cd95000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f037d15a000)

firedrake-install.log
make.log
configure.log

[connorjward]

@JDBetteridge do you think that perhaps setting LD_LIBRARY_PATH (or some other magic environment variable) might help resolve this?

[JDBetteridge]

This is all a little bit complicated, as you can see from the mpicc command that PETSc thinks it's using:

/apps/spack/opt/gcc-8.5.0/gcc-10.5.0-5hebf6wwre324ruklrqmzo6hr3t7ub2a/bin/gcc -I/apps/spack/opt/gcc-10.5.0/openmpi-3.1.6-tbfphb7v5cjlsolrf62eilq3p2h33v5b/include -pthread -L/apps/spack/opt/gcc-10.5.0/hwloc-1.11.13-l3lci54bnci3yrojvevmpdvbvxm7mnqi/lib -L/apps/spack/opt/gcc-10.5.0/pmix-4.2.2-tgxt7mxqpfzacrjaoeid4xdjvxmwur2r/lib -L/opt/pbs/lib -L/apps/spack/opt/gcc-10.5.0/ucx-1.14.1-ehivo6c246kzne2n7pgebiarodbppgd4/lib -Wl,-rpath,/apps/spack/opt/gcc-8.5.0/gcc-10.5.0-5hebf6wwre324ruklrqmzo6hr3t7ub2a/lib/gcc/x86_64-pc-linux-gnu/10.5.0 -Wl,-rpath,/apps/spack/opt/gcc-8.5.0/gcc-10.5.0-5hebf6wwre324ruklrqmzo6hr3t7ub2a/lib64 -Wl,-rpath -Wl,/apps/spack/opt/gcc-10.5.0/hwloc-1.11.13-l3lci54bnci3yrojvevmpdvbvxm7mnqi/lib -Wl,-rpath -Wl,/apps/spack/opt/gcc-10.5.0/pmix-4.2.2-tgxt7mxqpfzacrjaoeid4xdjvxmwur2r/lib -Wl,-rpath -Wl,/opt/pbs/lib -Wl,-rpath -Wl,/apps/spack/opt/gcc-10.5.0/ucx-1.14.1-ehivo6c246kzne2n7pgebiarodbppgd4/lib -Wl,-rpath -Wl,/apps/spack/opt/gcc-10.5.0/openmpi-3.1.6-tbfphb7v5cjlsolrf62eilq3p2h33v5b/lib -L/apps/spack/opt/gcc-10.5.0/openmpi-3.1.6-tbfphb7v5cjlsolrf62eilq3p2h33v5b/lib -lmpi

We need to find out where libmpi.so.12 is, because it looks like your GCC compiler configuration has been tweaked to always link against MPI, whether you asked for that or not.

Could you also share the output of

ldd $VIRTUAL_ENV/lib/python**/site-packages/mpi4py/*.so

I hope and suspect it will contain a line that looks like:

libmpi.so.12 => /apps/spack/opt/gcc-10.5.0/openmpi-3.1.6-tbfphb7v5cjlsolrf62eilq3p2h33v5b/lib/libmpi.so.12

If so I will share a method for building numpy that should work on this system

[JDBetteridge]

You should be able to run firedrake-update now and the installation will complete.

but we don't appear to be linking to fblaslapack

You might be lucky, if the BLAS/LAPACK being used is ABI compatible everything will be fine. If they aren't it will be pretty obvious when the tests are run and we'll try rebuilding numpy again.

[lindsayad]

It seems that firedrake-update does not do the same pinning of numpy as firedrake-install. ☹️ This led to an auto-update of numpy to 2.0 that led to an error in h5py related to mismatching data types

[lindsayad]

So I've learned that I need to run firedrake-install all the way to the point of libmpi.so linking failure. I naively tried to stop the firedrake-install script at the point that numpy was installing to do the manual numpy build and then follow with firedrake-update but that led to multiple issues. First was that firedrake-update relies on petsc already being cloned. After manually cloning the second issue was missing petscvariables

[JDBetteridge]

It seems that firedrake-update does not do the same pinning of numpy as firedrake-install. ☹️ This led to an auto-update of numpy to 2.0 that led to an error in h5py related to mismatching data types

Not quite. We recently merged #3546 which unpins numpy in both install and update (it's a feature not a bug). At this point you probably need to run firedrake-update --rebuild to update the compiled dependencies in the stack.

[lindsayad]

Thanks @JDBetteridge. I was wondering whether --download-cmake should be removed from Linux PETSc configure since libsupermesh also depends on cmake which forces the user to have a cmake available in their environment? I guess the cmake version requirement for PETSc could be higher than for libsupermesh. I only ask because I unloaded cmake from my module environment when I saw that PETSc was building it and then discovered later that libsupermesh needed it

[lindsayad]

When running firedrake-update, I am running into issues with firedrake finding a libsupermesh header

    firedrake/cython/supermeshimpl.c:777:10: fatal error: libsupermesh-c.h: No such file or directory
      777 | #include "libsupermesh-c.h"

firedrake-update.log

[JDBetteridge]

Ah, it looks like libsupermesh was cloned but not built the first time around, you can build it by running

cd $VIRTUAL_ENV/src/libsupermesh/build
cmake .. -DBUILD_SHARED_LIBS=ON
    -DCMAKE_INSTALL_PREFIX=$VIRTUAL_ENV \
    -DMPI_C_COMPILER= \
    -DMPI_CXX_COMPILER= \
    -DMPI_Fortran_COMPILER= \
    -DCMAKE_Fortran_COMPILER= \
    -DMPIEXEC_EXECUTABLE=
make
make install

You will have to set all the build flags to be the MPI compilers.

[lindsayad]

Alright, install completed successfully 🎉 Now at run-time there are some module import errors but I think this is an issue with one of our HPC python packages, so I'll work that out with them