From 16e8366ab57953e224291e4d30be74058956a9f7 Mon Sep 17 00:00:00 2001 From: Shane Canon Date: Sat, 11 May 2024 13:18:57 +0200 Subject: [PATCH] Update 14.hpc --- _episodes/14.hpc.md | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/_episodes/14.hpc.md b/_episodes/14.hpc.md index 34ba474..48e3b1c 100644 --- a/_episodes/14.hpc.md +++ b/_episodes/14.hpc.md @@ -34,11 +34,13 @@ Key concepts around the setup for running MPI and GPU enabled containers with Si ### Configure the MPI/interconnect bind approach -Before we start, let us cd to the `openfoam` example directory and set-up the MPI to use: +Before we start, let us cd to the `openfoam` example directory +and adjust the PATH: ```bash cd ~/isc-tutorial/exercises/openfoam -source set-mpi.sh +OLDPATH=PATH +PATH=/opt/intel/oneapi/mpi/latest/bin:$PATH ``` Now, suppose you have an MPI installation in your host and a containerised MPI application, built upon MPI libraries that are ABI compatible with the former. @@ -49,7 +51,7 @@ For this tutorial, we do have MPICH installed on the host machine: which mpirun ``` ```output -/opt/intel/oneapi/intelpython/latest/bin/mpirun +/opt/intel/oneapi/mpi/latest/bin/mpirun ``` and we're going to pull an OpenFoam container, which was built on top of MPICH as well: @@ -58,13 +60,13 @@ and we're going to pull an OpenFoam container, which was built on top of MPICH a singularity pull library://marcodelapierre/beta/openfoam:v2012 ``` -OpenFoam comes with a collection of executables, one of which is `simpleFoam`. We can use the Linux command `ldd` to investigate the libraries that this executable links to. As `simpleFoam` links to a few tens of libraries, let's specifically look for MPI (`libmpi*`) libraries in the command output: +Note that this image has already been pulled on the tutorial VMs. OpenFoam comes with a collection of executables, one of which is `simpleFoam`. We can use the Linux command `ldd` to investigate the libraries that this executable links to. As `simpleFoam` links to a few tens of libraries, let's specifically look for MPI (`libmpi*`) libraries in the command output: ```bash singularity exec openfoam_v2012.sif bash -c 'ldd $(which simpleFoam) |grep libmpi' ``` ```output -libmpi.so.12 => /usr/lib/libmpi.so.12 (0x00007f73a729b000) +libmpi.so.12 => /usr/lib/libmpi.so.12 (0x00007fa0836e6000) ``` This is the container MPI installation that was used to build OpenFoam. @@ -74,7 +76,7 @@ We can make use of Singularity-specific environment variables, to make these hos ```bash export SINGULARITY_BINDPATH="/opt/intel/oneapi" -export SINGULARITYENV_LD_LIBRARY_PATH="/opt/intel/oneapi/intelpython/latest/lib:/opt/intel/oneapi/intelpython/latest/lib/libfabric:\$LD_LIBRARY_PATH" +export SINGULARITYENV_LD_LIBRARY_PATH="/opt/intel/oneapi/mpi/latest/lib:/opt/intel/oneapi/mpi/latest/lib/libfabric:\$LD_LIBRARY_PATH" ``` Note the "escaping" of the dollar sign at the end. This is to keep the explicit name of the `LD_LIBRARY_PATH` within the definition (and not the evaluation of its value in the host). @@ -85,14 +87,14 @@ Note the "escaping" of the dollar sign at the end. This is to keep the explicit > echo $SINGULARITYENV_LD_LIBRARY_PATH > ``` > ``` output -> /opt/intel/oneapi/intelpython/latest/lib:/opt/intel/oneapi/intelpython/latest/lib/libfabric:$LD_LIBRARY_PATH +> /opt/intel/oneapi/mpi/latest/lib:/opt/intel/oneapi/mpi/latest/lib/libfabric:$LD_LIBRARY_PATH > ``` > Then, the evaluation of `LD_LIBRARY_PATH` within the container has, at the beginning, the additional path for the libraries in the host (as intended). And the rest is the original definition of the variable within the container (as intended). > ```bash > singularity exec openfoam_v2012.sif bash -c 'echo $LD_LIBRARY_PATH' > ``` > ``` output -> /opt/intel/oneapi/intelpython/latest/lib:/opt/intel/oneapi/intelpython/latest/lib/libfabric:/opt/OpenFOAM/ThirdParty-v2012/platforms/linux64Gcc/fftw-3.3.7/lib64:/opt/OpenFOAM/ThirdParty-v2012/platforms/linux64Gcc/CGAL-4.12.2/lib64:/opt/OpenFOAM/ThirdParty-v2012/platforms/linux64Gcc/boost_1_66_0/lib64:/opt/OpenFOAM/ThirdParty-v2012/platforms/linux64Gcc/ADIOS2-2.6.0/lib:/opt/OpenFOAM/ThirdParty-v2012/platforms/linux64Gcc/ParaView-5.6.3/lib:/opt/OpenFOAM/OpenFOAM-v2012/platforms/linux64GccDPInt32Opt/lib/sys-mpi:/opt/OpenFOAM/ThirdParty-v2012/platforms/linux64GccDPInt32/lib/sys-mpi:/home/ofuser/OpenFOAM/ofuser-v2012/platforms/linux64GccDPInt32Opt/lib:/2012/platforms/linux64GccDPInt32Opt/lib:/opt/OpenFOAM/OpenFOAM-v2012/platforms/linux64GccDPInt32Opt/lib:/opt/OpenFOAM/ThirdParty-v2012/platforms/linux64GccDPInt32/lib:/opt/OpenFOAM/OpenFOAM-v2012/platforms/linux64GccDPInt32Opt/lib/dummy:/.singularity.d/libs:/.singularity.d/libs +> /opt/intel/oneapi/mpi/latest/lib:/opt/intel/oneapi/mpi/latest/lib/libfabric:/opt/OpenFOAM/ThirdParty-v2012/platforms/linux64Gcc/fftw-3.3.7/lib64:/opt/OpenFOAM/ThirdParty-v2012/platforms/linux64Gcc/CGAL-4.12.2/lib64:/opt/OpenFOAM/ThirdParty-v2012/platforms/linux64Gcc/boost_1_66_0/lib64:/opt/OpenFOAM/ThirdParty-v2012/platforms/linux64Gcc/ADIOS2-2.6.0/lib:/opt/OpenFOAM/ThirdParty-v2012/platforms/linux64Gcc/ParaView-5.6.3/lib:/opt/OpenFOAM/OpenFOAM-v2012/platforms/linux64GccDPInt32Opt/lib/sys-mpi:/opt/OpenFOAM/ThirdParty-v2012/platforms/linux64GccDPInt32/lib/sys-mpi:/home/ofuser/OpenFOAM/ofuser-v2012/platforms/linux64GccDPInt32Opt/lib:/2012/platforms/linux64GccDPInt32Opt/lib:/opt/OpenFOAM/OpenFOAM-v2012/platforms/linux64GccDPInt32Opt/lib:/opt/OpenFOAM/ThirdParty-v2012/platforms/linux64GccDPInt32/lib:/opt/OpenFOAM/OpenFOAM-v2012/platforms/linux64GccDPInt32Opt/lib/dummy:/.singularity.d/libs:/.singularity.d/libs > ``` > {: .solution} @@ -120,7 +122,7 @@ Now, if we inspect `mpirun` dynamic linking again: singularity exec openfoam_v2012.sif bash -c 'ldd $(which simpleFoam) |grep libmpi' ``` ```output -libmpi.so.12 => /opt/intel/oneapi/intelpython/latest/lib/libmpi.so.12 (0x00007f34aa9e3000) +libmpi.so.12 => /opt/intel/oneapi/mpi/latest/lib/libmpi.so.12 (0x00007f34aa9e3000) ``` Now OpenFoam is picking up the host MPI libraries!