Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pelahi askapingest #405

Open
wants to merge 37 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
aec1b20
Adding askap ingest slews12sp5 specific cygnets
Oct 2, 2020
8926e9b
Update to clang cynget for askap ingest
Oct 6, 2020
537deb3
update to askap ingest cygnets
Oct 8, 2020
eb7c78a
Updates to cygnets, some bug fixes to mvapich
Oct 9, 2020
140b21d
Few more bug fixes for mvapich cygnet related to module file
Oct 9, 2020
3c37b29
Update to askap ingest specific cygnets
Oct 13, 2020
1ebebfa
Update to casacore to use build options
pelahi Oct 13, 2020
3f4556b
Update to casacore to use build options
pelahi Oct 13, 2020
0f05dbe
Update to casacore to use build options
pelahi Oct 13, 2020
2c5275f
Update to casacore to use build options
pelahi Oct 13, 2020
09e4534
Update to ice cygnet
pelahi Oct 13, 2020
47cfd04
Update to ice cygnet
pelahi Oct 13, 2020
eedb250
Update to ice cygnet
pelahi Oct 13, 2020
107548b
Update to ice cygnet
pelahi Oct 13, 2020
4e85a4c
Added casarest cygnet
pelahi Oct 13, 2020
0967a7e
Update to casarest cygnet
pelahi Oct 13, 2020
781b8b8
Update to casarest cygnet
pelahi Oct 13, 2020
243714e
Update to casarest cygnet
pelahi Oct 13, 2020
67c192f
Update to cyngets to install modules in askap-devel
Oct 19, 2020
bb5dbce
Update to ice cygnet, adding build options
pelahi Oct 20, 2020
dcc9a73
Update to ice cygnet, changed to build both cpp11 and cpp98 by default
pelahi Oct 20, 2020
ef384cd
Update to askap cygnets, added some python ones
Oct 21, 2020
90976b8
Added AOCC cygnet, still alpha
Dec 11, 2020
0458568
Update to AMD compiler cygnet
pelahi Dec 11, 2020
79ab274
Adding adios cynget
pelahi Jan 20, 2021
1618bac
Added openmpi cygnet
pelahi Jan 20, 2021
08827bc
Merge branch 'pelahi-askapingest' of https://github.com/PawseySC/maal…
pelahi Jan 20, 2021
a45da69
Some minor updates
pelahi Jan 21, 2021
18202b4
Update to handle adios2
pelahi Jan 21, 2021
221e0c4
Update to adios cygnet
pelahi Jan 25, 2021
b3ce8ed
Added openblas cygnet
pelahi Jan 27, 2021
42d1565
Update to openblas cygnet
pelahi Jan 27, 2021
2497ef4
Minor update to openblas
pelahi Jan 27, 2021
1802893
Update to casacore, fixing the build options handling
pelahi Jan 28, 2021
1f6fb5d
Bug fix for openblas usage in casacore cygnet
pelahi Jan 28, 2021
41dbfee
Updates to casarest to match cascacore build options
pelahi Jan 29, 2021
e5df297
Update to casarest cygnet
pelahi Jan 29, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 92 additions & 0 deletions askapingest_sles12sp5/boost.cyg
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
##############################################################################
# maali cygnet file for Boost
##############################################################################

read -r -d '' MAALI_MODULE_WHATIS << EOF

Boost provides free peer-reviewed portable C++ source libraries.

We emphasize libraries that work well with the C++ Standard Library. Boost
libraries are intended to be widely useful, and usable across a broad spectrum
of applications. The Boost license encourages both commercial and
non-commercial use.

For further information see http://www.boost.org/

EOF

# Request only the latest GNU and Intel compilers to be used
MAALI_TOOL_COMPILERS="$MAALI_DEFAULT_COMPILERS"

# needed for this URL
MAALI_BOOST_VERSION=`echo $MAALI_TOOL_VERSION| sed -e 's/\./_/g'`

# URL to download the source code from
MAALI_URL="http://sourceforge.net/projects/$MAALI_TOOL_NAME/files/$MAALI_TOOL_NAME/$MAALI_TOOL_VERSION/"$MAALI_TOOL_NAME"_"$MAALI_BOOST_VERSION".tar.bz2/download"

# location we are downloading the source code to
MAALI_DST="$MAALI_SRC/"$MAALI_TOOL_NAME"_"$MAALI_BOOST_VERSION".tar.bz2"

# where the unpacked source code is located
MAALI_TOOL_BUILD_DIR="$MAALI_BUILD_DIR/"$MAALI_TOOL_NAME"_"$MAALI_BOOST_VERSION

# type of tool (eg. apps, devel, python, etc.)
MAALI_TOOL_TYPE="devel"

# tool pre-requisites
MAALI_TOOL_PREREQ="$MAALI_DEFAULT_PYTHON"

# for auto-building module files
MAALI_MODULE_SET_LD_LIBRARY_PATH=1
MAALI_MODULE_SET_LIBRARY_PATH=1
MAALI_MODULE_SET_CPATH=1
MAALI_MODULE_SET_C_INCLUDE_PATH=1
MAALI_MODULE_SET_CPLUS_INCLUDE_PATH=1
MAALI_MODULE_SET_BOOST_ROOT='$MAALI_APP_HOME'

##############################################################################

function maali_build {

cd "$MAALI_TOOL_BUILD_DIR"

#https://github.com/HIT-SCIR/pyltp/pull/193
if [ "$MAALI_TOOL_VERSION" == "1.66.0" ]; then
sed -i -e 's;return PyUnicode_Check(obj) ? _PyUnicode_AsString(obj) : 0;return (void *)(PyUnicode_Check(obj) ? _PyUnicode_AsString(obj) : 0);g' libs/python/src/converter/builtin_converters.cpp
fi

export MAALI_CORES=12
# Intel-compiler specifics
if [[ "$MAALI_COMPILER_NAME" == "intel"* ]]; then

if [ "$MAALI_TOOL_MINOR_VERSION" -ge 66 ]; then
maali_run "./bootstrap.sh --prefix=$MAALI_INSTALL_DIR --with-toolset=intel-linux"
maali_run "./b2 -j$MAALI_CORES install toolset=intel --prefix=$MAALI_INSTALL_DIR"
else
maali_run "./bootstrap.sh --prefix=$MAALI_INSTALL_DIR --with-toolset=intel-linux"
maali_run "./bjam -j$MAALI_CORES toolset=intel release"
maali_run "./bjam install --prefix=$MAALI_INSTALL_DIR"
fi

# GNU compiler specifics
else

if [ "$MAALI_TOOL_MINOR_VERSION" -ge 66 ]; then
maali_run "./bootstrap.sh --prefix=$MAALI_INSTALL_DIR --with-toolset=gcc"
maali_run "./b2 -j$MAALI_CORES install toolset=gcc --prefix=$MAALI_INSTALL_DIR"
else
maali_run "./bootstrap.sh --prefix=$MAALI_INSTALL_DIR --without-libraries=python"
maali_run "./bjam -j$MAALI_CORES toolset=gcc release"
maali_run "./bjam install --prefix=$MAALI_INSTALL_DIR"
fi

fi

if [ "$MAALI_TOOL_VERSION" == "1.64.0" ]; then
sed -i -e 's;#include <boost/serialization/array.hpp>;#include <boost/serialization/array_wrapper.hpp>;g;' $MAALI_INSTALL_DIR/include/boost/numeric/ublas/matrix.hpp
sed -i -e 's;#include <boost/serialization/array.hpp>;#include <boost/serialization/array_wrapper.hpp>;g;' $MAALI_INSTALL_DIR/include/boost/numeric/ublas/storage.hpp
fi

}

##############################################################################
166 changes: 166 additions & 0 deletions askapingest_sles12sp5/casacore.cyg
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
##############################################################################
# maali cygnet file for casacore
##############################################################################

read -r -d '' MAALI_MODULE_WHATIS << EOF

The casacore package contains the core libraries of the old AIPS++/CASA
package. This split was made to get a better separation of core libraries and
applications. CASA is now built on top of casacore.

For further information see http://casacore.github.io/casacore/

EOF

#list extra build options in MAALI_TOOL_BUILD_OPTIONS_AVAIL
CASACORE_WITH_OPENMP=1
CASACORE_WITH_PTHREADS=1
CASACORE_WITH_MPI=0
CASACORE_WITH_HDF5=1
CASACORE_WITH_FFTW=1
CASACORE_WITH_PYTHON=0
CASACORE_WITH_PYTHON3=0
MAALI_TOOL_BUILD_OPTIONS_AVAIL=("CASACORE_WITH_OPENMP=1:0", "CASACORE_WITH_PTHREADS=1:0", "CASACORE_WITH_MPI=0:1", "CASACORE_WITH_HDF5=1:0", "CASACORE_WITH_FFTW=1:0", "CASACORE_WITH_PYTHON=0:1", "CASACORE_WITH_PYTHON3=0:1")

# specify which compilers we want to build the tool with
MAALI_TOOL_COMPILERS="gcc/8.3.0 gcc/5.5.0 "

# URL to download the source code from
MAALI_URL="https://github.com/${MAALI_TOOL_NAME}/${MAALI_TOOL_NAME}/archive/v${MAALI_TOOL_VERSION}.tar.gz"

# location we are downloading the source code to
MAALI_DST="$MAALI_SRC/$MAALI_TOOL_NAME-$MAALI_TOOL_VERSION.tar.gz"

# type of tool (eg. apps, devel, python, etc.)
MAALI_TOOL_TYPE="askap-devel"

# tool build pre-requisites - not added to the module, only needed for building (loaded after MAALI_TOOL_PREREQ)
MAALI_TOOL_BUILD_PREREQ="cmake/3.18.2"

# tool pre-requisites
MAALI_TOOL_PREREQ="cfitsio/3450 wcslib/7.3 intel-mkl/19.0.5 boost/1.66.0"

#location of casacore cata
CASACORE_DATA=/pawsey/askapingest_sles12sp5/casacore-data

# add additional configure options
# here are the default cmake options, directories point to appropriate.
MAALI_TOOL_CONFIGURE=""
MAALI_TOOL_CONFIGURE+=' -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCMAKE_BUILD_TYPE=RELEASE -DDATA_DIR=${CASACORE_DATA} '
MAALI_TOOL_CONFIGURE+=' -DCMAKE_EXE_LINKER_FLAGS="-lpthread -lcurl" '
MAALI_TOOL_CONFIGURE+=' -DBLAS_LIBRARIES=${MKLROOT}/lib/intel64/libmkl_rt.so -DLAPACK_LIBRARIES=${MKLROOT}/lib/intel64/libmkl_rt.so '
MAALI_TOOL_CONFIGURE+=' -DCFITSIO_ROOT_DIR=${MAALI_CFITSIO_HOME} -DWCSLIB_ROOT_DIR=${MAALI_WCSLIB_HOME} '

# parse the build options, would be nice to make this a function
# the issue currently is the local scope of variables declared
CASACORE_BUILD_NAME=""
if [[ ${#MAALI_TOOL_BUILD_OPTIONS[@]} -gt 0 ]]; then
for i in ${MAALI_TOOL_BUILD_OPTIONS[@]}
do
opt=(${i//=/ })
optname=${opt[0]}
optval=${opt[1]}
if [[ "$optname" -eq "CASACORE_WITH_MPI" ]]; then
CASACORE_WITH_MPI=$optval
fi
if [[ "$optname" -eq "CASACORE_WITH_OPENMP" ]]; then
CASACORE_WITH_OPENMP=$optval
fi
if [[ "$optname" -eq "CASACORE_WITH_PTHREADS" ]]; then
CASACORE_WITH_PTHREADS=$optval
fi
if [[ "$optname" -eq "CASACORE_WITH_HDF5" ]]; then
CASACORE_WITH_HDF5=$optval
fi
if [[ "$optname" -eq "CASACORE_WITH_FFTW" ]]; then
CASACORE_WITH_FFTW=$optval
fi
if [[ "$optname" -eq "CASACORE_WITH_PYTHON" ]]; then
CASACORE_WITH_PYTHON=$optval
fi
if [[ "$optname" -eq "CASACORE_WITH_PYTHON3" ]]; then
CASACORE_WITH_PYTHON3=$optval
fi
done
fi

#now parse build options, set build name
if [[ $CASACORE_WITH_OPENMP -eq 1 ]]; then
MAALI_TOOL_CONFIGURE+=' -DUSE_OPENMP=ON '
else
CASACORE_BUILD_NAME+="-noopenmp"
MAALI_TOOL_CONFIGURE+=' -DUSE_OPENMP=OFF '
fi
if [[ $CASACORE_WITH_PTHREADS -eq 1 ]]; then
MAALI_TOOL_CONFIGURE+=' -DUSE_THREADS=ON '
else
CASACORE_BUILD_NAME+="-nothreads"
MAALI_TOOL_CONFIGURE+=' -DUSE_THREADS=OFF '
fi
if [[ $CASACORE_WITH_MPI -eq 1 ]]; then
MAALI_TOOL_CONFIGURE+=' -DUSE_MPI=ON '
MAALI_TOOL_PREREQ+=" $MAALI_DEFAULT_MPI "
CASACORE_BUILD_NAME+="-mpi"
else
MAALI_TOOL_CONFIGURE+=' -DUSE_MPI=OFF '
fi
if [[ $CASACORE_WITH_HDF5 -eq 1 ]]; then
if [[ $CASACORE_WITH_MPI -eq 1 ]]; then
MAALI_TOOL_PREREQ+=" hdf5/1.10.0-parallel-api-v110 "
else
MAALI_TOOL_PREREQ+=" hdf5/1.10.0-c++-noparallel-api-v110 "
fi
MAALI_TOOL_CONFIGURE+=' -DUSE_HDF5=ON '
MAALI_TOOL_CONFIGURE+=' -DHDF5_ROOT_DIR=${HDF5_DIR} '
else
CASACORE_BUILD_NAME+="-nohdf5"
MAALI_TOOL_CONFIGURE+=' -DUSE_HDF5=OFF '
fi
if [[ $CASACORE_WITH_FFTW -eq 1 ]]; then
if [[ $CASACORE_WITH_MPI -eq 1 ]]; then
MAALI_TOOL_PREREQ+=" fftw/3.3.8-mpi "
else
MAALI_TOOL_PREREQ+=" fftw/3.3.8 "
fi
MAALI_TOOL_CONFIGURE+=' -DUSE_FFTW=ON '
MAALI_TOOL_CONFIGURE+=' -DFFTW3F_THREADS_LIBRARY=${FFTW_LIB}/libfftw3f_threads.so -DFFTW3_THREADS_LIBRARY=${FFTW_LIB}/libfftw3_threads.so -DFFTW3F_LIBRARY=${FFTW_LIB}/libfftw3f.so -DFFTW3_LIBRARY=${FFTW_LIB}/libfftw3.so MAALI_TOOL_CONFIGURE+=-DFFTW3_INCLUDE_DIR=${FFTW_INCLUDE} '
else
CASACORE_BUILD_NAME+="-nofftw"
MAALI_TOOL_CONFIGURE+=' -DUSE_FFTW=OFF '
fi

#python config options still not working, needs updates.
if [[ $CASACORE_WITH_PYTHON3 -eq 1 ]]; then
MAALI_TOOL_PREREQ+=" python/3.8.5 numpy/1.19.2 "
MAALI_TOOL_CONFIGURE+=' -DBUILD_PYTHON3=ON '
MAALI_TOOL_CONFIGURE+=' -DBUILD_PYTHON=OFF '
MAALI_TOOL_CONFIGURE+=' -DPYTHON3_EXECUTABLE=$PYTHON_DIR/bin/python3 -DPYTHON3_LIBRARY=$PYTHON_DIR/lib/libpython3.so '
MAALI_TOOL_CONFIGURE+=' -DPYTHON3_Boost_LIBRARIES=${MAALI_BOOST_HOME}/lib/libboost_python3.so -DPYTHON3_Boost_INCLUDE_DIRS=${MAALI_BOOST_HOME}/include/ '
MAALI_TOOL_CONFIGURE+=' -DBOOST_LIBRARYDIR=${MAALI_BOOST_HOME}/lib '
MAALI_MODULE_SET_PYTHONPATH=1
CASACORE_BUILD_NAME+="-python3"
else
if [[ $CASACORE_WITH_PYTHON -eq 1 ]]; then
MAALI_TOOL_PREREQ+=" python/2.7.18 numpy/1.19.2 "
MAALI_TOOL_CONFIGURE+=' -DBUILD_PYTHON=ON '
MAALI_TOOL_CONFIGURE+=' -DPYTHON_EXECUTABLE=$PYTHON_DIR/bin/python2 -DPYTHON_LIBRARY=$PYTHON_DIR/lib/libpython2.so '
MAALI_TOOL_CONFIGURE+=' -DPYTHON_Boost_LIBRARIES=${MAALI_BOOST_HOME}/lib/libboost_python2.so -DPYTHON_Boost_INCLUDE_DIRS=${MAALI_BOOST_HOME}/include/ '
CASACORE_BUILD_NAME+="-python2"
MAALI_MODULE_SET_PYTHONPATH=1
else
MAALI_TOOL_CONFIGURE+=' -DBUILD_PYTHON=OFF '
fi
MAALI_TOOL_CONFIGURE+=' -DBUILD_PYTHON3=OFF '
fi

#set the build option name
MAALI_TOOL_BUILD_OPTIONS_SUFFIX="$CASACORE_BUILD_NAME"

# where the unpacked source code is located
MAALI_TOOL_BUILD_DIR="$MAALI_BUILD_DIR/$MAALI_TOOL_NAME-$MAALI_TOOL_VERSION"

# for auto-building module files
MAALI_MODULE_SET_PATH=1
MAALI_MODULE_SET_CPATH=1
MAALI_MODULE_SET_LD_LIBRARY_PATH=1
MAALI_MODULE_SET_CASACORE_ROOT='$MAALI_APP_HOME'
Loading