Successful Installation Examples
In this document, we describe a number of EMsoft installations that involve non-standard environments or hardware. EMsoft has been installed on many platforms, and the CMake build files provide installation/configuration instructions for the more commonly available operating systems:
- Mac OS X: EMsoft has been installed on a variety of Apple laptops, as well as on iMacs and Mac Pros with OS versions starting around 10.10 or so. The CMake configuration files should be able to handle all these minor variations in hardware and OS versions.
- Windows 10: EMsoft, while originally developed with the gfortran compiler, has been successfully ported to Intel's ifort compiler (version 17) within the Visual Studio 2015 environment. CMake configuration files are included in the standard EMsoftSuperbuild for this OS.
- Linux: This OS presents a slightly more complicated compilation task due to the many available flavors of Linux and the large range of possible hardware configurations. The standard CMake Build files are known to produce a functional EMsoft distribution on CentOS 7 and Ubuntu 16. For other configurations, one will likely need to modify the configuration files a bit.
In the remainder of this document we describe examples of less common and/or more complex installation environments. These are provided by EMsoft users and we post them here as additional information for future users; the descriptions are mostly taken verbatim, from user email messages, with minor editing as needed.
At Carnegie Mellon University, we run EMsoft on a variety of Mac OS X systems, including MacBook, MacBook Pro, iMac, and Mac Pro, as well as on two high end Linux workstations:
- CentOS 7: 48 cores (Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz) with 4 Nvidia GeForce GTX 1080 GPU cards
- Ubuntu 16: 88 cores (Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz) with 3 Tesla K80 GPU cards (effectively 6 GPU devices)
This information was kindly provided by McLean Echlin [[email protected]], UCSB
EMSoft Install on a CentOS Linux release 7.5.1804 Computer Cluster
Added the following to the .bashrc & .bash_profile files:
% We add the EMsoft Binaries folder to path, so that they may be run from anywhere
PATH=$PATH:$HOME/.local/bin:$HOME/bin:$HOME/EMsoftBuild/Bin
% enable the redhat devtools gcc/gfortran gcc version 7.3.1 20180303 (Red Hat 7.3.1-5) (GCC)
. /opt/rh/devtoolset-7/enable
% sets/substitutes environment variables to point to gfortran
export F95=gfortran
export F90=gfortran
export FC=gfortran
% load the cmake 3 module
module load cmake-3
% we unload this module (not sure if this is critical)
module unload gnu/5.4.0
% download and build the EMsoft Superbuild (necessary libraries)
% replace TOPFOLDER with the folder inside which you wish to build everything
git clone -b develop https://github.com/EMsoft-org/EMsoftSuperbuild EMsoftSDK
cd EMsoftSDK
mkdir Debug
mkdir Release
cd Debug
% adding “-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON” to get more output
cmake -DEMsoft_SDK=/TOPFOLDER/EMsoftSDK -DCMAKE_BUILD_TYPE=Debug -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON ../
make -j8
cd ../Release
cmake -DEMsoft_SDK=/TOPFOLDER/EMsoftSDK -DCMAKE_BUILD_TYPE=Release -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON ../
% adding “-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON” to get more output%
make -j8
% next, clone the Public EMsoft repository
cd ~
git clone -b develop https://github.com/EMsoft-org/EMsoftPublic EMsoftPublic
mkdir EMsoftPublicBuild
cd EMsoftPublicBuild
% adding “-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON” to get more output%
cmake -DEMsoft_SDK=/TOPFOLDER/EMsoftSDK -DCMAKE_BUILD_TYPE=Release ../EMsoftPublic
make -j8This information was kindly provided by Chad Paris [[email protected]] and Christ Layton [[email protected]], ORNL
We are working with Docker and using HPC module environments. We did a scale-up experiment on the Nvidia DGX box using the EMTKDmaster binary. It has 20 physical cores and 40 apparent cores (hyperthreading). It was almost linear up to 20 OpenMP threads, and then rolled off when we tried 35.
We focused on the docker setup which we implemented via the dockerfile below. We then built this into a container that we can move between the DGX-DL (which is Ubuntu 16.04) and the Openstack Instance (which is CentOS 7.4) via a internal "registry" for this container.
-------------------------
DOCKER BUILD FILE
------------------------
FROM nvidia/cuda:9.0-devel-centos7
# Builder : Layton, Chris <[email protected]>
WORKDIR /workspace
# Setup the SDK enviroment
RUN mkdir /workspace/EMsoft_SDK &&\
curl -o /workspace/EMsoft_SDK/cmake-3.9.1-Linux-x86_64.tar.gz http://dream3d.bluequartz.net/binaries/EMSoft_SDK/cmake-3.9.1-Linux-x86_64.tar.gz &&\
cd /workspace/EMsoft_SDK &&\
tar -xzf cmake-3.9.1-Linux-x86_64.tar.gz
# We run this separate because there are deps on it below
RUN yum -y install epel-release
# ADD /root/.config/EMsoft/EMsoftConfig.json
COPY files/EMsoftConfig.json /root/.config/EMsoft/EMsoftConfig.json
# Install all needed packages and enable the EPEL repo for some that need it
RUN yum -y install --enablerepo=epel git \
wget \
cmake \
blas \
blas-devel \
blas64 \
blas64-devel \
make \
dkms \
clinfo \
xorg-x11-drv-nvidia-libs \
centos-release-scl \
cuda-misc-headers-9-0 \
xorg-x11-drv-nvidia-libs-devel \
opencl-headers \
unzip \
ocl-icd.x86_64 \
ocl-icd-devel.x86_64 \
lapack-devel \
nano
# This installs files from the SCL installed above.
RUN yum -y install devtoolset-6-gcc-gfortran.x86_64
# Setup the PATH so we use the SCL packages vs system fortran
ENV PATH=/opt/rh/devtoolset-6/root/usr/bin:/workspace/EMsoft_SDK/cmake-3.9.1-Linux-x86_64/bin:/opt/rh/devtoolset-6/root/usr/bin:${PATH}
# Grab the EMSoft Package and hack in the SCL fortran
RUN git clone https://github.com/marcdegraef/EMsoft.git &&\
sed -e 's/git\:/http\:/g' -i /workspace/EMsoft/Support/SDK_Build_Scripts/Linux_Build_Scripts/*.sh &&\
sed -e 's|/usr/bin/gfortran-5|/opt/rh/devtoolset-6/root/usr/bin/gfortran|g' -i /workspace/EMsoft/Support/SDK_Build_Scripts/Linux_Build_Scripts/Build_JsonFortran.sh
# Change the workdir due to the way paths are setup in the build scripts
WORKDIR /workspace/EMsoft/Support/SDK_Build_Scripts/Linux_Build_Scripts/
RUN /workspace/EMsoft/Support/SDK_Build_Scripts/Linux_Build_Scripts/Build_SDK.sh /workspace/EMsoft_SDK 50
# Build the software
WORKDIR /workspace/EMsoft
RUN mkdir /workspace/EMsoft/Build &&\
cd /workspace/EMsoft/Build &&\
/workspace/EMsoft_SDK/cmake-3.9.1-Linux-x86_64/bin/cmake -DEMsoft_SDK=/workspace/EMsoft_SDK/ ../ &&\
make -j $(nproc) &&\
make install