Skip to content

ubuntu 10.10

Jump to bottom Edit New page
dimasad edited this page Aug 18, 2012 · 7 revisions

StarCluster AMI Cookbook Ubuntu 10.10

Below are pseudo-instructions for creating the latest StarCluster AMI (Ubuntu 10.10 Maverick) from scratch:

Note

You should be root when executing any command in this guide

Launch AMI and Install Packages

  1. Launch an alestic ubuntu AMI (x86: ami-a6f504cf, amd64: ami-08f40561)

  2. Login via ssh as the ubuntu user and use "sudo -i" to switch to root

  3. Edit /etc/apt/sources.list, add multiverse to the end of all lines

  4. Update the packages to the latest versions:

    apt-get update
apt-get upgrade

  5. Install the necessary build tools:

    apt-get --reinstall install build-essential python-dev swig gfortran -y
apt-get remove python-nose g77 python-setuptools -y

  6. Install and configure blas/lapack:

    rm -rvf /usr/lib{,64}/atlas/
apt-get remove liblapack"*" -y
apt-get remove libblas"*" -y
apt-get -y --reinstall install libatlas-base-dev libatlas3gf-base

  7. Install and configure gfortran:

    apt-get --reinstall install gfortran-4.4 -y
ln -sf /usr/bin/gfortran{-4.4,}

  8. Remove any existing numpy/scipy installations:

    rm -rvf /usr/local/lib/python2.6/*-packages/numpy*
rm -rvf /usr/local/lib/python2.6/*-packages/scipy*

  9. Install and configure the base dependencies:

    apt-get install python-dev git-core vim mercurial subversion cvs encfs \
openmpi-bin libopenmpi-dev nfs-kernel-server python-django keychain \
screen tmux zsh ksh csh tcsh python-pip python-setuptools \
python-virtualenv python-imaging python-boto unzip rar unace \
build-essential gfortran ec2-api-tools ec2-ami-tools mysql-server \
mysql-client apache2 libapache2-mod-wsgi sysv-rc-conf pssh emacs cython \
irssi python-distutils-extra python-profiler htop vim-scripts \
python-ctypes

  10. Install latest ipython and nose:

    pip install ipython
pip install nose

Configure Root Login

The alestic AMI's have been configured to disable root logins. Follow the commands below to undo this behavior:

  1. Edit /etc/cloud/cloud.cfg and set disable_root: 0
  2. Edit /root/.ssh/authorized_keys and remove prefix (everything before ssh-rsa-....) commands from pubkey entry
  3. Edit /usr/bin/cloud-init, go to line 86 and change 'once-per-instance' to 'always', save and exit

Configure MOTD and Bash Completion

  1. Customize motd in /etc/update-motd.d/* and /etc/motd.tail
  2. Uncomment bash completion section in /etc/bash.bashrc

Configure NFS

StarCluster relies on the /etc/init.d/nfs script to start the NFS server. Run the command below to ensure this exists:

ln -s /etc/init.d/nfs-kernel-server /etc/init.d/nfs

StarCluster configures /etc/exports and then issues the server start command, so if the server is already running it will not restart. Therefore it must not be started automatically:

sudo update-rc.d -f nfs-kernel-server remove

Configure Sun Grid Engine

  1. Launch one of the 32bit or 64bit StarCluster AMI's and copy /opt/sge6-fresh to /opt/sge6-fresh on your new image host.

  2. Unpack python-drmaa (0.2) to /opt, set SGE6_ROOT='/opt/sge6-fresh' in setup.py and run:

    python setup.py install

  3. Unpack python-drmaa (0.4b3) to /opt and run the following command inside the source folder:

    python setup.py install

  4. Link the DRMAA library to /usr/lib64/libdrmaa.so:

    ln -s /opt/sge6-fresh/lib/lx24-{x86|amd64}/libdrmaa.so.1.0 /usr/lib{64}/libdrmaa.so.1.0
ln -s /usr/lib{64}/libdrmaa.so.1.0 /usr/lib{64}/libdrmaa.so

  5. Test that the following commands complete without error:

    python -c "import DRMAA"
python -c "import drmaa"

Rebuild Ubuntu's OpenMPI packages with Sun Grid Engine Support

  1. Install the build deps for libopenmpi-dev:

    apt-get build-dep libopenmpi-dev

  2. Get the source for the libopenmpi-dev debian package:

    cd /usr/local/src
apt-get source libopenmpi-dev

  3. Change into the libopenmpi-dev package's debian folder:

    cd openmpi-1.4.1/debian

  4. Modify the 'rules' file and add --with-sge to the configure arguments:

    vim rules
(add --with-sge to configure args)

  5. Rebuild the libopenmpi-dev package:

    cd ..
dpkg-buildpackage -rfakeroot -b

  6. Install the newly rebuild package:

    cd ..
dpkg -i *.deb

  7. Verify Sun Grid Engine support:

    ompi_info | grep -i grid
    MCA ras: gridengine (MCA v2.0, API v2.0, Component v1.4.1)

Configure Optimized Atlas/UMFPACK/AMD/FFTW/Scipy/Numpy

The next sections describe installing an optimized scipy/numpy using Atlas/UMFPACK/AMD/FFTW libraries from source. A lot of these instructions were merged from http://scipy.org/Installing_SciPy/Linux

Install lapack/atlas from source

  1. Checkout npinto-toolbox:

    cd /usr/local/src
git clone git://github.com/jtriley/npinto-toolbox
cd npinto-toolbox

  2. Run the ATLAS installer:

    ./install_atlas3.8.3_Ubuntu9.10-{x86,x86_64}-root.bash

  3. Remove /usr/lib/liblapack.so:

    rm /usr/lib/liblapack.so

  4. Update /usr/lib/gfortran.so symlink:

    ln -s /usr/lib/libgfortran.so.3 /usr/lib/gfortran.so

Install UMFPACK/AMD/UFconfig from source

  1. Create a directory to unpack the source into:

    mkdir /usr/local/src/umfpack
  2. Get the latest versions of AMD, UFconfig and UMFPACK from

    http://www.cise.ufl.edu/research/sparse/ and untar them into /usr/local/src/umfpack

  3. Modify /usr/local/src/umfpack/UFconfig/UFconfig.mk to look like (change

    -m64 to -m32 in the flags below for a 32bit system):

    CC = gcc
CFLAGS = -O3 -fexceptions -m64 -fPIC -DNCHOLMOD
F77 = gfortran
F77FLAGS = -O -m64 -fPIC

BLAS = -lptf77blas -lptcblas -latlas -lgfortran
LAPACK = -llapack -lptf77blas -lptcblas -latlas -lgfortran

  4. Run 'make' in the /usr/local/src/umfpack/UMFPACK directory:

    cd /usr/local/src/umfpack/UMFPACK
make

  5. Copy resulting libraries and include files:

    cp /usr/local/src/umfpack/AMD/Lib/libamd.a /usr/lib
cp /usr/local/src/umfpack/UMFPACK/Lib/libumfpack.a /usr/lib
cp /usr/local/src/umfpack/AMD/Include/amd.h /usr/include
cp /usr/local/src/umfpack/UFconfig/UFconfig.h /usr/include
cp /usr/local/src/umfpack/UMFPACK/Include/*.h /usr/include

  6. Create /usr/lib/libumfpack.so symlink:

    ln -s /usr/lib/libumfpack.a /usr/lib/libumfpack.so

Install FFTW from source

  1. Download fftw-3.2.2 from http://www.fftw.org into /usr/local/src and unpack

  2. Change into the fftw source folder:

    cd /usr/local/src/fftw-3.2.2

  3. Configure fftw (change -m64 to -m32 for 32bit):

    export CFLAGS="-O3 -fomit-frame-pointer -fstrict-aliasing -ffast-math -pthread -fPIC -m64"
export FFLAGS="-g -O2 -fPIC -m64"
export CXXFLAGS="-g -O2 -fPIC -m64"
./configure --enable-sse2 --enable-threads --with-combined-threads --prefix=/usr

  4. Build fftw:

    make install

Install Scipy/Numpy from source

  1. Download numpy-1.5.0b1/scipy-0.8.0 into /usr/local/src and untar them

  2. Copy numpy_scipy_site.cfg from npinto-toolbox:

    cp /usr/local/src/npinto-toolbox/install_scripts/numpy_scipy_site.cfg /usr/local/src/numpy-1.5.0b1/site.cfg
cp /usr/local/src/npinto-toolbox/install_scripts/numpy_scipy_site.cfg /usr/local/src/scipy-0.8.0/site.cfg

  3. Run the configure check for numpy:

    cd /usr/local/src/numpy-1.5.0b1
python setup.py config

  4. Copy the SMCFLAGS from the ATLAS VERSION output and export CFLAGS to be the same as SMCFLAGS

  5. Copy the F77FLAGS from the ATLAS VERSION output and export F77FLAGS to be the same

  6. Install numpy:

    python setup.py install

  7. Run the following command:

    python -c "import numpy; numpy.show_config()"

  8. The output of the above command should look something like:

    In [3]: import numpy

In [4]: numpy.show_config()
atlas_threads_info:
    libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib']
    language = f77

blas_opt_info:
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib']
    define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
    language = c

atlas_blas_threads_info:
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib']
    language = c

lapack_opt_info:
    libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib']
    define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
    language = f77

lapack_mkl_info:
  NOT AVAILABLE

blas_mkl_info:
  NOT AVAILABLE

mkl_info:
  NOT AVAILABLE

  9. Install scipy:

    cd /usr/local/src/scipy-0.8.0
python setup.py install

  10. Run the following command:

    python -c "import scipy; scipy.show_config()"

  11. Output of the above command should look something like:

    In [5]: import scipy

In [6]: scipy.show_config()
amd_info:
    libraries = ['amd']
    library_dirs = ['/usr/lib']
    define_macros = [('SCIPY_AMD_H', None)]
    swig_opts = ['-I/usr/include']
    include_dirs = ['/usr/include']

umfpack_info:
    libraries = ['umfpack', 'gfortran', 'amd']
    library_dirs = ['/usr/lib']
    define_macros = [('SCIPY_UMFPACK_H', None), ('SCIPY_AMD_H', None)]
    swig_opts = ['-I/usr/include', '-I/usr/include']
    include_dirs = ['/usr/include']

atlas_threads_info:
    libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib']
    language = f77

blas_opt_info:
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib']
    define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
    language = c

atlas_blas_threads_info:
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib']
    language = c

lapack_opt_info:
    libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib']
    define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
    language = f77

lapack_mkl_info:
  NOT AVAILABLE

blas_mkl_info:
  NOT AVAILABLE

mkl_info:
  NOT AVAILABLE

  12. Make sure the following commands both output "OK":

    python -c "import numpy; numpy.test()"
python -c "import scipy; scipy.test()"
Add a custom footer
Add a custom sidebar
Clone this wiki locally