Install notes for NASA/Pleiades: gcc stack¶
Note
Warning. These instructions for a gcc stack are quite outdated and have not been tested in well over a year. A lot has shifted in the stack since then (e.g., h5py, matplotlib) and using these is at your own risk. We have been using the intel compilers exclusively on Pleiades, so please see those instructions. These gcc instructions are kept for posterity and future use.
Old instructions¶
An initial Pleiades environment is pretty bare-bones. There are no
modules, and your shell is likely a csh varient. To switch shells,
send an e-mail to support@nas.nasa.gov; I’ll be using bash
.
Then add the following to your .profile
:
# Add your commands here to extend your PATH, etc.
module load gcc
module load git
module load openssl
export BUILD_HOME=$HOME/build
export PATH=$BUILD_HOME/bin:$BUILD_HOME:/$PATH # Add private commands to PATH
export LD_LIBRARY_PATH=$BUILD_HOME/lib:$LD_LIBRARY_PATH
export CC=mpicc
#pathing for Dedalus2
export MPI_ROOT=$BUILD_HOME/openmpi-1.8
export PYTHONPATH=$BUILD_HOME/dedalus2:$PYTHONPATH
export MPI_PATH=$MPI_ROOT
export FFTW_PATH=$BUILD_HOME
Note
We are moving here to a python 3.4 build. Also, it looks like scipy-0.14 and numpy 1.9 are going to have happier sparse matrix performance.
Doing the entire build took about 1 hour. This was with several (4) open ssh connections to Pleaides to do poor-mans-parallel building (of openBLAS, hdf5, fftw, etc.), and one was on a dev node for the openmpi and openblas compile.
Python stack¶
Interesting update. Pleiades now appears to have a python3 module.
Fascinating. It comes with matplotlib (1.3.1), scipy (0.12), numpy
(1.8.0) and cython (0.20.1) and a few others. Very interesting. For
now we’ll proceed with our usual build-it-from-scratch approach, but
this should be kept in mind for the future. No clear mpi4py, and the
mpi4py
install was a hangup below for some time.
Building Openmpi¶
The suggested mpi-sgi/mpt
MPI stack breaks with mpi4py; existing
versions of openmpi on Pleiades are outdated and suffer from a
previously identified bug (v1.6.5), so we’ll roll our own. This needs
to be built on a compute node so that the right memory space is identified.:
# do this on a main node (where you can access the outside internet):
cd $BUILD_HOME
wget http://www.open-mpi.org/software/ompi/v1.8/downloads/openmpi-1.8.tar.gz
tar xvf openmpi-1.7.3.tar.gz
# get ivy-bridge compute node
qsub -I -q devel -l select=1:ncpus=20:mpiprocs=20:model=ivy -l walltime=02:00:00
# once node exists
cd $BUILD_HOME
cd openmpi-1.7.3
./configure \
--prefix=$BUILD_HOME \
--enable-mpi-interface-warning \
--without-slurm \
--with-tm=/PBS \
--without-loadleveler \
--without-portals \
--enable-mpirun-prefix-by-default \
CC=gcc
make
make install
These compilation options are based on /nasa/openmpi/1.6.5/NAS_config.sh
,
and are thanks to advice from Daniel Kokron at NAS.
We’re using openmpi 1.7.3 here because something substantial changes in 1.7.4 and from that point onwards instances of mpirun hang on Pleiades, when used on more than 1 node worth of cores. I’ve tested this extensively with a simple hello world program (http://www.dartmouth.edu/~rc/classes/intro_mpi/hello_world_ex.html) and for now suggest we move forward until this is resolved.
Building Python3¶
Create $BUILD_HOME
and then proceed with downloading and installing Python-3.4:
cd $BUILD_HOME
wget https://www.python.org/ftp/python/3.4.0/Python-3.4.0.tgz --no-check-certificate
tar xzf Python-3.4.0.tgz
cd Python-3.4.0
./configure --prefix=$BUILD_HOME \
CC=mpicc \
CXX=mpicxx \
F90=mpif90 \
--enable-shared LDFLAGS="-lpthread" \
--with-cxx-main=mpicxx --with-system-ffi
make
make install
All of the intel patches, etc. are unnecessary in the gcc stack.
Note
We’re getting a problem on _curses_panel
and on _sqlite3
; ignoring for now.
Installing pip¶
Python 3.4 now automatically includes pip.
On Pleiades, you’ll need to edit .pip/pip.conf
:
[global]
cert = /etc/ssl/certs/ca-bundle.crt
You will now have pip3
installed in $BUILD_HOME/bin
.
You might try doing pip3 -V
to confirm that pip3
is built
against python 3.4. We will use pip3
throughout this
documentation to remain compatible with systems (e.g., Mac OS) where
multiple versions of python coexist.
Installing mpi4py¶
This should be pip installed:
pip3 install mpi4py
Note
Test that this works by doing a:
from mpi4py import MPI
This will segfault on sgi-mpi, but appears to work fine on openmpi-1.8, 1.7.3, etc.
Installing FFTW3¶
We need to build our own FFTW3, under intel 14 and mvapich2/2.0b:
wget http://www.fftw.org/fftw-3.3.4.tar.gz
tar -xzf fftw-3.3.4.tar.gz
cd fftw-3.3.4
./configure --prefix=$BUILD_HOME \
CC=mpicc \
CXX=mpicxx \
F77=mpif90 \
MPICC=mpicc MPICXX=mpicxx \
--enable-shared \
--enable-mpi --enable-openmp --enable-threads
make
make install
It’s critical that you use mpicc
as the C-compiler, etc.
Otherwise the libmpich libraries are not being correctly linked into
libfftw3_mpi.so
and dedalus failes on fftw import.
Installing nose¶
Nose is useful for unit testing, especially in checking our numpy build:
pip3 install nose
Installing cython¶
This should just be pip installed:
pip3 install cython
The Feb 11, 2014 update to cython (0.20.1) seems to work with gcc.
Numpy and BLAS libraries¶
Numpy will be built against a specific BLAS library. On Pleiades we will build against the OpenBLAS libraries.
All of the intel patches, etc. are unnecessary in the gcc stack.
Building OpenBLAS¶
From Stampede instructions:
# this needs to be done on a frontend
cd $BUILD_HOME
git clone git://github.com/xianyi/OpenBLAS
# suggest doing this build on a compute node, so we get the
# right number of openmp threads and architecture
cd $BUILD_HOME
cd OpenBLAS
make
make PREFIX=$BUILD_HOME install
Here’s the build report before the make install
:
OpenBLAS build complete. (BLAS CBLAS LAPACK LAPACKE)
OS ... Linux
Architecture ... x86_64
BINARY ... 64bit
C compiler ... GCC (command line : mpicc)
Fortran compiler ... GFORTRAN (command line : gfortran)
Library Name ... libopenblas_sandybridgep-r0.2.9.rc2.a (Multi threaded; Max num-threads is 40)
Building numpy against OpenBLAS¶
Now, acquire numpy
(1.8.1):
wget http://sourceforge.net/projects/numpy/files/NumPy/1.8.1/numpy-1.8.1.tar.gz
tar xvf numpy-1.8.1.tar.gz
cd numpy-1.8.1
Create site.cfg
with information for the OpenBLAS
library directory
Next, make a site specific config file:
cp site.cfg.example site.cfg
emacs -nw site.cfg
Edit site.cfg
to uncomment the [openblas]
section; modify the
library and include directories so that they correctly point to your
~/build/lib
and ~/build/include
(note, you may need to do fully expanded
paths). With my account settings, this looks like:
[openblas]
libraries = openblas
library_dirs = /u/bpbrown/build/lib
include_dirs = /u/bpbrown/build/include
where $BUILD_HOME=/u/bpbrown/build
. We may in time want to
consider adding fftw as well. Now build:
python3 setup.py config build_clib build_ext install
This will config, build and install numpy.
Test numpy install¶
Test that things worked with this executable script
numpy_test_full
. You can do this
full-auto by doing:
wget http://dedalus-project.readthedocs.org/en/latest/_downloads/numpy_test_full
chmod +x numpy_test_full
./numpy_test_full
We succesfully link against fast BLAS and the test results look normal.
Python library stack¶
After numpy
has been built
we will proceed with the rest of our python stack.
Installing Scipy¶
Scipy is easier, because it just gets its config from numpy. Dong a pip install fails, so we’ll keep doing it the old fashioned way:
wget http://sourceforge.net/projects/scipy/files/scipy/0.13.3/scipy-0.13.3.tar.gz
tar -xvf scipy-0.13.3.tar.gz
cd scipy-0.13.3
python3 setup.py config build_clib build_ext install
Note
We do not have umfpack; we should address this moving forward, but for now I will defer that to a later day.
Installing HDF5 with parallel support¶
The new analysis package brings HDF5 file writing capbaility. This needs to be compiled with support for parallel (mpi) I/O:
wget http://www.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.12.tar
tar xvf hdf5-1.8.12.tar
cd hdf5-1.8.12
./configure --prefix=$BUILD_HOME \
CC=mpicc \
CXX=mpicxx \
F77=mpif90 \
MPICC=mpicc MPICXX=mpicxx \
--enable-shared --enable-parallel
make
make install
Next, install h5py. For reasons that are currently unclear to me, this cannot be done via pip install.
Installing h5py with collectives¶
We’ve been exploring the use of collectives for faster parallel file writing.
git is having some problems, especially with it’s SSL version.
I suggest adding the following to ~/.gitconfig
:
[http]
sslCAinfo = /etc/ssl/certs/ca-bundle.crt
This is still not working, owing (most likely) to git being built on an outdated SSL version. Here’s a short-term hack:
export GIT_SSL_NO_VERIFY=true
To build that version of the h5py library:
git clone git://github.com/andrewcollette/h5py
cd h5py
git checkout mpi_collective
export CC=mpicc
export HDF5_DIR=$BUILD_HOME
python3 setup.py configure --mpi
python3 setup.py build
python3 setup.py install
Here’s the original h5py repository:
git clone git://github.com/h5py/h5py
cd h5py
export CC=mpicc
export HDF5_DIR=$BUILD_HOME
python3 setup.py configure --mpi
python3 setup.py build
python3 setup.py install
Note
This is ugly. We’re getting a “-R” error at link, triggered by
distutils not recognizing that mpicc is gcc or something like
that. Looks like we’re failing if self._is_gcc(compiler)
For now, I’ve hand-edited unixccompiler.py in
lib/python3.3/distutils
and changed this line:
- def _is_gcc(self, compiler_name):
- return “gcc” in compiler_name or “g++” in compiler_name
to:
- def _is_gcc(self, compiler_name):
- return “gcc” in compiler_name or “g++” in compiler_name or “mpicc” in compiler_name
This is a hack, but it get’s us running and alive!
Note
Ahh… I understand what’s happening here. We built with
mpicc
, and the test _is_gcc
looks for whether gcc appears
anywhere in the compiler name. It doesn’t in mpicc
, so the
gcc
checks get missed. This is only ever used in the
runtime_library_dir_option()
call. So we’d need to either
rename the mpicc wrapper something like mpicc-gcc
or do a
test on compiler --version
or something. Oh boy. Serious
upstream problem for mpicc wrapped builds that cythonize and go
to link. Hmm…
Installing Mercurial¶
On NASA Pleiades, we need to install mercurial itself:
wget http://mercurial.selenic.com/release/mercurial-2.9.tar.gz
tar xvf mercurial-2.9.tar.gz
cd mercurial-2.9
make install PREFIX=$BUILD_HOME
I suggest you add the following to your ~/.hgrc
:
[ui]
username = <your bitbucket username/e-mail address here>
editor = emacs
[web]
cacerts = /etc/ssl/certs/ca-bundle.crt
[extensions]
graphlog =
color =
convert =
mq =
Dedalus2¶
Preliminaries¶
With the modules set as above, set:
export BUILD_HOME=$BUILD_HOME
export FFTW_PATH=$BUILD_HOME
export MPI_PATH=$BUILD_HOME/openmpi-1.8
Then change into your root dedalus directory and run:
python setup.py build_ext --inplace
further packages needed for Keaton’s branch:
pip3 install tqdm
pip3 install pathlib
Running Dedalus on Pleiades¶
Our scratch disk system on Pleiades is /nobackup/user-name
. On
this and other systems, I suggest soft-linking your scratch directory
to a local working directory in home; I uniformly call mine workdir
:
ln -s /nobackup/bpbrown workdir
Long-term mass storage is on LOU.