Veros & Bohrium

Roman Nuterman

Personal web-page

Getting your own Python environment with Bohrium and Versatile Ocean Simulator on DC3

 

Getting Your Own Python Environment

 

On DC3, there are two Anaconda Python modules that you can use to run Python code. You can load those e.g. with

 

module load anaconda/4.2.0_python2

 

However, if you want to install additional Python packages (e.g. via pip), you will have to create your own Python environment in your home folder. One convenient way to do so is to create a so-called virtual environment off of one of the global Anaconda environments. Virtual environments are a powerful tool to create isolated Python environments that you have full control over.

 

It is advisable to gather all of your virtual environments in the same folder, say, ~/venvs. To create your first environment, execute

 

module load anaconda/4.2.0_python2

mkdir $HOME/venvs

python -m virtualenv $HOME/venvs/myvirtualenv

 

Instead of myvirtualenv you are free to choose whatever name you would like for your environment. You can create as many virtual environments as you want (e.g. to install different packages for different use cases, and to prevent dependency conflicts).

 

Before using a virtual environment, you will have to activate it, e.g. via

 

source ~/venvs/myvirtualenv/bin/activate

 

If you want, you can also add this line to your $HOME/.bashrc file, so the environment will be activated on login. You can run which python now, and you will see that the python alias now points to the interpreter in venvs/myvirtualenv. To install packages in your environment, you can just use the PyPI package manager, pip:

 

pip install numpy

 

will install NumPy into myenvironment.

 

Installing Bohrium & Veros

 

Bohrium is already installed and managed in the global Anaconda Python instances - however, it is not accessible from your local, virtual environment yet. You can make the module available simply be creating a symbolic link to it:

 

ln -s /groups/ocean/software/anaconda2/lib/python2.7/site-packages/bohrium $HOME/venvs/myvirtualenv/lib/python2.7/site-packages/

 

Before being able to use Bohrium, you will need to set the path to the Bohrium config file:

 

export BH_CONFIG=/groups/ocean/software/anaconda2/etc/bohrium/config.ini

 

(this can also be added to the $HOME/.bashrc file to make it permanent)

 

Now, you should be able to import Bohrium. To check whether everything was installed correctly, activate your environment and run:

 

python -c "import bohrium"

 

If this does not throw an error, the installation succeeded.

 

To install Veros, activate your virtual environment and run

 

cd $HOME

module load git

git clone https://github.com/dionhaefner/veros.git

pip install -e ./veros

 

The Veros code is now available in the folder ~/veros, and has been installed into your local Python environment.

 

For information on setting up a model, please refer to the Veros documentation.

 

Submitting Veros Jobs

 

After setting up a model, you can use a batch script to submit it to a compute node. In its simplest form, this batch script could read something like this:

 

#!/bin/bash

#

#SBATCH -p aegir

#SBATCH -A ocean

#SBATCH --job-name=veros_my_model

#SBATCH --nodes=1

#SBATCH --ntasks=1

#SBATCH --cpus-per-task=16

#SBATCH --exclusive

#SBATCH --mail-type=ALL

#SBATCH --mail-user=

 

export BH_CONFIG=/groups/ocean/software/anaconda2/etc/bohrium/config.ini

export BH_OPENMP_COMPILER_FLG="-x c -fPIC -shared -std=gnu99 -O3 -Werror -fopenmp"

source $HOME/venvs/myvirtualenv/bin/activate

 

srun -- python my_veros_model.py -b bohrium

 

All you need to do is to save this script into the setup folder (e.g. as ``veros-batch.sh``), adapt the names and paths, and submit it via

 

sbatch veros-batch.sh

 

Note, however, that your job will time out after the time limit is reached (e.g. after 24 hours). If your job runs longer than that, you will have to use a resubmit mechanism (see also the Veros documentation).

The corresponding batch script using veros resubmit looks like this:

 

#!/bin/bash

#

#SBATCH -p aegir

#SBATCH -A ocean

#SBATCH --job-name=veros_my_model

#SBATCH --nodes=1

#SBATCH --ntasks=1

#SBATCH --cpus-per-task=16

#SBATCH --exclusive

#SBATCH --mail-type=ALL

#SBATCH --mail-user=

 

export BH_CONFIG=/groups/ocean/software/anaconda2/etc/bohrium/config.ini

export BH_OPENMP_COMPILER_FLG="-x c -fPIC -shared -std=gnu99 -O3 -Werror -fopenmp"

source $HOME/venvs/myvirtualenv/bin/activate

 

veros resubmit mymodel 8 7776000 "python my_veros_model.py -b bohrium" --callback "sbatch veros-batch.sh"

 

Submitting this script with sbatch veros-batch.sh will create a job that automatically re-submits itself after successful termination (here, for a total of 8 times). Note that each individual run must finish before timing out.