Uenv are user environments that provide scientific applications, libraries and tools on Alps. This article will explain how to activate uenv, use them to build software and how to enable them in SLURM jobs.

Unlike the standard Cray Programming Environment (CPE), which aims to provide a complete software stack that can be configured to build all types of software on all node architectures, uenv are typically application-specific, domain-specific or tool-specific - each uenv contains only what is required for the application or tools that it provides.

User environments are packaged in a single file (in the  SquashFS file format), that stores a compressed directory tree that contains all of the software, tools and other information like modules, required to provide a rich environment.

Each environment contains a software stack, comprised of compilers, libraries, tools and scientific applications, built using Spack.

For help with uenv create a "Software Environment" request at the CSCS Service Desk.

For more information on how CSCS builds user environments, have a look at the documentation for the Stackinator tool that CSCS use to configure Spack to build environments with optimisations for the Alps architecture.

This tool can also be used by motivated users to provide their own software stacks.

Notices

Alps update January 2025

On the uenv tools provided on Daint, Santis and Clariden were updated. This update makes some significant changes, that can affect user workflows.

The uenv view  command has been removed:

  • it is no longer possible to modify the uenv environment once a uenv has been started.
  • the fix: provide the view using the --view  flag, see the examples below
using --view with the CLI tool
> uenv start prgenv-gnu/24.11:v1 --view=default

> uenv run prgenv-gnu/24.11:v1 --view=default -- ./run-application.sh
using --view with sbatch
#SBATCH --view=prgenv-gnu/24.11:v1
#SBATCH --view=default


Update uenv on EIger

The version of uenv installed on Eiger does not support new uenv features, such as improved Spack integration and module files.

We strongly recommend installing the latest version (see the Installing instructions.)

'ascii' code can't encode character '\u2261'

In uenv 5, installed on eiger and balfrin, if you see an error like the following when pulling an image:

> uenv image pull icon-wcp/v1:rc3
downloading image 8e3fccb3a887964c3b0f5cbf4f0cbd0937480f0042574b7515b3af1edcdad709 7.9GB
  [=                                                 ]   1% 125/8128 MB
[error] image pull failed with unknown exception: 'ascii' codec can't encode character '\u2261' in position 22: ordinal not in range(128)
the fix is to set the environment variable export PYTHONIOENCODING=utf-8

Getting Started

After logging into an Alps cluster, you can quickly check the availability of uenv with the following commands:

downloading and installing uenv
> uenv status
there is no uenv loaded
> uenv --version
7.0.0

Installing

The command line tool is pre-installed on the system. The develop version can be installed from source, do this only if you need to test a new version.

locally installing uenv
git clone https://github.com/eth-cscs/uenv2.git
cd uenv2

# to build the latest uenv, you need to first pull the following:
uenv image pull prgenv-gnu/24.11:v1

# run the installation script.
# this will install uenv2 in $HOME/.local/$(uname -m)/bin
./install-alps-local.sh

# add the following to your bashrc
echo "export PATH=$HOME/.local/$(uname -m)/bin:$PATH" >> $HOME/.bashrc
echo "unset -f uenv" >> $HOME/.bashrc
Once uenv has been installed, log out and back in again for it to be enabled in your environment

Naming uenv

Uenv are referred to using labels, where a label has the following form name/version:tag@system%uarch, for example prgenv-gnu/24.11:v2@todi%gh200.

name

the name of the uenv. In this case prgenv-gnu.

version

The version of the uenv. The format of version depends on the specific uenv. Often they use the yy.mm format, though they may also use the version of the software being packaged. For example the namd/3.0.1 uenv packages version 3.0.1 of the popular NAMD simulation tool.

tag

Used to differentiate between releases of a versioned uenv. Some examples of tags include:

  • rc1, rc2: release candidates.
  • v1: a first release typically made after some release candidates.
  • v2: a second release, that might fix issues in the first release.

system

The name of the Alps cluster for which the uenv was built.

uarch

The node type (microarchitecture) that the uenv is built for:

uarchCPUGPUcomment
gh2004 72-core NVIDIA Grace (aarch64)4 NVIDIA H100 GPUs
zen22 64-core AMD Rome (zen2)-used in Eiger
a1001 64-core AMD Milan (zen3)4 NVIDIA A100 GPUs
mi2001 64-core AMD Milan (zen3)4 AMD Mi250x GPUs
zen32 64-core AMD Milan (zen3)-only in MeteoSwiss systems

Using labels

The uenv command line can use partial uenv labels, where appropriate:

using uenv lables
# search for all uenv on the current system that have the name prgenv-gnu
uenv image find prgenv-gnu

# search for all uenv with version 24.11
uenv image find /24.11

# search for all uenv with tag v1
uenv image find :v1

# seach for a specific version
uenv image find prgenv-gnu/24.11:v1

By default, the uenv filters results to show those for the current cluster:

using uenv lables
# log into the eiger vCluster
ssh eiger

# this command will search for all pgrenv-gnu uenv on _eiger_
uenv image find prgenv-gnu

# use @ to search on a specific system, e.g. on daint:
uenv image find prgenv-gnu@daint

# this can be used to search for all uenv on another system:
uenv image find @daint

# the '*' is a wildcard used meaining "all systems"
# this will show all images on all systems
# WARNING: '*' must be in single quotes!
uenv image find @'*'

# search for all images on Alps that were built for gh200 nodes.
uenv image find @'*'%gh200

The wild card * used for "all systems" must always be escaped in single quotes: @'*'.

Finding uenv

Uenv for programming environments, tools and applications are provided by CSCS on each Alps system.

The same uenv are not installed on every system. Instead uenv that are supported for the users of that platform are provided.

The available uenv images are stored in a registry, that can be queried using the uenv image find  command:

uenv image find
> uenv image find
uenv                       arch  system  id                size(MB)  date
cp2k/2024.1:v1             zen2  eiger   2a56f1df31a4c196   2,693    2024-07-01
cp2k/2024.2:v1             zen2  eiger   f83e95328d654c0f   2,739    2024-08-23
cp2k/2024.3:v1             zen2  eiger   7c7369b64b5fabe5   2,740    2024-09-18
editors/24.7:rc1           zen2  eiger   e5fb284962908eed   1,030    2024-07-18
editors/24.7:v2            zen2  eiger   4f0f2770616135b1   1,062    2024-09-04
julia/24.9:v1              zen2  eiger   0ff97a74dfcaa44e     539    2024-11-09
linalg/24.11:rc1           zen2  eiger   b69f4664bf0cd1c4     770    2024-11-20
linalg/24.11:v1            zen2  eiger   c11f6c85028abf5b     776    2024-12-03
linalg-complex/24.11:v1    zen2  eiger   846a04b4713d469b     792    2024-12-03
linaro-forge/24.0.2:v1     zen2  eiger   65734ce35494a5f5     313    2024-07-18
linaro-forge/24.1:v1       zen2  eiger   b65d7c85adfb317a     344    2024-11-27
netcdf-tools/2024:v1       zen2  eiger   e7e508c34cf40ccd   3,706    2024-11-14
prgenv-gnu/24.11:rc4       zen2  eiger   811469b00f030493     570    2024-11-21
prgenv-gnu/24.11:v1        zen2  eiger   0b6ab5fc4907bb38     572    2024-11-27
prgenv-gnu/24.7:v1         zen2  eiger   7f68f4c8099de257     478    2024-07-01
quantumespresso/v7.3.1:v1  zen2  eiger   61d1f21881a65578     864    2024-11-08

# find all uenv with the tag "v2"
> uenv image find :v2
uenv                       arch   system  id                size(MB)  date
cp2k/2024.3:v2             gh200  daint   3e50a83255477ec0   3,987    2024-10-24
editors/24.7:v2            gh200  daint   e7b0d930df729da5   1,270    2024-09-04
quantumespresso/v7.3.1:v2  gh200  daint   475ffda1a162f2d8   9,313    2024-10-02
quantumespresso/v7.4:v2    gh200  daint   19c67f676d6ac0aa   6,693    2025-01-17

# find all uenv with the name "prgenv-gnu"
> uenv image find prgenv-gnu
uenv                 arch   system  id                size(MB)  date
prgenv-gnu/24.11:v1  gh200  daint   b81fd6ba25e88782   4,191    2024-11-27
prgenv-gnu/24.7:v3   gh200  daint   b50ca0d101456970   3,859    2024-08-23

The id  of the image is a unique identifier derived from the sha256 hash of the image.

The full name and id can both be used to refer to a specific uenv, e.g. uenv image find 3e50a82356599ec1

Downloading uenv

In order to pull images, a local directory for storing the images must first be created, and you will receive an error message. To create a repo in the default location, use the following command:

uenv image repo
> uenv repo create

To use a uenv, it first has to be pulled from the registry to local storage where you can access it. For example, to use the prgenv-gnu  uenv, use the uenv image pull  command:

uenv image pull
# The following commands have the same effect

# method 1: pull using the name of the uenv
> uenv image pull prgenv-gnu/24.2:v1

# method 2: pull using the id of the image
> uenv image pull 3ea1945046d884ee

Some images can be large, over 10 GB, and it can take a while to download from the registry.

To view all uenv that have been pulled, and are ready to use use the uenv image ls  command:

finding uenv images
> uenv image ls
uenv/version:tag                        uarch date       id               size
netcdf-tools/2024:v1                    gh200 2024-04-04 499c886f2947538e 1.2GB
linaro-forge/23.1.2:latest              gh200 2024-04-10 ea67dbb33801c7c3 342MB
icon-wcp/v1:v3                          gh200 2024-03-11 3e8f96370a4685a7 8.3GB
icon-wcp/v1:latest                      gh200 2024-03-11 3e8f96370a4685a7 8.3GB

Accessing Restricted Software

By default, uenv can be freely pulled by all users on a system, with no restrictions.

Some uenv are not available to all users, for example the vasp  images are only available for users who have been added to the vasp6  group once they have provided CSCS with a copy of their VASP6 license.

To be able to pull such images, you first need to configure the token for that specific software. This step only needs to be performed once - once set up you will only need to perform it again if the token is changed, or if you need to use a different token for another uenv.

uenv image pull --token=/capstor/scratch/cscs/bcumming/tokens/vasp6 --username=vasp6 vasp/v6.4.3:v1
As of , the only restricted software is VASP.

work in progress

This workflow is a bit of a hack - we are working on providing command line interface for and token location.

Starting uenv

To start a uenv, use the uenv start  command:

uenv start
# start the image using the name of the uenv
> uenv start --view=modules netcdf-tools/2024:v1

# or use the unqique id of the uenv
> uenv start 499c886f2947538e

# or provide 
> uenv start --view=develop $SCRATCH/my-uenv/gromacs.squashfs

It is possible to verify that the uenv was started using uenv status , for example:

# the contents of the uenv have been mounted at 
> uenv status
/user-environment:icon-wcp
  The tools required to build ICON
  modules: available
  views:
    icon (loaded)
    modules: activate modules
    spack: configure spack upstream

Loading an environment has no impact on other users – the mounted environment is only visible in your terminal. Under the hood, a new mount namespace is created with the image mounted, and the command is executed with that namespace. The environment is automatically cleaned up when you type exit  or uenv stop .

This means that multiple users on a login node can mount their own environment at the same mount point, without interfering with one-another.

Starting multiple images

By default, uenv images are mounted at the /user-environment path, and it is only possible to mount one image at a time at a location. This is a deliberate design decision, however there are some workflows where it is useful to be able to mount multiple images at different locations.

The lack of composability of images is deliberate. It fully isolates each uenv, and so that uenv can be updated independently of one another, without any side effects of changes to a frequently changing uenv on rarely updated stable uenv.

For example, to load a debugging or profiling uenv alongside an application or development uenv. We don't want to provide tools like debuggers and profilers in every single image, so we package them in separate tools uenv.

These second uenv images are mounted at the the /user-tools  path, while the main uenv is mounted at the /user-environment  path. For example, let's load the prgenv-gnu programming environment alongside editors (which provides text editors and other populare CLI tools):

# the uenv are separated using a comma
# views can be disambiguated using the environment name
> uenv start prgenv-gnu/24.11:v1,editors/25.1:test --view=ed,prgenv-gnu:spack

> uenv status
editors:/user-tools
  text editors and handy command line tools
  views:
    spack: configure spack upstream
    modules: activate modules
    ed (loaded):
prgenv-gnu:/user-environment
  GNU Compiler toolchain with cray-mpich, Python, CMake and other development tools.
  views:
    spack (loaded): configure spack upstream
    modules: activate modules
    default:

Views

Uenv provide views, which configure the environment by setting environment variables.

To use a view, use the --view  argument with uenv start  or uenv run  commands

# activate the view named default in prgenv-gnu
> uenv start --view=default prgenv-gnu/24.7:v3

# activate the spack and modules views in prgenv-gnu using a comma-separated list of view names
> uenv start --view=spack,modules prgenv-gnu/24.7:v3

# when starting multiple uenv, you can disambiguate using uenvname:viewname
> uenv start --view=prgenv-gnu:default,editors:ed prgenv-gnu/24.7:v3,editors

Modules

Most uenv provide modules, which can be activated using a special modules  view

Enabling uenv modules
# make modules available when starting the environment
> uenv start --view=modules prgenv-gnu/24.7:v2
loading the view prgenv-gnu:modules

# the modules are now available in your environment
> module avail

-------------------------- /user-environment/modules ---------------------------
   aws-ofi-nccl/master    gcc/13.2.0           ninja/1.11.1
   cmake/3.27.9           hdf5/1.14.3          openblas/0.3.26
   cray-mpich/8.1.29      libtree/3.1.1        osu-micro-benchmarks/5.9
   cuda/12.4.0            meson/1.3.2          python/3.12.1
   fftw/3.3.10            nccl-tests/2.13.6
   fmt/10.2.1             nccl/2.20.3-1

DEPRECATED

The old version of uenv currently provided on Eiger  uses a different method for loading modules.

We recommend updating the version of uenv on Eiger - see instructions at the top of this page.

Spack Upstream

The config  path in the environment path contains the Spack configuration files required, specifically:

  • compilers.yaml : the compilers that are provided by the user environment.
  • packages.yaml : the software provided by the system that was used to build the environment. This is typically the SLURM and libfabric libraries, though individual recipes may use additional software such as git
  • upstreams.yaml : use this to configure the software installed in the user environment as an upstream for your Spack installation.
  • repos.yaml : the custom software provided by the environment, e.g. the custom cray-mpich that provides optimised MPI.

A detailed guide on how to use Spack with uenv is provided here https://eth-cscs.github.io/alps-uenv/tutorial-spack/

Some images provide a spack view, which sets some useful environment variables:

Using the spack view
> uenv run --view=spack editors:/user-tools -- printenv | grep UENV_SPACK
UENV_SPACK_COMMIT=bd66dce2d668a6234504594661506cdd1eaca4adc
UENV_SPACK_CONFIG_PATH=/user-tools/config
UENV_SPACK_URL=https://github.com/spack/spack.git

Using SLURM

To use a user-environment for a SLURM job, the environment needs to be mounted on all of the compute nodes, which is managed using a SLURM plugin.

If an environment has been loaded on the login node when srun  or salloc  SLURM will load the image on all of the compute nodes used by the job. The example below illustrates that by default no environment is loaded by running ls on the compute node with srun , then re-executes the command after loading an environment.

# initially the /user-environment path is empty:
> ls -l /user-environment
total 0

# there is no image mounted on the compute nodes either:
> srun -n1 ls -l /user-environment
total 0

# start a new shell with the uenv mounted at /user-environment
> uenv start --uenv=prgenv-gnu/24.7:v3 --view=default

# the slurm plugin will automatically mount the loaded environment on all of
# the compute nodes used by the job.
> srun -n1 ls /user-environment
bin
config
env
linux-sles15-neoverse_v2
meta
modules

Using the SLURM plugin

The environment to load can be provided directly to slurm via two arguments:

  • --uenv  a comma-separated list of uenv to mount
  • --view  a comma-separated list of views to load


For example, the flags can be used with srun :

# mount the uenv prgenv-gnu with the view named default
> srun --uenv=prgenv-gnu/24.7:v3 --view=default ...

# mount an image at an explicit location (/user-tools)
> srun --uenv=$IMAGES/myenv.squashfs:/user-tools ...

# mount multiple images: use a comma to separate the different options
> srun --uenv=prgenv-gnu/24.7:v3,editors/24.7:v2 --view=default,editors:modules ...


The commands can also be used in sbatch scripts to have fine-grained control: 

sbatch script for uenv
#!/bin/bash

#SBATCH --uenv=editors/24.7:v2
#SBATCH --view=editors:ed
#SBATCH --ntasks=4
#SBATCH --nodes=1
#SBATCH --output=out-%j.out
#SBATCH --error=out-%j.out

echo "==== test in script ===="
# the fd command is provided by the ed view
# use it to inspect the meta data in the mounted image
fd . /user-tools/meta/recipe

echo "==== test in srun ===="
# use srun to launch the parallel job
srun -n4 bash -c 'echo $SLURM_PROCID on $(hostname): $(which emacs)'

echo "==== alternative mount ===="
srun -n4 --uenv=prgenv-gnu --view=prgenv-gnu:default bash -c 'echo $SLURM_PROCID on $(hostname): $(which mpicc)'
sbatch output
==== test in script ====
/user-tools/meta/recipe/compilers.yaml
/user-tools/meta/recipe/config.yaml
/user-tools/meta/recipe/environments.yaml
/user-tools/meta/recipe/modules.yaml
==== test in srun ====
1 on nid007144: /user-tools/env/ed/bin/emacs
3 on nid007144: /user-tools/env/ed/bin/emacs
0 on nid007144: /user-tools/env/ed/bin/emacs
2 on nid007144: /user-tools/env/ed/bin/emacs
==== alternative mount ====
0 on nid007144: /user-environment/env/default/bin/mpicc
1 on nid007144: /user-environment/env/default/bin/mpicc
2 on nid007144: /user-environment/env/default/bin/mpicc
3 on nid007144: /user-environment/env/default/bin/mpicc

In the example above, the #SBATCH --uenv  and SBATCH --view  parameters in the preamble of the sbatch script set the default uenv to editors  with the view ed .

  • editors is mounted and the view set in the script (the "test in script" part)
  • editors  is also mounted in the first call to srun (which does not provide a –uenv  flag)

it is possible to override the default uenv by passing a different --uenv  and --view  flags to an srun  call inside the script, as is done in the second srun  call. Note how the second call relies on mpicc being provided by prgenv-gnu.

Using the SLURM plugin on Eiger

An older version of the SLURM plugin is currently installed on Eiger, so only one uenv can be loaded at a time with the following arguments to srun/sbatch:

  • --uenv: the user environment image file or simple label (like cp2k/2024.3:v1 )

The --view  argument to srun/sbatch does not work. If you are using a sbatch script, you can use uenv view to load the view before calling srun .