Uenv are user environments that provide scientific applications, libraries and tools on Alps. This article will explain how to activate uenv, use them to build software and how to enable them in SLURM jobs.

Unlike the standard Cray Programming Environment (CPE), which aims to provide a complete software stack that can be configured to build all types of software on all node architectures, uenv are typically application-specific, domain-specific or tool-specific - each uenv contains only what is required for the application or tools that it provides.

User environments are packaged in a single file (in the  SquashFS file format), that stores a compressed directory tree that contains all of the software, tools and other information like modules, required to provide a rich environment.

Each environment contains a software stack, comprised of compilers, libraries, tools and scientific applications, built using Spack.

For help with uenv create a "Software Environment" request at the CSCS Service Desk.

For more information on how CSCS builds user environments, have a look at the documentation for the Stackinator tool that CSCS use to configure Spack to build environments with optimisations for the Alps architecture.

This tool can also be used by motivated users to provide their own software stacks.

Notices

known bugs and issues

'ascii' code can't encode character '\u2261'

If you see an error like the following when pulling an image:

> uenv image pull icon-wcp/v1:rc3
downloading image 8e3fccb3a887964c3b0f5cbf4f0cbd0937480f0042574b7515b3af1edcdad709 7.9GB
  [=                                                 ]   1% 125/8128 MB
[error] image pull failed with unknown exception: 'ascii' codec can't encode character '\u2261' in position 22: ordinal not in range(128)
The fix is to set the environment variable export PYTHONIOENCODING=utf-8

Update uenv on EIger

The version of uenv installed on Eiger does not support new uenv features, such as improved Spack integration and module files.

We strongly recommend installing the latest version (see the Installing instructions.)

Todi update: 2024-07-03

On the uenv tools provided on Todi were updated. This update makes some significant changes, that will affect user workflows.

There was a change to the database format used to record information about uenv images that have been pulled (uenv image pull )

  • an error message like the following will be printed the first time you use uenv: The repository /capstor/scratch/cscs/user/.uenv-images needs to be upgraded 
  • run the command uenv repo upgrade  to update your registry

The uenv module command was removed:

  • going forward, the same functionality will be provided as a view: uenv view modules  or uenv start --view=modules prgenv-gnu/24.7:v2  are two ways to make modules available.
  • if you are using an old image you will have to manually load the modules once the uenv has started:
    • module use /user-environment/modules
    • manual loading is only required when uenv status does not list modules  as an available view

Getting Started

After logging into an Alps cluster, you can quickly check the availability of uenv with the following commands:

downloading and installing uenv
> uenv status
there is no uenv loaded
> uenv --version
5.0.0

Installing

The command line tool can be installed from source, if you are working on a cluster that does not have uenv installed, or if you need to test a new version.

locally installing uenv
> git clone https://github.com/eth-cscs/uenv.git
> cd uenv
# run the installation script.
# if this is the first time that you are installing uenv,
# type "yes" or "y" when prompted whether to update your ~/.bashrc file.
> ./install --local

# log out then back in again and type "uenv --version" to verify that uenv has been installed.
Once uenv has been installed, log out and back in again for it to be enabled in your environment

Finding uenv

Uenv for programming environments, tools and applications are provided by CSCS on each Alps system.

The same uenv are not installed on every system. Instead uenv that are supported for the users of that platform are provided.

The available uenv images are stored in a registry, that can be queried using the uenv image find  command:

uenv image find
> uenv image find
uenv/version:tag                        uarch date       id               size
prgenv-gnu/24.2:v1                      gh200 2024-05-23 3ea1945046d884ee 3.7GB
prgenv-gnu/24.2:latest                  gh200 2024-05-23 3ea1945046d884ee 3.7GB
namd/3.0b6:v1                           gh200 2024-05-29 e8606d64eb8dcd2f 3.7GB
namd/3.0b6:latest                       gh200 2024-05-29 e8606d64eb8dcd2f 3.7GB
cp2k/2024.1:v1                          gh200 2024-05-29 af81b3bbbe68a62f 3.9GB
cp2k/2024.1:latest                      gh200 2024-05-29 af81b3bbbe68a62f 3.9GB
arbor/v0.9:v1                           gh200 2024-05-23 69fdbd75e246d439 3.6GB
arbor/v0.9:latest                       gh200 2024-05-23 69fdbd75e246d439 3.6GB

The output above shows that there are four uenv (prgenv-gnu, namd , cp2k and arbor ).

The full name of a uenv is of the form uenv/version:tag  

  • uenv  is the name of the uenv.
  • version  the version of the uenv, typically a in the form year , year.month or matching the version of the software package provided by the uenv (e.g version 3.0b6 of NAMD above)
  • tag : incremental tag applied to versions when they are deployed with an update, e.g v1 , v2  or latest .

The id  of the image is a unique identifier derived from the sha256 hash of the image.

The full name and id can both be used to refer to a specific uenv.

Pulling uenv

To use a uenv, it first has to be pulled from the registry to local storage where you can access it. For example, to use the prgenv-gnu  uenv, use the uenv image pull  command:

uenv image pull
# The following commands have the same effect

# method 1: pull using the name of the uenv
> uenv image pull prgenv-gnu/24.2:v1

# method 2: pull using the id of the image
> uenv image pull 3ea1945046d884ee

In order to pull images, a local directory for storing the images must first be created, and you will receive an error message. To create a repo in the default location, use the following command:

uenv image repo
> uenv repo create

Some images can be large, over 10 GB, and it can take a while to download from the registry.

To view all uenv that have been pulled, and are ready to use use the uenv image ls  command:

finding uenv images
> uenv image ls
uenv/version:tag                        uarch date       id               size
netcdf-tools/2024:v1                    gh200 2024-04-04 499c886f2947538e 1.2GB
linaro-forge/23.1.2:latest              gh200 2024-04-10 ea67dbb33801c7c3 342MB
icon-wcp/v1:v3                          gh200 2024-03-11 3e8f96370a4685a7 8.3GB
icon-wcp/v1:latest                      gh200 2024-03-11 3e8f96370a4685a7 8.3GB

Accessing Restricted Software

By default, uenv can be freely pulled by all users on a system, with no restrictions.

Some uenv are not available to all users, for exampl the vasp  images are only available for users who have a VASP license, who are added to the vasp6  group once then have provided CSCS with a copy of their license.

To be able to pull such images, you first need to configure the token for that specific software. This step only needs to be performed once - once set up you will only need to perform it again if the token is changed, or if you need to use a different token for another uenv.

cat /capstor/scratch/cscs/bcumming/tokens/vasp6 | /opt/cscs/uenv/libexec/uenv-oras login jfrog.svc.cscs.ch --username admin --password-stdin
As of , the only restricted software is VASP.

work in progress

This workflow is a bit of a hack - we are working on providing command line interface for and token location.

Starting uenv

To start a uenv, use the uenv start  command:

uenv start
# start the image using the name of the uenv
> uenv start netcdf-tools/2024:v1

# or use the unqique id of the uenv
> uenv start 499c886f2947538e

# legacy mode: if you have a squashfs image in a file
> uenv start $SCRATCH/my-uenv/gromacs.squashfs

The uenv will now be mounted at /user-environment . It is possible to verify that the uenv was started using uenv status , for example:

# the contents of the uenv have been mounted at 
> uenv status
/user-environment:icon-wcp
  The tools required to build ICON
  modules: available
  views:
    icon (loaded)
    modules: activate modules
    spack: configure spack upstream

Loading an environment has no impact on other users – the mounted environment is only visible in your terminal. Under the hood, a new mount namespace is created with the image mounted, and the command is executed with that namespace. The environment is automatically cleaned up when you type exit  or uenv stop .

This means that multiple users on a login node can mount their own environment at the same mount point, without interfering with one-another.

Loading multiple images

By default, uenv images are mounted at the /user-environment path, and it is only possible to mount one image at a time at a location. This is a deliberate design decision, however there are some workflows where it is useful to be able to mount multiple images at different locations.

The lack of composability of images is deliberate. It fully isolates each uenv, and so that uenv can be updated independently of one another, without any side effects of changes to a frequently changing uenv on rarely updated stable uenv.

For example, to load a debugging or profiling uenv alongside an application or development uenv. We don't want to provide tools like debuggers and profilers in every single image, so we package them in separate tools uenv.

These second uenv images are mounted at the the /user-tools  path, while the main uenv is mounted at the /user-environment  path. For example, let's load the prgenv-gnu programming environment alongside editors (which provides text editors and other populare CLI tools):

# start the two uenv
> uenv start prgenv-gnu/24.7 editors/24.7

# the status command shows both iamges, and where they are mounted.
> uenv status
/user-environment:prgenv-gnu
  GNU Compiler toolchain with cray-mpich, Python, CMake and other development tools.
  views:
    develop
    modules: activate modules
    spack: configure spack upstream

/user-tools:editors
  text editors and handy command line tools
  views:
    ed
    modules: activate modules
    spack: configure spack upstream

# note from above: by default the first uenv is mounted at /user-environment
# and the second at /user-tools
# custom mount points can be specified with a : on the cli.
# if mounting a tool image on its own, you want to override it's mount point this way:
> uenv start editors:/user-tools

# views can be disambiguated using the environment name
> uenv start --view=prgenv-gnu:default,editors:modules prgenv-gnu/24.7 editors/24.7
loading the views prgenv-gnu:develop editors:modules

Views

Uenv provide views, which configure the environment by setting environment variables.

To use a view, use the --view  argument with uenv start  or uenv run  commands

# activate the view named default in prgenv-gnu
> uenv start --view=default prgenv-gnu/24.7:v3

# activate the spack and modules views in prgenv-gnu using a comma-separated list of view names
> uenv start --view=spack,modules prgenv-gnu/24.7:v3

# when starting multiple uenv, you can disambiguate using uenvname:viewname
> uenv start --view=prgenv-gnu:default,editors:ed prgenv-gnu/24.7:v3,editors

Modules

Most uenv provide modules, which can be activated using a special modules  view

Enabling uenv modules
# make modules available when starting the environment
> uenv start --view=modules prgenv-gnu/24.7:v2
loading the view prgenv-gnu:modules

# the modules are now available in your environment
> module avail

-------------------------- /user-environment/modules ---------------------------
   aws-ofi-nccl/master    gcc/13.2.0           ninja/1.11.1
   cmake/3.27.9           hdf5/1.14.3          openblas/0.3.26
   cray-mpich/8.1.29      libtree/3.1.1        osu-micro-benchmarks/5.9
   cuda/12.4.0            meson/1.3.2          python/3.12.1
   fftw/3.3.10            nccl-tests/2.13.6
   fmt/10.2.1             nccl/2.20.3-1

DEPRECATED

The old version of uenv currently provided on Eiger  uses a different method for loading modules.

We recommend updating the version of uenv on Eiger - see instructions at the top of this page.

Spack Upstream

The config  path in the environment path contains the Spack configuration files required, specifically:

  • compilers.yaml : the compilers that are provided by the user environment.
  • packages.yaml : the software provided by the system that was used to build the environment. This is typically the SLURM and libfabric libraries, though individual recipes may use additional software such as git
  • upstreams.yaml : use this to configure the software installed in the user environment as an upstream for your Spack installation.
  • repos.yaml : the custom software provided by the environment, e.g. the custom cray-mpich that provides optimised MPI.

A detailed guide on how to use Spack with uenv is provided here https://eth-cscs.github.io/alps-uenv/tutorial-spack/

Some images provide a spack view, which sets some useful environment variables:

Using the spack view
> uenv run --view=spack editors:/user-tools -- printenv | grep UENV_SPACK
UENV_SPACK_COMMIT=bd66dce2d668a6234504594661506cdd1eaca4adc
UENV_SPACK_CONFIG_PATH=/user-tools/config
UENV_SPACK_URL=https://github.com/spack/spack.git

Using SLURM

To use a user-environment for a SLURM job, the environment needs to be mounted on all of the compute nodes, which is managed using a SLURM plugin.

If an environment has been loaded on the login node when srun  or salloc  SLURM will load the image on all of the compute nodes used by the job. The example below illustrates that by default no environment is loaded by running ls on the compute node with srun , then re-executes the command after loading an environment.

# initially the /user-environment path is empty:
> ls -l /user-environment
total 0

# there is no image mounted on the compute nodes either:
> srun -n1 ls -l /user-environment
total 0

# start a new shell with the uenv mounted at /user-environment
> uenv start --uenv=prgenv-gnu/24.7:v3 --view=default

# the slurm plugin will automatically mount the loaded environment on all of
# the compute nodes used by the job.
> srun -n1 ls /user-environment
bin
config
env
linux-sles15-neoverse_v2
meta
modules

Using the SLURM plugin

The environment to load can be provided directly to slurm via two arguments:

  • --uenv  a comma-separated list of uenv to mount
  • --view  a comma-separated list of views to load


For example, the flags can be used with srun :

# mount the uenv prgenv-gnu with the view named default
> srun --uenv=prgenv-gnu/24.7:v3 --view=default ...

# mount an image at an explicit location (/user-tools)
> srun --uenv=$IMAGES/myenv.squashfs:/user-tools ...

# mount multiple images: use a comma to separate the different options
> srun --uenv=prgenv-gnu/24.7:v3,editors/24.7:v2 --view=default,editors:modules ...


The commands can also be used in sbatch scripts to have fine-grained control: 

sbatch script for uenv
#!/bin/bash

#SBATCH --uenv=editors/24.7:v2
#SBATCH --view=editors:ed
#SBATCH --ntasks=4
#SBATCH --nodes=1
#SBATCH --output=out-%j.out
#SBATCH --error=out-%j.out

echo "==== test in script ===="
# the fd command is provided by the ed view
# use it to inspect the meta data in the mounted image
fd . /user-tools/meta/recipe

echo "==== test in srun ===="
# use srun to launch the parallel job
srun -n4 bash -c 'echo $SLURM_PROCID on $(hostname): $(which emacs)'

echo "==== alternative mount ===="
srun -n4 --uenv=prgenv-gnu --view=prgenv-gnu:default bash -c 'echo $SLURM_PROCID on $(hostname): $(which mpicc)'
sbatch output
==== test in script ====
/user-tools/meta/recipe/compilers.yaml
/user-tools/meta/recipe/config.yaml
/user-tools/meta/recipe/environments.yaml
/user-tools/meta/recipe/modules.yaml
==== test in srun ====
1 on nid007144: /user-tools/env/ed/bin/emacs
3 on nid007144: /user-tools/env/ed/bin/emacs
0 on nid007144: /user-tools/env/ed/bin/emacs
2 on nid007144: /user-tools/env/ed/bin/emacs
==== alternative mount ====
0 on nid007144: /user-environment/env/default/bin/mpicc
1 on nid007144: /user-environment/env/default/bin/mpicc
2 on nid007144: /user-environment/env/default/bin/mpicc
3 on nid007144: /user-environment/env/default/bin/mpicc

In the example above, the #SBATCH --uenv  and SBATCH --view  parameters in the preamble of the sbatch script set the default uenv to editors  with the view ed .

  • editors is mounted and the view set in the script (the "test in script" part)
  • editors  is also mounted in the first call to srun (which does not provide a –uenv  flag)

it is possible to override the default uenv by passing a different --uenv  and --view  flags to an srun  call inside the script, as is done in the second srun  call. Note how the second call relies on mpicc being provided by prgenv-gnu.

Using the SLURM plugin on Eiger

An older version of the SLURM plugin is currently installed on Eiger, so only one uenv can be loaded at a time with the following arguments to srun/sbatch:

  • --uenv: the user environment image file or simple label (like cp2k/2024.3:v1 )

The --view  argument to srun/sbatch does not work. If you are using a sbatch script, you can use uenv view to load the view before calling srun .