Daint (daint.alps) is a vCluster deployed on Alps for the HPC Platform. It replaces the Cray XC Piz Daint, which has reached end-of-life. 


Maintenance

  • Wednesday morning 8-12 is reserved for periodic updates, with services potentially unavailable during this timeframe. If the queues must be drained (redeployment of node images, rebooting of compute nodes, etc) then a Slurm reservation will be in place that will prevent jobs from running into the maintenance window. 
  • Exceptional and non-disruptive updates may happen outside this time frame and will be announced to the users mailing list

Change log

 

  1. The number of compute nodes has been increased to 1018
  2. The restriction on the number of running jobs per project has been lifted.
  3. A "low" priority partition has been added, which allows some project types to consume up to 130% of the project's quarterly allocation
  4. We have increased the power cap for the GH module from 624 to 660 W. You might see increased application performance as a consequence 
  5. Small changes in kernel tuning parameters

 

  1. Scheduler updated to enforce a maximum 10 running jobs per project. Jobs will pend with Reason=AssocGrpJobsLimit 
  2. The Enroot vService is updated from v2.11.0 to v2.12.0. Enroot is the container runtime that is used in the Container Engine on Daint.Alps. Changes in v2.12.0:
    • Re-introduced Enroot hook for PMIx support under Slurm.
      • This hook is a stripped-down version of the upstream Enroot hook for Slurm/PMI support: it contains only actions (e.g. env vars and mounts) specific to PMIx and does not affect the use of other PMI implementations.
    • SSH hook: Fixes for exposing environment variables to remote connections

 

The Enroot vService is updated from v2.10.0 to v2.11.0. Enroot is the container runtime that is used in the Container Engine on Daint.Alps. Changes in v2.11.0:

  • Removed the Slurm PMI/PMIx hook which was causing issues with MPICH containers when used with multiple ranks on the same node. The situation should be resolved, please let us know if you still encounter related problems. A revisited, dedicated hook to support PMIx and OpenMPI containers will be provided in the coming weeks.
  • Improvements to the SSH hook:
    • Enforced correct permissions to authorized key files within containers
    • The container environment variables are now reproduced on SSH connections as well, for a better interactive experience. The current implementation might cause issues with the CUDA forward compatibility provided by entrypoint scripts in some NVIDIA images; this will be addressed as soon as possible.

 

  1. Scheduler updated to enforce quarterly allocation compute budgets
  2. Slingshot Fabric Manager and switch firmware to Slingshot version 2.3

 

The Enroot vService is updated from v2.8.4 to v2.10.0. Enroot is the container runtime that is used in the Container Engine on Daint.Alps. Notable changes:

  • Enroot container filesystems are now writable by default: it's no longer necessary to force writable = true in EDFs
  • An Enroot hook for PMIx propagation in containers is now installed. The hook acts automatically when --mpi=pmix is used as a Slurm option. This makes working with OpenMPI containers easier.
  • The CUDA MPS hook for Enroot is now deployed correctly
  • The Container Engine documentation has been updated to reflect these changes
    • Changelog v2.10.0:
      • Updated default Enroot version to CSCS_2024_10_1:
        • Fixed startup errors on some systems by mounting /dev/shm with recursive bind by default
        • Fixed bugs causing the deployment script to exit abruptly without completing the installation process
        • Fixed installation of the CUDA MPS hook
        • Fixed activation condition of the CUDA MPS hook when used outside of a Slurm job
    • Changelog v2.9.0
      • Added hook adapter for the Sarus MPI hook
      • Slurm hook: Added dependencies for the new Slurm policies plugin
      • Install Enroot PMIx hook
      • Added creation of a symlink to the fusermount3 program
      • AWS OFI NCCL hook: Moved the mount of specific dependency libraries from the CXI hook to the AWS OFI NCCL hook

 

Slurm fairshare scheduling parameters are adjusted to better reflect project usage across quarterly allocation periods. 

 

The  $PROJECT  environment variable is now defined for all users of Daint.Alps and it targets the user's primary project folder. Please note that $USER  is no longer appended to $PROJECT: you can find more details on the dedicated page /capstor/store.

 

  1. The quota  command is now available on the login nodes.
  2. MPICH_GPU_SUPPORT_ENABLED=1 has been set as default for all users. This is a breaking change for applications that do not make use of GPU-aware MPI. If your application crashes with an error similar to MPIDI_CRAY_init: GPU_SUPPORT_ENABLED is requested, but GTL library is not linked you will need to export MPICH_GPU_SUPPORT_ENABLED=0  in your batch scripts.  For details of the behaviour of this variable see https://confluence.cscs.ch/x/40HJMw. 

Maintenance day is changed from Tuesday to Wednesday morning, see above. 

The version of uenv  on daint, clariden and santis was been updated this morning with new features, bug fixes and other improvements. It also introduces some breaking changes, the main one is that the uenv view command is no longer supported:

  • provide the view when starting the environment, e.g. uenv start --view=gromacs gromacs/2024:v1

Please check the updated documentation available on the dedicated page uenv user environments

The Slurm queue xfer  is available for the Internal Transfer on Daint. The data mover nodes mount temporarily with read-only access  /project.CrayXC, /store.CrayXC and /users.CrayXC that correspond to the GPFS mount points /project, /store and /users currently available on Piz Daint XC login nodes

Known issues

NCCL and libfabric incompatibility

We're aware of an incompatibility between the NCCL AWS plugin (aws-ofi-nccl version 1.9.2-aws) and Libfabric (libfabric version 1.22). This problem will cause codes to crash when using the prgenv-gnu/24.11:v1 uenv with NCCL. A fix is ready and should be deployed on 16 April 2025.

Container Engine: MPICH unable to allocate shared memory

We're aware of an issue affecting open source MPICH libraries from using shared memory when launching multiple ranks per node. See Container Engine#MPICHunabletoallocatesharedmemory for further details. 

Fabric-related slowdown

Following the update of the Slingshot fabric on 12 March 2025, Alps is experiencing fabric-related slowdowns which may negatively impact workloads. HPE engineers are investigating the issue.

Storage quotas on account.cscs.ch

The scratch and project-related storage paths shown in https://account.cscs.ch are not reporting capstor  usage, since the UI is deprecated and will be replaced by a new user management portal. Please use the quota command instead.

Quota command 

The quota command is unavailable on the login nodes. Issue resolved.

Jobs silently crashing in Slurm prolog

We are investigating an issue where jobs launched by srun  are silently crashing in the Slurm prolog and being removed from the queue. Issue resolved

$ [daint][user@daint-ln001 ~]$ srun [options] <executable>
srun: job 152105 queued and waiting for resources
<hanging, but the job has been cancelled by slurm>

Access

Log in as you would for other CSCS systems, by first configuring your SSH keys (see the MFA documentation for more information), then log into Daint via the front end server ela.cscs.ch:

$ ssh -A ela.cscs.ch
$ ssh daint.alps.cscs.ch


Simplifying log in

To log in directly to Daint without first logging into ela.cscs.ch , you can add the following configuration to the ~/.ssh/config  file on your laptop or PC:

~/.ssh/config
Host ela
    HostName ela.cscs.ch
    User <<username>>
    IdentityFile ~/.ssh/cscs-key
Host daint.alps
    HostName daint.alps.cscs.ch
    User <<username>>
    IdentityFile ~/.ssh/cscs-key
    ProxyJump ela
    AddKeysToAgent yes
    ForwardAgent yes

Where cscsusername is your CSCS account name.

Now you can access Daint directly from your laptop or PC:

ssh daint.alps


Cluster Specifications

All nodes are identical, with 4 Grace-Hopper modules per node. Specifically:

  • User Access Node (UAN): 4 repurposed GH compute nodes that serve as login nodes (daint-ln00[1-4])
  • Compute Node (CN): The number of compute nodes will change over time. You can get an up-to-date number using the command sinfo -s on the UAN. The majority of the nodes is provided in the Slurm partition normal, while a smaller number is accessible through the partition debug, meant for short test jobs with a quick turnaround   

Each node has approximately 800GB of free memory accessible from all sockets. Each Grace CPU has 72 cores with the following specification:

  • Arm V9.0 ISA compliant aarch64 (Neoverse V2 “Demeter”architecture)
  • Full SVE-2 Vector Extensions support, inclusive of NEON instructions
  • Supports 48-bit virtual and 48-bit physical address space

Each Hopper GPU has 96GB of RAM. NVLINK provides all-to-all cache-coherent memory between all host and device memory.

A login node is a shared resource. Do not run compute-intensive jobs on a login node, and do not start the CUDA MPS service there, as you might impact the work of others. Please have a look at the Policies that apply on CSCS computing systems 

Programming Environment

uenv

User environments uenv are used to provide programming environments and application software. Please refer to the uenv  documentation for detailed information on how to use the uenv  tools on the system.
You can list the uenv provided on the system with the command uenv image find. A non exhaustive list of the software provided by uenv images currently available on the system is shown below:


Please note that uenv images provided on the system Todi should also work on Daint. They can be accessed via

CLUSTER_NAME=todi uenv image find

Please always prepend CLUSTER_NAME=todi to uenv commands that should address an image available on Todi.

Cray Programming Environment (CPE)

CPE is provided on Daint, however CSCS does not officially support CPE or provide software built with CPE on the system.
The supported method for building software is uenv (see above): this is a key difference from the previous system Piz Daint (Cray XC).

To enable Cray Programming Environment (CPE), please run

$ module load cray
$ module load PrgEnv-cray

# to check that CPE is loaded:
$ ftn --version
Cray Fortran : Version 17.0.0

Container engine

The Container Engine (CE) is available on the system: this is a toolset is designed to enable computing jobs to seamlessly run inside Linux application containers, thus providing support for containerized user environments: please see the dedicated page to use CE.

File systems

The following mount points are available:

  • /users/$USER - $HOME
  • /capstor/scratch/cscs/$USER  - $SCRATCH
  • /capstor/store - $PROJECT  

The environment variable $HOME and $SCRATCH will give you have access to the user dedicated folders /users/$USER and /capstor/cscs/scratch/$USER respectively. The $PROJECT environment variable targets your personal folder /capstor/store/cscs/userlab/group_id>/$USER only for UserLab customers on the capstor storage: please note that users need to create their own sub-folders under the project folder, as they are not created automatically.

Please check the occupancy by your user (data volume and number of inodes) on the different file systems with the quota command, that is currently available on the frontend Ela.

Slurm configuration

Currently there is no fair share policy applied.

PartitionMax timeMax nodesBrief Description
normal24 h568Standard queue for production work
debug30 min10Quick turnaround for test jobs (one per user)
xfer24 h1Internal transfer queue

Nodes are not shared (except for the xfer queue) and at least 1 node must be allocated for your job.

Running jobs

Please have a look at an introduction to the Slurm workload manager in the page Running jobs.

The Slurm option --account=<project> or -A <project> selects the active <project> that will be charged for the allocation. On Daint.Alps , the user's primary account will be charged by default if the option is not set. See groups $USER

In the current Slurm configuration OMP threads are placed consecutively on the cores and MPI ranks are placed in the round-robin fashion between 4 sockets. Example below demonstrates the output of running 8 MPI ranks with 4 cores/rank:

$ srun -N1 -c4 -n8 ./a.out
MPI rank : 6, OMP thread id : 0, CPU core id : 149
MPI rank : 6, OMP thread id : 3, CPU core id : 148
MPI rank : 6, OMP thread id : 1, CPU core id : 150
MPI rank : 6, OMP thread id : 2, CPU core id : 151
MPI rank : 0, OMP thread id : 1, CPU core id : 2
MPI rank : 0, OMP thread id : 0, CPU core id : 1
MPI rank : 0, OMP thread id : 3, CPU core id : 0
MPI rank : 0, OMP thread id : 2, CPU core id : 3
MPI rank : 4, OMP thread id : 0, CPU core id : 5
MPI rank : 4, OMP thread id : 2, CPU core id : 7
MPI rank : 4, OMP thread id : 3, CPU core id : 4
MPI rank : 4, OMP thread id : 1, CPU core id : 6
MPI rank : 5, OMP thread id : 0, CPU core id : 76
MPI rank : 5, OMP thread id : 3, CPU core id : 79
MPI rank : 5, OMP thread id : 2, CPU core id : 78
MPI rank : 5, OMP thread id : 1, CPU core id : 77
MPI rank : 1, OMP thread id : 0, CPU core id : 73
MPI rank : 1, OMP thread id : 3, CPU core id : 72
MPI rank : 1, OMP thread id : 2, CPU core id : 75
MPI rank : 1, OMP thread id : 1, CPU core id : 74
MPI rank : 3, OMP thread id : 0, CPU core id : 216
MPI rank : 3, OMP thread id : 1, CPU core id : 217
MPI rank : 3, OMP thread id : 2, CPU core id : 218
MPI rank : 3, OMP thread id : 3, CPU core id : 219
MPI rank : 7, OMP thread id : 0, CPU core id : 220
MPI rank : 7, OMP thread id : 3, CPU core id : 221
MPI rank : 7, OMP thread id : 1, CPU core id : 223
MPI rank : 7, OMP thread id : 2, CPU core id : 222
MPI rank : 2, OMP thread id : 0, CPU core id : 147
MPI rank : 2, OMP thread id : 3, CPU core id : 144
MPI rank : 2, OMP thread id : 2, CPU core id : 145
MPI rank : 2, OMP thread id : 1, CPU core id : 146

Oversubscription of GPU cards

If you want to share GPU card(s) between multiple MPI ranks, you currently need to start multi-process daemon on the node yourself. This is due to the fact that CRAY_CUDA_MPS variable is no longer supported. To do so, you need to use a simple wrapper script:

MPS wrapper script
#!/bin/bash
# Example mps-wrapper.sh usage:
# > srun [srun args] mps-wrapper.sh [cmd] [cmd args]
export CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps
export CUDA_MPS_LOG_DIRECTORY=/tmp/nvidia-log
export CUDA_VISIBLE_DEVICES=$(( SLURM_LOCALID % 4 ))
# Launch MPS from a single rank per node
if [ $SLURM_LOCALID -eq 0 ]; then
    CUDA_VISIBLE_DEVICES=0,1,2,3 nvidia-cuda-mps-control -d
fi
# Wait for MPS to start
sleep 5
# Run the command
"$@"
# Quit MPS control daemon before exiting
if [ $SLURM_LOCALID -eq 0 ]; then
    echo quit | nvidia-cuda-mps-control
fi

and run your code using the following sample slurm script:

Batch submission script
#!/bin/bash -l
#SBATCH --job-name=<job_name>
#SBATCH --time=01:30:00 #HH:MM:SS
#SBATCH --nodes=2
#SBATCH --ntasks-per-core=1
#SBATCH --ntasks-per-node=32 #32 MPI ranks per node
#SBATCH --account=<account> 
#SBATCH --hint=nomultithread 
#SBATCH --hint=exclusive

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
export MPICH_MALLOC_FALLBACK=1

ulimit -s unlimited

srun --cpu-bind=socket ./mps-wrapper.sh <code> <args>