Internal Transfer

The Slurm queue xfer is available on Piz Daint only to address data transfers between internal CSCS file systems. The queue has been created to transfer files and folders from /users, /project or /store to the /capstor/scratch and /scratch file systems (stage-in) and vice versa (stage-out). Currently the following commands are available on the cluster supporting the queue xfer:

cp
mv
rm
rsync

You can adjust the Slurm batch script below to transfer your input data on $SCRATCH, setting the variable command to the unix command that you intend to use, choosing from the list given above:

#!/bin/bash -l
#
#SBATCH --time=02:00:00
#SBATCH --ntasks=1
#SBATCH --partition=xfer

command="rsync -av"
echo -e "$SLURM_JOB_NAME started on $(date):\n $command $1 $2\n"
srun -n $SLURM_NTASKS $command $1 $2
echo -e "$SLURM_JOB_NAME finished on $(date)\n"
 
if [ -n "$3" ]; then
 # unset memory constraint enabled on xfer partition
 unset SLURM_MEM_PER_CPU
 # submit job with dependency
 sbatch --dependency=afterok:$SLURM_JOB_ID $3
fi  

The template Slurm batch script above requires at least two command line arguments, which are the source and the destination files (or folders) to be copied. The stage script may take as third command line argument the name of the production Slurm batch script to be submitted after the stage job: the Slurm dependency flag --dependency=afterok:$SLURM_JOB_ID ensures that the production job can begin execution only after the stage job has successfully executed (i.e. ran to completion with an exit code of zero).

You can submit the stage job with a meaningful job name as below:

# stage-in and production jobs
$ sbatch --job-name=stage_in stage.sbatch ${PROJECT}/<source> ${SCRATCH}/<destination> production.sbatch

The Slurm flag --job-name will set the name of the stage job that will be printed in the Slurm output file: the latter is by default the file slurm-${SLURM_JOB_ID}.out, unless you set a specific name for output and error using the Slurm flags -e/--error and/or -o/--output (e.g.: -o %j.out -e %j.err, where the Slurm symbol %j will be replaced by $SLURM_JOB_ID). The stage script will also submit the Slurm batch script production.sbatch given as third command line argument. The production script can submit in turn a stage job to transfer the results back. E.g.:

# stage-out
sbatch --dependency=afterok:${SLURM_JOB_ID} --job-name=stage_out stage.sbatch ${SCRATCH}/<source> ${PROJECT}/<destination>