Versions Compared

Old Version 1

changes.mady.by.user Gwangmu Lee (CSCS)

Saved on

compared with

New Version 2

changes.mady.by.user Alberto Madonna (CSCS)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

It is required to keep the container writable (true by default) to use the hook.

The following EDF file shows an example of enabling the SSH hook and authorizing a user-provided public key:

Code Block
[<vcluster>][<username>@<vcluster>-ln001 ~]$ cat $HOME/.edf/ubuntu-ssh.toml
image = "ubuntu:latest"
writable = true

[annotations.com.hooks.ssh]
enabled = "true"
authorize_ssh_key = "<public key file>"

...

It is required to keep the container writable (true by default) to be able to use the hook.

...

Code Block
[<vcluster>][<username>@<vcluster>-ln001 ~]$ cat $HOME/.edf/vectoradd-cuda-mps.toml
image = "nvcr.io#nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04"
writable = true

[annotations]
com.hooks.nvidia_cuda_mps.enabled = "true"

[<vcluster>][<username>@<vcluster>-ln001 ~]$ srun -t2 -N1 -n8 --environment=vectoradd-cuda-mps /cuda-samples/vectorAdd | grep "Test PASSED" | wc -l
8

...

ParameterDescription

base_environment

Type: ARRAY or BASIC STRING

Default: N/A

Ordered list of EDFs that this file inherits from. Parameters from listed environments are evaluated sequentially. Supports up to 10 levels of recursion.

Expand
titleNotes
  • Parameters from the listed environments are evaluated sequentially, adding new entries or overwriting previous ones, before evaluating the parameters from the current EDF. In other words, the current EDF inherits the parameters from the EDFs listed in base_environment. When evaluating mounts or env parameters, values from downstream EDFs are appended to inherited values.
  • The individual EDF entries in the array follow the same search rules as the arguments of the --environment CLI option for Slurm; they can be either file paths or filenames without extension if the file is located in the EDF search path.
  • This parameter can be a string if there is only one base environment.

Examples

Expand
titleExamples

1. Single environment inheritance:

Code Block
base_environment = "common_env"

2. Multiple environment inheritance:

Code Block
base_environment = ["common_env", "ml_pytorch_env1"]

image

Type: BASIC STRING

Default: N/A

The container image to use. Can reference a remote Docker/OCI registry or a local Squashfs file as a filesystem path.

Expand
titleNotes
  • The full format for remote references is [USER@][REGISTRY#]IMAGE[:TAG].
    • [REGISTRY#] : (optional) registry URL, followed by #. Default: Docker Hub.
    • IMAGE : image name.
    • [:TAG] : (optional) image tag name, preceded by :.
  • The registry user can also be specified in the $HOME/.config/enroot/.credentials file.
  • The default registry is Docker Hub (docker.io).

Examples

Expand
titleExamples

1. Reference of Ubuntu image in the Docker Hub registry (default registry)

Code Block
image = "library/ubuntu:24.04"

2. Explicit reference of Ubuntu image in the Docker Hub registry

Code Block
image = "docker.io#library/ubuntu:24.04"

3. Reference to PyTorch image from NVIDIA Container Registry (nvcr.io)

Code Block
image = "nvcr.io#nvidia/pytorch:22.12-py3"

5. Image from third-party quay.io registry

Code Block
image = "quay.io#madeeks/osu-mb:6.2-mpich4.1-ubuntu22.04-arm64"

4. Reference to a manually pulled image stored in parallel FS

Code Block
image = "/path/to/image.squashfs"

workdir

Type: BASIC STRING

Default:

Inherited from image

Initial working directory when the container starts.

Examples

Expand
titleExamples

1. Workdir pointing to a user defined project path 

Code Block
workdir = "/home/user/projects"

2. Workdir pointing to the /tmp directory

Code Block
workdir = "/tmp"

entrypoint

Type: BOOL

Default: true

If true, run the entrypoint from the container image

Examples

Expand
titleExamples
Code Block
entrypoint = false

writable

Type: BOOL

Default: false true

If false, the container filesystem is read-only

Examples

Expand
titleExamples
Code Block
writable = truefalse

mounts

Type: ARRAY

Default: N/A

List of bind mounts in the format SOURCE:DESTINATION[:FLAGS]. Flags are optional and can include ro, private, etc.

Expand
titleNotes
  • Mount flags are separated with a plus symbol, for example: ro+private.
  • Optional flags from docker format or OCI (need reference)


Examples

Expand
titleExamples
  1. Literal fixed mount map
Code Block
mounts = ["/capstor/scratch/cscs/amadonna:/capstor/scratch/cscs/amadonna"]

2. Mapping path with env variable expansion

Code Block
mounts = ["/capstor/scratch/cscs/${USER}:/capstor/scratch/cscs/${USER}"]

3. Mounting the scratch filesystem using a host environment variable

Code Block
mounts = ["${SCRATCH}:/scratch"]

env

Type: TABLE

Default:

Inherited from

host and

image

Environment variables to set in the container. Null-string values will unset the variable.

Expand
titleNotes
  • By default, containers inherit environment variables from the container image and the host environment, with variables from the image taking precedence.
  • The env table can be used to further customize the container environment by setting, modifying, or unsetting variables.
  • Values of the table entries must be strings. If an entry has a null value, the variable corresponding to the entry key is unset in the container.

Examples

Expand
titleExamples

1. Basic env block

Code Block
[env]
MY_RUN = "production",
DEBUG = "false"
2. Use of environment variable expansion
Code Block
[env]
MY_NODE = "${VAR_FROM_HOST}",
PATH = "${PATH}:/custom/bin", 
DEBUG = "true"


annotations

Type: TABLE

Default: N/A

OCI-like annotations for the container.

 

Examples

Expand
titleExamples

1. Disabling the CXI hook

Code Block
[annotations]
com.hooks.cxi.enabled = "false"

2. Control of SSH hook parameters via annotation and variable expansion

Code Block
[annotations.com.hooks.ssh]
authorize_ssh_key = "/capstor/scratch/cscs/${USER}/tests/edf/authorized_keys"
enabled = "true"

3. Alternative example for usage of annotation with fixed path

Code Block
[annotations]
com.hooks.ssh.authorize_ssh_key = "/path/to/authorized_keys"
com.hooks.ssh.enabled = "true"

...