View Source

FirecREST is a RESTful API for managing High-Performance Computing resources developed at CSCS. Scientific platform developers can integrate Firecrest into web-enabled portals and applications, allowing them to securely access authenticated and authorized CSCS services such as job submissions and data transfer on HPC systems. Please refer to the FirecREST documentation for more details.

FirecREST features

Users can make HTTP requests to perform the following operations:

basic system utilities like ls, mkdir, mv, chmod, chown, among others
actions against the Slurm workload manager (submit, query, and cancel jobs of the user)
internal (between CSCS systems) and external (to/from CSCS systems) data transfers

For a full feature set, have a look at the latest FirecREST API specification deployed at CSCS.

FirecREST deployment at CSCS

For Alps systems, FirecREST has been split into Platforms:

FirecREST-HPC: for HPC Platform systems
FirecREST-ML: for Machine Learning Platform systems
FirecREST-CW: for Climate and Weather Platform systems

The public Gateway to FirecREST will depend on the platform of the system to access:

HPCP: https://api.cscs.ch/hpc/firecrest/v1
MLP: https://api.cscs.ch/ml/firecrest/v1
CWP: https://api.cscs.ch/cw/firecrest/v1

Users can access all file systems served by CSCS systems using FirecREST.

Alps systems exposed via FirecREST API

The following is a list of systems per CSCS platform. You should be able to use the X-Machine-Name header to select the system in the respective platform when calling FirecREST

HPC Platform
- Daint
- Eiger
ML Platform
- Bristen
- Clariden
CW Platform
- Santis

Getting started

One way to get started is by using pyFirecREST, which is a python package with a collection of wrappers for the different functionalities of FirecREST. This package simplifies the usage of FirecREST by making multiple requests in the background for more complex workflows as well as by refreshing the access token before it expires.

Here is an example of how one can try FirecREST through the python wrapper:

import firecrest as fc


client_id = <client_id>
client_secret = <client_secret>
token_uri = "https://auth.cscs.ch/auth/realms/firecrest-clients/protocol/openid-connect/token"

# Setup the client for the specific account
# For instance, for the Alps HPC Platform system Daint:
 
client = fc.Firecrest(
    firecrest_url="https://api.cscs.ch/hpc/firecrest/v1",
    authorization=fc.ClientCredentialsAuth(client_id, client_secret, token_uri)
)

print(client.all_systems())
# Output: (one dictionary per system)
# [{
#      'system': 'daint'
#      'status': 'available',
#      'description': 'System ready',
#  }]

print(client.list_files('daint', '/capstor/scratch/cscs/<username>'))
# Example output: (one dictionary per file)
# [{
#      'name': 'file.txt',
#      'user': 'username'
#      'last_modified': '2024-04-28T12:03:33',
#      'permissions': 'rw-r--r--',
#      'size': '2021',
#      'type': '-',
#      'group': 'project',
#      'link_target': '',
#  },
#  {
#       'name': 'test-dir',
#       'user': 'username'
#       'last_modified': '2024-04-20T11:22:41',
#       'permissions': 'rwxr-xr-x',
#       'size': '4096',
#       'type': 'd',
#       'group': 'project',
#       'link_target': '',
#  }]

The tutorial is written for a generic instance of FirecREST but if you have a valid user at CSCS you can test it directly with your resource allocation on the exposed systems.

CSCS Developer Portal

A client application that makes requests to FirecREST will not be using directly the credentials of the user for the authentication, but an access token instead. The access token is a signed JSON Web Token (JWT) which contains expiry information. Behind the API, all commands launched by the client will use the account of the user that registered the client, inheriting their access rights. You can manage your client application on the CSCS Developer Portal.

Every client will have a client ID (Consumer Key) and a secret (Consumer Secret) that will be used to get a short-lived access token with an HTTP request. Here is an example curl call to fetch the access token:

curl -s -X POST https://auth.cscs.ch/auth/realms/firecrest-clients/protocol/openid-connect/token \
     --data "grant_type=client_credentials" \
     --data "client_id=<your_client>" \
     --data "client_secret=<your_secret>"

Download large files through FirecREST

A staging area is used for external transfers and downloading/uploading a file from/to a CSCS filesystem.

Please follow the steps below to download a file:

Request FirecREST to move the file to the staging area: a download link will be provided
The file will remain in the staging area for 7 days or until the link gets invalidated with a request to the /storage/xfer-external/invalidate endpoint or through the pyfirecrest method
The staging area is common for all users, therefore users should invalidate the link as soon as the download has been completed

You can see the full process in this tutorial.

We may be forced to delete older files sooner than 7 days whenever large files are moved to the staging area and the link is not invalidated after the download, to avoid issues for other users: we will contact the user in this case.

When uploading files through the staging area, you don't need to invalidate the link. FirecREST will do it automatically as soon as it transfers the file to the filesystem of CSCS.

There is also a constraint on the size of a single file to transfer externally to our systems via FirecREST: 5 GB.

If you wish to transfer data bigger than the limit mentioned above, you can check the compress and extract endpoints

The limit on the time and size of files that can be download/uploaded via FirecREST might change if needed. You can check the current values in the parameters endpoint:

>>> print(json.dumps(client.parameters(), indent = 2))
{
  (...)

  "storage": [
    {
      "description": "Type of object storage, like `swift`, `s3v2` or `s3v4`.",
      "name": "OBJECT_STORAGE",
      "unit": "",
      "value": "s3v4"
    },
    {
      "description": "Expiration time for temp URLs.",
      "name": "STORAGE_TEMPURL_EXP_TIME",
      "unit": "seconds",
      "value": "604800"  ## <-------- 7 days
    },
    {
      "description": "Maximum file size for temp URLs.",
      "name": "STORAGE_MAX_FILE_SIZE",
      "unit": "MB",
      "value": "5120"   ## <--------- 5 GB
    }
  (...)
}

Note on Job Submission to the Workload Manager through FirecREST

FirecREST provides an abstraction for job submission using in the backend the SLURM scheduler of the vCluster (in the case of CSCS).

When submitting a job via the different endpoints, you should pass the -l option to the /bin/bash command on the batch file.

This option ensures that the job submitted uses the same environment as your login shell to access the system-wide profile (/etc/profile ) or to your profile (in files like ~/.bash_profile, ~/.bash_login, or ~/.profile).