Page History

FirecREST is a RESTful API for managing High-Performance Computing resources developed at CSCS. Scientific platform developers can integrate Firecrest into web-enabled portals and applications, allowing them to securely access authenticated and authorized CSCS services such as job submissions and data transfer on HPC systems. Please refer to the FirecREST documentation of FirecREST for more details.

FirecREST features

...

For a full feature set, have a look at the latest FirecREST API specification deployed at CSCS.

FirecREST deployment at CSCS

...

The following is a list of systems per CSCS platform. You should be able to use the X-Machine-Name header to select the system in the respective platform when calling FirecREST

• HPC Platform
  • Daint
  • Eiger (TBD)
• ML Platform
  • Bristen
  • Clariden (TBD)
• CW Platform
  • Santis (TBD)

Getting started

One way to get started is by using pyFirecREST, which is a python package with a collection of wrappers for the different functionalities of FirecREST. This package simplifies the usage of FirecREST by making multiple requests in the background for more complex workflows as well as by refreshing the access token before it expires.

...

If you wish to transfer data bigger than the limit mentioned above, you can check the compress and extract endpoints or follow the following example on how to split large files and download/upload them using FirecREST.

The limit on the time and size of files that can be download/uploaded via FirecREST might change if needed. You can check the current values in the parameters endpoint:

>>> print(json.dumps(client.parameters(), indent = 2))
{
  (...)

  "storage": [
    {
      "description": "Type of object storage, like `swift`, `s3v2` or `s3v4`.",
      "name": "OBJECT_STORAGE",
      "unit": "",
      "value": "s3v4"
    },
    {
      "description": "Expiration time for temp URLs.",
      "name": "STORAGE_TEMPURL_EXP_TIME",
      "unit": "seconds",
      "value": "604800"  ## <-------- 7 days
    },
    {
      "description": "Maximum file size for temp URLs.",
      "name": "STORAGE_MAX_FILE_SIZE",
      "unit": "MB",
      "value": "5120"   ## <--------- 5 GB
    }
  (...)
}

Note on Job Submission to the Workload Manager through FirecREST

FirecREST provides an abstraction for job submission using in the backend the SLURM scheduler of the vCluster (in the case of CSCS).

When submitting a job via the different endpoints, you should pass the -l option to the /bin/bash  command on the batch file.

This option ensures that the job submitted uses the same environment as your login shell to access the system-wide profile (/etc/profile ) or to your profile (in files like ~/.bash_profile, ~/.bash_login, or ~/.profile).

Further information

• Full list of FirecREST calls
• Documentation of FirecREST
• Documentation of pyFirecREST
• Demo FirecREST environment in Git
• https://www.oauth.com
• Python Requests

...