CSCS offers a public cloud object storage service, based on OpenStack Swift. The service can be accessed from REST APIs compatible with the Openstack Swift protocol.

Access

There are several ways of accessing the object storage service, from user-friendly graphical tools to software libraries which can be integrated with custom applications.

Web frontend

Users with the SwiftOperator role can use the OpenStack dashboard to access object storage, by selecting the Object Store tab on the top menu bar of the Horizon GUI. Users who don't have the SwiftOperator role cannot access the object store from this web interface and must use other means such as CLI or Cyberduck or the REST API.

Command line interface

A generic guide on how to connect to our OpenStack system Castor using the command line is available at this URL. You can find detailed documentation on how to use the Swift command line client at the official Openstack documentation. Please note that there are two ways of accessing Swift, either the old Swift client command line, which has the most features, but it's no longer actively developed: Object Storage service (swift) command-line client. Alternatively, you can use the newer OpenStack unified CLI, which does not cover all Swift features yet, but it's actively developed: OpenStackClient

Cyberduck

A guide for connecting to our object storage service using the graphical file browsing client Cyberduck can be found here.

Swift REST API

The object storage is accessible from a REST API, defined this official documentation page. You can access the REST API with curl, or alternatively you can use a software library such as the Python swiftclient module, which can be downloaded from GitHub or from PyPi.

Examples

You can find below a list of how-tos for typical use cases, which however don't cover all the possible operations of Openstack Swift. For a complete list of functionalities please refer to the official documentation of CLI and REST API.

Known issues and workarounds

You can find nelow a list of known issues with the relative workaround.

Access Control Lists

Users could be granted two different roles within a project:

  • SwiftOperator: can list, create and modify all containers and objects within a project, can configure read or write Access Control Lists (ACL)
  • member: can access objects in specific containers only if a read or write ACL was granted to them

In Swift, ACLs can be assigned to containers, not to objects. Our authentication system involves Keystone V3 and domains, which means that user names and project names might not be unique. Because of this, when creating an ACL rule users need to specify project IDs and user IDs instead of project names and user names. To facilitiate this, we are regularly populating an object called user_ids inside of a container project_info in each project. The object user_ids contains the IDs of members of the current project.

This is an example of how and operator can add a read ACL to a container:

swift post mycontainer --read-acl {PROJECT1_ID}:{USER1_ID},{PROJECT2_ID}:{USER2_ID

The option --write-acl is used to configure write permissions. Please note that PROJECT_ID and USER_ID are long alphanumerical strings, so the command in reality will look like the following:

swift post testcontainer --read-acl 62f7feebbfb94f3bbb501b0a060nfn2r:3bb7feebbfb94f3bb5mdob0a060b30eb

It is also possible to use the wildcard *, as described in the official ACL documentation.

Operators can use the swift stat command to list existing ACLs on a container.

If a user wants to use the CLI to access an object in a project he's member of he can run a command like the following, after having authenticated: swift list {container_name}.

Only operators can list all the containers in a project. Normal users cannot list which containers they have access to. However, once they are told by operators which containers they have access to, they can list their contents. If a user wants to access an object in a project he's not member of, as long as he was granted ACL access, he should instead use the following command:

swift --os-storage-url https://object.cscs.ch/v1/AUTH_{CHOSEN_PROJECT_ID} list {CONTAINER_NAME

You can find more documentation about Swift ACLs on the official documentation.

Roles and project memberships have to be requested contacting CSCS staff.

Data protection

Container versioning

Our object storage system automatically saves up to 3 versions of objects whenever they are modified or deleted. Object versions are automatically stored into the {your_container_name}_versions container and they are kept for 90 days. The user can recover an object by copying (CLI: swift copy) the desired version from the {your_container_name}_versions container into {your_container_name} or to a different one. The {your_container_name}_versions containers are automatically created by a daily cron job.

Backup

In addition to versioning, all data written in the object store is backed up to tape. This allows the recovery of the entire object store in case of major hardware or file system failures. The backup is taken once a day, and we configured a retention policy of 3 months. Whenever an object changes, a new copy is created in the backend, with a maximum of three copies stored on tape. In case of major outage, the entire storage will be restored within a few days.

Swift S3 API

In addition to the standard Swift API the object storage service exposes also a S3 API, still using the same service endpoint. The compatibility matrix between the Swift S3 API and the Amazon S3 API is described in this table.

These two APIs allow access to the same object storage service, so no matter which one you decide to use, you will always access to the same data. The S3 ACLs are disabled in order to allow both APIs to operate seamlessly. In order to set ACLs on your containers/buckets and objects, please use the Swift API.

To use the Swift S3 API, first you have to create a set of EC2 credentials. For this you need to obtain a standard keystone token and then use the OpensStacl CLI as follow:

openstack ec2 credentials create

You will obtain in output a pair of access and secret keys which can be used with any S3 client. Below a short list of the most common ones: