Documentation – EGI DataHub

Users: DataHub Use-Cases

Mon, 01 Jan 0001 00:00:00 +0000

An overview of the use cases and possible deployment scenarios of the EGI DataHub.

Build Data Spaces for Communities

Users can access with their EGI Check-in credentials data available at one or more providers which are abstracting over heterogeneous backends.

Different data access interfaces are available (Web, REST, POSIX, CDMI).

Integrate DataHub with EGI Notebooks

EGI Notebooks has been integrated with DataHub in order to easily access and store files from the JupyterLab interface.

On the central EGI Notebooks instance, a predefined space open to all users and VO specific spaces are available.

Make your data FAIR

Users can assign metadata to files, create shares and mint DOIs.

The EGI DataHub offers an OAI-PMH interface which enables standard discoverability and access to datasets.

Smart caching

Site A hosts data and computing resources
Site B hosts only data
Site X uses data from A and B without pre-staging
Pre-staging can also be done using APIs
Data is accessed locally via POSIX with FUSE

Users: DataHub Clients

Mon, 01 Jan 0001 00:00:00 +0000

EGI DataHub spaces can be accessed via web interface, the oneclient component or the API.

The official documentation for oneclient is hosted on the Onedata homepage, and a specific tutorial on how to install and use it from a Virtual Machine is also available.

Using the web interface

Using EGI Check-in allows you to connect with your institute’s credentials.

On this page it’s possible to have an overview of all the spaces and their supporting providers.

On this capture, the information about the spaces supported by a specific provider is displayed.

The data space can be managed (i.e. uploading/downloading/managing files and metadata, managing space access) using the web browser.

Generating tokens for using Oneclient or APIs

Important

In order to be able to access your spaces using oneclient or the API, it is required to generate an access token.

Tokens have to be generated from the EGI DataHub (Onezone) interface.

The access tokens can be created and managed using the EGI DataHub web interface.

Environment variables

The sections below assume you have defined the following variables in your environment:

ONECLIENT_ACCESS_TOKEN: access token allowing to access all the spaces
ONECLIENT_PROVIDER_HOST: name or IP of the Oneprovider the client should connect to.

Installing and testing Oneclient in a docker container

Important

In order to be able to use FUSE, the container should run in privileged mode.

A quick and simple solution for testing is to install the client on demand in a container for a supported Operating System flavor (mainly various CentOS and Ubuntu releases).

$ docker run -it --privileged centos:7 /bin/bash
root@81dbd7e84438 /]# curl -sS https://get.onedata.org/oneclient.sh | bash
# (...)
Complete!
Installation has been completed successfully.
Run 'oneclient --help' for usage info.
root@81dbd7e84438 /]# export ONECLIENT_ACCESS_TOKEN=<ACCESS_TOKEN_FROM_ONEZONE>
root@81dbd7e84438 /]# export ONECLIENT_PROVIDER_HOST=plg-cyfronet-01.datahub.egi.eu
root@81dbd7e84438 /]# mkdir /tmp/space
root@81dbd7e84438 /]# oneclient /tmp/space
root@81dbd7e84438 /]# ls /tmp/space

Here the data is mounted in /tmp/space, creating a file into it will push it to the Oneprovider and it will be accessible in the web interface and from other providers supporting the space.

For a real production usage it's preferable to use the Oneclient container as a source for a volume mounted into another container.

Testing Oneclient in a Oneclient docker container with NFS or samba

Docker containers for the Oneclient are available, the existing versions can be seen on the Oneclient docker hub.

It’s possible to use the most recent version by specifying the latest tag. We also recommend using the same version as shown on the Onezone and Oneprovider pages.

$ export ONECLIENT_ACCESS_TOKEN=<ACCESS_TOKEN_FROM_ONEZONE>
$ export ONECLIENT_PROVIDER_HOST=plg-cyfronet-01.datahub.egi.eu
$ docker run -it --privileged \
 -e ONECLIENT_ACCESS_TOKEN=$ONECLIENT_ACCESS_TOKEN \
 -e ONECLIENT_PROVIDER_HOST=$ONECLIENT_PROVIDER_HOST \
 onedata/oneclient:21.02.2
Connecting to provider 'plg-cyfronet-01.datahub.egi.eu:443' using session ID: '4138963898952098752'...
Getting configuration...
Oneclient has been successfully mounted in '/mnt/oneclient'

Now the client will run in the background and the data will be available through samba/CIFS or NFS protocols:

# Identifying the IP of the container
$ docker inspect --format "{{ .NetworkSettings.IPAddress }}" $(docker ps -ql)
172.17.0.2

So the data can be accessed at

smb://172.17.0.2/onedata
nfs://172.17.0.2/onedata

Testing Oneclient in a Oneclient docker container with local file access

Another solution is to mount a local directory as a volume in the container, allowing to access both the working directory as well as the Onedata spaces, thus allowing to easily exchange files between a local directory and a Onedata space.

In order to do this we will open a bash shell in the container then we will mount manually the Onedata spaces.

$ export ONECLIENT_ACCESS_TOKEN=<ACCESS_TOKEN_FROM_ONEZONE>
$ export ONECLIENT_PROVIDER_HOST=plg-cyfronet-01.datahub.egi.eu
$ docker run -it --privileged \
 -e ONECLIENT_ACCESS_TOKEN=$ONECLIENT_ACCESS_TOKEN \
 -e ONECLIENT_PROVIDER_HOST=$ONECLIENT_PROVIDER_HOST \
 -v $PWD:/mnt/src --entrypoint bash onedata/oneclient:21.02.2
root@aca612a84fb4:/tmp# oneclient /mnt/oneclient
Connecting to provider 'plg-cyfronet-01.datahub.egi.eu:443' using session ID: '1641165171427694510'...
Getting configuration...
Oneclient has been successfully mounted in '/mnt/oneclient'.
root@aca612a84fb4:/tmp# ls /mnt/oneclient
(...)
root@aca612a84fb4:/tmp# ls /mnt/src
(...)

Now it's possible to use the following mount points:

/mnt/oneclient: the Onedata spaces
/mnt/src: the local directory (any absolute path could have been used instead of $PWD that points to the working directory)

Testing Oneclient in a Virtual Machine

The following variables have to be exported:

ONECLIENT_ACCESS_TOKEN: access token allowing to access all the spaces.
ONECLIENT_PROVIDER_HOST: name or IP of the Oneprovider the client should connect to.

$ curl -sS https://get.onedata.org/oneclient.sh | bash
$ export ONECLIENT_ACCESS_TOKEN=<ACCESS_TOKEN_FROM_ONEZONE>
$ export ONECLIENT_PROVIDER_HOST=plg-cyfronet-01.datahub.egi.eu
$ mkdir /tmp/space
$ oneclient /tmp/space

Testing Oneclient in a Vagrant box

It's possible to quickly test Oneclient using Vagrant.

$ vagrant init ubuntu/xenial64
$ vagrant up
$ vagrant ssh
$ curl -sS https://get.onedata.org/oneclient.sh | bash
$ export ONECLIENT_ACCESS_TOKEN=<ACCESS_TOKEN_FROM_ONEZONE>
$ export ONECLIENT_PROVIDER_HOST=plg-cyfronet-01.datahub.egi.eu
$ mkdir /tmp/space
$ oneclient /tmp/space

Users: DataHub File Management

Mon, 01 Jan 0001 00:00:00 +0000

Web interface

In the EGI DataHub, all files are organized in spaces. The Web User interface allows for uploading new files as well as opening existing files.

In order to upload a file, open the directory in which the file should be placed and drag the file into the browser window:

Opening or downloading a file requires double clicking on the file in the file window.

Note

Make sure that the pop-ups for this browser window are not blocked, and unblock them if necessary.

Direct access via POSIX

Files can also be accessed directly via the POSIX interface, using Oneclient tool. Details on how to use are described in the official Oneclient documentation.

CDMI (Cloud Data Management Interface)

For more advanced use cases, files can be managed using the CDMI protocol, as described in details in the Onedata CDMI documentation.

File Permissions

You can control access to your data with a combination of:

classical (POSIX) file permissions
Access Control Lists (ACL)

POSIX Permissions

DataHub allows you to control access to your data in a POSIX-like fashion for users, group and others in terms of read, write and executable permissions.

An important nuance regarding file permissions is that all space members are treated as a virtual group which is the group owner of all files in the space. That means that whenever a file is accessed by a space member who is not the owner of the file, the group permissions are taken into consideration. Permissions for “others” are considered when accessing shares.

Consider the following example of a file’s POSIX permissions:

rwx r-- ---
 | | |
 | | guests
 | |
 | space members
 |
 owner user

In the above case, the creator of the file, its owner user, has a full access to the file. All space members have read access to the file. Other users, guests, who try to access the file through a share will fail to do so as all permissions are declined for “others”.

In order to edit permissions:

Click on Data on the left menu bar
Select the Space you want to access and the Data option
Select a file or a directory and right clicking on it
Click on the Permissions option
Select POSIX type of permissions radio button at the top
Enter privileges in octal form (e.g. 770)
Click OK in order to save changes

Access Control Lists

You can also setup permissions using more advanced Access Control Lists option to control permissions for individual users and groups.

In order to edit ACLs:

Navigate to Data tab
Select a file or a directory by right clicking on it
Click on the Permissions option
Click on the ACL radio button
Edit permissions by clicking on the appropriate checkbox
Click OK in order to save changes

Note, that access lists take precedence over POSIX permissions. If access list is set, POSIX are set to octal value of 000.

The order in which permissions take precedence is indicated with an arrow.

To limit the risk of making data inaccessible to yourself, it is advisable to first add the required ACL if new ones are needed, and if necessary remove the ones not needed. While using ACLs there should be at least one active, or no user might be able to access the data. To regain the access the space owner would then need to modify the permissions.

It is possible to share available DataHub data with other users or externally by generating a unique URL. This can be done, on the web interface, by selecting a directory or file, and right-clicking on it or clicking the three dots on the right like in the following screenshot and selecting “share”:

This will open a window allowing you to give a name to the share that you are about to create. This is shown in the following screenshot:

By clicking on the “Create” button, the share is created and you are taken to the last window where the URL of the share can be copied for further use. This is shown in the following screenshot:

To manage existing shares, in the space that is in use, it is sufficient to click the “Share” section, then a list of the existing ones is shown. By clicking on the three dots on the right of the share, you can rename, remove or copy the URL of the share. This is shown in the following screenshot:

By clicking on the share itself, it is possible to edit additional attributes of the share like the description and the possibility to publish it as Open Data trough one of the supported handle service.

An API is also available for the creation and administration of the shares.

Data transfer management

After uploading some data to DataHub, you can manage the replication and transfer to other DataHub providers supporting the same space. To do so, after uploading some files, select them, right click on the selection, and select “Data distribution”. This will open the following window:

This will show the providers available for the space being used, and the current operation being performed if any. By clicking on the three dots, next to the provider, the following operation can be performed:

Migrate the data if the provider selected contain the data. This will copy the data to the other provider, if not present, and remove them from the selected one.
Replicate the data, if the data is not present in the provider selected
Evict the data, if the data is present in the current provider, and al least in another one

Data transfer management API

The DataHub REST API structure gives an overview of the REST calls available. In particular the section on Transfer focus on the transfer operation which include the operation previously described using the web interface and include option to schedule transfers, a feature which is not available trough the web interface. Furthermore the status of the transfer can also be checked from the section Get transfer status of the same page.

The /transfers/ operations provide basic transfer management functionality based on the ID of transfer returned by /transfers [POST] operation.

With the API it is possible to get information about a specific transfer by simply querying the following resource:

$ curl -X GET -H "X-Auth-Token: $ACCESS_TOKEN" \
 https://$ONEPROVIDER_HOST/api/v3/oneprovider/transfers/<TRANSFER_ID>

or request all transfers for given space:

$ curl -X GET -H "X-Auth-Token: $ACCESS_TOKEN" \
 https://$ONEPROVIDER_HOST/api/v3/oneprovider/spaces/$SPACE_ID/transfers

Each transfer can be cancelled using the HTTP DELETE method:

$ curl -X DELETE -H "X-Auth-Token: $ACCESS_TOKEN" \
 https://$ONEPROVIDER_HOST/api/v3/oneprovider/transfers/<TRANSFER_ID>

Users: DataHub API

Mon, 01 Jan 0001 00:00:00 +0000

Most operations in EGI DataHub can be performed using one of the OneData Application Programming Interfaces (APIs).

Important

In order to be able to access the Onedata APIs, an access token is required. See below for instructions on how to generate one.

Getting an API access token

Tokens have to be generated from the EGI DataHub (Onezone) interface as documented in Generating tokens for using Oneclient or APIs or using a command-line call as documented hereafter.

Bear in mind that a single API token can be used with both Onezone, Oneprovider and other Onedata APIs.

It’s possible to retrieve the CLIENT_ID and REFRESH_TOKEN using the EGI Check-in Token Portal. See Check-in documentation for more information.

$ CLIENT_ID=<CLIENT_ID>
$ REFRESH_TOKEN=<REFRESH_TOKEN>
# Retrieving an OIDC token from Check-in
$ curl -X POST \
 -d "client_id=$CLIENT_ID&grant_type=refresh_token&refresh_token=$REFRESH_TOKEN&scope=openid%20email%20profile%20eduperson_entitlement" \
 'https://aai.egi.eu/auth/realms/egi/protocol/openid-connect/token' | python -m json.tool;
# Token is in the access_token field of the response

The following variables should be set:

OIDC_TOKEN: OpenID Connect Access token.
ONEZONE_HOST: name or IP of the Onezone host (to use Onezone API).

$ ONEZONE_HOST=https://datahub.egi.eu
$ OIDC_TOKEN=<OIDC_ACCESS_TOKEN>
$ curl -H "X-Auth-Token: egi:$OIDC_TOKEN" -X POST \
 -H 'Content-type: application/json' \
 "$ONEZONE_HOST/api/v3/onezone/user/tokens/named" \
 -d '{
 "name": "REST and CDMI access token",
 "type": {
 "accessToken": {}
 },
 "caveats": [
 {
 "type": "interface",
 "interface": "rest"
 }
 ]
 }'

Data access via CDMI and REST API

Below are example commands to learn how to access DataHub files and folders via CDMI and REST API using the command-line interface.

For more information please check the Onedata CDMI documentation and the Onedata Oneprovider REST API

Common configuration

Follow instructions above to get an API access token, and configure environment variables:

$ export DATAHUB_TOKEN=<DATAHUB_ACCESS_TOKEN>
$ export ONEPROVIDER_HOST=plg-cyfronet-01.datahub.egi.eu

Having jq installed is useful for better formatting of the JSON output.

CDMI

Configure a header to be passed in some operations.

$ export CDMI_VSN_HEADER='X-CDMI-Specification-Version: 1.1.1'

See examples on how to list a folder, and file download/upload using CDMI:

# List files in a folder
$ curl -H "X-Auth-Token: $DATAHUB_TOKEN" \
 -H "$CDMI_VSN_HEADER" \
 "https://$ONEPROVIDER_HOST/cdmi/PLAYGROUND/?children" | jq .

# Download "helloworld.txt" from DataHub to "downloadtest.txt" on your computer
$ curl -H "X-Auth-Token: $DATAHUB_TOKEN" \
 "https://$ONEPROVIDER_HOST/cdmi/PLAYGROUND/helloworld.txt" \
 -o downloadtest.txt

# Upload "helloworld.txt" from your computer to "uploadtest.txt" on DataHub
$ curl -H "X-Auth-Token: $DATAHUB_TOKEN" \
 -H "$CDMI_VSN_HEADER" \
 -X PUT "https://$ONEPROVIDER_HOST/cdmi/PLAYGROUND/uploadtest.txt" \
 -T helloworld.txt

REST API

See examples on how to list a folder, and file download/upload using REST API:

# Get base folder ID
$ curl -H "X-Auth-Token: $DATAHUB_TOKEN" \
 -X POST "https://$ONEPROVIDER_HOST/api/v3/oneprovider/lookup-file-id/PLAYGROUND"

# Add the folder ID to an environment variable
$ export DIR_ID=<ID_FROM_PREVIOUS_COMMAND>

# List files inside the folder with DIR_ID
$ curl -H "X-Auth-Token: $DATAHUB_TOKEN" \
 -X GET "https://$ONEPROVIDER_HOST/api/v3/oneprovider/data/$DIR_ID/children" \
 | jq .

# Add the ID of the file that you want to download
$ export FILE_ID=<ID_FROM_PREVIOUS_COMMAND>

# Download file with FILE_ID from DataHub to "helloworld.txt" on your computer
$ curl -H "X-Auth-Token: $DATAHUB_TOKEN" \
 -X GET "https://$ONEPROVIDER_HOST/api/v3/oneprovider/data/$FILE_ID/content" \
 -o helloworld.txt

# Upload "helloworld.txt" on your local computer to "uploadtest.txt" on DataHub
$ curl -H "X-Auth-Token: $DATAHUB_TOKEN" \
 -X POST \
 "https://$ONEPROVIDER_HOST/api/v3/oneprovider/data/$DIR_ID/children?name=uploadtest.txt" \
 -H "Content-Type: application/octet-stream" -d "@helloworld.txt"

Data access from Python

If your application is written in Python please check the documentation for the OnedataFS Python library

Testing the API with the REST client

A docker container with clients acting as wrappers around the API calls is available: onedata/rest-cli. It's very convenient for discovering and testing the Onezone and Oneprovider API.

$ docker run -it onedata/rest-cli
# Exporting env for Onezone API
$ export ONEZONE_HOST=https://datahub.egi.eu
$ export ONEZONE_API_KEY=<ACCESS_TOKEN>
# Checking current user
$ onezone-rest-cli getCurrentUSer | jq '.'
# Listing all accessible spaces
$ onezone-rest-cli listEffectiveUserSpaces | jq '.'

$ docker run -it onedata/rest-cli
# Exporting env for Oneprovider API
$ export ONEPROVIDER_HOST=https://plg-cyfronet-01.datahub.egi.eu
$ export ONEPROVIDER_API_KEY=<ACCESS_TOKEN>
# Listing all spaces supported by the Oneprovider
$ oneprovider-rest-cli getAllSpaces | jq '.'
# Listing content of a space
$ oneprovider-rest-cli listFiles path='EGI Foundation/'
$ oneprovider-rest-cli listFiles path='EGI Foundation/CS3_dataset'

Printing the raw REST calls of a wrapped command

Raw REST calls (used with curl) can be printed using the --dry-run switch.

$ docker run -it onedata/rest-cli
$ export ONEZONE_HOST=https://datahub.egi.eu
$ export ONEZONE_API_KEY=<ACCESS_TOKEN>
# Listing all accessible spaces
$ onezone-rest-cli listEffectiveUserSpaces | jq '.'
# Printing the curl command without running it
$ onezone-rest-cli listEffectiveUserSpaces --dry-run

Working with PID / Handle

It’s possible to mint a Permanent Identifier (PID) for a space or a subdirectory of a space using a handle service (like Handle.net) that is registered in the Onezone (EGI DataHub).

Once done, accessing the PID using its URL will redirect to the Onedata share allowing to retrieve the files.

Prerequisites: access to a Handle service registered in the Onezone. See the Handle Service API documentation for documentation on registering a new Handle service or ask a Onezone administrator to authorize you to use an existing Handle service already registered in the Onezone.

The following variables should be set:

API_ACCESS_TOKEN: Onedata API access token
ONEZONE_HOST: name or IP of the Onezone host (to use Onezone API).
ONEPROVIDER_HOST: name or IP of the Oneprovider host (to use Oneprovider API)

# Getting the IDs of the available Handle Services
$ curl -sS --tlsv1.2 -H "X-Auth-Token: $API_ACCESS_TOKEN" \
 "$ONEZONE_HOST/api/v3/onezone/user/handle_services"
HANDLE_SERVICE=<HANDLE_SERVICE_ID>

# Getting details about a specific Handle service
$ curl -sS --tlsv1.2 -H "X-Auth-Token: $API_ACCESS_TOKEN" \
 "$ONEZONE_HOST/api/v3/onezone/user/handle_services/$HANDLE_SERVICE"

# Listing all spaces
$ curl -sS --tlsv1.2 -H "X-Auth-Token: $API_ACCESS_TOKEN" \
 "$ONEZONE_HOST/api/v3/onezone/user/effective_spaces/" | jq '.'

# Displaying details of a space
$ curl -sS --tlsv1.2 -H "X-Auth-Token: $API_ACCESS_TOKEN" \
 "$ONEZONE_HOST/api/v3/onezone/spaces/$SPACE_ID" | jq '.'

# Listing content of a space
$ curl -sS --tlsv1.2 -H "X-Auth-Token: $API_ACCESS_TOKEN" \
 "$ONEPROVIDER_HOST/api/v3/oneprovider/files/EGI%20Foundation/" | jq '.'

# Creating a share of a subdirectory of a space
$ DIR_ID_TO_SHARE=<DIR_ID>
$ curl -sS --tlsv1.2 -H "X-Auth-Token: $API_ACCESS_TOKEN" \
 -X POST -H 'Content-Type: application/json' \
 -d '{"name": "input"}'
 "$ONEPROVIDER_HOST/api/v3/oneprovider/shares-id/$DIR_ID_TO_SHARE" | jq '.'

# Displaying the share
$ SHARE_ID=<SHARED_ID>
$ curl -sS --tlsv1.2 -H "X-Auth-Token: $API_ACCESS_TOKEN" \
 "$ONEZONE_HOST/api/v3/onezone/shares/$SHARE_ID" | jq '.'

# Registering a handle
# Proper Dublin Core metadata is required
# It can be created using https://nsteffel.github.io/dublin_core_generator/generator_nq.html
$ cat metadata.xml
# Escape double quotes and drop line return
$ METADATA=$(cat metadata.xml | sed 's/"/\\"/g' | tr '\n' ' ')
# On handle creation the created handles is provided in the Location header
$ curl -D - --tlsv1.2 -H "X-Auth-Token: $API_ACCESS_TOKEN" \
 -H "Content-type: application/json" -X POST \
 -d '{"handleServiceId": "'"$HANDLE_SERVICE_ID"'", "resourceType": "Share", "resourceId": "'"$SHARE_ID"'", "metadata": "'"$METADATA"'"}' \
 "$ONEZONE_HOST/api/v3/onezone/user/handles"

# Listing handles
$ curl --tlsv1.2 -H "X-Auth-Token: $API_ACCESS_TOKEN" \
 "$ONEZONE_HOST/api/v3/onezone/user/handles"

# Displaying a handle
$ HANDLE_ID=<HANDLE_ID>
$ curl --tlsv1.2 -H "X-Auth-Token: $API_ACCESS_TOKEN" \
 "$ONEZONE_HOST/api/v3/onezone/user/handles/$HANDLE_ID"

Following is an example of how to subscribe to DataHub to receive notification on file events which is described in details in the official documentation Subscribe to file events:

$ curl -N -H "X-Auth-Token: $TOKEN" \
 -X POST "https://$ONEPROVIDER_HOST/api/v3/oneprovider/changes/metadata/$SPACE_ID" \
 -H "Content-Type: application/json" -d "@./changes_req.json"

This requires the permission set as following:

For groups or single users. For single users, one way to add one is to select Effective members" -> in the user list search for the required user and “Make an owner”. In this case the user will have admins privileges in addition to the one required. As this might not be the desired configuration it will be enough to remove all the unwanted permissions, e.g.: make it the same as the VO to which the user belongs to, and leave only, as extra, the permission shown in the screenshot.

Users: DataHub Links

Mon, 01 Jan 0001 00:00:00 +0000

Documentation – EGI DataHub

Users: DataHub Use-Cases

Build Data Spaces for Communities

Integrate DataHub with EGI Notebooks

Make your data FAIR

Smart caching

Users: DataHub Clients

Using the web interface

Generating tokens for using Oneclient or APIs

Important

Environment variables

Installing and testing Oneclient in a docker container

Important

Testing Oneclient in a Oneclient docker container with NFS or samba

Testing Oneclient in a Oneclient docker container with local file access

Testing Oneclient in a Virtual Machine

Testing Oneclient in a Vagrant box

Users: DataHub File Management

Web interface

Note

Direct access via POSIX

CDMI (Cloud Data Management Interface)

File Permissions

POSIX Permissions

Access Control Lists

File sharing

Data transfer management

Data transfer management API

Users: DataHub API

Important

Getting an API access token

Data access via CDMI and REST API

Common configuration

CDMI

REST API

Data access from Python

Testing the API with the REST client

Printing the raw REST calls of a wrapped command

Working with PID / Handle

Subscribe to file events

Users: DataHub Links