Data Management

Overview

The data management services of EGI comprises two groups of services:

• Services that provide data management capabilities to enhance the raw storage available in the EGI infrastructure
• Specialized services that offer advanced organisation of data during ongoing research projects, as an integrated environment with data management and digital lab notebook

The EGI data management services offer both application programming interfaces (APIs) and command-line interfaces (CLIs) that are integrated with the advanced EGI services and platforms (such as development environments, machine learning, or cloud orchestrators), and can be accessed from most compute services.

Generic data management

This higher-level data management service is available to researchers:

• EGI DataHub is a high-performance data management solution that offers unified data access across multiple types of underlying storage, allowing users to share, collaborate and easily perform computations on the stored data.

Specialized data management

The following specialized data management service is also available:

• EGI Data Transfer is a low-level service to move data from one Grid or Object storage to another.

Use-cases for storing research data

Depending on the type of the employed compute services and the use-cases being addressed, users might need to choose different data service to store, access, and manage data.

The following sections offer detailed descriptions for each data management service.