Users: Online Storage

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Online Storage includes services that allow users to store, share and access data using the EGI infrastructure. Different categories of storage are available, depending on how data is stored, the technology used to access and consume data, and the foreseen usage.

Three major service offerings are available:

Block Storage is block-level storage that can be attached to virtual machines (VMs) as volumes, a simple solution for durable data that does not need to be shared beside a single VM.
Grid Storage is file storage for High Throughput Compute (HTC) and/or High Performance Compute (HPC) scenarios.
Object Storage is persistent, hierarchical blob storage for cloud native applications, archiving, or when data is shared between different VMs or multiple steps of processing workflows.

Comparison of storage types

The differences between Block, Grid, and Object Storage are summarized below:

Type	Sharing	Accounting	Usage
Block	From within VMs, only at the same site the VM is located	For the entire block	POSIX access, use as local disk
Grid	From any device connected to the internet	For the data stored	Grid protocols and HTTP/WebDAV
Object	From any device connected to the internet	For the data stored	HTTP requests to REST API

The following sections offer a more detailed description of each storage service.

Users: Data Management

Mon, 01 Jan 0001 00:00:00 +0000

Overview

The data management services of EGI comprises two groups of services:

Services that provide data management capabilities to enhance the raw storage available in the EGI infrastructure
Specialized services that offer advanced organisation of data during ongoing research projects, as an integrated environment with data management and digital lab notebook

The EGI data management services offer both application programming interfaces (APIs) and command-line interfaces (CLIs) that are integrated with the advanced EGI services and platforms (such as development environments, machine learning, or cloud orchestrators), and can be accessed from most compute services.

Generic data management

This higher-level data management service is available to researchers:

EGI DataHub is a high-performance data management solution that offers unified data access across multiple types of underlying storage, allowing users to share, collaborate and easily perform computations on the stored data.

Specialized data management

The following specialized data management service is also available:

EGI Data Transfer is a low-level service to move data from one Grid or Object storage to another.

Use-cases for storing research data

Depending on the type of the employed compute services and the use-cases being addressed, users might need to choose different data service to store, access, and manage data.

User	Data storage
Cloud user	Block and Object storage
HTC user	Grid storage
HPC user	High-performance parallel file systems or Object storage

The following sections offer detailed descriptions for each data management service.

Documentation – Data Services

Users: Online Storage

Overview

Comparison of storage types

Users: Data Management

Overview

Generic data management

Specialized data management

Use-cases for storing research data