EGI Data Transfer

Very large data transfers in the EGI infrastructure

What is it?

EGI Data Transfer allows scientists to move any type of data files asynchronously from one storage to another. The service includes dedicated interfaces to display statistics of on-going transfers and manage storage resource parameters.

EGI Data Transfer is ideal to move large amounts of files or very large files as the service has mechanisms to verify checksums and ensure automatic retry in case of failures.

The main features of EGI Data Transfer are:

  • Simplicity. Easy user interfaces for submitting transfers (command-line, Python bindings, WebFTS, Web Monitoring).
  • Reliability. Checksums are automatically calculated for each transfer and failed transfers are retried.
  • Flexibility. Multi-protocol support (WebDAV/HTTTPS, GridFTP, xrootd, SRM, S3, GCloud).
  • Intelligence. Parallel transfer optimization ensures users get the most from network without burning the storages. Transfers can be classified by priority and activity.

Components

FTS3 Server

The service is responsible of the asynchronous execution of the file transfer, checksumming and retries in case of errors

FTS3 REST

The RESTFul server which is contacted by clients via REST APIs, CLI and Python bindings

FTS3 Monitoring

A Web interface to monitor transfers activity and server parameters

WebFTS

A web interface that provides a file transfer and management solution in order to allow users to invoke reliable, managed data transfers on distributed infrastructures

Service Instances

EGI has signed OLAs with 2 Providers, CERN and UKRI-STFC, in order to access their FTS3 Service instances.

The following endpoints are available:

CERN

UKRI-STFC

N.B. if you access the endpoints via Browser the following CA certificates need to be installed:


Next topics:
Data Transfer Web Interface

The web interface of EGI Data Transfer

Data Tranfer Clients

Clients for accessing EGI Data Transfer

Data Transfer API

The programmatic interface of EGI Data Transfer

Monitoring Data Transfer

Monitoring file transfers of EGI Data Transfer

Last modified January 31, 2023 by Levente Farkas : Secrets Store documentation (#567)