Data Transfer

Documentation related to EGI Data Transfer

Overview

The EGI Data Transfer Service is based on the FTS3 service, developed at CERN. It allows you to move any type of data files asynchronously from one storage to another. The service includes dedicated interfaces to display statistics of on-going transfers and manage storage resources parameters.

The EGI Data Transfer is ideal to move large amounts of files or very large files as the service has mechanisms to verify checksums and ensure automatic retry in case of failures.

Features

Simplicity

Easy user interfaces for submitting transfers: CLI, Python Bindings, WebFTS and Web Monitoring.

Reliability

Checksums and retries are provided per transfer

Flexibility

Multi-protocol support (Webdav/https, GridFTP, xrootd, SRM, S3, GCloud).

Intelligence

Parallel transfers optimization to get the most from network without burning the storages. Priorities/Activities support for transfers classification.

Components

FTS3 Server

The service is responsible of the asynchronous execution of the file transfer, checksumming and retries in case of errors

FTS3 REST

The RESTFul server which is contacted by clients via REST APIs, CLI and Python bindings

FTS3 Monitoring

A Web interface to monitor transfers activity and server parameters

WebFTS

A web interface that provides a file transfer and management solution in order to allow users to invoke reliable, managed data transfers on distributed infrastructures

Service Instances

EGI has signed OLAs with 2 Providers, CERN and STFC, in order to access their FTS3 Service instances.

The following endpoints are available:

CERN

STFC

N.B. if you access the endpoints via Browser the following CA certificates need to be installed:


Clients

Documentation related to EGI Data Transfer Clients

WebFTS

Documentation related to EGI Data Transfer Web Interface

API

Documentation for EGI Data Transfer API

Monitoring

Documentation related to EGI Data Transfer Monitoring

Last modified January 11, 2021: Fix linter issues (#172) (2219631)