Documentation – Compute Services

Users: Federated Cloud Compute

Mon, 01 Jan 0001 00:00:00 +0000

The EGI Federated Cloud Compute (FedCloud) service offers a multi-cloud IaaS federation that brings together research clouds as a scalable computing platform for data and/or compute driven applications and services for research and science.

This documentation focuses on using the service. Those resource providers willing to integrate into the service, please see the EGI Federated Cloud Integration documentation.

Cloud Compute gives you the ability to deploy and scale virtual machines on-demand. It offers computational resources in a secure and isolated environment controlled via APIs without the overhead of managing physical servers.

Cloud Compute service is provided through a federation of IaaS cloud sites that offer:

Single Sign-On via EGI Check-in, users can log into every provider with their institutional credentials and use modern industry standards like OpenID Connect.
Global VM image catalogue at Artefact Registry with pre-configured Virtual Machine images that are automatically replicated to every provider based on your community needs.
Resource discovery features to easily understand which providers are supporting your community and what are their capabilities.
Global accounting that aggregates and allows visualisation of usage information across the whole federation.
Monitoring of Availability and Reliability of the providers to ensure SLAs are met.

The flexibility of the Infrastructure as a Service can benefit various use cases and usage models. Besides serving compute/data intensive analysis workflows, Web services and interactive applications can be also integrated with and hosted on this infrastructure. Contextualisation and other deployment features can help application operators fine tune services in the cloud, meeting software (OS and software packages), hardware (number of cores, amount of RAM, etc.) and other types of needs (e.g. orchestration, scalability).

Since the opening of the EGI Federated Cloud, the following usage models have emerged:

Service hosting: the EGI Federated Cloud can be used to host any IT service as web servers, databases, etc. Cloud features, as elasticity, can help users to provide better performance and reliable services.
- Example: NBIS Web Services, Peachnote analysis platform.
Compute and data intensive applications: for those applications needing considerable amount of resources in terms of computation and/or memory and/or intensive I/O. Ad-hoc computing environments can be created in the EGI cloud providers to satisfy extremely intensive HW resource requirements.
- Example: VERCE platform, The Genetics of Salmonella Infections, The Chipster Platform.
Datasets repository: the EGI Cloud can be used to store and manage large datasets exploiting the large amount of disk storage available in the Federation.
Disposable and testing environments: environments for training or testing new developments.
- Example: Training infrastructure

Eager to test this service? Have a look at how to create your first Virtual Machine in EGI.

Users: Cloud Container Compute

Mon, 01 Jan 0001 00:00:00 +0000

The EGI Cloud Container Compute service allows you to run container-based applications on the providers of the EGI Federated Cloud. There are two main ways of executing containers:

Using docker (or a similar container runtime) on a VM, so you can just interact directly with the container runtime to run your applications. This fits simpler applications that can easily fit on one node and are composed by a small number of containers.
Using a container orchestration platform, e.g. kubernetes on a set of VMs to manage the applications in an automated way for you. This is usually suited for more complex applications that spawn several nodes and are composed of several containers that need to cooperate to deliver the expected functionality.

Follow the guides below to learn more about them.

The EGI Cloud Container Compute service has been presented in the EGI Webinars.

January 2024: Introduction to the new generation EGI container execution platform. See slides and recording.
April 2021: Managing Singularity, Docker and udocker containers, Kubernetes clusters in the EGI Cloud. See slides and recording.

Users: High Throughput Compute

Mon, 01 Jan 0001 00:00:00 +0000

What is it?

High Throughput Compute (HTC) is a computing paradigm that focuses on the efficient execution of a large number of loosely-coupled tasks (e.g. data analysis jobs). HTC systems execute independent tasks that can be individually scheduled on many different computing resources, across multiple administrative boundaries. Users submit these tasks to the infrastructure as jobs. After a job have been scheduled and executed, the output can be collected from the service(s) that executed the job.

Target users

The target customers for EGI High Throughput Compute are research communities that need to share, store, process, and produce large sets of data. Typically, their research collaborations involve organizations across Europe and the World.Some may already have local resources (e.g. universities, research institutions) that can only be accessed by local users in accordance to the respective organisation’s access policies.

In case of local compute resources researchers can request access to the local compute cluster from their IT department. However, when researchers join collaborations that need to share their research activities, data collections, and repositories, they need a homogeneous and coordinated operation of the compute resources, which are not uniformly accessible. In addition, nowadays many research collaborations generate large amounts of data, and managing such data volumes is time consuming and error-prone.

The EGI High Throughput Compute service provides access to compute resources, and offers a set of high-level tools that allow managing large amounts of data in a collaborative way (e.g authorization and access control tools can be regulated by the research collaboration in a central manner, data can be uniformly distributed in the EGI Cloud, etc.).

Features

EGI High Throughput Compute provides easy, uniform access to shared computing and data services of EGI service providers. Most software deployed in the distributed resource centres is based on open standards and open source middleware services.

The main features of the EGI High Throughput Compute are:

Access to high-quality computing resources
Integrated monitoring and accounting tools provide information about the availability and resource consumption
Workload and data management tools to manage all computational tasks
Large amounts of processing capacity over long periods of time
Faster results for your research
Shared resources enable collaborative research

The EGI HTC infrastructure

The EGI High Throughput Compute infrastructure is the federation of GRID resources provided by EGI providers. Its aim is to share in a secure way the distributed IT resources that are part of the EGI Cloud. It comprises of:

Compute Resources – execution environment for computing tasks, organized into clusters distributed across multiple resource centres in Europe and the World.
Data Infrastructure – storage servers from different resource centres where users can store their data/files in a distributed manner.
Federated Operations – global operational tasks (e.g., AAI, Accounting, Helpdesk) needed to federate the heterogeneous resources of resource centres and their operational activities.
User Support – EGI provides the central user support and coordinates support activities of EGI providers, who offer user support for the services/resources they contribute to the EGI ecosystem.

Architecture and service components

The key components of the EGI High Throughput Compute architecture are:

Data Transfer service (FTS)
Online Storage services
Computing Elements (CEs) are compute resources made available through GRID interfaces. The most common implementations of CEs in the EGI infrastructure are HTCondor-CE and ARC-CE.

Access model

Access to HTC resources in the EGI infrastructure is based on X.509 certificates and Virtual Organisations (VOs).

VOs are fully managed by research communities, allowing communities to manage their users and grant access to their services and resources. This means communities can either own their resources and use EGI services to share (federate) them, or can use the resources available in the EGI infrastructure for their scientific needs.

Before users can access EGI HTC services, they have to:

Obtain an X.509 certificate. The certificates are issued by Certification Authorities (CAs) part of the European Policy Management Authority for Grid Authentication (EUGridPMA), which is also part of the International Global Trust Federation (IGTF).
Enrol into a VO having access to HTC resources, as users are not individually granted access to resources.
Add the certificate to their internet browser of choice, or import it into the appropriate certificate store of their local machine (on Windows).
Proceed to Workload Manager to submit HTC jobs or retrieve job results, login using EGI Check-in when prompted.

If you are interested in using command-line and direct submission to Compute Elements, there is a tutorial on HTC job submission.

Users: Compute Orchestration

Mon, 01 Jan 0001 00:00:00 +0000

Overview

One of the challenges for researchers in recent years is to manage an ever-increasing amount of compute and storage services, which then form ever more complex end user applications or platforms.

To address this need, EGI provides several orchestrators that facilitate the management of your workload using resources on the federation. These tools have different levels of abstractions and features. See below to understand which to choose (or gets used automatically) in specific scenarios:

Service Name	Workload Type	Use-Case
Infrastructure Manager	VMs, containers, storage	Used to run workloads on a single IaaS Cloud provider.
Workload Manager	Jobs	Used to efficiently distribute, manage, and monitor computing workloads.

The following sections offer more details about each of these orchestrators.

Users: Software Distribution

Mon, 01 Jan 0001 00:00:00 +0000

This page documents usage the CernVM-FS (CVMFS) service operated for EGI by UKRI-STFC. For information on how to install a client, follow the instruction in the CVMFS official documentation

Overview

The CernVM-File System (CVMFS) provides a scalable, reliable and low-maintenance software distribution service. It was developed to assist High Energy Physics collaborations to deploy software on the worldwide distributed computing infrastructure used to run data processing applications. CVMFS is implemented as a POSIX read-only file system in user space. Files and directories are hosted on standard web servers and mounted in the universal namespace /cvmfs. CernVM-FS uses outgoing HTTP connections only, thereby it avoids most of the firewall issues of other network file systems. It transfers data and metadata on demand and verifies data integrity by cryptographic hashes. CVMFS is actively used by small and large collaborations. In many cases, it replaces package managers and shared software areas on cluster file systems as means to distribute the software used to process experiment data.

The current list of EGI repositories is as follows (disclaimer, some of them are inactive, but we keep them for archival purposes):

Repository	Project’s URL
auger.egi.eu
biomed.egi.eu	web page
cernatschool.egi.eu
chipster.egi.eu
comet.egi.eu
config-egi.egi.eu
dirac.egi.eu	web page
eiscat.egi.eu	web page
eosc.egi.eu
extras-fp7.egi.eu
galdyn.egi.eu
ghost.egi.eu
glast.egi.eu
gridpp.egi.eu
hyperk.egi.eu
intertwin.egi.eu
km3net.egi.eu	web page
ligo.egi.eu
lucid.egi.eu
mice.egi.eu
na62.egi.eu
neugrid.egi.eu
notebooks.egi.eu	web page
omnibenchmark.egi.eu	web page
pheno.egi.eu
phys-ibergrid.egi.eu
pravda.egi.eu
researchinschools.egi.eu
seadatanet.egi.eu
snoplus.egi.eu	web page
solidexperiment.egi.eu
supernemo.egi.eu
t2k.egi.eu
unpacked.egi.eu
wenmr.egi.eu
west-life.egi.eu

The list of EGI repositories can also be found online via the CVMFS monitor.

This documentation is for the VO content managers.

Official CVMFS pages

Requesting the creation of a new repository

In the case of a new repository for EGI, steps are described in PROC22.

Onboarding new Content Managers

Steps for a new VO Content Manager to be granted access to the Stratum-0.

Requesting access

Request access to the service sending an email to cvmfs-support@gridpp.rl.ac.uk In the email, include the following information:

Name of the VO or CVMFS repository.
Your Check-in ID from EGI Check-in.

Mailing list

All VO content managers should join the CVMFS-UPLOADER-USERS mailing list in JISCMAIL.

Distributing new content

To log into the service, just use ssh. The hostname is cvmfs-uploader-egi.gridpp.rl.ac.uk. Also, to maintain backwards compatibility, the alias cvmfs-upload01.gridpp.rl.ac.uk can be used. You need to specify explicitly which username you want to use to log in. The username is composed as reponame+"sgm". For example, for the repository dirac.egi.eu, the username is diracsgm.

# Replace with the proper username
$ ssh diracsgm@cvmfs-upload01.gridpp.rl.ac.uk

To copy data:

# Replace with the proper username
$ scp source_file.txt diracsgm@cvmfs-upload01.gridpp.rl.ac.uk:destination_folder/

When running the ssh or scp commands, a message like this is displayed:

# Replace with the proper username
$ ssh diracsgm@cvmfs-upload01.gridpp.rl.ac.uk
Authenticate at
-----------------
https://aai.egi.eu/device?user_code=AAAAA-BBBBB
-----------------
Hit enter when you have finished authenticating

Copy and paste the URL into a browser, and follow the instructions to authenticate yourself using your home institution Identity Management Service.

After login, you will find a single directory in the home directory:

$ ls
cvmfs_repo

Add the new content you want to distribute into that directory.

Building your software

CVMFS is an infrastructure to distribute software world-wide. However, the uploader host should not be used for the purposes of building and compiling it prior to distribution.

The right approach is for you to have your own local building environment, and use the uploader host only to upload the new content for distribution.

If you have non-relocatable software, then you will need a /cvmfs/<myrepo>/ directory on your building host. One option is to use an actual CVMFS client, so you have ready all the existing content being already distributed by CVMFS. By default, the /cvmfs/ directory on a CVMFS client host is read-only, but that can be solved using an ephemeral writable container.