Documentation – Ad-hoc tutorials

Users: Create your first Virtual Machine (VM)

Mon, 01 Jan 0001 00:00:00 +0000

Overview

This tutorial describes how to create your first Virtual Machine in the EGI Federation.

Step 1: Signing up

Create an EGI account with Check-in.

Step 2: Enrolling to a Virtual Organisation

Once your EGI account is ready you need to join a Virtual Organisation (VO). Here are the steps to join a VO. Explore the list of available VOs in the Operations Portal. We have a dedicated VO called vo.access.egi.eu for piloting purposes. If you are not sure about which VO to enrol to, please request access to the vo.access.egi.eu VO with your EGI account by visiting the enrolment URL.

Step 3: Creating a VM

Once your membership to a VO has been approved you are ready to create your first Virtual Machine. There are several ways to achieve this. The simplest way is to use the Infrastructure Manager Dashboard. On the other hand, advanced users may prefer to use the command-line interface.

To know more about the Cloud Compute Service in EGI please visit its dedicated section.

Asking for help

If you find issues please do not hesitate to contact us.

Users: Accessing virtual machines with SSH

Mon, 01 Jan 0001 00:00:00 +0000

Overview

This page provides an introduction of connecting from a local computer to a cloud host via SSH. It provides general guidelines, SSH options, tips, and examples for setting up the OpenStack environment.

SSH Keys

The recommended method to access a cloud virtual machine is via ssh using SSH keys. You may inject your public key into the virtual machine, at deployment time, and use your private key to connect via ssh without any password.

Tip

If you are using ssh keys in GitHub your public keys are available at: https://github.com/${github_username}.keys. i.e.: wget https://github.com/github_username.keys

SSH username

The username to use to connect to a virtual machine is dependent on the virtual machine image and is generally different in each operating system image:

For images available in the Artefact Registry, you should be able to find the username in the image description.
For official OS virtual machine images you can use the general OpenStack reference documentation on obtaining images.
For custom virtual machine images you need to refer to your virtual machine image provider (i.e. it could be something specific like cloudadm).
For virtual machines deployed with Infrastructure Manager the default username is cloudadm.

It is also possible to change the username using cloud-init with a user-data configuration (i.e. see the cloud config examples) or inject some code to add additional users (i.e. with Ansible).

Local ssh key configuration

The private ssh-key stored on your local computer is required to have restrictive file permissions. Depending on your local operative system you may need to run:

$ chmod 600 ~/.ssh/id_rsa

(with id_rsa being the name of the private key associated with the public key in use).

Username and password

Warning

Username and password access to cloud virtual machine images is usually disabled for security reasons and it is strongly suggested not to be used.

In case you have no other option, and are conscious of the risks, in order to enable SSH password authentication, the destination virtual machine needs to have /etc/ssh/sshd_config configuration changed from PasswordAuthentication no to PasswordAuthentication yes.

If really needed, a custom image with PasswordAuthentication enabled can be used or that can be injected when the virtual machine is deployed.

Depending on your deployment method it could be done with Ansible, Terraform, Salt, Puppet, Chef, cloud-init, or your own deployment tool if supported (i.e. the Infrastructure Manager and a custom TOSCA template).

Warning

If you enable PasswordAuthentication, be sure to generate a strong and unique password or passphrase for your account, otherwise you virtual machines will be compromised, and your access may be suspended.

OpenStack networking

The OpenStack environment needs to be populated with the necessary configurations and virtual hardware. To access the virtual machine from outside the OpenStack project you have to associate a floating IP to the virtual machine (which will provide a public IP to the virtual machine), you also have to open the necessary ports and add or edit the security groups, (more details on that in the specific section below).

Depending on the default configuration of the OpenStack project in order to associate a floating IP to a virtual machine in a private network it may be necessary to set up a virtual router in OpenStack and attach it with an interface to the private network. This step is usually not required as the OpenStack router is usually pre-configured by the cloud provider.

Security Groups Rules

The Virtual Machine that you want to connect needs to have the SSH port (22) reachable by your local machine. For that, it is necessary that a specific Rule is set up in one of the Security Groups associated with the virtual machine. The rule has to open port 22 either to any IPs (with CIDR 0.0.0.0/0) or to a specific IP (or subnet) matching the IP of the local machine used to connect with the virtual machine.

Sites are often providing a default security group, that may already contain this rule. You can check this using openstack security group rule list default.

Private IP vs public IP

Virtual machines in OpenStack are configured in a private network (like in the subnet 192.168.0.0/24) but you can directly SSH-connect with them from the internet only using a Public IP, which has to be associated with a virtual machine in the private network.

Accessing virtual machines in the private network

In general, to reach all the virtual machines in a private network, only a single public IP is needed.

The virtual machine associated with a public IP is often referred to as a Bastion host, once you connect with the bastion host, you can connect with the other virtual machine in the same private network using the private IPs. Alternatively, it is also possible to set up a JumpHost configuration in your local ssh configuration to do that with a single command.

Example: ssh configuration for Jump host

$ cat ~/.ssh/config
# Bastion
Host bastion 193.1.1.2
 User ubuntu
 Hostname 193.168.1.2
 IdentityFile ~/.ssh/id_rsa
 IdentitiesOnly yes

# with ProxyJump
Host private_vm
 HostName 192.168.1.2
 ProxyJump bastion

# old-style with ProxyCommand and additional settings
Host private_vm 192.168.1.2
 Hostname 192.168.1.2
 ProxyCommand ssh -q -A bastion nc %h %p
 User ubuntu
 ServerAliveInterval 60
 TCPKeepAlive yes
 ControlMaster auto
 ControlPath ~/.ssh/mux-%r@%h:%p
 ControlPersist 8h
 IdentityFile ~/.ssh/dev
 CheckHostIP=no
 StrictHostKeyChecking=no

General considerations related to setting up the ssh configuration are valid also for the connection between hosts in the private network (i.e. the ssh destination host needs to have a public key in the ~/.ssh/known_hosts file of the destination user, matching the private key used for the connection).

SSH connection practical example

Network configuration of two virtual machines A and B :

A private IP 192.168.1.2, public IP 193.168.1.2
B private IP 192.168.1.3

Connecting from a local machine to `A`

# ssh VM_OS_username@PUBLIC_IP
$ ssh centos@193.1.1.2

If the ssh local key is not the default ~/.ssh/id_rsa it needs to be specified with:

# ssh -i /path_of_your_private_ssh_key VM_OS_username@PUBLIC_IP
$ ssh -i ~/private_key centos@193.1.1.2

Connecting from a local machine to `B`

# from your computer
# connect to A
$ ssh centos@193.1.1.2

# from the shell opened in 193.1.1.2
# connect from A to B
$ ssh centos@192.168.1.3

Infrastructure Manager (IM)

The Infrastructure Manager (IM) provides the SSH key that can be used to connect to the virtual machine in the VM info page of the IM-Dashboard.

The page shows the information related with the virtual machine: the IP, the username (usually cloudadm), and the SSH key.

Token-based authentication

If supported by your virtual machine, you can also use ssh-oidc which implements the authentication consuming under-the-hood tokens from a local demon installed on your local machine.

More details on that soon.

The Infrastructure Manager (IM) can Enable SSH OIDC access to the VM in virtual machines by selecting the related Optional Features.

Users: Access DataHub from a VM

Mon, 01 Jan 0001 00:00:00 +0000

Overview

This tutorial describes the access to EGI DataHub spaces from a virtual machine. In the following paragraphs you will learn how to access data remotely stored in EGI DataHub like if they were local, using traditional POSIX command-line commands, by:

installing the oneclient component
configuring access to an EGI DataHub Oneprovider via oneclient

Prerequisites

In order to access the EGI DataHub data you need an EGI Check-in account. If you don’t have one yet you can Sign up for an EGI account.

Oneclient installation

The installation of oneclient package is currently supported for:

Ubuntu 18.04 LTS (Bionic Beaver)
Ubuntu 20.04 LTS (Focal Fossa)
CentOS 7
CentOS 8 Stream

Alternatively a docker based installation is also provided.

Oneclient installation via packages

Use the following command in order to install the oneclient package in a supported OS:

$ curl -sS https://get.onedata.org/oneclient.sh | bash

This will also install the needed dependencies.

Oneclient installation via docker

In order to use the Dockerized version of oneclient (provided that you have docker installed), you can run the following command:

$ docker run -it --privileged -v $PWD:/mnt/src --entrypoint bash onedata/oneclient:20.02.15

This command will also expose the current folder to the container (as /mnt/src) to ease the transfer of data.

Getting the token to access data

In order to access data stored in EGI DataHub via oneclient, you need to get an API access token.

Using oneclient

Once you have acquired a token valid for oneclient you can configure it on the environment as follows:

$ export ONECLIENT_ACCESS_TOKEN=<ACCESS_TOKEN_FROM_ONEZONE>

You must also configure in the environment the provider you would like to connect to. The EGI DataHub offers a PLAYGROUND space hosted by the Oneprovider plg-cyfronet-01.datahub.egi.eu which is accessible for testing by anyone with a valid EGI Check-in account.

Therefore the access to that particular space can be configured as follows:

$ export ONECLIENT_PROVIDER_HOST=plg-cyfronet-01.datahub.egi.eu

Now in order to access data from a local folder you need to run the following commands:

$ mkdir /tmp/space
$ oneclient /tmp/space

and then all usual file and folder operations (POSIX) will be available:

$ root@222d3ceb86df:/tmp/space# ls -l
total 0

drwxrwxr-x 1 root root 0 Jan 28 16:56 PLAYGROUND

Creating a file into the folder will push it to the Oneprovider and it will be accessible in the web interface and from other providers supporting the space.

By using the default settings you can see all the spaces you have access to, but it will also be possible to specify a specific space to access using the option --space <space>.

Oneclient offers a lot of other options for configuration (e.g. buffer size, direct I/O, etc) which are listed when you type the oneclient command without any argument.

Users: Create a VM with Jupyter and DataHub

Mon, 01 Jan 0001 00:00:00 +0000

Overview

This tutorial describes how to start a Virtual Machine in the EGI Federation that runs a browser-accessible Jupyter server with DataHub spaces mounted. This setup can be used in the EGI Federation or in any other provider which synchronise images from AppDB but is not part of the federation.

Requirements

This tutorial assumes you have:

A valid EGI account: learn to can create one in Check-in.
Access to a cloud provider where the Jupyter DataHub VM is available. Alternatively, this VM can be run on your computer using a virtualisation tool like VirtualBox.

Create a VM with Jupyter and DataHub

Step 1: Start your VM

Start your VM on your cloud provider or virtualisation tool. You can check the tutorial on how to start a VM to learn how to start a VM at EGI’s Federated Cloud infrastructure.

This VM does not contain any default credentials, in order to access it you need a ssh key. Check this FAQ entry for more information. If you are starting this VM on VirtualBox, you will need to pass some valid context for cloud-init, see here how to prepare it.

The VM image is ready to listen on port 22 for ssh access and port 80 for accessing the notebooks interface. Make sure your have those ports open on your security groups, otherwise you will not be able to reach the Jupyter notebooks.

Once your instance is ready, assign it a public IP so you can reach it from your computer.

Step 2: Get a hostname and certificate for your VM

Your VM is ready to be accessible, but runs a plain HTTP server, which is not secure enough. If you try to connect with your browser to your VM, you will get a message as shown in the screenshot below:

You must enable HTTPS to encrypt requests and responses, thus making your VM safer and more secure.

Firstly, you need a valid name for your VM. You can use the FedCloud Dynamic DNS to create a name. See Dynamic DNS docs for more information on the service. Once you have your name ready, assign it your VM’s IP.

Secondly, you need to get a certificate to enable HTTPS. The VM has certbot already installed, you just need to run it with the hostname you have allocated and your email address as shown here:

# log into your VM
$ ssh ubuntu@<your VM's IP>
# now request the certificate
$ sudo certbot --nginx -d <your registered name> -m <your email>

Finally, open your browser and go to https://<your registered name>/ to see Jupyter started. Follow next steps for getting the credentials to access the service.

Enabling insecure access

If you really need to use HTTP (e.g. your VM is not accessible publicly and you cannot create a certificate for it), you can disable the error shown by default in the nginx configuration.

Open /etc/nginx/sites-enabled/default and comment out lines 14-16:

 # if ( $https != 'on' ) {
 # return 406;
 # }

And restart nginx:

$ sudo systemctl restart nginx

Step 3: Get your token for the Jupyter server

Your VM will spawn a Jupyter notebooks server upon starting. This server runs as an unprivileged user named jovyan with the software installed using micromamba. The server uses a randomly generated token for authentication that you can obtain by logging into the VM and becoming jovyan:

$ ssh ubuntu@<your VM's IP>
# become jovyan user and activate the default environment
$ sudo su - jovyan
$ micromamba activate
$ jupyter server list --jsonlist | jq -r .[].token
<your token>

Step 4: Start your notebooks

Now point your browser to http://<your VM's IP> and you will be able to enter the token to get started with Jupyter.

You can install additional packages with mamba from a terminal started from Jupyter or via ssh. For example for installing tensorflow:

$ micromamba activate
$ micromamba install -c conda-forge tensorflow

Step 5: Mount DataHub spaces

Log into EGI’s DataHub and create a token for mounting your data in the VM.

You will also need the IP or address of your closest Oneprovider for the spaces you are interested in accessing. This information is easily obtainable via DataHub’s web interface.

Go to your Jupyter session in your browser and edit the mount.sh file in your home directory. Set the ONECLIENT_ACCESS_TOKEN and ONECLIENT_PROVIDER_HOST values to get access to DataHub:

Open a terminal from the launcher screen and execute the mount.sh script:

You should now see a datahub folder with all your spaces available directly from your Jupyter interface

Users: Automate with oidc-agent, fedcloudclient, terraform and Ansible

Mon, 01 Jan 0001 00:00:00 +0000

Overview

This tutorial describes how to create a Virtual Machine in the EGI Federation, leveraging oidc-agent to retrieve OIDC tokens from EGI Check-in, fedcloudclient to simplify interacting with the EGI Cloud Compute service, terraform and Ansible to simplify deploying an infrastructure. EGI Dynamic DNS is also used to assign a domain name to the virtual machine, which can then be used to get a valid TLS certificate from Let’s Encrypt.

Step 1: Signing up for an EGI Check-in account

Create an EGI account with Check-in.

Step 2: Enrolling to a Virtual Organisation

This tutorial will assume you are using vo.access.egi.eu, adapt as required for your specific environment.

Step 3: Creating a VM

Once your membership to a VO has been approved you are ready to create your first Virtual Machine.

The OpenID Connect (OIDC) protocol is used to authenticate users and authorise access to Cloud Compute resources that are integrated with EGI Check-in.

While it’s not mandatory, a convenient way to manage the OIDC token is to use oidc-agent.

Setting up oidc-agent

oidc-agent is a set of tools to manage OpenID Connect tokens and make them easily usable from the command line.

Install oidc-agent according to official documentation, once oidc-agent is installed it can be used to retrieve an OIDC access token from EGI Check-in.

# Generating configuration for EGI Check-in
$ oidc-gen --pub --issuer https://aai.egi.eu/auth/realms/egi \
 --scope "email \
 eduperson_entitlement \
 eduperson_scoped_affiliation \
 eduperson_unique_id" egi
# Listing existing configuration
$ oidc-add -l
# Requesting an OIDC access token
$ oidc-token egi
# Exporting a variable with a Check-in OIDC access token to be used with OpenStack
# XXX access tokens are short lived, relaunch command to obtain a new token
# This is *not* required for following this tutorial, it's an example
$ export OS_ACCESS_TOKEN=$(oidc-token egi)

It’s possible to automatically start oidc-agent in your shell initialisation, example that can be added to ~/.bash_profile or ~/.zshrc:

if command -v oidc-agent-service &> /dev/null
 eval $(oidc-agent-service use)
 # for fedcloudclient, selecting egi configuration generated with oidc-gen
 export OIDC_AGENT_ACCOUNT=egi
fi

When using oidc-agent-service, fedcloudclient will be able to automatically request a new access token from oidc-agent.

See full documentation.

Installing fedcloudclient and ansible

fedcloudclient is an high-level Python package for a command-line client designed for interaction with the OpenStack services in the EGI infrastructure. The client can access various EGI services and can perform many tasks for users including managing access tokens, listing services, and mainly execute commands on OpenStack sites in EGI infrastructure.

fedcloudclient can leverage oidc-agent if it’s installed and properly configured.

fedcloudclient and openstackclient, the official OpenStack python client, will be used to interact with the EGI Cloud Compute service.

Required python dependencies are documented in a requirements.txt file (Ansible will be used at a later stage, but is installed at the same time):

openstackclient
fedcloudclient
ansible

For keeping the main system tidy and isolating the environment, the python packages will be installed in a dedicated <– cspell:disable-next-line –> python virtualenv:

# Creating an arbitrary directory where to store python virtual environments
$ mkdir -p ~/.virtualenvs
# Creating a python 3 virtual environment
$ python3 -m venv ~/.virtualenvs/fedcloud
# Activating the virtual environment
$ source ~/.virtualenvs/fedcloud
# Installing required python packages in the virtual environment
$ pip install -r requirements.txt

Identifying a suitable cloud site

It’s possible to deploy an OpenStack Virtual Machine (VM) on any of the sites supporting the Virtual Organisations (VO) you are a member of.

Once fedcloudclient is installed it’s possible to get information about the OIDC token accessed via oidc-agent.

# Listing the VO membership related to the OIDC access token
$ fedcloud token list-vos

In the following example, the IN2P3-IRES site supporting the vo.access.egi.eu VO will be used, see Step 2: Enrolling to a Virtual Organisation to request access.

Deploying the Virtual Machine with terraform

Instead of creating the server manually, it is possible to use terraform with EGI Cloud Compute.

The Terraform OpenStack provider provides official documentation.

Terraform provides installation instructions for all usual platforms.

Once terraform is installed locally, we will create a deployment as documented in the following sections.

Setting up the environment

The OS_* variables that will be used by terraform can be generated using fedcloudclient.

# Activating the virtual environment
$ source ~/.virtualenvs/fedcloudclient/bin/activate
# Exporting variable for VO and SITE to avoid having to repeat them
$ export EGI_VO='vo.access.egi.eu'
$ export EGI_SITE='IN2P3-IRES'
eval $(fedcloud site env)
# Obtaining an OS_TOKEN for terraform
# XXX this breaks using openstackclient: use fedcloudclient
# or unset OS_TOKEN before using openstackclient
$ export OS_TOKEN=$(fedcloud openstack token issue --site "$EGI_SITE" \
 --vo "$EGI_VO" -j | jq -r '.[0].Result.id')

Describing the terraform variables

The main terraform configuration file, main.tf is using variables that have to be described in a vars.tf file:

# Terraform variables definition
# Values to be provided in a *.tfvars file passed on the command line

variable "internal_net_id" {
 type = string
 description = "The id of the internal network"
}

variable "public_ip_pool" {
 type = string
 description = "The name of the public IP address pool"
}

variable "image_id" {
 type = string
 description = "VM image id"
}

variable "flavor_id" {
 type = string
 description = "VM flavor id"
}

variable "security_groups" {
 type = list(string)
 description = "List of security groups"
}

The SITE and VO specific values for those variables will be identified and documented in a $EGI_SITE.tfvars file.

Identifying the cloud resources

Once the environment is properly configure, fedcloudclient is used to gather information and identify flavor, image, network and security groups for the site you want to use.

fedcloud openstack currently requires an explicit --site parameter, this will be addressed in a future fedcloud release. In the meantime the $EGI_SITE environment variable can be reused using --site "$EGI_SITE".

# Selecting an image
$ fedcloud select image --image-specs "Name =~ 'EGI.*22'"
# Selecting a flavor
$ fedcloud select flavor --flavor-specs "RAM>=2096" \
 --flavor-specs "Disk > 10" --vcpus 2
# Identifying available networks
$ fedcloud openstack --site "$EGI_SITE" network list
$ fedcloud select network --network-specs default
# Identifying security groups
$ fedcloud openstack --site "$EGI_SITE" security group list
# Listing rules of a specific security group
$ fedcloud openstack --site "$EGI_SITE" security group rule list default

Documenting the cloud resources for the selected site

The chosen flavor, image, network and security group should be documented in a $EGI_SITE.tfvars file that will be passed as an argument to terraform commands.

The network configuration can be tricky, and is usually dependant on the site. For IN2P3-IRES, one has to request a floating IP from the public network IP pool ext-net, and assign this floating IP to the created instance. For another site it may not be needed, in that case the main.tf will have to be adjusted accordingly.

See the example IN2P3-IRES.tfvars below, to be adjusted according to the requirements and to the selected site and VO:

# Internal network
internal_net_id = "7ae7b0ca-f122-4445-836a-5fb7af524dcb"

# Public IP pool for floating IPs
public_ip_pool = "ext-net"

# Flavor: m1.medium
flavor_id = "ab1fbd4c-324d-4155-bd0f-72f077f0ebce"

# Image: EGI Ubuntu 22.04
# image_id = "fc6c83a3-845f-4f29-b44d-2584f0ca4177"

# Security groups
security_groups = ["default"]

Creating the main terraform deployment file

To be more reusable, the main.tf configuration file is referencing variables described in the vars.tf file created previously, and will take the values from the $EGI_SITE.tfvars file passed as an argument to the terraform command.

# Terraform versions and providers
terraform {
 required_version = ">= 0.14.0"
 required_providers {
 openstack = {
 source = "terraform-provider-openstack/openstack"
 version = "~> 1.35.0"
 }
 }
}

# Allocate a floating IP from the public IP pool
resource "openstack_networking_floatingip_v2" "egi_vm_floatip_1" {
 pool = var.public_ip_pool
}

# Creating the VM
resource "openstack_compute_instance_v2" "egi_vm" {
 name = "egi_test_vm"
 image_id = var.image_id
 flavor_id = var.flavor_id
 security_groups = var.security_groups
 user_data = file("cloud-init.yaml")
 network {
 uuid = var.internal_net_id
 }
}

# Attach the floating public IP to the created instance
resource "openstack_compute_floatingip_associate_v2" "egi_vm_fip_1" {
 instance_id = "${openstack_compute_instance_v2.egi_vm.id}"
 floating_ip = "${openstack_networking_floatingip_v2.egi_vm_floatip_1.address}"
}

# Create inventory file for Ansible
resource "local_file" "hosts_cfg" {
 content = templatefile("${path.module}/hosts.cfg.tpl",
 {
 ui = "${openstack_networking_floatingip_v2.egi_vm_floatip_1.address}"
 }
 )
 filename = "./inventory/hosts.cfg"
}

The last resource is relying on templatefile to populate the inventory file that will later be used by ansible.

Initial configuration of the VM using cloud-init

cloud-init is the industry standard multi-distribution method for cross-platform cloud instance initialization.

The initial configuration of the VM is done using a cloud-init.yaml file.

The curl call from the runcmd block in the cloud-init.yaml configuration below, will register the IP of the virtual machine in the DNS zone managed using the EGI Dynamic DNS service, allowing to access the virtual machine using a fully qualified hostname and allowing to retrieve a Let’s Encrypt certificate.

Please look at the EGI Dynamic DNS documentation for instructions on creating the configuration for a new host.

The users block in the cloud-init.yaml configuration below, will create a new user with password-less sudo access.

While this egi user can only be accessed via the specified SSH key(s), setting a user password and requesting password verification for using sudo should be considered, as a compromise of this user account would mean a compromise of the complete virtual machine.

Replace <NSUPDATE_HOSTNAME>, <NSUPDATE_SECRET>, <SSH_AUTHORIZED_KEY> (the content of your SSH public key) by the proper values.

---
# cloud-config
runcmd:
 - [
 curl,
 "https://<NSUPDATE_HOSTNAME>:<NSUPDATE_SECRET>@nsupdate.fedcloud.eu/nic/update",
 ]

users:
 - name: egi
 gecos: EGI
 primary_group: egi
 groups: users
 shell: /bin/bash
 sudo: ALL=(ALL) NOPASSWD:ALL
 ssh_authorized_keys:
 - <SSH_AUTHORIZED_KEY>

packages:
 - vim

package_update: true
package_upgrade: true
package_reboot_if_required: true

Launching the terraform deployment

Now that all the files have been created, it’s possible to deploy the infrastructure, currently only a single VM, but it can easily be extended to a more complex setup, using terraform:

# Initialising working directory, install dependencies
$ terraform init
# Reviewing plan of actions for creating the infrastructure
# Use relevant site-specific config file
$ terraform plan --var-file="${EGI_SITE}.tfvars"
# Creating the infrastructure
# Manual approval can be skipped using -auto-approve
# The SERVER_ID will be printed (openstack_compute_instance_v2.scoreboard)
$ terraform apply --var-file="${EGI_SITE}.tfvars"
# Wait a few minutes for the setup to be finalised
# Connecting to the server using ssh
$ ssh egi@$NSUPDATE_HOSTNAME

From here you can extend the cloud-init.yaml and/or use Ansible to configure the remote machine, as well as doing manual work via SSH.

Debugging terraform

The token used by Terraform for accessing OpenStack is short lived, it will have to be renewed from time to time.

# Creating a new token to access the OpenStack endpoint
$ export OS_TOKEN=$(fedcloud openstack token issue --site "$EGI_SITE" \
 --vo "$EGI_VO" -j | jq -r '.[0].Result.id')

It is possible to print a verbose/debug output to get details on interactions with the OpenStack endpoint.

# Debugging
$ OS_DEBUG=1 TF_LOG=DEBUG terraform apply --var-file="${EGI_SITE}.tfvars"

Destroying the resources created by terraform

# Destroying the created infrastructure
$ terraform destroy --var-file="${EGI_SITE}.tfvars"

Step 4: Using Ansible

Ansible can be used to manage the configuration of the crated virtual machine.

The terraform deployment generated an Ansible inventory, inventory/hosts.cfg, that can directly be used by Ansible.

Configure a basic Ansible environment in the ansible.cfg file:

[defaults]
# Use user created using cloud-init.yml
remote_user = egi
# Use inventory file generated by terraform
inventory = ./inventory/hosts.cfg

[privilege_escalation]
# Escalate privileges using password-less sudo
become = yes

Then you can verify that the Virtual Machine is accessible by Ansible:

# Confirming ansible can reach the VM
$ ansible all -m ping

Once this works, you can create advanced playbooks to configure your deployed host(s).

Various Ansible roles are available in the egi-qc/ansible-playbooks repository and in the EGI Federation GitHub organisation.

A style guide for writing Ansible roles is providing a skeleton that you can use fore creating new roles.

Additional resources

Additional resources are available, and can help with addressing different use cases, or be used as a source of inspiration:

egi-qc/deployment-howtos: Deployment recipes extracted from Jenkins builds for the UMD and CMD products
EGI-ILM/fedcloud-terraform: providing an advanced helper script allowing to interact with EGI Cloud Compute.
EGI-ILM/automated-containers: providing documentation for automated on-demand execution of Docker containers

Asking for help

If you find issues please do not hesitate to contact us.

Users: Data transfer with grid storage

Mon, 01 Jan 0001 00:00:00 +0000

Overview

This tutorial describes the EGI Data Transfer using FTS transfers services and WebFTS. In the following paragraphs you will learn how to use the FTS command-line client to perform data transfers between two Grid storage.

Prerequisites

As first step please make sure that you have installed the FTS client as described in Data Transfer, and in particular Clients for the command-line FTS and to have your certificate installed in your browser to use WebFTS browser based client.

To access services and resources in the EGI Federated Cloud, you will need:

An EGI Check-in account, you can sign up here
Enrolment into a Virtual Organisation (VO) that has access to the services and resources you need

FTS client usage

Step 1 Configuration check

To verify that everything is configured properly you can check with the following command and pointing to the certificates directly:

$ fts-rest-whoami --key ~/.globus/userkey.pem --cert ~/.globus/usercert.pem \
 -s https://fts3-public.cern.ch:8446/
User DN: /DC=org/DC=terena/DC=tcs/C=NL/O=Stichting EGI/CN=Jane Doe jd@egi.eu
VO: JaneDoejd@egi.eu@tcs.terena.org
VO id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
Delegation id: XXXXXXXXXXXXXXXX
Base id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX

In general the commands can be used by specifying the user public and private key like shown in the example or by creating a proxy certificate as described in the following section.

Step 2 Proxy creation

As you have seen in the previous section it is possible to use the FTS commands by specifying the location of the user public and private key. With the use of voms-proxy-init it is possible to create a proxy certificate for the user. With this you don’t need to specify the location of the public and private key for each FTS command. When running voms-proxy-init it’s possible to specify the location of the public and private key. If this are not included as options, the tool expect to find them in:

~/.globus/usercert.pem for the public key
~/.globus/userkey.pem for the private key with read access only for the owner

Following is an example of running this command with the public and private key already setup as described:

$ voms-proxy-init
Your identity: /DC=org/DC=terena/DC=tcs/C=NL/O=Stichting EGI/CN=Jane Doe jd@egi.eu
Creating proxy ........................................... Done

Your proxy is valid until Wed Aug 25 04:18:14 2021

The output of the command shows, a proxy certificate valid for 12 hours has been generated This is the default behaviour and can be usually increased, for example to 48 hours, with the following option:

$ voms-proxy-init -valid 48:00
Your identity: /DC=org/DC=terena/DC=tcs/C=NL/O=Stichting EGI/CN=Jane Doe jd@egi.eu
Creating proxy ................................... Done

Your proxy is valid until Thu Aug 26 16:23:01 2021

To verify for how long the proxy is still valid you can use the following command: command:

$ voms-proxy-info
subject : /DC=org/DC=terena/DC=tcs/C=NL/O=Stichting EGI/CN=Jane Doe jd@egi.eu/CN=1451339003
issuer : /DC=org/DC=terena/DC=tcs/C=NL/O=Stichting EGI/CN=Jane Doe jd@egi.eu
identity : /DC=org/DC=terena/DC=tcs/C=NL/O=Stichting EGI/CN=Jane Doe jd@egi.eu
type : RFC compliant proxy
strength : 1024 bits
path : /tmp/x509up_u1000
timeleft : 19:59:57

When the timeleft reaches zero the same command will produce the following message:

$ fts-rest-whoami -s https://fts3-public.cern.ch:8446/
Error: Proxy expired!

The last option that you need to use is specify the VO that you want to use for the proxy being created. In the following example the dteam VO has been used:

$ voms-proxy-init --voms dteam
Your identity: /DC=org/DC=terena/DC=tcs/C=NL/O=Stichting EGI/CN=Jane Doe jd@egi.eu
Creating temporary proxy ................................................................... Done
Contacting voms2.hellasgrid.gr:15004 [/C=GR/O=HellasGrid/OU=hellasgrid.gr/CN=voms2.hellasgrid.gr] "dteam" Done
Creating proxy .............................................................................. Done

Your proxy is valid until Wed Sep 8 04:37:07 2021

With a proxy now available for the user it is now possible to execute the FTS commands without specifying the public and private keys as it will be shown in the following sections.

Step 3 Find the storage

In general, the source and destination storage for a specific project should be already known. However, to discover the available source or destination endpoints to be used for a transfer, you can use the VAPOR service.

Once the page is loaded on the left column it is possible to filter by VO or scroll the list and click the desired VO as show in the following picture:

Once selected, you can see all the resources associated with the specific VO. In particular in this case you are interested in the information on the status, capacity, type of storage, etc. Following is a screenshot of the visualisation of the list of storage available to dteam.

Step 4 Starting a transfer

Once you have identified the source and destination storage needed for the transfer you can proceed with the transfer between the two endpoints. To do that you can use a command of this type, returning the job ID corresponding to the transfer that you started:

$ fts-transfer-submit -s https://fts3-public.cern.ch:8446/ \
 --source https://dc2-grid-64.brunel.ac.uk/dpm/brunel.ac.uk/home/dteam/1M \
 --destination https://golias100.farm.particle.cz/dpm/farm.particle.cz/home/dteam/1M \
 -o cfc884f8-1181-11ec-b9c7-fa163e5dcbe0

To check the status of the transfer you can use the returned job ID and use the following command specifying the server controlling the transfer, the source and the transfer itself:

$ fts-transfer-status -s https://fts3-public.cern.ch:8446/ \
 cfc884f8-1181-11ec-b9c7-fa163e5dcbe0
FINISHED

The last option -o specify that the file should be overwritten if present on the destination. If this option is not present and a file with the same name exists on the destination, the transfer itself will fail. If you use this option you should make sure that it is safe to do so.

Users: Submitting HTC Jobs

Mon, 01 Jan 0001 00:00:00 +0000

Overview

This tutorial describes how to submit High Throughput Compute (HTC) jobs using command-line.

This tutorial is meant for somewhat advanced users or the ones willing or needing to interact with the resources at a low level.

Prerequisites

To submit an EGI HTC job, you will have to:

Obtain an X.509 user certificate. The supported certificates are issued by Certification Authorities (CAs) part of the European Policy Management Authority for Grid Authentication (EUGridPMA), which is also part of the International Global Trust Federation (IGTF).
Enrol into a VO having access to HTC resources.

This tutorial will be using dteam a test Virtual Organisation that can be used by resource providers, commands should be adjusted to the appropriate VO.

Step 1: getting access to a User Interface (UI)

In order to interact with HTC resources, you should have access to a User Interface, often referred to as a UI. This software environment will provide all the tools required to interact with the different middleware, as different sites can be using different Computing Element (CE), such as HTCondorCE and ARC-CE (CREAM is a legacy software stack that is not officially supported).

Different possibilities are available to access an UI:

Having access to an UI provided by/for your community, please get in touch with them about this.
Deploying a UI, as documented below.

Deploying an UI

The UI is available as a package in the UMD software distribution, but it will also require additional software and configuration.

In order to help with deploying an UI, different solutions are possible:

Deploying an UI manually, using the packages available from UMD repositories. You will need to install at least the ui meta-package, the IGTF distribution, and configure the system to use voms-client.
Some Ansible roles are available in the EGI Federation GitHub organisation, mainly ansible-role-ui that should be used together with ansible-role-VOMS-client, providing software and material required for the authentication and authorisation, and ansible-role-umd configuring the software repositories from where all the software will be installed.
The repository ui-deployment provides a terraform based deployment allowing to deploy a User Interface (UI) in a Cloud Compute virtual machine. This integrated deployment is based on the Ansible modules, and should be adjusted to your environment and needs.

This tutorial is based on using a VM deployed using the ui-deployment repository, refer to the repository for detailed instructions on deploying the UI.

Step 2: creating a VOMS proxy

The Virtual Organization Membership Service (VOMS) enables Virtual Organisation (VO) access control in distributed services. A proxy allows limited delegation of rights, allowing remote services to securely interact with other resources and services on behalf of the user.

Configuring the system to use voms-client

When using ansible-role-VOMS-client, the full environment has been setup for you, and there is no need for manual configuration.

Before being able to use voms-client, it is required to configure access to the VOMS server of the chosen VO, using the proper .vomses and .lsc files, based on the information available on the VOMS server of the specific VO.

as an example with dteam, you can find the VOMS server address in the Operations Portal: https://operations-portal.egi.eu/vo/view/voname/dteam. Then looking at dteam VOMS configuration, you can create:
- /etc/vomses/dteam-voms2.hellasgrid.gr with the content of the VOMSES string.
- /etc/grid-security/vomsdir/dteam/voms2.hellasgrid.gr.lsc with the content for the LSC configuration.

If you cannot edit content in /etc/vomses and /etc/grid-security/vomsdir, you can respectively use ~/.glite/vomses and ~/.glite/vomsdir. You may have to export X509_VOMSES and X509_VOMS_DIR in your shell, as documented on CERN’s twiki:

$ export X509_VOMSES=~/.glite/vomses
$ export X509_VOMS_DIR=~/.glite/vomsdir

Preparing the X.509 credentials

Once you have obtained an X.509 user certificate issued by a Certification Authority (CA) part of the International Global Trust Federation (IGTF), you should extract the certificate and private key, and add them to a ~/.globus directory.

If the X.509 certificate is in your browser’s keyring, you should export it to a passphrase protected .p12 file, then using openssl pkcs12 you can extract the required PEM files:

# Creating and protecting ~/.globus directory
$ mkdir -p ~/.globus
$ chmod 750 ~/.globus
# Extracting the certificate from the p12 file "exported_cert.p12"
openssl pkcs12 -in exported_cert.p12 -out ~/.globus/usercert.pem -clcerts -nokeys
# Adjusting rights on the user certificate
$ chmod 640 ~/.globus/usercert.pem
# Extracting the certificate key from the p12, protecting it with a passphrase
$ openssl pkcs12 -in exported_cert.p12 -out ~/.globus/userkey.pem -nocerts
# Adjusting rights on the certificate key
$ chmod 400 ~/.globus/userkey.pem

If you are using a certificate provided by the GÉANT Trusted Certificate Service (TCS), in addition to the official documentation provided by your organisation, you may be interested by looking at the following documentation:

Generation 4 GEANT Trusted Certificate Service TCS, covering how to get and install your credentials, addressing potential issues with an improper .p12. Highly recommended.
SUNET TCS 2020- Information for administrators, an exhaustive documentation mainly for administrators but also covering client-related aspects.

Using voms-client

Once the configuration for the VOMS client has been completed, and when the X.509 credentials have been prepared, you can create a VOMS proxy for dteam VO:

# Creating the proxy
$ voms-proxy-init -voms dteam
Enter GRID pass phrase for this identity:
Contacting voms2.hellasgrid.gr:15004 [/C=GR/O=HellasGrid/OU=hellasgrid.gr/CN=voms2.hellasgrid.gr] "dteam"...
Remote VOMS server contacted successfully.


Created proxy in /tmp/x509up_u1001.

Your proxy is valid until Wed Oct 26 23:27:30 CEST 2022
# Checking the proxy
$ voms-proxy-info
subject : /DC=org/DC=terena/DC=tcs/C=NL/O=Stichting EGI/CN=Jane Doe/CN=123456319
issuer : /DC=org/DC=terena/DC=tcs/C=NL/O=Stichting EGI/CN=Jane Doe
identity : /DC=org/DC=terena/DC=tcs/C=NL/O=Stichting EGI/CN=Jane Doe
type : RFC3820 compliant impersonation proxy
strength : 2048
path : /tmp/x509up_u1001
timeleft : 11:58:48
key usage : Digital Signature, Key Encipherment

References

Step 3: identifying available resources

It is possible to identify available resources by querying the information system.

Two Computing Element (CE) “flavours” are used in production:

HTCondorCE, a Compute Entrypoint (CE) based on HTCondor.
ARC-CE, the ARC Compute Element (CE).

In this section we will document querying the EGI Information System to retrieve information about the available resources.

Tip

It’s also possible to use VAPOR to query resources using a graphical interface.

Use case: identifying all the Computing Elements supporting the dteam VO

As documented in the pages covering the querying of the Information System, in GLUE 2.0, the access granted to a given VO to a compute or storage resource, is published using the GLUE2Share and GLUE2Policy objects. The GLUE2ComputingShare object specifically documents sharing of compute resources.

# Querying GLUE2ComputingShare for all the computing resources available to dteam VO
$ ldapsearch -x -LLL -H ldap://lcg-bdii.egi.eu:2170 \
 -b "GLUE2GroupID=grid,o=glue" \
 '(&(objectClass=GLUE2ComputingShare)(GLUE2ShareID=*dteam*))'

It is possible to filter for the different types of Computing Element, and select only specific attributes.

Once you will have selected a site, using the ldapsearch queries from the next subsections, you will be able to send jobs to them, as documented in the Step4: submitting and managing jobs.

Information

The following Computing Elements have been arbitrarily chosen, like due to the site location, available resources, prior experience, or any other reason, and will be used in this tutorial:

HTCondorCE: condorce1.ciemat.es
ARC-CE: alex4.nipne.ro

Looking for a HTCondorCE for dteam

# Information about the HTCondorCE supporting dteam VO
$ ldapsearch -x -LLL -H ldap://lcg-bdii.egi.eu:2170 \
 -b "GLUE2GroupID=grid,o=glue" \
 '(&(objectClass=GLUE2ComputingShare)(GLUE2ShareID=*dteam*)(GLUE2ComputingShareComputingEndpointForeignKey=*HTCondorCE*))' \
 GLUE2ShareEndpointForeignKey \
 GLUE2ShareID \
 GLUE2ComputingShareTotalJobs \
 GLUE2ComputingShareRunningJobs \
 GLUE2ComputingShareWaitingJobs

# XXX Most HTCondorCE have the Endpoint ending in `HTCondorCE`, but some have
# it ending with `htcondorce`, like in this tutorial for `condorce1.ciemat.es`
# XXX The attribute `GLUE2ComputingShareComputingEndpointForeignKey` is matched
$ in a case sensitive way, and the filter should be updated to match them
$ ldapsearch -x -LLL -H ldap://lcg-bdii.egi.eu:2170 \
 -b "GLUE2GroupID=grid,o=glue" \
 '(&(objectClass=GLUE2ComputingShare)(GLUE2ShareID=*dteam*)(|(GLUE2ComputingShareComputingEndpointForeignKey=*HTCondorCE*)(GLUE2ComputingShareComputingEndpointForeignKey=*htcondorce*)))' \
 GLUE2ShareEndpointForeignKey \
 GLUE2ShareID \
 GLUE2ComputingShareTotalJobs \
 GLUE2ComputingShareRunningJobs \
 GLUE2ComputingShareWaitingJobs

As it was decided to go for condorce1.ciemat.es, the information about the CE can be requested using the following request, filtering on the GLUE2ShareID from the previous query: grid_dteam_condorce1.ciemat.es_ComputingElement.

# condor_submit needs CE (condorce1.ciemat.es) and pool (condorce1.ciemat.es:9619)
$ ldapsearch -x -LLL -H ldap://lcg-bdii.egi.eu:2170 \
 -b "GLUE2GroupID=grid,o=glue" \
 '(&(objectClass=GLUE2ComputingShare)(GLUE2ShareID=*grid_dteam_condorce1.ciemat.es_ComputingElement*))' \
 GLUE2ShareID \
 GLUE2ShareDescription \
 GLUE2ComputingShareExecutionEnvironmentForeignKey \
 GLUE2EntityOtherInfo

HTCondorCE are usually running on port 9619, this is confirmed by the results. Based on those results, it’s possible to guess the following parameters that will have to be used when submitting the job:

CE Name: condorce1.ciemat.es (reported in GLUE2ComputingShareExecutionEnvironmentForeignKey: condorce1.ciemat.es)
CE Pool: condorce1.ciemat.es:9619 (reported in GLUE2EntityOtherInfo: HTCondorCEId=condorce1.ciemat.es:9619/htcondorce-condor-group_dteam)

Looking for an ARC-CE for dteam

// jscpd:ignore-start

# Information about the ARC-CE supporting dteam VO
$ ldapsearch -x -LLL -H ldap://lcg-bdii.egi.eu:2170 \
 -b "GLUE2GroupID=grid,o=glue" \
 '(&(objectClass=GLUE2ComputingShare)(GLUE2ShareID=*dteam*)(GLUE2ComputingShareComputingEndpointForeignKey=*urn:ogf*))' \
 GLUE2ComputingShareComputingEndpointForeignKey \
 GLUE2ShareEndpointForeignKey \
 GLUE2ComputingShareTotalJobs \
 GLUE2ComputingShareRunningJobs \
 GLUE2ComputingShareWaitingJobs

As it was decided to go for alex4.nipne.ro, the information about the CE can be requested using the following request, filtering on the GLUE2ShareID from the previous query: urn:ogf:ComputingShare:alex4.nipne.ro:dteam_dteam.

# arcsub needs CE name (alex4.nipne.ro)
$ ldapsearch -x -LLL -H ldap://lcg-bdii.egi.eu:2170 \
 -b "GLUE2GroupID=grid,o=glue" \
 '(&(objectClass=GLUE2ComputingShare)(GLUE2ShareID=*urn:ogf:ComputingShare:alex4.nipne.ro:dteam_dteam*))' \
 GLUE2ShareID \
 GLUE2ShareDescription \
 GLUE2ComputingShareComputingServiceForeignKey \
 GLUE2ComputingShareExecutionEnvironmentForeignKey

// jscpd:ignore-end

CE Name: alex4.nipne.ro (exported from the GLUE2ComputingShareComputingServiceForeignKey: urn:ogf:ComputingService:alex4.nipne.ro:arex)

Step 4: submitting and managing jobs

To an HTCondorCE Computing Element

The HTCondor-CE software is a Compute Entrypoint (CE) based on HTCondor for sites that are part of a larger computing grid (e.g. EGI, Open Science Grid (OSG)).

The condor package will install all the required dependencies.

yum install condor

Condor will use the VOMS proxy created earlier.

While HTCondor provides an official HTCodnor Quick Start Guide, the main steps for managing a job will be highlighted below.

Create env.sub, the compute job to be executed on the remote Computing Element:

executable = /usr/bin/env
log = env.log
output = outfile.txt
error = errors.txt
should_transfer_files = Yes
when_to_transfer_output = ON_EXIT
queue

The format of the submit description file, is documented in HTCondor manual and condor_submit man page.

Submission of a job with the -spool option causes HTCondor to spool all input files, the job event log, and any proxy across a connection to the machine where the condor_schedd daemon is running. After spooling these files, the machine from which the job is submitted may disconnect from the network or modify its local copies of the spooled files.

Submit job using condor_submit:

# Submitting a job, spooling input and output files to
$ condor_submit --spool --name condorce1.ciemat.es \
 --pool condorce1.ciemat.es:9619 env.sub
Submitting job(s).
1 job(s) submitted to cluster 97412.

Monitor the status of the job using condor_q:

# Checking the status of a specific job
$ condor_q --name condorce1.ciemat.es --pool condorce1.ciemat.es:9619 97412


-- Schedd: condorce1.ciemat.es : <192.101.161.188:9619?... @ 10/26/22 16:31:00
OWNER BATCH_NAME SUBMITTED DONE RUN IDLE TOTAL JOB_IDS
dteam050 ID: 97412 10/26 16:21 _ _ _ 1 97412.0

Total for query: 1 jobs; 1 completed, 0 removed, 0 idle, 0 running, 0 held, 0 suspended
Total for all users: 884 jobs; 412 completed, 0 removed, 200 idle, 259 running, 13 held, 0 suspended


# Checking the status of all jobs running on that
$ condor_q --name condorce1.ciemat.es --pool condorce1.ciemat.es:9619


-- Schedd: condorce1.ciemat.es : <192.101.161.188:9619?... @ 10/26/22 16:25:03
OWNER BATCH_NAME SUBMITTED DONE RUN IDLE TOTAL JOB_IDS
dteam050 ID: 97400 10/26 15:46 _ _ _ 1 97400.0
dteam050 ID: 97412 10/26 16:21 _ _ _ 1 97412.0

Total for query: 2 jobs; 2 completed, 0 removed, 0 idle, 0 running, 0 held, 0 suspended
Total for dteam050: 2 jobs; 2 completed, 0 removed, 0 idle, 0 running, 0 held, 0 suspended
Total for all users: 883 jobs; 411 completed, 0 removed, 200 idle, 259 running, 13 held, 0 suspended

It is also possible to view the output of a running job using condor_tail.

Download the job output to the UI using condor_transfer_data:

# Retrieving the output of a specific job
 $ condor_transfer_data -name condorce1.ciemat.es -pool condorce1.ciemat.es:9619 97412.0
Fetching data files...

References

To an ARC-CE Computing Element

ARC Compute Element (CE) is a Grid front-end on top of a conventional computing resource (e.g. a Linux cluster or a standalone workstation). ARC CE is sometimes also called ARC server.

While there is an official documentation on How to submit a job, the main steps will be documented below.

If you haven’t already generated a credential with voms-proxy-init, you can generate a proxy certificate using ARC’s own tool: arcproxy, which is using the same credentials as voms-proxy-init, and should produce an equivalent proxy. To do this you first need to prepare your X.509 credentials.

# Generating a proxy for ARC
$ arcproxy --voms dteam
Enter pass phrase for private key:
Your identity: /DC=org/DC=terena/DC=tcs/C=NL/O=Stichting EGI/CN=Jane Doe
Contacting VOMS server (named dteam): voms2.hellasgrid.gr on port: 15004
Proxy generation succeeded
Your proxy is valid until: 2022-10-27 02:23:52

Create testjob.xrsl, a test job expressed in xRSL, showing environment where it will run:

&( executable = "/usr/bin/env" )
( jobname = "arctest" )
( stdout = "stdout" )
( join = "yes" )
( gmlog = "gmlog" )

Then review the ARC CE information and send the job using arcsub:

# Getting info about the selected CE
# Example CE: alex4.nipne.ro:2811/nordugrid-SLURM-dteam
$ ldapsearch -x -LLL -H ldap://lcg-bdii.egi.eu:2170
 -b "Mds-Vo-Name=local,o=grid" \
 '(&(objectClass=GlueCE)(GlueCEUniqueID=alex4.nipne.ro:2811/nordugrid-SLURM-dteam))'
# Submitting the job, the JOB_ID will be written on the output
$ arcsub --jobdescrfile testjob.xrsl --computing-element alex4.nipne.ro
Job submitted with jobid: gsiftp://alex4.nipne.ro:2811/jobs/....
# Export JOB_ID to be used for other commands
$ JOB_ID="gsiftp://alex4.nipne.ro:2811/jobs/...."

Then you can use arcstat to monitor the job:

# Monitoring the status of the job
$ arcstat "$JOB_ID"

# Use -l parameter with arcstat to get more information on the status of the Job
$ arcstat -l "$JOB_ID"

The jobs will be in state Finished once completed.

You can finally retrieve the output of a finished job using arcget:

# Retrieve output files of the finished job, removing them from the server
$ arcget "$JOB_ID"

Results stored at: Ow7KmRKv71nuvw3Vp3UrRNqABFKDmABFKDmfJKDmABFKDmoeT5zn
Jobs processed: 1, successfully retrieved: 1, successfully cleaned: 1

Instead of manually selecting a site, it’s possible to do some automatic selection from CE registered in a central registry, such as nordugrid.org.

# Automatic selection on the CE in the nordugrid.org registry
$ arcsub --jobdescrfile testjob.xrsl --registry nordugrid.org
(...)
gsiftp://vm3.tier2.hep.manchester.ac.uk:2811/jobs/...
JOB_ID="gsiftp://vm3.tier2.hep.manchester.ac.uk:2811/jobs/.."

# Use -l parameter to get more information on the status of the Job
$ arcstat -l "$JOB_ID"

References

To a CREAM Computing Element

The CREAM (Computing Resource Execution And Management) Service is a simple, lightweight service that implements all the operations at the Computing Element (CE) level.

The first step is to prepare a JDL as testjob.jdl. The CREAM JDL Guide, documents the creation of the JDL:

[
Type = "Job";
JobType = "Normal";
Executable = "/usr/bin/env";
StdOutput = "output.txt";
StdError = "error.txt";
OutputSandbox = {"output.txt", "error.txt"};
]

Then you can submit, monitor and retrieve the output of the job.

Those commands are broken on the UI installed from UMD via our Ansible module, but are provided here as a reference, and for users having access to UI maintained by or for their community and providing the required commands.

# Submitting a job, job ID would be printed to the output
$ glite-ce-job-submit 'lpsc-cream-ce.in2p3.fr:8443/cream-pbs-dteam' testjob.jdl
# Use a variable with the job ID to be reused later
JOB_ID='...'
# Monitoring the job
$ glite-ce-job-status "$JOB_ID"
# Retrieving the output of the job
$ glite-ce-job-output "$JOB_ID"

References

CREAM User’s guide

Via the EGI Workload Manager

The EGI Workload Manager is a service provided to the EGI community to efficiently manage and distribute computing workloads on the EGI infrastructure.

Using the Workload Manger web interface or the DIRAC command-line interface (CLI) is documented in the EGI Workload Manager.

Troubleshooting

In case you receive errors when submitting jobs to Computing Elements, it may be possible that the service is in Downtime for an intervention/upgrade or there is an issue already reported by the EGI Monitoring System ARGO.

To check the information about downtimes or issues you can browse the ARGO Issues Page and, as shown in the figure below, check if there are active Downtimes for the service you are trying to use (By clicking on the Downtime button) or issues (By clicking on the CRITICAL button).

Asking for help

If you find issues please do not hesitate to contact us.

Users: Data transfer with object storage

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Warning

This procedure has been tested with the FTS client 3.11. Older version do not support all the options necessary. To install the latest version please add the FTS3 Production repository to your configuration and update the client

Prerequisites

To access services and resources in the EGI Federated Cloud, you will need:

An EGI Check-in account, you can sign up here
Enrolment into a Virtual Organisation (VO) that has access to the services and resources you need
An Object Storage for which you need to have all the credentials available (any S3 compatible storage should work)
Permission to add the Object Storage credential to the FTS server or alternatively for this operation you may contact support at egi.eu.

FTS client usage

Step 1 Configuration check and Proxy creation

For this two steps please refer to the “Data transfer with grid storage” tutorial.

Step 2 Find the storage

As for the “Data transfer with grid storage” tutorial you can look for the available storage on VAPOR service while the Object Store can be one created as described in the Object Storage section or trough a provider such as Amazon, Azure, etc

Step 3 Add the Object Storage credential to the FTS server

Following is an example of the command that can be used to add the Object Store credential to the FTS server. The fist step is to register the Object Storage. The name of the storage, is S3: + the domain part of the URL (for example https://s3.cl2.du.cesnet.cz -> S3:s3.cl2.du.cesnet.cz)

$ curl -E "${X509_USER_PROXY}" \
 --cacert "${X509_USER_PROXY}" \
 --capath "/etc/grid-security/certificates" \
 https://fts3devel01.cern.ch:8446/config/cloud_storage \
 -H "Content-Type: application/json" \
 -X POST \
 -d '{"storage_name":"S3:s3.cl2.du.cesnet.cz"}'

And then, add the keys, so the requests can be signed.

$ curl -E "${X509_USER_PROXY}" \
 --cacert "${X509_USER_PROXY}" \
 --capath "/etc/grid-security/certificates" \
 "https://fts3devel01.cern.ch:8446/config/cloud_storage/S3:s3.cl2.du.cesnet.cz" \
 -H "Content-Type: application/json" \
 -X POST \
 --data @config.json

Where config.json is a JSON file with the following content:

{
 "vo_name": "/dteam/Role=NULL/Capability=NULL",
 "access_key": "ACCESS_KEY",
 "secret_key": "SECRET_KEY"
}

Where ACCESS_KEY and SECRET KEY are the corresponding key necessary to access the object storage. See also the S3 Support pages on the FTS docs pages.

Step 4 Transfer between a grid storage and an object storage

For the grid storage to use please follow the details described in the section “Find the storage” of the “Data transfer with grid storage” tutorial. In the following examples, an object storage available in s3://s3.cl2.du.cesnet.cz/ with an already available bucket is used. To manage the object storage is possible to use any compatible tool. Following will be only shown an example of transfer.

$ fts-transfer-submit --s3alternate \
 -s https://fts3-public.cern.ch:8446 \
 https://dc2-grid-64.brunel.ac.uk/dpm/brunel.ac.uk/home/dteam/1M \
 s3://s3.cl2.du.cesnet.cz/bucket-name/1M.3
247b7ca2-4c4d-11ec-84d0-fa163e5dcbe0

The command returns a job ID that we can use to check the status of the transfer itself:

$ fts-transfer-status -d \
 -s https://fts3-public.cern.ch:8446 \
 247b7ca2-4c4d-11ec-84d0-fa163e5dcbe0
FINISHED

Step 5 Transfer between two object storage

We can also use the data transfer service to perform transfers between two object storage. In this case the transfer will be controlled by the FTS service you can use a command like:

$ fts-transfer-submit --s3alternate \
 -s https://fts3-public.cern.ch:8446 \
 s3://s3.cl2.du.cesnet.cz/bucket-name/1M.3 \
 s3://s3.cl2.du.cesnet.cz/bucket-name/A
c1d4a8e6-4c81-11ec-8926-fa163e5dcbe0

In this case too we can verify the status of the transfer with the same command as before using the new job ID.

$ fts-transfer-status -d \
 -s https://fts3-public.cern.ch:8446 \
 c1d4a8e6-4c81-11ec-8926-fa163e5dcbe0
FINISHED

Users: Data redundancy

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

Data redundancy in cloud environments is crucial to ensure high availability, data integrity, and disaster recovery. In federated infrastructures like EGI, redundancy mitigates risks such as hardware failures, accidental deletions, and data corruption. Without proper redundancy, critical data may be lost permanently or become inaccessible during infrastructure failures. Redundancy enables resilience against failures and provides consistent service availability in Kubernetes (k8s) and OpenStack deployments. Relying on multiple geographically distributed sites supporting redundancy is essential for use cases that require cross-site data protection, compliance, and disaster recovery strategies.

This guide covers two approaches to achieving redundancy:

Using Rsync and Snapshot Replication between two OpenStack installations.
Using MinIO for Object Storage, providing an S3-compatible alternative for data redundancy.

Both solutions provide redundancy but cater to different use cases. The OpenStack rsync method is best for VM failover, while MinIO offers flexible object storage redundancy.

For a blend of security and granular control, Restic’s encrypted, deduplicated backups offer a compelling alternative to solely relying on OpenStack or MinIO redundancy. https://github.com/restic/restic

Solution 1: Redundant OpenStack Setup

Using Rsync and Snapshot Replication

Overview

This solution synchronizes virtual machine (VM) snapshots between two OpenStack instances, ensuring high availability and data redundancy.

Prerequisites

A virtual organization with access to two OpenStack sites (Source and Destination sites)
A migration instance on both sites
SSH access between sites
OpenStack command-line tools installed
Sufficient storage capacity

Step 1: Configure SSH for Passwordless Authentication

On the source migration instance:

# If you don't have an SSH key pair already
ssh-keygen -t rsa

# Replace user@destination_migration_host with the actual username and hostname/IP
ssh-copy-id ${DESTINATION_USER}@${DESTINATION_HOST}

Test the connection:

ssh user@destination_host "hostname"

This should execute hostname on the destination migration instance without asking for a password.

Step 2: Manual Snapshot Replication (Testing the Process)

Steps for testing the process manually before automating it.

2.1 Define Variables (Source Site)

On the source migration instance:

# --- Configuration ---
INSTANCE_ID="12345678-90ab-cdef-1234-567890abcdef" # ID of the VM to snapshot
BASE_SNAPSHOT_NAME="instance_snapshot" # Base name for snapshots
LOCAL_OPENRC_PATH="$HOME/source_openrc" # Path to the OpenStack RC file for the source site

DESTINATION_USER="cloudadm" # SSH user on the destination migration instance
DESTINATION_HOST="destination_host" # Hostname or IP of the destination migration instance
DESTINATION_OPENRC_PATH="~/destination_openrc" # Path to OpenStack RC file on destination instance
REMOTE_TMP_DIR="/tmp" # Temporary directory on destination for the image

# --- Dynamic Variables ---
TODAYS_DATE=$(date +%Y-%m-%d)
SNAPSHOT_NAME_WITH_DATE="${BASE_SNAPSHOT_NAME}-${TODAYS_DATE}"
LOCAL_IMAGE_FILE="/tmp/${SNAPSHOT_NAME_WITH_DATE}.img"
REMOTE_IMAGE_FILE="${REMOTE_TMP_DIR}/${SNAPSHOT_NAME_WITH_DATE}.img"

2.2 Create a Snapshot on the Source

On the source migration instance:

echo "Sourcing OpenStack RC file for source: ${LOCAL_OPENRC_PATH}"
source "${LOCAL_OPENRC_PATH}"

echo "Creating snapshot '${SNAPSHOT_NAME_WITH_DATE}' for instance '${INSTANCE_ID}'..."
SNAPSHOT_ID=$(openstack server image create \
 --name "${SNAPSHOT_NAME_WITH_DATE}" \
 "${INSTANCE_ID}" \
 -f value -c id)

echo "Wait until the snapshot becomes active"
STATUS=$(openstack image show "${SNAPSHOT_ID}" -f value -c status)
while [ "${STATUS}" != "active" ]; do
 echo "Waiting for snapshot to become active..."
 sleep 10
 STATUS=$(openstack image show "${SNAPSHOT_ID}" -f value -c status)
done

if [ -z "$SNAPSHOT_ID" ]; then
 echo "Error: Failed to create snapshot. SNAPSHOT_ID is empty."
 exit 1
fi
echo "Snapshot created with ID: ${SNAPSHOT_ID}"

2.3 Transfer the Snapshot to the Destination

On the source migration instance:

echo "Saving snapshot image to '${LOCAL_IMAGE_FILE}'..."
openstack image save --file "${LOCAL_IMAGE_FILE}" "${SNAPSHOT_ID}"

echo "Transferring image file to ${DESTINATION_USER}@${DESTINATION_HOST}:${REMOTE_TMP_DIR}/ ..."
rsync -avz --progress "${LOCAL_IMAGE_FILE}" "${DESTINATION_USER}@${DESTINATION_HOST}:${REMOTE_TMP_DIR}/"

2.4 Import Snapshot on Destination

On the destination migration instance:

echo "Sourcing OpenStack RC file on destination: '${DESTINATION_OPENRC_PATH}'"
source "${DESTINATION_OPENRC_PATH}"

echo "Creating image from transferred file: '${REMOTE_IMAGE_FILE}'"
openstack image create \\
 --file "${REMOTE_IMAGE_FILE}" \\
 --disk-format qcow2 \\
 --container-format bare \\
 "${SNAPSHOT_NAME_WITH_DATE}"

Cleanup old snapshots on destination (optional)

ssh "${DESTINATION_USER}@${DESTINATION_HOST}" bash -c "'
MAX_KEEP=3
OLD_SNAPS=\$(openstack image list --name \"${BASE_SNAPSHOT_NAME}-*\" -f value -c ID -c Created | sort -k2 | head -n -\$MAX_KEEP | awk \"{print \$1}\")
for SNAP in \$OLD_SNAPS; do
 echo \"Deleting old snapshot \$SNAP on destination\"
 openstack image delete \"\$SNAP\"
done
"'

2.5 Cleanup (Source)

On the source migration instance:

echo "Cleaning up local image file: ${LOCAL_IMAGE_FILE}"
rm -f "${LOCAL_IMAGE_FILE}"
# --- Optional, cleanup old snapshots on source ---
MAX_KEEP=3 # keep only the last 3 snapshots
OLD_SNAPS=$(openstack image list --name "${BASE_SNAPSHOT_NAME}-*" -f value -c ID -c Created | sort -k2 | head -n -$MAX_KEEP | awk '{print $1}')

for SNAP in $OLD_SNAPS; do
 echo "Deleting old snapshot $SNAP"
 openstack image delete "$SNAP"
done

Step 3: Automate Snapshot Replication with a Script

On the source migration instance, create a script file to combine all the above steps, for example, ~/replicate_vm_snapshot.sh Ensure INSTANCE_ID and other critical variables are correctly set within the script itself or loaded from a config file.

Make the script executable:

chmod +x ~/replicate_vm_snapshot.sh

Test the script manually:

~/replicate_vm_snapshot.sh

Check the ~/snapshot_replication.log file for output.

Add the script to Cron. Edit the crontab on the source migration instance:

crontab -e

Add an entry to run the script daily at, for example, 2:00 AM:

# Redirect all output (stdout and stderr) from cron to the log file handled by the script, or to /dev/null if confident.
0 2 * * * /bin/bash /home/your_user/replicate_vm_snapshot.sh >> /home/your_user/snapshot_replication.log 2>&1

Replace /home/your_user/ with the actual path to the script and log file.

Results

By utilizing rsync and snapshot replication, VM state consistency is ensured between sites, improving failover readiness.

Solution 2: Redundant Object Storage

with MinIO

Overview

MinIO is an open-source, high-performance object storage solution compatible with Amazon S3. It enables redundancy and data replication across multiple sites, making it a good alternative for OpenStack Swift or AWS S3 in a private or federated cloud.

Deployment in FedCloud

MinIO is available in FedCloud with different deployment options:

Preconfigured Virtual Machines using Infrastructure Manager.
Helm chart deployment in Kubernetes, with Cloud Container Compute, enabling easy setup with just a few clicks.

Prerequisites

MinIO installed on both source and destination sites
Network connectivity between sites
Storage capacity for object replication

MinIO Configuration

To configure MinIO for redundancy, follow these steps:

Step 1: Access the MinIO Web Interface

Open your browser and navigate to https://<minio_host>:9001.
Log in with your MinIO credentials (admin username and password).

Step 2: Create Buckets

In the MinIO Console, go to the “Buckets” section.
Click “Create Bucket” and enter a bucket name.
Configure access settings (private/public as required).

Step 3: Configure Replication

Go to “Buckets” and select the bucket you created.
Click on the “Replication” tab.
Add a remote target:
- Enter the remote MinIO server address.
- Provide access credentials for the remote instance.
- Enable “Active Sync” to keep the data continuously updated.
Save the configuration.

Step 4: Verify Replication

Upload an object to the source MinIO instance and check if it appears in the destination MinIO bucket.

Alternate MinIO Configuration using Command-line

Step 1: Install and Configure MinIO

On each site:

docker run -d --name minio \
 -p 9000:9000 \
 -p 9001:9001 \
 -e "MINIO_ROOT_USER=admin" \
 -e "MINIO_ROOT_PASSWORD=password" \
 quay.io/minio/minio server /data --console-address ":9001"

Step 2: Set Up Site-to-Site Replication

2.1 Configure Replication on Source MinIO

mc alias set source http://source_host:9000 admin password
mc alias set destination http://destination_host:9000 admin password
mc replicate add source/bucket destination/bucket --remote-bucket destination/bucket --sync

2.2 Verify Replication

mc mirror --watch source/bucket destination/bucket

Documentation – Ad-hoc tutorials

Users: Create your first Virtual Machine (VM)

Overview

Step 1: Signing up

Step 2: Enrolling to a Virtual Organisation

Step 3: Creating a VM

Asking for help

Users: Accessing virtual machines with SSH

Overview

SSH Keys

Tip

SSH username

Local ssh key configuration

Username and password

Warning

Warning

OpenStack networking

Security Groups Rules

Private IP vs public IP

Accessing virtual machines in the private network

Example: ssh configuration for Jump host

SSH connection practical example

Connecting from a local machine to A

Connecting from a local machine to B

Infrastructure Manager (IM)

Token-based authentication

Users: Access DataHub from a VM

Overview

Prerequisites

Oneclient installation

Oneclient installation via packages

Oneclient installation via docker

Getting the token to access data

Using oneclient

Users: Create a VM with Jupyter and DataHub

Overview

Requirements

Create a VM with Jupyter and DataHub

Step 1: Start your VM

Step 2: Get a hostname and certificate for your VM

Enabling insecure access

Step 3: Get your token for the Jupyter server

Step 4: Start your notebooks

Step 5: Mount DataHub spaces

Users: Automate with oidc-agent, fedcloudclient, terraform and Ansible

Overview

Step 1: Signing up for an EGI Check-in account

Step 2: Enrolling to a Virtual Organisation

Step 3: Creating a VM

Setting up oidc-agent

Installing fedcloudclient and ansible

Identifying a suitable cloud site

Deploying the Virtual Machine with terraform

Setting up the environment

Describing the terraform variables

Identifying the cloud resources

Documenting the cloud resources for the selected site

Creating the main terraform deployment file

Initial configuration of the VM using cloud-init

Launching the terraform deployment

Debugging terraform

Destroying the resources created by terraform

Step 4: Using Ansible

Additional resources

Asking for help

Users: Data transfer with grid storage

Overview

Prerequisites

FTS client usage

Step 1 Configuration check

Step 2 Proxy creation

Step 3 Find the storage

Step 4 Starting a transfer

Users: Submitting HTC Jobs

Overview

Prerequisites

Step 1: getting access to a User Interface (UI)

Deploying an UI

Step 2: creating a VOMS proxy

Configuring the system to use voms-client

Connecting from a local machine to `A`

Connecting from a local machine to `B`