Documentation – Federated Cloud Compute

Users: Authentication and Authorisation

Mon, 01 Jan 0001 00:00:00 +0000

Authentication

OpenID Connect (OIDC) is the main authentication protocol used on the EGI Cloud. It replaces the legacy VOMS-based authentication for all OpenStack providers.

Authentication to web based services (like the OpenStack dashboards) will redirect you to the EGI Check-in authentication page. Just select your institution or social login and follow the regular authentication process.

Access to APIs or via command-line interfaces (CLI) requires the use of OAuth2.0 tokens and interaction with the OpenStack Keystone OS-FEDERATION API. The process for authentication is as follows:

Obtain a valid OAuth2.0 access token from Check-in. Access tokens are short-lived credentials that can be obtained by recognised Check-in clients once a user has been authenticated.
Interchange the Check-in access token for a valid unscoped Keystone token.
Discover available projects from Keystone using the unscoped token.
Use the unscoped Keystone token to get a scoped token for a valid project. Scoped tokens will allow the user to perform operations on the provider.

Authorisation

Cloud Compute service is accessed through Virtual Organisations (VOs). Users with specific roles (by default, the vm_operator role) will have access to the providers supporting that VO: they will be able to manage VMs, block storage and object storage available to the VO. Resources (VMs and storage) are shared across all members of the VO, please do not interfere with the VMs of other users if you are not entitled to do so (specially do not delete them).

Some users roles may have additional permissions in VOs, check your VO Manager for the specific documentation.

Pilot VO

The vo.access.egi.eu Virtual Organisation serves as a test ground for users to try the Cloud Compute service and to prototype and validate applications. It can be used for up to 6 month by any new user.

Warning

After the 6-month long membership in the vo.access.egi.eu VO, you will need to move to a production VO, or establish a new VO.
The resources are not guaranteed and may be removed without notice by providers. Back-up frequently to avoid losing your work!

For joining this VO, please click on the enrolment URL using your EGI account.

Other VOs

Pre-existing VOs of EGI can be also used on IaaS cloud providers. Consult with your VO manager or browse the existing VOs at the EGI Operations Portal.

Check-in and access tokens

Access tokens can be obtained via several mechanisms, usually involving the use of a web server and a browser. Command-line clients/APIs without access to a browser or interactive prompt for user authentication can use refresh tokens. A refresh token is a special token that is used to generate additional access tokens. This allows you to have short-lived access tokens without having to collect credentials every single time one expires. You can request this token alongside the access and/or ID tokens as part of a user’s initial authentication flow.

If you need to obtain these kinds of tokens for using it in command-line tools or APIs, you can easily do so with the EGI Check-in Token Portal. You can access the EGI Check-in Token Portal and click on 'Authorise' to log in with your Check-in credentials to obtain:

a client ID (token-portal)
a refresh token

Refresh tokens

Refresh tokens MUST be treated with care! This is a secret that can be used to impersonate you in the infrastructure. The life time of refresh tokens is up to one year! It is recommended not to store them in plain text.

There are more secure alternatives for handling refresh tokens:

From your personal computer: use oidc-agent It is a tool that manages your tokens locally in a secure way refresh tokens are even encrypted in RAM).
fedcloud client executed inside EGI Notebooks.
mytoken Securely stores refresh tokens on the mytoken-server. Users obtain mytokens, which can be used to obtain access tokens. mytokens are more secure, because they can carry capabilities and restrictions to fine tune their power for specific use cases. Details here: https://mytoken-docs.data.kit.edu

Discovering projects in Keystone

The access token will provide you access to a cloud provider, but you may have access to several different projects within that provider (a project can be considered equivalent to a VO allocation). In order to discover which projects are available you can do that using the Keystone API.

You can use the fedcloud client to simplify the discovery of projects.

# Get a list of sites
$ fedcloud site list
# Get list of projects that you are allowed to access
# You can either specify the name of the account in your oidc-agent configuration
# or directly a valid access token
$ fedcloud endpoint projects --site=<name of the site> \
 [--oidc-agent-account <account name>|--oidc-access-token <access token>]
# You can also use environment variables for the configuration
$ export OIDC_ACCESS_TOKEN=<your access token>
$ fedcloud endpoint projects
# or with oidc-agent
$ export OIDC_AGENT_ACCOUNT=<account name>
$ fedcloud endpoint projects

Using the OpenStack API

Once you know which project to use, you can use your regular openstack cli commands for performing actual operations in the provider:

$ fedcloud openstack image list --site <NAME_OF_SITE> --vo <NAME_OF_VO>

For third-party tools that can use token based authentication in OpenStack, use the following command:

$ export OS_TOKEN=$(fedcloud openstack --site <NAME_OF_SITE> --vo <NAME_OF_VO> \
 token issue -c id -f value)

Using ssh-oidc

ssh-oidc is a set of tools that allows ssh with OIDC.

Users: Virtual Machine Images

Mon, 01 Jan 0001 00:00:00 +0000

Users of the EGI Cloud create Virtual Machines (VMs) on the providers. Those VMs are started from images: templates for the root volume of the running instances, i.e. operating system and applications available initially on a VM. The Artefact Registry collects the Virtual Machine Images available on the providers.

Artefact Registry

The Artefact Registry is a OCI-compliant catalogue of artefacts (including container images, VM images and helm charts) based on Harbor where users can upload their images.

Images in the Artefact Registry are organised into projects where images are pushed to. Access to projects is granted based on the membership and role of users within a VO. For a given project, only users with the appropriate roles can perform certain operations.

EGI images

EGI produces a set of basic images that are automatically synced to all the providers and available for all the supported VOs.

These images are built automatically using packer and Ansible and uploaded to a dedicated project named egi_vm_images at the Artefact Registry. Build scripts, packer templates, and ansible playbooks are available in the fedcloud-vmi-templates GitHub repository. The images are built regularly to avoid any potential vulnerabilities.

EGI images have minimal OS installation with cloud-init for contextualization and follow the registry.egi.eu egi_vm_images/<image name>:<version> naming convention.

Custom VO images

VO project in Artefact Registry

The Artefact Registry can store VM and container images. Besides the egi_vm_images projects, VOs can request their project in the Artefact Registry for storing custom images that include software needed to support the VOs activities. For doing so, open Helpdesk ticket as follows:

Group: Artefact Registry (select Second Level/Services/EGI Services and Service Components/Artefact Registry)

Ticket body:

Dear Artefact Registry,

I'd like to request a new project for VO `<name of the VO>`.
The entitlement to authorize read-only (list, pull) access is `<entitlement to use>`.
The entitlement to authorize write (list, pull, push) access is `<entitlement to use>`.

Thanks!

Once the project is created, you will be able to login and manage artefacts.

Uploading to the registry

The Artefact Registry can store arbitrary binary artifacts and those matching the expected metadata will be synced to the sites. For uploading the images, any OCI registry compliant tool can be used. We rely on oras for uploading the EGI images.

Images must have a eu.egi.cloud.tag annotation in their manifest for them to be synced and the following additional annotations are expected to be available:

org.openstack.glance.disk_format: with the disk format of the image (e.g. raw or qcow2)
org.openstack.glance.container_format: with the glance container format (bare should be used in most cases), see glance documentation.
eu.egi.cloud.image.title: Human readable title.
eu.egi.cloud.description: A description of the image.
org.openstack.glance.architecture: Image architecture.
org.openstack.glance.os_distro: OS distro name (e.g. ubuntu).
org.openstack.glance.os_version: OS version (e.g. 24.04 for Ubuntu 24.04).
org.openstack.glance.os_type: OS type (e.g. linux).

For the upload, you need to:

Prepare an annotation file with the expected metadata. You can find below an example for an AlmaLinux image file named alma-9.qcow2. It also includes a $manifest entry with annotation of the manifest itself.

{
 "$manifest": {
 "org.opencontainers.image.revision": "7b98c834862f2e7e342fad7f9e175ea8c74aa4f3",
 "org.opencontainers.image.source": "https://github.com/EGI-Federation/fedcloud-vmi-templates"
 },
 "alma-9.qcow2": {
 "eu.egi.cloud.image.title": "EGI Alma 9 image",
 "org.openstack.glance.architecture": "x86_64",
 "org.openstack.glance.os_distro": "alma",
 "org.openstack.glance.os_type": "linux",
 "org.openstack.glance.os_version": "9",
 "org.openstack.glance.disk_format": "qcow2",
 "eu.egi.cloud.tag": "2025-06-10-7b98c834",
 "org.openstack.glance.container_format": "bare"
 }
}

Upload the image and the annotation file:

oras push --annotation-file <annotation json> \
 registry.egi.eu/<project_name>/<repository>:<tag> \
 <image file>

Building your own image

Packaging your application in a custom VM image is a suggested solution in one of the following cases:

your particular OS flavor is not available;
installation of your application is very complex and time-consuming for being performed during contextualization; or
you want to reduce the number of 'moving-parts' of your software stack and follow an immutable infrastructure approach for deploying your application.

Custom VM images can be crafted in different ways. The two main possibilities are:

start from scratch, creating a virtual machine, installing an OS and the software on top of it, then taking the virtual machine OS disk as custom image; or
dump an existing disk from a running VM or physical server and modify it, if needed, to run on a virtualisation platform.

In this guide we will focus on the first option, because it tends to produce cleaner images and reduces the risks of hardware conflicts. Snapshotting may be also restricted by the cloud providers or by security policies.

Advantages:

Possibility to build the virtual disk directly from a legacy machine, dumping the contents of the disk.
Possibility to speed-up the deployment for applications with complex and big installation packages. This because you do not need to install the application at startup, but the application is already included in the machine.

Disadvantages:

Building a virtual disk directly from a legacy machine poses a set of compatibility issues with hardware drivers, which usually differs from a virtual and physical environment and even between different virtual environments.
You need to keep your machine updated. Outdated VM disk images may take a long time to startup due to the need to download and install the latest OS updates.
If you are using special drivers or you are not packaging correctly the disk, your custom VM image may not run (or run slowly) on different cloud providers based on different virtualisation technologies.
VM images on public clouds are sometimes public, thus be aware of installing proprietary software on custom images, since other users may be able to run the image or download it.
In general, the effort to implement this solution is higher than the basic contextualization.

Image size and layout

The larger the VM image, the longer it will take to be distributed to the providers and the longer it will take to be started on the infrastructure. As a general rule, always try to make images as smaller as possible following these guidelines:

DO NOT include (big) data in your image. There are other mechanisms for accessing data from your VM (block/ object storage, CVMFS)
DO NOT include (big) empty space or swap in your image. Extra space for your computation or swap can be added with block storage once the VM is booted or using VM flavors that have extra disk allocated for your VM.
DO NOT install un-needed software. Tools like GUI are of no-use in most cases since you will have no access to the graphical console of the VM.
DO adjust the size of the images as much as possible. As stated above, empty space can be allocated on runtime easily.
DO use compressed image formats, like qcow2 or vmdk (used in OVA) to minimize the size of the image. Preferred format for images in EGI is OVA as it's standardised.
DO fill with 0 the empty disk space of your image so when compressed it can be significantly reduced, e.g. using:
```
dd if=/dev/zero of=/bigemptyfile bs=4096k
rm -rf /bigemptyfile
```
DO use a single partition (no /boot, no swap) for the disk layout and avoid LVM. This will allow the cloud provider to easily resize your partition when instantiated and to modify files in it if needed.

Contextualization and credentials

Danger

Do NOT include any credentials on your images.

You should never include any kind of credentials on your images, instead you should use contextualization. cloud-init is a tool that will simplify the contextualization process for you. This is widely available as packages in major OS distributions and is supported by all the providers of the EGI Cloud and most of the commercial providers.

cloud-init documentation contains detailed examples on how to create users, run scripts, install packages and several other actions supported by the tool.

For complex setups, especially when applications involve multiple VMs it may be useful to use cloud-init to bootstrap some Configuration Management Software that will manage the configuration of the VMs during runtime.

Security

Always remove all default passwords and accounts from your VM.
Disable all services unless necessary for the intended tasks.
Make sure the firewall configuration (iptables for Linux, also on IPv6) is minimally open.
Put no shared credentials (passwords) in any image.

You should also follow best practice guides for each service that’s exposed to the outside world. See also AWS security Best Practices

Tools

Whenever possible, automate the process of creating your images. This will allow you to:

Get reproducible results
Avoid tedious manual installation steps
Quickly produce updated versions of your images

Check out the fedcloud-vmi-templates GitHub repository for examples of images that can be built in a completely automated workflow using packer and GitHub Actions.

Users: OpenStack in EGI Cloud

Mon, 01 Jan 0001 00:00:00 +0000

OpenStack providers of the EGI Cloud Compute service offer native OpenStack features via native APIs integrated with EGI Check-in accounts.

The extensive OpenStack user documentation includes details on every OpenStack project most providers offer access to:

Keystone, for identity
Nova, for VM management
Glance, for VM image management
Cinder, for block storage
Swift, for object storage
Neutron, for network management
Horizon, as a web dashboard

The Horizon Web-dashboard of the OpenStack providers can be accessed using your EGI Check-in credentials directly. See the Getting Started guide for more information. The rest of this guide will focus on CLI/API access.

Installation

The OpenStack client is a command-line client for OpenStack that brings the command set for Compute, Identity, Image, Object Storage and Block Storage APIs together in a single shell with a uniform command structure.

Linux / Mac
Windows

Installation of the OpenStack client can be done using:

pip install openstackclient

As there are non-pure Python packages needed for installation, the Microsoft C++ Build Tools is a prerequisite, please make sure it’s installed with the following options selected:

C++ CMake tools for Windows
C++ ATL for latest v142 build tools (x86 & x64)
Testing tools core features - Build Tools
Windows 10 SDK (<latest>)

In case you prefer to use non-Microsoft alternatives for building non-pure packages, please see here.

Installation of the OpenStack client can be done using:

pip install openstackclient

Add IGTF CA to python's CA store:

cat /etc/grid-security/certificates/*.pem >> $(python -m requests.certs)

Authentication

Check the documentation at the authentication and authorisation section on how to get the right credentials for accessing the providers.

OpenStack token for other clients

Most OpenStack clients allow authentication with tokens, so you can easily use them with EGI Cloud providers just doing a first step for obtaining the token. With the OpenStack client you can use the following command to set the OS_TOKEN variable with the needed token:

$ OS_TOKEN=$(openstack --os-auth-type v3oidcaccesstoken \
 --os-protocol openid --os-identity-provider egi.eu \
 --os-auth-url <keystone url> \
 --os-access-token <your access token> \
 --os-project-id <your project id> token issue -c id -f value)

You can easily obtain an OpenStack token with the FedCloud client:

fedcloud openstack --site <NAME_OF_THE_SITE> --vo <NAME_OF_VO> token issue -c id -f value

Useful commands with OpenStack CLI

Usage of the OpenStack client is described in detail here.

Please refer to the nova documentation for a complete guide on the VM management features of OpenStack. We list in the sections below some useful commands for the EGI Cloud.

Registering an existing ssh key

It’s possible to register an ssh key that can later be used as the default ssh key for the default user of the VM (via the --key-name argument to openstack server create):

openstack keypair create --public-key ~/.ssh/id_rsa.pub mykey

Creating a VM

openstack flavor list
FLAVOR=<FLAVOR_NAME>
openstack image list
IMAGE_ID=<IMAGE_ID>
openstack network list
# Pick FedCloud network
NETWORK_ID=<NETWORK_ID>
openstack security group list
openstack server create --flavor $FLAVOR --image $IMAGE_ID \
 --nic net-id=$NETWORK_ID --security-group default \
 --key-name mykey oneprovider
# Creating a floating IP
openstack floating ip create <NETWORK_NAME>
# Assigning floating IP to server
openstack server add floating ip <SERVER_ID> <IP>
# Removing floating IP from server
openstack server show <SERVER_ID>
# Deleting server
openstack server remove floating ip <SERVER_ID> <IP>
openstack server delete <SERVER_ID>
# Deleting floating IP
openstack floating ip delete <IP>

Using cloud-init

openstack server create --flavor m3.medium \
 --image d0a89aa8-9644-408d-a023-4dcc1148ca01 \
 --user-data userdata.txt --key-name My_Key server01.example.com

Shell script data as user data

#!/bin/sh
adduser --disabled-password --gecos "" clouduser

cloud-config data as user data

#cloud-config
hostname: mynode
fqdn: mynode.example.com
manage_etc_hosts: true

Creating a snapshot image from running VM

You can create a new image from a snapshot of an existing VM that will allow you to easily recover a previous version of your VM or to use it as a template to clone a given VM.

openstack server image create <your VM> --name <name of the snapshot>

Once the snapshot is ready openstack image show <name of the snapshot> will give your the details you can use it as any other image at the provider:

openstack server create --flavor <flavor> \
 --image <name of the snapshot> \
 <name of the new VM>

You can override files in the snapshot if needed, e.g. changing the SSH keys:

openstack server create --flavor <flavor> \
 --image <name of the snapshot> \
 --file /home/ubuntu/.ssh/authorized_keys=my_new_keys \
 <name of the new VM>

libcloud

Apache libcloud supports OpenStack and EGI authentication mechanisms by setting the ex_force_auth_version to 3.x_oidc_access_token. Check the libcloud docs on connecting to OpenStack for details. See below sample code:

import requests

from libcloud.compute.types import Provider
from libcloud.compute.providers import get_driver

refresh_data = {
 'client_id': '<your client_id>',
 'client_secret': '<your client_secret>',
 'grant_type': 'refresh_token',
 'refresh_token': '<your refresh_token>',
 'scope': 'openid email profile',
}

r = requests.post("https://aai.egi.eu/auth/realms/egi/protocol/openid-connect/token",
 auth=(client_id, client_secret),
 data=refresh_data)

access_token = r.json()['access_token']

OpenStack = get_driver(Provider.OPENSTACK)
# first parameter is the identity provider: "egi.eu"
# Second parameter is the access_token
# The protocol 'openid' is specified in ex_tenant_name
# and tenant/project cannot be selected :(
driver = OpenStack('egi.eu', access_token, ex_tenant_name='openid',
 ex_force_auth_url='https://keystone_url:5000',
 ex_force_auth_version='3.x_oidc_access_token')

Users: Block Storage

Mon, 01 Jan 0001 00:00:00 +0000

What is it?

Block storage provides block-level storage volumes for use within virtual machines (VMs). Block storage volumes are raw, unformatted block devices, which can be mounted as devices in VMs.

Block storage volumes that are attached to a VM are exposed as storage volumes that persist independently from the life of the VM, and need to be explicitly destroyed when data is not needed any more. Users can create a file system on top of these volumes, or use them in any way you would use a block device (such as a hard drive).

The content of block storage volumes can be accessed only from within the VM they are mounted to, and they can be mounted to a single VM at any given time.

The main features of block storage:

Block storage is recommended for data that must be quickly accessible and requires long-term persistence.
Block storage volumes are well suited to both database-style applications that rely on random reads and writes, and to throughput-intensive applications that perform long, continuous reads and writes.
Users can create point-in-time snapshots of block storage volumes, which protect data for long-term durability, and they can be used as the starting point for new block storage volumes.

Note

Block storage volumes can only be mounted to VMs running at the same provider where the block storage is located.

Note

Block storage usage is accounted for the entire block storage device, regardless how much of it is actually used.

Important

There is a limit on the number of block storage devices you can attach on a VM and there is a limit to the maximum size of such virtual disks. These values will depend on the particular provider and your SLA.

Manage volumes

The block storage in the EGI Cloud is offered via OpenStack deployments that implement the Cinder service.

Users can manage block storage using the OpenStack Horizon dashboard of a provider, from a command-line interface (CLI), or via the OpenStack Block Storage API.

Manage from the command-line

Multiple command-line interfaces (CLIs) are available to manage block storage:

The OpenStack CLI
The FedCloud Client is a high-level CLI for interaction with the EGI Federated Cloud (recommended)
The Cinder CLI has some advanced features and administrative commands that are not available through the OpenStack CLI

The main FedCloud commands for managing volumes are detailed below.

Note

For more information see the documentation about volume management.

List volumes

For example, to list the volumes in the site IN2P3-IRES via the Pilot VO (vo.access.egi.eu), use the following FedCloud command:

To avoid passing the site, VO, etc. each time, you can use FedCloud CLI environment variables to set them once and reuse them with each command invocation.

$ export EGI_SITE=IN2P3-IRES
$ export EGI_VO=vo.access.egi.eu
$ fedcloud openstack volume list --site $EGI_SITE
Site: IN2P3-IRES, VO: vo.access.egi.eu, command: volume list
+---------------------------+--------+-----------+------+--------------------------------+
| ID | Name | Status | Size | Attached to |
+---------------------------+--------+-----------+------+--------------------------------+
| aa711296-5cff-46ac-bbe... | Matlab | in-use | 50 | Attached to Moodle on /dev/vdb |
| b0abc762-a503-129d-3c1... | | available | 30 | |
+---------------------------+--------+-----------+------+--------------------------------+

To avoid passing the site, VO, etc. each time, you can use FedCloud CLI environment variables to set them once and reuse them with each command invocation.

> set EGI_SITE=IN2P3-IRES
> set EGI_VO=vo.access.egi.eu
> fedcloud openstack volume list --site %EGI_SITE%
Site: IN2P3-IRES, VO: vo.access.egi.eu, command: volume list
+---------------------------+--------+-----------+------+--------------------------------+
| ID | Name | Status | Size | Attached to |
+---------------------------+--------+-----------+------+--------------------------------+
| aa711296-5cff-46ac-bbe... | Matlab | in-use | 50 | Attached to Moodle on /dev/vdb |
| b0abc762-a503-129d-3c1... | | available | 30 | |
+---------------------------+--------+-----------+------+--------------------------------+

To avoid passing the site, VO, etc. each time, you can use FedCloud CLI environment variables to set them once and reuse them with each command invocation.

> $Env:EGI_SITE="IN2P3-IRES"
> $Env:EGI_VO="vo.access.egi.eu"
> fedcloud openstack volume list --site $Env:EGI_SITE
Site: IN2P3-IRES, VO: vo.access.egi.eu, command: volume list
+---------------------------+--------+-----------+------+--------------------------------+
| ID | Name | Status | Size | Attached to |
+---------------------------+--------+-----------+------+--------------------------------+
| aa711296-5cff-46ac-bbe... | Matlab | in-use | 50 | Attached to Moodle on /dev/vdb |
| b0abc762-a503-129d-3c1... | | available | 30 | |
+---------------------------+--------+-----------+------+--------------------------------+

Create volume

To create a new volume use the FedCloud command below:

$ fedcloud openstack volume create --size 10 my-volume
Site: IN2P3-IRES, VO: vo.access.egi.eu, command: volume create --size 10 my-volume
+---------------------+------------------------------------------------------------------+
| Field | Value |
+---------------------+------------------------------------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2021-08-05T13:10:42.000000 |
| description | None |
| encrypted | False |
| id | a711296-5cff-46ac-bbe3-58e00712ee3e |
| multiattach | False |
| name | my-volume |
| properties | |
| replication_status | None |
| size | 10 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| type | ceph |
| updated_at | None |
| user_id | 1a3ef4b64714f86ac71f1c9512345678c157a94ae1b37f167b6a663baa3b915b |
+---------------------+------------------------------------------------------------------+

The status of the new volume will probably be returned as creating. To check if the volume finished creating, look at the details of the volume, or list only the newly created volume (filter by volume name or ID):

$ fedcloud openstack volume list --name my-volume
Site: IN2P3-IRES, VO: vo.access.egi.eu, command: volume list --name my-volume
+--------------------------------------+-----------+-----------+------+-------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+-----------+-----------+------+-------------+
| aa711296-5cff-46ac-bbe3-58e00712ee3e | my-volume | available | 10 | |
+--------------------------------------+-----------+-----------+------+-------------+

When the status of the volume is available the volume is ready to be attached to a VM.

See volume details

To view details of a volume use the FedCloud command below:

Tip

The volume can be specified either by its ID or by its name (if it has one).

$ fedcloud openstack volume show Matlab
Site: IN2P3-IRES, VO: vo.access.egi.eu, command: volume show Matlab
+------------------------------+------------------------------------------------------------+
| Field | Value |
+------------------------------+------------------------------------------------------------+
| attachments | [{'server_id': 'a4ab00a1-d458-41a9-a091-ee707bc9357e', |
| | 'attachment_id': '291111d3-f43c-494c-b64c-5c0abcda3d37', |
| | 'attached_at': '2021-08-05T08:50:19.000000', |
| | 'host_name': 'sbgcsrv17.in2p3.fr', |
| | 'volume_id': '9a4000fb-0bcc-47e8-96fb-85a222295402', |
| | 'device': '/dev/vdb', |
| | 'id': '9a4000fb-0bcc-47e8-96fb-85a222295402'}] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2021-08-05T08:48:41.000000 |
| description | |
| encrypted | False |
| id | 9a4000fb-0bcc-47e8-96fb-85a222295402 |
| multiattach | False |
| name | Matlab |
| os-vol-tenant-attr:tenant_id | 7a910xxxxae74ed9yyyy7497zzzz9499 |
| properties | |
| replication_status | None |
| size | 50 |
| snapshot_id | None |
| source_volid | None |
| status | in-use |
| type | ceph |
| updated_at | 2021-08-05T08:50:20.000000 |
| user_id | babzz8c3b4cxxxx6286dacyyyy01e5a3 |
+------------------------------+------------------------------------------------------------+

Attach volume to VM

Mapping block devices to VMs is described in detail in the OpenStack documentation.

To attach a volume to a VM use the FedCloud command below:

Note

To be able to attach a volume to a VM, the volume must not be attached to any VM (volume status must be available).

Caution

The optional --device argument to specify the device name in the VM should not be used. It does not work properly, and will be removed in the near future.

$ fedcloud openstack server add volume my-server my-volume
Site: IN2P3-IRES, VO: vo.access.egi.eu, command: server add volume my-server my-volume

You can check that the volume got attached to the VM (and with what device name) by either looking at the details of the volume, the details of the VM, or by listing only the volume in question (filter by volume name or ID):

$ fedcloud openstack volume list --name my-volume
Site: IN2P3-IRES, VO: vo.access.egi.eu, command: volume list --name my-volume
+--------------------------------------+-----------+-----------+------+-----------------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+-----------+-----------+------+-----------------------+
| aa711296-5cff-46ac-bbe3-58e00712ee3e | my-volume | in-use | 10 | my-server on /dev/vdc |
+--------------------------------------+-----------+-----------+------+-----------------------+

When the volume status is in-use the volume is attached to a VM, and it can be used from the VM.

Detach volume from VM

To detach a volume from a VM use the FedCloud command below:

$ fedcloud openstack server remove volume my-server my-volume
Site: IN2P3-IRES, VO: vo.access.egi.eu, command: server remove volume my-server my-volume

You can check that the volume got detached by either looking at the details of the volume, the details of the VM, or by listing only the volume in question (filter by volume name or ID):

$ fedcloud openstack volume list --name my-volume
Site: IN2P3-IRES, VO: vo.access.egi.eu, command: volume list --name my-volume
+--------------------------------------+-----------+-----------+------+-------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+-----------+-----------+------+-------------+
| aa711296-5cff-46ac-bbe3-58e00712ee3e | my-volume | available | 10 | |
+--------------------------------------+-----------+-----------+------+-------------+

When the volume status is available the volume is not attached to any VM.

Resize volume

To resize a volume set its size property to the desired size. For example, if there is a volume named my-volume with a size of 10GB, it can be resized using the FedCloud command below:

Tip

Other volume properties can be altered in the same way.

Note

To be able to resize a volume, the volume must not be attached to any VM (volume status must be available), unless the volume driver supports in-use extend.

$ fedcloud openstack volume set --size 20 my-volume
Site: IN2P3-IRES, VO: vo.access.egi.eu, command: volume set --size 20 my-volume

You can check that the volume got resized by either looking at the details of the volume, or by listing only the volume in question (filter by volume name or ID):

$ fedcloud openstack volume list --name my-volume
Site: IN2P3-IRES, VO: vo.access.egi.eu, command: volume list --name my-volume
+--------------------------------------+-----------+-----------+------+-------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+-----------+-----------+------+-------------+
| aa711296-5cff-46ac-bbe3-58e00712ee3e | my-volume | available | 20 | |
+--------------------------------------+-----------+-----------+------+-------------+

Snapshot a volume

Users can create snapshots of a volume that can later be used to create other volumes or to rollback to a precedent point in time. Volume snapshots are pointers in the read-write history of a volume.

To take a snapshot of a volume, use the FedCloud command below:

Tip

See also the documentation about the other snapshot management commands.

Note

To be able to create a snapshot of a volume, the volume must not be attached to any VM (volume status must be available). To create a snapshot while the volume is attached to a VM, use the --force command flag, but be aware that there may be inconsistencies if the VM’s OS is not aware of the snapshot being taken.

$ fedcloud openstack volume snapshot create --volume my-volume my-snapshot --force
Site: IN2P3-IRES, VO: vo.access.egi.eu, command: volume snapshot create --volume my-volume my-snapshot
+-------------+--------------------------------------+
| Field | Value |
+-------------+--------------------------------------+
| created_at | 2021-08-06T16:08:37.631226 |
| description | None |
| id | 15149b42-032f-4cec-b6f3-41aa5c958081 |
| name | my-snapshot |
| properties | |
| size | 10 |
| status | creating |
| updated_at | None |
| volume_id | aa711296-5cff-46ac-bbe3-58e0712ee3e9 |
+-------------+--------------------------------------+

The status of the snapshot will probably be returned as creating. To check if the snapshot is ready, look at the details of the snapshot, or list only the newly created snapshot (filter by snapshot name or ID):

$ fedcloud openstack volume snapshot list --name my-snapshot
Site: IN2P3-IRES, VO: vo.access.egi.eu, command: volume snapshot list --name my-snapshot
+--------------------------------------+-------------+-------------+-----------+------+
| ID | Name | Description | Status | Size |
+--------------------------------------+-------------+-------------+-----------+------+
| 15149b42-032f-4cec-b6f3-41aa5c958081 | my-snapshot | None | available | 10 |
+--------------------------------------+-------------+-------------+-----------+------+

When the status of the snapshot is available the snapshot is ready, and can be used to create new volumes or to restore the source volume to the point-in-time when the snapshot was taken.

Backup a volume

Users can also create backups of a volume, but backups can only be used later to create or replace other volumes. A volume backup is a copy of a volume saved to cold storage, which is cheaper than the performant storage used for volumes and snapshots.

To make a backup of a volume, use the FedCloud command below:

Tip

See also the documentation of the other backup management commands.

Note

Not all OpenStack deployments support volume backups.

Note

Backups use as much storage as the source volume, albeit in a cheaper storage layer. This means backups take a long time to create (~4h for 500GB), and the space is accounted for the same way as for regular volumes (the entire size of the backed up volume, regardless of how much of the volume space is actually used). Thus backups should be deleted when no longer needed.

Note

To be able to make a backup of a volume, the volume must not be attached to any VM (volume status must be available). To make a backup while the volume is attached to a VM, use the --force command flag, but be aware that there may be inconsistencies if the VM’s OS is not aware of the backup being taken.

$ fedcloud openstack volume backup create --name my-backup my-volume --force
Site: IN2P3-IRES, VO: vo.access.egi.eu, command: volume backup create --name my-backup my-volume
+-------------+--------------------------------------+
| Field | Value |
+-------------+--------------------------------------+
| created_at | 2021-08-06T16:08:37.631226 |
| description | None |
| id | 158bcb42-032f-4cec-b6f3-890a5c958081 |
| name | my-backup |
| properties | |
| size | 10 |
| status | creating |
| updated_at | None |
| volume_id | aa711296-5cff-46ac-bbe3-58e0712ee3e9 |
+-------------+--------------------------------------+

The status of the backup will probably be returned as creating. To check if the backup has finished, look at the details of the backup, or list only the newly created backup (filter by backup name or ID):

$ fedcloud openstack volume backup list --name my-backup
Site: IN2P3-IRES, VO: vo.access.egi.eu, command: volume backup list --name my-backup
+--------------------------------------+-----------+-------------+-----------+------+
| ID | Name | Description | Status | Size |
+--------------------------------------+-----------+-------------+-----------+------+
| 158bcb42-032f-4cec-b6f3-890a5c958081 | my-backup | None | available | 10 |
+--------------------------------------+-----------+-------------+-----------+------+

When the status of the backup is available the backup operation is complete.

Delete volume

To delete a volume use the following FedCloud command:

Note

To be able to delete a volume, the volume must not be attached to any VM (volume status must be available).

$ fedcloud openstack volume delete my-volume
Site: IN2P3-IRES, VO: vo.access.egi.eu, command: volume delete my-volume

This starts deletion of the specified volume (the volume status changes to deleting):

$ fedcloud openstack volume list --name my-volume
Site: IN2P3-IRES, VO: vo.access.egi.eu, command: volume list --name my-volume
+--------------------------------------+-----------+----------+------+-------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+-----------+----------+------+-------------+
| aa711296-5cff-46ac-bbe3-58e00712ee3e | my-volume | deleting | 10 | |
+--------------------------------------+-----------+----------+------+-------------+

Once deletion of the specified volume is complete, it will no longer show up in the list of volumes.

Access from your VMs

Block storage volumes attached to a VM will appear as a block device in the VM.

To find out the device name that got assigned to a volume when it was attached to a VM, look at the details of the volume, or list only the volume in question (filter by volume name or ID):

$ fedcloud openstack volume list --name my-volume
Site: IN2P3-IRES, VO: vo.access.egi.eu, command: volume list --name my-volume
+--------------------------------------+-----------+--------+------+-----------------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+-----------+--------+------+-----------------------+
| aa711296-5cff-46ac-bbe3-58e00712ee3e | my-volume | in-use | 10 | my-server on /dev/vdb |
+--------------------------------------+-----------+--------+------+-----------------------+

In this example the device name is /dev/vdb. To validate this, run the following command in the VM:

$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 252:0 0 20G 0 disk
└─vda1 252:1 0 20G 0 part /
vdb 252:16 0 10G 0 disk

Usually these devices are empty upon creation. The first time you attach them to a VM, you will need to partition the device and create filesystem(s) on it, by running the following command in the VM:

Tip

Using a file system volume label is useful to avoid the need to find the out the device name, especially when multiple block storage volumes are attached to a VM. It is recommended to use a file system volume label in the VM that is the same as the name of the block storage volume.

Tip

The filesystem type can be one supported by the Linux distribution in the VM, but xfs and ext4 are the most widely used.

Caution

Only run this command the first time you use the device, as it deletes all data! Make sure you use the correct device name, otherwise you will destroy data on other devices!

$ sudo mkfs.ext4 -L my-volume /dev/vdb

Once you created a filesystem on the device, you can mount it at any desired path by running the following command in the VM:

$ sudo mount /dev/vdb1 /<path>

Continuing with the example above, if we check again the block devices by running the following command in the VM:

$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 252:0 0 20G 0 disk
└─vda1 252:1 0 20G 0 part /
vdb 252:16 0 10G 0 disk
└─vdb1 252:1 0 10G 0 part /<path>

If non-root users should be able to access the mounted volume similar to the way e.g. /tmp is accessible, set the sticky bit on the mount point, with this command in the VM:

$ sudo chmod +t /<path>

If the desired behaviour is to mount the file system automatically on VM restart, add it to /etc/fstab. Using the LABEL parameter will ensure the correct volume is chosen if multiple volumes are attached:

LABEL=my-volume /<path> ext4 noatime,nodiratime,user_xattr,nofail 0 0

Note

The use of option nofail is recommended in order to skip (and not block on) mounting the file system volume if it is unavailable, e.g. in case of network issues. Remove this option from the fstab line if you want the VM to block the boot process if the volume is unavailable.

Note

The use of option nobarrier is not recommended as volumes are accessed via a cache, and ignoring the correct ordering of journal commits may result in a corrupted file system in case of a hardware problem.

With that you can access /<path> inside the VM, where all data stored on the volume will be available. Applications will not see any difference between a block storage device and a regular disk, thus no major changes should be required in the application logic.

Storage Encryption

This section describes the usage of the tool cryptsetup to enable the permanent encryption of the data stored in the disk. The tool is available in standard linux distributions, and for this guide we assume the installation in an Ubuntu distribution.

$ sudo su -
$ apt -y install cryptsetup

To encrypt the disk, it must be first initialized correctly. In the example below, the disk named /dev/vdb is first filled with random data and then initialized using the cryptsetup luksFormat command below. This first step can be quite long.

$ dd if=/dev/urandom of=/dev/vdb bs=4k
$ cryptsetup -v --cipher aes-xts-plain64 --key-size 512 --hash sha512 \
 --iter-time 5000 --use-random luksFormat /dev/vdb

If this last command slows down or even blocks with the following message:

System is out of entropy while generating volume key.
Please move mouse or type some text in another window to gather some random events.
Generating key (0% done).

you can make the cryptsetup luksFormat command running faster by first installing the haveged program in your virtual machine.

The following command verifies that the disk is now of type LUKS:

$ cryptsetup luksDump /dev/vdb
LUKS header information for /dev/vdb

Version: 1
Cipher name: aes
Cipher mode: xts-plain64
Hash spec: sha512
Payload offset: 4096
MK bits: 512
MK digest: c4 f7 4b 02 2a 3f 12 c1 2c ba e5 c9 d2 45 9a cd 89 20 6c 73
MK salt: 98 58 3e f3 f6 88 99 ea 2a f3 cf 71 a0 0d e5 8b
 d5 76 64 cb d2 5c 9b d1 8a d3 1d 18 0e 04 7a eb
MK iterations: 81250
UUID: c216d954-199e-4eab-a167-a3587bd41cb3

Key Slot 0: ENABLED
 Iterations: 323227
 Salt: a0 45 3e 98 fa cf 60 74 c6 09 3d 54 97 89 be 65
 5b 96 7c 1c 39 26 47 b4 8b 0e c1 3a c9 94 83 c2
 Key material offset: 8
 AF stripes: 4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

The disk is now ready for use. The first time you use it, you must perform the following steps:

Step 1: Open the encrypted disk with the cryptsetup luksOpen command. The name storage1 is only indicative, you can choose what you want:

$ cryptsetup luksOpen /dev/vdb storage1

Step 2: Create a filesystem on disk:

$ mkfs.ext4 /dev/mapper/storage1

Step 3: Create the disk mount point:

$ mkdir /storage1

Step 4: Mount the disk:

$ mount -t ext4 /dev/mapper/storage1 /storage1

Step 5: Check available space (this may be slightly different from what was entered during the openstack volume create command):

$ df -h /storage1
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/storage1 2.0G 6.0M 1.9G 1% /storage1

Once the disk is operational, steps 2 and 3 are no longer necessary.

You can now send files (for example DATA.dat) from your personal computer to your virtual machine in a secure way, for example with scp:

$ scp -i ${HOME}/.ssh/cloudkey DATA.dat ubuntu@134.158.151.224:/storage1
DATA.dat 100% 82 0.1KB/s 00:00

When you are done with your work on the disk, you can remove it cleanly with the following commands:

$ umount /storage1
$ cryptsetup close storage1

For subsequent uses of the persistent disk, there will be no need to perform all these operations, only the following are necessary:

$ cryptsetup luksOpen /dev/vdb storage1
$ mkdir -p /storage1
$ mount -t ext4 /dev/mapper/storage1 /storage1

Note that directory /storage1 will be created only if it does not already exist.

Access via EGI Data Transfer

EGI Data Transfer allows you to move any type of data files asynchronously from one storage to another. If you want to copy data from/to one VM running on the EGI cloud, you will need to run a compatible server (Webdav/HTTPS, GridFTP, xrootd, SRM, S3, GCloud) that can interact with the FTS3 software.

An easy way to provide a GridFTP server on your VM is to use the gridftp-le ready2go docker stack for deploying a GridFTP Docker Container with certificates from Let’s Encrypt. Take into account:

Security groups for the VM must allow ports 80, 2811 and the 50000-50200 range.
The VM must have a valid DNS entry (you can use Dynamic DNS in FedCloud to get one)
The default setup uses /srv as path to expose and maps users to the nobody user. Make sure that nobody is able to read (and write if needed) on that location or set the mapping to the appropriate users.
The Let’s Encrypt certificates may not be accepted by some of the EGI infrastructure endpoints, you may want to consider using IGTF certificates instead. Check your CA for instructions on how to get those.
You can add direct mappings for specific DNs by adding a /etc/localgridmap.conf file in your running container. See the example below to map the /DC=org/DC=terena/DC=tcs/C=NL/O=Stichting EGI/CN=Enol Fernandez del Castillo DN to nobody. You can add as many lines as needed:
```
"/DC=org/DC=terena/DC=tcs/C=NL/O=Stichting EGI/CN=Enol Fernandez del Castillo" nobody
```
You need to specify the environment variables in the docker-compose.yml file for obtaining the Let’s Encrypt certificate. An extra variable GLOBUS_HOSTNAME must be also set:
```
environment:
 - TESTCERT
 - EMAIL=youremail@domain.com
 - DOMAIN=mygridftp.example.com
 - GLOBUS_HOSTNAME=mygridftp.example.com
```
If you are running on a site with MTU smaller than 1500 (e.g. CESNET-MCC), make sure that you set the MTU to a value smaller than the interface MTU(you can check this with ip addr). In your docker-compose.yml, add:
```
networks:
 default:
 driver: bridge
 driver_opts:
 com.docker.network.driver.mtu: 1434
```

Users: GPUs

Mon, 01 Jan 0001 00:00:00 +0000

GPU resources on EGI Cloud

GPUs resources are available on selected providers of the EGI Cloud. These are available as specific flavours that when used to instantiate a Virtual Machine will make the hardware available to the user.

The table below summarises the available options:

Site	VM configuration options	Flavors	Supported VOs with GPUs	Access conditions	More information
IISAS-FedCloud	up to 7 NVIDIA A100 40GB, up to 8 NVIDIA Tesla K20m	g1.c08r30-K20m, g1.c16r60-2xK20m, A100 GPUs accessible using private flavors	vo.access.egi.eu, training.egi.eu, eosc-synergy.eu, vo.ai4eosc.eu, icecube, vo.beamide.com	Sponsored, conditions to be negotiated
IFCA-LCG2	up to 80 NVIDIA T4, up to 20 NVIDIA V100			Pay-per-use	IFCA-LCG2 Documentation
CESNET-MCC	up to 2 NVIDIA Tesla T4, up to 4 NVIDIA A40, up to 2 GeForce RTX 1080/2080 (experimental use only)	hpc.8core-64ram-nvidia-1080-glados, hpc.19core-176ram-nvidia-1080-glados, hpc.19core-176ram-nvidia-2080-glados, hpc.32core-256ram-nvidia-t4-single-gpu, hpc.64core-238ram-nvidia-t4, hpc.64core-512ram-nvidia-t4, meta-hdm.16core-120ram-nvidia-a40, meta-hdm.64core-485ram-nvidia-a40-quad	vo.clarin.eu, biomed, eosc-synergy.eu, peachnote.com, cryoem.instruct-eric.eu, fusion, icecube, vo.carouseldancing.org, vo.pangeo.eu, vo.neanias.eu, umsa.cerit-sc.cz, vip, vo.notebooks.egi.eu, vo.emphasisproject.eu, vo.inactive-sarscov2.eu	Sponsored, conditions to be negotiated	CESNET-MCC Documentation
IN2P3-IRES	up to 1 NVIDIA Tesla T4 per VM, up to 1 NVIDIA A40 per VM, up to 4 GeForce RTX 2080 per VM (experimental use only)	g1.xlarge-4xmem, g2.xlarge-4xmem, g4.xlarge-4xmem	biomed, vo.emphasisproject.eu, vo.france-grilles.fr	Sponsored, conditions to be negotiated

Access to GPU resources on EGI Cloud

Check whether you belong to one of the supported Virtual Organisations (VOs). If you are not sure what VO to join, request access to the pilot VO vo.access.egi.eu by visiting the enrolment URL with your Check-In account. More information is available in the Check-in section.

GPUs sites can be accessed in different ways: via site-specific dashboards and endpoints or via common federated-cloud services like the OpenStack Horizon dashboards or a cloud orchestrator.

It is also possible to use the FedCloud Client for command-line access. Below is an example on how to use the fedcloud command to show the GPU properties of the available GPU flavors on all sites for the specific VO in the command:

fedcloud openstack flavor list --long --site ALL_SITES --vo vo.access.egi.eu --json-output | \
 jq -r 'map(select(."Error code" == 0)) |
 map(.Result = (.Result| map(select(.Properties."Accelerator:Type" == "GPU")))) |
 map(select(.Result | length > 0))'

Site-specific dashboards and endpoints are described in the following table:

Site	Openstack Horizon dashboard	Keystone endpoint
IISAS-FedCloud	`https://cloud.ui.savba.sk`	`https://cloud.ui.savba.sk:5000/v3/`
IFCA-LCG2	`https://portal.cloud.ifca.es`	`https://api.cloud.ifca.es:5000/`
CESNET-MCC	`https://dashboard.cloud.muni.cz`	`https://identity.cloud.muni.cz/`

Users: Automating Deployments

Mon, 01 Jan 0001 00:00:00 +0000

The OpenStack sites in the EGI Cloud that provide compute resources to run virtual machines (VMs) allow nearly everything to be done via an Application Programming Interface (API) or a command-line interface (CLI). This means that repetitive tasks or complex architectures can be turned into shell scripts.

But creating VMs happens so often in the EGI Cloud that tools were developed to capture the provisioning of these VMs, and allow users to recreate them in a flash, in a deterministic and repeatable way, using an Infrastructure-as-Code (IaC) approach.

Automating this activity will help researchers to:

Not forget important configuration (e.g. the size and type of the hardware resources needed).
Ensure the same steps are performed, in the same order (e.g. making sure the correct datasets are attached to each VM).
Easily share scientific pipelines with collaborators.
Make scientific applications cloud agnostic.

Infrastructure Manager

Infrastructure Manager is the recommended orchestration service to manage virtual infrastructures in EGI Cloud. Check the service documentation for more information

Other orchestrators

Out of band authentication

Orchestrators without native support for the EGI Cloud Compute service, but support out of band obtention with an OpenStack token for accessing the site can be used by obtaining this token with the help of the FedCloud client. You can set the OS_AUTH_URL, OS_PROJECT_ID and OS_TOKEN environment variables as follows:

# export OS_AUTH_URL and OS_PROJECT_ID with
$ eval "$(fedcloud site show-project-id --site <NAME_OF_SITE> --vo <NAME_OF_VO>)"

# now get a valid token for OpenStack
$ export OS_TOKEN=$(fedcloud openstack --site <NAME_OF_SITE> --vo <NAME_OF_VO> \
 token issue -c id -f value)

You will need an access token available for the fedcloud openstack command to work.

Terraform

Terraform supports EGI Cloud OpenStack providers by using valid access tokens for Keystone. For using this, just configure your provider as usual in Terraform, but do not include user/password information. Instead, set up your environment for out of band authentication

Here is a sample main.tf configuration file for Terraform:

terraform {
 required_providers {
 openstack = {
 source = "terraform-provider-openstack/openstack"
 }
 }
}

# Create a server
resource "openstack_compute_instance_v2" "vm" {
 name = "testvm"
 image_id = "..."
 flavor_id = "..."
 security_groups = ["default"]
}

Initialize Terraform with:

$ terraform init

Now check the deployment plan:

$ terraform plan

If you are happy with the plan, perform the deployment with:

$ terraform apply

For more information on how to use Terraform with OpenStack please check the OpenStack provider documentation.

Pulumi

Pulumi provides Infrastructure as Code using different programming languages. Similarly to Terraform, it can use credentials obtained out of band to interact with the OpenStack services. Check the documentation of the Pulumi OpenStack Provider for details on how to interact with sites.

Example: creating a VM with Pulumi and Python

You can follow these steps for getting started with Pulumi and EGI Cloud. It assumes you have a working installation of Pulumi and Python. We will be using IN2P3-IRES site with vo.access.egi.eu VO.

Create new project

Create a new project using pulumi new:

$ mkdir egi-pulumi
$ cd egi-pulumi
$ pulumi new openstack-python

The pulumi new command will interactively initialize a new project, once done you will have several configuration files and a ___main__.py file with some boilerplate code.

Define you infrastructure as code

Edit __main__.py to create a VM. In the example code below, we will use an alma 9 image from EGI’s registry and a 2 cpus VM flavor at IN2P3-IRES site. The site requires a network to be specified, which we can discover either using the dashboard or the FedCloud client.

"""An OpenStack Python Pulumi program"""

import pulumi
from pulumi_openstack import compute, images

# Create an OpenStack resource (Compute Instance)

# Get image EGI alma:9
alma = images.get_image(
 most_recent=True,
 properties={
 "os_distro": "alma",
 "os_version": "9",
 "image_list": "egi_vm_images",
 },
)

flavor = compute.get_flavor(vcpus=2)

# Create instance
# Network name vary for every provider, in this case using
# the name of the network for vo.access.egi.eu VO at IN2P3-IRES
# It can be discovered by login into the dashboard or with:
# `fedcloud openstack --site IN2P3-IRES --vo vo.access.egi.eu network list`
instance = compute.Instance(
 "pulumi-test",
 flavor_id=flavor.id,
 networks=[
 {
 "name": "egi-access-net",
 }
 ],
 image_id=alma.id,
)

# Export the IP of the instance
pulumi.export("instance_ip", instance.access_ip_v4)

Deploy

Follow instructions for out of band authentication to set your environment variables. The next step is to run pulumi login which by default requires an account on the Pulumi Cloud. For the sake of this example, we suggest using pulumi login file://. instead. Then you are ready to run pulumi up:

$ pulumi up
Enter your passphrase to unlock config/secrets
 (set PULUMI_CONFIG_PASSPHRASE or PULUMI_CONFIG_PASSPHRASE_FILE to remember):
Enter your passphrase to unlock config/secrets
Previewing update (dev):
 Type Name Plan Info
 + pulumi:pulumi:Stack egi-pulumi-dev create 1 warning
 + └─ openstack:compute:Instance pulumi-test create

Diagnostics:
 pulumi:pulumi:Stack (egi-pulumi-dev):
 warning: provider config warning: Users not using loadbalancer resources can ignore this message. Support for neutron-lbaas will be removed on next major release. Octavia will be the only supported method for loadbalancer resources. Users using octavia will have to remove 'use_octavia' option from the provider configuration block. Users using neutron-lbaas will have to migrate/upgrade to octavia.

Outputs:
 instance_ip: [unknown]

Resources:
 + 2 to create

Do you want to perform this update? yes

Updating (dev):
 Type Name Status Info
 + pulumi:pulumi:Stack egi-pulumi-dev created (15s) 1 warning
 + └─ openstack:compute:Instance pulumi-test created (13s)

Diagnostics:
 pulumi:pulumi:Stack (egi-pulumi-dev):
 warning: provider config warning: Users not using loadbalancer resources can ignore this message. Support for neutron-lbaas will be removed on next major release. Octavia will be the only supported method for loadbalancer resources. Users using octavia will have to remove 'use_octavia' option from the provider configuration block. Users using neutron-lbaas will have to migrate/upgrade to octavia.

Outputs:
 instance_ip: "172.16.21.167"

Resources:
 + 2 created

Duration: 17s

Clean up

Clean up the resources by using pulumi destroy:

$ pulumi destroy
Enter your passphrase to unlock config/secrets
 (set PULUMI_CONFIG_PASSPHRASE or PULUMI_CONFIG_PASSPHRASE_FILE to remember):
Enter your passphrase to unlock config/secrets
Previewing destroy (dev):
 Type Name Plan
 - pulumi:pulumi:Stack egi-pulumi-dev delete
 - └─ openstack:compute:Instance pulumi-test delete

Outputs:
 - instance_ip: "172.16.21.167"

Resources:
 - 2 to delete

Do you want to perform this destroy? yes
Destroying (dev):
 Type Name Status
 - pulumi:pulumi:Stack egi-pulumi-dev deleted (0.00s)
 - └─ openstack:compute:Instance pulumi-test deleted (11s)

Outputs:
 - instance_ip: "172.16.21.167"

Resources:
 - 2 deleted

Duration: 12s

The resources in the stack have been deleted, but the history and configuration associated with the stack are still maintained.
If you want to remove the stack completely, run `pulumi stack rm dev`.

Users: Dynamic DNS

Mon, 01 Jan 0001 00:00:00 +0000

The Dynamic DNS service provides a unified, federation-wide Dynamic DNS support for VMs in EGI infrastructure. Users can register their chosen meaningful and memorable DNS hostnames in given domains (e.g. my-server.vo.fedcloud.eu) and assign to public IPs of their servers.

By using Dynamic DNS, users can host services in EGI Cloud with their meaningful service names, can freely move VMs from sites to sites without modifying server/client configurations (federated approach), can request valid server certificates in advance (critical for security) and many other advantages.

A short demonstration video is available at fedcloud.eu YouTube channel.

Dynamic DNS GUI portal

The Dynamic DNS offers a web GUI portal where users can login using their Check-in credentials. For doing so, click on the Login link (top left) and then click on the egi button.

Once logged in, you will be presented with the following page:

To register a new DNS host name:

Click on Overview and then on Add Host.
Type in the hostname you’d like to register and select the domain to use.
The portal will then show you a secret than can be used for updating the host ip whenever needed. Note it down so you can use it later.
From the VM you’d like to assign the name to, run a command like follows:
```
curl "https://<hostname>:<secret>@nsupdate.fedcloud.eu/nic/update"
```
where <hostname> is the full hostname generated before, e.g. myserver.fedcloud-tf.fedcloud.eu and <secret> is the secret generated in the previous step. You can add that as a boot command in your cloud-init configuration:
```
#cloud-config

runcmd:
 - [curl, "https://<hostname>:<secret>@nsupdate.fedcloud.eu/nic/update"]
```
You can also manually edit your registered hostnames in the Overview menu by clicking on the hostname you’d like to manage

Note

Hostnames/IP addresses are not expired so no need to refresh IP addresses if no changes, it is enough to run once. You can add the following command curl https://HOSTNAME:SECRET@nsupdate.fedcloud.eu/nic/update to cloud-init as described above to assign hostname automatically at VM start.
DNS server set Time-to-Live (max time for caching DNS records) to 1 min for dynamic DNS, but MS Windows seems to not respect that. You can clear DNS cache in Windows with ipconfig /flushdns command with Administrator account.

Dynamic DNS support in Infrastructure Manager

Infrastructure Manager (IM) automatically creates hostname and assign the correct public IP of the head node of your infrastructure in the Dynamic DNS. Just fill in the DNS name to be used for the VM or DNS name to set to the Kubernetes Front-end (other infrastructures may have similar names for this field) with a FQDN within the supported domains of the Dynamic DNS:

IM will take care of registering the hostname, assigning the IP and eventually removing the hostname once the infrastructure is destroyed.

Wildcard hosts

For certain use cases, it is desirable for all hosts within a specific subdomain to resolve to the same IP address. For example, if the head node of a Kubernetes cluster is accessible at kubernetes.fedcloud.eu, all services within the cluster can share the same IP and be represented using a wildcard entry *.kubernetes.fedcloud.eu, where * stands for any of the services; e.g. dashboard.kubernetes.fedcloud.eu, api.kubernetes.fedcloud.eu, app1.kubernetes.fedcloud.eu. In such setup, request routing is handled internally by the cluster, typically through an Ingress or Gateway resource.

This kind of names can be registered using the API calls as described below.

API

The API is accessible via the API_BASE_URL endpoint (e.g. https://nsupdate.fedcloud.eu) and exposes several endpoints for domain management and related operations.

Authorization

All API requests require a valid ACCESS_TOKEN. This token must be included in the HTTP request Authorization header using the Bearer scheme for authentication.

List domains

Retrieves all private domains owned by the authenticated user, as well as all available public domains.

Endpoint

GET {{API_BASE_URL}}/nic/domains
Authorization: Bearer {{ACCESS_TOKEN}}

Response

Returns a JSON object containing two arrays:

private — Domains owned by the requesting user.
public — Public domains available for use.

Sample response

{
 "status": "ok",
 "private": [],
 "public": [
 {
 "name": "cloud.ai4eosc.eu",
 "public": true,
 "available": true,
 "comment": "Domain for stable services in AI4EOSC project",
 "owner": "viet02"
 },
 {
 "name": "cloud.eosc-siesta.eu",
 "public": true,
 "available": true,
 "comment": "Domain for production services in EOSC-SIESTA project",
 "owner": "root"
 }
 ]
}

Domain Fields Description

Each domain object (private or public) includes the following fields:

Field	Type	Description
`name`	string	The fully qualified domain name (FQDN) of the domain. Example: `cloud.ai4eosc.eu`.
`public`	boolean	Indicates whether the domain is public (`true`) or private (`false`). Public domains are available for multiple users, whereas private domains are owned exclusively by the requesting user.
`available`	boolean	Indicates whether the domain is available for new host registrations. `true` means the domain can be used; `false` means it is in use, reserved, or restricted.
`comment`	string	A short description or annotation about the domain. Typically provides context or usage information.
`owner`	string	The username of the domain owner. For public domains, this is the account responsible for the domain.

Register Host

This endpoint registers a host in the specified domain, optionally setting a wildcard and adding a comment. The response confirms whether the registration was successful or if an error occurred.

Endpoint

Option 1

GET {{API_BASE_URL}}/nic/register?fqdn={{HOSTNAME}}&ip={{IP}}&wildcard={{WILDCARD}}&comment={{COMMENT}}
Authorization: Bearer {{ACCESS_TOKEN}}

Parameters

Name	Type	Required	Description
`fqdn`	string	✅ Yes	Fully qualified domain name (FQDN) of the host to register.
`ip`	string	No	IP address to associate with the host. If omitted, the IP is inferred from the incoming request.
`wildcard`	boolean	No	Enables or disables a wildcard entry for the host. • `true` — enable wildcard • `false` — disable wildcard (default)
`comment`	string	No	Optional comment or description for the host.

Option 2

GET {{API_BASE_URL}}/nic/register?name={{NAME}}&domain={{DOMAIN}}&ip={{IP}}&wildcard={{WILDCARD}}&comment={{COMMENT}}
Authorization: Bearer {{ACCESS_TOKEN}}

Parameters

Name	Type	Required	Description
`name`	string	✅ Yes	Name of the host to register.
`domain`	string	✅ Yes	Domain under which the host is registered.
`ip`	string	No	IP address to associate with the host. If omitted, the IP is inferred from the incoming request.
`wildcard`	boolean	No	Enables or disables a wildcard entry for the host. • `true` — enable wildcard • `false` — disable wildcard (default)
`comment`	string	No	Optional comment or description for the host.

Sample Response

{
 "status": "ok",
 "message": "Host registered.",
 "host": {
 "fqdn": "test.vm.fedcloud.eu",
 "name": "test",
 "domain": "vm.fedcloud.eu",
 "wildcard": false,
 "comment": "test",
 "available": true,
 "client_faults": 0,
 "server_faults": 0,
 "abuse_blocked": false,
 "abuse": false,
 "last_update_ipv4": "2025-10-13T11:24:51.165433+00:00",
 "tls_update_ipv4": false,
 "ipv4": "10.10.0.253",
 "last_update_ipv6": null,
 "tls_update_ipv6": false,
 "ipv6": null,
 "update_secret": "some_secret",
 "IPv4_update_url_basic_auth": "https://test.vm.fedcloud.eu:some_secret@nsupdate.fedcloud.eu/nic/update",
 "IPv4_update_url_bearer_auth": "https://nsupdate.fedcloud.eu/nic/update?hostname=test.vm.fedcloud.eu&myip=${myip}"
 }
}

Response Fields

Field	Type	Description
`status`	string	Indicates the overall status of the request (e.g., `"ok"` for success or `"error"` for failure).
`message`	string	Human-readable message summarizing the result.
`host`	object	Contains detailed information about the registered host.

`host` Object

Field	Type	Description
`fqdn`	string	Fully qualified domain name of the host.
`name`	string	Name of the host.
`domain`	string	Domain under which the host is registered.
`wildcard`	boolean	Whether a wildcard entry is enabled for the host.
`comment`	string	Optional comment associated with the host.
`available`	boolean	Indicates if the host is active and reachable.
`client_faults`	integer	Number of client-related errors recorded.
`server_faults`	integer	Number of server-related errors recorded.
`abuse_blocked`	boolean	Whether the host has been blocked due to abuse reports.
`abuse`	boolean	Indicates if the host is currently flagged for abuse.
`last_update_ipv4`	string (ISO 8601)	Timestamp of the last IPv4 update.
`tls_update_ipv4`	boolean	Indicates if a TLS update is pending for the IPv4 record.
`ipv4`	string	The assigned IPv4 address.
`last_update_ipv6`	string \| null	Timestamp of the last IPv6 update, or `null` if none.
`tls_update_ipv6`	boolean	Indicates if a TLS update is pending for the IPv6 record.
`ipv6`	string \| null	The assigned IPv6 address, or `null` if none.
`update_secret`	string	Secret token used for authenticated updates.
`IPv4_update_url_basic_auth`	string	URL for updating the IPv4 record using Basic Auth credentials.
`IPv4_update_url_bearer_auth`	string	URL for updating the IPv4 record using Bearer token authentication.

Update DNS record

The dynamic DNS update server uses the dyndns2 protocol, compatible with commercial providers such as dyn.com and noip.com, and allows clients to update the IP address of a registered host. In addition to Basic authentication using an update secret, this endpoint also supports authentication with the Bearer scheme for enhanced security.

Endpoint

Update using Bearer token authentication

GET {{API_BASE_URL}}/nic/update?hostname={{HOSTNAME}}&myip={{IP_ADDRESS}}
Authorization: Bearer {{ACCESS_TOKEN}}

Update using Basic authentication with update secret

GET {{API_BASE_URL}}/nic/update?hostname={{HOSTNAME}}&myip={{IP_ADDRESS}}
Authorization: Basic {{BASE64_ENCODED_AUTH_STRING}}

Creating Basic authentication secret

To create the Basic authentication token, encode FQDN of the target host and corresponding update secret using the Base64 encoding. To retrieve the update secret, use the /nic/generate_secret endpoint.

$ echo -n "${HOSTNAME}:${UPDATE_SECRET}" | base64 -
c3Rldm8tZ3B1Og==

Parameters

Name	Type	Required	Description
`hostname`	string	✅ Yes	Fully qualified domain name (FQDN) of the host to update.
`myip`	string	No	IP address to set for the host. If omitted, the server uses the IP address of the requesting client.

Sample response

HTTP/2 200 OK
(Headers) ...content-type: text/plain...

good 123.45.67.89

The response body returns the status of the update followed by the current IP address assigned to the host.
Example: good 123.45.67.89 indicates the update succeeded and the host now points to 123.45.67.89.

List Hosts

Retrieves all hosts registered by the authenticated user. Optionally, you can filter hosts by a specific domain.

Endpoint

GET {{API_BASE_URL}}/nic/hosts?domain={{DOMAIN}}
Authorization: Bearer {{ACCESS_TOKEN}}

Parameters

Name	Type	Required	Description
`domain`	string	No	Optional domain to filter the hosts. If omitted, all hosts registered by the user are returned.

Sample response

{
 "status": "ok",
 "hosts": [
 {
 "fqdn": "myhost.cloud.ai4eosc.eu",
 "name": "myhost",
 "domain": "cloud.ai4eosc.eu",
 "wildcard": false,
 "comment": "comment",
 "available": true,
 "client_faults": 1,
 "server_faults": 0,
 "abuse_blocked": false,
 "abuse": false,
 "last_update_ipv4": "2025-04-10T10:33:06.930760+00:00",
 "tls_update_ipv4": false,
 "ipv4": "147.213.65.206",
 "last_update_ipv6": "2025-04-10T10:21:57.975176+00:00",
 "tls_update_ipv6": false,
 "ipv6": null
 }
 ]
}

Response Fields

Field	Type	Description
`status`	string	Indicates the overall status of the request (e.g., `"ok"` for success or `"error"` for failure).
`message`	string	Present only in failed responses; provides a human-readable explanation of the error.
`hosts`	array	Presents only in successful responses; provides a list of host objects registered by the user.

`hosts` Object

Field	Type	Description
`fqdn`	string	Fully qualified domain name of the host.
`name`	string	Name of the host.
`domain`	string	Domain under which the host is registered.
`wildcard`	boolean	Indicates whether a wildcard entry is enabled for the host.
`comment`	string	Optional comment associated with the host.
`available`	boolean	Indicates if the host is active and reachable.
`client_faults`	integer	Number of client-related errors recorded.
`server_faults`	integer	Number of server-related errors recorded.
`abuse_blocked`	boolean	Whether the host has been blocked due to abuse reports.
`abuse`	boolean	Indicates if the host is currently flagged for abuse.
`last_update_ipv4`	string (ISO 8601)	Timestamp of the last IPv4 update.
`tls_update_ipv4`	boolean	Indicates if a TLS update is pending for the IPv4 record.
`ipv4`	string	The assigned IPv4 address.
`last_update_ipv6`	string (ISO 8601)	Timestamp of the last IPv6 update, or `null` if none.
`tls_update_ipv6`	boolean	Indicates if a TLS update is pending for the IPv6 record.
`ipv6`	string \| null	The assigned IPv6 address, or `null` if none.

Unregister Host

Removes a previously registered host from the specified domain. The response confirms whether the unregistration was successful or if an error occurred.

Endpoint

Option 1

GET {{API_BASE_URL}}/nic/unregister?fqdn={{HOSTNAME}}
Authorization: Bearer {{ACCESS_TOKEN}}

Parameters

Name	Type	Required	Description
`fqdn`	string	✅ Yes	Fully qualified domain name (FQDN) of the host to unregister.

Option 2

GET {{API_BASE_URL}}/nic/unregister?name={{NAME}}&domain={{DOMAIN}}
Authorization: Bearer {{ACCESS_TOKEN}}

Parameters

Name	Type	Required	Description
`name`	string	✅ Yes	Name of the host to unregister.
`domain`	string	✅ Yes	Domain under which the host is registered.

Sample response

{
 "status": "ok",
 "message": "Host unregistered.",
 "host": {
 "fqdn": "test.vm.fedcloud.eu",
 "name": "test",
 "domain": "vm.fedcloud.eu",
 "wildcard": false,
 "comment": "host for testing purposes"
 }
}

Response Fields

Field	Type	Description
`status`	string	Indicates the overall status of the request (e.g., `"ok"` for success or `"error"` for failure).
`message`	string	Human-readable message summarizing the result.
`host`	object	Contains details of the unregistered host.

`host` Object

Field	Type	Description
`fqdn`	string	Fully qualified domain name (FQDN) of the unregistered host.
`name`	string	Name of the unregistered host.
`domain`	string	Domain under which the host was registered.
`wildcard`	boolean	Indicates whether a wildcard entry was enabled for the host.
`comment`	string \| null	Comment associated with the host, or `null` if none.

Generate update secret

Generates a new update secret for clients to update a registered host. The secret can be used for authenticated dynamic updates to the host’s IP records.

Endpoint

GET {{API_BASE_URL}}/nic/generate_secret?fqdn={{HOSTNAME}}
Authorization: Bearer {{ACCESS_TOKEN}}

Parameters

Name	Type	Required	Description
`fqdn`	string	✅ Yes	Fully qualified domain name (FQDN) of the host for which to generate the secret.

Sample response

{
 "status": "ok",
 "secret": "bMkzLXMr75"
}

Response Fields

Field	Type	Description
`status`	string	Indicates the overall status of the request (e.g., `"ok"` for success or `"error"` for failure).
`message`	string	Present only in failed responses; provides a human-readable explanation of the error.
`secret`	string	Presents only in successful responses; provides the newly generated update secret for the specified host.

Security

For updating IP address, only hostname and its secret are needed. No user information is stored on VM in any form for updating IP.
NS-update server uses HTTPS protocol, hostname/secret are encrypted as data and not visible during transfer so it is secure to use the update URL
NS-update portal does not store host secret in recoverable form. If you forget the secret of your hostname, simply generate new one via “Show configuration” button in the host edit page. The old secret will be invalid.

Support

Support for Dynamic DNS service is provided via EGI Helpdesk “Dynamic DNS” support unit, where users can ask questions, report issues or make requests for additional domains for specific projects or user communities.

Users: Frequently Asked Questions

Mon, 01 Jan 0001 00:00:00 +0000

Basics

How can I get access to the cloud compute service?

There is a VO available for 6 months piloting activities that any researcher in Europe can join. Just request access to the pilot Virtual Organisation.

How can I get an OAuth2.0 token?

Authentication via CLI or API requires a valid Access Token from Check-in. The EGI Check-in Token Portal allows you to get one as needed. Check the Authentication and Authorisation guide for more information.

Is OCCI still supported?

OCCI is now deprecated as API for the EGI Cloud providers using OpenStack. Some providers still support OCCI (a list of active endpoints can be queried at GOCDB) but it should note be used for any new developments.

Migration from rOCCI CLI to OpenStack CLI is quite straightforward, we summarize the main commands in rOCCI and OpenStack equivalent in the table below:

Action	rOCCI	OpenStack
List images	`occi -a list -r os_tpl`	`openstack image list`
Describe images	`occi -a describe -r <image_id>`	`openstack image show <image_id>`
List flavors	`occi -a list -r resource_tpl`	`openstack flavor list`
Describe flavors	`occi -a describe -r <template_id>`	`openstack flavor show <image_id>`
Create VM	`occi -a create -r compute -t occi.core.title="MyFirstVM" -M <flavor id> -M <image id> -T user_data="file://<file>"`	`openstack server create --flavor <flavor> --image <image> --user-data <file> MyFirstVM`
Describe VM	`occi -a describe -r <vm id>`	`openstack server show <vm id>`
Delete VM	`occi -a delete -r <vm id>`	`openstack server delete <vm id>`
Create volume	`occi -a create -r storage -t occi.storage.size='num(<site in GB>)' -t occi.core.title=<storage_resource_name>`	`openstack volume create --size <size in GB> <storage resource name>`
List volume	`occi -a list -r storage`	`openstack volume list`
Attach volume	`occi -a link -r <vm_id> -j <storage_resource_id>`	`openstack server add volume <vm id> <volume id>`
Detach volume	`occi -a unlink -r <storage_link_id>`	`openstack server remove volume <vm id> <volume id>`
Delete volume	`occi -a delete -r <volume id>`	`openstack volume delete <volume id>`
Attach public IP	`occi -a link -r <vm id> --link /network/public`	`openstack server add floating ip <vm id> <ip>`

If you still rely on OCCI for your access, please contact us at support _at_ egi.eu for support on the migration.

Discovery

How can I get the list of the EGI Cloud providers?

The list of certified providers is available in GOCDB. The following command with the fedcloud client can help you to get that list:

$ fedcloud site list
100IT
BIFI
CESGA
CESNET-MCC
CETA-GRID
CLOUDIFIN
CYFRONET-CLOUD
DESY-HH
GSI-LCG2
IFCA-LCG2
IISAS-FedCloud
IISAS-GPUCloud
IN2P3-IRES
INFN-CATANIA-STACK
INFN-CLOUD-BARI
INFN-PADOVA-STACK
Kharkov-KIPT-LCG2
NCG-INGRID-PT
SCAI
TR-FC1-ULAKBIM
UA-BITP
UNIV-LILLE
fedcloud.srce.hr

The providers also generate dynamic information which is accessible through the cloud-info-api.

How can I choose which site to use?

Sites offer their resources to users through Virtual Organisations (VO). First, you need to join a Virtual Organisation that matches your research interests, see authorisation section on how VOs work.

How can I get information about the available VM images?

The Artefact Registry contains information about the VM images available in the EGI Cloud. A set of predefined images is available on every provider. For VO-specific images, check the VM image guide.

Managing VMs

The disk on my VM is full, how can I get more space?

There are several ways to increase the disk space available at the VM. The fastest and easiest one is to use block storage, creating a new storage disk device and attaching it to the VM. Check the storage guide for more information.

How can I keep my data after the VM is stopped?

After a VM has been stopped and unless backed up in a block storage volume, all data in the VM is destroyed and cannot be recovered. To ensure your data will be available after the VM is deleted, you need to use some form of persistent storage.

How can I assign a public IP to my VM?

Some providers do not automatically assign a public IP address to a VM during the creation phase. In this case, you can attach a public IP by first allocating a new public IP and then assigning it to the VM.

How can I assign a DNS name to my VM?

If you need a domain name for your VMs, we offer a Dynamic DNS service that allows any EGI user to create names for VMs under the fedcloud.eu domain.

Just go to EGI Cloud nsupdate and login with your Check-in account. Once in, you can click on "Add host" to register a new hostname in an available domain.

What is contextualisation?

Contextualisation is the process of installing, configuring and preparing software upon boot time on a predefined virtual machine image. This way, the predefined images can be stored as generic and small as possible, since customisations will take place on boot time.

Contextualisation is particularly useful for:

Configuration not known until instantiation (e.g. data location).
Private Information (e.g. host certs)
Software that changes frequently or under development.

Contextualisation requires passing some data to the VMs on instantiation (the context) and handling that context in the VM.

How can I inject my public SSH key into the machine?

The best way to login into the virtual server is to use SSH keys. If you don't have one, you need to generate it with the ssh-keygen command:

ssh-keygen -f fedcloud

This will generate two files:

fedcloud, the private key. This file should never be shared
fedcloud.pub, the public key. That will be sent to your VM.

To inject the public SSH key into the VM you can use the key-name option when creating the VM in OpenStack. Check keypair management option in OpenStack documentation. This key will be available for the default configured user of the VM (e.g. ubuntu for Ubuntu, centos for CentOS).

You can also create users with keys with a contextualisation file:

#cloud-config
users:
 - name: cloudadm
 sudo: ALL=(ALL) NOPASSWD:ALL
 lock-passwd: true
 ssh-import-id: cloudadm
 ssh-authorized-keys:
 - <paste here the contents of your SSH key pub file>

Warning

YAML format requires that the spaces at the beginning of each line is respected in order to be correctly parsed by cloud-init.

How can I use a contextualisation file?

If you have a contextualisation file, you can use it with the --user-data option to server create in OpenStack.

openstack server create --flavor <your-flavor> --image <your image> \
 --user-data <your contextualisation file> \
 <server name>

Note

We recommend using cloud-init for contextualisation. EGI images do support cloud-init. Check the documentation for more information.

How can I pass secrets to my VMs?

EGI Cloud endpoints use HTTPS so information passed to contextualize the VMs can be assumed to be safe and only readable within your VM. However, take into account that anyone with access to the VM may be able to access also the contextualisation information.

Warning

Take into account that anyone with access to the VM may be able to access also the contextualisation information, so ensure that no sensitive data like clear text passwords is used during contextualisation.

How can I use ansible?

Ansible relies on ssh for accessing the servers it will configure. VMs at EGI Cloud can be also accessed via ssh, just make sure you inject the correct public keys in the VMs to be able to access.

If you don't have public IPs for all the VMs to be managed, you can also use one as a gateway as described in the Ansible FAQ.

How can I release resources without destroying my data?

Whenever you delete a VM, the ephemeral disks associated with it will be also deleted. If you don’t plan to use your VM for some time, there are several ways to release resources consumed by the VM (e.g. CPU, RAM) and recover the data or boot your VM in a previous state when you need it back. We list below the main strategies you can use:

Use a volume to store the data to be kept: Check the Storage section of the documentation to learn how to use volumes. If you start your VM from a volume, the VM can be destroyed and recreated easily. OpenStack documentation cover how to start a VM from a volume with CLI or using the Horizon dashboard
Suspend or shelve instance: Suspending a VM will pause a VM, releasing CPU and memory, and allowing to resume later in time at the exact same state. Shelving shuts down the VM, thus RAM contents will be lost but disk will be kept. This releases more resources from the provider while still allows to easily boot the VM back without losing disk contents.
Create snapshot of instance: a snapshot will create a new VM image at your provider that can be used to boot a new instance of the VM with the same disk content. You can use this technique for creating a base template image that can be later re-used to start similar VMs easily.

How can I find all the VMs that I own in the EGI Federated Cloud?

We suggest using the fedcloudclient:

# list the Virtual Organisations that you belong to
fedcloud token list-vos

# then, for each VO, run:
list-all-my-own-vms.sh --vo <virtual-organisation>

See the fedcloudclient documentation for more information.