Documentation – OpenStack

Providers: Check-in

Mon, 01 Jan 0001 00:00:00 +0000

The integration of OpenStack service providers into the EGI Check-in is a two-step process:

Test integration with the demo instance of EGI Check-in. This will allow you to check the complete functionality of the system without affecting the production Check-in service.
Once the integration is working correctly, register your provider with the production instance of EGI Check-in to allow members of the EGI User Community to access your service.

Registration into Check-in demo instance

Before your service can use the EGI Check-in OIDC Provider for user login, you need to register a client through the EGI Federation Registry in order to obtain OAuth2.0 credentials and register one or more redirect URIs.

Make sure that you fill in the following options:

General tab:
- Set Integration Environment to Demo and fill the form with the information about your Service.
Protocol Specific tab:
- Set Select Protocol to OIDC Service
- Set redirect URL to https://<your keystone endpoint>/v3/auth/OS-FEDERATION/websso/openid/redirect. Recent versions of OpenStack may deploy Keystone at /identity/, be sure to include that in the <your keystone endpoint> part of the URL if needed.
- Enable openid, profile, email, eduperson_entitlement in the Scope field
- Enable authorization code in the Grant Types field
- Set Proof Key for Code Exchange (PKCE) Code Challenge Method to SHA-256 hash algorithm (recommended)
- Make sure Allow calls to the Introspection Endpoint? is enabled in Introspection field

Submit the request for review by the Check-in operations team. Once the request has been approved, you will get a client ID and client secret. Save them for the following steps

Keystone setup

Pre-requisites

Keystone must run as a WSGI application behind an HTTP server (Apache is used in this documentation, but any server should be possible if it has OpenID connect/OAuth2.0 support). Keystone project has deprecated eventlet, so you should be already running Keystone in such way.
Keystone must be run with SSL
You need to install mod_auth_openidc for adding support for OpenID Connect to Apache.

IGTF CAs

EGI monitoring checks that your Keystone accepts clients with certificates from the IGTF CAs. Please ensure that your server is configured with the correct Certificate and Revocation path:

For Apache HTTPd: HTTPd is able to use CAs and CRLs contained in a directory:

SSLCACertificatePath /etc/grid-security/certificates
SSLCARevocationPath /etc/grid-security/certificates

For haproxy: CA and CRLS have to be bundled into one file.

Client verification should be set as optional otherwise accepted CAs won't be presented to the EGI monitoring:

# crt: concatenated cert, key and CA
# ca-file: all IGTF CAs, concatenated as one file
# crl-file: all IGTF CRLs, concatenated as one file
# verify: enable optional X.509 client authentication
bind XXX.XXX.XXX.XXX:443 ssl crt /etc/haproxy/certs/host-cert-with-key-and-ca.pem ca-file /etc/haproxy/certs/igtf-cas-bundle.pem crl-file /etc/haproxy/certs/igtf-crls-bundle.pem verify optional

For nginx: CA and CRLS have to be bundled into one file.

Client verification should be set as optional otherwise accepted CAs won't be presented to the EGI monitoring:

ssl_client_certificate /etc/ssl/certs/igtf-cas-bundle.pem;
ssl_crl /etc/ssl/certs/igtf-crls-bundle.pem;
ssl_verify_client optional;

Managing IGTF CAs and CRLs: IGTF CAs can be obtained from UMD, you can find repository files for your distribution at EGI CA repository

IGTF CAs and CRLs can be bundled using the examples command hereafter.

Please update CAs bundle after IGTF updates, and CRLs bundle after each CRLs update made by fetch-crl:

cat /etc/grid-security/certificates/*.pem > /etc/haproxy/certs/igtf-cas-bundle.pem
cat /etc/grid-security/certificates/*.r0 > /etc/haproxy/certs/igtf-crls-bundle.pem
# Some CRLs files are not ending with a new line
# Ensuring that CRLs markers are separated by a line feed
perl -pe 's/----------/-----\n-----/' -i /etc/haproxy/certs/igtf-crls-bundle.pem

Apache Configuration

Include this configuration on the Apache config for the virtual host of your Keystone service, using the client ID and secret obtained above:

OIDCResponseType "code"
OIDCClaimPrefix "OIDC-"
OIDCClaimDelimiter ;
OIDCScope "openid profile email eduperson_entitlement"
OIDCProviderMetadataURL https://aai-demo.egi.eu/auth/realms/egi/.well-known/openid-configuration
# PKCE method
OIDCPKCEMethod S256
OIDCClientID <client id>
OIDCClientSecret <client secret>
OIDCCryptoPassphrase <some crypto pass phrase>
OIDCRedirectURI https://<your keystone endpoint>/v3/auth/OS-FEDERATION/websso/openid/redirect

# OAuth for CLI access
OIDCOAuthIntrospectionEndpoint https://aai-demo.egi.eu/auth/realms/egi/protocol/openid-connect/token/introspect
OIDCOAuthClientID <client id>
OIDCOAuthClientSecret <client secret>

# Increase Shm cache size for supporting long entitlements
OIDCCacheShmEntrySizeMax 65536

<Location ~ "/v3/auth/OS-FEDERATION/websso/openid">
 AuthType openid-connect
 Require valid-user
</Location>

<Location ~ "/v3/OS-FEDERATION/identity_providers/egi.eu/protocols/openid/auth">
 Authtype oauth20
 Require valid-user
</Location>

If you have multiple keystone hosts, configure an alternative caching mechanism as per https://github.com/zmartzone/mod_auth_openidc/wiki/Caching

For example, using memcache

OIDCCacheType memcache
OIDCMemCacheServers "memcache1 memcache2 memcache3"

Be sure to enable the mod_auth_oidc module in Apache, in Ubuntu:

sudo a2enmod auth_openidc

Note

If running Keystone behind a proxy, make sure to correctly set the X-Forwarded-Proto and X-Forwarded-Port request headers, e.g. for haproxy:

http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
http-request set-header X-Forwarded-Port %[dst_port]

Multiple OIDC providers

If your OpenStack deployment needs to support multiple identity providers (besides EGI Check-in) you will need to configure mod_auth_openidc to support multiple providers and use an OAuth2.0 token introspection proxy like ESACO.

`mod_auth_openidc` configuration

First, create a directory to host each of the providers configuration, in our case we will use /var/lib/apache2/oidc/metadata, but adapt this to your specific needs. Ensure this directory is writable by the user running apache:

mkdir -p /var/lib/apache2/oidc/metadata
chown -R www-data:www-data /var/lib/apache2/oidc/metadata

Set in your Apache configuration the OIDCMetadataDir pointing to that directory

OIDCMetadataDir /var/lib/apache2/oidc/metadata

You may remove the OIDCProviderMetadataURL, OIDCClientID and OIDCClientSecret options from the Apache configuration as these will be now set in new files created in the metadata directory. For every provider you will support, you need to create 3 files:

<urlencoded-issuer-value-with-https-prefix-and-trailing-slash-stripped>.provider with the OpenID Connect Discovery OP JSON metadata. The easiest way to create this file is getting its content from the OIDC server itself. For EGI Check-in:
```
curl https://aai-demo.egi.eu/auth/realms/egi/.well-known/openid-configuration > \
 /var/lib/apache2/oidc/metadata/aai-demo.egi.eu%2Foidc.provider
```
<urlencoded-issuer-value-with-https-prefix-and-trailing-slash-stripped>.client with the client credentials. For EGI Check-in (aai-demo.egi.eu%2Foidc.client):
```
{
 "client_id": "<your client id>",
 "client_secret": "<your secret id>"
}
```
<urlencoded-issuer-value-with-https-prefix-and-trailing-slash-stripped>.conf with any extra configuration for the provider. This may not be needed if all your providers are similar. For example to specify the scopes to use for Check-in, use a aai-demo.egi.eu%2Foidc.conf as follows:
```
{
 "scope": "openid email profile eduperson_entitlement",
 "pkce_method": "S256"
}
```

Now add for the providers you support new configuration in Apache to facilitate the use of the dashboard. This is for a configuration of an egi.eu identity provider with openid as protocol:

<Location ~ "/v3/auth/OS-FEDERATION/identity_providers/egi.eu/protocols/openid/websso">
 AuthType openid-connect
 # This is your Redirect URI with a new iss=<your idp iss> option added
 OIDCDiscoverURL https://<your keystone endpoint>/v3/auth/OS-FEDERATION/websso/openid/redirect?iss=https%3A%2F%2Faai-demo.egi.eu%2Foidc%2F
 # Ensure that the user is authenticated with the expected iss
 Require claim iss:https://aai-demo.egi.eu/auth/realms/egi
 Require valid-user
</Location>

ESACO configuration

ESACO will handle OAuth tokens when users hit your Keystone from API/CLI. It needs to run as a daemon that listens (by default) on port 8156. We will use docker for facilitating the deployment:

Create a yaml file with the configuration of the different providers (application.yaml):

oidc:
 clients:
 - issuer-url: https://aai-demo.egi.eu/auth/realms/egi
 client-id: "<your check-in client id>"
 client-secret: "<your check-in client secret>"
 - issuer-url: <another idp>
 client-id: "<your client id for second idp>"
 client-secret: "<your client secret for second idp>"

Create a environment file with the ESACO credentials you want to use (esaco.env):

# User name credential requested from clients introspecting tokens
ESACO_USER_NAME=<esaco user name>

# Password credential requested from clients introspecting tokens
ESACO_USER_PASSWORD=<esaco password>

Run the ESACO server (adapt this as it better fits to run on your servers and make it run permanently):

docker run -p 8156:8156 -d -env-file=esaco.env \
 -v application.yml:/esaco/config/application.yml:ro \
 indigoiam/esaco:latest

Configure Keystone’s Apache to use ESACO as OAuth introspection endpoint:

# point this to the host where ESACO is running
OIDCOAuthIntrospectionEndpoint http://localhost:8156/introspect
OIDCOAuthClientID <esaco user name>
OIDCOAuthClientSecret <esaco password>
OIDCIDTokenIatSlack 3600

Configure also the locations in Apache that should use OAuth:

<Location ~ "/v3/OS-FEDERATION/identity_providers/egi.eu/protocols/openid/auth">
 Authtype oauth20
 Require valid-user
</Location>

<Location ~ "/v3/OS-FEDERATION/identity_providers/other_idp/protocols/openid/auth">
 Authtype oauth20
 Require valid-user
</Location>

Horizon configuration

In your Horizon configuration, set the list of providers and their mappings:

# this is the list that will show up in the dropdown menu
WEBSSO_CHOICES = (
 ("credentials", _("Keystone Credentials")),
 ("egi.eu", _("EGI Check-in")),
 ("other-idp", _("Other IdP")),
)

# this maps the options above to keystone's idps and protocols
WEBSSO_IDP_MAPPING = {
 "egi.eu": ("egi.eu", "openid"),
 "other-idp": ("other-idp.com", "openid")
}

Keystone Configuration

Configure your keystone.conf to include in the [auth] section openid in the list of authentication methods:

[auth]

# This may change in your installation
# add openid to the list of the methods you support
methods = password, token, openid

Add a [openid] section as follows:

[openid]
# this is the attribute in the Keystone environment that will define the
# identity provider
remote_id_attribute = HTTP_OIDC_ISS

Add your horizon host as trusted dashboard to the [federation] section:

[federation]
trusted_dashboard = https://<your horizon>/dashboard/auth/websso/

Finally copy the default template for managing the tokens in horizon to /etc/keystone/sso_callback_template.html. This template can be found in keystone git repository at https://github.com/openstack/keystone/blob/master/etc/sso_callback_template.html

curl -L https://raw.githubusercontent.com/openstack/keystone/master/etc/sso_callback_template.html \
 > /etc/keystone/sso_callback_template.html

Now restart your Apache (and Keystone if running in uwsgi) so you can configure the Keystone Federation support.

Keystone Federation Support

First, create a new egi.eu identity provider with remote id https://aai-demo.egi.eu/auth/realms/egi:

$ openstack identity provider create --remote-id https://aai-demo.egi.eu/auth/realms/egi egi.eu
+-------------+-----------------------------------------+
| Field | Value |
+-------------+-----------------------------------------+
| description | None |
| domain_id | 1cac7817dafb4740a249cc9ca6b14ea5 |
| enabled | True |
| id | egi.eu |
| remote_ids | https://aai-demo.egi.eu/auth/realms/egi |
+-------------+-----------------------------------------+

Check the name of the egi.eu domain name:

$ openstack domain show -f value -c name $(openstack identity provider show -f value -c domain_id egi.eu)

Set the name to egi.eu (if it was set to random auto-generated number):

$ openstack domain set --name egi.eu $(openstack identity provider show -f value -c domain_id egi.eu)

Create a group and add a domain-wide role for auditing purposes (see below):

# Support for https://operations-portal.egi.eu/vo/view/voname/cloud.egi.eu
$ openstack group create --domain egi.eu egi-staff
$ openstack role add --domain egi.eu --group egi-staff reader

Every VO you want to support should be mapped to a local project. The ops VO is used by EGI monitoring to ensure the correct functioning of your site. Create a group for this vo and add the group as a member of it:

# Support for https://operations-portal.egi.eu/vo/view/voname/ops
$ openstack group create ops
$ openstack role add --domain egi.eu --group ops --project <your local ops project> member

Now you can define the mapping of EGI Check-in users into the groups you just created and restrict with the OIDC-eduperson_entitlement attribute which users will be members of those groups. Substitute the group IDs to the adequate values for your deployment:

$ cat mapping.egi.json
[
 {
 "local": [
 {
 "user": {
 "name": "{0}",
 "email": "{1}"
 },
 "group": {
 "id": "_ops_group_ID_"
 }
 }
 ],
 "remote": [
 {
 "type": "HTTP_OIDC_SUB"
 },
 {
 "type": "HTTP_OIDC_EMAIL"
 },
 {
 "type": "HTTP_OIDC_ISS",
 "any_one_of": [
 "https://aai-demo.egi.eu/auth/realms/egi"
 ]
 },
 {
 "type": "OIDC-eduperson_entitlement",
 "regex": true,
 "any_one_of": [
 "^urn:mace:egi.eu:group:ops:role=vm_operator#aai.egi.eu$"
 ]
 }
 ]
 },
 {
 "local": [
 {
 "user": {
 "name": "{0}",
 "email": "{1}"
 },
 "group": {
 "id": "_egi-staff_group_ID_"
 }
 }
 ],
 "remote": [
 {
 "type": "HTTP_OIDC_SUB"
 },
 {
 "type": "HTTP_OIDC_EMAIL"
 },
 {
 "type": "HTTP_OIDC_ISS",
 "any_one_of": [
 "https://aai.egi.eu/auth/realms/egi"
 ]
 },
 {
 "type": "OIDC-eduperson_entitlement",
 "regex": true,
 "any_one_of": [
 "^urn:mace:egi.eu:group:cloud.egi.eu:role=auditor#aai.egi.eu$"
 ]
 }
 ]
 }
]

Note

Note the use of the HTTP_OIDC_EMAIL in the mapping will allow to store the user’s email in your local database.

Create the mapping in Keystone:

$ openstack mapping create --rules mapping.egi.json egi-mapping

Finally, create the federated protocol with the identity provider and mapping created before:

$ openstack federation protocol create \
 --identity-provider egi.eu \
 --mapping egi-mapping openid
+-------------------+-------------+
| Field | Value |
+-------------------+-------------+
| id | openid |
| identity_provider | egi.eu |
| mapping | egi-mapping |
+-------------------+-------------+

Keystone is now ready to accept EGI Check-in credentials.

VO auditing

Sometimes it is easy to leave behind Virtual Machines that are no longer used, consuming unnecessary resources. Owners of unused VMs should be notified to check whether occupied resources can be freed.

EGI Check-in users get an ePUID (i.e. a long hash ending in @egi.eu) which are translated into local OpenStack user IDs. When VMs are created the owner of the VM is set to the OpenStack user ID instead of the ePUID. However, only the ePUID is linked to the user email in order for the user to be notified. The mapping between OpenStack user IDs and ePUIDs is shown with:

$ openstack user list

Problem is that regular users will not have the permissions to execute the command above. The steps above to configure a mapping for the cloud.egi.eu VO grant access to selected staff at EGI.eu to execute the command, using the default keystone policy:

 "identity:list_users": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)"

This has been tested in production on OpenStack Ussuri thanks to the collaboration between EGI.eu and IISAS-Fedcloud. It should also work with newer versions of OpenStack.

EGI.eu staff belonging to the cloud.egi.eu VO having the auditor role should use the below setup to get the OpenStack user list:

export OS_INTERFACE=public
# get OS_AUTH_URL with "fedcloud site env --vo <vo> --site <site>"
export OS_AUTH_URL=https://cloud.ui.savba.sk:5000/v3
export OS_USERNAME=<ePUID> # get it from https://aai.egi.eu/
export OS_IDENTITY_PROVIDER=egi.eu
export OS_AUTH_TYPE=v3oidcaccesstoken
export OS_PROTOCOL=openid
export OS_IDENTITY_API_VERSION=3
# get OS_ACCESS_TOKEN following https://docs.egi.eu/users/aai/check-in/obtaining-tokens/
export OS_ACCESS_TOKEN=<token>
export OS_DOMAIN_NAME=egi.eu

$ openstack user list

With this configuration EGI.eu staff is able to proactively notify creators of long-running VMs that may not be making an effective use of the cloud resources.

Additional VOs

To configure additional VOs please follow steps in the VO Configuration guide.

Horizon Configuration

Edit your local_settings.py to include the following values:

# Enables keystone web single-sign-on if set to True.
WEBSSO_ENABLED = True

# Allow users to choose between local Keystone credentials or login
# with EGI Check-in
WEBSSO_CHOICES = (
 ("credentials", _("Keystone Credentials")),
 ("openid", _("EGI Check-in")),
)

Once horizon is restarted you will be able to choose "EGI Check-in" for login.

CLI Access

The OpenStack Client has built-in support for using OpenID Connect Access Tokens to authenticate. You first need to get a valid Access Token from EGI Check-in (e.g. from https://aai-demo.egi.eu/token/) and then use it in a command like:

$ openstack --os-auth-url https://<your keystone endpoint>/v3 \
 --os-auth-type v3oidcaccesstoken --os-protocol openid \
 --os-identity-provider egi.eu \
 --os-access-token <your access token> \
 token issue
+---------+---------------------------------------------------------------------------------------+
| Field | Value |
+---------+---------------------------------------------------------------------------------------+
| expires | 2017-05-23T11:24:31+0000 |
| id | gAAAAABZJA3fbKX....nEMAPi-IsFOCkU9QWGTISYElzYJsI3z0SJGs7QsTJv4aJQq0JDJUBz6uE85SqXDj3 |
| user_id | 020864ea9415413f9d706f6b473dbeba |
+---------+---------------------------------------------------------------------------------------+

Moving to EGI Check-in production instance

Once tests in the development instance of Check-in are successful, you can move to the production instance. Go to EGI Federation Registry and submit a Service Request for the production instance of EGI Check-in. After the approval of the request, you will need to update your configuration as follows:

Update the remote-id of the identity provider:

openstack identity provider set --remote-id https://aai.egi.eu/auth/realms/egi egi.eu

Update the HTTP_OIDC_ISS filter in your mappings, e.g.:

sed -i 's/aai-demo.egi.eu/aai.egi.eu/' mapping.egi.json
openstack mapping set --rules mapping.egi.json egi-mapping

Update Apache configuration to use aai.egi.eu instead of aai-demo.egi.eu, if you have multiple OIDC providers, you should as well update the providers metadata and ESACO configuration. For the basic Apache configuration you should set these values:
```
OIDCProviderMetadataURL https://aai.egi.eu/auth/realms/egi/.well-known/openid-configuration
OIDCOAuthIntrospectionEndpoint https://aai.egi.eu/auth/realms/egi/protocol/openid-connect/token/introspect
```

Changes in the client settings

If you want to make any changes to the client configuration, you need to submit a reconfiguration request through the Federation Registry.

Additional VOs

Once ops VO is working, you can include any further VOs you want to support as documented in the VO Configuration guide.

Providers: Accounting

Mon, 01 Jan 0001 00:00:00 +0000

There are two different processes handling the accounting integration:

cASO, which connects to the OpenStack services to get the usage information, and,
ssmsend, which sends that usage information to the central EGI accounting repository.

Installation

You can get cASO from the releases page, alternatively a container image is available in the fedcloud-caso repository.

SSM is also available in the releases or as a container in the ssm repository

Configuration

cASO configuration is stored at /etc/caso/caso.conf. Most default values should be OK, but you must set:

site_name (line 12), with the name of your site as defined in GOCDB.
credentials to access the OpenStack services to extract accounting data. Check cASO documentation for the expected permissions of the user configured here.
cASO will discover the projects from OpenStack using tags and properties. You can set specific tags and properties as needed as described in the Documentation.

cASO will write records to /var/spool/apel from where ssmsend will take them.

SSM configuration is available at /etc/apel. Set the destination to eu-egi-cloud-accounting:

[messaging]
# If using AMS this is the project that SSM will connect to. Ignored for STOMP.
ams_project: accounting

# Queue to which SSM will send messages
destination: eu-egi-cloud-accounting

Other default values available in the configuration file should be ok for your site. SSM expects /etc/grid-security to contain the host certificate (/etc/grid-security/hostcert.pem), private key (/etc/grid-security/hostkey.pem) and CAs (/etc/grid-security/certificates).

Running the services

Both caso and ssmsend should run periodically, e.g. with a cron job, at least once a day. We recommend running them every 4 hours.

Providers: Catch-all components

Mon, 01 Jan 0001 00:00:00 +0000

EGI manages the operations of two components for the sites so you don’t need to do it:

the information discovery provides a view about the actual images and flavors available at the OpenStack for the federation users. It runs as a single python application cloud-info-provider that pushes information through the Messaging Service.

BDII is deprecated
Cloud providers no longer need to provide BDII as the Argo Messaging Service is used instead for transferring information
the image-sync makes sure that images available in EGI’s Artefact Registry are available for your users in glance.

The deployment and configuration of these components is managed a the

EGI-Federation/fedcloud-catchall-operations repository. Every provider should have a single yaml file that describes its configuration as follows:

endpoint: <your endpoint as declared in GOCDB>
gocdb: <your site name as declared in GOCDB>
images:
 sync: true
 # optionally list the formats that your site supports/prefers (as supported by qemu)
 # if not specified, the sync will assume all formats are supported
 formats:
 - qcow2
 - raw
vos:
 # a list of VOs you support in your deployment as follows
 - auth:
 project_id: <local OpenStack project identifier>
 name: <name of the vo>
 - auth:
 project_id: <local OpenStack project identifier for second VO>
 name: <name of another vo>

You can create pull requests for adding and maintaining your site configuration to the catch-all operations. As soon as the pull request is merged, a new deployment will be triggered and your site should start publishing information and syncing images.

The catch-all operations uses a service account user, this user is member of the ops VO and should be enabled to manage VM images in glance.

Opting out of catch-all operations

You can completely opt-out from the catch-all operations by removing your site configuration from the repository. However, this will make your site not discoverable by fedcloudclient or Infrastructure Manager.

You can also disable the image sync by setting images.sync to false or by completely removing the images configuration. In that case, monitoring probes may fall as they rely on the images being present at the site.

Individual components are available in GitHub if you want to operate them by your own:

Providers: GPU flavours

Mon, 01 Jan 0001 00:00:00 +0000

Setting up GPU flavours

Support for GPU can be added to flavours using the PCI passthrough feature in OpenStack. This allows to plug any kind of PCI device to the Virtual Machines.

As a summary of the OpenStack documentation, these are the steps needed to add a GPU enabled flavour (be aware this may need tuning to your specific hardware/configuration!):

On computing node, get vendor/product ID of your hardware: lspci | grep NVIDIA to get pci slot of GPU, then virsh nodedev-dumpxml pci_xxxx_xx_xx_x
On computing node, unbind device from host kernel driver. Unbinding is system dependent, and can be done in many ways, e.g.:
- if the kernel does not uses the devices (no GPU drivers included in kernel, or drivers disable in GRUB), nothing to unbind
- via pci-stub grubby --args="pci-stub.ids=10de:11fa" --update-kernel DEFAULT (see RedHat manual, section 12.1, step 1-2; where the pci-stub.ids value is vendor_ID: product_id from lspci.
- via echo command: echo $dev > /sys/bus/pci/devices/$dev/driver/unbind where $dev is the PCI device ID xx:xx.x or xxxx:xx:xx.x from lspci
On computing node, add pci_passthrough_whitelist = {"vendor_id":"xxxx","product_id":"xxxx"} to nova.conf (see nova-compute)
On controller node, add pci_alias = {"vendor_id":"xxxx","product_id":"xxxx", "name":"GPU"} to nova.conf (see nova-api)
On controller node, enable PciPassthroughFilter in the scheduler (see nova-scheduler)
Create new flavours with pci_passthrough:alias (or add key to existing flavour), e.g. openstack flavor set m1.large --property "pci_passthrough:alias"="GPU:2"

GPU description in flavour metadata

Users should be able to easily discover the flavours that provide GPUs (or accelerators in general). The following table describes the agreed metadata for EGI providers to add to those flavours:

Metadata	Definition	Comments
Accelerator:Type	Type of accelerator (e.g. `GPU`)	Possible values: `GPU`, `MIC`, `FPGA`, `TPU`, `NPU`
Accelerator:Number	Number of accelerators available in the flavour (e.g. `1.0`)	Non integers allowed for the case of sharing GPU between VMs
Accelerator:Vendor	Name of accelerator Vendor (e.g. `NVIDIA`)
Accelerator:Model	Model of accelerator (e.g. `Tesla V100`)	Need to make consensus and enforce. A100 is usually marketed without “Tesla” class name. Similarly, RTX A6000 usually marketed without “GeForce”. For clarity, full names should be used: “Tesla A100” and “GeForce RTX A6000”
Accelerator:Version	Version of the accelerator	Some cards have different versions, e.g. A100 PCIe and NVLink. Openstack does not allow empty value, so we should give 0 if no version is specified
Accelerator:Memory	RAM in GB of the accelerator
Accelerator:VirtualizationType	Type of virtualisation used (e.g. `PCI passthrough`)	Not relevant for accounting, but may be still useful in some cases

There are some extra fields that are defined in the GLUE2.1 schema but not so relevant for GPUs and therefore not considered at the moment. These are listed below for completeness:

Metadata	Definition	Comments
Accelerator:ComputeCapability	Compute capabilities	Defined by GLUE2.1, e.g. floating point type, NVLink, … may be used informally so far
Accelerator:ClockSpeed	Clockspeed of accelerator	Defined by GLUE2.1, not so relevant, as ClockSpeed no longer related to performance. May be reserved for other types of accelerators
Accelerator:Cores	Number of cores of the accelerator	Not so useful as there are several types of cores now (CUDA, tensor). May be reserved for other types of accelerators

Adding metadata to flavours has no effects on site operations. End users can see the metadata easily via openstack flavor list --long or openstack flavor show <flavor id> commands without any additional tools, e.g.:

$ fedcloud openstack flavor show gpu1cpu2 --site IISAS-GPUCloud --vo eosc-synergy.eu -f json
Site: IISAS-GPUCloud, VO: eosc-synergy.eu
{
 "OS-FLV-DISABLED:disabled": false,
 "OS-FLV-EXT-DATA:ephemeral": 0,
 "access_project_ids": null,
 "disk": 40,
 "id": "a8082202-f647-4d1f-9b97-4f5ddb38ae8e",
 "name": "gpu1cpu2",
 "os-flavor-access:is_public": false,
 "properties": "Accelerator:Version='0', Accelerator:Memory='5', Accelerator:Model='Tesla K20m', Accelerator:Number='1.0', Accelerator:Type='GPU', Accelerator:Vendor='NVIDIA', Accelerator:VirtualizationType='PCI passthrough', pci_passthrough:alias='GPU:1'",
 "ram": 8192,
 "rxtx_factor": 1.0,
 "swap": "",
 "vcpus": 2
}

Providers: VO Configuration guide

Mon, 01 Jan 0001 00:00:00 +0000

In this page you can find a summary of the needed steps for supporting a new VO in your OpenStack infrastructure.

Local project creation

The usual method of supporting a VO is by creating a local project for it. You should assign quotas to this project as agreed in the OLA defining the support for the given VO.

Create a group where users belonging to the VO will be mapped to:
```
group_id=$(openstack group create -f value -c id <new_group>)
```

Add that group to the desired local project:

$ openstack role add member --group $group_id --project <your project>

Set the egi.VO property to the name of the VO that you are supporting:

$ openstack project set --property egi.VO=<name of the VO> <your project>

Keystone Mapping

Expand your mapping.json with the VO membership to the created group (substitute group_id and entitlement as appropriate). The expected mappings for the VOs are listed in vo-mappings.yaml of fedcloud-catchall-operations repository:

[
 <existing mappings>,
 {
 "local": [
 {
 "user": {
 "name": "{0}",
 "email": "{1}"
 },
 "group": {
 "id": "<group_id>"
 }
 }
 ],
 "remote": [
 {
 "type": "HTTP_OIDC_SUB"
 },
 {
 "type": "HTTP_OIDC_EMAIL"
 },
 {
 "type": "HTTP_OIDC_ISS",
 "any_one_of": [
 "https://aai.egi.eu/auth/realms/egi"
 ]
 },
 {
 "type": "OIDC-eduperson_entitlement",
 "regex": true,
 "any_one_of": [
 "^<entitlement>$"
 ]
 }
 ]
 }
]

And update the mapping in your Keystone IdP:

$ openstack mapping set --rules mapping.json egi-mapping

You can include as many mappings as needed in the json file. Users will be members of all the matching groups.

Accounting

Add the project supporting the VO to cASO:

In the projects field of /etc/caso/caso.conf :

projects = vo_project1, vo_project2, <your_new_vo_project>

and as a new mapping in /etc/caso/voms.json :

{
 "<your new vo>": {
 "projects": ["<your new vo project>"]
 }
}

Be sure to include the user running cASO at least as reader of the project if it does not have admin privileges:

openstack role add --user <your caso user> --project <your new vo project> reader

Information system / VM Image management

If you are correctly setting the egi.VO property to your projects, the configuration will be automatically retrieved by the catch-all components.

Providers: Useful Resources

Mon, 01 Jan 0001 00:00:00 +0000

In this page we gather a list of useful resources for OpenStack administrators.

OpenStack IDLE VM Detector (osidle)

Our partners from GRyCAP UPV have created a tool to analyze the usage of Virtual Machines running in OpenStack. For more information please visit: https://github.com/grycap/osidle/.

Would you like to share your favourite tool to make OpenStack management easier? Then contact us!

Documentation – OpenStack

Providers: Check-in

Registration into Check-in demo instance

Keystone setup

Pre-requisites

IGTF CAs

Apache Configuration

Note

Multiple OIDC providers

mod_auth_openidc configuration

ESACO configuration

Horizon configuration

Keystone Configuration

Keystone Federation Support

Note

VO auditing

Additional VOs

Horizon Configuration

CLI Access

Moving to EGI Check-in production instance

Changes in the client settings

Additional VOs

Providers: Accounting

Installation

Configuration

Running the services

Providers: Catch-all components

BDII is deprecated

Opting out of catch-all operations

Providers: GPU flavours

Setting up GPU flavours

GPU description in flavour metadata

Providers: VO Configuration guide

Local project creation

Keystone Mapping

Accounting

Information system / VM Image management

Providers: Useful Resources

OpenStack IDLE VM Detector (osidle)

Share yours

`mod_auth_openidc` configuration