Documentation – Cloud Compute

Providers: Requirements

Mon, 01 Jan 0001 00:00:00 +0000

Resource Centres are free to use any Cloud Management Platform as long as they are able to integrate with the EGI Federation components. At the moment this compliance is supported for OpenStack open-source cloud platform.

The general minimum requirements are described below.

Relevant parts of PROC09 Resource Centre Registration and Certification have been successfully completed
Hardware requirements greatly depend on your cloud infrastructure, EGI components in general do lightweight operations by interacting with your services APIs.
- Image sync requires enough space to store the community images into your local catalogue. The number and size of images which will be downloaded depends on the communities you plan to support. For the operations VO ops, a < 4GB image is used.
Servers need X.509 host certificates in order to authenticate to each other or to act as public endpoints in the EGI Federated Cloud context. For accounting purposes one IGTF-accredited X.509 certificate is needed per site, but the other public endpoints can use ordinary certificates (issued by Let’s Encrypt for example).
- For the IGTF-accredited certificates, a list of national / regional Certificate Authorities is available at EUGridPMA Membership; it is expected that the Resource Centre will obtain the IGTF certificate from a close Certificate Authority
For operational purposes, the ops VO needs to be supported by the Resource Centre as per Resource Centre OLA. Also at least one community VO that supports EGI users is expected to be supported by the Resource Centre (i.e. vo.access.egi.eu for piloting and prototyping FedCloud usage).
The public endpoints need to have proper firewall configuration (to allow inbound external access)
A flavor (a preset configuration that defines the compute, memory, and storage capacity - min 8 GB - of an instance) needs to be made available by the site for monitoring purposes.

Providers: GOCDB Registration

Mon, 01 Jan 0001 00:00:00 +0000

Site endpoints must be registered in EGI Configuration Management Database (GOCDB). Cloud services can coexist within an existing (Grid) site, but we are recommending to register a new site for the cloud services (as per PROC09 Resource Centre Registration and Certification procedure - see Joining as a provider section)

Expected Services

These are the expected services for a working site:

org.openstack.nova for the Nova endpoint of the site. The endpoint URL must contain the Keystone v3 URL: https://hostname:port/url/v3. Set the Host DN so the cloud-info-provider can be enabled.
eu.egi.cloud.accounting for the host sending the records to the accounting repository (executing SSM send). This host needs a valid IGTF-accredited X.509 certificate. Set the Host DN so the SSM can be enabled.
org.openstack.horizon for the dashboard endpoint of the site (optional). The endpoint URL field must contain the horizon URL: https://hostname:port/url/.
(Optional, if offering object storage) org.openstack.swift for the swift endpoint of the site. The endpoint URL field must contain the Keystone v3 URL: https://hostname:port/url/v3.

Deprecated services

Deprecated services for cloud providers:

Site-BDII. This service collects and publishes site's data for the Information System. There is no need to run a BDII for cloud providers. eu.egi.cloud.information.bdii is also deprecated and no longer in use.
eu.egi.cloud.vm-metadata.vmcatcher for the VMI replication mechanism.
eu.egi.cloud.vm-management.occi for the OCCI endpoint offered by the site. OCCI is no longer in use in FedCloud.

Providers: OpenStack

Mon, 01 Jan 0001 00:00:00 +0000

This section provides information on how to set up a Resource Centre providing cloud resources in the EGI infrastructure. Integration with FedCloud requires a working OpenStack installation as a pre-requirement. EGI supports any recent OpenStack version (tested from OpenStack Mitaka).

The following OpenStack services are expected to be available and accessible from outside the site:

Keystone
Nova
Cinder
Glance
Neutron
Swift (if providing Object Storage)

The integration is performed by a set of EGI components that interact with the OpenStack services APIs:

Authentication and authorization of EGI users into your system is performed by configuring the native OpenID Connect support of Keystone
cASO collects accounting data from OpenStack and uses SSM to send the records to the central accounting database on the EGI Accounting service (APEL). cASO and SSM are operated by the site.
EGI runs catch-all components – cloud-info-provider and cloudkeeper – information discovery and VM image synchronisation.

Installation options

cASO and SSM releases can be be obtained from GitHub:

cASO. cASO containers are available in the fedcloud-caso package from fedcloud-catchall-operations
SSM.

Open Ports

Inbound

The following services must be accessible to allow access to an OpenStack-based FedCloud site (default ports listed below, can be adjusted to your installation).

Port	Application	Note
5000/TCP	OpenStack/Keystone	Authentication to your OpenStack.
8776/TCP	OpenStack/cinder	Block Storage management.
8774/TCP	OpenStack/nova	VM management.
9696/TCP	OpenStack/neutron	Network management.
9292/TCP	OpenStack/glance	VM Image management.

Outbound

The EGI Cloud components require the following outgoing connections open:

Port	Host	Note
443/TCP	`msg.argo.grnet.gr`	ARGO Messaging System (used to send accounting records by SSM).
8443/TCP	`msg.argo.grnet.gr`	AMS authentication (used to send accounting records by SSM).

Users

Local Users

In order to get accounting information from your OpenStack, cASO needs to be run with a user that is a member of the projects to extract accounting information from and it’s allowed to access identity:list_users and identity:list_projects in Keystone. Check cASO documentation for further information.

Federated Users

Regular user accounts will be managed by the Federated Identity features of OpenStack. These users are created into a specific OpenStack domain for every configured identity provider. All users within the egi.eu domain will have a unique username. For users whose community identity is managed by Check-in, this identifier is of the form <uniqueID>@egi.eu. The <uniqueID> portion is an opaque identifier issued by Check-in, for example:

$ openstack domain list
+----------------------------------+----------------------------------+---------+---------------------------------------------------------------+
| ID | Name | Enabled | Description |
+----------------------------------+----------------------------------+---------+---------------------------------------------------------------+
| 0125ed0ebc8045a49ed8c34c2a78740d | 0125ed0ebc8045a49ed8c34c2a78740d | True | Auto generated federated domain for Identity Provider: egi.eu |
| default | Default | True | The default domain |
+----------------------------------+----------------------------------+---------+---------------------------------------------------------------+

$ openstack user list --domain 0125ed0ebc8045a49ed8c34c2a78740d
+------------------------------------------------------------------+-------------------------------------------------------------------------+
| ID | Name |
+------------------------------------------------------------------+-------------------------------------------------------------------------+
| 2c096b11a1410d44e3936fa40479ad26eaa649cfd6887f06b3c6669e5d6c03d0 | efb8534478028XXXXXXXXXXXXXXXfeed9766fafc@sram.surf.nl |
| 933c692b53192e4d893e5ed5c026aa444acb4d75f6ee6c304422861207ce1ea5 | e9c37aa0d1XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX2867bc43581b835c@egi.eu |
| d52112709a37975903576f80f37dde4604d1a227c53cb1fef43c45981673640c | 529a87e5ceXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXe714cb1309cc3907@egi.eu |
+------------------------------------------------------------------+-------------------------------------------------------------------------+

If you have set the email of the user in the mapping, you will be able to also get this information:

$ openstack user show d52112709a37975903576f80f37dde4604d1a227c53cb1fef43c45981673640c
+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+
| domain_id | 0125ed0ebc8045a49ed8c34c2a78740d |
| email | XXXX-redacted@example.com |
| enabled | True |
| federated | [{'idp_id': 'egi.eu', 'protocols': [{'protocol_id': 'openid', 'unique_id': '529a87e5ceXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXe714cb1309cc3907%40egi.eu'}]}] |
| id | d52112709a37975903576f80f37dde4604d1a227c53cb1fef43c45981673640c |
| name | 529a87e5ceXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXe714cb1309cc3907@egi.eu |
| options | {} |
| password_expires_at | None |
+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+

Every VO has a VO identity card available via the Operations Portal, where you can also get contact information for the VO managers.

VMs created by EGI’s Infrastructure Manager have additional metadata properties that can help to identify the workload:

$ openstack server show 0f3e1420-4480-4bea-95f1-9920a70b324d -c properties -f yaml
properties:
 eu.egi.cloud.orchestrator: es.upv.grycap.im
 eu.egi.cloud.orchestrator.id: 0afdc7ba-bf5d-11ed-9e89-86ce117c3fcf
 eu.egi.cloud.orchestrator.url: https://im.egi.eu/im
 eu.egi.cloud.orchestrator.user: __OPENID__XXXXXXredacted

Providers: Installation Validation

Mon, 01 Jan 0001 00:00:00 +0000

Once the site services are registered in GOCDB (and flagged as "monitored") they will appear in the EGI service monitoring tools. EGI will check the status of the services. Check if your services are present in the EGI service monitoring tools and passing the tests; if you experience any issues (services not shown, services are not OK...) please contact back EGI Operations or your reference Resource Infrastructure.

Extra checks for your installation:

Check in ARGO that your services are listed and are passing the tests. If all the tests are OK, your installation is already in good shape.
Check that all the images listed in the AppDB for the VOs you support (e.g. AppDB page for vo.access.egi.eu VO) are listed in your BDII. This sample query will return all the template IDs registered in your BDII: :
```
ldapsearch -x -h <site bdii host> -p 2170 -b Glue2GroupID=cloud,Glue2DomainID=<your site name>,o=glue objectClass=GLUE2ApplicationEnvironment GLUE2ApplicationEnvironmentRepository
```
Try to start one of those images in your cloud. You can do it with [onetemplate instantiate]{.title-ref} or OCCI commands, the result should be the same.
Execute the site certification manual tests against your endpoints.
Check in the accounting portal that your site is listed and the values reported look consistent with the usage of your site.

Providers: FAQ

Mon, 01 Jan 0001 00:00:00 +0000

Why joining the EGI Cloud?

To support international communities supported by EGI (e.g. research communities and applications or research infrastructures or business pilots in the EOSC Digital Innovation Hub.
To participate in e-Infrastructure projects (H2020, EOSC) as an EGI compliant IaaS cloud provider.
To participate in resource allocation and in pay-for-use campaigns run by EGI.
To align access policies and operational model of your cloud with international good practices.
To adopt best practices of multi-cloud federation for the benefit of your local users.

Do I lose control on who can access my resources if I join federated cloud?

No. EGI uses the concept of Virtual Organisation (VO) to group users. The resource provider has complete control on which VOs he wants to allow on its resources and which quotas or restrictions to assign to each VO. In the case of OpenStack, each VO is mapped to a regular OpenStack project that can be managed as any other and are isolated to other projects you may have configured in your deployment. Although not recommended, you can even restrict the automatic access of users within a VO and manually enable individual members.

How many components do I have to install?

Depending on your cloud management framework and the kind of integration this will vary.

In general, the federation requires your cloud management framework to be configured to support Federated AAI with EGI Check-in. This may require changes in your current setup.

Other components are designed to access your cloud management framework public APIs and do not require modification of your deployment. For OpenStack, these components can be run on a single VM that encapsulates them for convenience.

Which components of my cloud will interact with the federated cloud components?

For OpenStack they are:

Keystone
Nova
Glance
Swift (optional)

Users will also interact with:

Neutron
Cinder

to perform their regular activities.

How will my daily operational activities change?

For the most part daily operations will not change.

A resource centre part of the EGI Federation, and supporting international communities, needs to provide support through the EGI channels. This means following up GGUS tickets. This includes requests from user communities and tickets triggered by failures detected by the monitoring infrastructure.

A resource centre needs to maintain the services federated in EGI properly configured with the EGI AAI.

The resource centre will have to comply with the operational and security requirements. All the EGI policies aim at implementing service provisioning best practices and common requirements. EGI operations may conduct campaigns targeted to mitigate security vulnerabilities and to update unsupported operating system and software. These activities are part of the regular activities of a resource centre anyway (also for the non-federated ones). EGI and the Operations Centres coordinate these actions in order to have them implemented in a timely manner.

In summary, most of the site activities that are coordinated by EGI and the NGIs are already part of the work plan of a well-maintained resource centre, the additional task for a site manager is to acknowledge to EGI that the task has been performed.