Documentation – Infrastructure Manager

Users: Elastic Cloud Compute Clusters

Mon, 01 Jan 0001 00:00:00 +0000

What is it?

Elastic Cloud Compute Cluster (EC3) is a tool to create elastic virtual clusters on top of Infrastructure-as-a-Service (IaaS) providers.

Being based on Infrastructure Manager, EC3 supports the same wide choices of backends, either public (such as Amazon Web Services, Google Cloud or Microsoft Azure) or on-premises (such as OpenStack). EC3 can provision clusters running TORQUE, SLURM, HTCondor, Apache Mesos, Nomad, Kubernetes and others, which will be automatically resized to fit the load (e.g. number of jobs at the batch system).

Note

EC3 was presented in one of the EGI Webinars. Please see more details on the Indico page and check out the video recording on YouTube.

The following section of the documentation will guide you on how to:

Deploy a simple EC3 elastic cluster on top of the IaaS providers of the EGI Cloud, either using the web interface or the command-line interface, and
Run pre-configured scientific applications in the EC3 elastic cluster.

Users: Infrastructure Manager Command-Line Interface

Mon, 01 Jan 0001 00:00:00 +0000

You can find here documentation covering getting started with IM Command-Line Interface (CLI) on EGI Cloud Compute sites. Full documentation at IM CLI documentation

Getting started

Install with pip

You only have to call the install command of the pip tool with the IM-client package.

$ pip install IM-client

IM-Client Docker image

The IM Client has an official Docker container image available on Docker Hub that can be used instead of installing the CLI. You can use it by typing:

$ docker run --rm -ti -v "$PWD:/tmp/im" grycap/im-client \
 -r https://server.com:8800 -a /tmp/im/auth.dat list

Configuration

To avoid typing the parameters in all the client calls, the user can define a config file im_client.cfg in the current directory or a file .im_client.cfg in their home directory. In the config file the user can specify the following parameters:

[im_client]
restapi_url=https://im.egi.eu/im
auth_file=auth.dat

Authentication data file

An authentication file must be created to access the IM service. It must have one line per authentication element. It must have at least one line with the authentication data for the IM service and another one for the Cloud provider(s) the user want to access. Each line of the file is composed by pairs of key and value separated by semicolon, and refers to a single credential. The key and value should be separated by =, that is an equals sign preceded and followed by one whitespace at least. The following lines show the credentials needed to access an EGI Cloud Compute site:

type = InfrastructureManager; token = egi_aai_token_value
id = egi; type = EGI; host = CESGA; vo = vo.access.egi.eu; token = egi_aai_token_value

The value of egi_aai_token_value must be replaced with a valid EGI Check-in access token. Users of EGI Check-in can get all the information needed to obtain access tokens, by visiting EGI Check-in Token Portal.

oidc-agent can be used to get a valid access token:

id = im; type = InfrastructureManager; token = command(oidc-token OIDC_ACCOUNT)
id = egi; type = EGI; host = SCAI; vo = vo.access.egi.eu; token = command(oidc-token OIDC_ACCOUNT)

Create and Manage an infrastructure

To create a virtual infrastructure you have to describe a file documenting the required resources. IM supports its native language RADL and the OASIS TOSCA Simple Profile in YAML Version 1.0. You can find some examples in the IM GitHub repository.

For example, we can use RADL to define a simple VM with 1 CPU, 1 GB of RAM using the EGI Ubuntu 20.04 image.

network public (outbound = 'yes')

system node (
cpu.count>=2 and
memory.size>=4g and
net_interface.0.connection = 'public' and
disk.0.os.name='linux' and
disk.0.image.url = 'egi://IN2P3-IRES/egi_vm_images/ubuntu:22.04?vo.access.egi.eu'
)

configure wn (
@begin
---
 - tasks:
 - debug: msg="Configured!"
@end
)

deploy node 1

IM also supports TOSCA. For example this is an equivalent TOSCA document to deploy a single VM:

tosca_definitions_version: tosca_simple_yaml_1_0

imports:
- indigo_custom_types: https://raw.githubusercontent.com/indigo-dc/tosca-types/master/custom_types.yaml

topology_template:

 node_templates:

 simple_node:
 type: tosca.nodes.indigo.Compute
 capabilities:
 endpoint:
 properties:
 network_name: PUBLIC
 host:
 properties:
 num_cpus: 2
 mem_size: 4 GB
 os:
 properties:
 image: egi://IN2P3-IRES/egi_vm_images/ubuntu:22.04?vo.access.egi.eu

 outputs:
 node_ip:
 value: { get_attribute: [ simple_node, public_address, 0 ] }
 node_creds:
 value: { get_attribute: [ simple_node, endpoint, credential, 0 ] }

Then we can call the create operation of the IM client tool using a RADL or a TOSCA YAML file:

$ im_client.py create infra.radl
Secure connection with: https://im.egi.eu/im
Infrastructure successfully created with ID: 457273ea-85e4-11ec-aa81-faaae69bc911

Then we can call get the current state of infrastructure using the getstate operation of the IM client tool:

$ im_client.py getstate 457273ea-85e4-11ec-aa81-faaae69bc911
Secure connection with: https://im.egi.eu/im
The infrastructure is in state: pending
VM ID: 0 is in state: pending.

The valid VM and infrastructure states are the following:

pending, launched, but still in initialization stage;
configuring, created successfully and running, but still in the configuration stage;
configured, running and contextualized;
unconfigured, running but not correctly contextualized;
stopped, stopped or suspended;
off, shutdown or removed from the infrastructure;
failed, an error happened during the launching; or
unknown, unable to obtain the status.
deleting, in the deletion process.

Once the configuration step has started we can get the output of the Ansible process using the getcontmsg operation:

$ im_client.py getcontmsg 457273ea-85e4-11ec-aa81-faaae69bc911
Secure connection with: https://im.egi.eu/im
Connected with: http://localhost:8800
Msg Contextualizator:

2022-02-11 10:40:12.768523: Copying YAML, hosts and inventory files.
VM 0:
Contextualization agent output processed successfullyGenerate and copy the ssh key
Sleeping 0 secs.
Launch task: wait_all_ssh
...

Once the VM is booted we can access it via SSH using the ssh operation:

$ im_client.py ssh 457273ea-85e4-11ec-aa81-faaae69bc911

When using a TOSCA yaml document to create the infrastructure, we can get the TOSCA output values with the getoutputs operation:

$ im_client.py getoutputs 457273ea-85e4-11ec-aa81-faaae69bc911
Secure connection with: https://im.egi.eu/im
The infrastructure outputs:

node_ip = 8.8.8.8
node_creds = {'token': '...', 'user': 'cloudadm', 'token_type': 'private_key'}

Once we no more need the Infrastructure, we can destroy it using the destroy operation:

$ im_client.py destroy 457273ea-85e4-11ec-aa81-faaae69bc911
Secure connection with: https://im.egi.eu/im
Infrastructure successfully destroyed

Users: Infrastructure Manager Dashboard

Mon, 01 Jan 0001 00:00:00 +0000

The IM Dashboard is a graphical interface for the IM Server specially developed for users to access EGI Cloud Compute resources.

You can access EGI’s IM Dashboard at: https://im.egi.eu

Functionalities:

OIDC authentication
Display user’s infrastructures
Display infrastructure details, template and log
Delete infrastructure
Create new infrastructure
Add nodes to an infrastructure
Resize VMs

Tip

More details about installing and configuring the dashboard are available in the IM Dashboard documentation.

Usage

The dashboard of the IM enables non advanced users to manage their infrastructures by launching a set of predefined TOSCA templates on top of EGI Cloud Compute resources. The dashboard does not provide all the features provided by the IM service. In case you need more advanced features use the IM Web interface or the IM-CLI.

Users must use EGI Check-in to log into the dashboard. Once authenticated, they will be redirected to the portfolio of available TOSCA templates.

The main menu bar is located at the top of the pages:

The first button IM Dashboard enables the user to go to the portfolio of available TOSCA templates.
Second item Infrastructures redirects to the list of current user deployed infrastructures.
In the Advanced item the Settings sub-item displays some configuration settings as the URL of the IM service or the OIDC issuer.
External Links show a set of configurable information links (documentation, video tutorials, etc.)
Finally, on the right top corner, appears the User menu item. This item shows the full name of the logged user, and an avatar obtained from Gravatar. In this menu the user can access their Cloud Credentials with the cloud providers or log out from the application.

Cloud Credentials

To be able to access any Cloud site the user must specify the credentials to access them. This page allows the user to specify the credentials for accessing any cloud provider. In the list the user can edit, delete and enable or disable the selected cloud credentials.

Editing or adding the credentials will show a modal form where the user has the ability to specify all the parameters needed to access the supported cloud providers. In particular, for Cloud Compute sites the user only has to select one of the VOs he is member of and one of sites that supports that VO. These drop-down fields are generated using the information available from the sites and the list of VOs the user is member of.

TOSCA Templates

The list of available TOSCA templates enables the user to select the required topology to deploy. Each TOSCA template can be labelled by the TOSCA developer with any tag that will show a ribbon displayed on the right bottom corner. The special elastic tag is used to mark templates that are configured to automatically manage the elasticity of the deployed cluster.

The user must click on the Configure button to set the input values of the TOSCA template and select the VO, Site and Image to use for deploying the infrastructure.

Initially the user can set a name to describe the infrastructure to be deployed. It will simplify identifying infrastructures. In the firsts tabs, the user can introduce the set of input values of the topology. By default there is only one tab called Input Values, but the TOSCA developer can add or rename them to simplify the selection of input values.

The final tab will be the Cloud Provider Selection. In this tab the user first has to select one of the Cloud providers that has been previously added (and not disabled) in the Cloud Credentials page, then has to select the base image used to deploy the VMs. In case of EGI Cloud Compute sites, the user has two options: he can select an image from the list of images provided by the cloud information system or from the list provided directly by the Cloud site.

Other providers will only show a drop-down list with the available images. Only in the case of AWS Cloud provider the user has to specify manually the AMI ID of the image.

Infrastructures

This page lists the infrastructures deployed by the user. The first column shows the name set by the user on infrastructure creation, then shows the ID assigned by the IM service, the third column shows the current status of the infrastructure, the fourth show the list of VMs with their IDs and finally a button with a set of actions appears.

List of Actions

The following figure shows the list of actions available to manage the existing infrastructures:

Add nodes: The Add nodes action enables to add new VMs to the users' deployment. It will show the list of the different types of nodes currently deployed in the infrastructure and the user must set the number of nodes of each type he wants to deploy. It will also show a dropdown list with the available base images. It will enable changing the base image used to deploy the new nodes. In some cases it will be necessary because the original one has been removed.
Show template: This action shows the original TOSCA template submitted to create the infrastructure.
Log: Shows the error/contextualization log of the infrastructure.
Stop: Stops/Suspends all the VMs of the infrastructure.
Start: Starts/Resumes a previously stopped infrastructure.
Outputs: Shows the outputs of the TOSCA template. Private key of credentials can be downloaded as a file or copied to the clipboard.
Delete: Delete this infrastructure and all the associated resources. It also has the option to force the deletion. In this case the infrastructure will be removed from the IM service even if some cloud resources cannot be deleted. Only use this option if you know what you are doing.
Delete & Recreate: Delete this infrastructure as the previous option, but once it is deleted it will redirect to the infrastructure creation form, with all the input fields filled with the same set of values used to create the deleted infrastructure.
Reconfigure: Starts the reconfiguration of the infrastructure.
Change User: Add or change the ownership of the infrastructure at IM level. Providing a valid Access Token of another user, the infrastructure can be shared or transferred to them. If Overwrite is checked, the new user will be the unique owner of the infrastructure (transferring), otherwise it will be added to the list of current users (sharing). The new user must be member of the VO used to create the resources, otherwise he will not be able to manage them. The list of current owners of the infrastructure will also be displayed.

VM Info page

The VM Info page shows all the information about the selected VM and enables to manage the lifecycle of it. On the top right corner the Manage VM drop-down menu allows to Stop/Start, Reboot, Resize, Reconfigure and Terminate the VM. Furthermore the user can check the error or contextualisation log of this particular VM.

The VM information is split in two different tables, the first one with the main information: State, IPs, HW features and the SSH credentials needed to access it. The second table shows additional fields.

When resizing the VM the user must provide the new size of the VM in terms of number of CPUs and amount of memory as shown in the figure below:

Documentation – Infrastructure Manager

Users: Elastic Cloud Compute Clusters

What is it?

Note

Users: Infrastructure Manager Command-Line Interface

Getting started

Install with pip

IM-Client Docker image

Configuration

Authentication data file

Create and Manage an infrastructure

Users: Infrastructure Manager Dashboard

Tip

Usage

Login

Main menu bar

Cloud Credentials

TOSCA Templates

Infrastructures

List of Actions

VM Info page