Command Line Interface

EGI command line interface

Command line tools

The various public EGI services can be managed and used/accessed with a wide variety of command line interface (CLI) tools. The documentation of each service contains a summary of the CLIs that can be used wih that service, together with recommendations on which one to use in what context.

The FedCloud client

The FedCloud client is a high-level Python package for a command-line client designed for interaction with the OpenStack services in the EGI infrastructure.

FedCloud client has the following modules (features):

Installation

The FedCloud client can be installed with the pip3 Python package manager (without root or aministrator privileges).

To install the FedCloud client:

$ pip3 install fedcloudclient

This installs the latest version of the FedCloud client, together with its required packages (like openstackclient). It will also create executables fedcloud and openstack, adding them to the bin folder corresponding to your current Python execution environment ($VIRTUAL_ENV/bin for executing pip3 in a Python virtual environment, ~/.local/bin for executing pip3 as user (with --user option), and /usr/local/bin when executing pip3 as root).

As there are non-pure Python packages needed for installation, the Microsoft C++ Build Tools is a prerequisite, make sure it’s installed with the following options selected:

  • C++ CMake tools for Windows
  • C++ ATL for latest v142 build tools (x86 & x64)
  • Testing tools core features - Build Tools
  • Windows 10 SDK (<latest>)

In case you prefer to use non-Microsoft alternatives for building non-pure packages, please see here.

To install the FedCloud client:

> pip3 install fedcloudclient

This installs the latest version of the FedCloud client, together with its required packages (like openstackclient). It will also create executables fedcloud and openstack, adding them to the bin folder corresponding to your current Python execution environment.

Check if the installation is correct by executing the client:

$ fedcloud --version

Installing EGI Core Trust Anchor certificates

Some sites in the EGI infrastructure use certificates issued by Certificate Authorities (CAs) that are not included in the default OS distribution. If you receive error message “SSL exception connecting to…”, install the EGI Core Trust Anchor Certificates by running the following commands:

$ wget https://raw.githubusercontent.com/tdviet/python-requests-bundle-certs/main/scripts/install_certs.sh
$ bash install_certs.sh

Using via Docker container

The FedCloud client can also be used without installation, by running it in a Docker container. In this case, the EGI Core Trust Anchor certificates are preinstalled.

To run the FedCloud client in a container, make sure Docker is installed, then run the following commands:

$ docker pull tdviet/fedcloudclient
$ docker run -it tdviet/fedcloudclient bash

To run the FedCloud client in a container, make sure Docker is installed, then run the following commands:

$ docker pull tdviet/fedcloudclient
$ docker run -it tdviet/fedcloudclient bash

To run the FedCloud client in a container, make sure Docker is installed, then run the following commands:

> docker pull tdviet/fedcloudclient
> docker run -it tdviet/fedcloudclient bash

Once you have a shell running in the container with the FedCloud client, usage is the same as from the command line.

Using from EGI Notebooks

EGI Notebooks are integrated with access tokens so it simplifies using the FedCloud client. First make sure that you follow the installation steps above. Then, below are the commands that you need to run inside a terminal in JupyterLab:

export OIDC_ACCESS_TOKEN=`cat /var/run/secrets/egi.eu/access_token`
fedcloud token check

Please follow instructions below to learn how to use the fedcloud command.

Using from the command line

The FedCloud client has these subcommands:

  • fedcloud token for checking access tokens (see token subcommands)
  • fedcloud endpoint for querying the Configuration Database (see endpoint subcommands)
  • fedcloud site for manipulating site configurations (see site subcommands)
  • fedcloud openstack or fedcloud openstack-int for performing OpenStack commands on sites (see openstack subcommands)
  • fedcloud ec3 for provisioning elastic cloud compute clusters (see cluster subcommands)

Performing any OpenStack command on any site requires only three options: the site, the VO and the command. For example, to list virtual machine (VM) images available to members of VO fedcloud.egi.eu on the site CYFRONET-CLOUD, run the following command:

$ fedcloud openstack image list --vo fedcloud.egi.eu --site CYFRONET-CLOUD

Authentication

Many of the FedCloud client commands need access tokens for authentication. Users can choose whether to provide access tokens directly (via option --oidc-access-token), or generate them on the fly with oidc-agent (via option --oidc-agent-account) or from refresh tokens (via option --oidc-refresh-token, which must be provided together with option --oidc-client-id and option --oidc-client-secret).

If multiple methods of getting access tokens are given at the same time, the FedCloud client will try to get an access token from the oidc-agent first, then obtain one using the refresh token.

The default authentication protocol is openid. Users can change the default protocol via the option --openstack-auth-protocol. However, sites may have the protocol fixed in the site configuration (e.g. oidc for the site INFN-CLOUD-BARI).

The default OIDC identity provider is EGI Check-in (https://aai.egi.eu/oidc). Users can set another OIDC identity provider via option --oidc-url.

Environment variables

Most of the FedCloud client options can be set via environment variables:

Environment variableCommand line optionDefault value
OIDC_AGENT_ACCOUNT--oidc-agent-account
OIDC_ACCESS_TOKEN--oidc-access-token
OIDC_REFRESH_TOKEN--oidc-refresh-token
OIDC_CLIENT_ID--oidc-client-id
OIDC_CLIENT_SECRET--oidc-client-secret
OIDC_URL--oidc-urlhttps://aai.egi.eu/oidc
OPENSTACK_AUTH_PROTOCOL--openstack-auth-protocolopenid
OPENSTACK_AUTH_PROVIDER--openstack-auth-provideregi.eu
OPENSTACK_AUTH_TYPE--openstack-auth-typev3oidcaccesstoken
EGI_SITE--site
EGI_VO--vo

Getting help

The FedCloud client can display help for the commands and subcommands it supports. Try running the following command to see the commands supported by the FedCloud client:

$ fedcloud --help
Usage: fedcloud [OPTIONS] COMMAND [ARGS]...

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  ec3            EC3 related comands
  endpoint       endpoint command group for interaction with GOCDB and...
  openstack      Executing OpenStack commands on site and VO
  openstack-int  Interactive OpenStack client on site and VO
  site           Site command group for manipulation with site...
  token          Token command group for manipulation with tokens

Similarly, you can see help for e.g. the openstack subcommand by running the command below:

$ fedcloud openstack --help
Usage: fedcloud openstack [OPTIONS] OPENSTACK_COMMAND...

  Executing OpenStack commands on site and VO

Options:
  --oidc-client-id TEXT           OIDC client id
  --oidc-client-secret TEXT       OIDC client secret
  --oidc-refresh-token TEXT       OIDC refresh token
  --oidc-access-token TEXT        OIDC access token
  --oidc-url TEXT                 OIDC URL  [default: https://aai.egi.eu/oidc]
  --oidc-agent-account TEXT       short account name in oidc-agent
  --openstack-auth-protocol TEXT  Check-in protocol  [default: openid]
  --openstack-auth-type TEXT      Check-in authentication type  [default:
                                  v3oidcaccesstoken]
  --openstack-auth-provider TEXT  Check-in identity provider  [default:
                                  egi.eu]
  --site TEXT                     Name of the site or ALL_SITES  [required]
  --vo TEXT                       Name of the VO  [required]
  -i, --ignore-missing-vo         Ignore sites that do not support the VO
  -j, --json-output               Print output as a big JSON object
  --help                          Show this message and exit.

Using from Python

The FedCloud client can be used as a library for developing other services and tools for EGI services. Most of the functionalities can be called directly from Python code without side effects.

An usage example is available on GitHub. Just copy/download the code, add your access token and execute python demo.py to see how it works.

Using in scripts

The FedCloud client can also be used in scripts for simple automation, either for setting environment variables for other tools, or to process outputs from OpenStack commands.

Setting environment variables for external tools

Some FedCloud commands generate output that contains shell commands to set environment variables with the returned result, as exemplified below.

Run a command to get details of a project:

$ export EGI_SITE=IISAS-FedCloud
$ export EGI_VO=eosc-synergy.eu
$ fedcloud site show-project-id
export OS_AUTH_URL="https://cloud.ui.savba.sk:5000/v3/";
export OS_PROJECT_ID="51f736d36ce34b9ebdf196cfcabd24ee";

Run the same command but set environment variables with the returned values:

$ eval $(fedcloud site show-project-id)

The environment variables will have their values set to what the command returned:

$ echo $OS_AUTH_URL
https://cloud.ui.savba.sk:5000/v3/

$ echo $OS_PROJECT_ID
51f736d36ce34b9ebdf196cfcabd24ee

Run a command to get details of a project:

> set EGI_SITE=IISAS-FedCloud
> set EGI_VO=eosc-synergy.eu
> fedcloud site show-project-id
set OS_AUTH_URL=https://cloud.ui.savba.sk:5000/v3/
set OS_PROJECT_ID=51f736d36ce34b9ebdf196cfcabd24ee

If you copy the returned output and execute it as commands in a command prompt:

> set OS_AUTH_URL=https://cloud.ui.savba.sk:5000/v3/
> set OS_PROJECT_ID=51f736d36ce34b9ebdf196cfcabd24ee

The environment variables will have their values set to what the command returned:

> set OS_AUTH_URL
OS_AUTH_URL=https://cloud.ui.savba.sk:5000/v3/

> set OS_PROJECT_ID
OS_PROJECT_ID=51f736d36ce34b9ebdf196cfcabd24ee

Run a command to get details of a project:

> $Env:EGI_SITE = "IISAS-FedCloud"
> $Env:EGI_VO = "eosc-synergy.eu"
> fedcloud site show-project-id
$Env:OS_AUTH_URL="https://cloud.ui.savba.sk:5000/v3/";
$Env:OS_PROJECT_ID="51f736d36ce34b9ebdf196cfcabd24ee";

Run the same command but set environment variables with the returned values:

> fedcloud site show-project-id | Out-String | Invoke-Expression

The environment variables will have their values set to what the command returned:

> $Env:OS_AUTH_URL
https://cloud.ui.savba.sk:5000/v3/

> $Env:OS_PROJECT_ID
51f736d36ce34b9ebdf196cfcabd24ee

Processing output from OpenStack commands

The fedcloud openstack subcommand’s output can be converted to JavaScript Object Notation (JSON) format by using the --json-output option. This is useful for further machine processing of the command output.

$ export EGI_SITE=IISAS-FedCloud
$ export EGI_VO=eosc-synergy.eu
$ fedcloud openstack flavor list --json-output
[
{
  "Site": "IISAS-FedCloud",
  "VO": "eosc-synergy.eu",
  "command": "flavor list",
  "Exception": null,
  "Error code": 0,
  "Result": [
    {
      "ID": "0",
      "Name": "m1.nano",
      "RAM": 64,
      "Disk": 1,
      "Ephemeral": 0,
      "VCPUs": 1,
      "Is Public": true
    },
    {
      "ID": "2e562a51-8861-40d5-8fc9-2638bab4662c",
      "Name": "m1.xlarge",
      "RAM": 16384,
      "Disk": 40,
      "Ephemeral": 0,
      "VCPUs": 8,
      "Is Public": true
    },
    ...
  ]
}
]

# The following jq command selects flavors with VCPUs=2 and prints their names
$ fedcloud openstack flavor list--json-output | \
    jq -r '.[].Result[] | select(.VCPUs == 2) | .Name'
m1.medium