Documentation – EGI Secrets Store

Users: Secrets Store Web Interface

Mon, 01 Jan 0001 00:00:00 +0000

Although the recommended way to access and use EGI Secrets Store is via the command-line interface, the service can also be accessed via its web interface.

Tip

Tutorials about using the web interface are available on the Hashicorp Vault site.

Using the web interface

Follow the steps below to access the web interface of EGI Secrets Store:

Open https://secrets.egi.eu in your browser.
Choose the OIDC authentication method in the pull-down menu, then click Sign in with OIDC provider.
Login via EGI Check-in and authorize the Vault GUI.

The main page will open, with a top folder secrets that contains a space for each user.

Browse secrets

To browse and manage your secrets, click on Secrets in the top navigation bar, then click on folder secrets.

Each user has a private secret space, or a “home directory”, under the root folder secrets, with the user’s EGI Check-in ID as the folder name. You can only create secrets in your own secret space.

Note

Your “home directory” is not created automatically, you have to create it the first time you login using the web interface. Create a secret with your EGI Check-in ID as the path.

Tip

You can find your EGI Check-in ID via the user registry or by clicking the down arrow next to the user icon in the top right corner.

Once you identified your secret space, click on it to browse your secrets.

Create secrets

To create your first secret, click on Secrets in the top navigation bar, then click on folder secrets, then click Create secret on the right.

Use your EGI Check-in ID (e.g. e0b6...@egi.eu in the image below) as the path, followed by the name of your secret. Add at least one key to your secret, together with a value.

Click Save and your “home directory” will be created together with your first secret.

Once you created your first secret, you can create the subsequent secrets without having to enter your Check-in ID in the path. Just navigate to your “home folder” and create a secret there.

Users: Secrets Store Command-Line Interface

Mon, 01 Jan 0001 00:00:00 +0000

Multiple tools and command-line interfaces are available for accessing and working with EGI Secrets Store.

The FedCloud client is strongly recommended as it is tightly integrated with the service, it works out of the box without additional configuration, has a simple syntax and supports advanced features such as client-side encrypted secrets.

The Hashicorp Vault client can be used to access advanced Vault features not exposed via other tools.

Prerequisites

To access the EGI Secrets Store service from the command-line you need a valid EGI Check-in access token. Get it either from the EGI Check-in Token Portal, or from the oidc-agent (see cli guide for details), then set it to an environment variable:

$ export OIDC_ACCESS_TOKEN=<token>

> $env:OIDC_ACCESS_TOKEN="<token>"

> set OIDC_ACCESS_TOKEN=<token>

Access via FedCloud client

The FedCloud client is integrated with the EGI Secrets Store service, so that users can access the service immediately with simple commands. Below is a quick start to using the service.

Basic usage

Let’s assume you want to create a secret my_app_secrets and store passwords for two services in it:

$ fedcloud secret put my_app_secrets \
 mysql_password=123456 \
 redis_password=abcdef

> fedcloud secret put my_app_secrets `
 mysql_password=123456 `
 redis_password=abcdef

> fedcloud secret put my_app_secrets ^
 mysql_password=123456 ^
 redis_password=abcdef

Listing all your secrets is very simple:

$ fedcloud secret list
my_app_secrets

Using any of the keys (actually, their values) is straightforward too:

# Get all keys and their values
$ fedcloud secret get my_app_secrets
key value
-------------- -------
redis_password abcdef
mysql_password 123456

# Get the value of a specific key from a secret
$ fedcloud secret get my_app_secrets mysql_password
123456

# Using the value of a specific key from a secret
$ export ADMIN_PASSWORD=$(fedcloud secret get my_app_secrets admin_password)
$ echo $ADMIN_PASSWORD
abcdef

# Get all keys and their values
> fedcloud secret get my_app_secrets
key value
-------------- -------
redis_password abcdef
mysql_password 123456

# Get the value of a specific key from a secret
> fedcloud secret get my_app_secrets mysql_password
123456

# Using the value of a specific key from a secret
> $env:ADMIN_PASSWORD=$(fedcloud secret get my_app_secrets admin_password)
> echo $env:ADMIN_PASSWORD
abcdef

:: Get all keys and their values
> fedcloud secret get my_app_secrets
key value
-------------- -------
redis_password abcdef
mysql_password 123456

:: Get the value of a specific key from a secret
> fedcloud secret get my_app_secrets mysql_password
123456

:: Using the value of a specific key from a secret
> set ADMIN_PASSWORD=(fedcloud secret get my_app_secrets admin_password)
> echo %ADMIN_PASSWORD%
abcdef

Deleting a secret is easy, but it is irreversible:

$ fedcloud secret delete my_app_secrets

Secret values from small text files

If the secret value starts with @ the content of the file with that name is used as the value of the key. The following example creates a secret named certificate for storing the certificate file and its keyfile:

$ fedcloud secret put certificate cert=@hostcert.pem key=@hostkey.pem

You can get the certificate and its keyfile (e.g. when you want to use it on a virtual machine) as follows:

$ fedcloud secret get certificate cert > hostcert.pem
$ fedcloud secret get certificate key > hostkey.pem

Note

The size of the secret object (all the values in it) is limited to 512kB, which is sufficient for storing tokens, certificates, configuration files and so on. For larger datasets, please use encrypted cloud storage.

Secret values from small binary files

It is recommended to store secret values as text for compatibility and ease of manipulation. However, the FedCloud client supports storing small binary files as secret values by encoding/decoding the binary data to ASCII via base64.

Add option --binary-file or -b when using binary files as the secret value:

$ fedcloud secret put secret-image image=@secret-image.png -b
$ fedcloud secret get secret-image image -b > received-image.png

Modifying existing secrets

Secret values in secret objects cannot be edited individually. However, you can get the contents of existing secret objects, change them locally, and put the new contents back, overwriting the old secret. Some examples are shown below.

Important

You will probably want to take care of securely disposing of the local temporary file(s) involved in the update. Omitted here for brevity.

For a secret named certificate containing two keys named cert and key:

$ fedcloud secret get certificate
key value
-------------- -------
cert ...
key ...

To add new values to an existing secret:

$ fedcloud secret get certificate -f json > certificate.json
$ fedcloud secret put certificate @certificate.json \
 another_cert=@usercert.pem \
 another_key=@userkey.pem

To delete values from an existing secret:

$ fedcloud secret get certificate -f json |
 jq 'del (.another_cert, .another_key)' \
 > certificate.json
$ fedcloud secret put certificate @certificate.json

To replace existing values in an existing secret:

$ fedcloud secret get certificate -f json > certificate.json
$ fedcloud secret put certificate @certificate.json \
 cert=@new_hostcert.pem \
 key=@new_hostkey.pem

For a secret named certificate containing two keys named cert and key:

> fedcloud secret get certificate
key value
-------------- -------
cert ...
key ...

To add new values to an existing secret:

> fedcloud secret get certificate -f json > certificate.json
> fedcloud secret put certificate @certificate.json `
 another_cert=@usercert.pem `
 another_key=@userkey.pem

To delete values from an existing secret:

$ fedcloud secret get certificate -f json | `
 jq 'del (.another_cert, .another_key)' `
 > certificate.json
$ fedcloud secret put certificate @certificate.json

To replace existing values in an existing secret:

$ fedcloud secret get certificate -f json > certificate.json
$ fedcloud secret put certificate @certificate.json `
 cert=@new_hostcert.pem `
 key=@new_hostkey.pem

For a secret named certificate containing two keys named cert and key:

> fedcloud secret get certificate
key value
-------------- -------
cert ...
key ...

To add new values to an existing secret:

> fedcloud secret get certificate -f json > certificate.json
> fedcloud secret put certificate @certificate.json ^
 another_cert=@usercert.pem ^
 another_key=@userkey.pem

To delete values from an existing secret:

> fedcloud secret get certificate -f json ^
 | jq 'del (.another_cert, .another_key)' ^
 > certificate.json
> fedcloud secret put certificate @certificate.json

To replace existing values in an existing secret:

> fedcloud secret get certificate -f json > certificate.json
> fedcloud secret put certificate @certificate.json ^
 cert=@new_hostcert.pem ^
 key=@new_hostkey.pem

Export and import secrets

FedCloud client can output secrets in YAML or JSON format for further processing when using option --output-format or -f:

$ fedcloud secret get my_secrets -f json
$ fedcloud secret get my_secrets -f yaml > my_secrets.yaml

The YAML or JSON files created by FedCloud client can be imported back into EGI Secrets Store by using a single key and adding @ before its name, followed by the filename to load the value(s) from:

$fedcloud secret put my_other_secrets @my_other_secrets.yaml

As the YAML format is a superset of JSON, it is expected by default, unless the filename has .json extension. Try to export your secrets to both formats to see the differences between them.

Importing secret objects from files in text format with key=value lines is not supported, as the format is error-prone, especially for multi-line secret values or values with special characters. You can replace = with : for converting simple text files to YAML files.

Tip

Do not forget that in YAML files a blank space is required after the : separating keys and values.

Note

There is a difference between cert=@hostcert.pem for reading the content of the file hostcert.pem as the value for the key cert, and @my_secrets.yaml for reading the whole secret object with all key:value pairs from the YAML file.

Client-side encrypted secrets

EGI Secret Store encrypts secret objects both in transit and at rest. For highly-sensitive secrets, you can opt to also encrypt your secret values on the client-side, before storing them in EGI Secrets Store.

The client-side encryption is done on the fly by the FedCloud client if an encryption key (passphrase) is provided via option --encrypt-key or -e:

$ fedcloud secret put sensitive data=@sensitive-data.txt -e password

Decryption is done in a similar way, just by providing the passphrase via option --decrypt-key or -d. The secret values will be decrypted on the fly if the passphrase is correct:

$ fedcloud secret get sensitive data -d password

Note

Only secret values are encrypted, not the (names of) the keys.

Verifying what is actually stored in a secret can be done without providing the passphrase:

$ fedcloud secret get sensitive data
gAAAAAB...............................

Note

The encryption/decryption is done with the standard Python cryptography library. Security experts are invited to review the code (available on GitHub) and provide feedback and suggestions for improvements.

Reading data from standard inputs

Reading data from stdin may help in creating shorter scripts and to avoid storing secrets in intermediate files on disk, for security reasons. The symbol - in input parameters means the data will be read from the standard input in the same way as @ can be used to read from files. For example:

To read entire secret from standard input, which must to be in JSON or YAML format:

$ echo '{"mysql_pwd":"123"}' | fedcloud secret put my_secrets -

To read only a secret value from standard input:

$ echo "abcdef" | fedcloud secret put my_secrets admin_password=-

To copy a secret object export it to JSON, then import it as a new copy:

$ fedcloud secret get my_secrets -f json |
 fedcloud secret put my_secret_copy -

To add new values to an existing secret, without using intermediate files:

$ fedcloud secret get certificate -f json |
 fedcloud secret put certificate - \
 another_cert=@usercert.pem \
 another_key=@userkey.pem

> fedcloud secret get certificate -f json | `
 fedcloud secret put certificate - `
 another_cert=@usercert.pem `
 another_key=@userkey.pem

> fedcloud secret get certificate -f json | ^
 fedcloud secret put certificate - ^
 another_cert=@usercert.pem ^
 another_key=@userkey.pem

Access via Vault client

To access EGI Secrets Store using the Vault client, visit the Vault project’s site, download the correct version for your operating system, and install it.

To install the Vault client:

$ brew tap hashicorp/tap
$ brew install hashicorp/tap/vault

To install the Vault client:

$ wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg
$ echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
$ sudo apt update && sudo apt install vault

To install the Vault client:

$ sudo yum install -y yum-utils
$ sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
$ sudo yum -y install vault

To install the Vault client:

$ sudo yum install -y yum-utils shadow-utils
$ sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo
$ sudo yum -y install vault

To install the Vault client:

$ sudo dnf install -y dnf-plugins-core
$ sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/fedora/hashicorp.repo
$ sudo dnf -y install vault

To create/read secrets and to manage the EGI Secrets Store service, you need a Vault token. You can get one from your EGI Check-in access token.

To get a Vault token:

$ export VAULT_ADDR=https://secrets.egi.eu
$ export VAULT_TOKEN=$(vault write auth/jwt/login jwt=$OIDC_ACCESS_TOKEN |
 grep -Po 'token\s+\K[^\s]+$')

To get a Vault token:

> $env:VAULT_ADDR="https://secrets.egi.eu"
> $env:VAULT_TOKEN=$(vault write auth/jwt/login jwt=$env:OIDC_ACCESS_TOKEN `
 | Select-String -Pattern "(?<=token\s+)[^\s]+(?=$)" `
 | %{$_.Matches.value})

To get a Vault token:

> set VAULT_ADDR=https://secrets.egi.eu
> for /f "delims=" %a in ('vault write auth/jwt/login "jwt=%OIDC_ACCESS_TOKEN%" ^| findstr /r /c:"token[ ][ ]*[^^ ]*"') do @set VAULT_TOKEN=%a:token=%
> set VAULT_TOKEN=%VAULT_TOKEN: =%

For convenience, add the path to your secret space to an environment variable:

$ export VAULT_HOME=/secrets/users/$(curl -X POST https://aai.egi.eu/auth/realms/egi/protocol/openid-connect/userinfo -H "Authorization: Bearer $OIDC_ACCESS_TOKEN" | jq -r .voperson_id)

> $env:VAULT_HOME=/secrets/users/$(curl -X POST https://aai.egi.eu/auth/realms/egi/protocol/openid-connect/userinfo -H "Authorization: Bearer $env:OIDC_ACCESS_TOKEN" | jq -r .voperson_id)

> for /f "delims=" %a in ('curl -X POST https://aai.egi.eu/auth/realms/egi/protocol/openid-connect/userinfo -H "Authorization: Bearer %OIDC_ACCESS_TOKEN%" ^| jq -r .voperson_id') do set VAULT_HOME="/secrets/users/%a"

List secrets

After setting the environment variables VAULT_TOKEN and VAULT_HOME, you can list the secrets in your personal secret space:

To list your secrets:

$ vault list $VAULT_HOME

To list your secrets:

> vault list $env:VAULT_HOME

To list your secrets:

> vault list %VAULT_HOME%

Create secret

To create a new secret named test in your personal secret space, containing a key my-key having the value test value use the command below.

Important

The command below will replace, without warning, a previous secret named test, including all its keys, with a new one that will only include the keys you provide as part of the command.

To create or update a secret:

$ vault write $VAULT_HOME/test "my-key=test value" db-pass=1234

To create or update a secret:

> vault write $env:VAULT_HOME/test "my-key=test value" db-pass=1234

To create or update a secret:

> vault write %VAULT_HOME%/test "my-key=test value" db-pass=1234

Tip

Use quotes if the key or value includes white spaces. You can include in the same quote both the key and the value.

Note

You can add as many key to a secret as needed, but keep in mind that they will always be handled (read, written, or deleted) atomically.

Read secret

To read a secret named test from your personal secret space use the commands below.

To read all keys from a secret:

$ vault read $VAULT_HOME/test
Key Value
--- -----
refresh_interval 768h
my-key test value
db-pass 1234

To read specific keys from a secret:

$ vault read -field="my-key" $VAULT_HOME/test
test value

To read all keys from a secret:

> vault read $env:VAULT_HOME/test
Key Value
--- -----
refresh_interval 768h
my-key test value
db-pass 1234

To read specific keys from a secret:

$ vault read -field="my-key" $env:VAULT_HOME/test
test value

To read all keys from a secret:

> vault read %VAULT_HOME%/test
Key Value
--- -----
refresh_interval 768h
my-key test value
db-pass 1234

To read specific keys from a secret:

$ vault read -field="my-key" %VAULT_HOME%/test
test value

Tip

Alternative commands kv put and kv get exist for vault write and vault read. Be sure to check out the full list of Vault commands for more information.

Users: Secrets Store API

Mon, 01 Jan 0001 00:00:00 +0000

Hashicorp Vault, the service on which EGI Secrets Store is based, has a REST API with similar inputs like the Vault CLI. See the full documentation for more details.

Note

There is a long list of libraries, clients, and tools for working with Hashicorp Vault.

Users: Secrets Store Architecture

Mon, 01 Jan 0001 00:00:00 +0000

EGI Secrets Store is based on Hashicorp Vault and is fully integrated with EGI Check-in, allowing users to use it with their community credentials.

Important

When this service is adopted by an application, it becomes a critical dependency for that application, thus the EGI Secrets Store is implemented using a highly-available (HA) approach.

Service design

Multiple Vault servers in HA mode, located at different geographical locations (in different resource centres) are connected into a cluster, with one active and multiple standby instances. Data is replicated among servers using the Raft Consensus algorithm supported by the Vault Integrated Storage.

Should the active server fail, one of the standby servers will become active automatically. Each server does a periodic check, and the active one will update the main service endpoint to point to itself, via Dynamic DNS.

Service endpoints

During normal operation users can access any of the nodes of EGI Secrets Store directly:

Note

Accessing EGI Secrets Store via the node endpoints is not recommended: if a server is down, its endpoint is not accessible.

For convenience and high availability, you should use the generic endpoint to access the service. This generic endpoint will be pointed to one of the service nodes automatically via Dynamic DNS. A simple cron script periodically checks and assigns the generic hostname to a healthy server.

The recovery time of the generic endpoint, in the case of an unscheduled downtime of the generic endpoint’s server, is T+1 minutes, where T is the interval between cron checks (usually 1 minute). In the case of scheduled downtime for maintenance, administrators simply assign the generic endpoint to another server instance.