Secrets Store Command-Line Interface

Command-line interfaces for accessing EGI Secrets Store

Multiple tools and command-line interfaces are available for accessing and working with EGI Secrets Store.

The FedCloud client is strongly recommended as it is tightly integrated with the service, it works out of the box without additional configuration, has a simple syntax and supports advanced features such as client-side encrypted secrets.

The Hashicorp Vault client can be used to access advanced Vault features not exposed via other tools.

Prerequisites

To access the EGI Secrets Store service from the command-line you need a valid EGI Check-in access token. Get it either from the EGI Check-in Token Portal, or from the oidc-agent (see here for details), then set it to an environment variable:

$ export OIDC_ACCESS_TOKEN=<token>

> $env:OIDC_ACCESS_TOKEN="<token>"

> set OIDC_ACCESS_TOKEN=<token>

Access via FedCloud client

The FedCloud client is integrated with the EGI Secrets Store service, so that users can access the service immediately with simple commands. Below is a quickstart to using the service.

Basic usage

Let’s assume you want to create a secret my_app_secrets and store passwords for two services in it:

$ fedcloud secret put my_app_secrets \
       mysql_password=123456 \
       redis_password=abcdef

> fedcloud secret put my_app_secrets `
       mysql_password=123456 `
       redis_password=abcdef

> fedcloud secret put my_app_secrets ^
       mysql_password=123456 ^
       redis_password=abcdef

Listing all your secrets is very simple:

$ fedcloud secret list
my_app_secrets

Using any of the keys (actually, their values) is straightforward too:

# Get all keys and their values
$ fedcloud secret get my_app_secrets
key             value
--------------  -------
redis_password  abcdef
mysql_password  123456

# Get the value of a specific key from a secret
$ fedcloud secret get my_app_secrets mysql_password
123456

# Using the value of a specific key from a secret
$ export ADMIN_PASSWORD=$(fedcloud secret get my_app_secrets admin_password)
$ echo $ADMIN_PASSWORD
abcdef

# Get all keys and their values
> fedcloud secret get my_app_secrets
key             value
--------------  -------
redis_password  abcdef
mysql_password  123456

# Get the value of a specific key from a secret
> fedcloud secret get my_app_secrets mysql_password
123456

# Using the value of a specific key from a secret
> $env:ADMIN_PASSWORD=$(fedcloud secret get my_app_secrets admin_password)
> echo $env:ADMIN_PASSWORD
abcdef

:: Get all keys and their values
> fedcloud secret get my_app_secrets
key             value
--------------  -------
redis_password  abcdef
mysql_password  123456

:: Get the value of a specific key from a secret
> fedcloud secret get my_app_secrets mysql_password
123456

:: Using the value of a specific key from a secret
> set ADMIN_PASSWORD=(fedcloud secret get my_app_secrets admin_password)
> echo %ADMIN_PASSWORD%
abcdef

Deleting a secret is easy, but it is irreversible:

$ fedcloud secret delete my_app_secrets

Secret values from small text files

If the secret value starts with @ the content of the file with that name is used as the value of the key. The following example creates a secret named certificate for storing the certificate file and its keyfile:

$ fedcloud secret put certificate cert=@hostcert.pem key=@hostkey.pem

You can get the certificate and its keyfile (e.g. when you want to use it on a virtual machine) as follows:

$ fedcloud secret get certificate cert > hostcert.pem
$ fedcloud secret get certificate key  > hostkey.pem

Note

The size of the secret object (all the values in it) is limited to 512kB, which is sufficient for storing tokens, certificates, configuration files and so on. For larger datasets, please use encrypted cloud storage.

Secret values from small binary files

It is recommended to store secret values as text for compatibility and ease of manipulation. However, the FedCloud client supports storing small binary files as secret values by encoding/decoding the binary data to ASCII via base64.

Add option --binary-file or -b when using binary files as the secret value:

$ fedcloud secret put secret-image image=@secret-image.png -b
$ fedcloud secret get secret-image image -b > received-image.png

Modifying existing secrets

Secret values in secret objects cannot be edited individually. However, you can get the contents of existing secret objects, change them locally, and put the new contents back, overwriting the old secret. Some examples are shown below.

Important

You will probably want to take care of securely disposing of the local temporary file(s) involved in the update. Omitted here for brevity.

For a secret named certificate containing two keys named cert and key:

$ fedcloud secret get certificate
key             value
--------------  -------
cert            ...
key             ...

To add new values to an existing secret:

$ fedcloud secret get certificate -f json > certificate.json
$ fedcloud secret put certificate @certificate.json \
            another_cert=@usercert.pem \
            another_key=@userkey.pem

To delete values from an existing secret:

$ fedcloud secret get certificate -f json | 
            jq 'del (.another_cert, .another_key)' \
            > certificate.json
$ fedcloud secret put certificate @certificate.json

To replace existing values in an existing secret:

$ fedcloud secret get certificate -f json > certificate.json
$ fedcloud secret put certificate @certificate.json \
            cert=@new_hostcert.pem \
            key=@new_hostkey.pem

For a secret named certificate containing two keys named cert and key:

> fedcloud secret get certificate
key             value
--------------  -------
cert            ...
key             ...

To add new values to an existing secret:

> fedcloud secret get certificate -f json > certificate.json
> fedcloud secret put certificate @certificate.json `
            another_cert=@usercert.pem `
            another_key=@userkey.pem

To delete values from an existing secret:

$ fedcloud secret get certificate -f json | ` 
            jq 'del (.another_cert, .another_key)' `
            > certificate.json
$ fedcloud secret put certificate @certificate.json

To replace existing values in an existing secret:

$ fedcloud secret get certificate -f json > certificate.json
$ fedcloud secret put certificate @certificate.json `
            cert=@new_hostcert.pem `
            key=@new_hostkey.pem

For a secret named certificate containing two keys named cert and key:

> fedcloud secret get certificate
key             value
--------------  -------
cert            ...
key             ...

To add new values to an existing secret:

> fedcloud secret get certificate -f json > certificate.json
> fedcloud secret put certificate @certificate.json ^
            another_cert=@usercert.pem ^
            another_key=@userkey.pem

To delete values from an existing secret:

> fedcloud secret get certificate -f json ^
            | jq 'del (.another_cert, .another_key)' ^
            > certificate.json
> fedcloud secret put certificate @certificate.json

To replace existing values in an existing secret:

> fedcloud secret get certificate -f json > certificate.json
> fedcloud secret put certificate @certificate.json ^
            cert=@new_hostcert.pem ^
            key=@new_hostkey.pem

Export and import secrets

FedCloud client can output secrets in YAML or JSON format for further processing when using option --output-format or -f:

$ fedcloud secret get my_secrets -f json
$ fedcloud secret get my_secrets -f yaml > my_secrets.yaml

The YAML or JSON files created by FedCloud client can be imported back into EGI Secrets Store by using a single key and adding @ before its name, followed by the filename to load the value(s) from:

$fedcloud secret put my_other_secrets @my_other_secrets.yaml

As the YAML format is a superset of JSON, it is expected by default, unless the filename has .json extension. Try to export your secrets to both formats to see the differences between them.

Importing secret objects from files in text format with key=value lines is not supported, as the format is error-prone, especially for multi-line secret values or values with special characters. You can replace = with : for converting simple text files to YAML files.

Tip

Do not forget that in YAML files a blank space is required after the : separating keys and values.

Note

There is a difference between cert=@hostcert.pem for reading the content of the file hostcert.pem as the value for the key cert, and @my_secrets.yaml for reading the whole secret object with all key:value pairs from the YAML file.

Client-side encrypted secrets

EGI Secret Store encrypts secret objects both in transit and at rest. For highly-sensitive secrets, you can opt to also encrypt your secret values on the client-side, before storing them in EGI Secrets Store.

The client-side encryption is done on the fly by the FedCloud client if an encryption key (passphrase) is provided via option --encrypt-key or -e:

$ fedcloud secret put sensitive data=@sensitive-data.txt -e password

Decryption is done in a similar way, just by providing the passphrase via option --decrypt-key or -d. The secret values will be decrypted on the fly if the passphrase is correct:

$ fedcloud secret get sensitive data -d password

Note

Only secret values are encrypted, not the (names of) the keys.

Verifying what is actually stored in a secret can be done without providing the passphrase:

$ fedcloud secret get sensitive data
gAAAAAB...............................

Note

The encryption/decryption is done with the standard Python cryptography library. Security experts are invited to review the code (available on GitHub) and provide feedback and suggestions for improvements.

Reading data from standard inputs

Reading data from stdin may help in creating shorter scripts and to avoid storing secrets in intermediate files on disk, for security reasons. The symbol - in input parameters means the data will be read from the standard input in the same way as @ can be used to read from files. For example:

To read entire secret from standard input, which must to be in JSON or YAML format:

$ echo '{"mysql_pwd":"123"}' | fedcloud secret put my_secrets -

To read only a secret value from standard input:

$ echo "abcdef" | fedcloud secret put my_secrets admin_password=-

To copy a secret object export it to JSON, then import it as a new copy:

$ fedcloud secret get my_secrets -f json |
  fedcloud secret put my_secret_copy -

To add new values to an existing secret, without using intermediate files:

$ fedcloud secret get certificate -f json |
  fedcloud secret put certificate - \
              another_cert=@usercert.pem \
              another_key=@userkey.pem

> fedcloud secret get certificate -f json | `
  fedcloud secret put certificate - `
              another_cert=@usercert.pem `
              another_key=@userkey.pem

> fedcloud secret get certificate -f json | ^
  fedcloud secret put certificate - ^
              another_cert=@usercert.pem ^
              another_key=@userkey.pem

Access via Vault client

To access EGI Secrets Store using the Vault client, visit the Vault project’s site, download the correct version for your operating system, and install it.

To install the Vault client:

$ brew tap hashicorp/tap
$ brew install hashicorp/tap/vault

To install the Vault client:

$ wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg
$ echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
$ sudo apt update && sudo apt install vault

To install the Vault client:

$ sudo yum install -y yum-utils
$ sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
$ sudo yum -y install vault

To install the Vault client:

$ sudo yum install -y yum-utils shadow-utils
$ sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo
$ sudo yum -y install vault

To install the Vault client:

$ sudo dnf install -y dnf-plugins-core
$ sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/fedora/hashicorp.repo
$ sudo dnf -y install vault

To create/read secrets and to manage the EGI Secrets Store service, you need a Vault token. You can get one from your EGI Check-in access token.

To get a Vault token:

$ export VAULT_ADDR=https://vault.services.fedcloud.eu:8200
$ export VAULT_TOKEN=$(vault write auth/jwt/login jwt=$OIDC_ACCESS_TOKEN |
           grep -Po 'token\s+\K[^\s]+$')

To get a Vault token:

> $env:VAULT_ADDR="https://vault.services.fedcloud.eu:8200"
> $env:VAULT_TOKEN=$(vault write auth/jwt/login jwt=$env:OIDC_ACCESS_TOKEN `
           | Select-String -Pattern "(?<=token\s+)[^\s]+(?=$)" `
           | %{$_.Matches.value})

To get a Vault token:

> set VAULT_ADDR=https://vault.services.fedcloud.eu:8200
> for /f "delims=" %a in ('vault write auth/jwt/login "jwt=%OIDC_ACCESS_TOKEN%" ^| findstr /r /c:"token[ ][ ]*[^^ ]*"') do @set VAULT_TOKEN=%a:token=%
> set VAULT_TOKEN=%VAULT_TOKEN: =%

For convenience, add the path to your secret space to an environment variable:

$ export VAULT_HOME=/secrets/$(curl -X POST https://aai.egi.eu/auth/realms/egi/protocol/openid-connect/userinfo -H "Authorization: Bearer $OIDC_ACCESS_TOKEN" | jq -r .voperson_id)

> $env:VAULT_HOME=/secrets/$(curl -X POST https://aai.egi.eu/auth/realms/egi/protocol/openid-connect/userinfo -H "Authorization: Bearer $env:OIDC_ACCESS_TOKEN" | jq -r .voperson_id)

> for /f "delims=" %a in ('curl -X POST https://aai.egi.eu/auth/realms/egi/protocol/openid-connect/userinfo -H "Authorization: Bearer %OIDC_ACCESS_TOKEN%" ^| jq -r .voperson_id') do set VAULT_HOME="/secrets/%a"

List secrets

After setting the environment variables VAULT_TOKEN and VAULT_HOME, you can list the secrets in your personal secret space:

To list your secrets:

$ vault list $VAULT_HOME

To list your secrets:

> vault list $env:VAULT_HOME

To list your secrets:

> vault list %VAULT_HOME%

Create secret

To create a new secret named test in your personal secret space, containing a key my-key having the value test value use the command below.

Important

The command below will replace, without warning, a previous secret named test, including all its keys, with a new one that will only include the keys you provide as part of the command.

To create or update a secret:

$ vault write $VAULT_HOME/test "my-key=test value" db-pass=1234

To create or update a secret:

> vault write $env:VAULT_HOME/test "my-key=test value" db-pass=1234

To create or update a secret:

> vault write %VAULT_HOME%/test "my-key=test value" db-pass=1234

Tip

Use quotes if the key or value includes whitespaces. You can include in the same quote both the key and the value.

Note

You can add as many key to a secret as needed, but keep in mind that they will always be handled (read, written, or deleted) atomically.

Read secret

To read a secret named test from your personal secret space use the commands below.

To read all keys from a secret:

$ vault read $VAULT_HOME/test
Key                 Value
---                 -----
refresh_interval    768h
my-key              test value
db-pass             1234

To read specific keys from a secret:

$ vault read -field="my-key" $VAULT_HOME/test
test value

To read all keys from a secret:

> vault read $env:VAULT_HOME/test
Key                 Value
---                 -----
refresh_interval    768h
my-key              test value
db-pass             1234

To read specific keys from a secret:

$ vault read -field="my-key" $env:VAULT_HOME/test
test value

To read all keys from a secret:

> vault read %VAULT_HOME%/test
Key                 Value
---                 -----
refresh_interval    768h
my-key              test value
db-pass             1234

To read specific keys from a secret:

$ vault read -field="my-key" %VAULT_HOME%/test
test value

Tip

Alternative commands kv put and kv get exist for vault write and vault read. Be sure to check out the full list of Vault commands for more information.

Last modified March 8, 2023 by Baptiste Grenier : Import a GitHub action to check spell (#584)