Secrets Store Command-Line Interface
Multiple tools and command-line interfaces are available for accessing and working with EGI Secrets Store.
The FedCloud client is strongly recommended as it is tightly integrated with the service, it works out of the box without additional configuration, has a simple syntax and supports advanced features such as client-side encrypted secrets.
The Hashicorp Vault client can be used to access advanced Vault features not exposed via other tools.
Prerequisites
To access the EGI Secrets Store service from the command-line you need a valid EGI Check-in access token. Get it either from the EGI Check-in Token Portal, or from the oidc-agent (see here for details), then set it to an environment variable:
$ export OIDC_ACCESS_TOKEN=<token>
> $env:OIDC_ACCESS_TOKEN="<token>"
> set OIDC_ACCESS_TOKEN=<token>
Access via FedCloud client
The FedCloud client is integrated with the EGI Secrets Store service, so that users can access the service immediately with simple commands. Below is a quick start to using the service.
Basic usage
Let’s assume you want to create a secret my_app_secrets
and store passwords
for two services in it:
$ fedcloud secret put my_app_secrets \
mysql_password=123456 \
redis_password=abcdef
> fedcloud secret put my_app_secrets `
mysql_password=123456 `
redis_password=abcdef
> fedcloud secret put my_app_secrets ^
mysql_password=123456 ^
redis_password=abcdef
Listing all your secrets is very simple:
$ fedcloud secret list
my_app_secrets
Using any of the keys (actually, their values) is straightforward too:
# Get all keys and their values
$ fedcloud secret get my_app_secrets
key value
-------------- -------
redis_password abcdef
mysql_password 123456
# Get the value of a specific key from a secret
$ fedcloud secret get my_app_secrets mysql_password
123456
# Using the value of a specific key from a secret
$ export ADMIN_PASSWORD=$(fedcloud secret get my_app_secrets admin_password)
$ echo $ADMIN_PASSWORD
abcdef
# Get all keys and their values
> fedcloud secret get my_app_secrets
key value
-------------- -------
redis_password abcdef
mysql_password 123456
# Get the value of a specific key from a secret
> fedcloud secret get my_app_secrets mysql_password
123456
# Using the value of a specific key from a secret
> $env:ADMIN_PASSWORD=$(fedcloud secret get my_app_secrets admin_password)
> echo $env:ADMIN_PASSWORD
abcdef
:: Get all keys and their values
> fedcloud secret get my_app_secrets
key value
-------------- -------
redis_password abcdef
mysql_password 123456
:: Get the value of a specific key from a secret
> fedcloud secret get my_app_secrets mysql_password
123456
:: Using the value of a specific key from a secret
> set ADMIN_PASSWORD=(fedcloud secret get my_app_secrets admin_password)
> echo %ADMIN_PASSWORD%
abcdef
Deleting a secret is easy, but it is irreversible:
$ fedcloud secret delete my_app_secrets
Secret values from small text files
If the secret value starts with @
the content of the file with that name is
used as the value of the key. The following example creates a secret named
certificate
for storing the certificate file and its keyfile:
$ fedcloud secret put certificate cert=@hostcert.pem key=@hostkey.pem
You can get the certificate and its keyfile (e.g. when you want to use it on a virtual machine) as follows:
$ fedcloud secret get certificate cert > hostcert.pem
$ fedcloud secret get certificate key > hostkey.pem
Note
The size of the secret object (all the values in it) is limited to 512kB, which is sufficient for storing tokens, certificates, configuration files and so on. For larger datasets, please use encrypted cloud storage.Secret values from small binary files
It is recommended to store secret values as text for compatibility and ease of manipulation. However, the FedCloud client supports storing small binary files as secret values by encoding/decoding the binary data to ASCII via base64.
Add option --binary-file
or -b
when using binary files as the secret value:
$ fedcloud secret put secret-image image=@secret-image.png -b
$ fedcloud secret get secret-image image -b > received-image.png
Modifying existing secrets
Secret values in secret objects cannot be edited individually. However, you can get the contents of existing secret objects, change them locally, and put the new contents back, overwriting the old secret. Some examples are shown below.
Important
You will probably want to take care of securely disposing of the local temporary file(s) involved in the update. Omitted here for brevity.For a secret named certificate
containing two keys named cert
and key
:
$ fedcloud secret get certificate
key value
-------------- -------
cert ...
key ...
To add new values to an existing secret:
$ fedcloud secret get certificate -f json > certificate.json
$ fedcloud secret put certificate @certificate.json \
another_cert=@usercert.pem \
another_key=@userkey.pem
To delete values from an existing secret:
$ fedcloud secret get certificate -f json |
jq 'del (.another_cert, .another_key)' \
> certificate.json
$ fedcloud secret put certificate @certificate.json
To replace existing values in an existing secret:
$ fedcloud secret get certificate -f json > certificate.json
$ fedcloud secret put certificate @certificate.json \
cert=@new_hostcert.pem \
key=@new_hostkey.pem
For a secret named certificate
containing two keys named cert
and key
:
> fedcloud secret get certificate
key value
-------------- -------
cert ...
key ...
To add new values to an existing secret:
> fedcloud secret get certificate -f json > certificate.json
> fedcloud secret put certificate @certificate.json `
another_cert=@usercert.pem `
another_key=@userkey.pem
To delete values from an existing secret:
$ fedcloud secret get certificate -f json | `
jq 'del (.another_cert, .another_key)' `
> certificate.json
$ fedcloud secret put certificate @certificate.json
To replace existing values in an existing secret:
$ fedcloud secret get certificate -f json > certificate.json
$ fedcloud secret put certificate @certificate.json `
cert=@new_hostcert.pem `
key=@new_hostkey.pem
For a secret named certificate
containing two keys named cert
and key
:
> fedcloud secret get certificate
key value
-------------- -------
cert ...
key ...
To add new values to an existing secret:
> fedcloud secret get certificate -f json > certificate.json
> fedcloud secret put certificate @certificate.json ^
another_cert=@usercert.pem ^
another_key=@userkey.pem
To delete values from an existing secret:
> fedcloud secret get certificate -f json ^
| jq 'del (.another_cert, .another_key)' ^
> certificate.json
> fedcloud secret put certificate @certificate.json
To replace existing values in an existing secret:
> fedcloud secret get certificate -f json > certificate.json
> fedcloud secret put certificate @certificate.json ^
cert=@new_hostcert.pem ^
key=@new_hostkey.pem
Export and import secrets
FedCloud client can output secrets in YAML or
JSON format for further processing when using option
--output-format
or -f
:
$ fedcloud secret get my_secrets -f json
$ fedcloud secret get my_secrets -f yaml > my_secrets.yaml
The YAML or JSON files created by FedCloud client can be imported back into EGI
Secrets Store by using a single key and adding @
before its name, followed by
the filename to load the value(s) from:
$fedcloud secret put my_other_secrets @my_other_secrets.yaml
As the YAML format is a superset of JSON, it is expected by default, unless the filename has .json extension. Try to export your secrets to both formats to see the differences between them.
Importing secret objects from files in text format with key=value
lines is
not supported, as the format is error-prone, especially for multi-line secret
values or values with special characters. You can replace =
with :
for
converting simple text files to YAML files.
Tip
Do not forget that in YAML files a blank space is required after the:
separating keys and values.Note
There is a difference betweencert=@hostcert.pem
for reading the content of the file hostcert.pem as
the value for the key cert
, and @my_secrets.yaml
for reading the whole
secret object with all key:value pairs from the YAML file.Client-side encrypted secrets
EGI Secret Store encrypts secret objects both in transit and at rest. For highly-sensitive secrets, you can opt to also encrypt your secret values on the client-side, before storing them in EGI Secrets Store.
The client-side encryption is done on the fly by the FedCloud client if an
encryption key (passphrase) is provided via option --encrypt-key
or -e
:
$ fedcloud secret put sensitive data=@sensitive-data.txt -e password
Decryption is done in a similar way, just by providing the passphrase via
option --decrypt-key
or -d
. The secret values will be decrypted on the fly
if the passphrase is correct:
$ fedcloud secret get sensitive data -d password
Note
Only secret values are encrypted, not the (names of) the keys.Verifying what is actually stored in a secret can be done without providing the passphrase:
$ fedcloud secret get sensitive data
gAAAAAB...............................
Note
The encryption/decryption is done with the standard Python cryptography library. Security experts are invited to review the code (available on GitHub) and provide feedback and suggestions for improvements.Reading data from standard inputs
Reading data from stdin
may help in creating shorter scripts and to avoid
storing secrets in intermediate files on disk, for security reasons. The symbol
-
in input parameters means the data will be read from the standard input in
the same way as @
can be used to read from files. For example:
To read entire secret from standard input, which must to be in JSON or YAML format:
$ echo '{"mysql_pwd":"123"}' | fedcloud secret put my_secrets -
To read only a secret value from standard input:
$ echo "abcdef" | fedcloud secret put my_secrets admin_password=-
To copy a secret object export it to JSON, then import it as a new copy:
$ fedcloud secret get my_secrets -f json |
fedcloud secret put my_secret_copy -
To add new values to an existing secret, without using intermediate files:
$ fedcloud secret get certificate -f json |
fedcloud secret put certificate - \
another_cert=@usercert.pem \
another_key=@userkey.pem
> fedcloud secret get certificate -f json | `
fedcloud secret put certificate - `
another_cert=@usercert.pem `
another_key=@userkey.pem
> fedcloud secret get certificate -f json | ^
fedcloud secret put certificate - ^
another_cert=@usercert.pem ^
another_key=@userkey.pem
Access via Vault client
To access EGI Secrets Store using the Vault client, visit the Vault project’s site, download the correct version for your operating system, and install it.
To install the Vault client:
$ brew tap hashicorp/tap
$ brew install hashicorp/tap/vault
To install the Vault client:
$ wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg
$ echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
$ sudo apt update && sudo apt install vault
To install the Vault client:
$ sudo yum install -y yum-utils
$ sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
$ sudo yum -y install vault
To install the Vault client:
$ sudo yum install -y yum-utils shadow-utils
$ sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo
$ sudo yum -y install vault
To install the Vault client:
$ sudo dnf install -y dnf-plugins-core
$ sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/fedora/hashicorp.repo
$ sudo dnf -y install vault
To create/read secrets and to manage the EGI Secrets Store service, you need a Vault token. You can get one from your EGI Check-in access token.
To get a Vault token:
$ export VAULT_ADDR=https://secrets.egi.eu
$ export VAULT_TOKEN=$(vault write auth/jwt/login jwt=$OIDC_ACCESS_TOKEN |
grep -Po 'token\s+\K[^\s]+$')
To get a Vault token:
> $env:VAULT_ADDR="https://secrets.egi.eu"
> $env:VAULT_TOKEN=$(vault write auth/jwt/login jwt=$env:OIDC_ACCESS_TOKEN `
| Select-String -Pattern "(?<=token\s+)[^\s]+(?=$)" `
| %{$_.Matches.value})
To get a Vault token:
> set VAULT_ADDR=https://secrets.egi.eu
> for /f "delims=" %a in ('vault write auth/jwt/login "jwt=%OIDC_ACCESS_TOKEN%" ^| findstr /r /c:"token[ ][ ]*[^^ ]*"') do @set VAULT_TOKEN=%a:token=%
> set VAULT_TOKEN=%VAULT_TOKEN: =%
For convenience, add the path to your secret space to an environment variable:
$ export VAULT_HOME=/secrets/$(curl -X POST https://aai.egi.eu/auth/realms/egi/protocol/openid-connect/userinfo -H "Authorization: Bearer $OIDC_ACCESS_TOKEN" | jq -r .voperson_id)
> $env:VAULT_HOME=/secrets/$(curl -X POST https://aai.egi.eu/auth/realms/egi/protocol/openid-connect/userinfo -H "Authorization: Bearer $env:OIDC_ACCESS_TOKEN" | jq -r .voperson_id)
> for /f "delims=" %a in ('curl -X POST https://aai.egi.eu/auth/realms/egi/protocol/openid-connect/userinfo -H "Authorization: Bearer %OIDC_ACCESS_TOKEN%" ^| jq -r .voperson_id') do set VAULT_HOME="/secrets/%a"
List secrets
After setting the environment variables VAULT_TOKEN
and VAULT_HOME
, you can
list the secrets in your personal secret space:
To list your secrets:
$ vault list $VAULT_HOME
To list your secrets:
> vault list $env:VAULT_HOME
To list your secrets:
> vault list %VAULT_HOME%
Create secret
To create a new secret named test
in your personal secret space, containing a
key my-key
having the value test value
use the command below.
Important
The command below will replace, without warning, a previous secret namedtest
, including all its keys, with a
new one that will only include the keys you provide as part of the command.To create or update a secret:
$ vault write $VAULT_HOME/test "my-key=test value" db-pass=1234
To create or update a secret:
> vault write $env:VAULT_HOME/test "my-key=test value" db-pass=1234
To create or update a secret:
> vault write %VAULT_HOME%/test "my-key=test value" db-pass=1234
Tip
Use quotes if the key or value includes white spaces. You can include in the same quote both the key and the value.Note
You can add as many key to a secret as needed, but keep in mind that they will always be handled (read, written, or deleted) atomically.Read secret
To read a secret named test
from your personal secret space use the commands
below.
To read all keys from a secret:
$ vault read $VAULT_HOME/test
Key Value
--- -----
refresh_interval 768h
my-key test value
db-pass 1234
To read specific keys from a secret:
$ vault read -field="my-key" $VAULT_HOME/test
test value
To read all keys from a secret:
> vault read $env:VAULT_HOME/test
Key Value
--- -----
refresh_interval 768h
my-key test value
db-pass 1234
To read specific keys from a secret:
$ vault read -field="my-key" $env:VAULT_HOME/test
test value
To read all keys from a secret:
> vault read %VAULT_HOME%/test
Key Value
--- -----
refresh_interval 768h
my-key test value
db-pass 1234
To read specific keys from a secret:
$ vault read -field="my-key" %VAULT_HOME%/test
test value
Tip
Alternative commandskv put
and
kv get
exist for vault write
and vault read
. Be sure to check out the
full list of Vault commands
for more information.