Documentation – Notebooks

Providers: Architecture

Mon, 01 Jan 0001 00:00:00 +0000

The EGI Notebooks service relies on the following technologies to provide its functionality:

JupyterHub with custom EGI Check-in oauthentication configured to spawn pods on Kubernetes.
Kubernetes as container orchestration platform running on top of EGI Cloud resources. Within the service it is in charge of managing the allocated resources and providing the right abstraction to deploy the containers that build the service. Resources are provided by EGI Federated Cloud providers, including persistent storage for users notebooks.
CA authority to allocate recognised certificates for the HTTPS server
Prometheus for monitoring resource consumption.
Specific EGI hooks for monitoring, accounting and backup.
VO-Specific storage/Big data facilities or any pluggable tools into the notebooks environment can be added to community specific instances.

Kubernetes

A Kubernetes (k8s) cluster deployed into a resource provider is in charge of managing the containers that will provide the service. On this cluster there are:

1 master node that manages the whole cluster
Support for load balancer or alternatively 1 or more edge nodes with a public IP and corresponding public DNS name (e.g. notebooks.egi.eu) where a k8s ingress HTTP reverse proxy redirects requests from user to other components of the service. The HTTP server has a valid certificate from one CA recognised at most browsers (e.g. Let's Encrypt).
1 or more nodes that host the JupyterHub server, the notebooks servers where the users will run their notebooks. Hub is deployed using the JupyterHub helm charts. These nodes should have enough capacity to run as many concurrent user notebooks as needed. Main constraint is usually memory.
Support for Kubernetes PersistentVolumeClaims for storing the persistent folders. Default EGI-Notebooks installation uses NFS, but any other volume type with ReadWriteOnce capabilities can be used.
Prometheus installation to monitor the usage of resources so accounting records are generated.

All communication with the user goes via HTTPS and the service only needs a publicly accessible entry point (public IP with resolvable name)

Monitoring and accounting are provided by hooking into the respective monitoring and accounting EGI services.

There are no specific hardware requirements and the whole environment can run on commodity virtual machines.

EGI Customisations

EGI Notebooks is deployed as a set of customisations of the JupyterHub helm charts.

Authentication

EGI Check-in can be easily configured as a OAuth2.0 provider for JupyterHub's oauthenticator. See below a sample configuration for the helm chart using Check-in production environment:

hub:
 extraEnv:
 OAUTH2_AUTHORIZE_URL: https://aai.egi.eu/auth/realms/egi/protocol/openid-connect/auth
 OAUTH2_TOKEN_URL: https://aai.egi.eu/auth/realms/egi/protocol/openid-connect/token
 OAUTH_CALLBACK_URL: https://<your host>/hub/oauth_callback

auth:
 type: custom
 custom:
 className: oauthenticator.generic.GenericOAuthenticator
 config:
 login_service: "EGI Check-in"
 client_id: "<your client id>"
 client_secret: "<your client secret>"
 oauth_callback_url: "https://<your host>/hub/oauth_callback"
 username_key: "sub"
 token_url: "https://aai.egi.eu/auth/realms/egi/protocol/openid-connect/token"
 userdata_url: "https://aai.egi.eu/auth/realms/egi/protocol/openid-connect/userinfo"
 scope: ["openid", "profile", "email", "eduperson_scoped_affiliation", "eduperson_entitlement"]

To simplify the configuration and to add refresh capabilities to the credentials, we have created a new EGI Check-in authenticator that can be configured as follows:

auth:
 state:
 enabled: true
 cryptoKey: <some unique crypto key>
 type: custom
 custom:
 className: oauthenticator.egicheckin.EGICheckinAuthenticator
 config:
 client_id: "<your client id>"
 client_secret: "<your client secret>"
 oauth_callback_url: "https://<your host>/hub/oauth_callback"
 scope:
 - openid
 - profile
 - email
 - offline_access
 - eduperson_scoped_affiliation
 - eduperson_entitlement

The auth.state configuration allows to store refresh tokens for the users that will allow to get up-to-date valid credentials as needed.

Accounting

Warning

This is Work in progress, expect changes!

Accounting module generates VM-like accounting records for each of the notebooks started at the service. It's available as a helm chart that can be deployed in the same namespace as the JupyterHub chart. The only needed configuration for the chart is an IGTF-recognised certificate for the host registered in GOCDB as accounting.

ssm:
 hostcert: |-
  <hostcert>
 hostkey: |-
  <hostkey>

Monitoring

Monitoring is performed by trying to execute a user notebook every hour. This is accomplished by registering a new service in the hub that has admin permissions. Monitoring is then deployed as a helm chart that must be deployed in the same namespace as the JupyterHub chart. Configuration of JupyterHub must include this section:

hub:
 services:
 status:
 url: "http://status-web/"
 admin: true
 apiToken: "<a unique API token>"

Likewise the monitoring chart is configured as follows:

service:
 api_token: "<same API token as above>"

Docker images

Our service relies on custom images for the hub and the single-user notebooks. Dockerfiles are available at EGI Notebooks images git repository and automatically build for every commit pushed to the repository to eginotebooks @ dockerhub.

Hub image

Builds from the JupyterHub k8s-hub image and adds:

EGI and D4Science authenticators
EGISpawner
EGI look and feel for the login page

Single-user image

Builds from Jupyter datasicence-notebook and adds a wide range of libraries as requested by users of the services. We are currently looking into alternatives for better managing this image with CVMFS as a possible solution.

Sample helm configuration

If you want to build your own EGI Notebooks instance, you can start from the following sample configuration and adapt to your needs by setting:

secret tokens (for proxy.secretToken, hub.services.status.api_token, auth.state.cryptoKey). They can be generated with openssl rand -hex 32.
A valid hostname (<your notebooks host> below) that resolves to your Kubernetes Ingress
Valid EGI Check-in client credentials, these can be obtained by creating a new Service for the demo instance of Check-in through the EGI Federation Registry. When moving to EGI Check-in production environment, make sure to remove the hub.extraEnv.EGICHECKIN_HOST variable.

---
proxy:
 secretToken: "<some secret>"
 service:
 type: NodePort

ingress:
 enabled: true
 annotations:
 kubernetes.io/tls-acme: "true"
 hosts: [<your notebooks host>]
 tls:
 - hosts:
 - <your notebooks host>
 secretName: acme-tls-notebooks
 enabled: true
 hosts: [<your notebooks host>]

singleuser:
 storage:
 capacity: 1Gi
 dynamic:
 pvcNameTemplate: claim-{userid}{servername}
 volumeNameTemplate: vol-{userid}{servername}
 storageAccessModes: ["ReadWriteMany"]
 memory:
 limit: 1G
 guarantee: 512M
 cpu:
 limit: 2
 guarantee: .02
 defaultUrl: "/lab"
 image:
 name: eginotebooks/single-user
 tag: c1b2a2a

hub:
 image:
 name: eginotebooks/hub
 tag: c1b2a2a
 extraConfig:
 enable-lab: |-
  c.KubeSpawner.cmd = ['jupyter-labhub']
 volume-handling: |-
 from egispawner.spawner import EGISpawner
 c.JupyterHub.spawner_class = EGISpawner
 extraEnv:
 JUPYTER_ENABLE_LAB: 1
 EGICHECKIN_HOST: aai-demo.egi.eu
 services:
 status:
 url: "http://status-web/"
 admin: true
 api_token: "<monitor token>"

auth:
 type: custom
 state:
 enabled: true
 cryptoKey: "<a unique crypto key>"
 admin:
 access: true
 users: [<list of EGI Check-in users with admin powers>]
 custom:
 className: oauthenticator.egicheckin.EGICheckinAuthenticator
 config:
 client_id: "<your egi checkin_client_id>"
 client_secret: "<your egi checkin_client_secret>"
 oauth_callback_url: "https://<your notebooks host>/hub/oauth_callback"
 enable_auth_state: true
 scope:
 - openid
 - profile
 - email
 - offline_access
 - eduperson_scoped_affiliation
 - eduperson_entitlement

Providers: Service Operations

Mon, 01 Jan 0001 00:00:00 +0000

In this section you can find the common operational activities related to keep the service available to our users.

Initial set-up

Notebooks VO

The resources used for the Notebooks deployments are managed with the vo.notebooks.egi.eu VO. Operators of the service should join the VO with the admin role.

Clients installation

In order to manage the resources you will need these tools installed on your client machine:

fedcloudclient for discovering sites and managing tokens,
terraform to create the VMs at the providers,
ansible to configure the VMs and install kubernetes at the providers,
terraform-inventory to get the list of hosts to use from terraform.

Get the configuration repository

All the configuration of the notebooks is stored at a git repository available in GitHub. Start by cloning the repo:

git clone https://github.com/EGI-Federation/notebooks-operations

Kubernetes

We use terraform and ansible to build the cluster at one of the EGI Cloud providers. If you are building the cluster for the first time, create a new directory on your local git repository from the template, add it to the repo, and get terraform ready:

cp -a template <new provider>
git add <new provider>
cd <new provider>/terraform
terraform init

Using the fedcloud you can get set the right environment for interacting with the OpenStack APIs of a given site:

eval "$(fedcloud site show-project-id --site CESGA --vo vo.notebooks.egi.eu)"

Whenever you need to get a valid token for the site and VO, you can obtain it with:

OS_TOKEN=$(fedcloud openstack --site CESGA --vo vo.notebooks.egi.eu \
 token issue -c id -f value)

First get the network IDs and pool to use for the site:

$ fedcloud openstack --site CESGA --vo vo.notebooks.egi.eu network list
+--------------------------------------+-------------------------+--------------------------------------+
| ID | Name | Subnets |
+--------------------------------------+-------------------------+--------------------------------------+
| 1aaf20b6-47a1-47ef-972e-7b36872f678f | net-vo.notebooks.egi.eu | 6465a327-c261-4391-a0f5-d503cc2d43d3 |
| 6174db12-932f-4ee3-bb3e-7a0ca070d8f2 | public00 | 6af8c4f3-8e2e-405d-adea-c0b374c5bd99 |
+--------------------------------------+-------------------------+--------------------------------------+

In this case we will use public00 as the pool for public IPs and 1aaf20b6-47a1-47ef-972e-7b36872f678f as the network ID. Check with the provider which is the right network to use. Use these values in the terraform.tfvars file:

ip_pool = "public00"
net_id = "1aaf20b6-47a1-47ef-972e-7b36872f678f"

You may want to check the right flavors for your VMs and adapt other variables in terraform.tfvars. To get a list of flavors you can use:

$ fedcloud openstack --site CESGA --vo vo.notebooks.egi.eu flavor list
+--------------------------------------+----------------+-------+------+-----------+-------+-----------+
| ID | Name | RAM | Disk | Ephemeral | VCPUs | Is Public |
+--------------------------------------+----------------+-------+------+-----------+-------+-----------+
| 26d14547-96f2-4751-a686-f89a9f7cd9cc | cor4mem8hd40 | 8192 | 40 | 0 | 4 | True |
| 42eb9c81-e556-4b63-bc19-4c9fb735e344 | cor2mem2hd20 | 2048 | 20 | 0 | 2 | True |
| 4787d9fc-3923-4fc9-b770-30966fc3baee | cor4mem4hd40 | 4096 | 40 | 0 | 4 | True |
| 58586b06-7b9d-47af-b9d0-e16d49497d09 | cor24mem62hd60 | 63488 | 60 | 0 | 24 | True |
| 635c739a-692f-4890-b8fd-d50963bff00e | cor1mem1hd10 | 1024 | 10 | 0 | 1 | True |
| 6ba0080d-d71c-4aff-b6f9-b5a9484097f8 | small | 512 | 2 | 0 | 1 | True |
| 6e514065-9013-4ce1-908a-0dcc173125e4 | cor2mem4hd20 | 4096 | 20 | 0 | 2 | True |
| 85f66ce6-0b66-4889-a0bf-df8dc23ee540 | cor1mem2hd10 | 2048 | 10 | 0 | 1 | True |
| c4aa496b-4684-4a86-bd7f-3a67c04b1fa6 | cor24mem50hd50 | 51200 | 50 | 0 | 24 | True |
| edac68c3-50ea-42c2-ae1d-76b8beb306b5 | test-bigHD | 4096 | 237 | 0 | 2 | True |
+--------------------------------------+----------------+-------+------+-----------+-------+-----------+

Finally, ensure your public ssh key is also listed in the cloud-init.yaml file and then you are ready to deploy the cluster with:

terraform apply

Your VMs are up and running, it's time to get kubernetes configured and running with Ansible.

The following Ansible role needs to be installed first:

ansible-galaxy install grycap.kubernetes

and then:

cd .. # you should be now in <new provider>
ANSIBLE_TRANSFORM_INVALID_GROUP_CHARS=silently TF_STATE=./terraform \
 ansible-playbook --inventory-file=$(which terraform-inventory) \
 playbooks/k8s.yaml

Interacting with the cluster

As the master will be on a private IP, you won't be able to directly interact with it, but you can still ssh into the VM using the ingress node as a gateway host (you can get the different hosts with TF_STATE=./terraform terraform-inventory --inventory)

$ ssh -o ProxyCommand="ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -W %h:%p -q egi@<ingress ip>" \
 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null egi@<master ip>
egi@k8s-master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 33m v1.15.7
k8s-nfs Ready <none> 16m v1.15.7
k8s-w-ingress Ready <none> 16m v1.15.7
egi@k8s-master:~$ helm list
NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
certs-man 2 Wed Jan 8 15:56:58 2020 DEPLOYED cert-manager-v0.11.0 v0.11.0 cert-manager
cluster-ingress 3 Wed Jan 8 15:56:53 2020 DEPLOYED nginx-ingress-1.7.0 0.24.1 kube-system
nfs-provisioner 3 Wed Jan 8 15:56:43 2020 DEPLOYED nfs-client-provisioner-1.2.8 3.1.0 kube-system

Modifying/Destroying the cluster

You should be able to change the number of workers in the cluster and re-apply terraform to start them and then execute the playbook to get them added to the cluster.

Any changes in the master, NFS or ingress VMs should be done carefully as those will probably break the configuration of the Kubernetes cluster and of any application running on top.

Destroying the cluster can be done with a single command:

terraform destroy

Notebooks deployments

Once the k8s cluster is up and running, you can deploy a notebooks instance. For each deployment you should create a file in the deployments directory following the template provided:

cp deployments/hub.yaml.template deployments/hub.yaml

Each deployment will need a domain name pointing to your ingress host, you can create one at the FedCloud dynamic DNS service.

Then you will need to create an OpenID Connect client for EGI Check-in to authorise users into the new deployment. You can create a client by going to the EGI Federation Registry. You can find more information about registering an OIDC Client in EGI Check-in guide for SPs Use the following as redirect URL: https://<your host domain name>/hub/oauth_callback.

Then add offline_access to the list of scopes and submit the request. After the approval of the Service request, save the client and take note of the client ID and client secret for later.

Finally, you will also need 3 different random strings generated with openssl rand -hex 32 that will be used as secrets in the file describing the deployment.

Go and edit the deployment description file to add this information (search for # FIXME NEEDS INPUT in the file to quickly get there)

For deploying the notebooks instance we will also use ansible:

ANSIBLE_TRANSFORM_INVALID_GROUP_CHARS=silently \
 TF_STATE=./terraform ansible-playbook \
 --inventory-file=$(which terraform-inventory) playbooks/notebooks.yaml

The first deployment trial may fail due to a timeout caused by the downloading of the container images needed. You can retry after a while to re-deploy.

In the master you can check the status of your deployment (the name of the deployment will be the same as the name of your local deployment file):

$ helm status hub
LAST DEPLOYED: Thu Jan 9 08:14:49 2020
NAMESPACE: hub
STATUS: DEPLOYED

RESOURCES:
==> v1/ServiceAccount
NAME SECRETS AGE
hub 1 6m46s
user-scheduler 1 3m34s

==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hub ClusterIP 10.100.77.129 <none> 8081/TCP 6m46s
proxy-public NodePort 10.107.127.44 <none> 443:32083/TCP,80:30581/TCP 6m45s
proxy-api ClusterIP 10.103.195.6 <none> 8001/TCP 6m45s

==> v1/ConfigMap
NAME DATA AGE
hub-config 4 6m47s
user-scheduler 1 3m35s

==> v1/PersistentVolumeClaim
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
hub-db-dir Pending managed-nfs-storage 6m46s

==> v1/ClusterRole
NAME AGE
hub-user-scheduler-complementary 3m34s

==> v1/ClusterRoleBinding
NAME AGE
hub-user-scheduler-base 3m34s
hub-user-scheduler-complementary 3m34s

==> v1/RoleBinding
NAME AGE
hub 6m46s

==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
continuous-image-puller-flf5t 1/1 Running 0 3m34s
continuous-image-puller-scr49 1/1 Running 0 3m34s
hub-569596fc54-vjbms 0/1 Pending 0 3m30s
proxy-79fb6d57c5-nj8n2 1/1 Running 0 2m22s
user-scheduler-9685d654b-9zt5d 1/1 Running 0 3m30s
user-scheduler-9685d654b-k8v9p 1/1 Running 0 3m30s

==> v1/Secret
NAME TYPE DATA AGE
hub-secret Opaque 3 6m47s

==> v1/DaemonSet
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
continuous-image-puller 2 2 2 2 2 <none> 3m34s

==> v1/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
hub 1 1 1 0 6m45s
proxy 1 1 1 1 6m45s
user-scheduler 2 2 2 2 3m32s

==> v1/StatefulSet
NAME DESIRED CURRENT AGE
user-placeholder 0 0 6m44s

==> v1beta1/Ingress
NAME HOSTS ADDRESS PORTS AGE
jupyterhub notebooktest.fedcloud-tf.fedcloud.eu 80, 443 6m44s

==> v1beta1/PodDisruptionBudget
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
hub 1 N/A 0 6m48s
proxy 1 N/A 0 6m48s
user-placeholder 0 N/A 0 6m48s
user-scheduler 1 N/A 1 6m47s

==> v1/Role
NAME AGE
hub 6m46s


NOTES:
Thank you for installing JupyterHub!

Your release is named hub and installed into the namespace hub.

You can find if the hub and proxy is ready by doing:

kubectl --namespace=hub get pod

and watching for both those pods to be in status 'Running'.

You can find the public IP of the JupyterHub by doing:

kubectl --namespace=hub get svc proxy-public

It might take a few minutes for it to appear!

Note that this is still an alpha release! If you have questions, feel free to
1. Read the guide at https://z2jh.jupyter.org
2. Chat with us at https://gitter.im/jupyterhub/jupyterhub
3. File issues at https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues

Updating a deployment

Just edit the deployment description file and run ansible again. The helm will be upgraded at the cluster.