Documentation – Notebooks

Users: Notebooks Quick Start

Mon, 01 Jan 0001 00:00:00 +0000

The Notebooks Service is based on JupyterHub. It enables remote use of Jupyter Notebooks in a managed environment integrated with other EGI services. Notebooks main interface uses JupyterLab, a highly extensible environment for running and authoring computational notebooks.

You can find links below to the upstream documentation of the EGI Notebooks components:

JupyterLab Documentation for use of the integrated user environment (menus, working tabs, etc.)
JupyterHub Documentation to manage your remote resources (starting and stopping servers, collaborative sharing, etc.)
Documentation for your core programming languages:
- Python
- Julia
- R
- Octave

Getting Started

Start by creating an EGI account
Then enrol vo.notebooks.egi.eu VO
Go to https://notebooks.egi.eu/
Start the authentication process by clicking on Continue with EGI Check-in! button
Once logged in, you will prompted to select the environment, pick the “Default EGI environment”
Click on “Start” to get your JupyterLab instance You will see the Jupyter interface once your personal server is started

Launching a notebook

Click on the “Python 3 (ipykernel)” option to launch your notebook with Python 3 kernel. When you create this notebook, a new tab will be presented with a notebook named Untitled.ipynb. You can easily rename it by right-clicking on the current name.

Structure of a notebook

The notebook consists of a sequence of cells. A cell is a multi-line text input field, and its contents can be executed by using Shift-Enter, or by clicking either the “Play” button in the toolbar, or Cell -> Run in the menu bar.

The execution behaviour of a cell is determined by the cell’s type.

There are three types of cells: cells, Markdown, and raw cells. Every cell starts off being a code cell, but its type can be changed by using a drop-down on the toolbar (which will be “Code”, initially).

Code cells

A code cell allows you to edit and write new code, with full syntax highlighting and tab completion. The programming language you use depends on the kernel.

When a code cell is executed, its content is sent to the kernel associated with the notebook. The results that are returned from this computation are then displayed in the notebook as the cell’s output. The output is not limited to text, with many other possible forms of output are also possible, including figures and HTML tables.

Markdown cells

You can document the computational process in a literate way, alternating descriptive text with code, using rich text. This is accomplished by marking up text with the Markdown language. The corresponding cells are called Markdown cells. The Markdown language provides a simple way to perform this text markup, that is, to specify which parts of the text should be emphasized (italics), bold, form lists, etc.

If you want to provide structure for your document, you can also use markdown headings. Markdown headings consist of 1 to 6 hash # signs followed by a space and the title of your section. The Markdown heading will be converted to a clickable link for a section of the notebook. It is also used as a hint when exporting to other document formats, like PDF.

When a Markdown cell is executed, the Markdown code is converted into the corresponding formatted rich text. Markdown allows arbitrary HTML code for formatting.

Raw cells

Raw cells provide a place in which you can write output directly. Raw cells are not evaluated by the notebook.

Keyboard shortcuts

All actions in the notebook can be performed with the mouse, but keyboard shortcuts are also available for the most common ones. These are some of the most common:

Shift-Enter: run cell. Execute the current cell, show any output, and jump to the next cell below. If Shift-Enter is invoked on the last cell, it creates a new cell below. This is equivalent to clicking the Cell -> Run menu item, or the Play button in the toolbar.
Esc: Command mode. In command mode, you can navigate around the notebook using keyboard shortcuts.
Enter : Edit mode. In edit mode, you can edit text in cells.

Tutorials

You can find links to sample notebooks that we have used in past trainings that may be useful to explore the system:

Users: Notebooks Environments

Mon, 01 Jan 0001 00:00:00 +0000

JupyterHub is an extensible platform that supports different combination of software and hardware for running your applications. For the EGI service, you have the choice to run:

the default environment that provides a data-science ready stack with support for Python, R, Julia, and Octave.
the MATLAB environment, for running MATLAB.

After logging into the service, you will be shown a form for selecting the environment, pick the desired one and click start

The list of options is customised for your community, depending on the VOs you are member of, you will be able to select new options tuned to your needs, e.g. eiscat.se VO members are presented with a runtime environment built matching their needs:

If you have special needs that cannot be covered with the default environment environment, let us know by opening a ticket to the Notebooks Support Unit in GGUS.

Additionally, you can build your own environment via Replay. This will allow you to build reproducible and shareable environments for your notebooks.

Users: Data Management in Notebooks

Mon, 01 Jan 0001 00:00:00 +0000

Every user of the EGI Notebooks catch-all instance has a persistent home to store any notebooks and associated data. The content of this home directory will be kept even if your notebook server is stopped (which can happen if there is no activity for more than 1 hour). Modifications to the notebooks environment outside the home directory are not kept (e.g. installation of libraries). If you need those changes to persist, let us know via a GGUS ticket to the Notebooks Support Unit. You can also ask for increasing your default home allocation via ticket.

Note

Your home directory is backed up daily.

Import notebooks into your workspace

The Notebooks service default environment includes nbgitpuller, an extension to sync a git repository one-way to a local path. You can generate a shareable link by filling in the nbgitpuller link generator with your git repository.

Alternatively, you can also use Replay for providing a link to notebooks and their computing environment.

Getting data in/out

Your notebooks have outgoing internet connectivity so you can connect to any external service to bring data in for analysis. As with input data, you can connect to any external service to deposit the notebooks output.

This is convenient for smaller datasets but not practical for larger ones, for those cases we can offer integration with several data services.

EGI DataHub

EGI DataHub provides a scalable distributed data infrastructure. It offers a tight integration with Jupyter and notebooks with specific drivers that make the DataHub Spaces accessible from any notebook.

Whenever you log into the service, supported DataHub spaces will be available under the datahub folder. If you need support for any additional space, please open a ticket in GGUS to add it.

By default the notebooks-shared space is open for writing to any EGI Notebooks user part of the vo.notebooks.egi.eu VO. Please check the File Management section in the EGI DataHub documentation for more information on how to upload files.

EUDAT B2DROP

EUDAT B2DROP is a low-barrier, user-friendly and trustworthy storage environment which allows users to synchronise their active data across different desktops and to easily share this data with peers. EUDAT offers a free public instance of B2DROP for any researcher with a limited quota.

The data on B2DROP can be synchronised with EGI Notebooks so you can share content between the two services. This offers an easy-to-use storage and compute platform for the long-tail of science.

Here is how you can get them synchronised. First, make sure you have access to B2DROP. Then, configure app username and app password on B2DROP’s security settings. Now, back to EGI Notebooks, click on the B2DROP connection drop-down menu when you start your session:

Enter the app username and app password created previously, with the option to save them for future logins:

You will see a b2drop folder in the list of folders (left panel) of the EGI Notebooks that is synchronised with the content on B2DROP:

D4Science Workspace

D4Science VREs provide a shared workspace via a dedicated API. EGI Notebooks embedded in D4Science VREs will automatically show the user’s workspace at the workspace directory. You can browse and use as any regular file.

Shared folders

The Notebooks service can enable shared folders for users, either in read-only or read-write mode. These are specially meant for community instances for easing the sharing of data between all the users of the service. In the catch-all instance the datasets directory serves as an example of such feature.

Other services

We are open for integration with other services for facilitating the access to input and output data. Please contact support _at_ egi.eu with your request so we can investigate the best way to support your needs.

Users: Notebooks Integrations

Mon, 01 Jan 0001 00:00:00 +0000

Notebooks running on EGI can access other existing computing and storage services from EGI or other e-Infrastructures. For data services, check data section of the documentation

API access to notebooks

You can use the JupyterHub REST API with a valid Check-in token at https://notebooks.egi.eu/services/jwt.

The endpoint expects an Authorization header to be present in the requests with your token, e.g.:

$ curl -H "Authorization: bearer $TOKEN" https://notebooks.egi.eu/services/jwt
{"version":"5.3.0"}

You can use the Python requests library to interact with the API. The code below shows an API GET request to /user using the token available in the Notebooks session for authorization. The response contains information about the user.

import requests

api_url = "https://notebooks.egi.eu/services/jwt"

r = requests.get(api_url + "/users",
 headers={
 "Authorization": "Bearer {}".format(open("/var/run/secrets/oidc/access_token").read())
 }
)

r.raise_for_status()
users = r.json()

EGI services: access tokens

Most services integrated with EGI Check-in can handle valid access tokens for authorising users. These are short-lived (normally less than 1-hour) and need to be renewed for longer usage. With EGI Notebooks, you can get a ready-to-use token at any time.

You can obtain a new token at any time with the JupyterLab extension available from the Jupyter Lab interface. You have two options:

Copying the token to clipboard
Saving the token to disk so it’s accessible from your notebook files and/or terminal. Optionally you can request the token to be continuously refreshed on disk so you don’t need to request new tokens.

Whether you have selected to mount the token on the server creation step or if you save it to disk with the extension, the token will be available at /var/run/secrets/egi.eu/access_token. See for example below some python code to access cloud providers of the EGI cloud, where a list of VMs is obtained from CESGA’s OpenStack:

from keystoneauth1.identity import v3
from keystoneauth1 import session
from novaclient import client


with open("/var/run/secrets/egi.eu/access_token") as f:
 access_token = f.read()

auth = v3.OidcAccessToken(auth_url="https://fedcloud-osservices.egi.cesga.es:5000/v3",
 identity_provider="egi.eu",
 protocol="openid",
 project_id="3a8e9d966e644405bf19b536adf7743d",
 access_token=access_token)
sess = session.Session(auth=auth)
nova = client.Client(session=sess, version=2)
nova.servers.list()

fedcloud client

A direct benefit of the integration with access tokens in EGI Notebooks is that you can easily work with the fedcloud client. Once logged into the EGI Notebooks open a terminal and run:

export OIDC_ACCESS_TOKEN=`cat /var/run/secrets/egi.eu/access_token`
fedcloud token check

If the fedcloud command is not available, please follow the getting started guide to get it.

EGI Workload Manager (DIRAC)

If you are using a Notebooks instance integrated with EGI Workload Manager, you may access the DIRAC client via CVMFS with the following two command lines:

$ source /cvmfs/dirac.egi.eu/dirac/bashrc_egi_dev
$ dirac-login --issuer=https://dirac.egi.eu/auth

D4Science

If you are using a Notebooks instance integrated with D4Science, you can easily invoke DataMiner or any other D4Science functionality as the service will provide the GCUBE_TOKEN environment variable with a valid token.

This code will print the list of DataMiner methods available within your VRE:

import os
from owslib.wps import WebProcessingService

# init http header parameter and base folders for gCube REST API
gcube_vre_token_header = {'gcube-token': os.environ["GCUBE_TOKEN"]}
# init WPS access for DataMiner algorithms
dataminer_url = 'https://dataminer-prototypes.d4science.org/wps/WebProcessingService'
wps = WebProcessingService(dataminer_url, headers=gcube_vre_token_header)

for process in wps.processes:
 print('- Name: ', process.title)

DataMiner algorithms can be invoked also from Notebooks, this code shows a sample:

from owslib.wps import ComplexDataInput, monitorExecution

# define processid and inputs
processid = 'org.gcube.dataanalysis.wps.statisticalmanager.synchserver.mappedclasses.transducerers.WOFOST_CLOUD_V0_2_1'
inputs = [
 ('ClassToRun', 'nl.wur.wofostsystem.App'),
 ('FileInput',
 ComplexDataInput(
 'https://data.d4science.org/shub/E_eVhZTzBWWktOaVJxQjJkdTUxR3FHaTFFdE9BTDYrZkZxQnFWcGMyaVVJbXptejdDOEFpSVNmam82RllkRUJ6cA==',
 mimeType="text/xml")
 )
]

# execute the process
execution = wps.execute(processid, inputs)
monitorExecution(execution, sleepSecs=5, download=True)
print(execution.status)

Note that inputs that point to a URL should be specified using the ComplexDataInput class as shown above.

Other third-party services

We are open for integration with other services that may be relevant for your research. Please contact support <at> egi.eu with your request so we can investigate the best way to support your needs.

Users: Frequently Asked Questions

Mon, 01 Jan 0001 00:00:00 +0000

<– cSpell:words magics –>

How do I install library X?

You can install new software easily on the notebooks using conda or pip. The %conda and %pip magics can be used in a cell of your notebooks to do so, e.g. installing rdkit:

%conda install rdkit

Once installed you can import the library as usual.

Warning

Any modifications to the libraries/software of your notebooks will be lost when your notebook server is stopped (automatically after 1 hour of inactivity)!

Can I request library X to be installed permanently?

Yes! Just let us know what are your needs. You can contact us via:

Opening a ticket in the EGI Helpdesk, or
Creating a GitHub Issue

We will analyse your request and get back to you.