Users: Notebooks

Mon, 01 Jan 0001 00:00:00 +0000

The more you go in data analysis, the more you understand that the most suitable tool for coding and visualizing is not pure code, or some integrated development environment (IDE), nor data manipulation diagrams (such as workflows or flowcharts). From some point on you just need a mix of all these – that is what notebook platforms are, Jupyter being the most popular of them.

What is it?

EGI Notebooks is a service-like environment based on the Jupyter technology, offering a browser-based tool for interactive data analysis.

The Notebooks environment provides users with notebooks where they can combine text, mathematics, computations and rich media output. EGI Notebooks is a multi-user service that can scale on demand, being powered by the compute services of EGI.

EGI Notebooks provides the well-known Jupyter interface for notebooks, with the following added features:

Integration with EGI Check-in allows you to login with any eduGAIN or social accounts (e.g. Google, Facebook)
Persistent storage associated with each user is available in the notebooks environment
Customisable with new notebook environments, expose any existing notebook to your users
Can easily use EGI compute and storage services from your notebooks, as your notebooks run on EGI infrastructure

Service Modes

We offer different service modes depending on your needs

Notebooks for researchers

Individual users can use the centrally operated service from EGI. Users can log in, write and play and re-play notebooks by:

creating an EGI account
Enrolling to the one of the supported VOs such as vo.notebooks.egi.eu VO
accessing https://notebooks.egi.eu/

The central instance supports the following VOs:

Notebooks for communities

User communities can have their customised EGI Notebooks service instance. EGI offers consultancy, support, and can operate the setup as well. A community specific setup allows the community to use the community's own Virtual Organisation (i.e. federated compute and storage sites) for Jupyter, add custom libraries into Jupyter (e.g. discipline-specific analysis libraries) or have fine grained control on who can access the instance (based on the information available to the EGI Check-in AAI service).

EGI currently operates community instances for:

D4Science. These instances are accessed through specific Gateways: SoBigData, Blue-Cloud, D4Science Services and EOSC-Pillar. Check with D4Science support for more information.

Training instance

EGI can provide a custom and temporary instance of the Notebooks service for training events, if you have a specific event where you would like to use EGI Notebooks as platform for your training, let us know.

Note

This instance may not use the same software version as in production and may not be always available, as it is typically configured only for specific training events.

Users: Replay

Mon, 01 Jan 0001 00:00:00 +0000

What is it?

Replay allows the re-creation of a custom computing environment for reproducible execution of notebooks (and potentially many other types of applications). Users who create their own notebooks in the EGI Notebooks to analyze data can easily create a shareable link for those notebooks in the form of a GitHub repository. Based on this link, anyone can then reproduce the same data analysis using the link in the EGI Replay service.

The service builds on BinderHub, an Open Source tool that allows to build docker images from a Git repository and then makes them available through your browser.

EGI Replay offers a service similar to the publicly accessible mybinder.org site. However, EGI Replay has the following additional features:

Access with academic user accounts: login via Check-in that’s connected to eduGAIN and social media accounts.
Access to scalable storage: selected storage spaces of EGI DataHub are directly available under the datahub folder, simplifying the access to shared data from Binder notebooks.
Guaranteed capacity: environments have 2GB of RAM guaranteed and can reach 6GB as maximum.
Persistent sessions: There is no hard limit on the session time per user, although sessions will be shut down automatically after 1 hour of inactivity (see session limitations at the public mybinder.org service).
Access to the rest of EGI services: a personal access token is available in the Replay session to interact with the rest of the EGI infrastructure.
Community Replay environments: User communities can have their customized Replay service instance from EGI, with extra features as requested (such as access to GPUs, integration with community specific data repositories and services). EGI offers consultancy and support the setup of these instances, and provides operational oversight for them.

EGI Replay has been presented in the EGI Webinars. See slides and recording.

Reproducible research

Replay facilitates the sharing and reproducibility of digital data analysis:

Users can define their computational analysis in the EGI Notebooks service.
Once the notebook is ready for publishing, it can be shared in a GitHub repository.
Optionally, users can use the Zenodo-GitHub integration for generating DOIs that can be cited in publications and can be discovered by fellow researchers
Anyone can use the link to the GitHub repository or Zenodo DOI to reproduce the computational analysis in EGI Replay.

Access to the service

EGI’s Replay has the same access conditions as the centrally operated Notebooks service from EGI. Before using the service, you need to have an EGI account and be a member of one of the supported resource pools (alias Virtual Organisations). Follow the instructions on the EGI Replay login page for access.

The default capacity allocated for users includes up to 4 vCPU cores and 6GB of RAM for every user.

Creating a Binder repository

Replay starts from a code repository that contains the code or notebook you’d like to run and a set of configuration files that specify what’s the exact computational environment your code needs to run.

Replay then creates a reproducible container using repo2docker, and generates a user session to interact with the container in the browser.

The configuration for building the container supports specifying conda environments; installing Python, R and Julia environments; installing additional OS packages; and even complete custom Dockerfiles to bring any application to the system. The code repository can be hosted on popular git hosting platforms like GitHub and GitLab and can also be referenced with a DOI from Zenodo, FigShare or Dataverse. You can learn more on the configuration of your repository with Replay at the Binder user documentation.

You can start by forking the EGI-Federation/binder-example GitHub repository for creating your own reproducible environment. To run this directly on EGI’s Replay click on the button below:

You can create such link to share your notebooks from the Replay interface, as shown in the screenshot below, you can copy the URL shown when the building is in progress:

The binder examples organisation on GitHub contains more sample repositories for common configurations that can help you getting started.

Accessing data

Your notebooks running in Replay have outgoing internet connectivity, so you can connect to external services to bring data in for analysis or deposing the notebooks output.

Every session that you start will also provide access to your spaces in the DataHub under a folder named datahub. Only those spaces configured to be mounted locally will be made available automatically. Check the documentation for the Notebook’s DataHub support for more information.

Errors when launching

You can only have one notebook server instance running at any given time, if you try to launch a second instance while there is already one running you will get an error like shown in the screenshot:

In that case, you can either stop your existing server or connect to it from your Replay JupyterHub home:

Documentation – Development Environments

Users: Notebooks

What is it?

Service Modes

Notebooks for researchers

Notebooks for communities

Training instance

Note

Users: Replay

What is it?

Reproducible research

Access to the service

Creating a Binder repository

Accessing data

Errors when launching