Documentation – Tutorials & Guides

Users: Foundation level

Mon, 01 Jan 0001 00:00:00 +0000

Target Audience: Newcomers of the EGI Infrastructure

Are you new to EGI? And maybe a bit confused about what EGI is, what we do, how we are structured and how you can collaborate with us? Then this is the webinar for you!

"An introduction to EGI" (October, 2021)

Agenda, slides and recording: https://indico.egi.eu/event/5464/contributions/15771/

About: The EGI Federation is an international e-Infrastructure to provide advanced computing and data analytics services for research and innovation. EGI federates compute resources, data management services, support teams and various online, thematic services from over 30 countries and international institutes, to make those available for researchers, innovators and educators. This webinar will provide a basic introduction to EGI, covering all the fundamental topics that you need to know before deep-diving on the advanced services to conduct world-class research and innovation.

Target Audience:

Scientific communities, projects, research infrastructures who wish to learn about the services of EGI.
E-infrastructure and other service providers who want to be part of a pan-European federation to support international science.

"EGI Federation - Advanced Computing for Research" (April 2020)

Agenda, slides and recording: https://indico.egi.eu/event/5083/

About: The EGI federation is the largest distributed computing infrastructure in the world, and brings together hundreds of data centres worldwide and also includes the largest community cloud federation in Europe with tens of cloud providers across most of the European countries offering IaaS cloud and storage services. The current federated resources represent altogether more than 350 Petabytes of online storage and 380 Petabytes of archival storage supported by approximately 1 M Cores. EGI expanded the federation of its facilities with other non-European digital infrastructures in North America, South America, Africa-Arabia and the Asia-Pacific region, as such EGI fully realised the “Open to the World” vision. In order to interoperate at international level, EGI and its partners operate in the context of a lightweight collaboration framework defining rules of participations via a corpus of policies and technical guidelines.

EGI offering includes a federated IaaS cloud to run compute- or data-intensive tasks and host online services in virtual machines or docker containers on IT resources accessible via a uniform interface; high-throughput data analysis to run compute-intensive tasks for producing and analysing large datasets and store/retrieve research data efficiently across multiple service providers; federated operations to manage service access and operations from heterogeneous distributed infrastructures and integrate resources from multiple independent providers with technologies, processes and expertise offered by EGI; consultancy for user-driven innovation to assess research computing needs and provide tailored solutions for advanced computing.

The notion of a distributed infrastructure offering advanced resources and services for data-intensive processing in research and innovation has been part of the EGI mission and vision since the EGI design and implementation that started with the DataGrid project back in 2000 under the leadership of CERN. Distributed processing of data supported by a pan-European broadband network infrastructure, solutions for trust and identity management and the Grid middleware, have been the enablers of two Nobel prizes in Physics (2013 and 2017), and many more data-driven scientific discoveries in high energy physics, astronomy and astrophysics, health and medicine, and earth sciences resulting in more than 3,000 open access scientific publications enabled each year. In this webinar Gergely will provide an overview of EGI, the pan-european federation of national e-infrastructures and he will explain how EGI supports big data based Open Science and contributes to the implementation of the EOSC vision. The talk goes through the services that EGI provides for scientific communities, and for the members of the federation, and will show how these services benefit research, science and innovation in Europe and worldwide.

Users: Intermediate level

Mon, 01 Jan 0001 00:00:00 +0000

Target Audience: Researchers, and IT-service providers who support research and education.	"Webinar: EGI Cloud Services: Updates, Innovations, and Future Directions" (February, 2025) Agenda, slides and recording: https://www.egi.eu/event/webinar-egi-cloud-services-updates-innovations-and-future-directions/ About: This webinar highlights the latest developments in EGI Cloud Service, designed to support cutting-edge research and innovation. It provides an overview of the service features, recent updates to the EGI Cloud infrastructure, and new capabilities introduced to address the evolving needs of the scientific community. Suggested tutorials Create your first Virtual Machine (VM): Step by step guide to get your first Virtual Machine up and running Accessing virtual machines with SSH: Accessing virtual machines with SSH
Target Audience: This presentation is intended for site administrators, site managers, security professionals, EGI management, and anyone who could be affected by a security incident and required to manage or respond to it.	"EGI CSIRT IRTF 2024 in Review: Incidents, Learnings, and Plans for 2025" (January, 2025) Agenda, slides and recording: https://www.egi.eu/event/webinar-egi-irtf-2024-in-review-incidents-learnings-and-plans-for-2025/ About: We will start by presenting the role of the EGI CSIRT Incident Response Task Force (IRTF) in strengthening security across our community. Discover who we are, what drives our mission, and how we collaborate with key partners to protect the computing grid. Then, we’ll explain the initiatives from 2024, sharing the results of the Communications Challenges, the metrics of security monitoring, and present the various incidents that occurred during the year, along with the lessons that can be learned from them. Looking ahead, we will present our initiatives for 2025, designed to enhance visibility into the security status of sites, strengthen collaboration with VOs, and foster greater engagement from sites to improve their overall security posture.
Target Audience: Scientific communities, developers, IT service providers, and end users.	"Webinar: EGI DataHub: an innovative platform for ingestion, management and publishing of distributed scientific data" (September, 2024) Agenda, slides and recording: https://www.egi.eu/event/egi-datahub-platform-ingestion-management-publishing-distributed-scientific-data/ About: Watch our webinar on the latest advancements in EGI DataHub, an innovative distributed data access and management platform powered by Onedata technology. EGI DataHub unites 18 data providers across Europe, offering seamless data access and management capabilities.
Target Audience: e-Infrastructure platform/technology providers Domain scientists and research Data Scientists	"Webinar: Managing Secrets in the EGI Infrastructure with EGI Secrets Store" (July, 2024) Agenda, slides and recording: https://www.egi.eu/event/webinar-managing-secrets-in-the-egi-infrastructure-with-egi-secrets-store/ About: In this session, we will explore the EGI Secrets Store, a powerful solution designed to help you securely manage credentials, tokens, passwords, and other sensitive information required by your applications and services during deployment and operation. Traditional methods of storing these secrets in clear text within code repositories or configuration files pose significant security risks and make rotation cumbersome. EGI Secrets Store addresses these challenges by providing a secure, efficient, and standardised way to handle secrets.
Target Audience: Scientific communities, developers, IT service providers, and end users.	"Webinar: Reproducible Open Science with EGI Replay" (April, 2024) Agenda, slides and recording: https://www.egi.eu/event/reproducible-open-science-with-egi-replay/ About: In this webinar we showcase EGI Replay, a managed service for the EGI Community based on Binder. EGI Replay offers users the ability to effortlessly reproduce intricate calculations, simulations, and visualizations by importing Jupyter Notebooks and their respective runtime environments. With a single link, users can easily share their work with collaborators, making Replay an invaluable tool for workshops, scientific workflows, and team collaboration.
Target Audience: Scientific communities, developers, integrators and end users.	"Data Management in EGI with Rucio and FTS" (October, 2021) Agenda, slides and recording: https://indico.egi.eu/event/5711/ About: Rucio is a data management software, originally developed for ATLAS at CERN to supersede their previous data management software 10 years ago. Since then Rucio has been constantly developed by ATLAS and other communities that have come to use Rucio, ensuring that it is a feature rich, and well maintained open software. Multi-VO Rucio implemented by the STFC in the UK hosts Rucio as a service for many communities. This is to provide communities the opportunity to use Rucio for their data management solution, without having to learn about, and host their own instance of Rucio. FTS is a low level data movement service, responsible for reliable bulk transfer of files between storages. It's responsible for globally distributing the majority of the LHC data across the WLCG infrastructure and it supports many communities is EGI. In this webinar we introduce the main functionalities and show how to interact with the services in order to schedule transfers. Suggested tutorials Data transfer with grid storage: Use EGI Data transfer to handle data in grid storage
Target Audience: Scientific communities, developers, integrators and end users.	"Using EGI Cloud infrastructure with fedcloudclient" (September, 2021) Agenda, slides and recording: https://indico.egi.eu/event/5694/ About: The FedCloud client is a high-level Python package for a command-line client designed for interaction with the OpenStack services in the EGI infrastructure. The client can access various EGI services and can perform many tasks for users including managing access tokens, listing services, and mainly execute commands on OpenStack sites in EGI infrastructure. The webinar will provide tutorial and demonstration of using fedcloudclient in EGI Cloud infrastructure. Suggested tutorials Automate with oidc-agent, fedcloudclient, terraform and Ansible: Step by step guide to automating the deployment using Ansible with Terraform, oidc-agent and fedcloudclient
Target Audience: Scientific communities, developers, IT service providers, and end users.	"Using Dynamic DNS service in the EGI Cloud infrastructure" (June, 2021) Agenda, slides and recording: https://indico.egi.eu/event/5559/ About: Nowadays, more and more services are dynamically deployed in Cloud environments. Usually, the services hosted on virtual machines in Cloud are accessible only via IP addresses or pre-configured hostnames given by the target Cloud providers, making it difficult to provide them with meaningful domain names. The Dynamic DNS service provides a unified, federation-wide Dynamic DNS support for VMs in EGI infrastructure. Users can register their chosen meaningful and memorable DNS host names in given domains (e.g. my-server.vo.fedcloud.eu) and assign to public IPs of their servers. By using Dynamic DNS, users can host services in EGI Cloud with their meaningful service names, can freely move VMs from sites to sites without modifying server/client configurations (federated approach), and can request valid server certificates in advance (critical for security). The webinar will provide demonstration and tutorial, also practical advice on using Dynamic DNS service in realistic user scenarios.
Target Audience: Developers and administrators of relying parties that want to connect to Check-in for authenticating users and managing their access rights.	"Providing controlled access to distributed resources and services with EGI Check-in: the provider perspective"(May, 2021) Agenda, slides and recording: https://indico.egi.eu/event/5494/ About: This webinar will help new services to integrate with Check-in, the EGI Authentication & Authorisation Infrastructure enabling secure access to relying parties. The target group of the training are developers and administrators of services that want to connect to Check-in for user authentication and authorisation. The training will showcase the use of the EGI Check-In Federation Registry tool for managing the lifecycle of a relying party, i.e. registration, reconfiguration and de-registration. The training will include hands-on sessions for the participants to integrate their own relying party to Check-In.
Target Audience: Scientific communities and IT-service providers who support research and education.	"Access and analyze data from the EGI DataHub with Jupyter notebooks and MATLAB" (May, 2021) Agenda, slides and recording: https://indico.egi.eu/event/5499/ About: Good, clean data is hard to come by. The EGI provides scientists and researchers access to a large collection of public datasets from data centres globally. This data can be accessed using the EGI Jupyter Notebook service. MATLAB users can now analyse this data using the familiar MATLAB desktop, via a web browser, on the EGI’s resources. In this webinar, you will learn how to: Use your MATLAB licence to login to the EGI MATLAB installation Access data from the EGI DataHub Read in scientific data into MATLAB Analyse and visualise data using computational notebooks called MATLAB Live Scripts. Share your MATLAB code with your peers. Suggested tutorials Access DataHub from a VM: Use data in EGI DataHub from a virtual machine. Create a VM with Jupyter and DataHub: Step by step guide to get a Virtual Machine for Jupyter and DataHub in your cloud provider
Target Audience: Scientific communities and IT-service providers.	"Monitoring services with ARGO" (May, 2021) Agenda, slides and recording: https://indico.egi.eu/event/5496/ About: ARGO is a lightweight service for Service Level Monitoring designed for medium and large sized Research Infrastructures. Services are monitored with probes compatible with flexible and widely adopted Nagios plugin format. Besides basic availability checks, services can be monitored by emulating typical user scenarios that allows to derive the quality of service the actual user gets. ARGO offers near real-time status updates which allow both end-users and site admins to have an overview of the services offered at any given point in time via a web user interface and via enriched email notifications. ARGO generates custom Availability and Reliability reports based on the aggregated monitoring data. The rich monitoring data collected in ARGO service is actually stored in a highly flexible big data friendly form using state-of-the-art computational pipelines and formats. This provides the ability to reuse & analyse the data in different ways such as to highlight service usage patterns and provide a number of trends and insights. In this training session we are going to show the process we follow to monitor a new service with ARGO. In addition, the real time computations and the results via the alerts, API and UI will be shown. ARGO is a service jointly developed and maintained by CNRS, GRNET and SRCE.
Target Audience: Scientific communities, and programmers who support research and education.	"Managing Singularity, Docker and udocker containers, Kubernetes clusters in the EGI Cloud" (April, 2021) Agenda, slides and recording: https://indico.egi.eu/event/5492/ About: Containers provide a streamlined way to build, test, deploy, and redeploy applications on different environments: from the developer’s local machine to any cloud provider. Containers make it easy for developers to package applications and for operators to manage and deploy those applications on the infrastructure. Container orchestrators like Kubernetes facilitate the management of containerized workloads and services, using declarative configuration and automation. In this webinar we will introduce the different runtimes available for executing containers in EGI infrastructure and will show how to manage Kubernetes clusters to get your containers under control executed on EGI cloud providers.
Target Audience: Users and application experts of the EGI communities.	"DIRAC Services for EGI users" (October, 2020) Agenda, slides and recording: https://indico.egi.eu/event/5267/ About: DIRAC is a complete framework for building distributed computing systems of any level of complexity. Initially developed for the LHCb High Energy Physics experiment at the LHC collider at CERN, the framework was generalised for the use by multiple scientific communities in various domains. Services based on the DIRAC software are offered by several grid infrastructure projects such as France-Grilles or GridPP/UK. Since 2014, the DIRAC services have also been provided for the EGI users. During the webinar, an overview of the DIRAC framework will be presented together with a number of services offered to the users by EGI: how to manage user jobs in the EGI infrastructure, how to connect custom computing and storage resources, how to manage user data as well as automate regular tasks. Extending DIRAC with community custom services will also be discussed.
Target Audience: IT service providers, site and NGI operation managers (new member of staff).	"EGI Operations and responsibilities of an NGI" (October 2020) Agenda, slides and recording: https://indico.egi.eu/event/5268/ About: This webinar will give an overview of tried and tested approaches to federated operations, both at the level of the infrastructure as well as at the national level. It will cover the most important aspects and day-to-day work covered by staff - both at the international infrastructure level at EGI as well as at an example National Grid Initiative (NGI). Example scenarios will be presented along with the tools used to deal with the scenarios. Finally there will be an opportunity for questions and discussions arising from the topics covered.
Target Audience: Site administrators and cluster administrators; CVMFS power users.	"CernVM-FS for Containers" (October 2020) Agenda, slides and recording: https://indico.egi.eu/event/5251/ About: Delivering complex software stacks across a worldwide distributed system is a challenge in high-throughput scientific computing. The global-scale virtual file system CernVM-FS distributes more than a billion software binaries to hundreds of thousands of machines around the world. In this webinar, we will present the latest developments with regard to CernVM-FS container integration. Containers and CernVM-FS team up nicely: containers provide the isolation capabilities that decouple the application stack from the underlying platform and CernVM-FS provides efficient distribution means for the containerized software binaries. Containers are an enabling technology to harness opportunistic resources and HPC facilities. CernVM-FS enables the use of such resources at scale. In this webinar, we will show how existing repositories can be used with several popular container runtimes, such as docker, podman, singularity, and kubernetes. We will also show how operating system containers themselves can be efficiently distributed through CernVM-FS. Lastly, we will highlight an upcoming new way of publishing content from within a container. This makes it easy to set up, build and test and deploy-to-cvmfs pipelines on kubernetes.
Target Audience: Scientific communities and IT-service providers who operate IdP for them.	"The EGI AAI Check-In service for scientific communities" (May 2020) Agenda, slides and recording: https://indico.egi.eu/event/5088/ About: The EGI Check-in service (also called EGI AAI proxy) enables access to EGI services and resources using federated authentication mechanisms. Specifically, the proxy service is operated as a central hub between federated Identity Providers (IdPs) residing ‘outside’ of the EGI ecosystem, and Service Providers (SPs) that are part of EGI. The main advantage of this design principle is that all entities need to establish and maintain technical and trust relation only to a single entity, the EGI AAI proxy, instead of managing many-to-many relationships. In this context, the proxy acts as a Service Provider towards the Identity Providers and as an Identity Provider towards the Service Providers. Through the EGI AAI proxy, users are able to authenticate with the credentials provided by the IdP of their Home Organisation (e.g. via eduGAIN), as well as using social identity providers, or other selected external identity providers (support for eGOV IDs is also foreseen). To achieve this, the EGI AAI has built-in support for SAML, OpenID Connect and OAuth2 providers and already enables user logins through Facebook, Google, LinkedIn, and ORCID. In addition to serving as an authentication proxy, the EGI AAI provides a central Discovery Service (Where Are You From – WAYF) for users to select their preferred IdP. The EGI AAI proxy is also responsible for aggregating user attributes originating from various authoritative sources (IdPs and attribute provider services) and delivering them to the connected EGI service providers in a harmonised and transparent way. Service Providers can use the received attributes for authorisation purposes, i.e. determining the resources the user has access to. During this webinar we will give an overview about the service and provide guidelines to support the resource providers’ and communities' needs for federated access through the EGI AAI Check-In service. The webinar will also cover more advanced workflows for addressing non-web-based access use cases (e.g. command line and API).
Target Audience: Scientific communities, for programmers and IT-service providers who support research and education.	"The EGI Notebooks service: Support for analytics and big data visualisation in the cloud" (May 2020) Agenda, slides and recording: https://indico.egi.eu/event/5087/ About: The EGI Notebooks service is an environment based on Jupyter and the EGI cloud service that offers a browser-based, scalable tool for interactive data analysis. The notebooks environment provides users with notebooks where they can combine text, mathematics, computations and rich media output. The service, in production since late 2019, is offered in two options: (i) Notebooks for researchers: EGI offers a basic instance of the Notebooks as an open service. Any researcher can access this automatically to write and play notebooks on limited capacity cloud servers. (ii) Notebooks for communities: EGI offers customised Notebooks service to scientific communities. Such customised instances can be hosted on special hardware (for example with fat nodes and GPUs), can offer special libraries, data import/export and user authentication systems. During the webinar Enol will go through the main features of the EGI Notebooks service and he will explain how to use it with Binder and other open-source solutions to implement Open Science.
Target Audience: Scientific communities, for programmers and IT-service providers who support research and education.	"Introduction to Slurm" (March 2023) About: Slurm is an open-source, fault-tolerant, and highly scalable cluster management and job scheduling system for large and small Linux clusters. In this tutorial, we briefly discuss the benefits of using batch schedulers, the motivations to use Slurm and provide a list of commands to get started with Slurm. Suggested material Slides
Target Audience: Scientific communities, for programmers and IT-service providers who support research and education.	"Introduction to Snakemake" (December 2022) About: The Snakemake workflow management system is a tool for creating reproducible and scalable data analyses. Workflows are described via a human-readable, Python-based language. They can be seamlessly scaled to server, cluster, grid and cloud environments without the need to modify the workflow definition. Finally, Snakemake workflows can entail a description of the required software, which will be automatically deployed to any execution environment. Slides and code https://github.com/c-scale-community/c-scale-tutorial-snakemake
Target Audience: Scientific communities, for programmers and IT-service providers who support research and education.	"Leveraging the Onedata Platform for Long-Term Data Archiving" (June 2023) About: In this presentation, we will discuss the latest advancements in the Onedata platform, focusing on its new features for long-term data archiving and processing. We will demonstrate how the platform has been optimized to meet the Open Archival Information System (OAIS) standards, ensuring the reliable preservation and accessibility of archived information over time. Furthermore, we will explore the integration of Function as a Service (FaaS) capabilities in the platform, allowing for seamless and scalable data processing on demand. By combining the robust archiving capabilities of the OAIS standard with the flexibility of FaaS, the Onedata platform emerges as a powerful solution for organizations seeking efficient and reliable management of their long-term data storage and processing needs.

Users: Advanced level

Mon, 01 Jan 0001 00:00:00 +0000

Target Audience: Scientific communities, for programmers and IT-service providers who support research and education.	"Webinar: TOSCA-based cloud orchestration and deployment with Infrastructure Manager (IM)" (June, 2024) Agenda, slides and recording: https://www.egi.eu/event/tosca-based-cloud-orchestration-and-deployment-with-infrastructure-manager-im/ About: In this webinar we focus on the usage of the IM Dashboard, an easy-to-use web interface, designed for users to deploy a set of predefined and well-tested customisable virtual infrastructures on top of a wide range of cloud providers with a set of mouse clicks. Furthermore, it uses a two-step wizard that enables it to compose more complex templates, as shown in the demo.
Target Audience: Scientific communities, developers, integrators and end users.	"Webinar: Introduction to the new generation EGI container execution platform" (January, 2024) Agenda, slides and recording: https://www.egi.eu/event/webinar-introduction-to-the-container-platform/ About: The new generation EGI container execution platform is a system that delivers fully managed Kubernetes clusters for end users. The webinar covers how to build container images, how to store and share them through the Harbor repository, and how to run them on fully managed Kubernetes clusters with the Rancher service.
Target Audience: e-Infrastructure platform and technology providers and those researching or applying (distributed) deep learning at any level.	"Webinar: Distributed Deep Learning with Horovod" (March, 2023) Agenda, slides and recording: https://www.egi.eu/event/webinar-distributed-deep-learning-with-horovod/ About: This webinar is focused on the Horovod distributed deep learning framework and the reference architecture and service(s) built for supporting it. The aim of the presented works is to enable the efficient utilisation of cloud resources in the heavily resource intensive task of distributed deep learning. The concept of reference architectures is also briefly presented, along with the experiences gained from their continuous development.
Target Audience: Scientific communities, developers, integrators and end users.	"FAIR EVA (Evaluator, Validator & Advisor)" (April, 2022)” Agenda, slides and recording: https://indico.egi.eu/event/5876/ About: FAIR EVA (Evaluator, Validator and Advisor) has been developed to check the FAIRness level of digital objects from different repositories or data portals. It requires the object identifier and the repository to check and it can be adapted to different contexts and environments. Developed within the EOSC-Synergy project it aims at helping data producers and data managers to evaluate the adoption of the FAIR principles based on the RDA indicators. This webinar will present the tool as well as show how it can be deployed, how the different tests work and how it can be adapted to different data systems.
Target Audience: Scientific communities, developers, integrators and end users.	"OpenRDM - FAIR research data management as a service to the scientific community" (January, 2022) Agenda, slides and recording: https://indico.egi.eu/event/5753/ About: openRDM.eu provides FAIR research data management. It offers research data management as a service to the scientific community, based on the powerful openBIS platform. The service is available as a preview version containing an openBIS instance. Preview is intended for end-users to learn service & eventually plan on-premise and/or, own cloud based deployment. Alternatively, self-hosting using local IT infrastructure at the respective institution can also be agreed. Consulting & support for on-premise and/or own cloud based deployment of openBIS is also offered along with user support including data model generation, to be able to import data into openBIS & training for the use of openBIS as a data management platform. OpenBIS is designed to facilitate robust data management for a wide variety of experiment types and research subjects. It allows tracking, annotating, and sharing of data throughout distributed research projects in different quantitative sciences.
Target Audience: User communities that want to use GPUs in Clouds.	"How to train your AI models in EOSC", (Dec. 2021) Agenda, slides and recording: https://indico.egi.eu/event/5742/ About: Learn how you can train and develop AI models in EOSC using the distributed and federated computing infrastructure of EGI and the services developed during the Deep Hybrid DataCloud project. The webinar, sponsored by the EGI-ACE project, covers: How to prototype, build and train AI applications exploiting resources from EU e-infrastructures. Use and share AI trained models developed by other researchers (from and outside your communities) with the DEEP Marketplace.
Target Audience: Scientific communities, developers, integrators and end users.	"How to orchestrate services in the EOSC Compute Platform with the INDIGO PaaS" (Oct. 2021) Agenda, slides and recording: https://indico.egi.eu/event/5720/ About: The INDIGO PaaS implements an abstraction and federation layer on top of heterogeneous distributed computing environments: it allow to orchestrate and coordinate the provisioning of virtualized compute and storage resources on Cloud Management Frameworks, both private and public (like OpenStack, OpenNebula, AWS, etc.), and the deployment of dockerized long-running services and batch jobs on Container Orchestration Platforms like Apache Mesos and Kubernetes. In this webinar, we will describe the architecture of the INDIGO Platform and its main features. The demo will show how users can easily interact with this orchestration system using both the command line interface and the web dashboard.
Target Audience: Scientific communities, developers, integrators and end users.	"Analyze your data using DODAS generated cluster" (September, 2021) Agenda, slides and recording: https://indico.egi.eu/event/5695/ About: DODAS enables the execution of user analysis code both in batch mode and interactively via the Jupyter interface. DODAS is highly customizable and offers several building blocks that can be combined together in order to create the best service composition for a given use case. The currently available blocks allow to combine Jupyter and HTCondor as well as Jupyter and Spark or simply a jupyter interface. In addition, they allow the management of data via caches to optimise the processing of remote data. This can be done either via XCache or MinIO S3 object storage capabilities. DODAS is based on docker containers and the related orchestration relies on Kubernetes that enables the possibility to compose the building blocks via a web-based user interface thanks to Kubeapps. In this webinar we will explain the DODAS fundamentals and we will provide a user oriented demo.
Target Audience: Scientific communities, developers, integrators and end users.	"Running containers in your user space with udocker" (June, 2021) Agenda, slides and recording: https://indico.egi.eu/event/5541/ About: udocker enables the execution of docker containers in user space without requiring root privileges for installation or use. Udocker implements the pull, load, import and execution of containers by non-privileged users in Linux systems where docker is not available. It can be used in Linux batch systems and interactive clusters that are managed by other entities, such as grid infrastructures or externally managed batch or interactive systems. udocker does not require any type of privileges nor the deployment of services by system administrators. It can be downloaded and executed entirely by the end user. udocker offers several execution modes exploiting system call interception, library call interception and namespaces. udocker integrates several tools to provide a subset of the docker capabilities aimed at container execution. In this webinar we will explain the udocker fundamental, how to use udocker to execute Linux containers and how to best exploit the several execution engines.
Target Audience: Site administrators already familiar with an earlier version of ARC CE or planning a migration to ARC from the CREAM platform.	"Rolling out ARC6 CE" (July 2020) Agenda, slides and recording: https://indico.egi.eu/event/5157/ About: The Advanced Resource Connector (ARC) middleware integrates computing resources (usually, computing clusters managed by a batch system), making them available via a secure common layer. Conceptually, ARC provides an edge service to batch systems. Through this service, called ARC Compute Element (ARC-CE), scientific communities can launch and manage computational tasks in a uniform manner. A bit more than a year ago a non backward compatible major version, the ARC6.0 was released bringing new functionality, enhanced manageability and increased stability to the community. With the availability of the ARC6 release the support for the previous ARC5 deployments are to be discontinued with the end of June 2020. This interactive webinar will introduce the major new features of the ARC6 release and cover the deployment steps of an ARC6 CE both as a new installation and as an upgrade from a previous ARC5-based deployment. Special attention will be given to the accounting system related changes and the new one-shop-stop sysadmin toolbox built around arcctl.
Target Audience: Scientific communities, developers, integrators and end users.	"High performance software - Easy gains with simple CUDA" (April 2023) About: This tutorial provides an introduction to CUDA in high performance software, covering roughly these topics: Best practices for high performance software engineering, such as avoiding premature optimization, ensuring cache alignment, etc. A broad introduction to GPUs, including their hardware and which categories of problems they are/aren't best suited for. Installing and working with GPU frameworks An overview of profiler tools and how to use them A live coding session to implement and diagnose a basic CUDA program, with the level of detail dependent on available time Q&A and stories from the trenches Please note that the training will not cover multi-GPU setups or provide a detailed dive into GPU hardware and CUDA specifics. Participants should have basic knowledge of Python and matrix computation libraries like NumPy. Slides and code https://github.com/c-scale-community/cscale-gpet-workshop

Users: Scientific level

Mon, 01 Jan 0001 00:00:00 +0000

Target Audience: Data scientists Domain scientists and research (in Environmental, Climate research) e-Infrastructure platform/ technology & service providers	"Harnessing Advanced Research Infrastructures for Earth and Environmental Data Management and Analysis: A MATLAB Perspective" (May, 2024) Agenda, slides and recording: https://www.egi.eu/event/harnessing-advanced-research-infrastructures-for-earth-and-environmental-data-management-and-analysis-a-matlab-perspective/ About: The increasing volume and complexity of Earth and environmental data requires an efficient interdisciplinary collaboration and management. The use of advanced research infrastructures enables data integration, interoperability, and seamless data exchange, leveraging analysis and visualization tools and cloud facilities. In the first part of the presentation there is an introduction to MATLAB and a review of existing and new features relevant to the EGI users. In the second part, a case study is presented on the ENES Data Space, a cloud-based data science environment for climate data analysis, part of the European Open Science Cloud (EOSC) Compute Platform. Users can access and work with climate datasets, including past recordings and future projections from the CMIP project. The case study demonstrates geodata import, analysis, and visualization using MATLAB Online and Jupyter notebooks, emphasizing ease of use, data manipulation, and sharing capabilities within the ENES workspace without requiring local software installations or data downloads.
Target Audience: oceanographers climate researchers ocean modellers ocean data managers	"The EGI-ACE webODV – Online extraction, analysis and visualisation of SeaDataNet and Argo data" (Nov. 2022) Agenda, slides and recording: https://indico.egi.eu/event/5980/ About: ODV (Ocean Data View) is a widely used software package for the analysis, exploration and visualization of oceanographic and other environmental data with almost 100,000 registrations since the 1990’s and more than 10,000 active users. ODV has been further developed as part of the EU supported activities of SeaDataNet, the pan-European infrastructure for marine and ocean data management. To make working with web-based large community datasets easier, an online version of the ODV software called webODV has been developed, which provides typical ODV functionality in the form of web services. webODV has been deployed in the cloud in the framework of the EU EGI-ACE project, which seeks to promote scientific analytical cloud services in support of the European Open Science Cloud (EOSC) initiative. In this webinar, we will be able to learn about the webODV Explorer and Extractor components in combination with the large validated Temperature & Salinity data collections as provided by SeaDataNet and EuroArgo – Argo. Instructions on how to register as a user to perform analytics, extractions, and visualisations will also be presented.
Target Audience: Scientific communities and end users.	"Can deep learning models help accelerate electrostatics-driven protein pKa predictions?" (Nov. 2022) Agenda, slides and recording: https://indico.egi.eu/event/5981/ About: pH is a crucial physicochemical property that affects proteins molecular structure, folding, stability, and function. Many computational methods have been developed to calculate pKa values. In the highly accurate, but slow, Poisson–Boltzmann (PB)-based methods, proteins are represented by point charges in a low dielectric medium surrounded by an implicit solvent (high dielectric). Empirical methods rely on statistically fitting parameters over large datasets of experimental pKa values. These are much faster than the physics-based methods, although at the cost of less microscopic insights and unknown predictive power on mutations and proteins dissimilar to those in the training set. In this webinar we will present a novel strategy to combine the best features of PB models – accuracy and interpretability – with the speed of classical empirical methods. The deep learning pKa predictors obtained were trained on a database of 3M theoretical pKa values estimated from 50k structures using a PB method. With this approach, we can retrieve the physics-based predictions with an average error below 0.4 pK units while being up to 1000x faster.
Target Audience: Scientific communities and end users.	"The Virtual Imaging Platform: Scientific Applications as a Service and Beyond" (March, 2022) Agenda, slides and recording: https://indico.egi.eu/event/5824/ About: The Virtual Imaging Platform (VIP) is a web portal for medical simulation and image data analysis. It leverages resources available in the EGI biomed Virtual Organisation to offer an open service to academic researchers worldwide. In the last few years, VIP has addressed interoperability and reproducibility concerns, in the larger scope of a FAIR (Findable, Accessible, Interoperable, Reusable) approach to scientific data analysis. The presentation will give an overview of VIP and its main interests for the audience: Use existing applications as a service on VIP. Import your own applications to make them available to the community benefit from EGI biomed resources in a transparent way and Foster open and reproducible science
Target Audience: Scientific communities and end users.	"ENES Data Space Service" (March, 2022) Agenda, slides and recording: https://indico.egi.eu/event/5743/ About: The ENES Data Space delivers an open, scalable and cloud-enabled data science environment for climate data analysis on top of the EOSC Compute Platform. It provides access to a set of specific Coupled Model Intercomparison Project (CMIP) variable-centric collections to support meteorological and industrial researchers in realistic climate model analysis experiments. Data is downloaded and kept in sync with the Earth System Grid Federation (ESGF) federated data archive. This webinar will provide a general overview of the ENES Data Space service and its main features. Moreover, the webinar will include a tutorial on how to join the ENES Data Space service and a short demo for the participants to get started with the data analysis and visualisation features.
Target Audience: Scientific communities, end users and service providers who are interested in discovering how we have been making efficient use of EGI High Throughput Compute (HTC) resources over more than 10 years to provide services to worldwide researchers in structural biology and life sciences.	"WeNMR - Structural biology in the cloud - 10 years of experience of using EGI services" (April 2020) Agenda, slides and recording: https://indico.egi.eu/event/5084/ About: Structural biology deals with the characterization of the structural (atomic coordinates) and dynamic (fluctuation of atomic coordinates over time) properties of biological macromolecules and adducts thereof. Gaining insight into 3D structures of biomolecules is highly relevant with numerous applications in health and food sciences, with as current example unravelling the structural details of Sars-Cov2 in the COVID-19 pandemic. Since 2010, the WeNMR project has implemented numerous web-based services to facilitate the use of advanced computational tools by researchers in the field, using the grid computational infrastructure provided by EGI. These services have been further developed in subsequent initiatives under the H2020 EGI-ENGAGE West-Life project and the BioExcel Center of Excellence for Biomolecular Computational Research. The WeNMR services are currently operating under the European Open Science Cloud with the H2020 EOSC-Hub project, with the HADDOCK portal sending >10 millions jobs and using ~2700 CPU years per year. In my talk, I will summarise 10 years of successful use of e-infrastructure solutions to serve a large worldwide community of users (>16’000 to date), providing them with user-friendly, web-based solutions that allow to run complex workflows in structural biology. I will illustrate this with details of the HADDOCK service and how we could increase our capacity to serve COVID-related projects. Suggested tutorials Submitting HTC Jobs: Submitting High Throughput Compute jobs
Target Audience: Scientific communities working in the domain of Earth Observation.	"C-SCALE Notebooks for Earth Observation" (June 2023) About: The C-SCALE project has been federating compute and data resource providers around centralized EGI services, aiming at providing users with seamless access to processing capacities as well as source data for their analyses. Alongside the traditional IaaS and PaaS services, Jupyter Notebooks have been identified as an environment suitable not only for interactive analysis within C-SCALE, but also for documenting the different steps one needs to take in discovering and accessing geospatial data across Europe. The demonstration of C-SCALE's example notebooks and procedures will focus on those essential features: simple steps to get started using the federated resources for interactive resources of Earth Observation data. Slides and code https://github.com/c-scale-community/c-scale-notebooks
Target Audience: Scientific communities working in the domain of Earth Observation.	"C-SCALE Earth Observation - Metadata Query Service" (March 2023) About: The C-SCALE Earth Observation Metadata Query Service (EO-MQS) makes Copernicus data distributed across providers within the C-SCALE Data federation discoverable and searchable. Slides and code https://github.com/c-scale-community/c-scale-tutorial-eo-mqs
Target Audience: Scientific communities working in the domain of Earth Observation.	"Introduction to openEO Platform" (December 2022) About: openEO platform provides intuitive programming libraries to process a wide variety of earth observation datasets. This large-scale data access and processing is performed on multiple infrastructures, which all support the openEO API. This allows use cases from exploratory research to large-scale production of EO-derived maps and information. Slides and code https://github.com/c-scale-community/c-scale-tutorial-openeo
Target Audience: Scientific communities and end users.	"I-NERGY, European AI-on demand platform" (July 2023) About: I-NERGY aims to support and develop novel AI-based energy services as part of the enrichment of European AI-on demand platform. This webinar will present the objectives and scope of the project, its requirements in terms of resources and the successful utilisation of EGI infrastructure. The webinar will conclude with a demo of I-NERGY services.
Target Audience: Scientific communities and end users.	"Access to the EISCAT tools with help of EGI Check-in" (June 2023) About: The present era of rapid technological advances creates a challenge for data providers and scientists to create and maintain FAIR data and services not just for future operations but also for historical data gathered and analysed with technologies that are slowly phasing out of their usage. GUISDAP is an open-source software package, written in MATLAB, C and Fortran and provided and maintained by EISCAT, for analysis and visualisation of its incoherent scatter radar data as well as for some other radars in the world. One way how to preserve GUISDAP operability and accessibility by the user community is to make it accessible through a Jupyter notebook docker deployment through EISCAT resources and in the frame of an EOSC project. This will help to ensure the FAIRness of EISCAT data by providing tools for reanalysis and visualisation that will be accessible by any potential EISCAT user with the help of EGI check-in technology.

Users: Ad-hoc tutorials

Mon, 01 Jan 0001 00:00:00 +0000

Overview

The following guides and tutorials show you how to perform common tasks in the EGI infrastructure. Each of the following sections addresses a specific use case: it describes how to set up the EGI services needed for that use case, how to connect or operate them together, and will walk you step-by-step towards achieving a task or setup that can be immediately used by researchers and scientists.