Documentation – High Throughput Compute

Users: Grid Storage

Mon, 01 Jan 0001 00:00:00 +0000

What is it?

Grid storage enables storage of files in a fault-tolerant and scalable environment, and sharing it with distributed teams. Your data can be accessed through multiple protocols, and can be replicated across different providers to increase fault-tolerance. Grid storage gives you complete control over what data you share, and with whom you share the data.

The main features of grid storage:

Access highly-scalable storage from anywhere
Control the data you share
Organise your data using a flexible, hierarchical structure

Grid storage file access is based on the gridFTP and WebDav/HTTP protocols, together with XRootD and legacy SRM (under deprecation at some of the endpoints).

Several grid storage implementations are available in the EGI Infrastructure, the most common being:

dCache
DPM (N.B. under decommissioning)
StoRM
EOS

Endpoint discovery

The grid storage endpoints that are available to a user’s Virtual Organizations are discoverable via the EGI Information System (BDII).

The lcg-infosites command can be used to obtain VO-specific information on existing grid storages, using the following syntax:

$ lcg-infosites --vo voname -[v] -f [site name] [option(s)] [-h| --help] [--is BDII]

For example, to list the Storage Elements (SEs) available to the biomed VO, we could issue the following command:

$ lcg-infosites --vo biomed se

 Avail Space(kB) Used Space(kB) Type SE
------------------------------------------
 280375465082 n.a SRM ccsrm.ihep.ac.cn
 10995116266 11 SRM cirigridse01.univ-bpclermont.fr

Access from the command-line

Access to grid storage via a command-line interface (CLI) requires users to obtain a valid X.509 user VOMS proxy. Please refer to the Check-in documentation for more information.

Note

Integration via OpenID Connect to the EGI Check-in service is under piloting at some of the endpoints of the EGI Cloud infrastructure , but it has not yet reached the production stage.

The CLI widely used to access grid-storage is gfal2, which is available for installation both on RHEL and Debian compatible systems.

In particular, gfal2 provides an abstraction layer on top of several storage protocols (XRootD, WebDAV, SRM, gsiftp, etc), offering a convenient API that can be used over different protocols.

The gfal2 CLI can be installed as follows (for RHEL compatible systems):

$ yum install gfal2-util gfal2-all

where gfal2-all will install all the plug-ins (to deal with all the available protocols).

Below you can find examples of the usual commands needed to access storage via gfal2. For a complete list of available commands, and the guide on how to use them, please refer to the gfal2-util documentation.

Note

In the examples below, the used gsiftp protocol can be replaced by any other supported protocol.

List files on a given endpoint

$ gfal-ls gsiftp://dcache-door-doma01.desy.de/dteam
1G.header-1
domatest
gb
SSE-demo
test
tpctest

Create a folder

$ gfal-mkdir gsiftp://dcache-door-doma01.desy.de/dteam/test

Copy a local file

$ gfal-copy test.json gsiftp://dcache-door-doma01.desy.de/dteam/test
Copying file:///root/Documents/test.json [DONE] after 0s

Copy files between storages

$ gfal-copy gsiftp://prometheus.desy.de/VOs/dteam/public-file gsiftp://dcache-door-doma01.desy.de/dteam/test
Copying gsiftp://prometheus.desy.de/VOs/dteam/public-file [DONE] after 3s

Download a file to a local folder

$ gfal-copy gsiftp://prometheus.desy.de/VOs/dteam/public-file /tmp
Copying gsiftp://prometheus.desy.de/VOs/dteam/public-file [DONE] after 0s

Delete a file

$ gfal-rm gsiftp://dcache-door-doma01.desy.de/dteam/test/public-file
gsiftp://dcache-door-doma01.desy.de/dteam/test/public-file DELETED

Access via EGI Data Transfer

The EGI Data Transfer service provides mechanisms to optimize the transfer of files between EGI Online Storage endpoints. Both a graphical user interface (GUI) and command-line interfaces (CLI) are available to perform bulk movement of data. Please check out the related documentation for more information.

Integration with Data Management frameworks

Grid storage access, most of the time, is hidden from users by the integration with the Data Management Frameworks (DMFs) used by Collaborations and Experiments.

For example, EGI Workload Manager provides a way to efficiently access grid storage endpoints in order to read/store files, and to catalogue the existing file and related metadata.

When running computation via the EGI Workload Manager, users do not actually access the storage directly. However, users can retrieve the output of the computation once it has been stored on the grid.

Users: Querying the Information System

Mon, 01 Jan 0001 00:00:00 +0000

About the information system

Information about resources is documented according to the GLUE Schema.

Information in the Top BDII is provided for two GLUE schema versions:

GLUE 1.3: the legacy version of the specification, under the LDAP base Mds-Vo-Name=local,o=grid.
GLUE 2.0: the most recent version of the specification, under the LDAP base GLUE2GroupID=grid,o=glue.

Some resources may not yet be exposed via GLUE 2.0, and it may be required to use GLUE 1.3 for those ones.

Tip

You can use -o ldif-wrap=no to disable wrapping the results.

Those examples are relying on the Top BDII service maintained by EGI Foundation: ldap://lcg-bdii.egi.eu:2170.

# Dumping all the information from GLUE 2.0
$ ldapsearch -x -H ldap://lcg-bdii.egi.eu:2170 -b "GLUE2GroupID=grid,o=glue"

In order to retrieve information required to contact a given CE, you can look for GLUE2EndpointURL objects.
In order to retrieve information about the batch systems, you can look for GLUE2Manager objects.
In order to look for information about a specific CE, you can look into GLUE2Service.
In order to look for information about the resources available to VOs, you can look into GLUE2Share and GLUE2Policy.

The following queries can be used to retrieve information about all the Computing Elements of a given type. You will likely be able to use only a subset of them, only the ones supporting the Virtual Organisation you are a member of, and for which you have a valid VOMS proxy.

Computing Elements

Nowadays mainly two Computing Element (CE) “flavours” are used in production:

HTCondorCE, a Compute Entrypoint (CE) based on HTCondor.
ARC-CE, the ARC Compute Element (CE).

The CREAM CE is a legacy and no more supported middleware.

Querying for HTCondorCE compute resources

Most, if not all the HTCondorCE Computing Elements should be discoverable via GLUE 2.0.

# Querying GLUE2EndpointURL for all HTCondorCE compute endpoints
$ ldapsearch -x -LLL -H ldap://lcg-bdii.egi.eu:2170 \
 -b "GLUE2GroupID=grid,o=glue" \
 '(&(objectClass=GLUE2Endpoint)(GLUE2EndpointImplementationName=HTCondor))' \
 GLUE2EndpointInterfaceName \
 GLUE2EndpointImplementationVersion \
 GLUE2EndpointURL

# Querying GLUE2Manager for the batch systems' versions and number of CPUs
$ ldapsearch -x -LLL -H ldap://lcg-bdii.egi.eu:2170 \
 -b "GLUE2GroupID=grid,o=glue" \
 '(&(objectClass=GLUE2Manager)(GLUE2ManagerProductName=HTCondor))' \
 GLUE2ManagerProductName \
 GLUE2ManagerProductVersion \
 GLUE2ComputingManagerTotalLogicalCPUs \
 GLUE2ComputingManagerComputingServiceForeignKey

Once you have selected a given CE, you can look into getting more information about this one.

# Querying GLUE2Service for information about a specific CE
$ ldapsearch -x -LLL -H ldap://lcg-bdii.egi.eu:2170 \
 -b "GLUE2GroupID=grid,o=glue" \
 '(&(objectClass=GLUE2Service)(GLUE2ServiceID=*condorce1.ciemat.es*))'

# Querying GLUE2Share for a specific CE, filtering for details on the queues
$ ldapsearch -x -LLL -H ldap://lcg-bdii.egi.eu:2170 \
 -b "GLUE2GroupID=grid,o=glue" \
 '(&(objectClass=GLUE2Share)(GLUE2ShareID=*condorce1.ciemat.es*))' \
 GLUE2ComputingShareMappingQueue \
 GLUE2ShareIDGLUE2ComputingShareMaxWallTime \
 GLUE2ComputingShareMaxVirtualMemory \
 GLUE2ComputingShareMaxUserRunningJobs \
 GLUE2ComputingShareMaxRunningJobs \
 GLUE2ComputingShareMaxCPUTime \
 GLUE2ComputingShareWaitingJobs \
 GLUE2ComputingShareUsedSlots \
 GLUE2ComputingShareTotalJobsGLUE2ComputingShareRunningJobs

# Querying GLUE2Policy for a specific CE, filtering for supported VOs
$ ldapsearch -x -LLL -H ldap://lcg-bdii.egi.eu:2170 \
 -b "GLUE2GroupID=grid,o=glue" \
 '(&(objectClass=GLUE2Policy)(GLUE2PolicyID=*condorce1.ciemat.es*))' \
 GLUE2PolicyRule \
 GLUE2PolicyID \
 GLUE2MappingPolicyShareForeignKey

Querying for ARC-CE compute resources

Most, if not all the ARC-CE should be discoverable via GLUE 2.0.

# Querying for all ARC-CE compute resources
$ ldapsearch -x -LLL -H ldap://lcg-bdii.egi.eu:2170 \
 -b "GLUE2GroupID=grid,o=glue" \
 '(&(objectClass=GLUE2Endpoint)(GLUE2EndpointImplementationName=nordugrid-arc))' \
 GLUE2EndpointInterfaceName \
 GLUE2EndpointImplementationVersion \
 GLUE2EndpointURL

# Querying GLUE2Manager for the batch systems' versions and number of CPUs
$ ldapsearch -x -LLL -H ldap://lcg-bdii.egi.eu:2170 \
 -b "GLUE2GroupID=grid,o=glue" \
 '(&(objectClass=GLUE2Manager)(GLUE2ComputingManagerComputingServiceForeignKey=*urn:ogf:ComputingService*))' \
 GLUE2ManagerProductName \
 GLUE2ManagerProductVersion \
 GLUE2ComputingManagerTotalLogicalCPUs \
 GLUE2ComputingManagerComputingServiceForeignKey

Once you have selected a given CE, you can look into getting more information about this one.

# Querying GLUE2Service for information about a specific CE
$ ldapsearch -x -LLL -H ldap://lcg-bdii.egi.eu:2170 \
 -b "GLUE2GroupID=grid,o=glue" \
 '(&(objectClass=GLUE2Service)(GLUE2ServiceID=*alex4.nipne.ro*))'

# Querying GLUE2Share for a specific CE, filtering for details on the queues
$ ldapsearch -x -LLL -H ldap://lcg-bdii.egi.eu:2170 \
 -b "GLUE2GroupID=grid,o=glue" \
 '(&(objectClass=GLUE2Share)(GLUE2ShareID=*alex4.nipne.ro*))' \
 GLUE2ComputingShareMappingQueue \
 GLUE2ShareIDGLUE2ComputingShareMaxWallTime \
 GLUE2ComputingShareMaxVirtualMemory \
 GLUE2ComputingShareMaxUserRunningJobs \
 GLUE2ComputingShareMaxRunningJobs \
 GLUE2ComputingShareMaxCPUTime \
 GLUE2ComputingShareWaitingJobs \
 GLUE2ComputingShareUsedSlots \
 GLUE2ComputingShareTotalJobsGLUE2ComputingShareRunningJobs

# Querying GLUE2Policy for a specific CE, filtering for supported VOs
$ ldapsearch -x -LLL -H ldap://lcg-bdii.egi.eu:2170 \
 -b "GLUE2GroupID=grid,o=glue" \
 '(&(objectClass=GLUE2Policy)(GLUE2PolicyID=*alex4.nipne.ro*))' \
 GLUE2PolicyRule \
 GLUE2PolicyID \
 GLUE2MappingPolicyShareForeignKey

It’s also possible to look into GLUE 1.3.

Using GLUE 1.3

Deprecated

GLUE 1.3 is legacy

Querying for information about ARC-CE using GLUE 1.3.

# Querying for all ARC CE compute resources, using GLUE 1.3
$ ldapsearch -x -LLL -H ldap://lcg-bdii.egi.eu:2170 \
 -b "Mds-Vo-Name=local,o=grid" \
 '(&(objectClass=GlueCE)(GlueCEInfoJobManager=arc))' \
 GlueCEImplementationVersion GlueCEImplementationName \
 GlueCEStateStatus GlueCEUniqueID GlueServiceEndpoint GlueServiceType

# Querying information about the ARC CE via GLUE 1.3
$ ldapsearch -x -LLL -H ldap://lcg-bdii.egi.eu:2170 \
 -b "Mds-Vo-Name=local,o=grid" \
 '(&(objectClass=GlueCE)(GlueCEUniqueID=alex4.nipne.ro:2811/nordugrid-SLURM-dteam))'

# Limiting output for the ARC CE via GLUE 1.3
$ ldapsearch -x -LLL -H ldap://lcg-bdii.egi.eu:2170 \
 -b "Mds-Vo-Name=local,o=grid" \
 '(&(objectClass=GlueCE)(GlueCEUniqueID=alex4.nipne.ro:2811/nordugrid-SLURM-dteam))' \
 GlueCEInfoHostName GlueCEUniqueID \
 GlueCEInfoJobManager GlueCEImplementationName GlueCEImplementationVersion \
 GlueCEInfoLRMSType GlueCEInfoLRMSVersion \
 GlueCEAccessControlBaseRule \
 GlueCEInfoTotalCPUs \
 GlueCEStateStatus

Using legacy tools for GLUE 1.3

Using lcg-info it’s possible to easily do more targeted queries, like focusing on a specific VO.

lcg-info and lcg-infosites are only taking into account the GLUE 1.3 schema, and will lack information that is only published according to the GLUE 2 schema, like for most HTCondorCE Computing Elements.

# Identify compute resources available for dteam VO
$ lcg-info --list-ce --vo dteam --bdii ldap://lcg-bdii.egi.eu:2170
# Identify storage resources available for dteam VO
$ lcg-info --list-se --vo dteam --bdii ldap://lcg-bdii.egi.eu:2170

$ lcg-info --list-ce --vo dteam --bdii ldap://lcg-bdii.egi.eu:2170 \
 --attrs CEImpl --query 'CEImpl=*HTCondorCE'
(...)
- CE: ce01.knu.ac.kr:9619/ce01.knu.ac.kr-condor
 - CEImpl HTCondorCE

- CE: ce13.pic.es:9619/ce13.pic.es-condor
 - CEImpl HTCondorCE
(...)
- CE: condorce1.ciemat.es:9619/condorce1.ciemat.es-condor
 - CEImpl HTCondorCE

- CE: condorce2.ciemat.es:9619/condorce2.ciemat.es-condor
 - CEImpl HTCondorCE
(...)

We can see that the dteam VO should be able to access Computing Elements from the various types:

$ lcg-info --list-ce --vo dteam --bdii ldap://lcg-bdii.egi.eu:2170 \
 --attrs CEImpl --query 'CEImpl=*' | grep CEImpl | sort | uniq -c
 123 - CEImpl ARC-CE
 22 - CEImpl CREAM
 7 - CEImpl HTCondorCE

Identifying all the resources accessible by a given VO

In GLUE 2.0, the access granted to a given VO to a compute or storage resource, is published using the GLUE2Share and GLUE2Policy objects. There are also GLUE2ComputingShare and GLUE2StorageShare to specifically document sharing of compute or storage resources.

# Querying GLUE2Share for all the resources available to dteam VO
$ ldapsearch -x -LLL -H ldap://lcg-bdii.egi.eu:2170 \
 -b "GLUE2GroupID=grid,o=glue" \
 '(&(objectClass=GLUE2Share)(GLUE2ShareID=*dteam*))'

# Querying GLUE2ComputingShare for all the computing resources available to dteam VO
$ ldapsearch -x -LLL -H ldap://lcg-bdii.egi.eu:2170 \
 -b "GLUE2GroupID=grid,o=glue" \
 '(&(objectClass=GLUE2ComputingShare)(GLUE2ShareID=*dteam*))'

# Querying GLUE2StorageShare for all the storage resources available to dteam VO
$ ldapsearch -x -LLL -H ldap://lcg-bdii.egi.eu:2170 \
 -b "GLUE2GroupID=grid,o=glue" \
 '(&(objectClass=GLUE2StorageShare)(GLUE2ShareID=*dteam*))'

It is possible to filter for the different types of Computing Element, and select only specific attributes.

Looking for a HTCondorCE for dteam

# Information about the HTCondorCE supporting dteam VO
$ ldapsearch -x -LLL -H ldap://lcg-bdii.egi.eu:2170 \
 -b "GLUE2GroupID=grid,o=glue" \
 '(&(objectClass=GLUE2ComputingShare)(GLUE2ShareID=*dteam*)(GLUE2ComputingShareComputingEndpointForeignKey=*HTCondorCE*))' \
 GLUE2ShareEndpointForeignKey \
 GLUE2ShareID \
 GLUE2ComputingShareTotalJobs \
 GLUE2ComputingShareRunningJobs \
 GLUE2ComputingShareWaitingJobs

Assuming it was decided, based on the site location, available resources, prior experience, or any other reason, to go for condorce1.ciemat.es, the information about the CE can be requested using the following request, filtering on the GLUE2ShareID from the previous query: grid_dteam_condorce1.ciemat.es_ComputingElement.

# condor_submit needs CE (condorce1.ciemat.es) and pool (condorce1.ciemat.es:9619)
$ ldapsearch -x -LLL -H ldap://lcg-bdii.egi.eu:2170 \
 -b "GLUE2GroupID=grid,o=glue" \
 '(&(objectClass=GLUE2ComputingShare)(GLUE2ShareID=*grid_dteam_condorce1.ciemat.es_ComputingElement*))' \
 GLUE2ShareID \
 GLUE2ShareDescription \
 GLUE2ComputingShareExecutionEnvironmentForeignKey \
 GLUE2EntityOtherInfo

Looking for an ARC-CE for dteam

# Information about the ARC-CE supporting dteam VO
$ ldapsearch -x -LLL -H ldap://lcg-bdii.egi.eu:2170 \
 -b "GLUE2GroupID=grid,o=glue" \
 '(&(objectClass=GLUE2ComputingShare)(GLUE2ShareID=*dteam*)(GLUE2ComputingShareComputingEndpointForeignKey=*urn:ogf*))' \
 GLUE2ComputingShareComputingEndpointForeignKey \
 GLUE2ShareEndpointForeignKey \
 GLUE2ComputingShareTotalJobs \
 GLUE2ComputingShareRunningJobs \
 GLUE2ComputingShareWaitingJobs

# arcsub needs CE name (alex4.nipne.ro)
$ ldapsearch -x -LLL -H ldap://lcg-bdii.egi.eu:2170 \
 -b "GLUE2GroupID=grid,o=glue" \
 '(&(objectClass=GLUE2ComputingShare)(GLUE2ShareID=*urn:ogf:ComputingShare:alex4.nipne.ro:dteam_dteam*))' \
 GLUE2ShareID \
 GLUE2ShareDescription \
 GLUE2ComputingShareComputingServiceForeignKey \
 GLUE2ComputingShareExecutionEnvironmentForeignKey