Documentation – Operations manuals and How-Tos

Providers: Operations Start Guide

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

The Operations Start Guide will help you start with EGI Operations duties. It covers the responsibilities of the various parties involved in the running of the EGI infrastructure and guide how to join operations. As a newcomer, you need to understand the structure of the infrastructure and roles of operators at different levels. Reading the whole document will give you a complete overall picture of daily operations within EGI.

Resource Centres and Resource Infrastructures

Resources are geographically distributed and are contributed by Resource Centres. A Resource Centre is the smallest resource administration domain within EGI. It can be either localized or geographically distributed. A Resource Centre is also known as a site. It provides a minimum set of local or remote IT Services, such as High Throughput Compute (HTC), Cloud Compute, Storage and Data Transfer, compliant to well-defined IT Capabilities necessary to make resources accessible to authorised users. Users can access the services using common interfaces.

The EGI Federation is a Resource Infrastructure federating Resource Centres to constitute a homogeneous operational domain, and the Resource Infrastructure Provider is the legal organisation that is responsible of establishing, managing and of operating directly or indirectly the operational services to an agreed level of quality needed by the Resource Centres and the user community. It holds the responsibility of integrating them in EGI to enable uniform resource access and sharing for the benefit of their consuming end users. Examples of a Resource infrastructure Provider are the European Intergovernmental Research Organisations (EIRO) and the NGIs.

In Europe, Resource Centres are required to be affiliated to the respective NGIs, which (a) have a mandate to represent their national users community in all matters falling within the scope of the EGI Infrastructure, and (b) are the only organization having the mandate described in (a) for its country and thus provide a single contact point at the national level.

Operations Centres

A Resource Infrastructure Provider is responsible for delivering, to different groups of customers, a number of services, either technical or humans : services supporting the activities of the given Resource Infrastructure Provider and facilitating and making secure the access to the provided resources and services.

In order to contribute resources to EGI, a Resource Infrastructure Provider must be associated to an Operations Centre which will deliver the services on behalf of them.

A single Operations Centre can be federated and operate services for multiple Resource Infrastructure Providers. Federation of operations can be a cost-effective approach for Resource Infrastructure Providers wishing to share their effort and services, or in order to operate the infrastructures in an early stage of maturity.

The Operations Centre provides services in collaboration with the respective Resource Centres via the Resource Centre Operations Manager, and globally with EGI Foundation and other Operations Centres.

Locally, Operations Centres are responsible for supporting the Resource Centres, for monitoring their Quality of Service (QoS), for collecting requirements and representing them in the EGI operations boards.
Globally they are in charge of contributing to the evolution of the EGI Infrastructure.

Roles

The following sections covers the roles that are commonly involved in the operations of the EGI Infrastructure. The correspondent roles defined in Configuration Database (GOCDB) give specific rights in the Configuration Database itself and in other EGI services. There are roles whose scope is limited to the operation of a Resource Centre. A number of other roles act on a higher level, involving the operations activities related either to the NGI (Regional level) or to the EGI infrastructure as a whole (global level). Other terms and definitions can be found in the EGI Glossary.

Resource Centre level

Resource Centre Administrator

An individual responsible for installing, operating, maintaining and supporting one or more Resources or IT Services in a Resource Centre. In the scope of Operations, Resource Centres (RC) administrators primarily receive and react on incidents at their Resource Centre and on service requests notified through tickets created on the EGI Helpdesk service. They should respond to the tickets in a suitable time frame as defined in the Resource Centre Operational Level Agreement (OLA) and be aware of the alarms at their site, eg. through the Operations Dashboard. Sites MUST only operate supported middleware versions. This implies upgrading it regularly. Emergency releases are treated in a special way. Resource Centre Administrators MUST react to security issues that are at a global level, but affect their site. See SEC03 EGI-CSIRT Critical Vulnerability Handling.

Resource Centre Operations Manager

An individual who leads the Resource Centre operations, who is the official technical contact person in the connected organisation, and who is locally supported by a team of Resource Centre Administrators. The Resource Centre Operations Manager is responsible for the site at the political and legal levels and for signing the Operational Level Agreement) between the Resource Centre and its hosting NGI. The Resource Centre Operations Manager is also responsible for enforcing the EGI policies and procedures by the Resource Centre and for assigning and approving the other site roles in the Configuration Database. Further, they should ensure that administrators are subscribed to the relevant mailing lists.

Resource Centre Security Officer

The person responsible for keeping the site compliant with the EGI security policies. They are the primary contact for the NGI Security officer and EGI Computer Security Incident Response Team (CSIRT). The Site Security Officer deals with security incidents and shall respond to enquiries in a timely fashion as defined in the collection of security procedures and policies.

Regional level

Regional Operator on Duty (ROD)

A team responsible for solving problems and incidents in the infrastructure according to agreed procedures. ROD teams monitor the Resource Centres in their region, react to incidents identified by the monitoring tools, and oversee incidents and related problems through to their resolution. They ensure that incidents are properly recorded and that the solutions progress according to specified time lines. They also provide support to Resource Centres and Virtual Organisations (VOs) and provide support to oversight bodies in cases of unresponsive Resource Centres. They ensure that all the necessary information is available to all parties. The team is provided by each NGI and requires procedural knowledge on the process (rather than technical skills) for their work. New ROD team members are required to read the ROD Welcome page and be familiar with ROD wiki page.

NGI Security officer

The NGI Security Officer is member of the EGI-CSIRT.

The NGI Security Officer coordinates the security activities within its NGI and serves as the primary contact for all security related requests, in particular from EGI-CSIRT’s IRTF concerning issues with the sites within the NGI.

Further, the NGI Security Officer is responsible for overseeing the security related aspects of the operations of the NGI and coordinates the security activities within its NGI.

NGIs and Sites MUST respond in a timely manner to the security requests and alerts coming from the EGI-CSIRT’s IRTF.

The NGI Security Officer name and contact address needs to be registered in the Configuration Database (GOCDB) and the information maintained by the NGI.

NGI Operations manager

The NGI Operations manager is the contact point for all operational matters and represents the NGI within the Operations Management Board (OMB).

They are mainly responsible for:

keeping up to date the NGI entry in the Configuration Database and managing the status of all sites under their NGI, and ensuring their information is also kept current
addressing problems with Site availability or reliability. The reports are issued on a monthly basis and the NGI operations managers have 10 days to respond to identified problems
attending regular Operations Management Board meetings

All NGI operations management responsibilities are listed in the Resource Infrastructure Provider OLA document.

Global level

Chief Operations Officer

The Chief Operations Officer leads EGI Operations, and is responsible for coordinating the operations of the infrastructure across the project.

EGI-CSIRT

EGI-CSIRT is the official security coordination team and contact point at project level.

EGI-CSIRT’s IRTF

The Incident Response Team for the Federation (IRTF) is a subgroup of EGI-CSIRT which acts as the primary contact for all security related requests concerning the Federation and the projects the EGI Foundation is involved in.

EGI-CSIRTs IRTF provides the Security Officer on Duty role on a weekly rota basis. Details are in the EGI-CSIRT Terms of References.

EGI Security Officer

The EGI Security Officer, having the EGI CSIRT Officer role in the Configuration Database, is leading and coordinating EGI-CSIRT.

The role of the EGI Security Officer is provided by a member of EGI-CSIRT’s IRTF on a weekly rota basis.

EGI Foundation SDIS team

The EGI Foundation Service Delivery and Information Security (SDIS) team, formerly known as the EGI Operations team, is responsible of coordinating and supporting the operational activities of all the EGI Infrastructure.

VO

A Virtual Organisation (VO) is a group of users and, optionally, resources, often not bound to a single institution or national borders, who, by reason of their common membership and in sharing a common goal, are given authority to use a set of resources. Each VO member signs the VO Acceptable Usage Policy (AUP) (during registration) which is the policy describing the goals of the VO thereby defining the expected and acceptable use of the resources by the users of the VO. User documentation can be found in the users section.

VO manager

An individual responsible for the management of the membership registry of the VO ensuring its accuracy and integrity.

Joining operations

In order to join any of the organisational groups in your NGI, you will need to go through the following steps in order:

Authentication

In general the authentication in EGI infrastructure works either with X509 personal certificates or through federated identities using Check-in. For some services (HTC and Storage) the access is granted only by using a X509 personal certificate due to legacy reasons: the process of moving the authentication and authorisation mechanism to federated identities has started but it will takes time before having everything compliant with federated identities.

Obtaining a X509 personal certificate

If you do not already have a X509 personal certificate the EUGridPMA world map provides a map of all certification authorities according to the country (or NGI). Select your country on the map to find out who is your local CA. Follow the procedure of your local CA to request a certificate. When you have received your certificate, install it into your web browser.

If case of setting up a new Resource Centre please request Host certificates.

EUGridPMA provides a web page allowing to test your certificate. Please use this resource and contact your CA if your certificate does not work.

Create a federated identity: registration in EGI Check-in

As soon as you try to access an EGI service with your federated identity, you will be requested to register an account in EGI Check-in if not existing yet. The Check-in sign up guide explains how to sign up for an EGI account. If you already own an account, you will be simply asked to login through EGI Check-in.

Joining dteam VO

It is recommended to join the dteam VO at the dteam Registration page. You should request group membership for /dteam and /dteam/YOUR_NGI. The dteam group manager will then be notified and review your application. The membership to dteam VO is possible only by using a X09 personal certificate and it is useful to test the RCs and to debug related issues.

Requesting GOCDB access

Read Input System User Documentation first.
Go to the Configuration Database and follow the instruction.

All members should notify their NGI operations manager about their role requests, to be sure they are considered on time.

Registering into GGUS

GGUS can be accessed with your federated identity. To register as a supporter in GGUS please submit a GGUS ticket to TPM to request the “Supporter” role in GGUS.

If your NGI also have a local helpdesk interfaced with GGUS, please ensure that you are properly registered also there: your NGI managers will take care of that.

Subscribing to mailing lists

NGIs and Sites have local mailing lists for ROD team members and Site Administrators respectively. Please ensure that you subscribe to them. Depending on your role ask your NGI operations manager or Site operations manager to have you included on the necessary mailing lists if there is no automatic subscription process.

NGI operations manager should contact operations@egi.eu and state that wish to be subscribed to noc-managers mailing list noc-managers@mailman.egi.eu.

Documentation

Procedures and policies are accessible on the EGI Policies and Procedures space. Additional documentation relevant to EGI operations is available at EGI Documentation.

Tools

A list of services relevant to EGI operations can be found at the section about internal services.

Providers: Resource Centre Integration Check List

Mon, 01 Jan 0001 00:00:00 +0000

This page provides an overview of what are the required steps to get a new Resource Centre integrated into the EGI Federation.

Be aware of the security policies.
Acceptance of the RC OLA (agreed with the NGI the RC belongs to).
Registration on the Configuration Database:
- an entry for the resource centre was created
- Required mailing lists are registered
- people operating the services are registered
- service endpoints are registered associated to the proper service types (e.g. for cloud providers), and the flags monitored and production are set to yes
Support through the EGI Helpdesk service:
- The RC name is listed in the GGUS fields “Affected site” and “Notify Site”
- The site administrators can modify and reply to the tickets assigned to their RC
  - The Supporter role can be requested either directly on GGUS by using the own X509 personal certificate or by enrolling to the ggus-supporters group in Check-in.
Security:
- HTC: pakiti is installed and the outcome of the EGI CSIRT assessment is positive;
- Cloud: the EGI security Survey was sent to the EGI CSIRT and the outcome was positive.
AAI:
- HTC/Storage Compliant with X509 certificates and VOMS attributes
- OpenStack clouds integration with Check-in
- General integration with Check-in for SPs
Monitoring: the registered endpoints are automatically detected by ARGO and monitored according to their type registered in the Configuration Database.
Accounting: The accounting records are properly sent to the accounting repository and displayed by the Accounting Portal.
- Accounting probes and APEL SSM are properly installed and configured:
  - HTC: APEL client
  - Cloud: cASO.
Information discovery:
- HTC/Storage: the information about compute and storage endpoints are published by the site-bdii into the Top-BDIIs.
- Cloud: site is added to the fedcloud-catchall-operations repository
Middleware: latest version of technology products are installed using the UMD and CMD releases.
Software distribution:
- HTC (Optional): CVMFS
- Cloud: VM image synchronisation is configured with a HEPIX VM image list compliant software (e.g. cloudkeeper)

Providers: MAN01 How to publish site information

Mon, 01 Jan 0001 00:00:00 +0000

Document control

Property	Value
Title	How to publish Site Information
Policy Group	Operations Management Board (OMB)
Document status	Approved
Procedure Statement	Publishing site information in the Information Discovery System
Owner	SDIS team

EGI profile for the use of the GLUE 2.0 Information Schema specifies how the GLUE 2.0 information schema should be used in EGI. It gives detailed guidance on what should be published, how the information should be interpreted, what kinds of uses are likely, and how the information may be validated to ensure accuracy.

Configuring a site BDII

The site BDII needs to be configured to read from every node in the site which publishes information (meaning that it runs a so-called resource BDII). In YAIM this is defined with the BDII_REGIONS variable, which contains a list of node names which in turn refer to variables called BDII_<NODE>_URL which specify the LDAP URL of each resource BDII.

Some services may have DNS aliases for multiple hosts, but the BDII_REGIONS must contain the real hostnames for each underlying node - the information in the resource BDII is different for each node, so reading it via an alias would produce inconsistent results. However, it will usually be desirable for the published endpoint URLs to contain the alias rather than the real hostname; that can often be defined with a YAIM variable for the service. For the site BDII itself this variable is SITE_BDII_HOST. (If multiple site or top BDIIs are configured identically their content will also be identical, so reading via an alias does not produce any inconsistencies.)

Most services now publish themselves, so sites should check that all relevant services are included. In particular, VOMS servers have only published themselves comparatively recently so may be missing from the configuration. If the glite-CLUSTER node type is used this must also be included. Publication has been enabled for Argus in EMI 2, so this may also need to be added. Common services which do not currently publish are APEL and Squid. See the table below for more detailed information.

It is important to realise that the site BDII itself has a resource BDII, and this must be explicitly included in the configuration, e.g. with something like

BDII_REGIONS="CE SE BDII"
(...)
BDII_BDII_URL="ldap://$SITE_BDII_HOST:2170/mds-vo-name=resource,o=grid"

In the past it was common for the site BDII to be colocated with the CE so it did not need to be listed explicitly, but if installed on a dedicated node (which is now the recommended deployment) it must be included.

To check that all expected services are published the following command can be used:

$ ldapsearch -x -h $SITE_BDII_HOST -p 2170 -b mds-vo-name=$SITE_NAME,o=grid \
 objectclass=GlueService \
 | perl -p00e 's/\r?\n //g' | grep Endpoint:

(replacing SITE_BDII_HOST; and SITE_NAME with the values for your site), which should list all the service URLs.

In addition, most services should now be published in GLUE 2 format. There is no explicit configuration needed for GLUE 2, but one thing to be aware of is that the site name (and the other parts like o=grid) in the GIIS URL field in the GOCDB must have the correct case as GLUE 2 is case-sensitive.

To verify the GLUE 2 publication use the command:

$ ldapsearch -x -h $SITE_BDII_HOST -p 2170 -b GLUE2DomainID=$SITE_NAME,o=glue \
 objectclass=GLUE2Endpoint \
 | perl -p00e 's/\r?\n //g' | grep URL:

Some services, notably storage elements, may be missing or incomplete in GLUE 2 if they are older than the EMI 2 release. The following table shows the publishing status for gLite and WLCG node types (ARC and Unicore have a different structure).

Node type	GLUE 1	GLUE 2	Notes
LCG-CE	Yes	No	Obsolete
CREAM	Yes	Yes	Full publication only in EMI 2
CLUSTER	Yes	Yes	Full publication only in EMI 2
WMS	Yes	Yes
LB	Yes	Yes
DPM	Yes	EMI 2
dCache	Yes	EMI 2
StoRM	Yes	EMI 2
LFC	Yes	EMI 2
FTS	Yes	EMI 2	Channels not yet published in GLUE 2
Hydra	EMI 2	EMI 2	Not yet released in EMI 2
AMGA	Yes	EMI 2
VOMS	Yes	Yes
MyProxy	Yes	Yes
Argus	No	EMI 2	Internal service, publication for deployment monitoring
Site BDII	Yes	Yes
Top BDII	Yes	Yes
R-GMA	Yes	No	Obsolete
VOBOX	Yes	Yes
Apel	No	No	Internal service, publishing not yet requested
Squid	No	No	Configuration exists but not enabled
Nagios	Yes	Yes

Federated Cloud BDII configuration

For information about configuration of a Federated Cloud BDII, please look at the EGI Information System.

GlueSite Object

These are the existing well established attributes in the GlueSite object. All of these MUST remain.

Attribute	Example	Schema	Notes
GlueSiteName	RAL-LCG2	Free text, no whitespace	Same as GOCDB name if in GOCDB, your choice.
GlueSiteUniqueID	RAL-LCG2	Identical to your !GlueSiteName	Same as GlueSiteName
GlueSiteWeb	`https://cern.ch/it`	Free Text	Valid URL about the site.
GlueSiteLatitude	52.42	NN.NN	Site Latitude.
GlueSiteLongitude	16.91	NN.NN	Longitude of Site.
GlueSiteDescription	Rutherford Lab	Free Text	A long name for the site.
GlueSiteLocation	Dublin, Ireland	Town, City, Country	An decreasing resolution ending with Country, agree a country name within a country. i.e UK != United Kingdom. Scotland and the Balkans should write a dynamic provider.
!GlueSiteUserSupportContact	`mailto:helpdesk@example.com`	Valid URL	URL for getting support. A ticket
system if available.
!GlueSiteSysAdminContact	`xmpp://admins@jabber.org`	Valid URL	How to contact the admins.
!GlueSiteSecurityContact	`mailto:security@example.com`	Valid URL	How to contact for security related matters.

The GlueSite object in the 1.3 Glue Schema contains an attribute GlueSiteOtherInfo. To quote.

The attribute is to be used to publish data that does not fit any other attribute of the site entity. A name=value pair or an XML structure are example[s] of usage.

All this extra configuration will be with in the static information for the glue site within the Grid Information Provider system.

Guidelines for GlueSite Object

A format for publishing useful information about sites within the !GlueSiteOtherInfo is needed, as shown in the following table.

Key	Example	Type	Notes
GRID	EGI	[#validgrid List of valid grid names]	Multiple ones can be defined.
WLCG_TIER	1	Tier level of site in WLCG context.	Either 0, 1 , 2 , 3 , 4
WLCG_PARENT	UK-T1-RAL	Name of the higher (administrative) tier site in WLCG	The WLCG_NAME of the site at a higher tier with WLCG
WLCG_NAME	IT-ATLAS-federation	[#lcgnames Valid WLCG Names]	An official WLCG name.
WLCG_NAMEICON	`https://example.com/tier2.png`	Valid URL	URL to WLCGNAME icon, ideally 80x80 pixels.
EGEE_ROC	Russia	Valid federated Operations Centre name	Only applicable if your site is still part of a federated Operations Centre (“ROC” according to the old EGEE terminology). Name MUST match the Operations Centre name declared in GOCDB. Note. If the site is now part of a NGI, then EGI_NGI MUST be used (see below).
EGI_NGI	NGI_CZ	Valid NGI	Must agree with the GOC DB
EGEE_SERVICE	prod	prod, pps or cert	Which EGEE grid your site is part of, multiple attributes is okay. Obsolete in EGI.
OLDNAME	Bristol	text	If your !GlueSiteName changes at some point please record your old name here.
ICON	`https://example.com/icon.png`	Valid URL	Icon Image for your site, ideally 80x80 pixels
BLOG	`https://scotgrid.blogspot.com/feeds/posts/default`	Valid RSS or Atom Feed	Your site blog if you have one
CONFIG	yaim	yaim, puppet, quattor, …	The configuration tool(s) used at the site

Note. Keywords starting with one of the grid names are to some extent reserved for that grid.

Example

GlueSiteName: RAL-LCG2
GlueSiteOtherInfo: BLOG=https://example.com/blog/feed
GlueSiteOtherInfo: EGI_NGI=NGI_UK
GlueSiteOtherInfo: GRID=EGI
GlueSiteOtherInfo: GRID=GRIDPP
GlueSiteOtherInfo: GRID=WLCG
GlueSiteOtherInfo: ICON=https://example.com/images/tierOneSmall.png
GlueSiteOtherInfo: WLCG_PARENT=CERN-PROD
GlueSiteOtherInfo: WLCG_TIER=1

Distributed Tier1s and Tier2s

Within an WLCG context for instance there are instances of distributed Tier2s and Tier1s. If separate component sites want to exist as a single WLCG tier then they might contain common values for their WLCGNAME.

GlueSiteName: CSCS-LCG2
GlueSiteOtherInfo: CONFIG=yaim
GlueSiteOtherInfo: EGI_NGI=NGI_CH
GlueSiteOtherInfo: GRID=EGI
GlueSiteOtherInfo: GRID=WLCG
GlueSiteOtherInfo: WLCG_NAME=CH-CHIPP-CSCS
GlueSiteOtherInfo: WLCG_PARENT=FZK-LCG2
GlueSiteOtherInfo: WLCG_TIER=2

Note that WLCG_PARENT is an accounting unit defined in the MOU document, as shown in WLCG CRIC.

Established Grid Name

Short Name	Long Name	URL
EGI	European Grid Initiative	https://www.egi.eu
EELA	Europe and Latin America	https://www.eu-eela.eu/
WLCG	World LHC Computing Grid	https://cern.ch/lcg
GRIDPP	UK Particle Physics Grid	https://www.gridpp.ac.uk
UKNGS	National UK Grid Service	https://www.ngs.ac.uk
OSG	Open Science Grid (US)	https://www.opensciencegrid.org/
NDGF	Nordic DataGrid Facility	https://www.ndgf.org/
LondonGrid	London Grid	https://www.gridpp.ac.uk/tier2/london/
NORTHGRID	Northern (UK) Grid	https://www.gridpp.ac.uk/northgrid/
SCOTGRID	Scottish Grid	https://www.scotgrid.ac.uk/
SOUTHGRID	Southern (UK) Grid	https://www.gridpp.ac.uk/southgrid/
Academic Grid Malaysia	Malaysian Grid
UPM Campus Grid	Universiti Putra Malaysia	https://www.upm.edu.my/
AEGIS	Academic and Educational Grid Initiative of Serbia	https://www.aegis.rs/
BIGGRID	Dutch e-science Grid	https://www.biggrid.nl/
Consorzio Cometa	Consorzio Multi-Ente per la promozione e l’adozione di Tecnologie di calcolo Avanzato (Italy)	https://www.consorzio-cometa.it/en
D-Grid	German Grid	https://www.d-grid-gmbh.de/index.php?id=1&L=1
EUMED	EU/Mediterranean Grid	https://www.eumedgrid.eu/
GILDA	Grid INFN Laboratory for Dissemination Activities (Italy)	https://gilda.ct.infn.it/
GISELA	Grid Initiative for e-Science virtual communities in Europe and Latin America	https://www.gisela-grid.eu/
GRISU	Griglia del Sud (Southern Italy Grid)	https://www.grisu-org.it/
NEUGRID	Neuroscience Grid	https://neugrid4you.eu/background
RDIG	Russian Data Intensive Grid	https://grid-eng.jinr.ru/?page_id=43
SEE-GRID	South Eastern European GRid-enabled eInfrastructure Development	https://www.see-grid.org/

Important: The EGEE Grid name was decommissioned on [[Agenda-14-02-2011|14-02-2011]]. All sites need to replace this grid name with EGI.

Being part of a grid is just a reference that your site is in some way associated with a particular Resource Infrastructure Provider either technically or as part of a collaboration. The list of Grids can be extended. Please contact operations@egi.eu to request changes.

Valid WLCG Names

The WLCG names are the site names that appear within the LCG MOU concerning commitments to LHC computing.

WLCG Name	Current GlueSiteName
CA-TRIUMF	TRIUMF-LCG2
CERN	CERN-PROD
DE-KIT	FZK-LCG2
ES-PIC	pic
FR-CCIN2P3	IN2P3-CC
IT-INFN-CNAF	INFN-T1
NDGF	NDGF-T1
NL-T1	SARA-MATRIX
TW-ASGC	Taiwan-LCG2
UK-T1-RAL	RAL-LCG2
US-FNAL-CMS	USCMS-FNAL-W1
US-T1-BNL	BNL-LCG2

For the tier two names please consult WLCG CRIC. The column marked Accounting Name are the WLCG Names which in the case of Tier2s are the GOCDB names. Use your site GOCDB name as your WLCG_NAME.

Also some tier2s live under more than 1 tier1 perhaps for different for different VOs. If your tier2 has more that one WLCG_PARENT then just add two distinct records to show this. Also some tier2s do not have a WLCGNAME at all.

GlueSiteUniqueId: EENet
GlueSiteName: EENet
GlueSiteOtherInfo: GRID=WLCG
GlueSiteOtherInfo: GRID=EGI
GlueSiteOtherInfo: EGI_NGI=NGI_NL
GlueSiteOtherInfo: WLCG_TIER=2
GlueSiteOtherInfo: WLCG_PARENT=UK-T1-RAL
GlueSiteOtherInfo: WLCG_PARENT=NL-T1

Valid EGI NGI Names

The valid names are those published on GOCDB.

YAIM Instructions

YAIM will have to be updated for those sites using yaim. This will be done and submitted to sites in the normal way.

YAIM Variable and Value	Resulting Glue Attribute and Value
SITE_NAME=RAL_LCG2	GlueSiteName: RAL-LCG2
SITE_DESC=“Rutherford Lab”	GlueSiteDescription: Rutherford Lab
SITE_EMAIL= steve@example.com	GlueSiteSysAdminContact: mailto:steve@example.com
SITE_SUPPORT_EMAIL= steve@example.com	GlueSiteUserSupportContact: mailto:steve@example.com
SITE_SECURITY_EMAIL= steve@example.com	GlueSiteSecurityContact: mailto:steve@example.com
SITE_LOC=“Soho, London, United Kingdom”	GlueSiteLocation: Soho, London, United Kingdom
SITE_LONG=52.45	GlueSiteLongitude: 52.45
SITE_LAT=-12.34	GlueSiteLatitude: -12.34
SITE_WEB=“https://example.com/"	GlueSiteWeb: https://example.com/
SITE_OTHER_GRID=“EGI\|WLCG”	GlueSiteOtherInfo: GRID=EGI GlueSiteOtherInfo: GRID=WLCG
SITE_OTHER_EGEE_ROC=“UK/I”	GlueSiteOtherInfo: EGEE_ROC=UK/I
SITE_OTHER_EGI_NGI=“NGI_CZ”	GlueSiteOtherInfo: EGI_NGI=NGI_CZ
SITE_OTHER_EGEE_SERVICE=“prod”	GlueSiteOtherInfo: EGEE_SERVICE=prod
SITE_OTHER_WLCG_TIER=2	GlueSiteOtherInfo: WLCG_TIER=2
SITE_OTHER*="\|”	GlueSiteOtherInfo: KEY=GlueSiteOtherInfo: KEY=

If multiple values for GlueSiteOtherInfo are needed, then just delimit your values with a |. The character | must be avoided in values.

Check your own GlueSite Object

The information published can be checked through an ldap search:

$ ldapsearch -x -H ldap://$SITE_BDII_HOST:2170 \
 -b 'Mds-Vo-Name=$SITE_NAME,o=Grid' \
 '(ObjectClass=GlueSite)'

In addition, VAPOR is a tool which provides a GUI for different views of published information, including a LDAP view.

Site information in GLUE 2

The GLUE 2 equivalent of the GlueSite object is the GLUE2AdminDomain. The same information should be present although in a slightly different format, and there are separate GLUE2Contact and GLUE2Location objects.

Providers: MAN02 Service intervention management

Mon, 01 Jan 0001 00:00:00 +0000

Document control

Property	Value
Title	Service Intervention Management
Policy Group	Operations Management Board (OMB)
Document status	Approved
Procedure Statement	Managing Service interventions
Owner	SDIS team

Service Intervention

A service intervention is defined as an action which will involve or lead to the possibility of a loss, or noticeable degradation of a service. Depending on the planning of the outage, we have two types of intervention:

Scheduled interventions: planned and agreed in advance
Unscheduled interventions: unplanned, usually triggered by an unexpected failure

How to manage an intervention

Interventions are recorded through the Configuration Database. For more information, have a look at the description.

Scheduled interventions

Scheduled interventions MUST be declared at least 24 h in advance, specifying reason and duration.
Existing scheduled interventions CAN be extended, provided that it’s done 24 hours in advance.

Unscheduled interventions

Any intervention declared less than 24 h in advance will be considered unscheduled.
Sites MUST declare unscheduled interventions as soon as they are detected to inform the users. Unscheduled interventions CAN be declared up to 48 hours in the past (retroactive information to the user community).

Required information

The required information to fill in when declaring an intervention is:

Severity (Outage or Warning)
Description
Timezone
Starting and ending dates
Affected site / Affected services and endpoints

Recommendations

For interventions that impact end users, the downtime SHOULD be declared 5 working days in advance, specifying reason and duration.
A post−mortem SHOULD be included in the downtime report.

Notifications

Intervention notifications (through broadcasts, RSS feeds, etc) as specified in the following procedures are automatically sent when declaring a downtime in Configuration Database: at declaration time, 24 h in advance and 1 h before the intervention.

Suspension policy

Sites on downtime for more than 1 month will be suspended/uncertified. AT_RISK downtime declarations are only for providing warnings to users, and are ignored for calculating site availability (actual status will be used).

Providers: MAN04 Tool Intervention Management

Mon, 01 Jan 0001 00:00:00 +0000

Property	Value
Title	Tool Intervention Management
Policy Group	Operations Management Board (OMB)
Document status	Approved
Procedure Statement	How to manage central operational tool unscheduled downtimes
Owner	SDIS team

The purpose of this document is to describe the intervention in case of unscheduled failure of central operational tool.

Scope

This manual only applies to unscheduled downtimes of central operational tools. The list of central operational tool is available here.

Note: Scheduled downtimes are management according to existing procedures (MAN02).

Announcements

All announcements should be sent with the Operations Portal Broadcast tool.

When using Operations Portal Broadcast tool the following groups should be included:

LCG Rollout Mailing List
Operators Mailing lists
OSG Mailing list
Tool Admins Mailing List
WLCG Tier 1 contacts
NGI managers
VO managers
VO users
Site administrators
Operation tools

Notice: Individual notification templates together with targets are predefined in Operations Portal Broadcast tool. Administrators are advised to use such predefined templates.

Procedure

In the following sections several relevant scenarios are covered.

Case 1: short “undetected” downtime

Description: Service fails and recovers before administrator manages to react (e.g. short power or network outage).

Action: Administrator announces the failure by using the following template:

Subject:
 [SERVICE_NAME] unscheduled downtime

Message:
 Dear all,

 [SERVICE_NAME] experienced unscheduled downtime between [START] and [END].

 [DETAILED_FAILURE_DESCRIPTION]

 Apologies for any inconvenience caused.

 Best Regards
 [SERVICE_TEAM]

Case 2: long “detected” outage

Service fails and administrator detects the problem. The problem takes at least 1 hour time to recover. In the sections below individual situations are described.

1. Outage

Description: Service failure is detected.

Action: Administrator announces the failure by using the following template:

Subject:
 [SERVICE_NAME] outage

Message:
 Dear all,

 [SERVICE_NAME] is experiencing unscheduled downtime.

 [ADDITIONAL_INFO]

 Apologies for any inconvenience caused.

 Best Regards
 [SERVICE_TEAM]

2. Extended downtime

Description: Service recovery is delayed. Update should be sent at least every 24h.

Action: The administrator announces that recovery is taking longer by using the following template:

Subject:
 [SERVICE_NAME] extended outage

Message:
 Dear all,

 Outage of [SERVICE_NAME] is extended.

 [ADDITIONAL_INFO]

 Apologies for any inconvenience caused.

 Best Regards
 [SERVICE_TEAM]

Note: In this template [ADDITIONAL_INFO] should indicate the estimated time of recovery.

3. Recovery

Description: Service is recovered.

Actions: At the time of recovery the administrator announces the recovery by using the following template:

Subject:
 [SERVICE_NAME] recovery

Message:
 Dear all,

 [SERVICE_NAME] is back online.

 [ADDITIONAL_INFO]

 Best Regards
 [SERVICE_TEAM]

4. Post mortem analysis

Description: Service failure required further time to investigate the source of the problem. This action is required only if the post mortem analysis is needed.

Actions: The administrator announces the post mortem analysis of failure by using the following notification template:

Subject:
 [SERVICE_NAME] outage analysis

Message:
 Dear all,

 [SERVICE_NAME] experienced unscheduled downtime between [START] and [END].

 [DETAILED_FAILURE_DESCRIPTION]

 Best Regards
 [SERVICE_TEAM]

Providers: MAN05 top and site BDII High Availability

Mon, 01 Jan 0001 00:00:00 +0000

Document control

Property	Value
Title	MAN05 top and site BDII High Availability
Policy Group	Operations Management Board (OMB)
Document status	Approved
Procedure Statement	Deploying top or site BDII service in High Availability
Owner	SDIS team

Top BDII and Site BDII with High Availability

This document objective is to provide guidelines to improve the availability of the information system, addressing three main areas:

Requirements to deploy a top or site BDII service
High Availability from a client perspective
Configuration of a High Availability top or site BDII service

Requirements to deploy a top or site BDII service

Hardware

dual core CPU
10GB of hard disk space
2-3 GB RAM. If you decide to set BDII_RAM_DISK=yes in your YAIM configuration, it’s advisable to have 4GB of RAM.

Co-hosting

Due to the critical nature of the information system with respect to the operation of the grid, the top or site BDII should be installed as a stand-alone service to ensure that problems with other services do not affect the BDII. In no circumstances should the BDII be co-hosted with a service which has the potential to generate a high load.

Physical vs Virtual Machines

There is no clear vision on this topic. Some managers complain that there are performance issues related to deploying a top or site BDII service under a virtual machine. Others argue that such performance issues are related to the configuration of the service itself. The only agreed feature is that the management and disaster recovery of any service deployed under a virtual machine is more flexible and easier. This could be an important point to take into account considering the critical importance of the top or site BDII service.

Best practices from a client perspective for top BDII

In gLite 3.2 and EMI you can set up redundancy of top BDIIs for the clients (WNs and UIs) setting up a list of top BDII instances to support the automatic failover in the GFAL clients. If the first Top level BDII fails to be contacted, the second will be used in its place, and so on. This mechanism is implemented defining the BDII_LIST YAIM variable according to the following syntax:

BDII_LIST=topbdii.domain.one:2170[,topbdii.domain.two:2170[...]].

After running YAIM, the client enviroment should contain the following definition:

LCG_GFAL_INFOSYS=topbdii.domain.one:2170,topbdii.domain.two:2170

The data management tools (lcg_utils) contact the information system for every operation (lcg-cr, lcg-cp, …). So, if you have your client properly configured with redundancy for the information system, the lcg_utils tools will use that mechanism in a transparent way. Be aware that lcg-infosites doesn’t work with multiple BDIIs. Only gfal, lcg_utils, lcg-info and glite-sd-query.
Site administrators should configure their services with this failover mechanism where the first top BDII of the list should be the default top BDII provided by their NGI.

Best practices for a top or site BDII High Availability service

The best practice proposal to provide a high availability site or top BDII service is based on two mechanisms working as main building blocks:

DNS round robin load balancing
A fault tolerance DNS Updater

We will provide a short introduction to some of these DNS mechanisms but for further information on specific implementations, please contact your DNS administrator.

DNS round robin load balancing

Load balancing is a technique to distribute workload evenly across two or more resources. A load balancing method, which does not necessarily require a dedicated software or hardware node, is called round robin DNS.
We can assume that all transactions (queries to top or site BDII generate the same resource load. For an effective load balancing, all top or site BDII instances should have the same hardware configurations. In other case, a load balancing arbiter is needed.
Simple round robin DNS load balancing is easy to deploy. Assuming that there is a primary DNS server (dns.domain.tld) where the DNS load balancing will be implemented, one simply has to add multiple A records mapping the same hostname to multiple IP addresses under the core.top.domain DNS zone. It is equally applicable to site BDII.

# In dns.domain.tld: Add multiple A records mapping the same hostname to multiple IP addresses
Zone core.domain.tld
topbdii.core.domain.tld IN A x.x.x.x
topbdii.core.domain.tld IN A y.y.y.y
topbdii.core.domain.tld IN A z.z.z.z

The 3 records are always served as answer but the order of the records will rotate in each DNS query
This does NOT provide fault tolerance against problems in the top or site BDIIs themselves

if one top or site BDII fails its DNS “A” record will still be served
one in each three DNS queries will provide the failed top or site BDII first answer

Fault tolerance DNS Updater

The DNS Updater is a mechanism (to be implemented by you) which tests the different top or site BDIIs and decides to remove or add DNS entries through DNS dynamic updates. The fault tolerance is implemented by dynamically nsupdate introduced in bind V8 offers the possibility of changing DNS records dynamically:

The nsupdate tool connects to a bind server on port 53 (TCP or UDP) and can update zone records
Updates are authorized based on keys
Updates can only be performed on the DNS primary server
In the DNS bind implementation, the entire zone is rewritten by the DNS server upon “stop” to reflect the changes. Therefore, the zone should not be managed manually; and the changes are kept in a zone journal file until a “stop” happens.

Implementation

There are several alternatives to implement the DNS Updater:

NAGIOS based tests
a demonized service
scripts running as crons

What to test: BDII metrics

Status information about the BDII is available by querying the o=infosys root for the UpdateStats object. This entry contains a number of metrics relating to the latest update such as the time to update the database and the total number of entries. And example of such entry is shown below.

$ ldapsearch -x -h <TopBDII/siteBDII> -p 2170 -b "o=infosys"

(...)

dn: Hostname=localhost,o=infosys
objectClass: UpdateStats
Hostname: lxbra2510.cern.ch
FailedDeletes: 0
ModifiedEntries: 4950
DeletedEntries: 1318
UpdateTime: 150
FailedAdds: 603
FailedModifies: 0
TotalEntries: 52702
QueryTime: 8
NewEntries: 603
DBUpdateTime: 11
ReadTime: 0
PluginsTime: 4
ProvidersTime: 113

More extensive information can be obtained (modifyTimestamp,createTimestamp) adding the +:

$ ldapsearch -x -h <TopBDII/siteBDII> -p 2170 -b "o=infosys" +

(...)

# localhost, infosys
dn: Hostname=localhost,o=infosys
structuralObjectClass: UpdateStats
entryUUID: 09bf40e0-7b23-4992-af55-fd74f036a454
creatorsName: o=infosys
createTimestamp: 20110612223435Z
entryCSN: 20110615120723.216201Z#000000#000#000000
modifiersName: o=infosys
modifyTimestamp: 20110615120723Z
entryDN: Hostname=localhost,o=infosys
subschemaSubentry: cn=Subschema
hasSubordinates: FALSE

The following table shows the meaning of the most relevant metrics:

Metric	Description
ModifiedEntries	The number of objects to modify
DeletedEntries	The number of objects to delete
UpdateTime	To total update time in seconds
FailedAdds	The number of add statements which failed
FailedModifies	The number of modify statements which failed
TotalEntries	The total number of entries in the database
QueryTime	The time taken to query the database
NewEntries	The number of new objects
DBUpdateTime	The time taken to update the database in seconds
ReadTime	The time taken to read the LDIF sources in seconds
PluginsTime	The time taken to run the plugins in seconds
ProvidersTime	The time taken to run the information providers in seconds

Previous BDII metrics can be checked to take a decision regarding the reliability and availability of a top or site BDII instance.
More information is available in gLite-BDII_top Monitoring.

DNS caching

DNS records obtained in queries are cached by the DNS servers (usually during 24 hours). Therefore to propagate DNS changes fast enough it is important to have very short TTL lifetimes.
DNS has not been built to have very short TTL values and these may increase highly the number of queries and as result increase the load of the DNS server
The TTL lifetime to be used will have to be tested.
If the top BDII are only used by sites in the region and if queries are only from the DNS servers of these few sites then the number of queries may be low enough to allow for a very small TTL.
This value should not be lower than 30s - 60s.

Example 1: The IGI Nagios based mechanism

In IGI, the DNS update of the number of instances participating in the DNS round robin mechanism depends on the results provided by a Nagios instance.
When Nagios needs to check the status of a service it will execute a plugin and pass information about what needs to be checked. The plugin verifies the operational state of the service and reports the results back to the Nagios daemon.
Nagios will process the results of the service check and take appropriate action as necessary (e.g. send notifications, run event handlers, etc).
Each instance is checked every 5 minutes. If a failure occurs, Nagios runs the event handler to restart the BDII service AND remove the instance from the DNS round robin set using dnsupdate:

an email is sent as notification;
If 4 (out of 5) instances are failing, a SMS message is sent as notification;

If a failed instance appears to be restored, Nagios will re-add it to the DNS round robin mechanism.

This approach has some single points of failures:

The Nagios instance can fail
The master DNS where the DNS entries are updated can fail

Example 2: The IBERGRID scripting based mechanism

In IBERGRID, an application (developed by LIP) verifies the health of each top BDII. The application can connect to the DNS servers and remove the “A” records of top BDIIs that become unavailable (non responsive to tests).
The monitoring application (nsupdater) is a simple program that performs tests, and based on their result acts upon DNS entries

Written in perl
Can be run as daemon or at the command prompt
The tests are programs that are forked
Tests are added in a “module” fashion way
Can be used to manage several DNS round robin scenarios
Can manage multiple DNS servers

To remove the DNS single point of failure as in previous example, one could configure all DNS servers serving the core.ibergrid.eu domain as primary

Three primary servers would then exist for core.ibergrid.eu
All three DNS servers could be dynamically updated independently
The monitoring application should also have three instances, one running at each site
The downside is that DNS information can become incoherent. It would be up to the monitoring application to manage the three DNS servers content and their coherence

Providers: MAN06 Failover for MySQL grid-based services

Mon, 01 Jan 0001 00:00:00 +0000

Document control

Property	Value
Title	MAN06 Failover for MySQL grid-based services
Policy Group	Operations Management Board (OMB)
Document status	Approved
Procedure Statement	Implementing failover of MySQL grid-based services.
Owner	SDIS team

Introduction

Several critical grid services such as the VO Management Service (VOMS) server represent single points of failure in a grid infrastructure. When unavailable, a user can no longer access to the infrastructure since it is prevented from issuing new proxies, or is no longer able to access to the physical location of his data.
However, those services rely on MySQL backends which opens a window to replicate the relevant databases to different / backup services which could be used when the primary instances are unavailable. The MySQL DB replication process is one of the ways to get scalability and higher availability.

Architecture

In this document we propose to follow a Master-Slave architecture for the MySQL replication, consisting in keeping DB copies of a main host (MASTER) in a secondary host (SLAVE). The slave host will only have read access to the Database entries.

Security

For better availability, it is preferable to deploy the Master and the Slave services in different geographical locations, which normally means exposing the generated traffic to the internet. In that case, you will have to find a mechanism to encrypt the communication between the two hosts.
In this document, we propose to use Stunnel:

Stunnel is a free multi-platform computer program, used to provide universal TLS/SSL tunneling service.
Stunnel can be used to provide secure encrypted connections for clients or servers that do not speak TLS or SSL natively. It runs on a variety of operating systems, including most Unix-like operating systems and Windows.
Stunnel relies on a separate library such as OpenSSL or SSLeay to implement the underlying TLS or SSL protocol.
Stunnel uses Public-key cryptography with X.509 digital certificates to secure the SSL connection. Clients can optionally be authenticated via a certificate too
For more references, please check www.stunnel.org

There are other possibilities for encryption, like enabling SSL Support directly in MySQL, but these approach was not tested. Details can be obtained here.

MySQL replication

Assumptions

We are assuming that both grid services instances were previously installed and configured (manually or via YAIM) so that they support the same VOs.

Generic information

Description	Value
MASTER hostname	server1.domain.one
MASTER username	root
MASTER mysql superuser	root
MASTER mysql replication user	db_rep
SLAVE hostname	server2.domain.two
SLAVE username	root
SLAVE mysql superuser	root
DB to replicate	DB_1 DB_2 …

Setup the MySQL MASTER for replication

Step 1: Install stunnel. It is available in the SL5 repositories.

$ yum install stunnel

(...)

$ date; rpm -qa | grep stunnel

Wed Jun 15 16:00:23 WEST 2011
stunnel-4.15-2.el5.1

Step 2: Configure stunnel (via /etc/stunnel/stunnel.conf) to:

Accept incoming connections on port 3307, and allow to connect to port 3306
Use the server X509 certificates to encrypt data

$ cat /etc/stunnel/stunnel.conf

# Authentication stuff verify = 2
CApath = /etc/grid-security/certificates
cert = /etc/grid-security/hostcert.pem
key = /etc/grid-security/hostkey.pem

# Auth fails in chroot because CApath is not accessible
#chroot = /var/run/stunnel
#debug = 7
output = /var/log/stunnel.log
pid = /var/run/stunnel/master.pid

setuid = nobody
setgid = nobody

# Service-level configuration
[mysql]
accept = 3307
connect = 127.0.0.1:3306

Step 3: Start the stunnel service and add it to rc.local

$ mkdir /var/run/stunnel
$ chown nobody:nobody /var/run/stunnel
$ stunnel /etc/stunnel.conf
$ echo 'stunnel /etc/stunnel.conf' >> /etc/rc.local

Step 4: Setup the firewall to allow connections from the SLAVE instance (server2.domain.two with IP XXX.XXX.XXX.XXX) on port 3307 TCP

# MySQL replication
-A INPUT -s XXX.XXX.XXX.XXX -m state --state NEW -m tcp -p tcp \
 --dport 3307 -j ACCEPT

Step 5: Configure MySQL (via /etc/my.cnf) setting up the Master server-id (usually 1), and declaring the path for the MySQL binary log and the DBs to be replicated. Please make sure that the path declared under log-bin has the mysql:mysql ownerships, and that the binary log exists.

$ cat /etc/my.cnf

(...)

[mysqld]
server-id=1
log-bin = /path_to_log_file/log_file.index
binlog-do-db=DB_2
binlog-do-db=DB_1

Step 6: Restart MySQL

$ /etc/init.d/mysqld restart

Step 7: Add a specific user for the MySQL replication

mysql > GRANT REPLICATION SLAVE ON *.* TO 'db_rep'@'127.0.0.1' \
 IDENTIFIED BY '16_char_password';

Step 8: Backup the databases using the following command. Please note tha the option --master-data=2 writes a comment on dump.sql that shows the log file and the ID to be used on the Slave setup. This option also locks all the tables while they are being copied, avoiding problems.

$ mysqldump -u root -p --default-character-set=latin1 \
 --master-data=2 --databases DB_1 DB_2 > dump.sql

Setup the MySQL SLAVE for replication

Step 1: Configure stunnel (via /etc/stunnel/stunnel.conf) to:

Accept incoming SSL connections on port 3307, and allow to connect to server1.domain.one on port 3307
Use the server X509 certificates to encrypt data

$ cat /etc/stunnel/stunnel.conf

# Authentication stuff verify = 2
CApath = /etc/grid-security/certificates
cert = /etc/grid-security/hostcert.pem
key = /etc/grid-security/hostkey.pem

# Auth fails in chroot because CApath is not accessible
#chroot = /var/run/stunnel
#debug = 7
output = /var/log/stunnel.log
pid = /var/run/stunnel/master.pid

setuid = nobody setgid = nobody

# Use it for client mode client = yes

# Service-level configuration [mysql] accept = 127.0.0.1:3307
connect = server1.domain.one:3307

Step 2: Start stunnel and add it to rc.local

$ mkdir /var/run/stunnel $ chown nobody:nobody /var/run/stunnel
$ stunnel /etc/stunnel.conf
$ echo 'stunnel /etc/stunnel.conf' >> /etc/rc.local

Step 3: Configure MySQL to include the SLAVE server-id (typically 2) and the replicated databases

$ cat /etc/my.cnf

(...)
[mysqld]
server-id=2
replicate-do-db=DB_1
replicate-do-db=DB_2

Step 4: Restart MySQL and insert the dump.sql created on the Master

$ /etc/init.d/mysql restart
$ mysql -u root -p < dump.sql

Step 5: Start the Slave logging in the mysql of slave, and running the following queries, changing the values xxxxx and yyyyy by the values on top of dump.sql file:

mysql > CHANGE MASTER TO MASTER_HOST='127.0.0.1', MASTER_PORT=3307, \
 MASTER_USER='db_rep', MASTER_PASSWORD='16_char_password', \
 MASTER_LOG_FILE='xxxxx', MASTER_LOG_POS=yyyyy;
mysql > SLAVE START;

Step 6: Your replication should be up and running. In case of troubles, check the Troubleshooting section.

Troubleshooting

On the Slave

Check if you can connect to the Master on port 3307 from the Slave

#root@server2.domain.two]$ telnet server1.domain.one 3307

Check the /var/log/stunnel.log on the Slave to see if stunnel is working. For established connections, you should find a message like:

2011.05.30 11:57:45 LOG5[9000:1076156736]: mysql connected from XXX.XXX.XXX.XXX:PORTY
2011.05.30 11:57:45 LOG5[9000:1076156736]: VERIFY OK: depth=1, /DC=COUNTRY/DC=CA/CN=NAME
2011.05.30 11:57:45 LOG5[9000:1076156736]: VERIFY OK: depth=0, /DC=COUNTRY/DC=CA/O=INSTITUTE/CN=host/HOSTNAME

Check the MySQL process list. You should get an answer like the one bellow:

mysql> SHOW PROCESSLIST;
+----+-------------+------+------+---------+---------+-----------------------------------------------------------------------------+------+
| Id | User | Host | db | Command | Time | State | Info |
+----+-------------+------+------+---------+---------+-----------------------------------------------------------------------------+------+
| 1 | system user | | NULL | Connect | 1236539 | Waiting for master to send event | NULL |
| 2 | system user | | NULL | Connect | 90804 | Slave has read all relay log; waiting for the slave I/O thread to update it | NULL |
+----+-------------+------+------+---------+---------+-----------------------------------------------------------------------------+------+

Check the status of the slave

mysql> SHOW SLAVE STATUS;
+----------------------------------+-------------+------------------+---------------------+
| Slave_IO_State | Master_Port | Master_Log_File | Read_Master_Log_Pos |
+----------------------------------+-------------+------------------+---------------------+
| Waiting for master to send event | 3307 | mysql-bin.000001 | 126167682 |
+----------------------------------+-------------+------------------+---------------------+

On the Master

Check the /var/log/stunnel.log on the Slave to see if stunnel is working. For established connections, you should find a message like:

2011.05.30 11:57:45 LOG5[9000:1076156736]: mysql connected from XXX.XXX.XXX.XXX:PORTY
2011.05.30 11:57:45 LOG5[9000:1076156736]: VERIFY OK: depth=1, /DC=COUNTRY/DC=CA/CN=NAME
2011.05.30 11:57:45 LOG5[9000:1076156736]: VERIFY OK: depth=0, /DC=COUNTRY/DC=CA/O=INSTITUTE/CN=host/HOSTNAME

Check if you have established connections from the Slave on port 3307

$ netstat -tapn | grep 3307

tcp 0 0 0.0.0.0:3307 0.0.0.0:* LISTEN 9000/stunnel
tcp 0 0 XXX.XXX.XXX.XXX:3307 YYY.YYY.YYY.YYY:34378 ESTABLISHED 9000/stunnel

Check the Master status on MySQL. You should get an answer like:

mysql> SHOW MASTER STATUS;
+------------------+-----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+-----------+--------------+------------------+
| mysql-bin.000001 | 126167682 | DB_1 | |
+------------------+-----------+--------------+------------------+
1 row in set (0.00 sec)

Check the MySQL process list. You should get an answer like the one bellow:

mysql> SHOW PROCESSLIST;
+------+-----------+-----------------+------+-------------+---------+----------------------------------------------------------------+------+
| Id | User | Host | db | Command | Time | State | Info |
+------+-----------+-----------------+------+-------------+---------+----------------------------------------------------------------+------+
| 2778 | janedoe | localhost:42281 | NULL | Binlog Dump | 1400477 | Has sent all binlog to slave; waiting for binlog to be updated | NULL |
+------+-----------+-----------------+------+-------------+---------+----------------------------------------------------------------+------+

The VOMS case

A working example

It is also possible to deploy a backup VOMS server with a MySQL replica of the main VOMS server. This will enable users to still start proxies even if the main VOMS server is down.
The VOMS Admin interface of the backup VOMS server should be switched off so that new users can only request registration via the main VOMS Admin Web interface.
User interfaces should be configured to use both VOMS servers

$ voms-proxy-init --voms ict.vo.ibergrid.eu

Enter GRID pass phrase:
Your identity: /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=Goncalo Borges
Creating temporary proxy .............................................................. Done
Contacting voms01.ncg.ingrid.pt:40008 [/C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=voms01.ncg.ingrid.pt] "ict.vo.ibergrid.eu" Done
Creating proxy .......................................................................................................................................................... Done
Your proxy is valid until Thu Jun 16 07:27:31 2011

$ voms-proxy-init --voms ict.vo.ibergrid.eu

Enter GRID pass phrase:
Your identity: /C=PT/O=LIPCA/O=LIP/OU=Lisboa/CN=Goncalo Borges
Creating temporary proxy ............................................................. Done
Contacting ibergrid-voms.ifca.es:40008 [/DC=es/DC=irisgrid/O=ifca/CN=host/ibergrid-voms.ifca.es] "ict.vo.ibergrid.eu" Done
Creating proxy ............................................................................ Done
Your proxy is valid until Thu Jun 16 07:27:37 2011

Providers: MAN07 VOMS Replication

Mon, 01 Jan 0001 00:00:00 +0000

Document control

Property	Value
Title	MAN07 VOMS Replication
Policy Group	Operations Management Board (OMB)
Document status	Approved
Procedure Statement	How to implement a MySQL VOMS server replication
Owner	SDIS team

Introduction

In this manual we will show you how to implement a MySQL VOMS server replication: you need one master server, on which you can perform writing operations, and you can have from 1 to “n” replica servers that will work in read-only mode. In such a scenario you can do a whatever intervention on one of the servers without breaking the service, i.e. proxies creation and grid-mapfile downloads: just the users registration and the usual VOs management operations might be forbidden during an intervention on the master server (because it is the only server in writing mode).

This failover procedure is simply based on MySQL replication therefore every MySQL setting is referred to the current MySQL version (5.0.77 in this moment)

Settings on the MASTER SERVER

In order to allow the replica server to read the master database, you have to create an user with which the slave will connect to the master. Suppose the replica hostname is vomsrep.cnaf.infn.it, the user is janedoe and the password is always. What you have to launch on the master server is:

$ mysql -p -e "grant super, reload, replication slave, replication client \
 on *.* to janedoe@'vomsrep.cnaf.infn.it' identified by 'always'" ;

Then for each DB (VO) you want to replicate, you have to assign the right permissions, by launching:

mysql -p -e "grant select, lock tables on voms_myvo.* to \
 janedoe@'vomsrep.cnaf.infn.it'"

Eventually you have to modify the file /etc/my.cnf by adding the following lines into the section [mysqld]:

log-bin=mysql-bin
server-id=1
innodb_flush_log_at_trx_commit=1
sync_binlog=1

It is important that on the master server it is set server-id=1: it is the identification number that distinguish a master from its several slaves (each slave will have a unique number starting from 2)

For example, the content of my.cnf file may appear like this:

# less /etc/my.cnf

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql

# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
old_passwords=1

max_connections = 800
log-bin=mysql-bin server-id=1
innodb_flush_log_at_trx_commit=1
sync_binlog=1
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line: # symbolic-links=0

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

At this point, you have to restart MySQL, by launching:

$ service mysqld restart

In order to check that on the master side the mechanism is working, you can launch for example:

mysql> show master status;
+------------------+----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000001 | 24844 | | |
+------------------+----------+--------------+------------------+
1 row in set (0.00 sec)

Eventually, through the web interface, in the ACL section of each VO you want to replicate, add an entry granting all the permissions to the slave server:

select “a non VO member” from the menu
fill in the replica server DN and a reference email address
select “all” for the permissions and tick the “Propagate entry to children contexts” option

In this way, when the slave server copies the DB, it will have the proper permissions on acting on the DB. Moreover, in order to avoid the sending of notification to the email address you filled in before, connect to the MySQL database and do the following:

mysql> use voms_myvo;
mysql> update admins set email_address=NULL where \
 email_address="what you filled before";

Settings on the SLAVE SERVER

Install a VOMS server as usual, configuring the VOs you want to replicate: keep in mind that every modification done on the slave DB breaks the replica mechanism, so that on this server disable the users registration, by setting the yaim variable:

VOMS_ADMIN_WEB_REGISTRATION_DISABLE=true

Then ask the VO managers to not perform any action on the slave server web interface.

Then launch the following scripts:

first_replica.sh for the first database you want to replicate or in the case it is the only one
next_replicas.sh for the next databases (one database for each launch)

For both the scripts, set the following variables:

master_host, master_mysql_user, master_mysql_pwd that refers to the master server and to the user created on it
mysql_username_admin and mysql_password_admin that refers to the slave

Example:

voms_database="" # VOMS database (leave unset)
master_host="voms.cnaf.infn.it" # Master hostname
master_mysql_user="janedoe" # Master MySQL admin user for replication
master_mysql_pwd="always" # Master MySQL admin pass for replication user
master_log_file="" # Master LOG file (leave unset)
master_log_pos="" # Master LOG file (leave unset)
mysql_username_admin="root" # Slave MySQL admin username
mysql_password_admin="secret" # Slave MySQL admin pass

With the launch of first-replica.sh, the file /etc/my.cnf will be properly written; if you need to replicate further databases, modify /etc/my.cnf adding the following lines related to the db you are replicating (similar to the first db you’ve replicated):

replicate-do-db=<master_vo_db_name>
replicate-ignore-table=<master_vo_db_name>.seqnumber
replicate-ignore-table=<master_vo_db_name>.realtime
replicate-ignore-table=<master_vo_db_name>.transactions
replicate-ignore-table=<slave_vo_db_name>.seqnumber
replicate-ignore-table=<slave_vo_db_name>.realtime
replicate-ignore-table=<slave_vo_db_name>.transactions

Having set the variables in the way shown above, for replicating the first database the scripts launch syntax is the following:

$ ./first_replica.sh --master-db=voms_myvo --db=voms_myvo

In your /etc/my.cnf file you will find lines like the following:

# Connection with master
server-id=2
master-host=voms.cnaf.infn.it
master-user=janedoe
master-password=always

# Replicas settings
replicate-do-db=voms_myvo
replicate-ignore-table=voms_myvo.seqnumber
replicate-ignore-table=voms_myvo.realtime
replicate-ignore-table=voms_myvo.transactions
replicate-ignore-table=voms_myvo.seqnumber
replicate-ignore-table=voms_myvo.realtime
replicate-ignore-table=voms_myvo.transactions

Now you may want to replicate a second database, let’s say voms_hervo: therefore in my.cnf file add the following lines:

replicate-do-db=voms_hervo
replicate-ignore-table=voms_hervo.seqnumber
replicate-ignore-table=voms_hervo.realtime
replicate-ignore-table=voms_hervo.transactions
replicate-ignore-table=voms_hervo.seqnumber
replicate-ignore-table=voms_hervo.realtime
replicate-ignore-table=voms_hervo.transactions

Modify the script next_replicas.sh in according to the VO parameters and launch it:

$ ./next_replicas.sh --master-db=voms_hervo --db=voms_hervo

When you finished to replicate all the desired VOs, in order to make active the database modifications, restart voms and voms-admin:

$ /etc/init.d/voms-admin stop
$ /etc/init.d/voms stop
$ /etc/init.d/voms start
$ /etc/init.d/voms-admin start

Keep in mind that every modification done on the slave DB breaks the replica mechanism, so that on this server disable the users registration, by setting the yaim variable:

VOMS_ADMIN_WEB_REGISTRATION_DISABLE=true

And ask the VO managers to not perform any action on the slave server web interface.

Providers: MAN09 Accounting data publishing

Mon, 01 Jan 0001 00:00:00 +0000

Document control

Property	Value
Title	Accounting data publishing
Policy Group	Operations Management Board (OMB)
Document status	Approved
Procedure Statement	How to publish accounting information for different middleware
Owner	SDIS team

Introduction

In this manual we will show you how to publish accounting information from different middleware.

General information

Publishing with the APEL Client/SSM

To start sending accounting records:

register each endpoint sending the accounting records to the central repository as ‘gLite-APEL’ endpoint in GOCDB with host DN information
- either HTCondorCE or ARC-CE endpoints
- it is needed to authorise the endpoint to use the ARGO Message Service (AMS)
- Changes in GOCDB can take up to 4 hours to make it to AMS.
install the APEL client and APEL SSM on your publisher host and edit client.cfg and sender.cfg
install the APEL parser relevant to your batch system on each CE and edit parser.cfg
- documentation for APEL here
  - GitHub apel/apel
  - GitHub apel/ssm
The old ActiveMQ network was dismissed: have a look at the new settings to properly publish the accounting records via AMS
configure the client to publish data from the start of the current month.
Run the parser(s) and client.
If there are no errors messages and the client says it has sent messages then wait a day and you should see summaries here
If it doesn’t open a GGUS ticket for the APEL Team.

Monitoring the accounting data publication

In order to monitor the regular publication of accounting data, it is enough registering only one CE with the APEL service type.

See here the details about the related Nagios probes.

Publishing summarized records or single ones

Sites can send either but the preferred option is summaries. Larger sites are recommended to send summaries.

The frequency for sending aggregated/summary records to APEL database?

We recommend sending data daily for all sites, whether sending summaries or individual records. I think this is in the interests of people who are using the portal so that what they see is accurate. The summaries are for a complete month or the current month so far.

Authorization and authentication

Authorization and authentication is made by the host certificate (which is signed by a trusted CA from the ca_policy_core package). Certificate should be registered in GOCDB. The certificate must not have the X509 extension: Netscape Cert Type: SSL Server because the message brokers will reject it.

ARC

ARC uses its own system to publish the accounting data through AMS, so please refer to the NorduGrid ARC 6 documentation:

Information relevant only for 6.4 ARC releases and beyond: Accounting-NG
The old ActiveMQ network was dismissed. ARC 6.12 introduces new settings for publishing the accounting records via AMS.

HTCondor-CE

To collect and publish the accounting data you need to install the APEL software as explained in the general information section. In addition, HTCondor-CE must be configured to create accounting records:

Information on configuring HTCondor-CE for APEL accounting: APEL Accounting for HTCondor-CE

Providers: HOWTO01 Using IGTF CA distribution

Mon, 01 Jan 0001 00:00:00 +0000

To ensure interoperability within and outside of EGI, the Policy on Acceptable Authentication Assurance defined a common set of trust anchors (in a PKIX implementation “Certification Authorities”) that all sites in EGI should install. In short, all CAs accredited to the Interoperable Global Trust Federation under the classic, MICS or SLCS Authentication Profiles are approved for use in EGI. When installing the ‘combined-assurance’ bundle, also IOTA issuers complying with assurance level DOGWOOD are included. Of course, sites may add additional CAs as long as the integrity of the infrastructure as a whole is not compromised. Also, if there are site or national policies/regulations that prevent you from installing a CA, these regulations take precedence – but you then must inform the EGI Security Officer (see EGI CSIRT) about this exception.

Release notes

Review the release notes containing important notices about the current release, as well as a list of changes to the trust fabric.

Installation

To install the EGI trust anchors on a system that uses the RedHat Package Manager (RPM) based package management system, we provide a convenience package to manage the installation. To install the currently valid distribution, all RPM packages are provided at

https://repository.egi.eu/sw/production/cas/1/current/

The current version is based on the IGTF release with the same version number. Install the meta-package ca-policy-egi-core (or ca-policy-egi-cam) and its dependencies to implement the core EGI policy on trusted CAs.

Using YUM package management

Add the following repo-file to the /etc/yum.repos.d/ directory:

[EGI-trustanchors]
name=EGI-trustanchors
baseurl=http://repository.egi.eu/sw/production/cas/1/current/
gpgkey=http://repository.egi.eu/sw/production/cas/1/GPG-KEY-EUGridPMA-RPM-4
gpgcheck=1
enabled=1

and then update your installation. How to update depends on your previous activity:

if you have previously ever installed the lcg-CA package, remove any references to https://linuxsoft.cern.ch/LCG-CAs/current from your YUM setup, and run

$ yum clean cache metadata
$ yum update lcg-CA

and you are done. This will update the packages installed to the latest version, and also install the new ca-policy-egi-core package as well as a ca-policy-lcg package. All packages encode the same set of dependencies
if you are upgrading from a previous EGI version only, just run

$ yum update ca-policy-egi-core

although at times you may need to clean the yum cache using

$ yum clean cache metadata

if you are installing the EGI trust anchors for the first time, run

$ yum install ca-policy-egi-core

Using the distribution on a Debian or Debian-derived platform

The 1.39+ releases experimentally add the option to install the trust anchors from Debian packages using the APT dependency management system. Although care has been taken to ensure that this distribution is installable and complete, no guarantees are given, but you are invited to report your issues through GGUS. You may have to wait for a subsequent release of the Trust Anchor release to solve your issue, or may be asked to use a temporary repository. To use it:

Install the EUGridPMA PGP key for apt:

$ wget -q -O - \
 https://repository.egi.eu/sw/production/cas/1/current/GPG-KEY-EUGridPMA-RPM-4 \
 | gpg --dearmor -o /etc/apt/keyrings/GPG-KEY-EUGridPMA-RPM-4.gpg

Add the following egi-trustanchors.list to your sources.list.d:

#### EGI Trust Anchor Distribution ####
deb [signed-by=/etc/apt/keyrings/GPG-KEY-EUGridPMA-RPM-4.gpg] http://repository.egi.eu/sw/production/cas/1/current egi-igtf core

Populate the cache and install the meta-package

$ apt-get update
$ apt-get install ca-policy-egi-core

Using the distribution on other (non-RPM) platforms

The trust anchors are provided also as simple ’tar-balls’ for installation on other platforms. Since there is no dependency management in this case, please review the release notes carefully for any security issues or withdrawn CAs. The tar files can be found in the EGI repository at

https://repository.egi.eu/sw/production/cas/1/current/tgz/

Once you have downloaded the directory, you can unpack all the CA tar,gz as follows to your certificate directory:

$ for tgz in $(ls <ca download dir>); do
 tar xzf <ca download dir>/$tgz --strip-components=1 \
 -C /etc/grid-security/certificates
 done

Installing the distribution using Quattor

Quattor templates are provided as drop-in replacements for both QWG and CDB installations. Update your software repository (re-generating the repository templates as needed) and obtain the new CA templates from:

https://repository.egi.eu/sw/production/cas/1/current/meta/ca-policy-egi-core.tpl for QWG
https://repository.egi.eu/sw/production/cas/1/current/meta/pro_software_meta_ca_policy_egi_core.tpl for CDB

Make sure to mirror (or refer to) the new repository at https://repository.egi.eu/sw/production/cas/1/current/ and create the appropriate repository definition file.

For WLCG sites that are migrating from the lcg-CA package: the WLCG policy companion of the EGI templates can be found at QWG and CDB and can be included in the profile in parallel with the EGI core template. All packages needed are also included in the EGI repository, so only a single repository reference is necessary.

Combined Assurance/Adequacy Model

The release contains a “cam” (combined assurance/adequacy) package based on the approved policy on differentiated assurance. Technically, this means that you must ONLY install the new ca-policy-egi-cam packages if you ALSO at the same time implement VO-specific authorization controls in your software stack. This may require reconfiguration or a software update. Otherwise, just only install or update the regular ca-policy-egi-core package. There are no changes in this case. The ca-policy-egi-core package is approved for all VOs membership and assurance models. No configuration change is needed.

Acceptable Authentication Assurance

If a VO registration service or e-infrastructure registration service is accredited by EGI to meet the approved authentication assurance, an IGTF “DOGWOOD” accredited Authority - used solely in combination with said registration service - is also adequate for user authentication. See the policy for details.

This additional restriction policy must be implemented by each service in the authorization software. The “combined assurance” model package MUST NOT be installed unless the additional authorization is in place. You will need to reconfigure and may need to install upgrades. Not installing the new “cam” package does not have any detrimental effect on current users - only a new class of users (that can only obtain an opaque identifiers and do not do full vetting at their electronic identity provider) could be affected, and then only those users that are member of one of the communities that has part of the combined-assurance programme: LCG-Atlas, LCG-Alice, LCG-LHCb, and LCG-CMS.

Patches and workarounds

Reminder notice for VOMS AA operators

Several updates to this trust anchor distribution incorporate changes to the name of the issuing authority, but the name of the end-entities and the users remains exactly the same. To make the change transparent, all operators of VOMS and VOMS-Admin services are requested to enable the subject-only name resolution mechanisms in VOMS and VOMS Admin, see additional documentation in VOMS services configuration reference:

on the VOMS core Attribute Authority service, configure the -skipcacheck flag on start-up.
Set voms.skip_ca_check=True in the service properties.

Concerns, issues and verification

If you experience problems with the installation or upgrade of your trust anchors, or with the repository, please report such an issue through the GGUS system. For issues with the contents of the distribution, concerns about the trust fabric, or with questions about the integrity of the distribution, please contact the EGI IGTF liaison at egi-igtf-liaison@nikhef.nl.

You can verify the contents of the EGI Trust Anchor (CA) release with those of the International Grid Trust Federation, or its mirror. See the IGTF and EUGridPMA web pages for additional information.

Providers: HOWTO02 Site Certification Required Documentation

Mon, 01 Jan 0001 00:00:00 +0000

Documents required for Resource Centre registration and certification.

The Resource Centre administrators should read and understand the first five documents. It is a further requirement that the OLA between the NGI and RC, and the Service Operations Policy are accepted by the Site Operations Manager. Further, documents referred within the Service Operations Policy should also be understood.

Documentation

PROC09 Resource Centre Registration and Certification: Procedure document describing the steps required to register new Resource Centres and certify new and suspended ones.
Service Operations Policy: This security policy presents the conditions that apply to anyone running a Service on the Infrastructure, or to anyone providing a Service that is part of the Infrastructure.
Resource Centre Operational Level Agreement (OLA)
PROC11 Resource Centre Decommissioning Procedure: Procedure document describing the steps required to decommission a Resource Centre.
PROC12 Production Service Decommissioning Procedure: Procedure document describing the steps required to decommission a production service.
Helpdesk: A collection of documentation relevant to the EGI Helpdesk (GGUS).

Tools

Configuration Database (GOCDB)
Operations Portal
ARGO Monitoring: RCs status, A/R reports: The ARGO Monitoring service periodically checks the functionality of the production services.
EGI Operations Architecture: Document describing the roles and responsibilities of the various layers in EGI and also the architectural nomenclature.

Providers: HOWTO03 Site Certification GIIS Check

Mon, 01 Jan 0001 00:00:00 +0000

Be sure that Resource Centre GIIS URL is contained in the BDII you use for certification.

Check the consistency of the published information

These are the main branches of the LDAP tree:

GlueSiteUniqueID
GlueSubClusterUniqueID
GlueCEUniqueID
GlueCESEBind
GlueSEUniqueID
GlueServiceUniqueID

It is recommended to use the Apache Studio LDAP browser, although in this page ldapsearch queries are shown.

Contact information

Under the branch GlueSiteUniqueID check the values of the following fields:

GlueSiteName
GlueSiteUserSupportContact
GlueSiteSysAdminContact
GlueSiteSecurityContact
GlueSiteOtherInfo

Example:

$ ldapsearch -x -LLL -H ldap://sibilla.cnaf.infn.it:2170 \
 -b mds-vo-name=INFN-CNAF,o=grid 'objectClass=GlueSite' \
 GlueSiteName GlueSiteUserSupportContact GlueSiteSysAdminContact \
 GlueSiteSecurityContact GlueSiteOtherInfo

dn: GlueSiteUniqueID=INFN-CNAF,Mds-Vo-name=INFN-CNAF,o=grid
GlueSiteSecurityContact: mailto:grid-sec@cnaf.infn.it
GlueSiteSysAdminContact: mailto:grid-operations@lists.cnaf.infn.it
GlueSiteName: INFN-CNAF
GlueSiteOtherInfo: CONFIG=yaim
GlueSiteOtherInfo: EGEE_SERVICE=prod
GlueSiteOtherInfo: EGI_NGI=NGI_IT
GlueSiteOtherInfo: GRID=WLCG
GlueSiteOtherInfo: GRID=EGI
GlueSiteOtherInfo: WLCG_TIER=3
GlueSiteUserSupportContact: mailto:grid-operations@lists.cnaf.infn.it

Under the branch GlueSubClusterUniqueID check the values of the following fields:

Check that the GlueHostApplicationSoftwareRunTimeEnvironment contains a list of software tags supported by the site.
- The list can include VO-specific software tags.
- In order to ensure backwards compatibility it should include the entry ‘LCG-2’, the current middleware version and the list of previous middleware tags (i.e. LCG-2 LCG-2_1_0 LCG-2_1_1 LCG-2_2_0 LCG-2_3_0 LCG-2_3_1 LCG-2_4_0 LCG-2_5_0 LCG-2_6_0 LCG-2_7_0 GLITE-3_0_0 GLITE-3_1_0 GLITE-3_2_0 R-GMA).
GlueHostProcessorOtherDescription (see FAQ HEP SPEC06)
GlueHostOperatingSystemName, GlueHostOperatingSystemVersion and GlueHostOperatingSystemRelease (see publishing the OS name).

Example:

$ ldapsearch -x -LLL -H `<ldap://virgo-ce.roma1.infn.it:2170>` \
 -b mds-vo-name=resource,o=grid 'objectClass=GlueSubCluster' \
 GlueHostProcessorOtherDescription

dn: GlueSubClusterUniqueID=virgo-ce.roma1.infn.it,GlueClusterUniqueID=virgo-ce.roma1.infn.it,Mds-Vo-name=resource,o=grid
GlueHostProcessorOtherDescription: Cores=4, Benchmark=7.83-HEP-SPEC06

Publishing the OS name

It has been decided that the 3 fields

GlueHostOperatingSystemName
GlueHostOperatingSystemRelease
GlueHostOperatingSystemVersion

should be parsed from the output of /usr/bin/lsb_release, like this:

GlueHostOperatingSystemName: lsb_release -i | cut -f2
GlueHostOperatingSystemRelease: lsb_release -r | cut -f2
GlueHostOperatingSystemVersion: lsb_release -c | cut -f2

yielding values like

GlueHostOperatingSystemName: CentOS
GlueHostOperatingSystemRelease: 7.9.2009
GlueHostOperatingSystemVersion: Core

This has been tested on various Linux flavours and should work on every serious GNU/Linux distribution.

Information about the batch system

Under the branch GlueCEUniqueID check the values of the following fields:

GlueCEInfoTotalCPUs: Check that the value is higher than 0.
GlueCEStateWaitingJobs: If there is a “44444”, the information providers are not working properly.
GlueCEInfoLRMSType: any supported batch system (sge, pbs, lsf…)
GlueCEStateStatus: Production, Draining, Queuing or Closed are accepted values.
GlueCEAccessControlBaseRule: VOs enabled on the queue
GlueCECapability

Example:

$ ldapsearch -x -LLL -H ldap://virgo-ce.roma1.infn.it:2170 \
 -b mds-vo-name=INFN-ROMA1-VIRGO,o=grid 'objectClass=GlueCE' \
 GlueCEInfoTotalCPUs GlueCEInfoJobManager GlueCEImplementationName

dn: GlueCEUniqueID=virgo-ce.roma1.infn.it:2119/jobmanager-lcgpbs-theophys,Mds-Vo-name=INFN-ROMA1-VIRGO,o=grid
GlueCEImplementationName: LCG-CE
GlueCEInfoJobManager: lcgpbs
GlueCEInfoTotalCPUs: 8

dn: GlueCEUniqueID=virgo-ce.roma1.infn.it:2119/jobmanager-lcgpbs-cert,Mds-Vo-name=INFN-ROMA1-VIRGO,o=grid
GlueCEImplementationName: LCG-CE
GlueCEInfoJobManager: lcgpbs
GlueCEInfoTotalCPUs: 8

dn: GlueCEUniqueID=virgo-ce.roma1.infn.it:2119/jobmanager-lcgpbs-virgoglong,Mds-Vo-name=INFN-ROMA1-VIRGO,o=grid
GlueCEImplementationName: LCG-CE
GlueCEInfoJobManager: lcgpbs
GlueCEInfoTotalCPUs: 8

dn: GlueCEUniqueID=virgo-ce.roma1.infn.it:2119/jobmanager-lcgpbs-argo,Mds-Vo-name=INFN-ROMA1-VIRGO,o=grid
GlueCEImplementationName: LCG-CE
GlueCEInfoJobManager: lcgpbs
GlueCEInfoTotalCPUs: 8

dn: GlueCEUniqueID=virgo-ce.roma1.infn.it:2119/jobmanager-lcgpbs-virgogshort,Mds-Vo-name=INFN-ROMA1-VIRGO,o=grid
GlueCEImplementationName: LCG-CE
GlueCEInfoJobManager: lcgpbs
GlueCEInfoTotalCPUs: 8

$ ldapsearch -x -LLL -H ldap://cmsrm-bdii.roma1.infn.it:2170 \
 -b mds-vo-name=INFN-ROMA1-CMS,o=grid 'objectclass=GlueCE' GlueCECapability

dn: GlueCEUniqueID=cmsrm-ce01.roma1.infn.it:2119/jobmanager-lcglsf-cmsgcert,Mds-Vo-name=INFN-ROMA1-CMS,o=grid
GlueCECapability: CPUScalingReferenceSI00=1515
GlueCECapability: Share=cms:100

dn: GlueCEUniqueID=cmsrm-ce01.roma1.infn.it:2119/jobmanager-lcglsf-cmsglong,Mds-Vo-name=INFN-ROMA1-CMS,o=grid
GlueCECapability: CPUScalingReferenceSI00=1515
GlueCECapability: Share=cms:100

dn: GlueCEUniqueID=cmsrm-ce01.roma1.infn.it:2119/jobmanager-lcglsf-cmsgshort,Mds-Vo-name=INFN-ROMA1-CMS,o=grid
GlueCECapability: CPUScalingReferenceSI00=1515
GlueCECapability: Share=cms:100

dn: GlueCEUniqueID=cmsrm-ce02.roma1.infn.it:2119/jobmanager-lcglsf-cmsglong,Mds-Vo-name=INFN-ROMA1-CMS,o=grid
GlueCECapability: CPUScalingReferenceSI00=1515
GlueCECapability: Share=cms:100

dn: GlueCEUniqueID=cmsrm-ce02.roma1.infn.it:2119/jobmanager-lcglsf-cmsgcert,Mds-Vo-name=INFN-ROMA1-CMS,o=grid
GlueCECapability: CPUScalingReferenceSI00=1515
GlueCECapability: Share=cms:100

dn: GlueCEUniqueID=cmsrm-ce02.roma1.infn.it:2119/jobmanager-lcglsf-cmsgshort,Mds-Vo-name=INFN-ROMA1-CMS,o=grid
GlueCECapability: CPUScalingReferenceSI00=1515
GlueCECapability: Share=cms:100

Information on Computing Element about Storage Elements

For each SE, on the CEs the following values must be present:

GueCESEBindSEUniqueID.
- GlueCESEBindCEAccesspoint and GlueCESEBindMountInfo.

Example:

$ ldapsearch -x -LLL -H ldap://cremino.cnaf.infn.it:2170 \
 -b mds-vo-name=resource,o=grid 'objectClass=GlueCESEBind' \
 GlueCESEBindSEUniqueID GlueCESEBindCEUniqueID GlueCESEBindMountInfo

dn: GlueCESEBindSEUniqueID=sunstorm.cnaf.infn.it,GlueCESEBindGroupCEUniqueID=cremino.cnaf.infn.it:8443/cream-pbs-cert,Mds-Vo-name=resource,o=grid
GlueCESEBindSEUniqueID: sunstorm.cnaf.infn.it
GlueCESEBindMountInfo: n.a
GlueCESEBindCEUniqueID: cremino.cnaf.infn.it:8443/cream-pbs-cert

dn: GlueCESEBindSEUniqueID=sunstorm.cnaf.infn.it,GlueCESEBindGroupCEUniqueID=cremino.cnaf.infn.it:8443/cream-pbs-prod,Mds-Vo-name=resource,o=grid
GlueCESEBindSEUniqueID: sunstorm.cnaf.infn.it
GlueCESEBindMountInfo: n.a
GlueCESEBindCEUniqueID: cremino.cnaf.infn.it:8443/cream-pbs-prod

Information on Storage Elements

Under the branch GlueSEUniqueID check the values of the following fields:

GlueSALocalID: VO information
GlueSEAccessProtocolLocalID : rfio, srm_v2, gsiftp, gsidcap
GlueSEImplementationName (deprecated)
GlueSEArchitecture
GlueSAStateUsedSpace
GlueSAStateAvailableSpace
GlueSACapability

Example:

$ ldapsearch -x -LLL -H ldap://grid-se.pv.infn.it:2170 \
 -b mds-vo-name=resource,o=grid 'objectclass=GlueSE'

dn: GlueSEUniqueID=grid-se.pv.infn.it,Mds-Vo-name=resource,o=grid
GlueSEImplementationVersion: 1.7.4
GlueSETotalOnlineSize: 8795
GlueSEStatus: Production
objectClass: GlueTop
objectClass: GlueSE
objectClass: GlueKey
objectClass: GlueSchemaVersion
GlueSETotalNearlineSize: 0
GlueSEArchitecture: multidisk
GlueSESizeTotal: 8795
GlueSESizeFree: 5458
GlueSEName: INFN-PAVIA DPM server
GlueSchemaVersionMinor: 3
GlueSEUsedNearlineSize: 0
GlueForeignKey: GlueSiteUniqueID=INFN-PAVIA
GlueSEUsedOnlineSize: 3336
GlueSchemaVersionMajor: 1
GlueSEImplementationName: DPM
GlueSEUniqueID: grid-se.pv.infn.it

$ ldapsearch -x -LLL -H ldap://grid-se.pv.infn.it:2170 \
 -b mds-vo-name=resource,o=grid 'objectclass=GlueSA' \
 GlueSAAccessControlBaseRule GlueSACapability

dn: GlueSALocalID=storage:replica:online,GlueSEUniqueID=grid-se.pv.infn.it,Mds-Vo-name=resource,o=grid
GlueSAAccessControlBaseRule: VO:atlas
GlueSAAccessControlBaseRule: VO:dteam
GlueSAAccessControlBaseRule: VO:infngrid
GlueSAAccessControlBaseRule: VO:ops
GlueSACapability: InstalledOnlineCapacity=8258
GlueSACapability: InstalledNearlineCapacity=0

dn: GlueSALocalID=ATLASHOTDISK:SR:replica:online,GlueSEUniqueID=grid-se.pv.infn.it,Mds-Vo-name=resource,o=grid
GlueSAAccessControlBaseRule: VOMS:/atlas/Role=production
GlueSACapability: InstalledOnlineCapacity=536
GlueSACapability: InstalledNearlineCapacity=0

$ ldapsearch -x -LLL -H ldap://grid-se.pv.infn.it:2170 \
 -b mds-vo-name=resource,o=grid
 '(&(objectclass=GlueSA)(GlueSALocalID=storage:replica:online))' \
 GlueSAReservedNearlineSize GlueSAFreeNearlineSize \
 GlueSATotalNearlineSize GlueSAUsedNearlineSize GlueSACapability \
 GlueSATotalOnlineSize GlueSAFreeOnlineSize \
 GlueSAReservedOnlineSize GlueSAStateAvailableSpace \
 GlueSAUsedOnlineSize GlueSAStateUsedSpace

dn: GlueSALocalID=storage:replica:online,GlueSEUniqueID=grid-se.pv.infn.it,Mds-Vo-name=resource,o=grid
GlueSATotalNearlineSize: 0 GlueSAFreeOnlineSize: 4921
GlueSAUsedNearlineSize: 0 GlueSAFreeNearlineSize: 0
GlueSAReservedNearlineSize: 0 GlueSAStateAvailableSpace: 4921376492
GlueSAReservedOnlineSize: 0 GlueSAUsedOnlineSize: 3336
GlueSAStateUsedSpace: 3336991982 GlueSATotalOnlineSize: 8258
GlueSACapability: InstalledOnlineCapacity=8258
GlueSACapability: InstalledNearlineCapacity=0

Information about other services

There is a branch GlueServiceUniqueID for each service published by the site (WMS, LFC, DPM, GRIDICE, LB, MYPROXY, BDII, etc): what discriminates the services are the values of GlueServiceType, example:

lcg-file-catalog
org.glite.wms.WMProxy
org.glite.lb.Server
srm_v1, SRM

Example:

$ ldapsearch -x -LLL -H ldap://sibilla.cnaf.infn.it:2170 \
 -b mds-vo-name=INFN-CNAF,o=grid 'objectClass=GlueService' \
 GlueServiceType GlueServiceEndpoint GlueServiceName

dn: GlueServiceUniqueID=lfcserver.cnaf.infn.it,Mds-Vo-name=INFN-CNAF,o=grid
GlueServiceEndpoint: lfcserver.cnaf.infn.it
GlueServiceName: INFN-CNAF-lfc
GlueServiceType: lcg-file-catalog

dn: GlueServiceUniqueID=local-lfcserver.cnaf.infn.it,Mds-Vo-name=INFN-CNAF,o=grid
GlueServiceEndpoint: lfcserver.cnaf.infn.it
GlueServiceName: INFN-CNAF-lfc
GlueServiceType: lcg-local-file-catalog

dn: GlueServiceUniqueID=<http://lfcserver.cnaf.infn.it:8085/,Mds-Vo-name=INFN-CNAF,o=grid>
GlueServiceEndpoint: <http://lfcserver.cnaf.infn.it:8085/>
GlueServiceName: INFN-CNAF-lfc-dli
GlueServiceType: data-location-interface

dn: GlueServiceUniqueID=myproxy.cnaf.infn.it_MyProxy_4027652676,Mds-Vo-name=INFN-CNAF,o=grid
GlueServiceEndpoint: myproxy://myproxy.cnaf.infn.it:7512/
GlueServiceName: INFN-CNAF-MyProxy
GlueServiceType: MyProxy

dn: GlueServiceUniqueID=sibilla.cnaf.infn.it_bdii_site_3877936872,Mds-Vo-name=INFN-CNAF,o=grid
GlueServiceEndpoint: <ldap://sibilla.cnaf.infn.it:2170/mds-vo-name=INFN-CNAF,o=grid>
GlueServiceName: INFN-CNAF-bdii_site
GlueServiceType: bdii_site

dn: GlueServiceUniqueID=local-http://lfcserver.cnaf.infn.it:8085/,Mds-Vo-name=INFN-CNAF,o=grid
GlueServiceEndpoint: <http://lfcserver.cnaf.infn.it:8085/>
GlueServiceName: INFN-CNAF-lfc-dli
GlueServiceType: local-data-location-interface

dn: GlueServiceUniqueID=mon-it.cnaf.infn.it_Regional-NAGIOS_2937827985,Mds-Vo-name=INFN-CNAF,o=grid
GlueServiceEndpoint: <https://mon-it.cnaf.infn.it:443/nagios>
GlueServiceName: INFN-CNAF-Regional-NAGIOS
GlueServiceType: Regional-NAGIOS

dn: GlueServiceUniqueID=httpg://sunstorm.cnaf.infn.it:8444/srm/managerv2,Mds-Vo-name=INFN-CNAF,o=grid
GlueServiceEndpoint: httpg://sunstorm.cnaf.infn.it:8444/srm/managerv2
GlueServiceName: INFN-CNAF-SRM
GlueServiceType: SRM

dn: GlueServiceUniqueID=albalonga.cnaf.infn.it_org.glite.lb.server_889826742,Mds-Vo-name=INFN-CNAF,o=grid
GlueServiceEndpoint: <https://albalonga.cnaf.infn.it:9003/>
GlueServiceName: INFN-CNAF-server
GlueServiceType: org.glite.lb.server

dn: GlueServiceUniqueID=gridit-ce-001.cnaf.infn.it_org.edg.gatekeeper_715226072,Mds-Vo-name=INFN-CNAF,o=grid
GlueServiceEndpoint: gram://gridit-ce-001.cnaf.infn.it:2119/
GlueServiceName: INFN-CNAF-gatekeeper
GlueServiceType: org.edg.gatekeeper

dn: GlueServiceUniqueID=egee-wms-01.cnaf.infn.it_org.glite.wms.WMProxy_2200630265,Mds-Vo-name=INFN-CNAF,o=grid
GlueServiceEndpoint: <https://egee-wms-01.cnaf.infn.it:7443/glite_wms_wmproxy_server>
GlueServiceName: INFN-CNAF-WMProxy
GlueServiceType: org.glite.wms.WMProxy

dn: GlueServiceUniqueID=cremino.cnaf.infn.it_org.glite.ce.CREAM_860197007,Mds-Vo-name=INFN-CNAF,o=grid
GlueServiceEndpoint: <https://cremino.cnaf.infn.it:8443/ce-cream/services>
GlueServiceName: INFN-CNAF-CREAM
GlueServiceType: org.glite.ce.CREAM

dn: GlueServiceUniqueID=cremino.cnaf.infn.it_org.glite.ce.Monitor_2670664997,Mds-Vo-name=INFN-CNAF,o=grid
GlueServiceEndpoint: <https://cremino.cnaf.infn.it:8443/ce-monitor/services/CEMonitor>
GlueServiceName: INFN-CNAF-Monitor
GlueServiceType: org.glite.ce.Monitor

dn: GlueServiceUniqueID=top-bdii01.cnaf.infn.it_bdii_top_1813027130,Mds-Vo-name=INFN-CNAF,o=grid
GlueServiceEndpoint: <ldap://egee-bdii.cnaf.infn.it:2170/mds-vo-name=local,o=grid>
GlueServiceName: INFN-CNAF-bdii_top
GlueServiceType: bdii_top

[...]

See the Site Certification Manual tests HOWTO04

See to Resource Centre registration and certification procedure PROC09.

Providers: HOWTO04 Site Certification Manual tests

Mon, 01 Jan 0001 00:00:00 +0000

This page provides instructions on how to test manually the functionality of the grid and cloud services offered by a site. These checks are executed by the EGI Operations Team for sites that want to be formally included into EGI Production infrastructure. The site must successfully pass either the grid or the cloud certification tests to become part of the EGI Production infrastructure.

Check the functionality of the grid elements

Be sure that the site’s GIIS URL is contained in the Top level BDII/Information System your NGI will use for your certification.

Note that the examples here use the Italian NGI and sites. Please substitute with YOUR OWN NGI and site credentials when running the test.

ARC CE checks

A first test can be done using ARC’s ngstat command:

$ /usr/bin/ngstat -q -l -c <CE hostname> -t 20
...
... plenty of output
...

If a monitoring host of your NGI is available, then the probes can easily be executed from there:

Check the status of the CE with:

$ /usr/libexec/grid-monitoring/probes/org.ndgf/ARCCE-status \
 -H <CE hostname> -x /etc/nagios/globus/userproxy.pem-ops

Status is active

Test gsiftp:

$ /usr/libexec/grid-monitoring/probes/org.ndgf/ARCCE-auth -H <CE hostname> \
 -x /etc/nagios/globus/userproxy.pem-ops

gsiftp OK

Test the versions of the CA’s:

$ /usr/libexec/grid-monitoring/probes/org.ndgf/ARCCE-caver -H <CE hostname> \
 -x /etc/nagios/globus/userproxy.pem-ops

version = 1.38 - All CAs present

Check the versions of ARC and Globus:

$ /usr/libexec/grid-monitoring/probes/org.ndgf/ARCCE-softver \
 -H <CE hostname> \
 -x /etc/nagios/globus/userproxy.pem-ops

nordugrid-arc-0.8.3.1, globus-5.0.3

Copy a file:

$ /usr/libexec/grid-monitoring/probes/org.ndgf/ARCCE-gridftp -H <CE hostname> \
 -x /etc/nagios/globus/userproxy.pem-ops

Job finished successfully

Submit a test job:

$ /usr/libexec/grid-monitoring/probes/org.ndgf/ARCCE-jobsubmit \
 -H <CE hostname> \
 --vo ops -x /etc/nagios/globus/userproxy.pem-ops

Job submission successful

Check the LFC:

$ /usr/libexec/grid-monitoring/probes/org.ndgf/ARCCE-lfc -H <CE hostname> \
 -x /etc/nagios/globus/userproxy.pem-ops

Job finished successfully

Check the SRM:

$ /usr/libexec/grid-monitoring/probes/org.ndgf/ARCCE-srm -H <CE hostname> \
 -x /etc/nagios/globus/userproxy.pem-ops

Job finished successfully

Before continuing, you may want to make sure that the probes for all services which the CE intends to offer, do actually succeed.

Storage Element (SE) checks

Check if gridftp server on SE works:

$ uberftp inaf-se-01.ct.pi2s2.it

For STORM SE: check if SRM client works (on the published information you can find the right port to use)

$ /opt/storm/srm-clients/bin/clientSRM \
 ping -e httpg://sunstorm.cnaf.infn.it:8444

============================================================
Sending Ping request to: httpg://sunstorm.cnaf.infn.it:8444
============================================================ Request status:
statusCode="SRM_SUCCESS"(0) explanation="SRM server successfully contacted"
============================================================ SRM Response:
versionInfo="v2.2" otherInfo (size=2) [0] key="backend_type"
[0] value="StoRM" [1] key="backend_version"
[1] value="<FE:1.5.0-1.sl4><BE:1.5.3-4.sl4>"
============================================================

Try to write on SE. Be sure your UI is pointing to an IS the SE is contained in (you may use your certification BDII)

1) Setting a top-bdii that is publishing the SE you have to test

$ export LCG_GFAL_INFOSYS=<TopBDII hostname>:2170

2) Copy a file from the local filesystem to the SE, registering it in the LFC. This command output will return a SURL that you can use latter for other tests.

A SURL is a path of the type: srm://srm01.ncg.ingrid.pt/ibergrid/iber/generated/2011-02-01/file4034a935-8d7a-48f4-914f-16f2634d4802

$ lcg-cr -v --vo <VO>-d<Your SE> \
 -l lfn:/grid/<VO>/test.txt file:</path/to/your/local/file>

3) Create a new replica in other SE (to check the third-party transfer between 2 SEs)

$ lcg-rep -v --vo <VO>-d<Other SE> <SURL>

4) List Replicas

$ lcg-lr -v --vo <VO> lfn:/grid/<VO>/test.txt

5) Delete all replicas

$ lcg-del -v --vo <VO>-a<guid>

Job submission

Submit a test job to Cream-CE through the WMS, i.e. using the glite-wms-job-submit command. In case, submit a mpi test job. The NGI_IT certification WMS is gridit-cert-wms.cnaf.infn.it.

Registration into 1st level HLR

NOTE: this step is needed if your infrastructure uses DGAS as accounting system

After the site entered in production, it needs to register the site resources in the HLR. Ask the site admins to open a ticket towards the HLR administrators, passing them the following information:

grid queues names, in the form:
- gridit-ce-001.cnaf.infn.it:2119/jobmanager-lcgpbs-cert
not-grid queues names, in the form:
- hostname:queue
Name, surname ad certificate subject of each site admin
Certificate subject of Computing Element

Eventually, the site admins have to open a ticket to DGAS support unit asking to enable the forwarding of accounting data from the 2° level HLR to APEL.

Certification Job

The test job checks several things, like the environment on WN and installed RPMs. Moreover it performs some replica management tests. With a grep TEST you may get a summary of the results: in case of errors, you have to see in detail what is gone wrong!

Globus checks

These checks should be executed depending on the services registered in GOCDB under a Resource Centre. Not all services are compulsory for a RC, but upon registration of new ones, the corresponding tests should be executed.

GSISSH

Initialize grid proxy and check if GSISSH works:

$ grid-proxy-init
$ gsissh USER@HOST -p 2222 /bin/date
(Debug with: USER@HOST -vvv -p 2222 /bin/date)

GridFTP

Check if upload works:

$ globus-url-copy file:/tmp/test.txt gsiftp://HOST:2811/tmp/test.txt
(Debug with: globus-url-copy -dbg -v -vb file:/tmp/test.txt gsiftp://HOST:2811/tmp/test.txt)

Check if download works:

$ globus-url-copy gsiftp://HOST:2811/tmp/test.txt file:/tmp/test.txt
(Debug with: globus-url-copy -dbg -v -vb gsiftp://HOST:2811/tmp/test.txt file:/tmp/test.txt)

Delete the remote file:

$ uberftp HOST 'rm /tmp/test.txt'
(Debug with: uberftp HOST 'rm /tmp/test.txt' -debug 3)

GRAM

Check authentication:

$ globusrun -a -r HOST:2119

Check job submission:

$ globusrun -s -r HOST:2119 "&(executable="/bin/date")"

QCG checks

QCG Computing checks

The presented tests of QCG-Computing service use the qcg-comp, the client program for QCG-Computing, that may be installed from provided RPMS. In order to connect to QCG-Computing the grid proxy must be created.

Generate user’s proxy:

$ grid-proxy-init

Your identity: /C=PL/O=GRID/O=PSNC/CN=Jane Doe
Enter GRID pass phrase for this identity:
Creating proxy ............................ Done
Your proxy is valid until: Fri Jun 10 06:23:32 2011

Query the QCG-Computing service:

# the xmllint is used only to present the result in more pleasant way
$ qcg-comp -G | xmllint --format -

<bes-factory:FactoryResourceAttributesDocument xmlns:bes-factory="http://schemas.ggf.org/bes/2006/08/bes-factory">
… a lot of information …   
</bes-factory:FactoryResourceAttributesDocument>

Submit a sample job:

$ qcg-comp -c -J /opt/plgrid/qcg/share/qcg-comp/doc/examples/date.xml

Activity Id: ccb6b04a-887b-4027-633f-412375559d73

Query its status:

$ qcg-comp -s -a ccb6b04a-887b-4027-633f-412375559d73

status = Executing

$ qcg-comp -s -a ccb6b04a-887b-4027-633f-412375559d73

status = Finished
exit status = 0

QCG Notification checks

The tests of QCG-Notification require qcg-ntf-client program to be installed in a system. The program is provided in RPM package.

Create a sample subscription:

$ qcg-ntf-client -d \
 -S "cons=http://127.0.0.1:2212 top=http://schemas.qoscosgrid.org/comp/2011/04/notification/topic;//*;Full"

...
INF May 17 14:15:51 1128 0xa0262720 [qcg-client-gsoa] Subscribed, subRef: '810917963'
...

Remove the created subscription:

$ qcg-ntf-client -d -U "id=810917963"

...
INF May 17 14:41:48 3318 0xa0262720 [qcg-client-gsoa] Unsubscribed: '810917963'
…

Checking the connection with QCG-Computing: In one shell run ‘tail -f’ on the QCG-Computing log file and in the other try to submit a sample job using the qcg-comp program (as described above). Check the tail output if there are no error messages on sending notifications. E.g. the following lines means that the connection problems occurred:

$ tail -f /opt/qcg/var/log/qcg-comp/qcg-compd.log

INF Oct 04 10:55:33 18929 0x2adadc2abe30 [notification_ws] Sending notify: 320f014c-3181-4daf-bbd9-1824b7d8216a -> Queued
NOT Oct 04 10:55:33 18929 0x2adadc2abe30 [.....ntf_client] FaultCode: 'SOAP-ENV:Client'
NOT Oct 04 10:55:33 18929 0x2adadc2abe30 [.....ntf_client] FaultString: 'smcm:ActivityState'
NOT Oct 04 10:55:33 18929 0x2adadc2abe30 [.....ntf_client] FaultDetail: '<SOAP-ENV:Detail xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">connect failed in tcp_connect()</SOAP-ENV:Detail>'
ERR Oct 04 10:55:33 18929 0x2adadc2abe30 [notification_ws] Failed to send notification to http://grass1.man.poznan.pl:19011/

QCG Broker checks

The basic tests of QCG-Broker service may be proceeded with help of qcg-simple-client, the software that provides a set of commands for interaction with QCG-Broker. qcg-simple-client may be installed from RPMs.

Create a sample job description:

$ cat > sleep.qcg << EOF
#!/bin/bash

#QCG queue=plgrid
#QCG host=nova.wcss.wroc.pl
#QCG persistent

sleep 30
EOF

Submit a job:

$ qcg-sub sleep.qcg

https://qcg-broker.man.poznan.pl:8443/qcg/services/
/C=PL/O=GRID/O=PSNC/CN=qcg-broker/qcg-broker.man.poznan.pl
Your identity: C=PL,O=GRID,O=PSNC,CN=Jane Doe
Enter GRID pass phrase for this identity:
Creating proxy, please wait...
Proxy verify OK
Your proxy is valid until Tue Mar 12 14:50:27 CET 2013
UserDN = /C=PL/O=GRID/O=PSNC/CN=Jane Doe
ProxyLifetime = 24 Days 23 Hours 59 Minutes 58 Seconds

jobId = J1360936230540__0152

Check the job statuses:

$ qcg-info

https://qcg-broker.man.poznan.pl:8443/qcg/services/
/C=PL/O=GRID/O=PSNC/CN=qcg-broker/qcg-broker.man.poznan.pl
UserDN = /C=PL/O=GRID/O=PSNC/CN=Jane Doe
ProxyLifetime = 24 Days 23 Hours 59 Minutes 49 Seconds

Command translated to:
"task_info" "J1360936230540__0152" "task"
Note:
UserDN: /C=PL/O=GRID/O=PSNC/CN=Jane Doe
TaskType: SINGLE
SubmissionTime: Fri Feb 15 14:50:31 CET 2013
FinishTime:
ProxyLifetime: PT0S
Status: PREPROCESSING
StatusDesc:
StartTime: Fri Feb 15 14:50:33 CET 2013

Allocation:
HostName: nova.wcss.wroc.pl
ProcessesCount: 1
ProcessesGroupId:
Status: PREPROCESSING
StatusDescription:
SubmissionTime: Fri Feb 15 14:50:32 CET 2013
FinishTime:
LocalSubmissionTime: Fri Feb 15 14:50:37 CET 2013
LocalStartTime:
LocalFinishTime:

$ qcg-info

https://qcg-broker.man.poznan.pl:8443/qcg/services/
/C=PL/O=GRID/O=PSNC/CN=qcg-broker/qcg-broker.man.poznan.pl
UserDN = /C=PL/O=GRID/O=PSNC/CN=Jane Doe
ProxyLifetime = 24 Days 23 Hours 59 Minutes 23 Seconds

Command translated to:
"task_info" "J1360936230540__0152" "task"
Note:
UserDN: /C=PL/O=GRID/O=PSNC/CN=Jane Doe
TaskType: SINGLE
SubmissionTime: Fri Feb 15 14:50:31 CET 2013
FinishTime:
ProxyLifetime: PT0S
Status: RUNNING
StatusDesc:
StartTime: Fri Feb 15 14:50:33 CET 2013

Allocation:
HostName: nova.wcss.wroc.pl
ProcessesCount: 1
ProcessesGroupId:
Status: RUNNING
StatusDescription:
SubmissionTime: Fri Feb 15 14:50:32 CET 2013
FinishTime:
LocalSubmissionTime: Fri Feb 15 14:50:37 CET 2013
LocalStartTime: Fri Feb 15 14:50:47 CET 2013
LocalFinishTime:
$ qcg-info
https://qcg-broker.man.poznan.pl:8443/qcg/services/
/C=PL/O=GRID/O=PSNC/CN=qcg-broker/qcg-broker.man.poznan.pl
UserDN = /C=PL/O=GRID/O=PSNC/CN=Jane Doe
ProxyLifetime = 24 Days 23 Hours 56 Minutes 10 Seconds

Command translated to:
"task_info" "J1360936230540__0152" "task"
Note:
UserDN: /C=PL/O=GRID/O=PSNC/CN=Jane Doe
TaskType: SINGLE
SubmissionTime: Fri Feb 15 14:50:31 CET 2013
FinishTime: Fri Feb 15 14:52:17 CET 2013
ProxyLifetime: PT0S
Status: FINISHED
StatusDesc:
StartTime: Fri Feb 15 14:50:33 CET 2013

Allocation:
HostName: nova.wcss.wroc.pl
ProcessesCount: 1
ProcessesGroupId:
Status: FINISHED
StatusDescription:
SubmissionTime: Fri Feb 15 14:50:32 CET 2013
FinishTime: Fri Feb 15 14:52:12 CET 2013
LocalSubmissionTime: Fri Feb 15 14:50:37 CET 2013
LocalStartTime: Fri Feb 15 14:50:47 CET 2013
LocalFinishTime: Fri Feb 15 14:52:09 CET 2013

Check the functionality of the cloud elements

Sites can provide any (not necessarily all) of the interfaces listed below:

OpenStack Compute for VM Management
CDMI for Object Storage

Cloud Compute checks prerequisites

AppDB integration

Go to AppDB and look for a OS image member of the fedcloud.egi.eu VO (all sites should support), e.g. EGI CentOS 7 image

Check that the site is visible into the AppDB “Availability and Usage” panel for the image. If not, probably the site has not registered the FedCloud VO into their middleware (vmcatcher) or it did not properly configured the BDII provider script.

From that “Availability and Usage” panel, click on the Site name, then on the latest VM Image version, select a resource template (preferably with the smallest quantity of resources (RAM & CPU)) and click on the “get IDs” button on the right of the resource template. You will get the “Site Endpoint”, “Template ID” and “OCCI ID”. Save these values since they will be needed in the next steps.

Credentials

Generate a set of keys for your user (it is not required to set a passphrase for the keys, since these are just temporary keys for the test), make sure to set key permissions to 400:

$ ssh-keygen -t rsa -b 2048 -f tempkey

Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in tempkey.
Your public key has been saved in tempkey.pub.
The key fingerprint is:
(...)

$ chmod 400 tempkey

Create a simple contextualization script, to setup access keys on the machine and test contextualization

$ cat << EOF > ctx.txt
Content-Type: multipart/mixed; boundary="===============4393449873403893838=="
MIME-Version: 1.0

--===============4393449873403893838==
Content-Type: text/x-shellscript; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="deploy.sh"

#!/bin/bash
echo "OK" > /tmp/deployment.log

--===============4393449873403893838==
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="userdata.txt"

#cloud-config
users:
- name: testadm
 sudo: ALL=(ALL) NOPASSWD:ALL
 lock-passwd: true
 ssh-import-id: testadm
 ssh-authorized-keys:
 - `cat tempkey.pub`

--===============4393449873403893838==--
EOF

Create a proxy in RFC format for your tests:

$ voms-proxy-init -rfc -voms fedcloud.egi.eu

Your identity: /DC=es/DC=irisgrid/O=ifca/CN=Enol-Fernandez-delCastillo
Creating temporary proxy .......................................................................... Done
Contacting voms2.grid.cesnet.cz:15002 [/DC=org/DC=terena/DC=tcs/OU=Domain Control Validated/CN=voms2.grid.cesnet.cz] "fedcloud.egi.eu" Done
Creating proxy .............................................................................. Done

Your proxy is valid until Fri Nov 14 04:59:26 2014

OpenStack Compute checks (org.openstack.nova service type)

Export the following variables on your shell (keystone URL can be obtained from GOCDB URL of the endpoint)

$ export OS_AUTH_URL= <keystone URL>
$ export OS_AUTH_TYPE=v2voms
$ export OS_X509_USER_PROXY=$X509_USER_PROXY

Get the list of tenants supporting your proxy

$ keystone_tenants

Tenant id: 999f045cb1ff4684a15ebb338af69460
Tenant name: VO:fedcloud.egi.eu Enabled: True
Description: VO fedcloud.egi.eu

Export the tenant name as shown from the command in the following variable:

$ export OS_PROJECT_NAME="VO:fedcloud.egi.eu"

Describe the available flavors, check also that the template ID provided by AppDB is listed:

$ openstack flavor list

+----+-----------+-------+------+-----------+-------+-----------+
| ID | Name | RAM | Disk | Ephemeral | VCPUs | Is Public |
+----+-----------+-------+------+-----------+-------+-----------+
| 1 | m1.tiny | 512 | 0 | 0 | 1 | True |
| 2 | m1.small | 2000 | 10 | 20 | 1 | True |
| 3 | m1.medium | 4000 | 10 | 40 | 2 | True |
| 4 | m1.large | 7000 | 20 | 80 | 4 | True |
| 5 | m1.xlarge | 14000 | 30 | 160 | 8 | True |
+----+-----------+-------+------+-----------+-------+-----------+

Describe available images, again check that the ID provided by AppDb is listed

$ openstack image list

+--------------------------------------+----------------------------------------------+
| ID | Name |
+--------------------------------------+----------------------------------------------+
| 1414f242-d6d1-4a8c-8b26-8c0ada32f343 | IFCA Fedora Cloud 23 |
| 8a700834-04b4-4e91-a3d5-9246ef95167e | LW Jupyter-R Ubuntu 14.04 |
| 7f361fba-21d6-40ca-892d-17aa60b63a66 | IFCA CentOS 7 |
| f3544cc8-421f-4d93-ac35-eba7fdc75329 | IFCA CentOS 6 |
...
+--------------------------------------+----------------------------------------------+

Start a VM (using the flavor and images checked above and the context file created previously). The returned ID will be used in the following commands:

$ openstack server create \
 --flavor <flavor> --image <image id> \
 --user-data ctx.txt test

+--------------------------------------+------------------------------------------------------+
| Field | Value |
+--------------------------------------+------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | None |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | |
| config_drive | |
| created | 2016-03-01T13:07:10Z |
| flavor | m1.tiny (1) |
| hostId | |
| id | 5d3ed7d6-d5ac-4f09-a353-0c2bd0fbd0ea |
| image | IFCA CentOS 7 (7f361fba-21d6-40ca-892d-17aa60b63a66) |
| key_name | None |
| name | test |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| project_id | 999f045cb1ff4684a15ebb338af69460 |
| properties | |
| security_groups | [{u'name': u'default'}] |
| status | BUILD |
| updated | 2016-03-01T13:07:11Z |
| user_id | a31b8c452b594369a49a8329103e241a |
+--------------------------------------+------------------------------------------------------+

Check that the VM is active by describing it (it may take a few minutes):

$ openstack server show <vm id>

+---------------------+--------+
| Field | Value |
+---------------------+--------+
(...)
| OS-EXT-STS:vm_state | active |
(...)
+---------------------+--------+

If the VM does not have a public IP, you will need to get an IP for it

$ openstack ip floating pool list

+-------------+
| Name |
+-------------+
| nova |
+-------------+

$ openstack ip floating pool create <pool name>

+-------------+----------------+
| Field | Value |
+-------------+----------------+
| fixed_ip | None |
| id | 1265 |
| instance_id | None |
| ip | 193.146.75.245 |
| pool | nova |
+-------------+----------------+

$ openstack ip floating add <ip> <vm id>

The VM should have now a public IP available when shown:

$ openstack server show <vm id>
+-----------+-------------------------------------+
| Field | Value |
+-----------+-------------------------------------+
(...)
| addresses | private=172.16.8.14, 193.146.75.245 |
(...)
+-----------+-------------------------------------+

ssh to the machine to the provided IP (the options avoid problems when different VMs have the same IP, don’t use them in production) and check that contextualization script was executed:

$ ssh -i tempkey -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
 testadm@ <ip> "cat /tmp/deployment.log"

Warning: Permanently added '193.146.75.245' (ECDSA) to the list of known hosts.

OK

Create a storage volume:

$ openstack volume create --size 1 test

+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| created_at | 2016-03-01T13:29:44.392423 |
| display_description | None |
| display_name | test |
| id | 8f46046d-cd9b-4219-9ce7-f0abe30ad992 |
| properties | |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| type | None |
+---------------------+--------------------------------------+

And show its status:

$ openstack volume show <vol id>

Attach it to the running VM:

$ openstack server add volume <vm id> <vol id>

And check it’s attached:

$ openstack volume show 8f46046d-cd9b-4219-9ce7-f0abe30ad992

+---------------------+---------------------------------------------------------+
| Field | Value |
+---------------------+---------------------------------------------------------+
| attachments | [{u'device': u'/dev/vdb', |
| | u'server_id': u'21f6123f-926c-4816-b4fb-53df91907a63', |
| | u'id': u'8f46046d-cd9b-4219-9ce7-f0abe30ad992', |
| | u'volume_id': u'8f46046d-cd9b-4219-9ce7-f0abe30ad992'}] |
| availability_zone | nova |
| bootable | false |
| created_at | 2016-03-01T13:29:44.000000 |
| display_description | None |
| display_name | test |
| id | 8f46046d-cd9b-4219-9ce7-f0abe30ad992 |
| properties | |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | in-use |
| type | None |
+---------------------+---------------------------------------------------------+

And login into the machine to create a filesystem, mount it, create a file, and umount:

$ ssh -i tempkey -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
 testadm@ <ip_addr>

testadm $ sudo mke2fs <device_id>
(...)

testadm $ sudo mount <device_id> /mnt
testadm $ touch /mnt/test

testadm $ ls -l /mnt/

total 16 drwx------ 2 root root 16384 Nov 13 17:43 lost+found
-rw-r--r-- 1 root root 0 Nov 13 17:45 test

testadm $ sudo umount /mnt
testadm $ exit

Delete the VM:

$ openstack server delete <vm id>

And create a new one with the volume attached directly (substitute vdb for the same device name shown above):

$ openstack server create \
 --flavor <flavor> --image <image id> \
 --block-device-mapping vdb= <volume id> \
 --user-data ctx.txt test

+-----------------------------+---------------------------------------------+
| Field | Value |
+-----------------------------+---------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| accessIPv4 | |
| accessIPv6 | |
| addresses | |
| adminPass | veryverysecret |
| config_drive | |
| created | 2016-03-01T13:45:21Z |
| flavor | m1.tiny (1) |
| hostId | |
| id | 10000d50-239b-4e86-bbd8-224143d6d346 |
| image | Image for EGI Centos 6 [CentOS/6/KVM]_egi |
| key_name | None |
| name | test |
| progress | 0 |
| project_id | fffd98393bae4bf0acf66237c8f292ad |
| properties | |
| security_groups | [{u'name': u'default'}] |
| status | BUILD |
| updated | 2016-03-01T13:45:23Z |
| user_id | 6c254b295af64644904a813db0d3d88a |
+-----------------------------+---------------------------------------------+

Assign the public IP (if it does not have one already):

$ openstack ip floating add <ip> <vm id>

And check that the volume is attached and usable:

$ ssh -i tempkey -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
 testadm@ <new_vm_ip_addr>

testadm $ sudo mount <device_id> /mnt
testadm $ ls -ltr /mnt/

total 16
drwx------ 2 root root 16384 Nov 13 17:43 lost+found
-rw-r--r-- 1 root root 0 Nov 13 17:45 test

testadm $ sudo umount /mnt
testadm $ exit

Finally delete VM, volume and IP address (NOTE: use the id of the IP as returned
by openstack ip floating list\!):

$ openstack server delete <vm id>
$ openstack volume delete <volume id>
$ openstack ip floating delete <ip id>

Cloud Storage (CDMI) checks (eu.egi.cloud.storage-management.cdmi service type)

List the content of the repository:

$ bcdmi -e <cdmi_endpoint> list /

Create a test directory:

$ bcdmi -e <cdmi_endpoint> mkdir test
{
 "completionStatus": "Complete",
 "objectName": "test/",
 "capabilitiesURI": "/cdmi/AUTH_df37f5b1ebc94604964c2854b9c0551f/cdmi_capabilities/container/",
 "parentURI": "/cdmi/AUTH_df37f5b1ebc94604964c2854b9c0551f/",
 "objectType": "application/cdmi-container",
 "metadata": {}
}

Create a test file and upload it to the created directory:

$ echo "TEST OK" > testfile
$ bcdmi -e <cdmi_endpoint> put -T testfile
test/test.txt**

Try to download back the file and compare to previous:

$ bcdmi -e <cdmi_endpoint> get test/test.txt -o testfile.downloaded
$ diff testfile testfile.downloaded && echo "Files are equal" \
 || echo "Files differ"

Files are equal

Delete the file:

$ bcdmi -e <cdmi_endpoint> delete test/test.txt

Check that the file is not present anymore

$ bcdmi -e <cdmi_endpoint> list test/

Upload the file again and recursively delete the directory:

$ bcdmi -e <cdmi_endpoint> put -T testfile test/test.txt
$ bcdmi -e <cdmi_endpoint> delete -r test/

Check that the folder does not exist anymore

$ bcdmi -e <cdmi_endpoint> list /

See Site Certification GIIS Check HOWTO03.

See to Resource Centre registration and certification procedure PROC09.

Providers: FAQ HEP SPEC 06

Mon, 01 Jan 0001 00:00:00 +0000

Transition to HEP SPEC, a new CPU benchmark

Q1: Why adopting HEP SPEC 06?

The traditional si2k CPU benchmark is now obsolete and it is time to move to HEP SPEC, a new CPU benchmark that will replace si2k and will become the reference benchmark for accounting purposes.

Detailed description of the reasons are provided on the transition to a new CPU benchmarking unit for the WLCG.

Q2: What is HEP SPEC 06?

The HEP-SPEC06 benchmark is designed to scale with the performances of the high-energy physics codes on similar machines. The goal was to have an accuracy of ± 5% but for the moment the agreement is significantly higher.

The measurement of HEP-SPEC repeated on identical machines varies less than 1%. If the computing machines are similar, i.e. same processors and at least 2 GB per core, the results obtained are very close, within some percent, so that it is unnecessary to perform measures on all computing hosts. It is enough to do the measurement for one type of processor and consider it valid for all the machines with the same processor.

If you are using different OS and specially different compilers, the data will change.

Q3: Where can I find information about HEP SPEC 06 measurements?

Some example results are available on the HEPIX group-page, where one can see the differences between gcc3.4.x and gcc4.1.x.

Additional results tables are available from various EGI partners:

GRIDPP

If you don’t find your computing machine in that table, then it is better to try to do the measurement because extrapolating the results increases further the error.

Q4: How can I run the HEP SPEC 06 benchmark?

If you want to make HEP-SPEC06 on your own own, detailed instructions are available at CERN wiki.

In Short you need the following:

A machine with any version of Linux compatible with Scientific Linux (RHEL, SL, SLC, CentOS)
The gcc compiler should be installed
Configuration files and run script (available as a gzipped tar archive from the CERN Wiki). The archive’s md5sum is 9fed92b8d515b88904705f76809c4028
A tar ball of the SPECcpu2006 DVD called SPEC2006_v11.tar.bz2 that should be in the same directory as the run script

Q5: My site already adopted HEP SPEC 06. Do I still need to publish SpecInt2000?

The transition to HEP-SPEC does not eliminate the need to publish the computing power in SpecInt2000 (due to backward compatibility with sites not publishing yet HEP-SPEC). In this case you may calculate the value SpecInt2000 starting from HEP-SPEC through the following relation:

value_kSI2K = value_HEP-SPEC / 4 (or value_HEP-SPEC = 4 * value_kSI2K)

For the GlueHostBenchmarkSI00 attribute in the GLUE v1.3 schema the following relation is easier to use:

value_SI00 = value_HEP-SPEC * 250 rounded to the nearest integer

Q6: How are HEP SPEC 06 results set in YAIM?

The YAIM variable CE_OTHERDESCR is used to set the GlueHostProcessorOtherDescription attribute. The value of this variable MUST be defined in your site-info.def file as:

Cores=<CE_LOGCPU/CE_PHYSICALCPU> [, Benchmark=<value>-HEP-SPEC06]

where the ratio CE_LOGCPU / CE_PHYSICALCPU means the average number of cores per physical CPU in a sub-cluster; in the case of (slightly) heterogeneous sub-clusters it could be non-integer. The second value of this attribute MUST be published only in the case the CPU power of the sub-cluster has been computed using the HEP-SPEC06 benchmark. The Benchmark value must be the average HEP_SPEC06 result per core, in the sub-cluster.

These variables are set in your site-info.def file. After this, the variables need to be published by the CE’s resource BDII, configured e.g. by standard YAIM commands.

The total CPU capacity of the cluster is computed as Benchmark * CE_LOGCPU.

Documentation – Operations manuals and How-Tos

Providers: Operations Start Guide

Introduction

Resource Centres and Resource Infrastructures

Operations Centres

Roles

Resource Centre level

Resource Centre Administrator

Resource Centre Operations Manager

Resource Centre Security Officer

Regional level

Regional Operator on Duty (ROD)

NGI Security officer

NGI Operations manager

Global level

Chief Operations Officer

EGI-CSIRT

EGI-CSIRT’s IRTF

EGI Security Officer

EGI Foundation SDIS team

VO

VO manager

Joining operations

Authentication

Obtaining a X509 personal certificate

Create a federated identity: registration in EGI Check-in

Joining dteam VO

Requesting GOCDB access

Registering into GGUS

Subscribing to mailing lists

Documentation

Tools

Providers: Resource Centre Integration Check List

Providers: MAN01 How to publish site information

Document control

Configuring a site BDII

Service-related documentation

Federated Cloud BDII configuration

GlueSite Object

Guidelines for GlueSite Object

Example

Distributed Tier1s and Tier2s

Established Grid Name

Valid WLCG Names

Valid EGI NGI Names

YAIM Instructions

Check your own GlueSite Object

Site information in GLUE 2

Providers: MAN02 Service intervention management

Document control

Service Intervention

How to manage an intervention

Scheduled interventions

Unscheduled interventions

Required information

Recommendations

Notifications

Suspension policy

Providers: MAN04 Tool Intervention Management

Scope

Announcements

Procedure

Case 1: short “undetected” downtime

Case 2: long “detected” outage

1. Outage

2. Extended downtime

3. Recovery

4. Post mortem analysis

Providers: MAN05 top and site BDII High Availability

Document control

Top BDII and Site BDII with High Availability

Requirements to deploy a top or site BDII service

Hardware

Co-hosting

Physical vs Virtual Machines

Best practices from a client perspective for top BDII

Best practices for a top or site BDII High Availability service

DNS round robin load balancing

Fault tolerance DNS Updater

Implementation