Documentation – Helpdesk features

Internal: Issue type values

Mon, 01 Jan 0001 00:00:00 +0000

USER tickets

// jscpd:ignore-start

For generic user tickets selectable issue types are explained in the table below.

Issue type	Description
Accounting	All kind of issues related to accounting tools (APEL, Accounting Portal and so on).
AppDB	All kind of issues related to the AppDB or to the interactions with the AppDB.
Authorization/Authentication	All issues related to authorization and authentication e.g. certificates
COD Operations	WLCG Coordinator On Duty Operations
Catalogue	Issues related to file catalogues, like LFC
Computing Services	All issues related to execution of jobs. It combines two other type of problems (‘Workload management’ and ‘Local batch system’). The ‘Computing Services’ should be used when it is not clear to which subcategory the ticket refersConfiguration Issues of system configuration.
Data Management - generic	All kind of issues related to data management tools (GFAL, LCG_Util and so on).
Databases	Issues related to Databases.
Deployment - other	Issues of software deployment that do not fit any other category.
Documentation	Issues of missing, wrong, outdated documentation.
File Access	File access issues.
File Transfer	File transfer issues, e.g. related to FTS
GGUS	Bugs and feature requests related to GGUS system.
IaaS Operations	Issues related to VMs and clients (OCCI, rOCCI, etc.), block storage, networking.
Information System	Issues related to the BDII.
Installation	Installation issues.
Local Batch System	Issues related to the local batch systems.
Middleware	Issues of middleware stacks like gLite, globus and others.
Monitoring	Issues of infrastructure and service monitoring.
Network problem	Issues of network connectivity.
Operations	Issues of general processes, procedures, information and so on of the entire infrastructure.
Other	Requests that do not match to any other issue type.
Security	Issues of infrastructure and operation security.
Storage Systems	Issues concerning storage systems, like EOS, dCache, DPM
VO Specific Software	Issues of VO specific software tools and packages.
Virtual Appliance Management	Issues related to vmcatcher/vmcaster tools, and VA management in general.
Workload Management	Issues related to workload management system and tools.

TEAM tickets

Team and alarm tickets have a reduced number of issue types in the drop-down menu.

Issue type	Description
Databases	Issues related to Databases.
File Access	File access issues.
File Transfer	File transfer issues, e.g. related to FTS
Local Batch System	Issues related to the local batch systems.
Middleware	Issues of middleware stacks like gLite, globus and others.
Monitoring	Issues of infrastructure and service monitoring.
Network problem	Issues of network connectivity.
Other	Requests that do not match to any other issue type
Storage Systems	Issues related to storage systems, like EOS, dCache or DPM

ALARM tickets

Issue type	Description
Databases	Issues related to Databases.
File Access	File access issues.
File Transfer	File transfer issues, e.g. related to FTS
Local Batch System	Issues related to the local batch systems.
Middleware	Issues of middleware stacks like gLite, globus and others.
Monitoring	Issues of infrastructure and service monitoring.
Network problem	Issues of network connectivity.
Other	Requests that do not match to any other issue type
Storage Systems	Issues related to storage systems, like EOS, dCache or DPM

CMS tickets

The CMS VO has an own ticket submit form in the GGUS system. Although this form provides CMS specific issue types no special privilege is required to use it. This does not apply for the TEAM ticket submit form: Here the CMS specific issue types are only selectable for registered CMS TEAM members.

Issue type	Description
CMS_AAA WAN Access	Issues around WAN
CMS_CAF Operations	Issues around CAF operations at CERN, incl EOS space requests
CMS_Central Workflows	Issues around centrally managed MC production and processing
CMS_Data Transfers	Issues around data transfers, e.g. via Phedex or ASO
CMS_Facilities	Typically issues at CMS sites
CMS_HammerCloud	Issues around CMS HammerCloud
CMS_Register New CMS Site	Chosen when a new site gets registered with CMS
CMS_SAM tests	Issues around CMS SAM tests
CMS_Submission Infrastructure	Issues around CMS_Submission Infrastructure
CMS_Tier-1 Tape Families	For creation of Tape families/groups at Tier-1 archives

ATLAS tickets

The ATLAS specific issue types are only selectable for registered ATLAS TEAM members.

Issue type	Description
ATLAS_ADC Central	Issues around ADC in general
ATLAS_Databases	Issues around databases
ATLAS_Deletion	Issues around file deletions
ATLAS_File Access/Transfer	Issues around file access or file transfers
ATLAS_Frontier-Squid	Issues around frontier or squid
ATLAS_Local Batch System	Issues around batch systems
ATLAS_Middleware	Issues related to middleware
ATLAS_Monitoring	Issues around monitoring
ATLAS_Network Problem	Issues with network infrastructure
ATLAS_Staging	Issues around file staging
ATLAS_Storage Systems	Issues around storage

// jscpd:ignore-end

Internal: 'SITES' field and tickets to multiple sites

Mon, 01 Jan 0001 00:00:00 +0000

SITES field

The list of sites included in the drop-down menu of the “SITES” field is taken from GOCDB (EGI sites) and from OIM DB (OSG sites).
- Sites registered in OIM are only visible if they have certification status “enabled”.
- Sites registered in GOC DB are only visible if they have certification status “certified”.
- Sites in other statuses (e.g. “suspended”) are not visible in the “SITES” drop-down list.
the sites information is synchronized once per night.
In case a site’s status changes, the site disappears from the “SITES” drop-down list. Existing GGUS tickets related to this site get the “SITES” field flushed. The NGI to which the site belongs inherits the ticket from the site and is in charge of further processing this ticket.

Tickets to multiple sites option: the “Multisite” ticket

The Multisite ticket is a feature for submitting tickets with the same topic against an unlimited number of sites in GGUS. After the submission, the ticket is cloned and sent as separate ticket to each of the sites selected during the ticket creation.
- All the sites displayed in GGUS can be notified using this option.
This feature can be used only by users owning the “Multisites” role.
There is a specific submit form linked from GGUS ticket submit area. In the “Notify SITE” drop-down menu sites can be checked for receiving a ticket.

Creation of a Multisite ticket

When the users with the “Multisites” role click on the “+” button to create a ticket, they will see the option “New Multisite Ticket”.

By clicking on that option, they will end up in an interface allowing the submission to multiple sites.

In addition to the usual fields, the users can also select proper values for:

SITE TYPE: EGI or OSG
TIER LEVELS
NGIs

Then the “NOTIFIED SITES” field will be filled automatically based on the selections made for the aforementioned fields (it is also possible to further edit manually the values of this field).

When all the necessary fields of the ticket creation form have been filled in properly, the users can click on the “Create” button to submit the tickets.

At this point, a notification email for each submitted ticket is sent to the submitter (to be changed in the future to a single email containing all the created tickets in order to reduce the spam to the submitter). At the same time, all the selected sites in the list are notified and the tickets are assigned to the parent NGIs.

The list of submitted tickets is available through the link “Search related tickets”:

Internal: Alarm tickets

Mon, 01 Jan 0001 00:00:00 +0000

Introduction: purpose and conditions

The purpose of ALARM tickets is to notify WLCG Tier-0 and Tier-1 administrators about serious problems of the site at any time, independent from usual office hours.
Only experts, nominated by the WLCG VOs are allowed to submit alarm tickets. They need to have the appropriate permissions in GGUS user database.
The involved VOs are:
- Alice
- Atlas
- Cms
- Lhcb
Only the Tier-0 & Tier-1 sites are involved in the alarm tickets process. The WLCG Tier-0/Tier-1 site names can be used as well as the relevant GOC DB site names.
Alarm tickets are routed to the NGI/ROC the tier site belongs to automatically. They do not need a routing by the TPM. The NGI/ROC is notified about the ticket in the usual way. In parallel the site receives an alarm email signed with a GGUS certificate. This alarm email is processed at the Tier-0/Tier-1 and notifies the relevant people at any time.
Alarm email addresses are taken from GOC DB for the EGI sites and from OIM for the OSG sites. VOMS is used by GGUS as the information source for authorised alarmers.

Becoming an Alarm-ticket member

People who want to become an alarmer have to

register in GGUS first
be added to the appropriate group in a VOMS server.

GGUS system synchronizes its user database once per night with the VOMS servers. The synchronization is based on the DN string. Please make sure the DN of your GGUS account is the same than the one registered in VOMS.

Tier-1 site admins can become alarmers for testing the alarm process for their site. They have to fill in the registration form for supporters. Please add an appropriate comment (e.g. “I’m a xxx tier-1 site admins and I want to become an alarmer for testing purposes.”) in the registration form.

Technical description

This section describes the workflows of alarm tickets from a technical point of view.

ALARM ticket submission

Alarm tickets can be submitted using the GGUS web portal. On top of the ticket submit form in GGUS web portal there is a link to the submit form for alarm tickets.

As alarm ticket submitters are experts who will hopefully provide all necessary information, the number of fields on the alarm ticket submit form is reduced to a minimum compared to the number of fields on the user ticket submit form.

Three fields on this form are mandatory:

Subject
MoU Area
Notified Site

All other fields are optional.

ALARM ticket processing

The processing of a team ticket consists of two main parts: the notification of the notified site and the routing of the ticket to the NGI/ROC the site belongs to.

Tier-1 site notification

In parallel to the creation of an alarm ticket, the GGUS system sends an alarm email directly to the tier 1 site specified in field “Notify SITE”. This email is sent to a specific site alarm mail address and signed with the GGUS certificate. The tier-0 alarm mail address is based on the VO name. It is “voname”-operator-alarm"atnospam"cern.ch. Tier-1 site alarm mail addresses are taken from the “Emergency Email” field in GOC DB. For tier-1 sites registered in the OSG OIM DB the alarm email address is taken from field “SMSAddress” of the “Administrative Contact” in OIM DB. The DN of the GGUS certificate is /C=DE/O=GermanGrid/OU=KIT/CN=ggusmail/ggus.eu. The alarm mail looks like the following figure.

Ticket routing

Alarm tickets are bypassing the TPMs and routed to the appropriate NGI/ROC automatically. The decision to which NGI/ROC a ticket has to be routed is done automatically, based on the value of the “Notify SITE” field. The “Notify SITE” drop-down menu shows both the tier 1 site names from GOC DB and the tier 1 site names used by WLCG.

ALARM confirmation

Once the tier 1 site received the alarm email the receipt should be confirmed. Sending a reply mail containing the typical GGUS identifier and the ticket ID “GGUS-Ticket-ID: #00000000” in the subject is sufficient. Such a reply will be added to the alarm ticket.

Working on ALARM tickets

For working on alarm tickets and resolving them please use the GGUS portal. A reference link to the alarm ticket is given in the alarm email notification.

Periodic ALARM ticket testing rules

Alarm ticket testing is documented in WLCG twiki.

Schema of the Alarm tickets process

Internal: Quality of Support (QoS) levels

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

QoS stands for Quality of Support. It describes the level of support provided by Support Units in GGUS system.

It has an impact on the ticket priority colour in GGUS and the warnings are sent to SUs if 75% of the maximum response time of the QoS level are over.

QoS levels

There are three different QoS levels, each defining different response times for given Ticket Priority.

Base
Medium
Advanced

The default QoS level, if not declared differently, is Base.

Base Level

Base QoS level defines a response time of 5 working days regardless of the ticket priority.

Medium Level

Ticket Priority	Response time
less urgent	5 working days
urgent	5 working days
very urgent	1 working day
top priority	1 working day

Advanced service

Ticket Priority	Response time
less urgent	5 working days
urgent	1 working day
very urgent	1 working day
top priority	4 working hours

QoS level declaration

The QoS level for all of the SUs are available here

For the several EGI services, the QoS levels are defined in the specific Operational Level Agreements linked in the EGI OLA SLA framework page

Internal: Report generator

Mon, 01 Jan 0001 00:00:00 +0000

Definitions and prerequisites

The GGUS Report Generator is available from the support section.

The implementation of the report generator started in October 2011. Hence, the report generator does not provide data for tickets submitted before December 2011!

Timestamps and metrics

submit timestamp: timestamp when the ticket is submitted
assign timestamp: timestamp when a ticket gets assigned to a support unit
response timestamp: timestamp when the ticket status changes from “assigned” to any other status value or the ticket gets re-assigned to another support unit
expected response timestamp: timestamp when a ticket should have changed the status to any other status value than “assigned” or be re-assigned to another support unit at latest
solution timestamp: timestamp when the status changes to either “solved” or “unsolved”

Response time

The response time is a performance figure calculated from the support unit’s point of view. It describes how quick a support unit is reacting on tickets. Response time is the time from

either assigning a ticket to a support unit and the support unit is kept until the first status changes to any other value than “assigned”, or
assigning a ticket to a support unit and the status value “assigned” is kept until the support unit changes to any other support unit (re-assign).

The response time is calculated as difference between the timestamp changing the status or re-assigning the ticket and the assign timestamp. While assigning a ticket to a support unit the expected response timestamp is calculated by adding an amount of time to the assign timestamp. The amount of time added depends on the ticket priority and the kind of support unit. For support units that have declared a quality of service level the response times are defined by the QoS level. For all the other support units a medium QoS is assumed for calculating the expected response timestamp. In case the actual response timestamp is greater than the “expected response” timestamp for middleware support units the “violate” flag is set.

Response times are based on office hours. Hence the results unit is working days.

Solution time

The solution time is a performance figure also calculated from the support unit’s point of view. It describes how long it took the support unit for providing a solution. Solution time is the time from assigning a ticket to a support unit until it provides a solution for the problem described. “Providing a solution” means setting the ticket status to “solved” or “unsolved”. The solution time is calculated as the difference between the solution timestamp and the assign timestamp.

Solution times are based on office hours. Hence the results unit is working days.

Waiting time

Waiting time is the sum of all time slots the ticket was set to “waiting for reply”. Calculating the waiting time has started in July 2012. For tickets submitted before July 2012 no waiting time calculation was done. The waiting time for these tickets may be zero. The waiting time can be excluded when calculation solution times by ticking the checkbox “exclude waiting time”.

Ticket lifetime

The ticket lifetime is calculated from the user’s point of view. It describes how long it takes to provide a meaningful solution for a problem reported by the user. Ticket lifetime is the time from ticket submission to ticket solution (status “solved/unsolved”).

The ticket lifetime is based on calendar days.

Time zones

GGUS support units are spread over a wide range of time zones. Some of the support units themselves are spread over several time zones. However most support units are located in European time zones. Support units and their time zones are listed on a dedicated page. For middleware product teams GGUS assumes time zone “UTC +1” for all support units. For all the other support units the system uses the relevant time zone for calculating timestamps as far as possible.

Office hours

The systems assumes usual office hours from 09:00 to 17:00 Monday to Friday for all support units. National holidays are not taken into account. For middleware product teams the timezone UTC+1 is used as default timezone.

Ticket priorities

For the calculation of performance figures the original priority set during ticket submission is used. This priority value is kept as long as the support unit is in charge of the ticket. Updating the priority value during ticket lifetime doesn’t affect the calculation of performance figures.

Unit abbreviations

[wd] means working days
[d] means calendar days

Reports description

Although some of the drop-down lists request the selection of values a query can be started anyway. In this case the query is equivalent to a query over all tickets!

The “Reset” button resets all fields to their default settings besides the time frame.

Tickets submitted

This metric gives the number of tickets submitted within the specified time frame. Major criteria is the submit timestamp. The result lists shows the current status of the tickets.

Open tickets time

The open tickets time report calculates the time from ticket submission until now in calendar days. The submit timestamp must match the specified time frame for the report.

Tickets closed

This metric is focused on the ticket life time. It gives the number of tickets that reached the status “solved” or “unsolved” within the specified time frame. Major criteria is the solution timestamp which is the timestamp setting a ticket to “solved” or “unsolved”. Tickets in other terminal status like “verified” or “closed” appear in the result list as long as they have been set to “solved/unsolved” in the given time frame. Besides date, status and the number of closed tickets the results list displays the average ticket lifetime and the median ticket lifetime. The result lists shows the current status of the tickets.

Response time

This metric focuses on the responsiveness of support units. Major criteria for this report is the submit timestamp which must match the selected time frame. The result list shows:

the number of tickets responded
the number of responses
average response time and
median response time

for the specified time frame. The result list shows all support units that have ever been in charge of a ticket. Hence the same ticket ID may appear several times. The number of responses may be greater than the number of tickets.

Response times are based on office hours. Hence an average response time of 1d 3h 23min means 13 hours and 23 minutes in total.

Violated response time

Specific privileges are required for this report. Only people belonging to a dedicated technology provider (TP) are able doing reports for this TP. Major criteria for this report are the expected response time defined in SLAs and the TP.

Solution time

Major criteria for this report is the solution timestamp. The result list shows:

the number of solutions
average response time and
median response time

for the specified time frame. The result lists shows the current status of the tickets.

Solution times are based on office hours. Hence, an average solution time of 12d 3h 5min means 99 hours and 5 minutes in total.

The waiting time can be excluded by ticking the checkbox “exclude waiting time”. In case a ticket gets re-opened the metrics calculation starts again from scratch. Hence, the same ticket can appear several times in the solution times calculation.

Input parameters and results

Input parameters

The input parameters vary depending on the report type chosen. Possible input parameters are:

Time frame
Responsible Unit
Status
Priority
Concerned VO
Ticket type
Ticket category
Ticket Scope
Notified site
Technology provider
Date aggregation

Time frame

The time frame defines begin and end date of the report. The begin date starts at 00:00:00. The end date ends at 00:00:00. As the implementation of the report generator started in October 2011 the report generator does not provide data for tickets submitted before December 2011!

Responsible Units

The drop-down list offers all responsible units integrated in GGUS system. They can be filtered by keywords. Responsible Units can be either selected all, one by one or by checking the boxes in front of the responsible unit groups. In case no responsible units are selected all responsible units are considered in the reports.

Status

The drop-down list offers all status values available in GGUS system. Multiple selections are possible.

Priority

The drop-down list offers all priority values available in GGUS system. Multiple selections are possible.

Concerned VO

“Concerned VO” provides a drop-down list of all VOs supported by GGUS. Multiple selections are possible.

Ticket type

This drop-down list lists all ticket types in GGUS. Multiple selections are possible.

Ticket category

Selectable categories are “Incident”, “Change request”, “Documentation”, “Test”. Multiple selections are possible. The category “Test” should not be considered for all reports and therefore, if necessary, be excluded from the selection.

Ticket scope

To distinguish between tickets under the responsibility of EGI or WLCG. See ticket scope.

Notified site

“Notified site” lists all sites integrated in GGUS. They are derived from GOC DB and OIM DB. In case of reporting exclusively on tickets without any site value specified please ticket the “blank” value in the drop-down list.

Technology Provider

This input parameter is only visible if choosing the “violated response time” report. For accessing this report specific privileges are required. The drop-down list offers all technology providers currently integrated in GGUS and the “DMSU”. Multiple selections are possible. In case no technology provider is selected the reports will be done for tickets without any technology provider specified.

Date aggregation

The results aggregation level can be chosen from the drop down list “Choose date aggregation”.

Group by

Results can be grouped by one or more of the input parameters.

Results

The results are displayed below the input parameter area. They can be sorted in different ways by clicking on the column labels. A drill-down is possible by clicking on any row of the results list. The detail results open in a new window. They can be sorted by clicking the column labels too. Clicking on the ticket ID opens the ticket in GGUS system. At the bottom of the result panel there are various icons offering features like:

Search
Refresh
Export
Export what you see

Doing a simple “Export” saves the data in a csv file stripping all column headers. For exporting the data including the column headers please use “Export what you see”. The “Search” feature allows searching in the result list. Possible search parameters are the columns of the result list.

Internal: Team tickets

Mon, 01 Jan 0001 00:00:00 +0000

Introduction: purpose and conditions

The purpose of TEAM tickets is to allow all team members to modify all tickets of their team, although they may not have support access, and they do not have submitted the ticket themselves.
Only members of a few VOs are allowed to submit team tickets:
- Alice
- Atlas
- CMS
- LHCb
- Biomed
- Belle
The VO members need to have the appropriate permissions in GGUS user database
Other VOs can request this functionality by opening a GGUS ticket
Team tickets are routed to the NGI/ROC the site belongs to automatically. They do not need a routing by the TPM. The NGI/ROC is notified about the ticket in the usual way. In parallel, the site receives a notification email.

Becoming a Team-ticket member

People who want to become a team member have to:

register in GGUS first
be added to the appropriate group in a VOMS server.

Technical description

This section describes the workflows of team tickets from a technical point of view.

Team ticket submission

Team tickets can either be submitted using the GGUS web portal. On top of the ticket submit form in GGUS web portal there is a link to the submit form for team tickets.

As team ticket submitters are expected to be experts who will hopefully provide all necessary information, the number of fields on the team ticket submit form is reduced to a minimum, compared to the number of fields on the user ticket submit form

Three fields on this form are mandatory:

Subject
MoU Area
Notified Site

All of the other fields are optional.

Team ticket processing

The processing of a team ticket consists of two main parts, the notification of the notified site and the routing of the ticket to the NGI/ROC the site belongs to.

Site notification

In parallel to the creation of a team ticket the GGUS system sends an email notification directly to the site specified in field “Notify Site”. This email is sent to a specific site contact mail address.

Ticket routing

Team tickets are bypassing the TPMs and routed to the appropriate NGI/ROCs automatically as long as field “Routing type” is not changed. The decision to which NGI/ROC a ticket has to be routed is done automatically, based on the value of the “Notify Site” field.

Working on team tickets

For working on team tickets and resolving please use the GGUS portal. A reference link to the team ticket is given in the team notification mail. Team tickets can be upgraded to alarm tickets clicking on the button “Click to convert to ALARM”.

Team tickets process schema

See the schema of the Team Tickets process.

Internal: Ticket Priority

Mon, 01 Jan 0001 00:00:00 +0000

Priority definition

If you plan to adjust the ticket priority of a ticket we kindly ask you to provide also justification for this change in the reply text box of the ticket.

The following table will help you to find an appropriate priority value:

Priority	Comment
very urgent	service interrupted; needs to be addressed as soon as possible
urgent	service degraded; a workaround might or might not be available
less urgent	wishes and enhancements that are “nice to have”

In particular, be very economical when choosing ‘very urgent’. This value, when reaching the supporters via another ticketing system interface, might become a beep or phone alert even in the middle of the night. This level of support is ONLY committed by WLCG Tier0 and Tier1s and ONLY for ALARM tickets.

Finally, please be aware that the supporter who will try to solve your problem may change the value you have chosen to a more realistic one, putting their justification in the ticket.

Internal: Ticket Scope

Mon, 01 Jan 0001 00:00:00 +0000

Priority definition

The GGUS ticket scope filter is available in GGUS ticket search. It offers filtering for tickets in EGI or WLCG scope. The ticket scope field is automatically set by the system based on the qualifications listed below.

WLCG scope tickets are either:

of type TEAM or ALARM related to VOs alice, atlas, cms, lhcb, belle
assigned to VOSupport related to VOs alice, atlas, cms, lhcb, belle
assigned to OSG Software Support, USCMS, USATLAS, USBELLE, CRIC
with issue type like CMS_, ATLAS_

All other tickets are EGI scope.