Storage accounting

Introduction

Storage space usage accounting is based on the StAR (Storage Accounting Record) developed during the EMI project in conjunction with the OGF Usage Record Work Group (UR-WG). The format is documented in GFD-I.201.

EMI delivered StAR solutions for dCache and DPM in EMI-3. In both cases the storage service queries its database at a site and extracts data to populate StAR usage records. The site then uses SSM as a transport method to send the StAR records via the EGI Messaging Service to EGI Accounting repository.

Deployment instructions

Accounting script for dCache

The configuration follows the same configuration system that is common to all dCache components. As with other dCache components, the default values are exposed and documented.

In general, a site should review the star.properties file (in the standard location for configuration defaults) and update their site-specific configuration, if necessary.

Information on how to set the star.properties file can be found in the dCache repository.

Map local groupid to the VO names as appropriate, for example:

star.gid-mapping = 12345=/atlas,12346=/cms,12347=/lhcb (EXAMPLE)

Running the dcache-star command generates records according to that site’s configuration.

Edit SSM’s sender.cfg to have path:/var/spool/dcache/star so that it can find and publish the records dCache generated.

Accounting script for DPM

You need to install DPM-DMLITE 1.8.7 or higher. The Storage Accounting is implemented as a puppet module that adds cron configuration to execute the script daily.

Please be sure to have installed the star-accounting.py script v1.0.4 at least.

Accounting script for EOS

The script generating the required space accounting information is available in the eos-server package starting with release 5.0.15.

Please have a look at the EOS documentation for more information.

Install the APEL SSM software

The APEL SSM software can be installed from the UMD-4 repository.

Add the information to EGI Configuration Database

You need to add a new service endpoint for that host to EGI Configuration Database with the service type eu.egi.storage.accounting and the correct host certificate DN. The Accounting Repository takes up to an hour to update its ACL from the configuration Database and the EGI Messaging Service take up to 4 hours. If you get warnings in your SSM log about invalid username or password you can retry again after a delay. If this persists for over 4 hours, then do open a Helpdesk ticket.

Configure SSM

Set the configuration files as explained in the general documentation and in the migration instructions.

Running the Accounting Software

Create a cron job to run your accounting script followed by calling the SSM sender. We recommend that you send storage accounting data once per day. There will be a delay of up to 24 hours before you see the data you have sent reflected in the Accounting Portal.

• Example:

$ cat /etc/cron.daily/dmlite-StAR-accounting
#!/bin/sh
set -e
mkdir -p /var/spool/apel/outgoing/`date +%Y%m%d`
/usr/share/dmlite/StAR-accounting/star-accounting.py --reportgroups \
  --dbhost=[hostname] \
  --dbuser=[username] \
  --dbpwd=[password] \
  --nsdbname=cns_db \
  --dpmdbname=dpm_db \
  --site=[site name] > /var/spool/apel/outgoing/`date +%Y%m%d`/`date +%Y%m%d%H%M%S`
ssmsend

You can confirm if the Accounting Repository receives your data by looking at the sites publishing storage accounting records

The page is updated on a daily basis.

Storage Accounting Data at the EGI Accounting Portal

The storage accounting view is currently available on the development instance of the Accounting Portal.