GPU flavours

Setting up GPU flavours

Support for GPU can be added to flavours using the PCI passthrough feature in OpenStack. This allows to plug any kind of PCI device to the Virtual Machines.

As a summary of the OpenStack documentation, these are the steps needed to add a GPU enabled flavour (be aware this may need tuning to your specific hardware/configuration!):

1. On computing node, get vendor/product ID of your hardware: lspci | grep NVIDIA to get pci slot of GPU, then virsh nodedev-dumpxml pci_xxxx_xx_xx_x
2. On computing node, unbind device from host kernel driver. Unbinding is system dependent, and can be done in many ways, e.g.:
   • if the kernel does not uses the devices (no GPU drivers included in kernel, or drivers disable in GRUB), nothing to unbind
   • via pci-stub grubby --args="pci-stub.ids=10de:11fa" --update-kernel DEFAULT (see RedHat manual, section 12.1, step 1-2; where the pci-stub.ids value is vendor_ID: product_id from lspci.
   • via echo command: echo $dev > /sys/bus/pci/devices/$dev/driver/unbind where $dev is the PCI device ID xx:xx.x or xxxx:xx:xx.x from lspci
3. On computing node, add pci_passthrough_whitelist = {"vendor_id":"xxxx","product_id":"xxxx"} to nova.conf (see nova-compute)
4. On controller node, add pci_alias = {"vendor_id":"xxxx","product_id":"xxxx", "name":"GPU"} to nova.conf (see nova-api)
5. On controller node, enable PciPassthroughFilter in the scheduler (see nova-scheduler)
6. Create new flavours with pci_passthrough:alias (or add key to existing flavour), e.g. openstack flavor set m1.large --property "pci_passthrough:alias"="GPU:2"

GPU description in flavour metadata

Users should be able to easily discover the flavours that provide GPUs (or accelerators in general). The following table describes the agreed metadata for EGI providers to add to those flavours:

There are some extra fields that are defined in the GLUE2.1 schema but not so relevant for GPUs and therefore not considered at the moment. These are listed below for completeness:

Adding metadata to flavours has no effects on site operations. End users can see the metadata easily via openstack flavor list --long or openstack flavor show <flavor id> commands without any additional tools, e.g.:

$ fedcloud openstack flavor show gpu1cpu2  --site IISAS-GPUCloud --vo eosc-synergy.eu -f json
Site: IISAS-GPUCloud, VO: eosc-synergy.eu
{
  "OS-FLV-DISABLED:disabled": false,
  "OS-FLV-EXT-DATA:ephemeral": 0,
  "access_project_ids": null,
  "disk": 40,
  "id": "a8082202-f647-4d1f-9b97-4f5ddb38ae8e",
  "name": "gpu1cpu2",
  "os-flavor-access:is_public": false,
  "properties": "Accelerator:Version='0', Accelerator:Memory='5', Accelerator:Model='Tesla K20m', Accelerator:Number='1.0', Accelerator:Type='GPU', Accelerator:Vendor='NVIDIA', Accelerator:VirtualizationType='PCI passthrough', pci_passthrough:alias='GPU:1'",
  "ram": 8192,
  "rxtx_factor": 1.0,
  "swap": "",
  "vcpus": 2
}