How to match OpenCL devices with a specific GPU given PCI vendor, device and bus IDs in a multi-GPU system?

16
votes

I would like to be able to match OpenCL devices with GPUs in the system on multi-GPU systems identified by PCI IDs.

For example, if I have a system with multiple GPUs, possibly from different vendors, I can list the devices by enumerating the PCI bus. This gives me PCI vendor, device and bus IDs. If I choose one of these (GPU) PCI devices to use for OpenCL computation based on some selection criteria, how do I match it to the OpenCL device?

I can enumerate GPU devices in OpenCL using clGetDeviceIDs() but there is no obvious way to match OpenCL devices to PCI devices. The OpenCL function clGetDeviceInfo() provides access to the PCI vendor ID and device name but not PCI device or bus IDs. I could try to match the PCI device name with the OpenCL device name but it's possible that you have more than one of the same type of device and the names are not always the same anyway.

Why is this necessary? Say I know that program X is running CUDA or something else on GPU A. I want to avoid also using GPU A for an OpenCL operation so I choose GPU B. I then need to figure out which OpenCL device is GPU A and which is GPU B. PCI IDs seem to be the only consistent and cross platform way of identifying GPU devices.

BTW, the CUDA API does give you PCI, bus and slot IDs (CU_DEVICE_ATTRIBUTE_PCI_BUS_ID, CU_DEVICE_ATTRIBUTE_PCI_DEVICE_ID) but CUDA only works with NVidia devices.

Ideally I need a solution using either C or C++.

5
openclgpupci
The spec says CL_DEVICE_VENDOR_ID "could be the PCIe ID". If that doesn't get what you want, then I don't think there's anything in the spec that will. Still not sure why you need this, though. Sounds like premature optimization. – vocaro
@vocaro: Yes, I can get the Vendor ID. I don't think you understand the question. – jcoffland
You say you want to know the PCI device ID to avoid contention with another process which may be using a specific PCI device ID. I wondered how you know which PCI devices are in use? I guess you aren't using OpenCL for that? – Matt

5 Answers

10
votes

The way to do it is to use two vendor-specific extensions. For AMD, you have to use CL_DEVICE_TOPOLOGY_AMD which works on Windows and Linux and will return the PCIe bus id, which is unique for a GPU. On NVIDIA, query the device for CL_DEVICE_PCI_BUS_ID_NV. See also: https://anteru.net/2014/08/01/2483/

1
votes

Unfortunately the answer you're looking for is not pretty due to the abstracted nature of openCL.

The only way I have found to reliably do it is to assign a demanding workload to the platform + device ID in openCL, and then monitor the process usage via tools such as AMD's ADL and Nvidia's NVML. Even mature applications like cgminer have issues with this and often mix up openCL workloads with card metrics, so much so that they assign configuration variables to correct it manually ("gpu-map").

I wish there was a better answer for now because it would be great to know, through openCL, which device is behind the endpoint! This may change in the future, as AMD is working to add this layer to openCL as arsenm pointed out.

1
votes

Seems Anteru answer is correct, but only if you are running linux/mac. After some testing I did, it seems windows does not recognize these vendor specific extensions. (I've tested it both on Geforce GTX Titan & ATI Radeon R9)

My solution to you is to use clGetGLContextInfoKHR() function (available since openCL spec 1.1) with "CL_CURRENT_DEVICE_FOR_GL_CONTEXT_KHR" parameter, and that will ensure you get an openCL device ID that matcth the same GPU that performs the render.

True, that won't give you physical bus slot, but that will ensure the same GPU that renders is the same GPU that compute !

Also, assuming one works with Nvidia Quadro cards, then he can use the wgl_nv_gpu_affinity to ensure openGL access a specific GPU, and then use the GL context & get from it openCL device ID.

0
votes

The most recent AMD release has the cl_device_topology_amd extension on Linux,, which adds the CL_DEVICE_TOPOLOGY_AMD option to clGetDeviceInfo(), but that's a pretty narrow solution.

0
votes

I developed a library to do just that: keep OpenCL simulations from stepping on each others toes.

You will find it here: https://github.com/nbigaouette/oclutils/

It first enumerate all platforms and every devices for each platform present on the machine. You select the wanted platform and it will pick the best device available. I use it on my workstation with 3 nvidia cards: two GTX 580 for OpenCL calculations and one GT 210 for the display. Running two simulations at the same time will run on the two GTX separately. without intervention.

There is also a nice class which will keep two buffers in sync: one on the host and one on the device. Calling OpenCL_Array::Host_to_Device() and OpenCL_Array::Device_to_Host() makes the transfers back and forth simple.

It works with these platforms:

  • nvidia (GPU only)
  • amd (CPU and/or GPU)
  • intel (CPU only)
  • apple (CPU and/or GPU)

Note that it won't let you choose which device to use, but pick one for you. If two instances of a program uses the library, they will know it and won't run on the same device (if you have too, of course). It is also not able, right now, to detect if the video card is used for the display. But at least it's a start!