Linux PCIe DMA driver

1
votes

I'm currently writing a driver for a PCIe device that should send data to a Linux system using DMA. As far as I can understand my PCIe device needs a DMA controller (DMA master) and my Linux system too (DMA slave). Currently the PCIe device has no DMA controller and should not get one. That confuses me.

A. Is the following possible?

  1. PCIe device sends interrupt
  2. Wait for interrupt in the Linux driver
  3. Start DMA transfer from memory mapped PCIe registers to Linux system DMA.
  4. Read the data from memory in userspace

I have everything setup for this, the only thing I miss is how to transfer the data from the PCIe registers to the memory.

B. Which system call (or series of) do I need to call to do a DMA transfer?

C. I probably need to setup the DMA on the Linux system but what I find points to code that assumes there is a slave, e.g. struct dma_slave_config.

The use case is collecting data from the PCIe device and make it available in memory to userspace.

Any help is much appreciated. Thanks in advance!

1
linux-kernellinux-device-driverdmapci-e
Some of PCI transactions are done using DMA (bus mastering from the device). So, if you don't have such, your device, for example, won't be able to send MSI. – 0andriy
And your confusion starts from the point where one has decide which DMA controller you are talking about. If it's device's one (same as bus mastering), then your task is impossible to fulfill. Otherwise you need to program host controller to do transfers and PCIe device will be slave. In this case you also need to be sure you have proper DMA request line(s) provided and all necessary signaling on hardware level. – 0andriy
It might be one or two DMA controllers. Usually the device side controller is completely hidden and used only for bus mastering (messages initiated by device). Of course no-one forbids either to add more DMA controllers on device side or unhide the one used for bus mastering to do other transfers driven by OS, – 0andriy
It is pretty inefficient to transfer data by reading PCIe MMIO registers. All PCIe devices that handle more than tiny amounts of data use bus mastering by the device itself. If the amount of data to be transferred is small enough, you might as well have the CPU read it itself, rather than using a separate DMA controller to transfer it to memory. – prl
You can if your host even has a DMA controller; many don’t, these days, since every PCIe device can do DMA itself. Is there a specific system / DMA controller you are thinking of using? – prl

1 Answers

3
votes

DMA, by definition, is completely independent of the CPU and any software (i.e. OS kernel) running on it. DMA is a way for devices to perform memory reads and writes against host memory without the involvement of the host CPU.

The way DMA usually works is something like this: software will allocate a DMA accessible region in memory and share the physical address with the device, say, by performing memory writes against the address space associated with one of the device's BARs. Then, the device will perform a DMA read or write against that block of memory. When that operation is complete, the device will issue an interrupt to the device driver so it can handle the data and/or free the memory.

If your device does not have the capability of issuing a DMA read or write against host memory, then you'll have to interact with it with using the CPU only. Discrete DMA controllers have not been a thing for a very long time.