Linux 4.4 PCIe DMA into userspace pages not working - highmem not usable for DMA?

Question

I am updating an older linux driver that transfers data via DMA to userspace pages which are passed down from the application via get_user_pages().

My hardware is a new x86 Xeon based board with 12GB of RAM.

The driver gets data from a VME to PCIe FPGA, and is supposed to write it into the main memory. I do a dma_map_pages() for each page, I check it with dma_mapping_error() and write the returned physical DMA address into the buffer descriptors of the DMA controller. Then I kick off the DMA. (We also can see the transfer starting in the FPGA tracer).

However, when I get the DMA finish IRQ I see no data. For control, I have the same VME address space accessible via PIO mode and that works. I also tried writing values to page_address(page) of the userpages and the application can see these. All ok.

Digging deeper into the matter I checked the usual documentation like DMA-API.txt, but I could not find any other approach, also not in other drivers.

My a kernel is a self compiled 4.4.59 64bit with all kinds of debugs (debug DMA-API etc..) set to yes.

I also tried to dig through drivers/iommu/ to see debug possibilities here but just a few pr_debugs there.

The interesting thing: I have another driver, an ethernet driver, which supports a NIC connected to PCI. This one works without problems!

When dumping and comparing the retrieved DMA dma_addr_t's I see this:

The NIC driver allocates memory via dma_alloc_coherent() for buffer descriptors etc., it's addresses are in the "lower 4 GB":

 [ 3127.800567] dma_alloc_coherent: memVirtDma = ffff88006eeab000, memPhysDma = 000000006eeab000
 [ 3127.801041] dma_alloc_coherent: memVirtDma = ffff880035d9b000, memPhysDma = 0000000035d9b000
 [ 3127.801373] dma_alloc_coherent: memVirtDma = ffff88006ecd4000, memPhysDma = 000000006ecd4000

The VME driver, dma_map_page'ing the user space pages are > 4GB, the DMA address looks different: 0xffffe010 (with an offset from the application).

pageAddr=ffff88026b4b1000 off=10 dmaAddr=00000000ffffe010 length=100

DMA_BIT_MASK(32) is set in both drivers, our FPGA cores are 32bit wide.

Question: do I have to have special prerequisites in order for this DMA to work? I read that highmem memory can not be used for DMA, is this still so?

Part of dmesg:

[    0.539839] debug: unmapping init [mem 0xffff880037576000-0xffff880037ab2fff]
[    0.549502] DMA-API: preallocated 65536 debug entries
[    0.549509] DMA-API: debugging enabled by kernel config
[    0.549545] DMAR: Host address width 46
[    0.549550] DMAR: DRHD base: 0x000000fbffc000 flags: 0x1
[    0.549573] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap     8d2078c106f0466 ecap f020df
[    0.549580] DMAR: RMRR base: 0x0000007bc14000 end: 0x0000007bc23fff
[    0.549585] DMAR: ATSR flags: 0x0
[    0.549590] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x0
[    0.549779] DMAR: dmar0: Using Queued invalidation
[    0.549784] DMAR: dmar0: Number of Domains supported <65536>
[    0.549796] DMAR: Setting RMRR:
[    0.549809] DMAR: Set context mapping for 00:14.0
[    0.549812] DMAR: Setting identity map for device 0000:00:14.0     [0x7bc14000 - 0x7bc23fff]
[    0.549820] DMAR: Mapping reserved region 7bc14000-7bc23fff
[    0.549829] DMAR: Set context mapping for 00:1d.0
[    0.549831] DMAR: Setting identity map for device 0000:00:1d.0     [0x7bc14000 - 0x7bc23fff]
[    0.549838] DMAR: Mapping reserved region 7bc14000-7bc23fff
[    0.549845] DMAR: Prepare 0-16MiB unity mapping for LPC
[    0.549853] DMAR: Set context mapping for 00:1f.0
[    0.549855] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 -     0xffffff]
[    0.549861] DMAR: Mapping reserved region 0-ffffff
[    0.549892] DMAR: Intel(R) Virtualization Technology for Directed I/O
...
[    0.551725] iommu: Adding device 0000:00:00.0 to group 10
[    0.551753] iommu: Adding device 0000:00:01.0 to group 11
[    0.551780] iommu: Adding device 0000:00:01.1 to group 12
[    0.551806] iommu: Adding device 0000:00:02.0 to group 13
[    0.551833] iommu: Adding device 0000:00:02.2 to group 14
[    0.551860] iommu: Adding device 0000:00:03.0 to group 15
[    0.551886] iommu: Adding device 0000:00:03.2 to group 16
[    0.551962] iommu: Adding device 0000:00:05.0 to group 17
[    0.551995] iommu: Adding device 0000:00:05.1 to group 17
[    0.552027] iommu: Adding device 0000:00:05.2 to group 17
[    0.552059] iommu: Adding device 0000:00:05.4 to group 17
[    0.552083] iommu: Adding device 0000:00:14.0 to group 18
[    0.552134] iommu: Adding device 0000:00:16.0 to group 19
[    0.552166] iommu: Adding device 0000:00:16.1 to group 19
[    0.552191] iommu: Adding device 0000:00:19.0 to group 20
[    0.552216] iommu: Adding device 0000:00:1d.0 to group 21
[    0.552272] iommu: Adding device 0000:00:1f.0 to group 22
[    0.552305] iommu: Adding device 0000:00:1f.3 to group 22
[    0.552332] iommu: Adding device 0000:01:00.0 to group 23
[    0.552360] iommu: Adding device 0000:03:00.0 to group 24
[    0.552437] iommu: Adding device 0000:04:00.0 to group 25
[    0.552473] iommu: Adding device 0000:04:00.1 to group 25
[    0.552510] iommu: Adding device 0000:04:00.2 to group 25
[    0.552546] iommu: Adding device 0000:04:00.3 to group 25
[    0.552575] iommu: Adding device 0000:05:00.0 to group 26
[    0.552605] iommu: Adding device 0000:05:00.1 to group 27

When you say you cannot see the data, is that before or after unmapping the DMA transfer? If IOMMU mapping cannot be used for highmem for some reason, the DMA mapping routines should still be able to use bounce buffers. If bounce buffers are used, there will be no data in the memory you mapped to DMA until it is either unmapped or synced to CPU. — Ian Abbott
Hi, ist after dma_unmap_page(). I call dma_mapping_error() after dma_map_page() as recommended, shouldn't that give me an error when the kernel can't provide a valid DMA page ? — Thomas S.
Yes, if dma_mapping_error(dmaaddr) returns 0 for a dmaaddr returned by dma_map_page, then the DMA address is valid. (dma_mapping_error returns a simple true/false value rather than a negative errno value.) — Ian Abbott
The DMA address should be reachable as a PCI address from the FPGA. But some FPGAs use special mapping tables to map PCI addresses into the local FPGA address space. For example, Altera PCIe hard IP uses an "Avalon-MM to PCI Express Address Translation Table". (I think some third party DMA controller IP for Altera FPGAs can work with PCI addresses directly without going through this annoying translation table.) — Ian Abbott
Hi Ian, is there any possibility to debug further into the IOMMU to find out how this obviously virtual DMA address 0xFFFFe000 is further mapped to main memory ? I think somewhere there must be some if(...) clause or a check like that which fails in my case. I would like to see or debug which main memory address gets assigned for this DMA. I tought of adding debugs to dma_map_page() first but quickly dropped the idea since that function is called a million times in the kernel... — Thomas S.

Thomas S. Thomas S. · Accepted Answer · 2017-10-05T11:47:48

for completeness here the answer, we found it. Totally different reason: PCIe protocol bug in the FPGA PCIe core...

Linux 4.4 PCIe DMA into userspace pages not working - highmem not usable for DMA?

1 Answers