DMA on FPGA Cannot Access Kernel Memory Allocated with GFP_KERNEL Flag

Question

I would first like to give a brief description of the scenario that I am working on.

What I am trying to accomplish is to load image data from my user space application and transfer it over PCIe to a custom acceleration engine located inside a FPGA board.

The specifications of my host machine are:

• Intel Xeon Processor with 16G ram.
• 64 Bit Debian Linux with kernel version 4.18.

The FPGA is a Virtex 7 KC705 development board. The FPGA uses a PCIe controller (bridge) for the communication between the PCIe infrastructure and the AXI interface of the FPGA. In addition, the FPGA is equiped with a DMA engine which is supposed to read data through the PCIe controller from the kernel memory and forward them to the accelerator.

Since in future implementations I would like to make multiple kernel allocations up to 256M, I have configured my kernel to support CMA and DMA Contiguous Allocator. According to dmesg I can verify that my system reserves at startup the CMA area.

Regarding the acceleration procedure:

1. The driver initially allocates 4M kernel memory by using the dma_alloc_coherent() with GFP_KERNEL flag. This allocation is inside the range of the CMA.
2. Then from my user space application I call mmap with READ_PROT/WRITE_PROT and MAP_SHARED/MAP_LOCKED flags to map the previously allocated CMA memory and load the image data in there.
3. Once the image data is loaded I forward the dma_addr_t physical address of the CMA allocated memory and I start the DMA to transfer the data to the accelerator. When the acceleration is completed the DMA is supposed to write the processed data back to the same CMA kernel allocated memory.
4. On completion the user space application reads the processed data from the CMA memory and saves it to a .bmp file. When I check the "processed" image it is the same as the original one. I suppose that the processed data were never written to the CMA memory.

Is there some kind of memory protection that does not allow writing to the CMA memory when using GFP_KERNEL flag?

An interesting fact is that when I allocate kernel memory with dma_alloc_coherent but with either GFP_ATOMIC or GFP_DMA the processed data are written correctly to the kernel memory but unfortunately the allocated memory does not belong to the range of the CMA area.

What is wrong in my implementation? Please let me know if you need more information!

Answer 1

In order to use mmap() I have adopted the debugfs file operations method. Initially, I open a debugfs file as follows:

shared_image_data_file = open("/sys/kernel/debug/shared_image_data_mmap_value", O_RDWR);

The shared_image_data_mmap_value is my debugfs file which is created in my kernel driver and the shared_image_data_file is just an integer.

Then, I call mmap() from userspace as follows:

    kernel_address = (unsigned int *)mmap(0, (4 * MBYTE), PROT_READ | PROT_WRITE, MAP_SHARED | MAP_LOCKED, shared_image_data_file, 0);

When I call the mmap() function in user space the mmap file operation of my debugfs file executes the following function in the kernel driver:

dma_mmap_coherent(&dev->dev, vma, shared_image_data_virtual_address, shared_image_data_physical_address, length);

The shared_image_data_virtual_address is a pointer of type uint_64_t while the shared_image_data_physical_address is of type dma_addr_t and they where created earlier when I used the following code to allocate memory in kernel space:

shared_image_data_virtual_address = dma_alloc_coherent(&dev->dev, 4 * MBYTE, &shared_image_data_physical_address, GFP_KERNEL);

The address that I pass to the DMA of the FPGA is the shared_image_data_physical_address.

I hope that the above are helpful.

Thank you!