Detect transaction from PCIe endpoint to host memory in Linux

2
votes

I'm building an FPGA design in which some devices (CPU, UART, GPIO controller) are connected to an AXI4 bus. The AXI4 bus is connected to the host via Xilinx' "AXI Memory Mapped to PCI Express (PCIe) Gen2 v2.6 LogiCORE IP", which acts as a bridge. The subsystem residing on the FPGA interfaces via PCIe with a Linux driver. This driver creates a device file to provide an interface to a computer system simulator running on the host, a so-called virtual platform (VP). Part of the system components will be simulated in the VP, whereas the rest of the components are implemented on the FPGA. The system bus of the VP is connected transparently to the AXI4 bus on the FPGA via the driver and the aforementioned bridge. Transactions originating on the host are already implemented and working correctly.

I'd like to know if there's a way of detecting a PCIe read or write transaction within a Linux driver, whereas the transaction is issued by the PCIe endpoint acting as a bus master.

So far, my ideas are: (i) to instantiate a DMA controller on the FPGA, which issues a message-signaled interrupt (MSI) to the host at the end of each transfer. The driver can then retrieve the starting address and size of the transfer from the DMA's control registers. (ii) to allocate a buffer on the host, from / to which the PCIe bus master reads / writes. This buffer would need to be polled in order to detect changes, which to me seems inefficient, especially for large buffers.

Does somebody have an idea for a better solution? This is my first question here. Please excuse that my question became a bit lengthy. Any feedback will be highly appreciated.

1
linuxlinux-kernellinux-device-driverpci-e
mwait is frequently a good solution for this sort of thing, if you don’t want to use interrupts and you don’t mind tying up a CPU. – prl
If you don’t want to tie up a CPU, because there is other work they need to be doing, you pretty much have to use an interrupt. – prl
Thanks for your comments. I'll implement an RTL block which will reside on the FPGA and generate an interrupt whenever a bus master on the FPGA issues a memory request to the host. This block will have registers to hold the destination address, size and actual data of a request. The interrupt signals to the Linux driver that there is "something to be done". The driver can then retrieve the data from the RTL block and take the right actions. My question just turned into a hardware question and is out of the scope of Stackoverflow. If somebody is interested I'll be happy to share my solution. – Thomas Grass
Why not? Write the answer and ask moderators to move your question to hardware part of Stack family. – 0andriy

1 Answers

1
votes

I'm answering my question from two months ago, in case somebody else faces a similar problem in the future.

I implemented an IP block in VHDL which is instantiated as many times as there are AXI master ports which want to communicate with the virtual platform (VP) running on the host. This IP block has two AXI slave ports: one is connected to the AXI master port issuing the requests, the other can be read out via a PCIe / AXI bridge from the host. Whenever the IP block intercepts a request, an interrupt is generated in order to notify the host driver. The driver reads the address, data, type (R/W), and size of the request, which in turn are read by the VP via ioctl(). The VP processes the request and communicates the response to the driver, also via ioctl(). Then, the driver forwards the response to the IP block connected to the corresponding AXI master port and the request is completed.

In the future I'll probably implement a new version of the IP block which will use a DMA controller to write the request information to a buffer in the host memory, in order to reduce the number of PCIe round trips. The availability of new data in the host buffer will be signalled by an interrupt.