BAR regions unallocated after PCIe rescan on Linux

Question

I have an FPGA card attached to PCIe on a Linux system. I can re-program the FPGA and then echo 1 > /sys/bus/pci/rescan and my card shows up in lspci.

However the BAR regions aren't allocated any memory:

Region 0: Memory at <unassigned> (64-bit, prefetchable) [disabled] [size=32M]
Region 2: Memory at <unassigned> (64-bit, prefetchable) [disabled] [size=64M]

I've tried various pci= flags on the Linux boot command line without much effect.

My suspicion is that the BAR regions are too big and therefore Linux is unable to find an appropriately aligned range in the address space to map the BARs, since they weren't present at boot.

Is this a correct theory? Is it possible to tell Linux to reserve space for my device?

Answer 1

I am seeing this too in Linux. There is an answer on the Xilinx forums at http://forums.xilinx.com/t5/PCI-Express/Incorrect-assignment-of-64bit-BAR-addresses/td-p/347577 - your memory spaces are smallish though. Try reducing the size until you get an allocation.

Most likely your BIOS does not allow such a big memory space BAR. Actually I have never seen a real PCIe device with more than 512MB mapped into memory space. Why would you want to do this? If you think to have a reason for that, check again whether your application really requires this amount of memory being always accessible at the same time or if you could also handle this with a paging mechanism.

Your suspicion is close but it is the BIOS that is allocating the regions, not linux. As I see it the root port is given a fixed chunk of memory, if you're asking for more than is free in that chunk then you go unallocated.

Answer 2

Just a wild guess: these regions are assigned addresses when a driver first maps them.

Answer 3

A Tom Keddie says, BIOS, not the linux kernel, allocates the address regions for the device. I have found that sometimes you need to reboot the Linux system in order to get it to map the device. Depending on how you configure your FPGA, you may need to provide external power to the FPGA so that its configuration is not cleared on reboot.