Linux kernel point of view
A good way to learn something is to interact with it, so let's use the Linux kernel for that.
Here is a minimal PCI example on a QEMU emulated device: https://github.com/cirosantilli/linux-kernel-module-cheat/blob/366b1c1af269f56d6a7e6464f2862ba2bc368062/kernel_module/pci.c
The first 64 bytes of the PCI configuration are standardized as:
Image from LDD3.
So we can see that there are 6 BARs. The wiki page then shows the contents of each BAR:
The region width requires a magic write however: How is a PCI / PCIe BAR size determined?
This memory is setup by the PCI device, and gives information to the kernel.
Each BAR corresponds to an address range that serves as a separate communication channel to the PCI device.
The length of each region is defined by the hardware, and communicated to software via the configuration registers.
Each region also has further hardware defined properties besides length, notably the memory type:
IORESOURCE_IO: must be accessed with inX and outXIORESOURCE_MEM: must be accessed with ioreadX and iowriteXSeveral Linux kernel PCI functions take the BAR as a parameter to identify which communication channel is to be used, e.g.:
mmio = pci_iomap(pdev, BAR, pci_resource_len(pdev, BAR));
pci_resource_flags(dev, BAR);
pci_resource_start(pdev, BAR);
pci_resource_end(pdev, BAR);
By looking into the QEMU device source code, we see that QEMU devices register those regions with:
memory_region_init_io(&edu->mmio, OBJECT(edu), &edu_mmio_ops, edu,
"edu-mmio", 1 << 20);
pci_register_bar(pdev, 0, PCI_BASE_ADDRESS_SPACE_MEMORY, &edu->mmio);
and it is clear that properties of the BAR are hardware defined, e.g. the BAR number 0, has type memory PCI_BASE_ADDRESS_SPACE_MEMORY, and the memory region is 1MiB long 1 << 20.
See also: http://wiki.osdev.org/PCI#Base_Address_Registers of course.
I think this is a very basic question and I would suggest to read:
A Base Address Register (BAR) is used to:
- specify how much memory a device wants to be mapped into main memory, and
- after device enumeration, it holds the (base) address, where the mapped memory block begins.
A device can have up to six 32-bit BARs or combine two BARs to a 64-bit BAR.
BAR is record of the device address starting at memory.
root@Ubuntu:~$ lspci -s 00:04.0 -x
00:04.0 USB controller: Intel Corporation 82801DB/DBM (ICH4/ICH4-M) USB2 EHCI Controller (rev 10)
00: 86 80 cd 24 06 00 00 00 10 20 03 0c 10 00 00 00
10: 00 10 02 f3 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 f4 1a 00 11
30: 00 00 00 00 00 00 00 00 00 00 00 00 05 04 00 00
root@Ubuntu:~$ lspci -s 00:04.0 -v
00:04.0 USB controller: Intel Corporation 82801DB/DBM (ICH4/ICH4-M) USB2 EHCI Controller (rev 10) (prog-if 20 [EHCI])
Subsystem: Red Hat, Inc QEMU Virtual Machine
Physical Slot: 4
Flags: bus master, fast devsel, latency 0, IRQ 35
Memory at f3021000 (32-bit, non-prefetchable) [size=4K]
Kernel driver in use: ehci-pci
root@Ubuntu:~$ grep 00:04.0 /proc/iomem
f3021000-f3021fff : 0000:00:04.0
0xfff equals to 4095, which is 4K. Memory starts at 0xf3021000 is this USB device seen by CPU. This address is init during BIOS and in this example it is on BAR0. Why is BAR0?
Before that, one need to understand PCI spec, especially the below, type 0 and type 1:
Notice the header type is both defined at 0x0c third field, that is how BARs differ. In this example, it is 00, which means it is type 0. Thus BAR0 stores the address, which is 00 10 02 f3.
One may wonder why this is not exactly f3021000, this is because lspci goes with Little Endian. What is Endian? One may need to read "Gulliver's Travels".
BAR0 generally has three states, uninitialized, all 1s, and written address. And we now in the third since the device already init. Bit 11 ~ 4 is set to 0 at uninitialized state; Bit 3 means NP when set to 0, P when set to 1; Bit 2 ~ 1 means 32 bit when set to 00, 64 bit when set to 10; Bit 0 means memory request when set to 0, IO request when set to 1.
0xf3021000
====>>>>
11110011000000100001000000000000
From this, we can know this device is 32-bit, non-prefetchable, memory request. The uninitialized address is 32 ~ 12, since 2 ^ 12 = 4K.
For more device and vendor, one can find via https://pcilookup.com/
Roughly speaking, the root-complex (aka the host computer) acts as the "dealer" and talks to each end-point device in a process called enumeration, where each device has its own set of configuration registers. It does this access using configuration space, rather than normal memory space. memory space for the pci device doesn't exist until the bar registers are setup and mapped by the root complex. Using configuration space, the root-complex sequentially writes all 1's the bar register, in each PCI device, and read them back to determine the size of the bar address space assigned to each device. If the root complex sees zeros in the lower order bits above bit 4, this means that these are addressable space, then it picks a physical memory address and assigned it to the non-zero bits in the bar register...
For PCIe device with 32-bit bars the configuration space has the following 32-bit DWORDS:
UInt32 PCIEBAR32_0, PCIEBAR32_1, PCIEBAR32_2,
PCIEBAR32_3, PCIEBAR32_4, PCIEBAR32_5;
bool cond32_0 = (PCIeBAR32_0 & 0x7) == 0x00);
bool cond32_1 = (PCIeBAR32_1 & 0x7) == 0x00);
bool cond32_2 = (PCIeBAR32_2 & 0x7) == 0x00);
bool cond32_3 = (PCIeBAR32_3 & 0x7) == 0x00);
bool cond32_4 = (PCIeBAR32_4 & 0x7) == 0x00);
bool cond32_5 = (PCIeBAR32_5 & 0x7) == 0x00);
For PCIe device with 64-bit bars, the two adjacent 32-bit DWORDS are concatenated to form a 64-bit bar:
UInt64 PCIEBAR64_0, PCIEBAR64_1, PCIEBAR64_2;
bool cond64_0 = (PCIEBAR32_0 & 0x7) == 0x4);
bool cond64_1 = (PCIEBAR32_2 & 0x7) == 0x4);
bool cond64_2 = (PCIEBAR32_4 & 0x7) == 0x4);
if (!(cond64_0 && cond64_1 && cond64_2)) {
Console.Writeline("Whoops, we don't have 3 adjacent 64-bit bars");
return -1;
}
PCIEBAR64_0 = (UInt64)PCIEBAR32_1<<32 | (UInt64)PCIEBAR32_0;
PCIEBAR64_1 = (UInt64)PCIEBAR32_3<<32 | (UInt64)PCIEBAR32_2;
PCIEBAR64_2 = (UInt64)PCIEBAR32_5<<32 | (UInt64)PCIEBAR32_4;
//note that since lower 4-bits of Least significant
//bar indicate its a 64-bit bar, this means the
//next adjacent 32-bit bar doesn't knockout
//the bottom 4-bits of the bar. so that it can be concatenated.
Not really sure what happens for a system with a mix of 32-bit, and 64-bit bars... maybe you need to check the bars in order from 0 to 5 to find non-aligned cases...
