How to know or set a PCI/PCIe device address map?

Question

I'm trying to understand how to instruct the CPU to instruct a PCI device to map its memory to the physical address of the CPU.

I've read https://wiki.osdev.org/PCI#Memory_Mapped_PCI_Configuration_Space_Access and I understand how to configure a PCI device through the IO ports of the CPU:

Two 32-bit I/O locations are used, the first location (0xCF8) is named CONFIG_ADDRESS, and the second (0xCFC) is called CONFIG_DATA. CONFIG_ADDRESS specifies the configuration address that is required to be accesses, while accesses to CONFIG_DATA will actually generate the configuration access and will transfer the data to or from the CONFIG_DATA register.

So in order to configure a PCI device I just need to put the configuration address that I want to configure and CONFIG_DATA is where I put the data. Both of these are I/O locations, so I ise I/O instructions in the CPU.

These two I/O locations give me access to a 256byte "big register" that I can use to configure the PCI device. All the fields of this big register are described in the OSDEV page. These 2 bits are of interest:

Memory Space - If set to 1 the device can respond to Memory Space accesses; otherwise, the device's response is disabled.

I/O Space - If set to 1 the device can respond to I/O Space accesses; otherwise, the device's response is disabled.

However apart from this, it's not clear on how to get or set the physical address space where each PCI device is going to respond

Answer 1

These two I/O locations give me access to a 256byte "big register" that I can use to configure the PCI device.

That "big register" is a structure with many fields. One of the fields is "Command Register", which contains global enable/disable flags, that can be used to disable the device's ability to send things to the bus (IRQs, writes from device to memory, etc) and respond to things from the bus (writes from CPU to device, etc).

Other fields include "BARs" (Base Address Registers, starting at offset 0x10 in a device's configuration space) which tell the device which address range (in IO port space or physical address space) the device should accept. For these, a device will be hard-wired to use either IO ports or physical addresses (and that can not be changed) and the size of the area will also be hard-wired (and that can also not be changed). Because these things are hardwired, you can write zeros to a BAR and find out what was hardwired (to determine the size of the area, and if the device needs IO port space or physical address space).

Fortunately the firmware is responsible for configuring BARs, so (excluding hot-plug devices and "unusual circumstances") an OS doesn't need to set BARs itself, and can just read the value/s from the BARs (that were set by firmware) to determine the address (in IO port space or physical address space) that the device was already configured to use. Sadly, if the OS needs to determine the size of the area (e.g. and can't rely on the device driver knowing the size for its device already), an OS may need to read the firmware's value, then do the "write zeros to find out the size of the area", then restore the firmware's value.

Of course PCI configuration space won't say what the BARs are used for - it's up to the device driver to know what each area (described by each BAR) is used for.

If an OS wants to (re)configure BARs itself for whatever reason; then it's significantly more complex because you have to also make sure any bridges forward accesses to the device correctly (and if there's multiple devices behind a bridge then those devices must use sub-ranges in a single larger range that the bridge will forward); and you have to ensure that the CPU's caches don't ruin everything (which mostly means re-configuring the MTRRs/Memory Type Range Register on all CPUs at the same time with special synchronization); and you have to make sure you don't stomp on anything (e.g. ranges already used by firmware, RAM, CPU, etc); and have to make sure everything is in part of the physical address space that the CPU can access (even when the CPU has errata and reports the wrong "number of physical address bits" via. CPUID).