Understanding PCI address mapping

Question

I'm studying PC architecture and feel that I'm not getting the fundamentals of PCI addresses.

We have three address spaces in PCI: memory, in-out ports and configuration. I know that CPU can distinguish between memory and ports using different commands, but what happens in PCI? We have several commands in bus (reading/writing to these spaces, interruption processing etc). I thought that when reading memory space we address to physical RAM address but after reading some manuals it looks like we address internal devices' memory.

1. Why use memory mapping? Does it mean that when some program writes to RAM address mapped to some PCI device it actually writes to device memory? Why not use standard IO ports write?
2. How to access real memory if needed? For example, if device wants to store some data in RAM, how this request will be distinguished from "memory space" access?

Answer 1

A PCI endpoint device which has its own memory(on the end point device itself) apart from the 256 bytes of configuration space defined by the PCI specification. This memory has to be brought into systems address space inorder to allow a system access it. There are two ways this memory can be brought into systems address space -

1. Port Mapped IO - Address space meant to access IO device and will be accessed via special instructions IN & OUT (on x86). x86 uses 0xCFC(config data port) & 0xCF8(config address port) to access the PCI end point device.

2. Memory Mapped IO - A region of physical address space is reserved for memory mapped region and will be accessed by memory instructions like LOAD & STORE. Access to this MMIO region will be targeting associated end point device.

So when an instruction comes out of processor, it would either go into memory or it will be an IO operation. In systems we have very limited IO address space (64KB for x86 machines) but a very large amount of Physical address space. Also accessing to PCI end point device via port mapped IO is slower comparatively since it requires more CPU cycles. Access to memory is faster since it requires lesser number of CPU cycles, hence MMIO is a better approach Than port Mapped IO.

Note: During the enumeration process the BIOS has to access the device's configuration space to do the initialization. At this point in time there is only one way in PCI to access the device is called "PCI configuration space access" mechanism which would use Port 0xCFC & 0xCF8 to access the config space, Then later BIOS sets up the MMIO range requested by End point device.

Note: When we say "Physical address space" it means that the address range a CPU can access. for example a 64 bit processor can have upto 2^64 = 16EB (depends on the address lines are being used) which is enormous amount of address range. "Physical address space" does not mean that its an address ranges within the RAM.

Note: The PCI Express specification by PCI-SIG(PCI - Special interest group) suggests not to implement an IO address space, IO address space may be completely removed in future PCIe revision/specifications.

Answer 2

Do not think of memory addresses as "RAM addresses". The memory space occupied by RAM is a small fraction of the total address space of the CPU. The remainder of the memory address space is used for MMIO, including PCI devices. Any CPU accesses to these addresses are routed by the processor to the appropriate device registers.

Peter's answer regarding the reason for using MMIO is correct: I/O instructions are serializing, while memory accesses within the CPU are highly optimized, so there is much less overhead for accessing memory space than for accessing I/O space.

A device access to memory space (DMA) is routed in a similar way to CPU accesses--if the access is in the memory address space range of RAM, the access is routed to RAM, and if the access is in the address space range of another device, it is routed to that device.