PCIe Understanding

Question

As this domain is new for me, I have some confusions understanding PCIe. I was previously working on some protocols like I2c,spi,uart,can and most of these protocols have well defined docs(a max of 300 pages). In almost all these protocols mentioned, from a software perspective, the application had to just write to a data register and the rest will be taken care by the hardware. Like for example, in Uart, we just load data into the data register and the data is sent out with a start, parity and stop bit.

I have read a few things about PCIe online and here is the understanding i have so far.

During system boot, the BIOS firmware will figure out the memory space required by the PCIe device by a magic write and read procedure to the BAR in the PCIe device(endpoint). Once it figures out that, it will allocate an address space for the device in the system memory map(no actual RAM is used in the HOST, memory resides only in the endpoint.The enpoint is memory mapped into the Host). I see that the PCIe has a few header fields that the BIOS firmware figures out during the bus enumeration phase.

Now,if the Host wants to set a bit in a configuration register located at address 0x10000004(address mapped for the enpoint), the host would do something like(assume just 1 enpoint exists with no branches):

*(volatile uint32 *)0x10000004 |= (1<<Bit_pos);

1.How does the Root complex know where to direct these messages because the BAR is in the enpoint. Does the RC broadcast to all enpoints and then the enpoints each compare the address to the address programmed in BAR to see if it must accept it or not?(like an acceptence filter in CAN).

2. Does the RC add all the PCIe header related info(the host just writes to the address)?

3. If Host writes to 0x10000004, will it write to register at location 0x4 in the endpoint?

4. How does the host know the enpoint is given an address space starting from 0x10000000?

5. Is the RC like a router?

The above queries were related to, only if a config reg in the enpoint was needed to be read or written to. The following queries below are related to data transfer from the host to the enpoint.

1.Suppose the host asks the enpoint to save a particular data present in the dram to a SSD,and since the SSD is conneted to the PCIe slot, will PCIe also perform DMA transfers? Like, are the special BAR in the enpoint that the host writes with a start address in the Dram that has to be moved to ssd, which in turn triggers the PCIe to perform a DMA tranfer from host to enpoint?

I am trying to understand PCIe relative any other protocols i have worked on so far. This seems a bit new to me.

Answer 1

The RC is generally part of the CPU itself. It serves as a bridge that routes the request of the CPU downstream, and also from the endpoint to the CPU upstream.

PCIe endpoints have Type 0 headers and Bridges/Switches have Type 1 header. Type 1 headers have base(min address) and limit registers(max address). Type 0 headers have BAR registers that are programmed during the enumeration phase.

After the enumeration phase is complete, and all the endpoints have their BARs programmed, the Base and Limit registers in the Type 1 header of the RC and Bridges/Switches are programmed.

Ex: Assume a system that has only 1 endpoint connected directly to the RC with no intermediate Bridges/Switches, whose BAR has the value A00000. If it requests 4Kb of address space in the CPU(MMIO), the RC would have its Base register as A00000 and Limit register as AFFFFF(It is always 1 MB aligned,though the space requested by the endpoint is much less than 1MB). If the CPU writes to the register A00004, the RC will look at the base and limit register to find out if the address falls in its range and route the packet downstream to the endpoint.

Endpoints use BAR to find out if they must accept the packets or not. RC, Bridges and Switches use Base and Limit registers to route packets to the correct downstream port. Mostly, a switch can have multiple downstream ports and each port will have its own Type 1 header,whose Base and Limit register will be programmed with respect to the endpoints connected to its port. This is used for routing the packets.

Data transfer between CPU memory and endpoints is via PCIe Memory Writes. Each PCIe packet has a max payload capacity of 4K. If more than 4K has to be sent to the endpoint, it is via multiple Memory Writes. Memory Writes are posted transactions(no ACK from the endpoint is needed).