0
votes

PCIE uses Producer/Consumer ordering model, but AXI4 uses a different ordering model. AXI4's read and write channels are independent.

For instance, a system like this,

CPU <-> PCIE Controller <-> PCIE AXI Bridge <-> AXI4 with DEC and DDR slaves (DEC module is connected to AXI4 slave data port0 and APB register port, DDR memory module is connected to AXI4 slave data port1)

CPU performs the follwing two operations,

1.CPU writes data to DDR
2.CPU writes DEC APB register to start DEC

Because PCIE memory writes (both prefetchable and non-prefetchable) are posted, i.e. without responses, PCIE AXI Bridge will perform the above two operations successively with the same ID but without waiting for BRESP. Before data reaches DDR, DEC may have seen the APB register write and start read the data, so the data may be old and invalid.

This ordering issue can be solved by writing a data then reading it back as follows,

1.CPU writes data to DDR
2.CPU read the same data back
3.CPU writes DEC APB register to start DEC

But it's inefficient.

So how can we solve this ordering issue when PCIE and AXI4 are connected? Is it better to solve this ordering issue in the PCIE side or AXI interconnect side?

1

1 Answers

1
votes

When I look at Linux Kennel code device drivers I see the same problem, write to this register then this one but how do I know the first is done? In a lot of cases the driver will simply read either that same register back or a register which should show a side effect. For example nvme devices can be reset via one register write but you do not know that operation is complete until another register read returns a specific value. Notice the nvme_enable_ctrl function which writes a 32 reg and then immediately stalls until a read returns a specific value. As such I would advise following the decisions of past development instead of wrapping your own solution.

https://elixir.bootlin.com/linux/latest/source/drivers/nvme/host/core.c#L2169