Are cache operations atomic?

Question

I'm learning about CPU cache and now I still have misunderstanding of cache coherence protocol (MESI). Imagine that we have 2 cores have a cache line in a shared state. And one of them performs read, another one performs write:

 ;mem is cached in Shared state
 Thread 1 (core 1)          Thread 2 (core 2)
  mov rax, [mem]            mov [mem], dword 1

Can the core 1 observe some intermediate state. I mean something like the following:

Core 2 marks the cache line in L1D as modified and writes change into it.
The cache line in core 1's L1D cache is still in Shared state, so reading happens to read the stale value.
After the reading of the stale value has happened already the line in core 1's L1D cache is marked as Invalid.

Is some scenario like this possible in Intel's MESI/MESIF implementation?

It doesn't really matter because there is no global ordering of instructions so you don't know if thread1 reads the value before or after thread2. According to Peter's answer though, thread2 will not perform the write until it got a reply for its RFO request, so no, that should not happen. — Jester
Are we assuming that [mem] resides in a single cache line? It could easily be split over two, and get a different result. — Bo Persson
@BoPersson Definitely yes. Actually I did not think about misaligning it... — Linda Turasco

Peter Cordes Peter Cordes · Accepted Answer · 2018-02-25T19:30:47

The cache line in core 1's L1D cache is still in Shared state

This is the part of scenario that violates MESI. The store can't commit until the RFO sent by core 2 has completed, so core 1 has the line in Invalid state.

In your example it wouldn't really be an "intermediate" step, though. Without synchronization, there's no way to distinguish your impossible scenario from simply having the load by core 1 happen before the line was invalidated. i.e. core 1's load can appear before core 2's store in the global order.

Stores don't become globally visible until well after they execute locally (they have to retire and then the store queue can commit them to L1D), and x86's memory model allows StoreLoad reordering, so stores can be delayed (kept hidden in the private store queue) until after later loads by core 2 become globally visible. (See Jeff Preshing's Memory Barriers Are Like Source Control Operations for more background on memory reordering, and what StoreLoad reordering means).

In MESI (and all variants like MESIF or MOESI), if one cache has a line in E or M state, no other caches can have a copy of that line. The state table in the MESI wikipedia article makes this perfectly clear: if one cache has E or M state, the others all have Invalid.

It's never possible for two caches to both have valid copies of a line with differing data. This is what it means for caches to be coherent, and stopping that from happening is the whole point of the MESI protocol.

If a core wants to modify a cache line, it takes Exclusive ownership of the line so no other cores can observe stale values. This has to happen before a store can commit into L1D. Store queues exist to hide the latency of the Read-For-Ownership (among other things), but data in the store queue is not yet committed to L1D. (Related: What happens when different CPU cores write to the same RAM address without synchronization? has more about the store queue).

And BTW, lets assume that [mem] is naturally-aligned so loads / stores to it are atomic (as guaranteed by the x86 architecture). Why is integer assignment on a naturally aligned variable atomic on x86?.

Multi-level caches and Modified lines

With multi-level caches, dirty cache lines can propagate up the hierarchy. So a line can be in Modified state in L1D and L2 of the same core. This is fine because write-back from L1D goes through L2.

As I understand it, the shared inclusive L3 cache in Intel CPUs doesn't have to write-back to DRAM before it can share out copies of the cache line to multiple cores. So as far as normal / simple MESI is concerned, think of L3 as the backing store, not DRAM.

Making this work on multi-socket systems is tricky; I'm not sure if things are set up so the L3 in a socket can only cache physical addresses that correspond to DRAM attached to that socket. In any case, snoop requests are sent between sockets on L3 cache miss, and there are lots of complicated settings you can configure to tweak this on a Xeon system. (See an Anandtech article about Haswell Xeon for example.)

Are cache operations atomic?

2 Answers