2
votes

My code has a user mode mapping (set up via mmap()) which I need to flush after writing to it from the CPU but before I dispatch the data by DMA’ing the underlying physical memory. Also I need to invalidate the cache after data has arrived via a DMA to the underlying physical memory but before I attempt to read from it with the CPU.

In my mind “cache flushing” and “cache invalidating” mean two different things. Roughly “cache flushing” means writing what’s in the cache out to memory (or simply cache data goes to memory) whereas “cache invalidating” means subsequently assuming all cache contents are stale so that any attempts to read from that range will provoke a fresh read from memory (or simply memory data goes to cache).

However in the kernel I do not find two calls but instead just one: flush_cache_range().

This is the API I use for both tasks and it “seems to work”… at least it has up until the present issue I'm trying to debug.

This is possibly because the behavior of flush_cache_range() just might be to: 1) First write any dirty cache entries to memory- THEN 2) Invalidate all cache entries

IF is this is what this API really does then my use of it in this role is justified. After all it’s how I myself might implement it. The precise question for which I seek a confident answer is:

IS that in fact how flush_cache_range() actually works?

1
It depends on architecture of the system you are running on. x86, IIRC, cache coherent, thus the command does nothing. - 0andriy
Linux running on the CPU can ensure cache coherency for all the DMA operations it conducts (and therefore knows about). However our card is an autonomous agent outside the CPU. And if it DMA's to memory which has has entries in one of the host's caches the cache will then be stale- and the CPU won't know about it. Hence the need for explicit cache flushing. - user1967844
So, can you mark that piece of memory as non-cacheable from CPU side? - 0andriy
@0andriy Sure, but at significant performance cost compared to writeback plus appropriate CLWB (newer CPUs) or even CLFLUSHOPT / CLFLUSH (older CPUs). UC memory mode is really, really slow. It doesn't just prevent caching, but also prevents pipelined or merged accesses from the CPU — all accesses are word sized and serialized. For fast non-coherent devices, it is often better to use ordinary WB memory with explicit flushing in the driver. - Conrad Meyer

1 Answers

2
votes

Whether caches need to be invalidated or flushed is architecture dependent.

You should always use the Linux DMA functions to handle these issues correctly. Read DMA-API-HOWTO.txt and DMA-API.txt.