CLFLUSH in virtualization environment

Question

I have read the document about the CLFLUSH instruction of Intel x86 machine. I understand that CLFLUSH m8 means flush the cache line containing linear address m8 if I run the instruction inside a Linux module on the bare metal.

However, what if I want to run the instruction inside a VM in virtualization environment? What the parameter of CLFLUSH is? To be specific, suppose I want to flush a cache line that hold the content for virtual address va of a program in a VM. The virtual address va is mapped to physical address pa in the VM, and pa will be mapped to the machine address ma in the VMM/hypervisor. Which address should I use for the CLFLUSH instruction, va, pa or ma?

Thank you very much!

Let's answer this way. I'm sure you'll agree that software should run identically in virtualized and non-virtualized environments, and that ideally it should be unable to tell it is being virtualized. The answer to your question immediately follows; You use whichever address you would have used had it been running bare-metal, simply because you (should) have no way to tell whether you are, in fact, running bare-metal or not. — Iwillnotexist Idonotexist
This is a nice observation! I agree. However, the reason why software have no idea if it runs in virtualized or non-virtualized environment is because the VMM/hypervisor virtualized the instruction. If the VMM does not support this instruction and I want to extend the VMM's hypercall to support it, I will have to know which address the hypervisor should use for the CLFLUSH instruction. Am I correct? — Mike
If hypervisor can use the va address, I'm wondering how hardware can figure out which cache line should be flushed based on the va? On bare metal, I can understand that the hardware can use CR3 register to transfer the va to machine address to find the cache line to flush. But in virtualized environment, will the hardware do the translation from va to ma automatically? — Mike
I am leafing it right now, but I strongly suspect you'll find the nitty-gritty details in the Intel Software Developer's Manual, Volume 3, Section 23-33 but especially 28 "VMX SUPPORT FOR ADDRESS TRANSLATION". Long story short, an Intel processor has the option of maintaining additional bookkeeping information, in the shape of a VPID for each linear address translation. Then there can exist multiple mappings for the same linear address, but they differ in VPID. — Iwillnotexist Idonotexist
yeah, that's true... intel uses VPID to distinguish the linear address from different VMs. This may indicate that I have to issue CLFLUSH when the owner program of the address is still running. Otherwise, I will have to load the VPID and CR3 to issue CLFLUSH. Loading CR3 has already flush the cache. — Mike

Peter Cordes Peter Cordes · Accepted Answer · 2015-11-02T19:26:22

clflush is not a privileged instruction. User processes can run it, just like they can run load, store, prefetch, and movnt (store with cache eviction) instructions which also affect the cache.

I expect it virtualizes just fine, without any hypervisor support, since it uses addresses the same way other memory references do.

Since hardware virtualization allows the CPU to do a full translation from guest virtual addresses to hardware physical addresses, the desired behaviour (cache line synced to physical memory) should happen without hypervisor intervention.

CLFLUSH in virtualization environment

1 Answers