How do the sync() call and the sync command differ?

0
votes

I've used mmap() with fopen("/dev/mem") to create a mapping to a block of physical memory shared between two processor cores in an ARM system. When the processor running Linux writes the memory, there can be a lag of over one second before the other non-Linux processor sees the written data. The long delay disappears if the Linux process makes this system call just after writing to memory:

system("sync; echo 3 > /proc/sys/vm/drop_caches" );

I've tried to duplicate that logic directly in code, but the long delay persists:

int fd;
char* data = "3";
sync();
fd = open("/proc/sys/vm/drop_caches", O_WRONLY);
write(fd, data, sizeof(char));
close(fd);

Why does the sync() call differ in behavior from the sync system command? Does the sync command effect virtual memory flushes that the sync() call does not?

I know the manual says that the sync program does nothing but exercise the sync(2) system call, but does the fact that I call sync() from userspace affect its behavior? It acts as though a call to sync from userspace merely schedules the sync rather than blocking until its completion.

2
clinuxsyncmmap
This feels like a long shot, but your echo writes 2 bytes (3 and \n) to drop_caches, while your other code only writes the 3. Could that be the difference? - user2404501
@WumpusQ.Wumbley sizeof(char) == 1 - wildplasser
I tried both methods, with an without the trailing newline, but there was no difference. The system echo command as coded in my example does output the (implied) newline. - edj

2 Answers

0
votes

You forgot the newline.

echo 3 outputs "3\n".

Additionally, you are taking an exceptionally circuitous route to implementing shared memory, and imposing massive costs on the rest of the operating system in doing so.

Every time you call sync-the-command or sync-the-system-call, you cause the OS to flush every filesystem on the entire computer; worse, you're telling the OS to forget every filesystem buffer it has, forcing the OS to re-read everything from disk. It's atrocious to performance to the entire operating system in just about every way you can think of.

There is a much, much easier way.

Use shm_open() to create a named shared memory region. Use mmap to access it. Use memory barriers or shared/named mutexes on just that chunk of memory to ensure that you can read and write it consistently and safely.

In terms of complexity, your current approach is probably 1,000,000 times more costly than normal shared memory.

0
votes

Neither drop_caches nor sync are appropriate here, as both of them deal with file system caches - which aren't actually what you are running into here. The fact that sync appears to solve it is probably coincidental. (It is probably incidentally flushing the data cache when the sync tool is launched.)

Your application is most likely running into cache synchronization issues across the two processor cores on your system. Try using the cacheflush() system call to solve this:

#include <unistd.h>
#include <asm/unistd.h>
...
syscall(__ARM_NR_cacheflush, mapping_ptr, mapping_ptr + mapping_length, 0);

Note that you will probably need to flush the cache in both processes to see the correct results.

Flushing changes to mapped memory is often necessary for other mapped devices, but I think it may not be needed in this case. Trying an msync() as well couldn't hurt, though:

msync(mapping_ptr, mapping_length, MS_SYNC); // on process writing to memory
msync(mapping_ptr, mapping_length, MS_INVALIDATE); // on process reading from memory

Finally, make sure you are mapping this memory with the MAP_SHARED flag.