Exec format error when compiling in nfs shared directory inside a VM

Question

I set up a 32 bits Debian VM on my host 64 bits OS. I shared the source code directory using NFS.

When I compile inside that shared folder, I am unable to run the executable:

nicolas@vdeb32:~/Shared$ cc hello.c -o hello
nicolas@vdeb32:~/Shared$ ./hello
-bash: ./hello: cannot execute binary file: Exec format error
nicolas@vdeb32:~/Shared$ file hello
hello: data

But when the output is out of that shared folder, everything works fine:

nicolas@vdeb32:~/Shared$ cc hello.c -o /tmp/hello
nicolas@vdeb32:~/Shared$ /tmp/hello
hello
nicolas@vdeb32:~/Shared$ file /tmp/hello
/tmp/hello: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.32

What is the cause of that behavior? I found that askubuntu reply but there is no explanation and anyway my host is not running windows neither my executable is a link.

Thanks.

[EDIT] I found the reason why the file cannot be executed. Editing hello.c, I noticed that the file, once saved, was only contaning zeros:

nicolas@vdeb32 ~/Shared $ xxd hello.c 
00000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000030: 0000 0000 0000 0000                      ........

After some retries: when creating a file, everything works fine. But when I edit a file (in the shared directory), it is overwritten with zeros (same number of bytes as the original file). When compiling, the executable produced contains also only zeros. I am left with a different problem but the same question: what is the cause of that behavior?

[EDIT2] No trouble with the same architecture when the guest is running Mint 64 bits.

Are you compiling on the same system you are running on? Or you are compiling on the host when trying to run on VM (or vise-versa)? In the latter case you need to cross-compile. — Eugene Sh.
@EugeneSh. I am indeed compiling on the same machine I am trying to execute the output (the guest). — Nicolas Garnier
I'd be very curious whether the resulting executable differs (in terms of cmp or diff) in the NFS dir vs in /tmp. Check to make sure that your nfs share is not mounted noexec, but I don't think that will change the file output — Sam Hartman
Interesting edit. Is your host machine doing some funny stuff on the background? — Eugene Sh.

Nicolas Garnier Nicolas Garnier · Accepted Answer · 2017-09-28T13:42:08

Great! I eventually managed to solve the problem. This has nothing to do with compiling, editing or whatsoever. It was a network issue.

I tcpdumped NFS frames on both sides (btw that made me notice my NFS was using TCP -- I was sure UDP was the default).

From the guest:

Data written is what was actually written
Wrong TCP checksum

From the host:

Message zeroed
TCP checksum OK

Solution: I disabled offloading according to that procedure. I have still to understand the details, but at last I got something functional.

PS: The same behavior was happening when creating a file, but from the client side, as the file was not refreshed, everything was looking fine. But on the host, the same file was already zeroed.

Exec format error when compiling in nfs shared directory inside a VM

1 Answers