execve file not found when stracing the very same file!

Question

someone i know encountered a problem when running 'lmutil' so i asked them to strace -f lmutil. Why is execve failing with "No such file"!!! It makes no sense, since I am straceing the very same file!! What exactly is going on here???

strace -f /home/tabitha/Starprogram/FLEXlm_11.7/linux-x86_64-2.3.4/bin/lmutil

Output:

execve("/home/tabitha/Starprogram/FLEXlm_11.7/linux-x86_64-2.3.4/bin/lmutil", ["/home/tabitha/Starprogram/FLEXlm"...], [/* 38 vars */]) = -1 ENOENT (No such file or directory)
dup(2)                                  = 3
fcntl(3, F_GETFL)                       = 0x8002 (flags O_RDWR|O_LARGEFILE)
fstat(3, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 1), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd7cb8b0000
lseek(3, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
write(3, "strace: exec: No such file or di"..., 40strace: exec: No such file or directory
) = 40
close(3)                                = 0
munmap(0x7fd7cb8b0000, 4096)            = 0
exit_group(1)                           = ?

ldd output

$ ldd ./lmutil
        linux-vdso.so.1 =>  (0x00007fffcd5ff000)
        libpthread.so.0 => /lib/libpthread.so.0 (0x00007fe40ebbe000)
        libm.so.6 => /lib/libm.so.6 (0x00007fe40e93b000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007fe40e724000)
        libc.so.6 => /lib/libc.so.6 (0x00007fe40e3a1000)
        libdl.so.2 => /lib/libdl.so.2 (0x00007fe40e19d000)
        /lib64/ld-lsb-x86-64.so.3 => /lib64/ld-linux-x86-64.so.2 (0x00007fe40edf5000)

$ find . -name lmutil -exec file {} \;
./bin.linux.x86_64/lmutil: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.4.0, dynamically linked (uses shared libs), for GNU/Linux 2.4.0, stripped
./bin.linux.x86/lmutil: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), for GNU/Linux 2.2.5, stripped
./lmutil: Bourne shell script text executable

Answer 1

The file you're trying to execute (…/lmutil) exists but its “loader” doesn't exist, where

• the loader of a native executable is its dynamic loader, for example /lib/ld-linux.so.2;
• the loader of a script is the program mentioned on its shebang line, e.g., /bin/sh if the script begins with #!/bin/sh.

From the name of the directory, there's a good chance that lmutil is an amd64 Linux binary, looking for /lib64/ld-linux-x86-64.so.2 as its loader, but you have an amd64 Linux kernel running a 386 (i.e. 32-bit) userland. You need to get suitable binaries for your platform.

I consider this situation to be Unix's most misleading error message. Unfortunately fixing it would be hard: the kernel can only report a numeric error code to the caller of the program, so it only has room for “command not found” (ENOENT) and not for the name of the loader it's looking for. This is one of these rare cases where strace doesn't help.

Answer 2

Your ldd output refers to /lib64/ld-lsb-x86-64.so.3, but this loader may not actually exist unless (on Ubuntu) you've installed lsb-core package. The postinst script for the package creates the relevant symbolic links in /lib* directories.

Answer 3

Just a bit of speculation, but my first question would be if the user who is having this problem can run the executable by itself without strace.

Also the execve manual page says that ENOENT will occur if either the file or a required script interepreter or shared library cannot be found. (I notice there is 64-bit-ness involved here. Are all the right libraries available?)

Is the file a native executable or could it be a script of some sort?

This looks like a licensing manager - any chance it has made itself intentionally hard to debug?

Speaking of users, is 'tabitha' in whose directory the executable resides the user having the problem? Or are we looking at a possible complication of trying to run a program installed by another ordinary user rather than in a normal system-wide fashion by root?

Answer 4

You can use readelf (any readelf should do, you don't need one from a special crosscompiler toolchain) to check which loader is expected by the dynamically loaded or executable.

$ readelf -l <filename> |grep -i interp
...
[Requesting program interpreter: /system/bin/linker]

Answer 5

From the execve manpage:

On success, execve() does not return, on error -1 is returned, and errno is set appropriately.

strace is assuming that -1 means "file not found" as the errno value ENOENT is -1 and strace doesn't make a distinction.

Essentially, then, you can ignore this: the -1 just means that some error occurred. the strace output doesn't tell you what the value of errno is.

I write this as a warning call to not jump to conclusions with strace and return values, even though it may turn out that errno is ENOENT here anyway.