dlopen fails in chroot

Question

I'm attempting this: I have an environment defined in /chroot/debian6.0/ where I have bound some directories and created other ones. One is libs/ which contains the library libOne.so and its dependencies So:

/chroot/debian6.0/
               \--- libs/
                       \--- libOne.so
                       \--- other dependencies (*.so)

This library has been compiled in the chroot environment, and I want to open it with a process run from the containing environment.

This is the code:

remote.c

#include <unistd.h>
#include <dlfcn.h>
#include <stdio.h>

int main(int argc, char **argv)
{
    int res = 0;
    void* handle;

    /*chdir to first argument(path1)*/
    res = chdir(argv[1]);
    if (res == 0) {
        printf("\nchdir: %s", argv[1]);
    } else {
        printf("\nError chdir %s\n", argv[1]);
        return 1;
    }

    /*chroot to path in first argument*/
    res = chroot(argv[1]);
    if (res == 0) {
        printf("\nchroot: %s", argv[1]);
    } else {
        printf("\nError chroot %s\n", argv[1]);
        return 1;
    }

    /*Define path for dependencies*/
    putenv("LD_LIBRARY_PATH=/libs/");

    /*Opens library*/
    handle = dlopen(argv[2], RTLD_NOW);
    if (handle == NULL) {
        printf("\nError opening %s\n", argv[2]);
        return 1;
    } else {
        printf("\ndlopen: library %s opened\n", argv[2]);
    }

    return 0;

}

And I execute with the following command:

./remote "/chroot/debian6.0/Debian-6.0-chroot/" "/libs/libOne.so"

The result is an error when trying to dlopen the library. Last lines of strace:

read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0P#\0\000"..., 1024) = 1024
fstat64(3, {st_mode=S_IFREG|0644, st_size=116600, ...}) = 0
old_mmap(NULL, 119656, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xb7644000
mprotect(0xb7661000, 872, PROT_NONE)    = 0
old_mmap(0xb7661000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x1c000) = 0xb7661000
close(3)                                = 0
munmap(0xb7f01000, 6291)                = 0
munmap(0xb7770000, 6496996)             = 0
munmap(0xb7757000, 98784)               = 0
munmap(0xb7662000, 1000332)             = 0
munmap(0xb7644000, 119656)              = 0
write(1, "chroot: /chroot/debian6.0/Deb"..., 48chroot: /chroot/debian6.0/Debian-6.0-chroot/
) = 48
write(1, "Error opening /libs/libD"..., 50Error opening /libs/libOne.so
) = 50
munmap(0xb7f03000, 4096)                = 0
_exit(1)                                = ?

The library appears to have all its dependencies - inside the chroot I can ldd libOne.so and I get

linux-gate.so.1 => (0xb7f16000) libpthread.so.0 => /lib/libpthread.so.0 (0xb78c6000) libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0xb77d1000) libm.so.6 => /lib/libm.so.6 (0xb77aa000) libc.so.6 => /lib/libc.so.6 (0xb7665000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb7647000) /lib/ld-linux.so.2 (0xb7f17000)

Any idea why the dlopen fails? Or how to make it work?

---

Added perror and dlerror and I get:

Error opening library /lib/libc.so.6: version `GLIBC_2.4' not found (required by /libs/libOne.so)) = 117 

---

UPDATE:

Although I have copied the libc version which I use to compile the program, to my containing environment and chroot' environment (both in /lib), i still get the same message on execution:

Error opening library /lib/libc.so.6: version `GLIBC_2.4' not found (required by /libs/libOne.so)) = 117

So i went to each lib folder and that's what i see:

Containing Env:

ll /lib
libc-2.2.4.so
libc.so.6 -> libc-2.2.4.so

ldd libc.so.6
    /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0xb7dcc000)

Chroot's env:

ll /lib
libc.so.6 -> libc-2.11.3.so
libc-2.11.3.so

ldd libc.so.6
    /lib/ld-linux.so.2 (0xb7fb4000)
    linux-gate.so.1 =>  (0xb7fb3000)

In my Compiling Environment:

ll /lib
libc.so.6 -> libc-2.11.3.so
libc-2.11.3.so

ldd libc.so.6
    /lib/ld-linux.so.2 (0xf770f000)
    linux-gate.so.1 =>  (0xf770c000)

So it seems that my compiling and execution versions of libc.so.6 are the same, and linked to a second one which is the same, that is libc-2.11.3.so

So, I don't get why I'm getting GLIBC_2.4 message

Last update: right now, after Petesh's guidance and using dlmopen, my strace throws next messages (at the end):

writev(2, [{"./remote", 5}, {": ", 2}, {"/lib/libdl.so.2", 15}, {": ", 2}, {"version \`GLIBC_2.3.4\' not found "..., 51}, {"\n", 1}], 6./remote: /lib/libdl.so.2: version `GLIBC_2.3.4' not found (required by ./remote) ) = 76

Answer 1

The error in relation to GLIBC_2.4 not being found is most likely caused because the invocation of dlopen occurs in a process that has already mapped in a copy of libc that doesn't expose that version.

So, we need to be a little bit creative, and use the secondary routine dlmopen rather than simple old dlopen.

dlmopen takes a parameter lm_id which is a namespace where it loads all the relevant libraries - if you use the default (LM_ID_BASE), it starts with the namespace that the program runs in; which for example would contain the libc that was loaded already when the application was launched.

So, rather than just calling handle = dlopen(argv[2], RTLD_NOW); we replace it with a call to:

handle = dlmopen(LM_ID_NEWLM, argv[2], RTLD_NOW | RTLD_LOCAL);

This will load fresh copies of all the libraries that argv[2] needs into the link address space used by that handle.

You can then reference symbols in the library using:

symbol = dlsym(handle, "symname");

and it should work.

I've changed your example slightly to make use of this:

#define _GNU_SOURCE
#include <dlfcn.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char **argv)
{
    int res = 0;
    void* handle;

    /*chdir to first argument(path1)*/
    res = chdir(argv[1]);
    if (res == 0) {
        printf("\nchdir: %s", argv[1]);
    } else {
        printf("\nError chdir %s\n", argv[1]);
        return 1;
    }

    /*chroot to path in first argument*/
    res = chroot(argv[1]);
    if (res == 0) {
        printf("\nchroot: %s", argv[1]);
    } else {
        printf("\nError chroot %s\n", argv[1]);
        return 1;
    }

    /*Define path for dependencies*/
    putenv("LD_LIBRARY_PATH=/libs/");

    /*Opens library*/
    handle = dlmopen(LM_ID_NEWLM, argv[2], RTLD_NOW); // needs _GNU_SOURCE
    if (handle == NULL) {
        printf("\nError opening %s: %s\n", argv[2], dlerror());
        return 1;
    } else {
        printf("\ndlopen: library %s opened\n", argv[2]);
    }

    int (*foo)(void);
    foo = (int (*)(void))dlsym(handle, argv[3]);
    if (foo == NULL) {
        printf("\nError referencing %s: %s\n", argv[3], dlerror());
        return 1;
    }
    printf("%d\n", foo());

    return 0;

}

Now secondsly, the putenv("LD_LIBRARY_PATH=/libs/"); doesn't do the needful - you cannot alter LD_LIBRARY_PATH at the run-time of a program; the change will simply not be detected. Because your program is so simple we could do something at the start of the main like:

if (0 == getenv("RE_EXEC")) {
    putenv("RE_EXEC=1");
    putenv("LD_LIBRARY_PATH=/libs/");
    execvp(argv[0], argv);
    perror("execvp failed");
    exit(1);
}

which re-executes with LD_LIBRARY_PATH set to /libs so it will search there.

There are a few other potential solutions - if the path setting is not suitable with the re-exec

You could make sure that libOne.so has $ORIGIN in the rpath for the binary. You can do this at compile/link time with:

-Wl,-rpath,'$ORIGIN'

The quoting is to prevent the $ORIGIN from getting escaped by the shell. If this is in a makefile then you need to use the $$ syntax for the target actions.

You can post-hoc patch the library with a tool like patchelf (on my system I did a simple sudo apt-get install patchelf):

patchelf --set-rpath '$ORIGIN' libOne.so

This, of course relies on support for $ORIGIN in your ld.so (I cannot remember when it was added to ld.so; but it's been a few years); if your ld.so does not support $ORIGIN then you can always replace it with /libs/, and if you can't find patchelf for your distro, then you can always build it yourself