A way to have GDB load libraries from local sysroot and remote gdbserver

Question

When remote debugging using gdbserver, I'd like to get gdb to load some shared libraries of the programm being debugging from the local sysroot, but also allow download feature from gdbserver to load others, which are not present in the sysroot.

It seems that gdb can use only one method to find libraries, local files or remote download, and not both.

Example, if I set sysroot to target:/ to use remote files, everything will be downloaded:

(gdb) set sysroot target:/
(gdb) run
Starting program:  
Reading /root/a.out from remote target...
warning: File transfers from remote targets can be slow. Use "set sysroot" to access files locally instead.
Reading /root/a.out from remote target...
Reading symbols from target:/root/a.out...
Reading /lib/ld-linux-armhf.so.3 from remote target...

Unfortunately, the system libraries on the remote system do not have debug symbols. It's an embedded system with limited flash space. Debugging symbols increase the total file system size a great deal and simply don't fit.

However, I have a local sysroot tree of all the system libraries and this does include debugging symbols. But if I set sysroot to this tree, gdb will no longer consider remote downloads.

(gdb) set sysroot /bsp/sysroot
(gdb) run
Starting program:  
Reading symbols from /bsp/sysroot/root/a.out...
warning: Could not load shared library symbols for /lib/libm.so.6.

In this example, libm.so.6 is not present in the sysroot, but could have been downloaded from the target. But there seems to be now way to add target:/ back to the search path. Putting it in solib-search-path has no effect.

This situation arises from the use of a board support package (BSP) in embedded systems development. The BSP contains many libraries, which are stripped on the target, as they wouldn't fit otherwise, but have unstripped copies for the host. Users of the BSP build their own software it, but this software isn't part of the BSP and isn't present in the BSP's sysroot. It is however on the target system.

There seems no way to tell gdb to first try to find libraries locally, but then fall back to remote download if they are not found.

Answer 1

GDB does not support multiple sysroots, but there are some work-arounds.

• One way to fix the issue is by patching GDB to actually support multiple sysroots. Function solib_find_1 in gdb/solib.c (link to source code) handles the library lookup. Currently, it checks the sysroot (and if that starts with target:, uses the libraries from the target). Otherwise it takes the basename (i.e. takes the file name from the given absolute path) and looks up the libraries in solib-search-path.
  To get the desired behavior, the solib_find_1 function should instead fall back to a different directory, e.g. this patch: https://gitlab.com/gbenson/binutils-gdb/commit/0ebe17076406a85a35eb0c4f362850ed9efb843e
• A simpler approach without patching GDB: Copy the target libraries to the host. Variants include:
  • Directly inside the local sysroot. This is quick and simple.
  • With symlinks inside the local sysroot to the location of the target libraries.
  • Use a filesystem that merges multiple locations, such as OverlayFS.
  • Use a bind mount to mount the directories or files over the sysroot.
• If you don't want to copy files: Establish an automatic local mirror of libraries from the target (e.g. with sshfs), and use the approaches from the previous point to fall back to the target when needed.
• If you don't have the ability to alter the filesystem, then you can put a proxy between your GDB client and the GDB server:
  1. Use set sysroot target:
  2. Create a proxy for GDB's File-I/O Remote Protocol, and let it transparently forward every message, except for requests for target files. If the files of interest are available locally, reply with that. Otherwise forward the message as-is.
  3. Attach the debugger using target remote :[your_proxy_port] instead of target remote :[actual_gdbserver_port], to let your proxy connect to the gdbserver on your behalf.

I found that a combination of the "Copy the target libraries to the host" variants was the most effective, as it has minimal requirements (I only need a way to copy/receive files from the target once):

• I copied the system libraries from the target to a local directory. I actually copied all of them so I didn't have to find out which one I really needed.
• I recreated the directory structure on the target of the application and added a symlink to the build output directory of my project. The use of symlinks efforts to get the setup working without recurring maintenance.

The content of the sysroot didn't actually have to be identical to the target's remote files: The target had stripped libraries without debugging symbols, while I used the non-stripped libraries in the sysroot.