Why is it that gdb “can't compute CFA” when using a separate debug symbols file?

3
votes

I'm trying to invoke gdb with a stripped executable and a separate debug symbols file, on a core dump generated from running the stripped executable.

But when I use the separate debug symbols file, gdb is unable to give information on local variables for me.

Here is a log showing entirely how I produce my 3 ELF files and the core file and then run them through gdb 3 times.

  1. First I just run gdb with the stripped executable and of course can't see any file names or line numbers, and can't inspect variables.

  2. Then I run gdb using the stripped executable and grabbing the debug symbols from the original unstripped executable. This works pretty well but does give a disturbing and apparently unwarranted warning about the core and executable possibly mismatching.

  3. Finally I run gdb with the stripped executable and the separate debug file. This still gives filenames and line numbers, but I can't inspect local variables and I get a "can't compute CFA for this frame" error.

Here is the log:

2016-09-16 16:01:45 barry@somehost ~/proj/segfault/segfault
$ cat segfault.c
#include <stdio.h>
int main(int argc, char **argv) {
    char *badpointer = (char *)0x2398723;
    printf("badpointer: %s\n", badpointer);
    return 0;
}

2016-09-16 16:03:31 barry@somehost ~/proj/segfault/segfault
$ gcc -g -o segfault segfault.c

2016-09-16 16:03:37 barry@somehost ~/proj/segfault/segfault
$ objcopy --strip-debug segfault segfault.stripped

2016-09-16 16:03:40 barry@somehost ~/proj/segfault/segfault
$ objcopy --only-keep-debug segfault segfault.debug

2016-09-16 16:03:43 barry@somehost ~/proj/segfault/segfault
$ ./segfault.stripped
Segmentation fault (core dumped)

2016-09-16 16:03:48 barry@somehost ~/proj/segfault/segfault
$ ll /tmp/core.segfault.stripp.11
-rw------- 1 barry bsm-it 188416 2016-09-16 16:03 /tmp/core.segfault.stripp.11

2016-09-16 16:03:51 barry@somehost ~/proj/segfault/segfault
$ gdb ./segfault.stripped /tmp/core.segfault.stripp.11
GNU gdb (GDB) Fedora (7.0.1-50.fc12)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/barry/proj/segfault/segfault/segfault.stripped...(no debugging symbols found)...done.

warning: core file may not match specified executable file.
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/a6/8dce9115a92508af92ac4ccac24b9f0cc34d71
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Core was generated by `./segfault.stripped'.
Program terminated with signal 11, Segmentation fault.
#0  0x00000035fec47cb7 in vfprintf () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.11.2-3.x86_64
(gdb) bt
#0  0x00000035fec47cb7 in vfprintf () from /lib64/libc.so.6
#1  0x00000035fec4ec4a in printf () from /lib64/libc.so.6
#2  0x00000000004004f4 in main ()
(gdb) up
#1  0x00000035fec4ec4a in printf () from /lib64/libc.so.6
(gdb) up
#2  0x00000000004004f4 in main ()
(gdb) p argc
No symbol table is loaded.  Use the "file" command.
(gdb) q

2016-09-16 16:04:19 barry@somehost ~/proj/segfault/segfault
$ gdb -q -e ./segfault.stripped -s ./segfault -c /tmp/core.segfault.stripp.11
Reading symbols from /home/barry/proj/segfault/segfault/segfault...done.

warning: core file may not match specified executable file.
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/a6/8dce9115a92508af92ac4ccac24b9f0cc34d71
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Core was generated by `./segfault.stripped'.
Program terminated with signal 11, Segmentation fault.
#0  0x00000035fec47cb7 in vfprintf () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.11.2-3.x86_64
(gdb) bt
#0  0x00000035fec47cb7 in vfprintf () from /lib64/libc.so.6
#1  0x00000035fec4ec4a in printf () from /lib64/libc.so.6
#2  0x00000000004004f4 in main (argc=1, argv=0x7fffd1c0a728) at segfault.c:4
(gdb) up
#1  0x00000035fec4ec4a in printf () from /lib64/libc.so.6
(gdb) up
#2  0x00000000004004f4 in main (argc=1, argv=0x7fffd1c0a728) at segfault.c:4
4       printf("badpointer: %s\n", badpointer);
(gdb) p argc
$1 = 1
(gdb) q

2016-09-16 16:04:39 barry@somehost ~/proj/segfault/segfault
$ gdb -q -e ./segfault.stripped -s ./segfault.debug -c /tmp/core.segfault.stripp.11
Reading symbols from /home/barry/proj/segfault/segfault/segfault.debug...done.

warning: core file may not match specified executable file.
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/a6/8dce9115a92508af92ac4ccac24b9f0cc34d71
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Core was generated by `./segfault.stripped'.
Program terminated with signal 11, Segmentation fault.
#0  0x00000035fec47cb7 in vfprintf () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.11.2-3.x86_64
(gdb) bt
#0  0x00000035fec47cb7 in vfprintf () from /lib64/libc.so.6
#1  0x00000035fec4ec4a in printf () from /lib64/libc.so.6
#2  0x00000000004004f4 in main (argc=can't compute CFA for this frame
) at segfault.c:4
(gdb) up
#1  0x00000035fec4ec4a in printf () from /lib64/libc.so.6
(gdb) up
#2  0x00000000004004f4 in main (argc=can't compute CFA for this frame
) at segfault.c:4
4       printf("badpointer: %s\n", badpointer);
(gdb) p argc
can't compute CFA for this frame
(gdb) q

I have some questions about this:

  1. Why does it display the warning "warning: core file may not match specified executable file.", even though I'm using the exact same executable path as was used when the core dump was originally generated?
  2. Why does using the separate debug symbols (-s ./segfault.debug) result in the error "can't compute CFA for this frame" when attempting to inspect local variables?

What is a CFA anyway?

Am I using an incorrect method to product the debug symbol file? I confirmed that using "objcopy --strip-debug" gives the same result as "strip -g". Am I using the right options to feed the debug info into gdb?

My intention is that the stripped executables will be installed on a binary-compatible production system and any core dumps generated due to segfaults can be copied back to the devel system where we can feed them into gdb with the debug info and analyse the crash position and stack variables. But as a first step I'm trying to sort out the issues with using separate debug info files on the devel system.

It seems that using a separate debug symbols file causes the "can't compute CFA for this frame" error, even when a core file is not used.

My gcc version:

2016-09-16 16:07:39 barry@somehost ~/proj/segfault/segfault
$ gcc -v
Using built-in specs.
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --disable-dssi --enable-plugin --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre --enable-libgcj-multifile --enable-java-maintainer-mode --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libjava-multilib --with-ppl --with-cloog --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.4.4 20100630 (Red Hat 4.4.4-10) (GCC)

I suspect that gdb might be looking for symbols related to the variables in the segfault.debug file when objcopy actually only put them in the segfault.stripped file. If this is the case, perhaps some small adjustment to the options to objcopy could put those symbols in the place gdb is looking?

2
cdebugginggccgdb
Did you notice there are other symbols missing,too? Notably for libc which includes printf etc. - too honest for this site
I noticed that we don't seem to have the debug symbols installed for all libraries, but I'm interested in finding bugs in our own code, not the system libraries. I'm more concerned about not being able to inspect the variables in our own code. - barryd
And it did not come into mind this might be relevant? Did you try with debug symbols in the system libs or that main is somewhat special? - too honest for this site
I can ask my sysadmin to install the debug symbols for those libs on Monday. But do you really think this will make a difference? Note that in my 2nd gdb test, I was able to inspect argc despite the lack of debug info for those libs. - barryd
Just noting that I've done a simpler test which doesn't call printf. I also tried using a function other than main. The results are the same, except this time there aren't any stack frames from library code to worry about. I still get the CFA error when trying to access local vars or args in my program, when I try to use the separated debug symbol file. I won't update the question until I've had a chance to see about installing the library debuginfos. - barryd

2 Answers

7
votes

I commend you for wanting to keep a set of symbol files for everything that is deployed to the production server; in my opinion this is an often overlooked practice, but you will not regret it -- one day it will save you a lot of debugging trouble.

As I have had similar issues in the past, I will try to answer some of your questions, although you have quite an ancient toolchain, if you don't mind me saying so, so I'm not sure how much that really applies here. I'll put up here anyway.

CFA = Canonical Frame Address. This is the base pointer to the stack frame that every local variable is addressed relative to. If you have done some traditional x86 assembly programming, the BP register was used for this. So "can't compute CFA for this frame" basically says "I know of these local variables, but I don't know where they are located on the stack".

There used to be code in GDB that worked only for the DWARF-2 debugging format, and non-conformance triggered this particular error at least. That restriction was lifted some time ago, but that change won't be in your version.

The other thing is there are debug information regarding how variables may be moved around is not always generated. This usually happens in newer compilers though, as they get better at optimizing.

I was able to get rid of my problems by compiling like this:

gcc -g3 -gdwarf-2 -fvar-tracking -fvar-tracking-assignments -o segfault segfault.c

you can try to see if this solves your problem, too.

Regarding the message about the location of the symbol file; it seems that the debugger wants to load it from the system directory. Maybe you have to link the executable to the symbol file with:

objcopy --add-gnu-debuglink=segfault.debug segfault
2
votes

I found this question while searching for an answer to the following part of the original question:

Why does it display the warning "warning: core file may not match specified executable file.", even though I'm using the exact same executable path as was used when the core dump was originally generated?

There was not an answer to this particular question but through experimentation and research I believe I have found the answer.

Below is a transcript of using gdb to debug a core file. Notice that the "warning: core file may not match specified executable file." error appears when the executable file that caused the core is greater than 15 characters in length.

[~/t]$cat do_abort.c 
#include <stdlib.h>

int func4(int f) { if(f) {abort();} return 0;}
int func3(int f) { return func4(f); }
int func2(int f) { return func3(f); }
int func1(int f) { return func2(f); }
int main(void) {  return func1(1); }

[~/t]$gcc -g -o 123456789012345 do_abort.c 
[~/t]$./123456789012345 
Aborted (core dumped)
[~/t]$ll core*
-rw-------. 1 dev wheel 240K Apr 22 03:19 core.42697
[~/t]$gdb -q -c core.42697 123456789012345 
Reading symbols from /home/dev/t/123456789012345...done.
[New LWP 42697]
Core was generated by `./123456789012345'.
Program terminated with signal 6, Aborted.
#0  0x00007f0be67631d7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56    return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt
#0  0x00007f0be67631d7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f0be67648c8 in __GI_abort () at abort.c:90
#2  0x0000000000400543 in func4 (f=1) at do_abort.c:3
#3  0x000000000040055f in func3 (f=1) at do_abort.c:4
#4  0x0000000000400576 in func2 (f=1) at do_abort.c:5
#5  0x000000000040058d in func1 (f=1) at do_abort.c:6
#6  0x000000000040059d in main () at do_abort.c:7
(gdb) quit
[~/t]$rm core.42697 
[~/t]$
[~/t]$mv 123456789012345 1234567890123456
[~/t]$./1234567890123456 
Aborted (core dumped)
[~/t]$ll core*
-rw-------. 1 dev wheel 240K Apr 22 03:20 core.42721
[~/t]$gdb -q -c core.42721 1234567890123456 
Reading symbols from /home/dev/t/1234567890123456...done.

warning: core file may not match specified executable file.
[New LWP 42721]
Core was generated by `./1234567890123456'.
Program terminated with signal 6, Aborted.
#0  0x00007f5b271fa1d7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56    return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt
#0  0x00007f5b271fa1d7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f5b271fb8c8 in __GI_abort () at abort.c:90
#2  0x0000000000400543 in func4 (f=1) at do_abort.c:3
#3  0x000000000040055f in func3 (f=1) at do_abort.c:4
#4  0x0000000000400576 in func2 (f=1) at do_abort.c:5
#5  0x000000000040058d in func1 (f=1) at do_abort.c:6
#6  0x000000000040059d in main () at do_abort.c:7
(gdb) quit

[~/t]$mv 1234567890123456 123456789012345
[~/t]$gdb -q -c core.42721 123456789012345 
Reading symbols from /home/dev/t/123456789012345...done.
[New LWP 42721]
Core was generated by `./1234567890123456'.
Program terminated with signal 6, Aborted.
#0  0x00007f5b271fa1d7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56    return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt
#0  0x00007f5b271fa1d7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f5b271fb8c8 in __GI_abort () at abort.c:90
#2  0x0000000000400543 in func4 (f=1) at do_abort.c:3
#3  0x000000000040055f in func3 (f=1) at do_abort.c:4
#4  0x0000000000400576 in func2 (f=1) at do_abort.c:5
#5  0x000000000040058d in func1 (f=1) at do_abort.c:6
#6  0x000000000040059d in main () at do_abort.c:7
(gdb) quit

Following through the gdb source code I discovered that the ELF core file structure only reserves sixteen bytes to hold the executable filename, pr_fname[16], including the nul terminator (reference):

  35 struct elf_external_linux_prpsinfo32_ugid32
  36   {
  37     char pr_state;                      /* Numeric process state.  */
  38     char pr_sname;                      /* Char for pr_state.  */
  39     char pr_zomb;                       /* Zombie.  */
  40     char pr_nice;                       /* Nice val.  */
  41     char pr_flag[4];                    /* Flags.  */
  42     char pr_uid[4];
  43     char pr_gid[4];
  44     char pr_pid[4];
  45     char pr_ppid[4];
  46     char pr_pgrp[4];
  47     char pr_sid[4];
  48     char pr_fname[16];                  /* Filename of executable.  */
  49     char pr_psargs[80];                 /* Initial part of arg list.  */
  50   };

The "warning: core file may not match specified executable file." warning will be issued by gdb when the name of the executable passed on the command-line to gdb doesn't match the value stored in pr_fname[] in the core file (references here, here, and here).

Using the demonstration I showed at the start of this answer, when the filename is 1234567890123456 the filename stored in the core file as pr_fname[] is 123456789012345 (truncated to 15 characters). If gdb is started using gdb -c core.XXXX 1234567890123456 then the warning will be issued. If gdb is started using gdb -c core.XXXX 123456789012345 then the warning will not be issued.

It should follow that in the example from the original question, if segfault.stripped was renamed to segfault.stripp and gdb was run using gdb ./segfault.stripp /tmp/core.segfault.stripp.11 then the warning should not be issued.