How to get more detailed backtrace [duplicate]

20
votes

I am trying to print a backtrace when my C++ program terminated. Function printing backtrace is like below;

   void print_backtrace(void){

       void *tracePtrs[10];
       size_t count;

       count = backtrace(tracePtrs, 10);

       char** funcNames = backtrace_symbols(tracePtrs, count);

       for (int i = 0; i < count; i++)
           syslog(LOG_INFO,"%s\n", funcNames[i]);

       free(funcNames);

}

It gives an output like ;

   desktop program: Received SIGSEGV signal, last error is : Success
   desktop program: ./program() [0x422225]
   desktop program: ./program() [0x422371]
   desktop program: /lib/libc.so.6(+0x33af0) [0x7f0710f75af0]
   desktop program: /lib/libc.so.6(+0x12a08e) [0x7f071106c08e]
   desktop program: ./program() [0x428895]
   desktop program: /lib/libc.so.6(__libc_start_main+0xfd) [0x7f0710f60c4d]
   desktop program: ./program() [0x4082c9]

Is there a way to get more detailed backtrace with function names and lines, like gdb outputs?

7
clinux
Do you have a debugging libc installed? IIRC Linux will use a libc with debug symbols in it for this purpose if you pass -g on the command line to GCC. – Matthew Iselin
Why not use gdb, may I ask? Also the Backtraces section of the GNU libc manual looks useful. – Roman A. Taycher

7 Answers

20
votes

Yes - pass the -rdynamic flag to the linker. It will cause the linker to put in the link tables the name of all the none static functions in your code, not just the exported ones.

The price you pay is a very slightly longer startup time of your program. For small to medium programs you wont notice it. What you get is that backtrace() is able to give you the name of all the none static functions in your back trace.

However - BEWARE: there are several gotchas you need to be aware of:

  1. backtrace_symbols allocates memory from malloc. If you got into a SIGSEGV due to malloc arena corruption (quite common) you will double fault here and never see your back trace.

  2. Depending on the platform this runs on (e.g. x86), the address/function name of the exact function where you crashed will be replaced in place on the stack with the return address of the signal handler. You need to get the right EIP of the crashed function from the signal handler parameters for those platforms.

  3. syslog is not an async signal safe function. It might take a lock internally and if that lock is taken when the crash occurred (because you crashed in the middle of another call to syslog) you have a dead lock

If you want to learn all the gory details, check out this video of me giving a talk about it at OLS: http://free-electrons.com/pub/video/2008/ols/ols2008-gilad-ben-yossef-fault-handlers.ogg

4
votes

Feed the addresses to addr2line and it will show you the file name, line number, and function name.

3
votes

If you're fine with only getting proper backtraces when running through valgrind, then this might be an option for you:

VALGRIND_PRINTF_BACKTRACE(format, ...):

It will give you the backtrace for all functions, including static ones.

3
votes

The better option I have found is libbacktrace by Ian Lance Taylor:

https://github.com/ianlancetaylor/libbacktrace

backtrace_symbols() does prints only exported symbols and could not be less portable as it requires the GNU libc.

addr2line is nice as it includes file names and line numbers. But it fails as soon as the loader performs relocations. Nowadays as ASLR is common, it will fail very often.

libunwind alone will not allow one to print file names and line numbers. To do this, one needs to parse DWARF debugging information inside the ELF binary file. This can be done using libdwarf, though. But why bother when libbacktrace gives you everything required for free?

2
votes
  1. Create a pipe
  2. fork()
  3. Make child process execute addr2line
  4. In parent process, convert the addresses returned from backtrace() to hexadecimal
  5. Write the hex addresses to the pipe
  6. Read back the output from addr2line and print/log it

Since you're doing all this from a signal handler, make sure to not use functionality which is not async-signal-safe. You can see a list of async-signal-safe POSIX functions here.

1
votes

If you don't want to take the "signal a different process that runs gdb on you" approach, which I think gby is advocating, you can also slightly alter your code to call open() on a crash log file and then backtrace_symbols_fd() with the fd returned by open() - both functions are async signal safe according to the glibc manual. You'll need still -rdynamic, of course. Also, from what I've seen, you still sometimes need to run addr2line on some addresses that the backtrace*() functions won't be able to decode.

Also note fork() is not async signal safe: http://article.gmane.org/gmane.linux.man/1893/match=fork+async, at least not on Linux. Neither is syslog(), as somebody already pointed out.

0
votes

If ou want a very detailled backtrace, you should use ptrace(2) to trace the process you want the backtrace.

You will be able to see all functions your process used but you need some basic asm knowledge