gdb how to break in new thread when debugging multi threaded daemon program on linux

Question

Before asking this question - I referred to How does gdb multi-thread debugging coordinate with Linux thread scheduling? for added context

I have a c++ daemonized multithreaded program running on linux. It launches a thread every time It processes a transaction . The code to be debugged is part of the transaction processing code invoked in new thread.

This is the process I follow to debug.

I start gdb, turn off pagination, turn on synchronous command execution and set non-stop mode.

Then I attach gdb to running daemon program and set breakpoints on "all" threads (See illustration below) and send a new transaction.

The problem is that gdb seeems to inform me clearly that breakpoint is in thread 1. I don't get a break on the breakpoint in the new thread. A new thread is created when I send a new transaction and exits without breaking at breakpoint.

Please help me understand what is the possible reason (what am I missing)

(gdb) set pagination off
(gdb) set target-async on
(gdb) set non-stop on
(gdb) attach 11067 # <daemon pid>  
(gdb) thread apply all b foo 
Thread 1 (Thread 0x7f94bb9f3740 (LWP 11067)): Breakpoint 1 at <filename> , 
line <line#>
(gdb) c
 Continuing.
 [New Thread 0x7f94b725e700 (LWP 15750)]
 [Thread 0x7f94b725e700 (LWP 15750) exited]

The problem is - it did not break in function "foo" ( my breakpoint )
What is it that I am missing ? How do I instruct gdb to follow new child threads. Isn't "thread apply all" supposed to apply to "all" threads ?

if instead of debugging I were to just determine if my code was called ( i could infer if it ran ok from other symptoms ) would it be easier to just profile the program ? Any comments ? — Yogesh Devi
"It launches a thread every time It processes a transaction" You might want to think about that design. — stark
Are you sure the foo is actually called in the new thread? Try adding abort() inside foo, execute the transaction again. — Employed Russian
b foo should be enough to set breakpoint on all existing threads and all not yet created threads. — ks1322

Yogesh Devi Yogesh Devi · Accepted Answer · 2019-03-11T08:37:35

I found a workaround that works :-)

While I still can NOT get gdb to break at break-point of my interest (occurs in a short lived thread spawed by main thread), I found a way to switch to that thread explicitly soon enough before that thread ended.

The problem is that gdb needs you to explicitly switch to the thread you are interested in, to stop at break-point.

since the thread that I was trying to debug was rather short lived I can not switch to it soon enough (before it finished) .

In addition to break-point in function of my interest I placed another break-point in a logger function that occurred often in all threads and kept continuing ( annoying for sure) till the thread of my interest was spawned and then switched to that explicitly when new thread was spawned and by listing threads.

(gdb) info threads
(gdb) Thread <thread id>

I am posting this so if you have similar problem you may try this workaround.

Your comments, better answers :-) more responses welcome

gdb how to break in new thread when debugging multi threaded daemon program on linux

1 Answers