Debugging Linux Kernel Module

Question

I have built a linux kernel module which helps in migrating TCP socket from one server to another. The module is working perfectly except when the importing server tries to close the migrating socket, the whole server hangs and freezes.

I am not able to find out the root of the problem, I believe it is something beyond my kernel module code. Something I am missing when I am recreating the socket in the importing machine, and initializes its states. It seems that the system is entering an endless loop. But when I close the socket from client side, this problem does not appear at all.

So my question, what is the appropriate way to debug the kernel module and figure out what is going on, why is it freezing? How to dump error messages especially in my case I am not able to see anything, once I close the file descriptor related to the migrated socket in the server side, the machines freezes.

Note: I used printk to print all the values, and I am not able to find something wrong in the code.

If you suspect a deadlock, then you can try using lockdep: stackoverflow.com/questions/20892822/… — brokenfoot
Thanks for your answer. I was reading through lockdep, but for my case the whole machine freezes and I am not able to type anything to check the deadlock. So how can I log these info? I am running my demo on a virtual machine and I assign two CPUs for the machine, but when I close the socket. The whole machine freezes and I am not able to dump anything. So any additional suggestion? — user3849502
If the whole system froze, it might be a kernel crash, for example. You can try to make the system output the logs via serial port (COM) to a file on your host system and see if there is anything special there at the time of freeze. It depends on the virtual machine you use, how to do all this exactly. I personally use this technique in VirtualBox from time to time and it has worked OK so far. — Eugene

askb askb · Accepted Answer · 2014-07-26T05:21:32

Considering your system is freezing, have you checked if your system is under heavy load while migrating the socket, have you looked into any sar reports to confirm this, see if you can take a vmcore (after configuring kdump) and use crash-tool to narrow down the problem. First, install and configure kdump, then you may need add the following lines to /etc/sysctl.conf and running sysctl -p

kernel.hung_task_panic=1
kernel.hung_task_timeout_secs=300

Next get a vmcore/dump of memory: echo 'c' > /proc/sysrq-trigger # ===> 1

If you still have access to the terminal, use the sysrq-trigger to dump all the stack traces of kernel thread in the syslog: echo 't' > /proc/sysrq-trigger

If you system is hung try using the keyboard hot keys Alt+PrintScreen+'c' ====> same as 1

Other things you may want to try out, assuming you would have already tried some of the below: 1. dump_stack() in your code 2. printk(KERN_ALERT "Hello msg %ld", err); add these lines in the code. 3. dmesg -c; dmesg

Debugging Linux Kernel Module

1 Answers