GDB: find the thread which has locked the mutex

Q. In Linux if a multi threaded code seems hanged how to find which thread has locked the concerned mutex?
Ans:
1. Attach the gdb to the concerned process.

$sudo gdb -p pid

2. Get the information of all the running threads.

(gdb) info threads
 ......
 20 Thread 0x7f3804dde700 (LWP 19453) "XYZ" 0x00007f38db3afb9d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
 19 Thread 0x7f37f82dd700 (LWP 19454) "XYZ" 0x00007f38db3afb9d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
 18 Thread 0x7f37f7adc700 (LWP 19455) "XYZ" 0x00007f38db3af6dd in accept () at ../sysdeps/unix/syscall-template.S:81
 17 Thread 0x7f37f70d0700 (LWP 19460) "XYZ" __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
 16 Thread 0x7f37f68cf700 (LWP 19461) "XYZ" 0x00007f38dbcecc0b in __memp_get_bucket () from /usr/lib/x86_64-linux-gnu/libdb_cxx-5.1.so
 15 Thread 0x7f37f60ce700 (LWP 19463) "XYZ" __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
 14 Thread 0x7f37f58cd700 (LWP 19464) "XYZ" 0x00007f38db3af7eb in __libc_recv (fd=27, buf=0x7f37f58ccd88, n=4, flags=-616892437)
 at ../sysdeps/unix/sysv/linux/x86_64/recv.c:33
 13 Thread 0x7f37f50cc700 (LWP 19466) "XYZ" __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
 12 Thread 0x7f37f48cb700 (LWP 19467) "XYZ" 0x00007f38db3af7eb in __libc_recv (fd=30, buf=0x7f37f48cad88, n=4, flags=-616892437)
 at ../sysdeps/unix/sysv/linux/x86_64/recv.c:33
 .......

3. Lets say thread 17 is not able to lock a mutex. Now check the details of the thread 17.

(gdb) thread 17
(gdb) where
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f38db3aa664 in _L_lock_952 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f38db3aa4c6 in __GI___pthread_mutex_lock (mutex=0x7f37e8004da8) at ../nptl/pthread_mutex_lock.c:114
#3  0x000000000041602f in LinuxMutex::lock (this=0x7f37e8004da0) at linux/mutex.cpp:34

Print the details of the locked mutex. In my case Mutex I’m trying to lock is of type LinuxMutex which in turn is a wrap over class around pthread_mutex_t. So I’m diplaying the class object, which has a private member variable m_mutex, which is of type pthread_mutex_t . In your case, you can directly print the pthread_mutex_t variable itself.

(gdb) p *((LinuxMutex *) 0x7f37e8004da0)
$4 = { = {_vptr.Mutex = 0x433130 }, m_mutex = {__data = {__lock = 2, __count = 1, __owner = 19461, __nusers = 1, __kind = 1,
      __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\002\000\000\000\001\000\000\000\005L\000\000\001\000\000\000\001", '\000' ,
    __align = 4294967298}}

The pthread_mutex_t variable has __data.__owner variable, which indicates the thread ID of the thread which has currently locked the mutex.
In our case the thread which has locked the mutex is 19461.

4. Check again the info threads to find out the thread which has locked the mutex.

(gdb) info threads
 ......
 20 Thread 0x7f3804dde700 (LWP 19453) "XYZ" 0x00007f38db3afb9d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
 19 Thread 0x7f37f82dd700 (LWP 19454) "XYZ" 0x00007f38db3afb9d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
 18 Thread 0x7f37f7adc700 (LWP 19455) "XYZ" 0x00007f38db3af6dd in accept () at ../sysdeps/unix/syscall-template.S:81
 17 Thread 0x7f37f70d0700 (LWP 19460) "XYZ" __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
 16 Thread 0x7f37f68cf700 (LWP 19461) "XYZ" 0x00007f38dbcecc0b in __memp_get_bucket () from /usr/lib/x86_64-linux-gnu/libdb_cxx-5.1.so
 15 Thread 0x7f37f60ce700 (LWP 19463) "XYZ" __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
 14 Thread 0x7f37f58cd700 (LWP 19464) "XYZ" 0x00007f38db3af7eb in __libc_recv (fd=27, buf=0x7f37f58ccd88, n=4, flags=-616892437)
 at ../sysdeps/unix/sysv/linux/x86_64/recv.c:33
 13 Thread 0x7f37f50cc700 (LWP 19466) "XYZ" __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
 12 Thread 0x7f37f48cb700 (LWP 19467) "XYZ" 0x00007f38db3af7eb in __libc_recv (fd=30, buf=0x7f37f48cad88, n=4, flags=-616892437)
 at ../sysdeps/unix/sysv/linux/x86_64/recv.c:33
 .......

So here thread 16 is the thread which has locked the mutex needed by the thread 17.

Advertisements

GDB: Get stack trace of all threads of running process

Q. How to get the stack trace of a running c/c++ process on Unix.

Ans:

1. Attach GDB to a running process:

Get the pid of running process foo. $ps -aef | grep foo
Attach gdb to the pid of process foo
$sudo gdb -p 21884

2. Save the stack trace of all running threads in a log file.

(gdb) set pagination off
(gdb) set logging on
 Copying output to gdb.txt.
(gdb) thread apply all bt

3. Detach gdb from running process

(gdb) detach
 Detaching from program: foo, process 21884

After this you’ll find the stack trace of all the threads of your running process in file text gdb.txt This is extremely useful to debug the hung process.

GDB: print the c++ STL object

For me debugging becomes painful in Unix compared to Windows.

I came across an interesting problem. I had to display the contents of unordered_set of a running process on Ubuntu server. I attached the GDB to running process and tried to display the contents. Here is what I got:

1. Attach GDB to a running process:

Get the pid of running process foo. $ps -aef | grep foo Attach gdb to the pid of process foo $sudo gdb -p pid

2. Display the unordered_set contents

(gdb) print set_contents
$1 = {<std::tr1::__unordered_set<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::tr1::hash<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, false>> = {<std::tr1::_Hashtable<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Identity<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::tr1::hash<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::tr1::__detail::_Mod_range_hashing, std::tr1::__detail::_Default_ranged_hash, std::tr1::__detail::_Prime_rehash_policy, false, true, true>> = {<std::tr1::__detail::_Rehash_base<std::tr1::__detail::_Prime_rehash_policy, std::tr1::_Hashtable<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Identity<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::tr1::hash<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::tr1::__detail::_Mod_range_hashing, std::tr1::__detail::_Default_ranged_hash, std::tr1::__detail::_Prime_rehash_policy, false, true, true> >> = {<No data fields>}, <std::tr1::__detail::_Hash_code_base<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_Identity<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::tr1::hash<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::tr1::__detail::_Mod_range_hashing, std::tr1::__detail::_Default_ranged_hash, false>> = {
        _M_extract = {<std::unary_function<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<No data fields>}, <No data fields>},
        _M_eq = {<std::binary_function<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool>> = {<No data fields>}, <No data fields>},
        _M_h1 = {<std::unary_function<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long>> = {<No data fields>}, <No data fields>},
        _M_h2 = {<No data fields>}}, <std::tr1::__detail::_Map_base<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_Identity<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, true, std::tr1::_Hashtable<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Identity<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::tr1::hash<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::tr1::__detail::_Mod_range_hashing, std::tr1::__detail::_Default_ranged_hash, std::tr1::__detail::_Prime_rehash_policy, false, true, true> >> = {<No data fields>},
      _M_node_allocator = {<__gnu_cxx::new_allocator<std::tr1::__detail::_Hash_node<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, false> >> = {<No data fields>}, <No data fields>}, _M_buckets = 0x1fc6550, _M_bucket_count = 11, _M_element_count = 1, _M_rehash_policy = {
        _M_max_load_factor = 1, _M_growth_factor = 2,
        _M_next_resize = 11}}, <No data fields>}, <No data fields>}

There is no way you can make out what are the contents of STL object with this output.

Then I came across this article: https://sourceware.org/gdb/wiki/STLSupport

Following steps helped me to achieve my goal:

1. Check-out the latest Python libstdc++ printers to a place on your machine. In a local directory, do:
svn co svn://gcc.gnu.org/svn/gcc/trunk/libstdc++-v3/python

2. Add the following to your ~/.gdbinit. The path needs to match where the python module above was checked-out. So if checked out to: /home/maude/gdb_printers/, the path would be as written in the example:

python
import sys
sys.path.insert(0, '/home/maude/gdb_printers/python')
from libstdcxx.v6.printers import register_libstdcxx_printers
register_libstdcxx_printers (None)
end

After this GDB started displaying the contents of unordered_set as expected:

(gdb) print set_contents 
$1 = std::unordered_set with 1 elements = {"2014-03-15/449782CE-10E6-4233-A143-61CB6314E103" }