来自 linux 核心转储的线程特定数据

Thread specific data from linux core dump

提问by Vishwanath Sungal

How do i get pointer to thread's local storage or thread specific datawhile analyzing core dump for linux?

在分析linux 的核心转储时，如何获得指向线程本地存储或线程特定数据的指针？

i use pthread_setspecific to store some data in the pthread's local stoare.

我使用 pthread_setspecific 将一些数据存储在 pthread 的本地存储中。

my multi threaded program on linux crashed, and i want to see what is stored in current running thread's local storage.

我在 linux 上的多线程程序崩溃了，我想看看当前运行线程的本地存储中存储了什么。

If i get pointer to thread's local storage i can use key to get the data that is stored.

如果我获得指向线程本地存储的指针，我可以使用密钥来获取存储的数据。

Is there a command in gdbto get the pointer to thread's local storage?

gdb 中是否有命令可以获取指向线程本地存储的指针？

回答by Andy Lutomirski

If you're debugging a live program, you can:

如果您正在调试实时程序，您可以：

print pthread_getspecific(i)

If you have access to the pthread_t of the thread, you can:

如果您有权访问线程的 pthread_t，则可以：

print ((struct pthread*)pth)->specific[i/32][i%32]

where i in the index you want and pth is the pthread_t. See nptl/pthread_getspecific.c in the glibc sources.

i 在你想要的索引中，pth 是 pthread_t。请参阅 glibc 源代码中的 nptl/pthread_getspecific.c。

To do this without calling a function, you need to find the struct pthread. On x86-64, it's stored in the fs base, which is set using arch_prctl(ARCH_SET_FS_BASE, ...). I don't know how to access this from gdb, but you can get it with eu-readelf. Run eu-readelf --notes core_fileand look through the records for fs.base. That number is the pthread_t value. (To figure out which one it is, you can match up the pidfield in the same record with the LWP shown in gdb's info threadscommand.)

要在不调用函数的情况下执行此操作，您需要找到结构 pthread。在 x86-64 上，它存储在 fs 基础中，该基础使用 arch_prctl(ARCH_SET_FS_BASE, ...) 设置。我不知道如何从 gdb 访问它，但是您可以通过 eu-readelf 获得它。运行eu-readelf --notes core_file并查看 的记录fs.base。该数字是 pthread_t 值。（要找出它是哪一个，您可以pid将同一记录中的字段与 gdbinfo threads命令中显示的 LWP进行匹配。）

Good luck!

祝你好运！

回答by Tanner Sansbury

As far as I know, there is no command in gdb to get a pointer to data stored via pthread_setspecific(). However, there are a few options to obtain the memory address:

据我所知，gdb 中没有命令可以通过pthread_setspecific(). 但是，有几个选项可以获取内存地址：

• Examine each thread's backtrace, checking each frame to see if the result of pthread_getspecific()is still on the stack.
• Modify existing code to log both thread id and the result of pthread_getspecific().
• Locate the thread-specific data list within the threads internal data, then use the key to find the record that will contain the address. This approach is dependent on the pthread library implementation; thus, it requires either knowledge of the pthread implementation being used or reverse engineering with a bit of patience.

• 检查每个线程的回溯，检查每个帧以查看 的结果pthread_getspecific()是否仍在堆栈中。
• 修改现有代码以记录线程 ID 和pthread_getspecific().
• 在线程内部数据中找到特定于线程的数据列表，然后使用关键字查找将包含该地址的记录。这种方法依赖于 pthread 库的实现；因此，它需要了解正在使用的 pthread 实现或具有一点耐心的逆向工程。

Below is an demonstration with a simple program on a 32-bit machine:

下面是一个在 32 位机器上使用简单程序的演示：

• g++ (GCC) 4.1.2 20080704 (Red Hat 4.1.2-48)
• libpthread-2.5.so
• GNU gdb Red Hat Linux (6.5-25.el5rh)

• g++ (GCC) 4.1.2 20080704（红帽 4.1.2-48）
• libpthread-2.5.so
• GNU gdb Red Hat Linux (6.5-25.el5rh)

---

---

$cat example.cpp
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

void* the_thread( void* ); 
void get_position();

struct position_t
{
  int x;
  int y;
};

namespace {
  pthread_key_t position_key;

  enum {
    NUMBER_OF_THREADS = 2
  };
} // unnamed

int main(int argc, char **argv)
{
  int result = pthread_key_create( &position_key, NULL );
  printf( "pthread_key_create -- key: %u, result: %i\n",
          position_key, result );

  pthread_t threads[NUMBER_OF_THREADS];
  for (unsigned int i = 0; i < NUMBER_OF_THREADS; ++i )
  {
     // Allocate a position per threads.
     position_t* position = new position_t();

     // Set position values.
     position->x = ( 1 + i ) * 11;
     position->y = ( 1 + i ) * 13;

     // Create the thread.
     result = pthread_create( &threads[i], NULL, the_thread, position );
  }

  // Give time for threads to enter their forever loop.
  sleep( 5 );

  // Abort.
  abort();
  return 0; 
}

void* the_thread( void* position )
{
   int result = pthread_setspecific( position_key, position );

   printf( "Thread: 0x%.8x, key: %u, value: 0x%.8x, result: %i\n",
           pthread_self(), position_key, position, result );

   get_position();
   return 0;
}

void get_position()
{
   position_t* position =
        reinterpret_cast< position_t* >( pthread_getspecific( position_key ) );

   printf( "Thread: 0x%.8x, key: %u, position: 0x%.8x, x: %i, y: %i\n",
           pthread_self(), position_key, position, position->x, position->y );

   // Wait forever.
   while( true ) {};
}

$ g++ -g -lpthread example.cpp && gdb -q ./a.out 
Using host libthread_db library "/lib/libthread_db.so.1".
(gdb) r
Starting program: /tmp/a.out 
[Thread debugging using libthread_db enabled]
[New Thread -1209043248 (LWP 17390)]
pthread_key_create -- key: 0, result: 0
[New Thread -1209046128 (LWP 17393)]
Thread: 0xb7ef6b90, key: 0, value: 0x09a35008, result: 0
Thread: 0xb7ef6b90, key: 0, position: 0x09a35008, x: 11, y: 13
[New Thread -1219535984 (LWP 17394)]
Thread: 0xb74f5b90, key: 0, value: 0x09a350b0, result: 0
Thread: 0xb74f5b90, key: 0, position: 0x09a350b0, x: 22, y: 26

Program received signal SIGABRT, Aborted.
[Switching to Thread -1209043248 (LWP 17390)]
0x00377402 in __kernel_vsyscall ()

---

---

Using addresses still on the stack:

使用仍在堆栈中的地址：

(gdb) info threads
  3 Thread -1219535984 (LWP 17394)  get_position () at example.cpp:71
  2 Thread -1209046128 (LWP 17393)  get_position () at example.cpp:71
* 1 Thread -1209043248 (LWP 17390)  0x00377402 in __kernel_vsyscall ()
(gdb) thread 3
[Switching to thread 3 (Thread -1219535984 (LWP 17394))]#0  get_position () 
 at example.cpp:71
71         while( true ) {};
(gdb) list get_position
57
58         get_position();
59         return 0;
60      }
61       
62      void get_position()
63      {
64         position_t* position =
65              reinterpret_cast< position_t* >( pthread_getspecific( 
 position_key ) );
66
(gdb) info locals
position = (position_t *) 0x9a350b0
(gdb) p position->x
 = 22
(gdb) p position->y
 = 26

---

---

Using addresses printed from stdout:

使用从标准输出打印的地址：

(gdb) p ((position_t*)(0x09a350b0))->x 
 = 22
(gdb) p ((position_t*)(0x09a350b0))->y
 = 26

---

---

Locate the thread-specific data list within the threads internal data:

在线程内部数据中找到特定于线程的数据列表：

This approach is much easier if you have the value of keyand pthread_t.

这种方法是容易得多，如果你有值key和pthread_t。

I will introduce details about the pthread implementation I am using as they are needed:

我将在需要时介绍有关我正在使用的 pthread 实现的详细信息：

• pthreadstruct is thread descriptor structure used internally by pthread.
• pthread_create()returns pthread_t, an unsigned int, that contains the address of the associated pthreadstruct.

• pthreadstruct 是 pthread 内部使用的线程描述符结构。
• pthread_create()返回pthread_t, an unsigned int，其中包含关联pthread结构的地址。

First, locate the pthreadstruct for the thread.

首先，找到pthread线程的结构。

(gdb) info threads
* 3 Thread -1219535984 (LWP 17394)  get_position () at example.cpp:71
  2 Thread -1209046128 (LWP 17393)  get_position () at example.cpp:71
  1 Thread -1209043248 (LWP 17390)  0x00377402 in __kernel_vsyscall ()
(gdb) thread 1
[Switching to thread 1 (Thread -1209043248 (LWP 17390))]#0  0x00377402 in
 __kernel_vsyscall ()
(gdb) bt
#0  0x00377402 in __kernel_vsyscall ()
#1  0x0080ec10 in raise () from /lib/libc.so.6
#2  0x00810521 in abort () from /lib/libc.so.6
#3  0x0804880f in main () at example.cpp:47
(gdb) frame 3
#3  0x0804880f in main () at example.cpp:47
47        abort();
(gdb) info locals
result = 0
threads = {3085921168, 3075431312}
(gdb) p/x threads[1]
 = 0xb74f5b90

Ignoring many of the fields, the pthreadstruct definition looks like:

忽略许多字段，pthread结构定义如下所示：

struct pthread
{
  ...
  pid_t tid; // Thread ID (i.e. this thread descriptor).
  pid_t pid; // Process ID.
  ...
  struct pthread_key_data
  {
    uintptr_t seq;
    void *data;
  } specific_1stblock[PTHREAD_KEY_2NDLEVEL_SIZE];
  struct pthread_key_data *specific[PTHREAD_KEY_1STLEVEL_SIZE];
  ...
};

• pthread_key_data.seq: contains the sequence number that should be fairly low and match __pthread_keys[key].seq.
• pthread_key_data.data: contains the value provided to pthread_setspecific()
• pthread.specific_1stblockis a block that is used to store thread specific data before trying to dynamically allocate more blocks.
• pthreadis a two-level array for thread-specific data. Index 0will contain the memory address of pthread.specific_1stblock.
• PTHREAD_KEY_2NDLEVEL_SIZEhas a size of 32.

• pthread_key_data.seq: 包含应该相当低和匹配的序列号__pthread_keys[key].seq。
• pthread_key_data.data: 包含提供给的值 pthread_setspecific()
• pthread.specific_1stblock是用于在尝试动态分配更多块之前存储线程特定数据的块。
• pthread是线程特定数据的两级数组。索引0将包含 的内存地址pthread.specific_1stblock。
• PTHREAD_KEY_2NDLEVEL_SIZE大小为 32。

The definition gives a fairly good idea as to what to look for in memory:

该定义给出了在内存中寻找什么的一个很好的主意：

• An integer with a value of the pthreadmemory address (tid), followed by an integer with the process id (pid). This is helpful to indicate if the memory being examined is a pthreadstruct.
• cancelhandlingand flagsare flags. The specific values are not important. These fields may be helpful because their values are potentially visibly distinguishable from other fields, such as those containing memory addresses or counters.
• specific_1stblockis an array of size 32. If the pthreadstruct has been zero-initialized, then there should repeating 0s for 62~ words because the example code only has one thread-specific data position_keywhich has a size of two words.
• specificis an array containing memory addresses. If the pthreadstruct has been zero-initialized, then there should repeating 0s, but the first value should be the memory address of specific_1stblock.

• 一个带有pthread内存地址值( tid) 的整数，后跟一个带有进程 ID ( pid)的整数。这有助于指示正在检查的内存是否为pthread结构体。
• cancelhandling并且flags是标志。具体值并不重要。这些字段可能会有所帮助，因为它们的值可能与其他字段（例如包含内存地址或计数器的字段）明显不同。
• specific_1stblock是一个大小为 32 的数组。如果pthread结构体已经零初始化，那么应该重复0s 62~个字，因为示例代码只有一个线程特定的数据position_key，它的大小为两个字。
• specific是一个包含内存地址的数组。如果pthread结构体已经被零初始化，那么应该有重复的0s，但第一个值应该是 的内存地址specific_1stblock。

Print a chunk of pthread's memory:

打印一块pthread的内存：

(gdb) p/x *((int*)threads[1])@150
 = {0xb74f5b90, 0x9a350c8, 0xb74f5b90, 0x1, 0x377400, 0x7fb99100,
  0xcb40329e, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xb7ef6bd0,
  0x96b118, 0x43f2, 0x43ee, 0xb74f5be0, 0xffffffec, 0x0, 0x0, 0xb74f5470,
  0x0, 0x1, 0x9a350b0, 0x0 <repeats 62 times>, 0xb74f5bf8,
  0x0 <repeats 31 times>, 0x1000101, 0x0, 0x0, 0x0, 0xc2342345, 0xe0286,
  0x0, 0x0, 0x0, 0x0, 0x0, 0x80486ca, 0x9a350b0, 0x0 <repeats 13 times>, 
  0xb6af5000, 0xa01000}

By analyizing patterns in the memory, some words become good candidates for specific pthread fields:

通过分析内存中的模式，一些词成为特定 pthread 字段的良好候选词：

0xb74f5b90, 0x9a350c8, 0xb74f5b90 (pthread.tid), 0x1, 0x377400 (pthread.pid) ...
0x1, 0x9a350b0, 0x0 <repeats 62 times> (pthread.specific_1stblock) ...
0xb74f5bf8, 0x0 <repeats 31 times>  (pthread.specific)

Some light-weight sanity checks can be done, such as checking if pthread.specific[0]contains the address of pthread.specific_1stblock:

可以进行一些轻量级的完整性检查，例如检查是否pthread.specific[0]包含以下地址pthread.specific_1stblock：

(gdb) p/x *((int*)0xb74f5bf8)@64
 = {0x1, 0x9a350b0, 0x0 <repeats 62 times>} ## matches specific_1stblock

Now that pthread.specifichas been identified, obtain its memory address by counting the word offset from &pthread. In this case, it is 90:

现在pthread.specific已经确定了，通过计算从 的字偏移量来获取其内存地址&pthread。在这种情况下，它是 90：

(gdb) set $specific=(int*)threads[1] + 90

Calculate the first and second index via position_key:

通过position_key以下方式计算第一个和第二个索引：

• The index into the first array is key / PTHREAD_KEY_2NDLEVEL_SIZE.

• The index into the second array is key % PTHREAD_KEY_2NDLEVEL_SIZE.

  (gdb) set $index1=position_key/32
  (gdb) set $index2=position_key%32
  

• 第一个数组的索引是key / PTHREAD_KEY_2NDLEVEL_SIZE。

• 第二个数组的索引是key % PTHREAD_KEY_2NDLEVEL_SIZE。

  (gdb) set $index1=position_key/32
  (gdb) set $index2=position_key%32
  

Locate the pthread_key_datafor position_key:

找到pthread_key_data用于position_key：

(gdb) set $level2=(int*)*($specific + $index1)
(gdb) p/x *($level2 + (2*$index2))@2
 = {0x1, 0x9a350b0}

Thus:

因此：

pthread_key_data.seq = 1
pthread_key_data.data = 0x9a350b0

pthread_key_data.seq = 1
pthread_key_data.data = 0x9a350b0

The first word is the seqwhich should match pthread_key_struct[position_key].seq. Due to dealing with raw memory, __pthread_keyswill be cast to int*and pointer arithmetic will have to occur to account for the sizeof pthread_key_struct:

第一个词是seq哪个应该匹配pthread_key_struct[position_key].seq。由于处理原始内存，__pthread_keys将被强制转换为，int*并且必须进行指针运算以考虑 sizeof pthread_key_struct：

(gdb) p *(&((int*)&__pthread_keys)[2*position_key])@2
 = {1, 0}

Thus:

因此：

pthread_key_struct[position_key].seq = 1
pthread_key_struct[position_key].destr = NULL

pthread_key_struct[position_key].seq = 1
pthread_key_struct[position_key].destr = NULL

The seqnumbers match, so everything looks good. pthread_key_data.datacontains the value that would be returned from pthread_getspecific( position_key ).

该seq数字匹配，所以一切看起来不错。pthread_key_data.data包含将从 返回的值pthread_getspecific( position_key )。

(gdb) set $position=(position_t*)0x9a350b0
(gdb) p $position->x
 = 22
(gdb) p $position->y
 = 26

---

---

It is technically still possible to locate thread-specific data without having knowledge of the keyand pthread_tvalues:

从技术上讲，在不了解key和pthread_t值的情况下仍然可以定位特定于线程的数据：

• If a destructor function was provided to pthread_key_create(), then its memory address may will reside within the __pthread_keysarray. Examine the memory, and calculate the offset and divide by the sizeof pthread_key_struct. This should result in the index, which also happens to be the key:

  void* destr_fn( void* );
  pthread_key_create( key, destr_fn )
  __pthread_keys[key].destr == destr_fn
  

• If the pthread_tis unknown, it may exists within a register on the thread's stack. This may require examining various different memory addresses trying to locate a section in memory that contains the pthreadstruct.

  (gdb) info thread
    3 Thread -1219535984 (LWP 17394)  get_position () at example.cpp:71
    2 Thread -1209046128 (LWP 17393)  get_position () at example.cpp:71
  * 1 Thread -1209043248 (LWP 17390)  0x00377402 in __kernel_vsyscall ()
  (gdb) thread 3
  [Switching to thread 3 (Thread -1219535984 (LWP 17394))]#0 g
   get_position () at example.cpp:71
  71         while( true ) {};
  (gdb) bt
  #0  get_position () at example.cpp:71
  #1  0x0804871d in the_thread (position=0x9a350b0) at example.cpp:58
  #2  0x0095c43b in start_thread () from /lib/libpthread.so.0
  #3  0x008b3fde in clone () from /lib/libc.so.6
  (gdb) frame 2
  #2  0x0095c43b in start_thread () from /lib/libpthread.so.0
  (gdb) info register
  eax            0x3f     63
  ecx            0xb74f52ac       -1219538260
  edx            0x0      0
  ebx            0x96aff4 9875444
  esp            0xb74f53c0       0xb74f53c0
  ebp            0xb74f54a8       0xb74f54a8
  esi            0x0      0
  edi            0xb74f5b90       -1219535984
  eip            0x95c43b 0x95c43b <start_thread+203>
  eflags         0x200286 [ PF SF IF ID ]
  cs             0x73     115
  ss             0x7b     123
  ds             0x7b     123
  es             0x7b     123
  fs             0x0      0
  gs             0x33     51
  

  In this instance, the ediregister contains the address of the pthreadstruct.

• 如果向 提供了析构函数pthread_key_create()，则其内存地址可能会驻留在__pthread_keys数组中。检查内存，计算偏移量并除以 sizeof pthread_key_struct。这应该导致索引，它也恰好是关键：

  void* destr_fn( void* );
  pthread_key_create( key, destr_fn )
  __pthread_keys[key].destr == destr_fn
  

• 如果pthread_t未知，它可能存在于线程堆栈上的寄存器中。这可能需要检查各种不同的内存地址，尝试在内存中定位包含该pthread结构的部分。

  (gdb) info thread
    3 Thread -1219535984 (LWP 17394)  get_position () at example.cpp:71
    2 Thread -1209046128 (LWP 17393)  get_position () at example.cpp:71
  * 1 Thread -1209043248 (LWP 17390)  0x00377402 in __kernel_vsyscall ()
  (gdb) thread 3
  [Switching to thread 3 (Thread -1219535984 (LWP 17394))]#0 g
   get_position () at example.cpp:71
  71         while( true ) {};
  (gdb) bt
  #0  get_position () at example.cpp:71
  #1  0x0804871d in the_thread (position=0x9a350b0) at example.cpp:58
  #2  0x0095c43b in start_thread () from /lib/libpthread.so.0
  #3  0x008b3fde in clone () from /lib/libc.so.6
  (gdb) frame 2
  #2  0x0095c43b in start_thread () from /lib/libpthread.so.0
  (gdb) info register
  eax            0x3f     63
  ecx            0xb74f52ac       -1219538260
  edx            0x0      0
  ebx            0x96aff4 9875444
  esp            0xb74f53c0       0xb74f53c0
  ebp            0xb74f54a8       0xb74f54a8
  esi            0x0      0
  edi            0xb74f5b90       -1219535984
  eip            0x95c43b 0x95c43b <start_thread+203>
  eflags         0x200286 [ PF SF IF ID ]
  cs             0x73     115
  ss             0x7b     123
  ds             0x7b     123
  es             0x7b     123
  fs             0x0      0
  gs             0x33     51
  

  在这种情况下，edi寄存器包含pthread结构的地址。

References: descr.h, pthread_key_create.c, pthread_setspecific.c, pthreadP.h, internaltypes.h

参考：descr.h、pthread_key_create.c、pthread_setspecific.c、pthreadP.h、 internaltypes.h