debugging threads vs processes

[Weiqi Gao]

On 12 Jul 2001 17:20:29 -0700, Per Bothner wrote:
> 
> (Slightly off-topic, I admit ...)
> 
> I'm trying to debug Java modules loaded into Apache 2.
> Thing seem to work ok with the "prefork" module, but when
> I configure Apache with the "threaded" module I can only
> "reload" a few dozen times before the server freezes.
> While trying to figure out what is going on, I'm getting
> a bit confused wrt process ids vs thread ids.

The LinuxThread pthreads implementation (bundled with glibc on Red Hat
5, 6, 7) uses slots in the process table to record threads.  The results
of ps and top listings are a consequence of this.  Another consequence
is that creating too many Java threads will fill up the process table.

> One problem is how I can get a list of running processes vs threads?
> On my RedHat 7.1 (Pentium3) ps -m says "Thread display not
> implemented."  My impression is that both ps and top
> list all threads, not just all processes.  This is my
> conclusion from looking at various 'top' displays which show
> a bunch of "processes" using more memory in lock-step,
> which only seems plausible of these "processes" are
> actually threads in the same process.  But it sure would be
> nice to have a way of showing threads vs processes more clearly.
> 
> Another confusing thing is that getpid seems to return s thread id,
> not a process id.  What leads me to believe this is the following
> debug output from running Apache: (Each line results from one request
> to the server from the browser.)
> 
> handle: text/html, pid:16694,prev:0, created:0
> handle: text/html, pid:16694,prev:16694, created:1
> handle: text/html, pid:16694,prev:16694, created:1
> handle: text/html, pid:16697,prev:16694, created:1
> handle: text/html, pid:16697,prev:16697, created:1
> handle: text/html, pid:16697,prev:16697, created:1
> handle: text/html, pid:16697,prev:16697, created:1
> handle: text/html, pid:16697,prev:16697, created:1
> handle: text/html, pid:16697,prev:16697, created:1
> handle: text/html, pid:16697,prev:16697, created:1
> handle: text/html, pid:16697,prev:16697, created:1
> handle: text/html, pid:16697,prev:16697, created:1
> handle: text/html, pid:16697,prev:16697, created:1
> handle: text/html, pid:16697,prev:16697, created:1
> 
> The pid field is the result of getpid(), the prev field is the value
> saved by the last request in a static variable.  ps -H claims that
> 16694 and 16697 are siblings, not parent/child.  But then how can
> 16697 remember a static value set by 16694?  Unless they are two
> threads in the same process, in which case getpid is returning the
> thread id, rather than a process id.
> 
> Does someone on this list understand Linux threads enough to explain
> what is going on and/or suggest useful debugging tools?

Nathan Meyers' book Java Programming on Linux gives several suggestions
for distinguishing threads from processes:

"... The short answer is that you can't."

Hints:
+ Identical memory size.
+ /proc/<pid>/maps are identical for threads from the same process.
+ /proc/<pid>/status report identical values for all the Vm* fields.
+ /proc/<pid>/environ match, /proc/<pid>/fds/ match.

-- 
Weiqi Gao
weiqigao@networkusa.net