Why is Linux thread locking so slow?

[Jeff Sturm]

Godmar Back wrote:
> 
> >
> > My earlier question stands: why does boehm-gc use spinlocks and not
> > pthread mutexes, or condition variables? Was it a deliberate decision, or
> > ignorance on the part of the Linux porters?  That choice suprised me a
> > little, since garbage collection can run for long durations and cause
> > extensive spinning on waiting processors.
> 
> The version of boehm I downloaded a while ago says this:

[deleted]

> Furthermore, note that this lock protects both allocation and collection.
> Allocation is non-blocking and quick.
> If a collection happens ("GC_collecting" is set) the code jumps right
> to the "yield:" label where it does not spin.  So the long durations of
> gc is not an issue.  (At least in the version I'm looking at - 4.13alpha3)

I see... I hadn't considered allocation locks.  Besides, the GC attempts
to suspend all threads during collection, so collection locks really
shouldn't need to be used.

> > For that matter, Linux pthreads is optimized for uniprocessors.  It could
> > be improved by reading /proc/cpuinfo on startup to determine if it is
> > running on a SMP kernel.  I hacked libpthread.so once for MP use, and
> > observed up to a 30% speedup on some of my real-world Java code.
> 
> It would be nice to see these hacks.
> What exactly did you change to optimize for MP use?

If I remember right, the changes were simple... mostly I commented out
the sched_yield() stuff in spinlock.c which is ineffective anyway on my
SMP box.  That lets the CPU spin a few times in hope the lock is quickly
released.  It still falls back on nanosleep() for stubborn spinlocks.

My Java application code runs a producer-consumer model in which one
thread writes to a queue object and one or more threads read from it.  I
used short synchronized blocks to access the queue head/tail indexes. 
System calls should be unnecessary on an adequate MP machine, unless the
queue becomes full or empty. Lock contention should be low, though I'd
like to try it with spinlocks alone to see if it runs noticeably better.

> >
> > What exactly do you wish to improve in pthread_mutex_lock?  There are so
> > many variables, such as average latency with or without lock contention,
> > lock concurrency, resource utilization, predictability (ratio between
> > average and worst case), etc.  Clearly the technique you favor depends on
> > whether you are designing for maximum throughput, real-time, or other
> > needs.
> >
> 
> >From what I can see, pthreads already does a good job for latency
> in the uncontended case.  I'd like to see the latency contended case improved.
> I don't care about lock concurrency or any measure of fairness (for now).
> I'm not sure what you mean by resource utilization (do you mean CPU resource
> utilization, i.e., not wasting cycles due to spinning?)  I think I'd care
> about that too.

Yes, resource utilization means CPU and/or bus contention... I find that
the system bus becomes a significant bottleneck on my quad-PPro test
machine.

The benchmarks I posted yesterday were intended to reveal the limits of
the hardware... whatever scheme is used to share read/write access to a
data object amongst two threads, whether system calls are used or not,
bus locking and cache synchronization must take place somewhere.  Two
threads competing for a critical section seem to run an order of
magnitude slower than a single thread (3443 iters/msec vs. 414).  I
don't think it is possible to improve on that, not on this hardware
anyway... some more advanced SMP machines will likely do better.  (The
mutex numbers are artifically high because LinuxThread mutexes inhibit
sharing on MP machines.)

Oddly, I could not duplicate Matt Welsh's results, which were down in
the double-digits I believe.

> Basically, all I want is to not have applications with multiple threads and
> highly contended locks (java runtime systems) run slower on two processors
> than on one processor.  If they don't run twice as fast, I can live with that.

That might be impossible without reducing lock contention, which (as
Xavier said) must be done in the application.  Or use a cooperative
thread model, which defeats the purpose of SMP for multithreaded
applications.

-- 
Jeff Sturm
jsturm@sigma6.com