gomp - cost of threadprivate data access

[Matevz Tadel]

Hello,

Andrew Pinski wrote:
> On Mon, Mar 16, 2009 at 12:07 PM, Matevz Tadel <matevz.tadel@cern.ch> wrote:
>> The "fortran object-files" are linked into a shared library.
>>
>> I put the output of
>>  gfortran -O2 -g -fno-second-underscore -m64 -fno-f2c -fPIC -DCERNLIB_LXIA64
>>    -DCERNLIB_BLDLIB -DCERNLIB_CZ -DCERNLIB_GFORTRAN -I. -Iminicern -fPIC
>>    -fopenmp -pthread -ffixed-line-length-none -c gtrak/gtelec.F -S -o
>> gtelec.s
>> here:
>>  <http://mtadel.home.cern.ch/mtadel/gtelec.s>
>>
>> There are 142 lines with:
>>        call    __tls_get_addr@PLT
>>
>> Can these calls be transformed during linking phase?
> 
> There is nothing can be done if this is in a shared library as TLS
> (private data) has to be allocated via ld.so at runtime and then
> figured out the offsets also.  This is part of the ABI, there is not
> much can be done really.  Thread local data will always be slower in
> shared libraries than the use in executables.

I've come up with a simple example that reproduces the reported issue:
http://mtadel.home.cern.ch/mtadel/gomp/

In short it is just a loop doing basic operations on a common (3 variables), 
running in 4 threads. It provides dynamically and statically linked executables.

Now, running the test with various optimization levels (and with callgrind) 
seems to bring up even more questions. Here's a report.

In static executable case __tls_get_addr() is never called.
Normal timing tests were run with 2*10^9 loop iterations.
Callgrind timing - 10^6 loop iterations.

-O0
===
Running-time ratio dynamic/static = 2 (with callgrind 17.3)
Callgrind reports 32M calls to __tls_get_addr() (73% time spent there).
Every access to a common variable goes via __tls_get_addr().

The 73% in __tls_get_addr() seems utterly wrong - is this some internal 
callgrind problem?

-O1
===
Running-time ratio dynamic/static = 1.03 (with callgrind 3.7)
Callgrind reports 4M calls to __tls_get_addr() (53% time spent there).
One call to __tls_get_addr() per loop iteration.

The 4M calls to __tls_get_addr() seem to contradict the very small speed 
difference (3%). Again, 53% time reported by valgrind seems completely bogus.

-O2
===
Running-time ratio dynamic/static = 1 (also 1 with callgrind)
Callgrind reports 8 calls to __tls_get_addr().
Looking at assembler code one sees that thread-specific common variable 
addresses were stored in registers.

In full geant3 example the common addresses can not be stored in registers as 
the code is much more complex - there are intermittent function calls and the 
variable / common count is much higher.

However - it also seems that the time usage reports of callgrind are completely 
off - so I'm beginning to doubt (almost) everything.

Any comment appreciated!

Best,
Matevz