Re: The size of thread level storage under OpenMP.

[Toon Moene <toon at moene dot indiv dot nluug dot nl>]

Jakub Jelinek wrote:

> say ulimit -s 2097152 for 2GB stacks.  On 32-bit arches you should use
> ulimit -s not much larger than what you actually need for each threads,
> because otherwise you'll run out of virtual address space.

Well, I've tried everything from OMP_NUM_THREADS 1 .. 4 and ulimit -s 
100000 .. 2000000, but I always hit a heap corruption (I even tried with 
a computational domain 12 times smaller than the one I wanted - same 
results).

The following is typical:

hirlam@super:~/hl_home/EXP$ *** glibc detected *** 
/scratch/hirlam/hl_home/EXP/lib/linuxgfortran/bin/hlprog.x: free(): 
invalid next size (fast): 0x00000000054f7c20 ***
======= Backtrace: =========
/lib/libc.so.6[0x2b8c2479301d]
/lib/libc.so.6(cfree+0x76)[0x2b8c24794d26]
/usr/rel/lib64/libgomp.so.1[0x2b8c240ee7c8]
/scratch/hirlam/hl_home/EXP/lib/linuxgfortran/bin/hlprog.x[0x53235c]
/scratch/hirlam/hl_home/EXP/lib/linuxgfortran/bin/hlprog.x[0x50e819]
/scratch/hirlam/hl_home/EXP/lib/linuxgfortran/bin/hlprog.x[0x4b5fb9]
/scratch/hirlam/hl_home/EXP/lib/linuxgfortran/bin/hlprog.x[0x44e165]
/scratch/hirlam/hl_home/EXP/lib/linuxgfortran/bin/hlprog.x[0x4ef9f0]
/scratch/hirlam/hl_home/EXP/lib/linuxgfortran/bin/hlprog.x[0x4b50a2]
/scratch/hirlam/hl_home/EXP/lib/linuxgfortran/bin/hlprog.x[0x41574a]
/scratch/hirlam/hl_home/EXP/lib/linuxgfortran/bin/hlprog.x[0x40239a]
/scratch/hirlam/hl_home/EXP/lib/linuxgfortran/bin/hlprog.x[0x68879c]
/lib/libc.so.6(__libc_start_main+0xf4)[0x2b8c247421c4]
/scratch/hirlam/hl_home/EXP/lib/linuxgfortran/bin/hlprog.x[0x401cf9]
======= Memory map: ========
00400000-0069c000 r-xp 00000000 08:11 18204392 
 /scratch/hirlam/hl_home/EXP/lib/linuxgfortran/bin/hlprog.x
0089c000-0089d000 rw-p 0029c000 08:11 18204392 
 /scratch/hirlam/hl_home/EXP/lib/linuxgfortran/bin/hlprog.x
0089d000-06311000 rw-p 0089d000 00:00 0 
 [heap]
2b8c236b8000-2b8c236d3000 r-xp 00000000 08:02 6176877 
 /lib/ld-2.7.so
2b8c236d3000-2b8c2388f000 rw-p 2b8c236d3000 00:00 0
2b8c238d2000-2b8c238d4000 rw-p 0001a000 08:02 6176877 
 /lib/ld-2.7.so
2b8c238d4000-2b8c2398b000 r-xp 00000000 08:02 2075006 
 /usr/lib/libfftw3.so.3.1.2
2b8c2398b000-2b8c23b8b000 ---p 000b7000 08:02 2075006 
 /usr/lib/libfftw3.so.3.1.2
2b8c23b8b000-2b8c23b91000 rw-p 000b7000 08:02 2075006 
 /usr/lib/libfftw3.so.3.1.2
2b8c23b91000-2b8c23c69000 r-xp 00000000 08:02 2377403 
 /usr/rel/lib64/libgfortran.so.3.0.0
2b8c23c69000-2b8c23e68000 ---p 000d8000 08:02 2377403 
 /usr/rel/lib64/libgfortran.so.3.0.0
2b8c23e68000-2b8c23e6a000 rw-p 000d7000 08:02 2377403 
 /usr/rel/lib64/libgfortran.so.3.0.0
2b8c23e6a000-2b8c23ee8000 r-xp 00000000 08:02 6176885 
 /lib/libm-2.7.so
2b8c23ee8000-2b8c240e7000 ---p 0007e000 08:02 6176885 
 /lib/libm-2.7.so
2b8c240e7000-2b8c240e9000 rw-p 0007d000 08:02 6176885 
 /lib/libm-2.7.so
2b8c240e9000-2b8c240ea000 rw-p 2b8c240e9000 00:00 0
2b8c240ea000-2b8c240f1000 r-xp 00000000 08:02 2377445 
 /usr/rel/lib64/libgomp.so.1.0.0
2b8c240f1000-2b8c242f1000 ---p 00007000 08:02 2377445 
 /usr/rel/lib64/libgomp.so.1.0.0
2b8c242f1000-2b8c242f2000 rw-p 00007000 08:02 2377445 
 /usr/rel/lib64/libgomp.so.1.0.0
2b8c242f2000-2b8c24308000 r-xp 00000000 08:02 2377400 
 /usr/rel/lib64/libgcc_s.so.1
2b8c24308000-2b8c24507000 ---p 00016000 08:02 2377400 
 /usr/rel/lib64/libgcc_s.so.1
2b8c24507000-2b8c24508000 rw-p 00015000 08:02 2377400 
 /usr/rel/lib64/libgcc_s.so.1
2b8c24508000-2b8c2451e000 r-xp 00000000 08:02 6177960 
 /lib/libpthread-2.7.so
2b8c2451e000-2b8c2471d000 ---p 00016000 08:02 6177960 
 /lib/libpthread-2.7.so
2b8c2471d000-2b8c2471f000 rw-p 00015000 08:02 6177960 
 /lib/libpthread-2.7.so
2b8c2471f000-2b8c24724000 rw-p 2b8c2471f000 00:00 0
2b8c24724000-2b8c24862000 r-xp 00000000 08:02 6176880 
 /lib/libc-2.7.so
2b8c24862000-2b8c24a62000 ---p 0013e000 08:02 6176880 
 /lib/libc-2.7.so
2b8c24a62000-2b8c24a65000 r--p 0013e000 08:02 6176880 
 /lib/libc-2.7.so
2b8c24a65000-2b8c24a67000 rw-p 00141000 08:02 6176880 
 /lib/libc-2.7.so
2b8c24a67000-2b8c24a6c000 rw-p 2b8c24a67000 00:00 0
2b8c24a6c000-2b8c24a74000 r-xp 00000000 08:02 6177974 
 /lib/librt-2.7.so
2b8c24a74000-2b8c24c73000 ---p 00008000 08:02 6177974 
 /lib/librt-2.7.so
2b8c24c73000-2b8c24c75000 rw-p 00007000 08:02 6177974 
 /lib/librt-2.7.so
2b8c24c75000-2b8c31893000 rw-p 2b8c24c75000 00:00 0
2b8c34000000-2b8c34021000 rw-p 2b8c34000000 00:00 0
2b8c34021000-2b8c38000000 ---p 2b8c34021000 00:00 0
7fff871eb000-7fff873f2000 rw-p 7fffffdf8000 00:00 0 
 [stack]
7fff873fe000-7fff87400000 r-xp 7fff873fe000 00:00 0 
 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 
 [vsyscall]

Do you have an idea how to debug this ?

Thanks in advance,

--
Toon Moene - e-mail: toon@moene.indiv.nluug.nl - phone: +31 346 214290
Saturnushof 14, 3738 XG  Maartensdijk, The Netherlands
At home: http://moene.indiv.nluug.nl/~toon/
Progress of GNU Fortran: http://gcc.gnu.org/ml/gcc/2008-01/msg00009.html