Bug 60038

Summary: AddressSanitizer CHECK failed ... "((*tls_addr + *tls_size)) <= ((*stk_addr + *stk_size))" on CentOS 5.10
Product: gcc Reporter: Uroš Bizjak <ubizjak>
Component: sanitizerAssignee: Not yet assigned to anyone <unassigned>
Status: RESOLVED FIXED    
Severity: normal CC: dodji, dvyukov, jakub, kcc
Priority: P3    
Version: 4.9.0   
Target Milestone: ---   
Host: Target: x86_64-linux-gnu
Build: Known to work:
Known to fail: Last reconfirmed:
Attachments: gcc49-pr60038.patch
Proposed patch

Description Uroš Bizjak 2014-02-03 07:53:14 UTC
Many tests in address sanitizer testsuite fail on CentOS 5.10, on x86_64 64bit target:

FAIL: g++.dg/asan/asan_test.C  -O2  AddressSanitizer_PthreadExitTest execution test
FAIL: g++.dg/asan/asan_test.C  -O2  AddressSanitizer_ThreadedStressStackReuseTest execution test
FAIL: g++.dg/asan/asan_test.C  -O2  AddressSanitizer_ThreadNamesTest execution test
FAIL: g++.dg/asan/asan_test.C  -O2  AddressSanitizer_ThreadedTest ThreadedTestSpawn() output pattern test
FAIL: g++.dg/asan/asan_test.C  -O2  AddressSanitizer_ThreadStackReuseTest execution test
FAIL: g++.dg/asan/asan_test.C  -O2  AddressSanitizer_ManyThreadsTest execution test
FAIL: g++.dg/asan/asan_test.C  -O2  AddressSanitizer_ThreadedMallocStressTest execution test
FAIL: g++.dg/asan/deep-thread-stack-1.C  -O0  output pattern test, is ==28946==AddressSanitizer CHECK failed: ../../../../gcc-svn/trunk/libsaniti
zer/sanitizer_common/sanitizer_linux_libcdep.cc:260 "((*tls_addr + *tls_size)) <= ((*stk_addr + *stk_size))" (0x2ba95e3ff240, 0x2ba95e3ff000)
FAIL: g++.dg/asan/deep-thread-stack-1.C  -O1  output pattern test, is ==28972==AddressSanitizer CHECK failed: ../../../../gcc-svn/trunk/libsaniti
zer/sanitizer_common/sanitizer_linux_libcdep.cc:260 "((*tls_addr + *tls_size)) <= ((*stk_addr + *stk_size))" (0x2b904ae5e240, 0x2b904ae5e000)
FAIL: g++.dg/asan/deep-thread-stack-1.C  -O2  output pattern test, is ==28990==AddressSanitizer CHECK failed: ../../../../gcc-svn/trunk/libsaniti
zer/sanitizer_common/sanitizer_linux_libcdep.cc:260 "((*tls_addr + *tls_size)) <= ((*stk_addr + *stk_size))" (0x2b7261726240, 0x2b7261726000)
FAIL: g++.dg/asan/deep-thread-stack-1.C  -O3 -fomit-frame-pointer  output pattern test, is ==29015==AddressSanitizer CHECK failed: ../../../../gc
c-svn/trunk/libsanitizer/sanitizer_common/sanitizer_linux_libcdep.cc:260 "((*tls_addr + *tls_size)) <= ((*stk_addr + *stk_size))" (0x2b7f30b2f240
, 0x2b7f30b2f000)
FAIL: g++.dg/asan/deep-thread-stack-1.C  -O3 -g  output pattern test, is ==29038==AddressSanitizer CHECK failed: ../../../../gcc-svn/trunk/libsan
itizer/sanitizer_common/sanitizer_linux_libcdep.cc:260 "((*tls_addr + *tls_size)) <= ((*stk_addr + *stk_size))" (0x2ae7a3e97240, 0x2ae7a3e97000)
FAIL: g++.dg/asan/deep-thread-stack-1.C  -Os  output pattern test, is ==29078==AddressSanitizer CHECK failed: ../../../../gcc-svn/trunk/libsaniti
zer/sanitizer_common/sanitizer_linux_libcdep.cc:260 "((*tls_addr + *tls_size)) <= ((*stk_addr + *stk_size))" (0x2b8e40e38240, 0x2b8e40e38000)
FAIL: g++.dg/asan/deep-thread-stack-1.C  -O2 -flto -flto-partition=none  output pattern test, is ==29134==AddressSanitizer CHECK failed: ../../..
/../gcc-svn/trunk/libsanitizer/sanitizer_common/sanitizer_linux_libcdep.cc:260 "((*tls_addr + *tls_size)) <= ((*stk_addr + *stk_size))" (0x2b1d04
5f7240, 0x2b1d045f7000)
FAIL: g++.dg/asan/deep-thread-stack-1.C  -O2 -flto  output pattern test, is ==29196==AddressSanitizer CHECK failed: ../../../../gcc-svn/trunk/lib
sanitizer/sanitizer_common/sanitizer_linux_libcdep.cc:260 "((*tls_addr + *tls_size)) <= ((*stk_addr + *stk_size))" (0x2b99bff03240, 0x2b99bff0300
0)

All these fail on the same check, in the same way. The same tests also fail on i686 32bit targets, with different addresses, e.g.:

FAIL: g++.dg/asan/deep-thread-stack-1.C  -O0  output pattern test, is ==2673==AddressSanitizer CHECK failed: ../../../../../gcc-svn/trunk/libsani
tizer/sanitizer_common/sanitizer_linux_libcdep.cc:260 "((*tls_addr + *tls_size)) <= ((*stk_addr + *stk_size))" (0xf6592020, 0xf6592000)
Comment 1 Kostya Serebryany 2014-02-03 08:19:28 UTC
I expect this also happens with the clang version of ASAN, please confirm.

This is related to the hackish way we extract stack and tls bounds,
and CentOS 5.10 may have different version of glibc or some other difference.
What is the version of glibc on your OS?

We are trying to solve it in a general way in glibc
(https://sourceware.org/bugzilla/show_bug.cgi?id=16291),
but that will not happen for the existing distros.
Comment 2 Uroš Bizjak 2014-02-03 08:26:44 UTC
(In reply to Kostya Serebryany from comment #1)
> I expect this also happens with the clang version of ASAN, please confirm.

Unfortunately, I don't have clang installed on this (fairly old) machine, so I'm not able to confirm this issue.

> This is related to the hackish way we extract stack and tls bounds,
> and CentOS 5.10 may have different version of glibc or some other difference.
> What is the version of glibc on your OS?

$ /lib/libc.so.6 
GNU C Library stable release version 2.5, by Roland McGrath et al.
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.1.2 20080704 (Red Hat 4.1.2-54).
Compiled on a Linux 2.6.9 system on 2013-10-08.
Available extensions:
        The C stubs add-on version 2.1.2.
        crypt add-on version 2.1 by Michael Glad and others
        GNU Libidn by Simon Josefsson
        GNU libio by Per Bothner
        NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
        Native POSIX Threads Library by Ulrich Drepper et al
        BIND-8.2.3-T5B
        RT using linux kernel aio
Thread-local storage support included.
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.
Comment 3 Uroš Bizjak 2014-02-03 08:30:23 UTC
> This is related to the hackish way we extract stack and tls bounds,
> and CentOS 5.10 may have different version of glibc or some other difference.
> What is the version of glibc on your OS?

Looking at libsanitizer/sanitizer_common/sanitizer_linux_libcdep.cc

#if defined(__x86_64__) || defined(__i386__)
// sizeof(struct thread) from glibc.
// There has been a report of this being different on glibc 2.11 and 2.13. We
// don't know when this change happened, so 2.14 is a conservative estimate.
#if __GLIBC_PREREQ(2, 14)
const uptr kThreadDescriptorSize = FIRST_32_SECOND_64(1216, 2304);
#else
const uptr kThreadDescriptorSize = FIRST_32_SECOND_64(1168, 2304);
#endif

We probably just have to fill in correct values for glibc 2.5.
Comment 4 Kostya Serebryany 2014-02-03 08:31:03 UTC
> GNU C Library stable release version 2.5

2.5 is way too old. 
You may try to comment out this CHECK and see if the rest works
The main ASAN's functionality will probably not notice the lack of correct data about TLS, but LSAN (LeakSanitizer) may start reporting false positives.
Comment 5 Kostya Serebryany 2014-02-03 08:33:10 UTC
> We probably just have to fill in correct values for glibc 2.5.
This may help. A patch is welcome, please check 
https://code.google.com/p/address-sanitizer/wiki/HowToContribute
Comment 6 Jakub Jelinek 2014-02-03 08:46:55 UTC
(In reply to Kostya Serebryany from comment #5)
> > We probably just have to fill in correct values for glibc 2.5.
> This may help. A patch is welcome, please check 
> https://code.google.com/p/address-sanitizer/wiki/HowToContribute

I have already gathered those values, but nothing happened since then, see
http://gcc.gnu.org/ml/gcc-patches/2013-12/msg00287.html

BTW, you could supposedly use
#include <unistd.h>
...
  char buf[64];
  size_t len = confstr (_CS_GNU_LIBC_VERSION, buf, sizeof buf);
  if (strncmp (buf, "glibc 2.", 8) == 0)
    {
      char *end;
      int minor = strtoul (buf + 8, &end, 10);
      if (end != buf + 8 && (*end == '\0' || *end == '.')
        {
          if (minor <= 3)

        }
    }
Comment 7 Jakub Jelinek 2014-02-03 08:53:05 UTC
(In reply to Jakub Jelinek from comment #6)
BTW, you could supposedly use
#include <unistd.h>
...
  char buf[64];
  size_t len = confstr (_CS_GNU_LIBC_VERSION, buf, sizeof buf);
  int result = 
  if (strncmp (buf, "glibc 2.", 8) == 0)
    {
      char *end;
      int minor = strtoul (buf + 8, &end, 10);
      if (end != buf + 8 && (*end == '\0' || *end == '.')
        {
          if (minor <= 3)
            kThreadDescriptorSize = FIRST_32_SECOND_64(1104, 1696);
          else if (minor == 4)
            kThreadDescriptorSize = FIRST_32_SECOND_64(1120, 1728);
          else if (minor == 5)
            kThreadDescriptorSize = FIRST_32_SECOND_64(1136, 1728);
          else if (minor <= 9)
            kThreadDescriptorSize = FIRST_32_SECOND_64(1136, 1712);
          else if (minor == 10)
            kThreadDescriptorSize = FIRST_32_SECOND_64(1168, 1776);
          else if (minor <= 12)
            kThreadDescriptorSize = FIRST_32_SECOND_64(1168, 2288);
          else
            kThreadDescriptorSize = FIRST_32_SECOND_64(1216, 2304);
        }
    }
Comment 8 Kostya Serebryany 2014-02-03 08:56:21 UTC
(In reply to Jakub Jelinek from comment #6)
>   size_t len = confstr (_CS_GNU_LIBC_VERSION, buf, sizeof buf);
>   if (strncmp (buf, "glibc 2.", 8) == 0)

Yea, such patch is even more welcome. 
I was thinking about using __gnu_get_libc_version, 
to extract the libc version numbers, but confstr (_CS_GNU_LIBC_VERSION
sounds good too.
Comment 9 Jakub Jelinek 2014-02-03 09:26:18 UTC
Created attachment 32021 [details]
gcc49-pr60038.patch

This seems to work for me on glibc 2.17, Uros, can you please try it on your CentOS 5?
Comment 10 Uroš Bizjak 2014-02-03 09:27:51 UTC
Created attachment 32022 [details]
Proposed patch

Jakub's solution from Comment #7 in the form of a patch.

Tested with RUNTESTFLAGS=asan.exp. The patch works for me on CentOS 5.10 without any problem.
Comment 11 Uroš Bizjak 2014-02-03 09:33:02 UTC
(In reply to Jakub Jelinek from comment #9)
> Created attachment 32021 [details]
> gcc49-pr60038.patch
> 
> This seems to work for me on glibc 2.17, Uros, can you please try it on your
> CentOS 5?

Uh, that was quick, I didn't noticed your submission. ;)

I'll test your patch in a moment.
Comment 12 Uroš Bizjak 2014-02-03 10:00:36 UTC
(In reply to Uroš Bizjak from comment #11)
> (In reply to Jakub Jelinek from comment #9)
> > Created attachment 32021 [details]
> > gcc49-pr60038.patch
> > 
> > This seems to work for me on glibc 2.17, Uros, can you please try it on your
> > CentOS 5?
> 
> Uh, that was quick, I didn't noticed your submission. ;)
> 
> I'll test your patch in a moment.

sanitizer re-build and regression test with

RUNTESTFLAGS="--target_board=unix\{,-m32\} asan.exp"

worked without problems on CentOS 5.10.
Comment 13 Uroš Bizjak 2014-02-03 21:22:25 UTC
(In reply to Jakub Jelinek from comment #9)
> Created attachment 32021 [details]
> gcc49-pr60038.patch
> 
> This seems to work for me on glibc 2.17, Uros, can you please try it on your
> CentOS 5?

The patch was bootstrapped and regression tested OK [1] on CentOS 5.10.

[1] http://gcc.gnu.org/ml/gcc-testresults/2014-02/msg00160.html
Comment 14 Kostya Serebryany 2014-02-04 04:18:45 UTC
Landed upstream: http://llvm.org/viewvc/llvm-project?view=revision&revision=200733
Note that the patch is slightly different from Jakub's: it uses memory_order_relaxed.

My understanding is that now is not the best time to do full merge from upstream,
so feel free to commit this patch to GCC separately.
Comment 15 Jakub Jelinek 2014-02-04 07:38:16 UTC
Author: jakub
Date: Tue Feb  4 07:37:44 2014
New Revision: 207452

URL: http://gcc.gnu.org/viewcvs?rev=207452&root=gcc&view=rev
Log:
	PR sanitizer/60038
	* sanitizer_common/sanitizer_linux_libcdep.cc: Include
	sanitizer_atomic.h and unistd.h.
	(kThreadDescriptorSize): Made static, remove initializer and const,
	change type to atomic_uintptr_t.
	(ThreadDescriptorSize): Use confstr(_CS_GNU_LIBC_VERSION, ...) to
	query glibc version, compute kThreadDescriptorSize depending on
	glibc version minor number.
	(GetThreadStackAndTls): Use ThreadDescriptorSize() instead of
	kThreadDescriptorSize directly.

Modified:
    trunk/libsanitizer/ChangeLog
    trunk/libsanitizer/sanitizer_common/sanitizer_linux_libcdep.cc
Comment 16 Jakub Jelinek 2014-02-04 07:42:40 UTC
Hopefully fixed.