Bug 66742 - abort on sorting list with custom allocator that is not stateless
Summary: abort on sorting list with custom allocator that is not stateless
Status: RESOLVED FIXED
Alias: None
Product: gcc
Classification: Unclassified
Component: libstdc++ (show other bugs)
Version: 4.9.2
: P3 normal
Target Milestone: 12.0
Assignee: Not yet assigned to anyone
URL:
Keywords: patch
: 78371 (view as bug list)
Depends on:
Blocks: 66792
  Show dependency treegraph
 
Reported: 2015-07-02 14:46 UTC by Marc Di Luzio
Modified: 2022-05-06 12:30 UTC (History)
4 users (show)

See Also:
Host:
Target:
Build:
Known to work:
Known to fail:
Last reconfirmed: 2015-07-03 00:00:00


Attachments
main.ii (28.44 KB, text/plain)
2015-07-02 14:46 UTC, Marc Di Luzio
Details
main.ii (28.48 KB, text/plain)
2015-07-03 08:13 UTC, Marc Di Luzio
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Marc Di Luzio 2015-07-02 14:46:05 UTC
Created attachment 35899 [details]
main.ii

When using a custom defined allocator that contains state, sorting a list will cause an abort.

This seems to be due to the __carry list in sort having a null allocator, that then gets compared to the main list's allocator within splice in _M_check_equal_allocators.

GCC information:
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.9/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.9.2-10ubuntu13' --with-bugurl=file:///usr/share/doc/gcc-4.9/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.9 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.9 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.9-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.9.2 (Ubuntu 4.9.2-10ubuntu13) 
COLLECT_GCC_OPTIONS='-v' '-save-temps' '-Wall' '-Wextra' '-std=c++11' '-g' '-shared-libgcc' '-mtune=generic' '-march=x86-64'
 /usr/lib/gcc/x86_64-linux-gnu/4.9/cc1plus -E -quiet -v -imultiarch x86_64-linux-gnu -D_GNU_SOURCE main.cpp -mtune=generic -march=x86-64 -std=c++11 -Wall -Wextra -g -fworking-directory -fpch-preprocess -fstack-protector-strong -Wformat-security -o main.ii
ignoring duplicate directory "/usr/include/x86_64-linux-gnu/c++/4.9"
ignoring nonexistent directory "/usr/local/include/x86_64-linux-gnu"
ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../../x86_64-linux-gnu/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/include/c++/4.9
 /usr/include/x86_64-linux-gnu/c++/4.9
 /usr/include/c++/4.9/backward
 /usr/lib/gcc/x86_64-linux-gnu/4.9/include
 /usr/local/include
 /usr/lib/gcc/x86_64-linux-gnu/4.9/include-fixed
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
COLLECT_GCC_OPTIONS='-v' '-save-temps' '-Wall' '-Wextra' '-std=c++11' '-g' '-shared-libgcc' '-mtune=generic' '-march=x86-64'
 /usr/lib/gcc/x86_64-linux-gnu/4.9/cc1plus -fpreprocessed main.ii -quiet -dumpbase main.cpp -mtune=generic -march=x86-64 -auxbase main -g -Wall -Wextra -std=c++11 -version -fstack-protector-strong -Wformat-security -o main.s
GNU C++ (Ubuntu 4.9.2-10ubuntu13) version 4.9.2 (x86_64-linux-gnu)
	compiled by GNU C version 4.9.2, GMP version 6.0.0, MPFR version 3.1.2-p11, MPC version 1.0.3
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
GNU C++ (Ubuntu 4.9.2-10ubuntu13) version 4.9.2 (x86_64-linux-gnu)
	compiled by GNU C version 4.9.2, GMP version 6.0.0, MPFR version 3.1.2-p11, MPC version 1.0.3
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
Compiler executable checksum: d311308152d93b993d290cf64f65ff6f
COLLECT_GCC_OPTIONS='-v' '-save-temps' '-Wall' '-Wextra' '-std=c++11' '-g' '-shared-libgcc' '-mtune=generic' '-march=x86-64'
 as -v --64 -o main.o main.s
GNU assembler version 2.25 (x86_64-linux-gnu) using BFD version (GNU Binutils for Ubuntu) 2.25
COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.9/:/usr/lib/gcc/x86_64-linux-gnu/4.9/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.9/:/usr/lib/gcc/x86_64-linux-gnu/
LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.9/:/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../../lib/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-v' '-save-temps' '-Wall' '-Wextra' '-std=c++11' '-g' '-shared-libgcc' '-mtune=generic' '-march=x86-64'
 /usr/lib/gcc/x86_64-linux-gnu/4.9/collect2 -plugin /usr/lib/gcc/x86_64-linux-gnu/4.9/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-linux-gnu/4.9/lto-wrapper -plugin-opt=-fresolution=main.res -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc --sysroot=/ --build-id --eh-frame-hdr -m elf_x86_64 --hash-style=gnu --as-needed -dynamic-linker /lib64/ld-linux-x86-64.so.2 -z relro /usr/lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/crt1.o /usr/lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.9/crtbegin.o -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -L/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.9/../../.. main.o -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /usr/lib/gcc/x86_64-linux-gnu/4.9/crtend.o /usr/lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/crtn.o
Comment 1 Daniel Krügler 2015-07-02 19:40:06 UTC
Your definition of operator!= for the allocator test_alloc violates the allocator requirements, which impose that != behaves the same as the negation of == (see Table 28 — Allocator requirements, [allocator.requirements]). This violates causes undefined behaviour. Therefore it seems to me as if this is an invalid bug report.
Comment 2 Jonathan Wakely 2015-07-02 20:47:42 UTC
Indeed. Splicing lists has a precondition that l1.get_allocator() == l2.get_allocator(), and so we check:

  if (l1.get_allocator() != l2.get_allocator())
    abort();

and obviously that fails for your invalid allocator type.
Comment 3 Marc Di Luzio 2015-07-03 08:13:59 UTC
Created attachment 35901 [details]
main.ii

Updating with a valid ==/!= operator pair and state handling, apologies for the original bad test case.
gcc output is the same verbatim as above.

However, sort still aborts with any allocator that provides ==/!= operators based on internal state that is non-default.


sort contains a default constructed list __carry, which will have a different allocator instance to the parent list if an allocator was passed in on construction -
list.tcc:402
        list __carry;

__carry is then spliced with the parent list -
list.tcc:409
	    __carry.splice(__carry.begin(), *this, begin());


A check then occurs if the two allocators are equal -
stl_list.h:1374
	if (this != &__x)
	  _M_check_equal_allocators(__x);

allocator.h:188
      static bool
      _S_do_it(const _Alloc& __one, const _Alloc& __two)
      { return __one != __two; }


This final check will fail if the != operator for the allocator is dependent on internal state comparison, which has differs from the default.

This appears to be incorrect, unless I'm misunderstanding the purpose of the ==/!= operators for allocators?

If this is still considered an issue, I would suggest that __carry be copy constructed.
Comment 4 Marc Di Luzio 2015-07-03 08:20:08 UTC
I've verified with 5.1 as well now, and the abort still occurs
Comment 5 Jonathan Wakely 2015-07-03 10:20:06 UTC
(In reply to Marc Di Luzio from comment #3)
> If this is still considered an issue, I would suggest that __carry be copy
> constructed.

No, it needs to be constructed empty, but with the same allocator:

    list __carry(get_allocator());

Unfortunately so does every element of the array __tmp:

    list __tmp[64] = { get_allocator(), get_allocator(), get_allocator(),...
Comment 6 Jonathan Wakely 2015-07-03 10:29:22 UTC
So we need something ugly like this:

--- a/libstdc++-v3/include/bits/list.tcc
+++ b/libstdc++-v3/include/bits/list.tcc
@@ -398,6 +398,8 @@ _GLIBCXX_BEGIN_NAMESPACE_CONTAINER
          }
       }
 
+#define _GLIBCXX_REPEAT_8(_X) _X, _X, _X, _X, _X, _X, _X, _X
+
   template<typename _Tp, typename _Alloc>
     void
     list<_Tp, _Alloc>::
@@ -407,8 +409,10 @@ _GLIBCXX_BEGIN_NAMESPACE_CONTAINER
       if (this->_M_impl._M_node._M_next != &this->_M_impl._M_node
          && this->_M_impl._M_node._M_next->_M_next != &this->_M_impl._M_node)
       {
-        list __carry;
-        list __tmp[64];
+        list __carry(get_allocator());
+        list __tmp[64] = {
+           _GLIBCXX_REPEAT_8(_GLIBCXX_REPEAT_8(list(get_allocator())))
+       };
         list * __fill = &__tmp[0];
         list * __counter;
 
@@ -484,8 +488,10 @@ _GLIBCXX_BEGIN_NAMESPACE_CONTAINER
        if (this->_M_impl._M_node._M_next != &this->_M_impl._M_node
            && this->_M_impl._M_node._M_next->_M_next != &this->_M_impl._M_node)
          {
-           list __carry;
-           list __tmp[64];
+           list __carry(get_allocator());
+           list __tmp[64] = {
+               _GLIBCXX_REPEAT_8(_GLIBCXX_REPEAT_8(list(get_allocator())))
+           };
            list * __fill = &__tmp[0];
            list * __counter;
 
@@ -511,6 +517,7 @@ _GLIBCXX_BEGIN_NAMESPACE_CONTAINER
            swap(*(__fill - 1));
          }
       }
+#undef _GLIBCXX_REPEAT_8
 
 _GLIBCXX_END_NAMESPACE_CONTAINER
 } // namespace std
Comment 7 Jonathan Wakely 2015-07-03 15:52:44 UTC
A candidate patch has been posted to https://gcc.gnu.org/ml/gcc-patches/2015-07/msg00217.html

With that fix your testcase segfaults, because you allocate n bytes not n*sizeof(Tp).
Comment 8 Jonathan Wakely 2016-11-16 11:29:44 UTC
*** Bug 78371 has been marked as a duplicate of this bug. ***
Comment 9 TC 2016-11-16 22:12:14 UTC
The ugly fix in Comment #6 should be performant, if, well, ugly.

It may be worth considering holding the nodes via a different type. There's no real reason why the temporary holders need to be a `list` or have a copy of the allocator as long as we make sure that everything in there is spliced back into *this when we exit sort() (by exception or normal return).

Speaking of exceptions, merge() appears to have exception safety problems. I'll file a separate bug.
Comment 10 Eric Gallager 2018-02-20 05:18:38 UTC
(In reply to Jonathan Wakely from comment #7)
> A candidate patch has been posted to
> https://gcc.gnu.org/ml/gcc-patches/2015-07/msg00217.html
> 
> With that fix your testcase segfaults, because you allocate n bytes not
> n*sizeof(Tp).

adding "patch" keyword
Comment 11 Eric Gallager 2019-05-20 11:50:08 UTC
Are you still working on this, Jonathan?
Comment 12 Jonathan Wakely 2019-05-20 11:51:30 UTC
Yes.
Comment 13 Alexander Huszagh 2021-04-10 00:53:38 UTC
Any chance we can merge the proposed patch on this? This still triggers in GCC-10.2.1, and the proposed patch works if applied to both `sort()` and `sort(cmp)` implementations in `include/bits/list.tcc`.

Version Info:
gcc (GCC) 10.2.1 20201125 (Red Hat 10.2.1-9)
Comment 14 Jakub Jelinek 2021-04-27 11:37:47 UTC
GCC 11.1 has been released, retargeting bugs to GCC 11.2.
Comment 15 Jonathan Wakely 2021-05-25 20:52:43 UTC
Fixed downstream in my fork: https://gitlab.com/jonathan-wakely/gcc/-/commit/18d78721bf3afaed243252a01a4f4909c17531b2
Comment 16 GCC Commits 2021-10-01 19:40:08 UTC
The master branch has been updated by Jonathan Wakely <redi@gcc.gnu.org>:

https://gcc.gnu.org/g:ff7793bea465019683b3a07d7ffceb6eae22def5

commit r12-4082-gff7793bea465019683b3a07d7ffceb6eae22def5
Author: Jonathan Wakely <jwakely@redhat.com>
Date:   Tue May 25 14:33:15 2021 +0100

    libstdc++: Allow stateful allocators in std::list::sort [PR 66742]
    
    The temporary lists used by std::list::sort are default constructed,
    which means they use default constructed allocators. The sort operation
    is defined in terms of merge and splice operations, which have undefined
    behaviour (and abort) if the allocators do not compare equal. This means
    it is not possible to sort a list that uses an allocator that compares
    unequal to an default constructed allocator.
    
    The solution is to avoid using temporary std::list objects at all. We do
    not need to be able to allocate memory because no nodes are allocated,
    only spliced from one list to another. That means the temporary lists
    don't need an allocator at all, so whether it would compare equal
    doesn't matter.
    
    Instead of temporary std::list objects, we can just use a collection of
    _List_node_base objects that nodes can be spliced onto as needed. Those
    objects are wrapped in a _Scratch_list type that implements the splicing
    and merging operations used by list::sort.
    
    We also don't need to update the list size during the sort, because
    sorting doesn't alter the number of nodes. Although we move nodes in and
    out of the scratch lists, at the end of the function all nodes are back
    in the original std::list and the scratch lists are empty.  So for the
    cxx11 ABI we can avoid the _M_size modifications usually done when
    splicing nodes.
    
    Signed-off-by: Jonathan Wakely <jwakely@redhat.com>
    
    libstdc++-v3/ChangeLog:
    
            PR libstdc++/66742
            * include/bits/list.tcc (list::sort()): Use _Scratch_list
            objects for splicing and merging.
            (list::sort(StrictWeakOrdering)): Likewise.
            * include/bits/stl_list.h (__detail::_Scratch_list): New type.
            * src/c++98/list.cc (_List_node_base::_M_transfer): Add
            assertion for --enable-libstdcxx-debug library.
            * testsuite/23_containers/list/operations/66742.cc: New test.
Comment 17 Jonathan Wakely 2021-10-01 19:56:38 UTC
Fixed for GCC 12.
Comment 18 Jonathan Wakely 2021-11-02 10:16:31 UTC
My fix caused a regression for this (IMHO dumb) case:

#include <list>
struct X {
  bool operator<(X&) /* non-const */ { return false; }
};
struct Cmp {
  bool operator()(X&, X&) /* non-const */ { return false; }
};
int main() {
  std::list<X> l;
  l.sort();
  Cmp c;
  l.sort(c);
}
Comment 19 GCC Commits 2021-11-03 16:51:30 UTC
The master branch has been updated by Jonathan Wakely <redi@gcc.gnu.org>:

https://gcc.gnu.org/g:1e7a269856fd67aff78ac874bec96d31a54b2fd9

commit r12-4873-g1e7a269856fd67aff78ac874bec96d31a54b2fd9
Author: Jonathan Wakely <jwakely@redhat.com>
Date:   Tue Nov 2 10:21:01 2021 +0000

    libstdc++: Fix regression in std::list::sort [PR66742]
    
    The standard does not require const-correct comparisons in list::sort.
    
    libstdc++-v3/ChangeLog:
    
            PR libstdc++/66742
            * include/bits/list.tcc (list::sort): Use mutable iterators for
            comparisons.
            * include/bits/stl_list.h (_Scratch_list::_Ptr_cmp): Likewise.
            * testsuite/23_containers/list/operations/66742.cc: Check
            non-const comparisons.
Comment 20 Jakub Jelinek 2022-05-06 08:29:54 UTC
GCC 12.1 is being released, retargeting bugs to GCC 12.2.
Comment 21 Jonathan Wakely 2022-05-06 12:30:04 UTC
This was fixed for 12.1