Re: libstdc++-v3 PATCH: STL configuration change

[Loren James Rittle <rittle at latour dot rsch dot comm dot mot dot com>]

In article <Pine.SOL.3.91.1010530113608.7768B-100000@taarna.cygnus.com>,
Benjamin Kosnik <bkoz@redhat.com> writes:

> I would appreciate it if you'd add links to your last message and
> initial performance analysis in the faq/howtos.

> Please check this patch in, to both trunk and branch.

Done and committed.  I moved more information to
docs/html/23_containers/howto.html and docs/html/17_intro/howto.html.
I attempted to better express the problem with changing the internal
STL ABI with user flags and header file manipulation.

> I'd love it if some kind of regression went it to test this stuff, but 
> I'm not quite sure how to do this.

I am not sure either.  Since gcc doesn't expose a standard threading
interface to application code, it will be somewhat hard to do in
general.  Even if it did, as you know, anything involving threading
can appear non-deterministic.  Although I am trying very hard to break
the POSIX dependency in libstdc++-v3 before gcc 3.0 is released, I
think a testsuite for POSIX threads as used with libstdc++-v3 and the
STL would be in order.  I will give some thought to this.

> We really need to think of a new catagory of testsuite cases, that
> test for memory leaks (the string memory issue comes immediately to
> mind as another test case in this category). [...]

Yup.

Here is what I committed to mainline and then carefully moved to the
3.0 branch.

2001-05-30  Loren J. Rittle  <ljrittle@acm.org>

	* include/bits/c++config (__USE_MALLOC): Do not define it.
	Document why not and give pointers to more information.

	* docs/html/23_containers/howto.html: Update documentation
	to reflect recent understanding of problem.
	* docs/html/17_intro/howto.html: Likewise.

Index: docs/html/23_containers/howto.html
===================================================================
RCS file: /cvs/gcc/egcs/libstdc++-v3/docs/html/23_containers/howto.html,v
retrieving revision 1.4
diff -c -r1.4 howto.html
*** howto.html	2001/05/30 21:55:01	1.4
--- howto.html	2001/05/31 02:36:32
***************
*** 199,204 ****
--- 199,225 ----
        (&quot;For most clients,&quot;...), pointing
        out that locking must nearly always be done outside the container,
        by client code (that'd be you, not us *grin*).
+       <EM>However, please take caution when considering the discussion
+       about the user-level configuration of the mutex lock
+       implementation inside the STL container-memory allocator on that
+       page.  For the sake of this discussion, libstdc++-v3 configures
+       the SGI STL implementation, not you.  We attempt to configure
+       the mutex lock as is best for your platform.  In particular,
+       past advice was for people using g++ to explicitly define
+       _PTHREADS on the command line to get a thread-safe STL.  This
+       may or may not be required for your port.  It may or may not be
+       a good idea for your port.  Extremely big caution: if you
+       compile some of your application code against the STL with one
+       set of threading flags and macros and another portion of the
+       code with different flags and macros that influence the
+       selection of the mutex lock, you may well end up with multiple
+       locking mechanisms in use which don't impact each other in the
+       manner that they should.  Everything might link and all code
+       might have been built with a perfectly reasonable thread model
+       but you may have two internal ABIs in play within the
+       application.  This might produce races, memory leaks and fatal
+       crashes.  In any multithreaded application using STL, this
+       is the first thing to study well before blaming the allocator.</EM>
     </P>
     <P>You didn't read it, did you?  *sigh*  I'm serious, go read the
        SGI page.  It's really good and doesn't take long, and makes most
Index: docs/html/17_intro/howto.html
===================================================================
RCS file: /cvs/gcc/egcs/libstdc++-v3/docs/html/17_intro/howto.html,v
retrieving revision 1.4
diff -c -r1.4 howto.html
*** howto.html	2001/05/30 21:54:57	1.4
--- howto.html	2001/05/31 02:36:32
***************
*** 67,73 ****
--- 67,115 ----
        means for a library (not a general program).  We currently use the
        <A HREF="http://www.sgi.com/tech/stl/thread_safety.html";>same
        definition that SGI</A> uses for their STL subset.
+       <EM>Please see the many cautions given in HOWTOs on containers.</EM>
     </P>
+    <P>
+       Here is another attempt at explaining the dangers of using the
+       STL with threading support without understanding some important
+       details.  The STL implementation is currently configured to use
+       the high-speed caching memory allocator.  If you absolutely
+       think you must change this on a global basis for your platform
+       to support multi-threading, then please consult all commentary
+       in include/bits/c++config and the HOWTOs on containers.  Be
+       fully aware that you may change the external or internal ABI of
+       libstdc++-v3 when you provide -D__USE_MALLOC on the command line
+       or make a change to that configuration file. [Placeholder in
+       case other patches don't make it before the 3.0 release: That
+       memory allocator can appear buggy in multithreaded C++ programs
+       (and has been reported to leak memory), if STL is misconfigured
+       for your platform.  You may need to provide -D_PTHREADS on the
+       command line in this case to ensure the memory allocator for
+       containers is really protected by a mutex.  Also, be aware that
+       you just changed the ABI of libstdc++-v3 when you did that thus
+       your entire application and all libraries must be compiled with
+       compatible flags.  The STL implementation doesn't currently
+       protect you from changing the mutex locking implementation to
+       one that doesn't really play together with the implementation
+       you may have compiled other application code with.]
+    </P>
+    <P>
+       If you don't like caches of objects being retained inside the
+       STL, then you might be tempted to define __USE_MALLOC either on
+       the command line or by rebuilding c++config.h.  Please note,
+       once you define __USE_MALLOC, only the malloc allocator is
+       visible to application code (i.e. the typically higher-speed
+       allocator is not even available in this configuration).  There
+       is a better way: It is possible to force the malloc-based
+       allocator on a per-case-basis for some application code even
+       when the above macro symbol is not defined.  The author of this
+       comment believes that is a better way to tune an application for
+       high-speed using this implementation of the STL.  Here is one
+       possible example displaying the forcing of the malloc-based
+       allocator over the typically higher-speed default allocator:
+ 
+       std::list <void*, std::malloc_alloc> my_malloc_based_list;
+    </P>
     <P>A recent journal article has described &quot;atomic integer
        operations,&quot; which would allow us to, well, perform updates
        on integers atomically, and without requiring an explicit mutex
***************
*** 82,88 ****
        &quot;Thread Next&quot; to move down the thread.  This farm is in
        latest-to-oldest order.
        <UL>
!         <LI>
        </UL>
        <BR>
        Here are discussions that took place before the current snapshot;
--- 124,136 ----
        &quot;Thread Next&quot; to move down the thread.  This farm is in
        latest-to-oldest order.
        <UL>
!         <LI><A HREF="http://gcc.gnu.org/ml/libstdc++/2001-05/msg00384.html";>
! 	inspired this most recent updating of issues with threading
! 	and the SGI STL library.  It also contains some example
! 	POSIX-multithreaded STL code.</A>
! 	<LI> <A HREF="http://gcc.gnu.org/ml/libstdc++/2001-05/msg00136.html";>
! 	an early analysis of why __USE_MALLOC should be disable for
! 	the 3.0 release of libstdc++.</A>
        </UL>
        <BR>
        Here are discussions that took place before the current snapshot;
Index: include/bits/c++config
===================================================================
RCS file: /cvs/gcc/egcs/libstdc++-v3/include/bits/c++config,v
retrieving revision 1.24
diff -c -r1.24 c++config
*** c++config	2001/05/24 20:51:33	1.24
--- c++config	2001/05/31 02:36:32
***************
*** 101,109 ****
  # define __STL_UNWIND(action) 
  #endif
  
! // This is the "underlying allocator" for STL.  The alternatives are
! // homegrown schemes involving a kind of mutex and free list; see stl_alloc.h.
! #define __USE_MALLOC
  
  // The remainder of the prewritten config is mostly automatic; all the
  // user hooks are listed above.
--- 101,114 ----
  # define __STL_UNWIND(action) 
  #endif
  
! // Default to the typically high-speed, pool-based allocator (as
! // libstdc++-v2) instead of the malloc-based allocator (libstdc++-v3
! // snapshots).  See libstdc++-v3/docs/html/17_intro/howto.html for
! // details on why you don't want to override this setting.  Ensure
! // that threads are properly configured on your platform before
! // assigning blame to the STL container-memory allocator.  After doing
! // so, please report any possible issues to libstdc++@gcc.gnu.org .
! // Do not blindly #define __USE_MALLOC here or on the command line.
  
  // The remainder of the prewritten config is mostly automatic; all the
  // user hooks are listed above.