Patch RFC: Use internal qsort function in libbacktrace

[Ian Lance Taylor]

On Wed, Mar 5, 2014 at 9:17 AM, Ondřej Bílka <neleai@seznam.cz> wrote:
> On Wed, Mar 05, 2014 at 08:05:25AM -0800, Ian Lance Taylor wrote:
>> On Wed, Mar 5, 2014 at 1:25 AM, Richard Biener
>> <richard.guenther@gmail.com> wrote:
>> > On Wed, Mar 5, 2014 at 4:34 AM, Ian Lance Taylor <iant@google.com> wrote:
>> >> The GNU glibc qsort function will call malloc in some cases.  That makes
>> >> it unsuitable for libbacktrace, which is intended to work when called
>> >> from a signal handler.  This patch changes libbacktrace to use an
>> >> internal qsort function.
>> >>
> Another solution is use a malloc wrapper that makes to make malloc signal
> safe. Problem with this solution is that its on application layer, not
> library one. I am trying to add a signal safety to glibc.

I don't see any reasonable way to change libbacktrace to work that
way.  The libbacktrace library may be being invoked by a signal
handler that was run because of a signal that was sent while malloc
was executing.  At that point it is too late for libbacktrace to avoid
calling malloc recursively, which won't work.  It would not be
reasonable for libbacktrace, a small library, to override malloc for
the entire application.

Ian