Bug 71291

Summary:	Firefox with GCC reports stack-buffer-overflow but clang does not
Product:	gcc	Reporter:	Georg Koppen <gk>
Component:	sanitizer	Assignee:	Not yet assigned to anyone <unassigned>
Status:	RESOLVED INVALID
Severity:	normal	CC:	dodji, jakub, kcc, tavianator
Priority:	P3
Version:	5.2.0
Target Milestone:	---
Host:		Target:
Build:		Known to work:
Known to fail:		Last reconfirmed:
Attachments:	Patch for building mozilla-central with GCC and ASan ASan stack trace .ii file mozconfig

Description Georg Koppen 2016-05-26 11:48:10 UTC

Created attachment 38572 [details]
Patch for building mozilla-central with GCC and ASan

I recently reported a bug to the Mozilla team about a reproducible stack buffer overflow. To my surprise this ASan report only happens with GCC. Using different clang versions is working fine. The Mozilla folks concluded this is a GCC bug and closed the report as WORKSFORME.

I've tested with GCC 5.2.0/5.3.1 and with the latest code on gcc-6-branch (which contains the latest ASan changes in GCC if I got that right) both Firefox 45 ESR and latest mozilla-central. The ASan crash is always reproducible for me.

STR:

1) Build Firefox with a .mozconfig file like:

. $topsrcdir/browser/config/mozconfig

export CFLAGS="-fsanitize=address -Dxmalloc=myxmalloc"
export CXXFLAGS="-fsanitize=address -Dxmalloc=myxmalloc"
# We need to add -ldl explicitely due to bug 1213698
export LDFLAGS="-fsanitize=address -ldl"

ac_add_options --enable-address-sanitizer
ac_add_options --disable-jemalloc
ac_add_options --disable-elf-hack

(if you build mozilla-central you need the attached asan.patch as well)

2) Go to http://lab.hakim.se/meny/ and move with the mouse to the left corner

3) The build crashes with a srack-buffer-overflow report.

Comment 1 Georg Koppen 2016-05-26 11:50:32 UTC

I wonder if there are any good things I could do to debug this problem before looking at the stack trace itself. Any ideas?

Comment 2 Jakub Jelinek 2016-05-26 11:59:46 UTC

You haven't mentioned any details, like in which function it is, the exact diagnostic you get, reference to the upstream bugreport, preprocessed source + g++ command line for the problematic source file, ...

Comment 3 Georg Koppen 2016-05-26 12:22:51 UTC

Created attachment 38573 [details]
ASan stack trace

This is https://bugzilla.mozilla.org/show_bug.cgi?id=1268854 and attached is the ASan stack trace to begin with.

Comment 4 Georg Koppen 2016-05-26 13:32:52 UTC

Created attachment 38574 [details]
.ii file

Here comes the .ii file.

Comment 5 Georg Koppen 2016-05-26 13:58:16 UTC

And that's the g++ command line:

/usr/bin/g++ -std=gnu++11 -o Unified_cpp_gfx_layers2.o -c -I/home/thomas/Arbeit/Tor/mozilla-central/obj-x86_64-pc-linux-gnu/dist/stl_wrappers -I/home/thomas/Arbeit/Tor/mozilla-central/obj-x86_64-pc-linux-gnu/dist/system_wrappers -include /home/thomas/Arbeit/Tor/mozilla-central/config/gcc_hidden.h -DNDEBUG=1 -DTRIMMED=1 -DGOOGLE_PROTOBUF_NO_RTTI -DOS_POSIX=1 -DOS_LINUX=1 -DSTATIC_EXPORTABLE_JS_API -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -I/home/thomas/Arbeit/Tor/mozilla-central/gfx/layers -I/home/thomas/Arbeit/Tor/mozilla-central/obj-x86_64-pc-linux-gnu/gfx/layers -I/home/thomas/Arbeit/Tor/mozilla-central/obj-x86_64-pc-linux-gnu/ipc/ipdl/_ipdlheaders -I/home/thomas/Arbeit/Tor/mozilla-central/ipc/chromium/src -I/home/thomas/Arbeit/Tor/mozilla-central/ipc/glue -I/home/thomas/Arbeit/Tor/mozilla-central/docshell/base -I/home/thomas/Arbeit/Tor/mozilla-central/layout/base -I/home/thomas/Arbeit/Tor/mozilla-central/layout/generic -I/home/thomas/Arbeit/Tor/mozilla-central/gfx/skia -I/home/thomas/Arbeit/Tor/mozilla-central/gfx/skia/skia/include/config -I/home/thomas/Arbeit/Tor/mozilla-central/gfx/skia/skia/include/core -I/home/thomas/Arbeit/Tor/mozilla-central/gfx/skia/skia/include/gpu -I/home/thomas/Arbeit/Tor/mozilla-central/gfx/skia/skia/include/utils -I/home/thomas/Arbeit/Tor/mozilla-central/obj-x86_64-pc-linux-gnu/dist/include  -I/home/thomas/Arbeit/Tor/mozilla-central/obj-x86_64-pc-linux-gnu/dist/include/nspr -I/home/thomas/Arbeit/Tor/mozilla-central/obj-x86_64-pc-linux-gnu/dist/include/nss       -fPIC  -DMOZILLA_CLIENT -include /home/thomas/Arbeit/Tor/mozilla-central/obj-x86_64-pc-linux-gnu/mozilla-config.h -MD -MP -MF .deps/Unified_cpp_gfx_layers2.o.pp  -Wall -Wc++11-compat -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wwrite-strings -Wc++14-compat -Wno-invalid-offsetof -Wno-error=maybe-uninitialized -Wno-error=deprecated-declarations -Wno-error=array-bounds -fsanitize=address -Dxmalloc=myxmalloc -fno-exceptions -fno-strict-aliasing -fno-rtti -fno-exceptions -fno-math-errno -pthread -pipe  -g -freorder-blocks -Os -fno-omit-frame-pointer   -I/home/thomas/Arbeit/Tor/mozilla-central/obj-x86_64-pc-linux-gnu/dist/include/cairo -I/home/thomas/Arbeit/Tor/mozilla-central/widget/gtk/compat-gtk3 -pthread -I/usr/include/gtk-3.0/unix-print -I/usr/include/gtk-3.0 -I/usr/include/at-spi2-atk/2.0 -I/usr/include/at-spi-2.0 -I/usr/include/dbus-1.0 -I/usr/lib/x86_64-linux-gnu/dbus-1.0/include -I/usr/include/gtk-3.0 -I/usr/include/gio-unix-2.0/ -I/usr/include/cairo -I/usr/include/pango-1.0 -I/usr/include/harfbuzz -I/usr/include/pango-1.0 -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libpng16 -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/libpng16 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -Wno-error=shadow  /home/thomas/Arbeit/Tor/mozilla-central/obj-x86_64-pc-linux-gnu/gfx/layers/Unified_cpp_gfx_layers2.cpp

Let me know if you need something more or I could do something to help tracking this down.

Comment 6 Martin Liška 2016-05-27 08:36:14 UTC

(In reply to Georg Koppen from comment #3)
> Created attachment 38573 [details]
> ASan stack trace
> 
> This is https://bugzilla.mozilla.org/show_bug.cgi?id=1268854 and attached is
> the ASan stack trace to begin with.

Even thought I'm logged in, I'm not authorized to see the issue.
Can you please past the stack trace here?

Comment 7 Georg Koppen 2016-05-27 08:44:59 UTC

(In reply to Martin Liška from comment #6)
> (In reply to Georg Koppen from comment #3)
> > Created attachment 38573 [details]
> > ASan stack trace
> > 
> > This is https://bugzilla.mozilla.org/show_bug.cgi?id=1268854 and attached is
> > the ASan stack trace to begin with.
> 
> Even thought I'm logged in, I'm not authorized to see the issue.
> Can you please past the stack trace here?

It's already attached to this bug: https://gcc.gnu.org/bugzilla/attachment.cgi?id=38573.

Comment 8 Georg Koppen 2016-05-29 20:33:35 UTC

Looking at the stack trace again. It says "Memory access at offset 112 underflows this variable" yet ASan reports stack-buffer-overflow. I am confused. Shouldn't it report a stack-buffer-underflow then?

Comment 9 Maxim Ostapenko 2016-05-31 17:03:27 UTC

Hm, looking to corresponding source code (dist/include/mozilla/gfx/Matrix.h):

 705   template<class F>
 706   size_t TransformAndClipRect(const RectTyped<SourceUnits, F>& aRect,
 707                               const RectTyped<TargetUnits, F>& aClip,
 708                               PointTyped<TargetUnits, F>* aVerts) const
 709   {
 710     // Initialize a double-buffered array of points in homogenous space 
 711     // with the input rectangle, aRect.
 712     Point4DTyped<UnknownUnits, F> points[2]kTransformAndClipRectMaxVerts];
 713     Point4DTyped<UnknownUnits, F>* dstPoint = points[0];
............
 727     // Iterate through each clipping plane and clip the polygon.
 728     // In each pass, we double buffer, alternating between points[0] and
 729     // points[1].
 730     for (int plane=0; plane < 4; plane++) {
 731       planeNormals[plane].Normalize();
 732 
 733       Point4DTyped<UnknownUnits, F>* srcPoint = points[plane & 1];
 734       Point4DTyped<UnknownUnits, F>* srcPointEnd = dstPoint;
 735       dstPoint = points[~plane & 1];
 736 
 737       Point4DTyped<UnknownUnits, F>* prevPoint = srcPointEnd - 1;
 738       F prevDot = planeNormals[plane].DotProduct(*prevPoint);

............

I suspect this scenario to happen:

1) On iteration 2 (i == 1) dstPoint becomes points[0] at line 735.
2) Later on iteration 1 dstPoint doesn't change for some reason.
3) On iteration 3 (i == 2) srcPointEnd becomes srcPointEnd = dstPoint (== point[0]) at line 734.
4) Later on iteration 3 prevPoint = srcPointEnd - 1 (point[-1]) at line 737.
5) At line 738 we use *prevPoint (points[-1]) that leads to ASan report (valid, because points[-1] overflows).

Could you check this? If this is what happens, than ASan is innocent and something else went wrong here.

Comment 10 Maxim Ostapenko 2016-05-31 18:50:45 UTC

I've build Firefox locally with clang with optimizations disabled (CFLAGS="-fsanitize=address -fsanitize-recover=address -O0") and got pretty the same backtrace:

==12707==ERROR: AddressSanitizer: stack-buffer-underflow on address 0x7fe22dbc417c at pc 0x7fe25e3b9ee5 bp 0x7fe22dbc40b0 sp 0x7fe22dbc40a8
READ of size 4 at 0x7fe22dbc417c thread T39 (Compositor)
    #0 0x7fe25e3b9ee4 in mozilla::gfx::BasePoint4D<float, mozilla::gfx::Point4DTyped<mozilla::gfx::UnknownUnits, float> >::DotProduct(mozilla::gfx::Point4DTyped<mozilla::gfx::UnknownUnits, float> const&) const /home/max/src/firefox/gfx/2d/BasePoint4D.h:101:68
    #1 0x7fe25e3b7e6e in unsigned long mozilla::gfx::Matrix4x4Typed<mozilla::gfx::UnknownUnits, mozilla::gfx::UnknownUnits>::TransformAndClipRect<float>(mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits, float> const&, mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits, float> const&, mozilla::gfx::PointTyped<mozilla::gfx::UnknownUnits, float>*) const /home/max/src/firefox/gfx/2d/Matrix.h:738:19
    #2 0x7fe25e3a2329 in mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits, float> mozilla::gfx::Matrix4x4Typed<mozilla::gfx::UnknownUnits, mozilla::gfx::UnknownUnits>::TransformAndClipBounds<float>(mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits, float> const&, mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits, float> const&) const /home/max/src/firefox/gfx/2d/Matrix.h:675:24
    #3 0x7fe25ea08163 in mozilla::layers::BasicCompositor::DrawQuad(mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits, float> const&, mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits, float> const&, mozilla::layers::EffectChain const&, float, mozilla::gfx::Matrix4x4Typed<mozilla::gfx::UnknownUnits, mozilla::gfx::UnknownUnits> const&, mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits, float> const&) /home/max/src/firefox/gfx/layers/basic/BasicCompositor.cpp:311:23
    #4 0x7fe25e888778 in mozilla::layers::Compositor::DrawQuad(mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits, float> const&, mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits, float> const&, mozilla::layers::EffectChain const&, float, mozilla::gfx::Matrix4x4Typed<mozilla::gfx::UnknownUnits, mozilla::gfx::UnknownUnits> const&) /home/max/src/firefox/objdir-ff-asan-O0/dist/include/mozilla/layers/Compositor.h:331:7
    #5 0x7fe25eb0b14f in mozilla::layers::ColorLayerComposite::RenderLayer(mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const&)::$_1::operator()(mozilla::layers::EffectChain&, mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits, float> const&) const /home/max/src/firefox/gfx/layers/composite/ColorLayerComposite.cpp:31:5
    #6 0x7fe25ead8bdd in void mozilla::layers::RenderWithAllMasks<mozilla::layers::ColorLayerComposite::RenderLayer(mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const&)::$_1>(mozilla::layers::Layer*, mozilla::layers::Compositor*, mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const&, mozilla::layers::ColorLayerComposite::RenderLayer(mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const&)::$_1) /home/max/src/firefox/gfx/layers/composite/LayerManagerComposite.h:616:5
    #7 0x7fe25ead8437 in mozilla::layers::ColorLayerComposite::RenderLayer(mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const&) /home/max/src/firefox/gfx/layers/composite/ColorLayerComposite.cpp:28:3
    #8 0x7fe25eb749e6 in void mozilla::layers::RenderLayers<mozilla::layers::ContainerLayerComposite>(mozilla::layers::ContainerLayerComposite*, mozilla::layers::LayerManagerComposite*, mozilla::gfx::IntRectTyped<mozilla::RenderTargetPixel> const&) /home/max/src/firefox/gfx/layers/composite/ContainerLayerComposite.cpp:662:7
...................
Address 0x7fe22dbc417c is located in stack of thread T39 (Compositor) at offset 28 in frame
    #0 0x7fe25e3b6b8f in unsigned long mozilla::gfx::Matrix4x4Typed<mozilla::gfx::UnknownUnits, mozilla::gfx::UnknownUnits>::TransformAndClipRect<float>(mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits, float> const&, mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits, float> const&, mozilla::gfx::PointTyped<mozilla::gfx::UnknownUnits, float>*) const /home/max/src/firefox/gfx/2d/Matrix.h:709

  This frame has 20 object(s):
    [32, 1056) 'points' <== Memory access at offset 28 underflows this variable
    [1184, 1200) 'ref.tmp'
    [1216, 1232) 'ref.tmp2'
    [1248, 1264) 'ref.tmp4'
    [1280, 1296) 'ref.tmp5'
    [1312, 1328) 'ref.tmp11'
    [1344, 1360) 'ref.tmp12'
    [1376, 1392) 'ref.tmp18'
    [1408, 1424) 'ref.tmp19'
    [1440, 1504) 'planeNormals'
    [1536, 1552) 'ref.tmp32'
    [1568, 1584) 'ref.tmp35'
    [1600, 1616) 'ref.tmp38'
    [1632, 1648) 'ref.tmp42'
    [1664, 1680) 'ref.tmp68'
    [1696, 1712) 'coerce'
    [1728, 1744) 'ref.tmp71'
    [1760, 1768) 'p'
    [1792, 1800) 'ref.tmp99'
    [1824, 1832) 'ref.tmp100'
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-underflow /home/max/src/firefox/gfx/2d/BasePoint4D.h:101:68 in mozilla::gfx::BasePoint4D<float, mozilla::gfx::Point4DTyped<mozilla::gfx::UnknownUnits, float> >::DotProduct(mozilla::gfx::Point4DTyped<mozilla::gfx::UnknownUnits, float> const&) const
Shadow bytes around the buggy address:
  0x0ffcc5b707d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ffcc5b707e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ffcc5b707f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ffcc5b70800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ffcc5b70810: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0ffcc5b70820: 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 f1[f1]
  0x0ffcc5b70830: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ffcc5b70840: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ffcc5b70850: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ffcc5b70860: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ffcc5b70870: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb

So, I'm pretty sure this is a bug in Firefox code, not in GCC. I don't know why clang didn't detect this on -O2 level (optimized out?), perhaps you'll want file an issue in LLVM bugzilla. Anyway, I believe GCC is fine here.

Comment 11 Georg Koppen 2016-05-31 19:05:39 UTC

Thanks a lot! I've informed the Mozilla people and updated their bug. Sorry for the noise.

Comment 12 Georg Koppen 2016-06-01 12:12:03 UTC

(In reply to Maxim Ostapenko from comment #10)
> I've build Firefox locally with clang with optimizations disabled
> (CFLAGS="-fsanitize=address -fsanitize-recover=address -O0") and got pretty
> the same backtrace:

What clang version/firefox version and build commands did you use for getting a clang build that crashes? Using the clang I have on my Debian machine + mozilla central does not seem to be enough.

Comment 13 Maxim Ostapenko 2016-06-01 12:20:26 UTC

Created attachment 38620 [details]
mozconfig

This .mozconfig with current trunk clang 3.9.0. The source code I've downloaded from here:

$ hg clone https://hg.mozilla.org/mozilla-central/ firefox