[Bug sanitizer/71291] Firefox with GCC reports stack-buffer-overflow but clang does not

Tue May 31 19:17:00 GMT 2016

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71291

--- Comment #10 from Maxim Ostapenko <m.ostapenko at samsung dot com> ---
I've build Firefox locally with clang with optimizations disabled
(CFLAGS="-fsanitize=address -fsanitize-recover=address -O0") and got pretty the
same backtrace:

==12707==ERROR: AddressSanitizer: stack-buffer-underflow on address
0x7fe22dbc417c at pc 0x7fe25e3b9ee5 bp 0x7fe22dbc40b0 sp 0x7fe22dbc40a8
READ of size 4 at 0x7fe22dbc417c thread T39 (Compositor)
    #0 0x7fe25e3b9ee4 in mozilla::gfx::BasePoint4D<float,
mozilla::gfx::Point4DTyped<mozilla::gfx::UnknownUnits, float>
>::DotProduct(mozilla::gfx::Point4DTyped<mozilla::gfx::UnknownUnits, float>
const&) const /home/max/src/firefox/gfx/2d/BasePoint4D.h:101:68
    #1 0x7fe25e3b7e6e in unsigned long
mozilla::gfx::Matrix4x4Typed<mozilla::gfx::UnknownUnits,
mozilla::gfx::UnknownUnits>::TransformAndClipRect<float>(mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits,
float> const&, mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits, float>
const&, mozilla::gfx::PointTyped<mozilla::gfx::UnknownUnits, float>*) const
/home/max/src/firefox/gfx/2d/Matrix.h:738:19
    #2 0x7fe25e3a2329 in mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits,
float> mozilla::gfx::Matrix4x4Typed<mozilla::gfx::UnknownUnits,
mozilla::gfx::UnknownUnits>::TransformAndClipBounds<float>(mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits,
float> const&, mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits, float>
const&) const /home/max/src/firefox/gfx/2d/Matrix.h:675:24
    #3 0x7fe25ea08163 in
mozilla::layers::BasicCompositor::DrawQuad(mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits,
float> const&, mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits, float>
const&, mozilla::layers::EffectChain const&, float,
mozilla::gfx::Matrix4x4Typed<mozilla::gfx::UnknownUnits,
mozilla::gfx::UnknownUnits> const&,
mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits, float> const&)
/home/max/src/firefox/gfx/layers/basic/BasicCompositor.cpp:311:23
    #4 0x7fe25e888778 in
mozilla::layers::Compositor::DrawQuad(mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits,
float> const&, mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits, float>
const&, mozilla::layers::EffectChain const&, float,
mozilla::gfx::Matrix4x4Typed<mozilla::gfx::UnknownUnits,
mozilla::gfx::UnknownUnits> const&)
/home/max/src/firefox/objdir-ff-asan-O0/dist/include/mozilla/layers/Compositor.h:331:7
    #5 0x7fe25eb0b14f in
mozilla::layers::ColorLayerComposite::RenderLayer(mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits>
const&)::$_1::operator()(mozilla::layers::EffectChain&,
mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits, float> const&) const
/home/max/src/firefox/gfx/layers/composite/ColorLayerComposite.cpp:31:5
    #6 0x7fe25ead8bdd in void
mozilla::layers::RenderWithAllMasks<mozilla::layers::ColorLayerComposite::RenderLayer(mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits>
const&)::$_1>(mozilla::layers::Layer*, mozilla::layers::Compositor*,
mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const&,
mozilla::layers::ColorLayerComposite::RenderLayer(mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits>
const&)::$_1)
/home/max/src/firefox/gfx/layers/composite/LayerManagerComposite.h:616:5
    #7 0x7fe25ead8437 in
mozilla::layers::ColorLayerComposite::RenderLayer(mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits>
const&) /home/max/src/firefox/gfx/layers/composite/ColorLayerComposite.cpp:28:3
    #8 0x7fe25eb749e6 in void
mozilla::layers::RenderLayers<mozilla::layers::ContainerLayerComposite>(mozilla::layers::ContainerLayerComposite*,
mozilla::layers::LayerManagerComposite*,
mozilla::gfx::IntRectTyped<mozilla::RenderTargetPixel> const&)
/home/max/src/firefox/gfx/layers/composite/ContainerLayerComposite.cpp:662:7
...................
Address 0x7fe22dbc417c is located in stack of thread T39 (Compositor) at offset
28 in frame
    #0 0x7fe25e3b6b8f in unsigned long
mozilla::gfx::Matrix4x4Typed<mozilla::gfx::UnknownUnits,
mozilla::gfx::UnknownUnits>::TransformAndClipRect<float>(mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits,
float> const&, mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits, float>
const&, mozilla::gfx::PointTyped<mozilla::gfx::UnknownUnits, float>*) const
/home/max/src/firefox/gfx/2d/Matrix.h:709

  This frame has 20 object(s):
    [32, 1056) 'points' <== Memory access at offset 28 underflows this variable
    [1184, 1200) 'ref.tmp'
    [1216, 1232) 'ref.tmp2'
    [1248, 1264) 'ref.tmp4'
    [1280, 1296) 'ref.tmp5'
    [1312, 1328) 'ref.tmp11'
    [1344, 1360) 'ref.tmp12'
    [1376, 1392) 'ref.tmp18'
    [1408, 1424) 'ref.tmp19'
    [1440, 1504) 'planeNormals'
    [1536, 1552) 'ref.tmp32'
    [1568, 1584) 'ref.tmp35'
    [1600, 1616) 'ref.tmp38'
    [1632, 1648) 'ref.tmp42'
    [1664, 1680) 'ref.tmp68'
    [1696, 1712) 'coerce'
    [1728, 1744) 'ref.tmp71'
    [1760, 1768) 'p'
    [1792, 1800) 'ref.tmp99'
    [1824, 1832) 'ref.tmp100'
HINT: this may be a false positive if your program uses some custom stack
unwind mechanism or swapcontext
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-underflow
/home/max/src/firefox/gfx/2d/BasePoint4D.h:101:68 in
mozilla::gfx::BasePoint4D<float,
mozilla::gfx::Point4DTyped<mozilla::gfx::UnknownUnits, float>
>::DotProduct(mozilla::gfx::Point4DTyped<mozilla::gfx::UnknownUnits, float>
const&) const
Shadow bytes around the buggy address:
  0x0ffcc5b707d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ffcc5b707e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ffcc5b707f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ffcc5b70800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ffcc5b70810: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0ffcc5b70820: 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 f1[f1]
  0x0ffcc5b70830: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ffcc5b70840: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ffcc5b70850: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ffcc5b70860: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ffcc5b70870: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb

So, I'm pretty sure this is a bug in Firefox code, not in GCC. I don't know why
clang didn't detect this on -O2 level (optimized out?), perhaps you'll want
file an issue in LLVM bugzilla. Anyway, I believe GCC is fine here.