This is the mail archive of the gcc-patches@gcc.gnu.org mailing list for the GCC project.

Index Nav:	[Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav:	[Date Prev] [Date Next]	[Thread Prev] [Thread Next]
Other format:	[Raw text]

[PATCH] Fix qsort ordering violation in tree-vrp.c

From: Yuri Gribov <tetra2005 at gmail dot com>
To: GCC Patches <gcc-patches at gcc dot gnu dot org>
Cc: Richard Biener <rguenther at suse dot de>, Klaus Kruse Pedersen <klauskpedersen at rdamicro dot com>
Date: Sat, 15 Jul 2017 09:43:02 +0100
Subject: [PATCH] Fix qsort ordering violation in tree-vrp.c
Authentication-results: sourceware.org; auth=none

Hi,

This is a follow-up on https://gcc.gnu.org/ml/gcc/2017-07/msg00078.html

compare_assert_loc in tree-vrp.c could return unpredictable results
due to implicit conversion of unsigned subtraction to int here:
  return ha - hb;

This could return inconsistent results for objects with the following hashes:
  a: 0xdc8e4f72U
  b: 0x912ab538U
  c: 0x5ae66e3bU
Then
  a > b because (int)(0xdc8e4f72U - 0x912ab538U) > 0
  b > c because (int)(0x912ab538U - 0x5ae66e3bU) > 0
but
  a < c because (int)(0xdc8e4f72U - 0x5ae66e3bU) == (int)0x81a7e137U < 0

Bug was found with https://github.com/yugr/sortcheck

Bootstrapped and regtested in x64, ok for trunk?

-Y

Attachment: fix-order-1.patch
Description: Binary data

Follow-Ups:
- Re: [PATCH] Fix qsort ordering violation in tree-vrp.c
  - From: Richard Biener

Index Nav:	[Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav:	[Date Prev] [Date Next]	[Thread Prev] [Thread Next]