misbehaviour with md5_process_bytes and maybe in optimization

[Pierre Vittet <piervit at pvittet dot com>]

Hello,

I recently asked for some help as I got a problem when using
md5_process_bytes (in libiberty/md5.c):
http://gcc.gnu.org/ml/gcc-help/2011-09/msg00126.html,
http://gcc.gnu.org/ml/gcc-help/2011-09/msg00127.html and it appears that
there is a bug in md5_process_bytes.

The bug can conduct to a miscomputed md5 result.

It tooks time to me to make the bug reproducible but I was finally able
to do so. The fact is that it only appears in very particular situation.
I have written a small gcc plugin, allowing to reproduce it (see
attachment).
The bad news is that the bug only appears when use libiberty compiled in
-g -O0 (it works well with -O2). It is quite sad, because It could means
another bug in an optimization function.

I have attached a README which detail how to use the plugin and how to
explain the bug. I have tried to explain as good as possible (and I
apologize for my very bad english).

The bug appears when:
	1) We use libiberty compiled with -O0
	2) We first call md5_process_bytes with a less than 64 bits buffer (we
call his size len1).
	3) We make a new call of md5_process_bytes with a buffer which has a
size len2 such as:
	len2 > 127 + 65 (so test in line 228 of md5.C will be true)
	128 -len1 != Mulint with Mulint %  __alignof__ (md5_uint32) != 0 (so
condition on line 238 is true)
	len2 - (128 - len1) = Mul64 and Mul64 such as Mul %64=0 (so the loop of
line 239 is broken with len = 64, this leads to the bug as, line 249,
(len & ~63) = 64 and we shift the buffer without processing the data).


Please, can you reproduce the bug? Is there any useful informations I
can add? Must I contact somebody from libiberty (I don't know the status
of this library (is this part of gcc or from another project?)).

I already sent a patch correcting this issue (it does not correct the
fact that we don't get the bug with an optimised libiberty):
http://gcc.gnu.org/ml/gcc-patches/2011-09/msg01098.html. It has not been
reviewed, could someone reviews this?

Thanks!

Pierre Vittet

Attachment: md5sum_plugin.tar.gz
Description: application/gzip