This is the mail archive of the
gcc@gcc.gnu.org
mailing list for the GCC project.
Re: Benchmarking theory
- To: David Konerding <dek_ml at konerding dot com>
- Subject: Re: Benchmarking theory
- From: "Joseph S. Myers" <jsm28 at cam dot ac dot uk>
- Date: Sun, 27 May 2001 15:40:34 +0100 (BST)
- cc: Toon Moene <toon at moene dot indiv dot nluug dot nl>, <gcc at gcc dot gnu dot org>
On Sat, 26 May 2001, David Konerding wrote:
> I think Joe's point is that people aren't doing real statistics on the
> results. For example, with just 2 data points (time on one compiler
> and the other) you don't have what is known as "power" to distinguish
> whether the difference is actually significant or not. I certainly
Yes. It seems to me - as a mathematician, not a computer scientist - that
benchmark results should be quoted with quantitative results of
statistical significance tests, even if the release criteria don't state
significance levels at which performance must not have decreased. But I
presume that at least some of the people quoting results without this
information are computer scientists, and so know what they're doing with a
solid basis in computer science theory.
--
Joseph S. Myers
jsm28@cam.ac.uk