Benchmarking theory

[Joseph S. Myers]

On Sat, 26 May 2001, David Konerding wrote:

> I think Joe's point is that people aren't doing real statistics on the
> results. For example, with just 2 data points (time on one compiler
> and the other) you don't have what is known as "power" to distinguish
> whether the difference is actually significant or not.  I certainly

Yes.  It seems to me - as a mathematician, not a computer scientist - that
benchmark results should be quoted with quantitative results of
statistical significance tests, even if the release criteria don't state 
significance levels at which performance must not have decreased.  But I 
presume that at least some of the people quoting results without this 
information are computer scientists, and so know what they're doing with a 
solid basis in computer science theory.

-- 
Joseph S. Myers
jsm28@cam.ac.uk