This is the mail archive of the mailing list for the GCC project.

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]

RE: Benchmarking theory

I'm not sure I want to jump into *this* thread, although I did reply to the
post with actual numbers. As I said there, for all practical purposes, on
that benchmark set, the two compilers have identical performance. That said,
there is plenty of theory behind benchmarking, both mathematical and
computer science. I personally think compiler writers waste a lot of effort
on fancy optimizers at the cost of hideously complex compilers and time to
market (if that concept means anything with free software :-). I didn't say
any performance payoff in the data presented here.

M. Edward (Ed) Borasky, Chief Scientist, Borasky Research

If there's nothing to astrology, how come so many famous men were born on

> -----Original Message-----
> From: Joseph S. Myers []
> Sent: Sunday, May 27, 2001 7:41 AM
> To: David Konerding
> Cc: Toon Moene;
> Subject: Re: Benchmarking theory
> On Sat, 26 May 2001, David Konerding wrote:
> > I think Joe's point is that people aren't doing real statistics on the
> > results. For example, with just 2 data points (time on one compiler
> > and the other) you don't have what is known as "power" to distinguish
> > whether the difference is actually significant or not.  I certainly
> Yes.  It seems to me - as a mathematician, not a computer scientist - that
> benchmark results should be quoted with quantitative results of
> statistical significance tests, even if the release criteria don't state
> significance levels at which performance must not have decreased.  But I
> presume that at least some of the people quoting results without this
> information are computer scientists, and so know what they're
> doing with a
> solid basis in computer science theory.
> --
> Joseph S. Myers

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]