This is the mail archive of the gcc@gcc.gnu.org mailing list for the GCC project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]

RE: Benchmarking theory


I'm not sure I want to jump into *this* thread, although I did reply to the
post with actual numbers. As I said there, for all practical purposes, on
that benchmark set, the two compilers have identical performance. That said,
there is plenty of theory behind benchmarking, both mathematical and
computer science. I personally think compiler writers waste a lot of effort
on fancy optimizers at the cost of hideously complex compilers and time to
market (if that concept means anything with free software :-). I didn't say
any performance payoff in the data presented here.

--
M. Edward (Ed) Borasky, Chief Scientist, Borasky Research
http://www.borasky-research.net  http://www.aracnet.com/~znmeb
mailto:znmeb@borasky-research.com  mailto:znmeb@aracnet.com

If there's nothing to astrology, how come so many famous men were born on
holidays?

> -----Original Message-----
> From: Joseph S. Myers [mailto:jsm28@cam.ac.uk]
> Sent: Sunday, May 27, 2001 7:41 AM
> To: David Konerding
> Cc: Toon Moene; gcc@gcc.gnu.org
> Subject: Re: Benchmarking theory
>
>
> On Sat, 26 May 2001, David Konerding wrote:
>
> > I think Joe's point is that people aren't doing real statistics on the
> > results. For example, with just 2 data points (time on one compiler
> > and the other) you don't have what is known as "power" to distinguish
> > whether the difference is actually significant or not.  I certainly
>
> Yes.  It seems to me - as a mathematician, not a computer scientist - that
> benchmark results should be quoted with quantitative results of
> statistical significance tests, even if the release criteria don't state
> significance levels at which performance must not have decreased.  But I
> presume that at least some of the people quoting results without this
> information are computer scientists, and so know what they're
> doing with a
> solid basis in computer science theory.
>
> --
> Joseph S. Myers
> jsm28@cam.ac.uk
>
>


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]