Re: [GCC 3.x] Performance testing for QA

[Andreas Jaeger <aj at suse dot de>]

I've trimmed the CC list, no need to cross-post between GCC and
mplayer lists.

Sasi Péter <Peter.Sasi@t-systems.co.hu> writes:

> Dear GCC people,
>
> Allow me the humble question to ask you if it would possible to
> employ a simple yet efficient testing method from real life as
> opposed to the SPEC* (artificial) test?

SPEC* are not really artificial benchmarks, those are real life
programs that have been changed into tests.

> Specifically I was thinking of running a few encoding/decoding/manipulation runs with mplayer on sample multimedia streams.
>
> Main reasons I could think of are:
> - Easy to test: all you need (sources, sample files) is available free at www.mplayerhq.hu (for Linux(intel, alpha), Solaris, Irix, etc.)
> - Important for masses: it is one of the most widely used free software in the area gaining probably the most momentum nowadays (amongst performace hungry software)
> - There is a lot of space for improvements: even the 2.95 series perform better than the ongoing 3.x series, but if going for the long term Intel's compiler currently wins hands down
>
> I know it is not a very scientific measurement, still it does apply to a lot of people, andd is a reason why a lot of people still stick to the older versions.
> New features are really an important thing to have, still they should not make it instead of the good performance, but exploiting it.
>
> Summarizing the above:
> - Do you think such measurements are feasible/worth and can supply usable input for the development?

They can.

> - Do you lack the interest, the resources, or what else?

Resources and volunteers looking at the tests.

> - Ifyou are not interested, is there a reasoning behind that? (point me to RTFM if applicable)
> - In case of positive answers do you think profiling, code analysis or what else could be the way to improve the current situation? (get 3.x to the perf. level of 2.95.x)

It might be.  But it's not something you can just "give" us.  If you
set up some automatic tests - like I do with my SPEC tests -, you
should make the data available to developers and work with them on
performance issues.  Bug reports for specific problems or just: Your
patch causes a regression are helpful.

Note that GCC is a volunteer effort.  If you're not doing this, I fear
that nobody does it.

>
> Please CC my personal address in your reply, since I am not subscribed to any of your mailing lists.
>
> Thank you for your precious time.

Andreas
-- 
 Andreas Jaeger
  SuSE Labs aj@suse.de
   private aj@arthur.inka.de
    http://www.suse.de/~aj