Re: testsuite & Tornado (continued)

[mrs at wrs dot com (Mike Stump)]

> Date: Fri, 28 Aug 1998 09:52:32 +0200
> From: sxthree@penfeld.tls.mms.fr (SXTHREE)

> I stil haven't run the testsuite but I can issue a few notes about
> the process. Hey Mike, running the testsuite is quite a hard work !

It would help if you wrote down you experience and did a simple, how
to put it together and get it to work, so the next person don't have
such a hard time with it.

The advantage is, once done, it should be really easy to run a
testsuite.

> -> DejaGnu reboots the board between tests. It takes about 2
> minutes to load the BSP onto the board. It's not very reasonable
> to spend years for testing the compiler.

I don't know what you mean by `between tests'.  It only should reboot
between testsuite runs, and between testcases that fail that it cannot
recover from.  There should be so few (<10 for 3300 tests) of these
that 2 minutes shouldn't be a problem.  For example with:

                === g++ Summary ===
 
# of expected passes            6704
# of unexpected failures        46
# of unexpected successes       2
# of expected failures          164
# of untested testcases         20

I had just 9 reboots of the board, while testing took 3.5 hours.
Reboot time, if it took 2 minutes, would be 18 minutes or 8% of the
total test time for me.

Are you loading kernels over the ethernet or serial line?  I'd
recommend loading them via the ethernet.  If via ethernet, it should
go fairly quickly, 2 minutes sounds kinda long to me.

Ah, yes, I have some mods that try sending a ^X to reboot the board,
before I try powercycling it, as that is usually faster and works ok.

> -> tests are real programs (with a main). If the program uses
> arguments (argc, argv), those arguments must be provided in
> the way we expect we will find them. I mean that VxWorks does
> not recognize "main" as being a special function. Generally
> something like crtbegin handles these parameters for you. We
> do not use any crtxxxx on VxWorks. Manually generating the 
> parameters is not so easy when you can't pre-suppose how many
> they'll be.

This should all be handled by the status wrappers and testglue.c.

> I'm amazed I've never seen any message on the egcs list concerning
> test results for embedded platforms. I cannot believe I'm the first
> to try this kind of things...

In someways, you are the first.  :-)

Various folks generate testing results and usually keep them
internally.  I'v been doing a bit of testing (17 boards, 9 CPUs, 3
hosts), but it has been for various internal projects using
nonstandard compilers so far.  At some point I hope that I'll be able
to do more direct testing on raw egcs in a timely fashion and send out
results, but that's going to be down the road some.  That's why it's
useful to have people that have cross setups on this list, and doing
testing and letting us know the results.  Hopefully we'll see more
people generating testresults.

Thanks for taking the time, seting it up, and trying it out.
Hopefully you can help test egcs...

I have some hacks to dejagnu that I've collected.  If you want to see
them, I can send them to you.  The are kinda in a state of flux, and
are still being worked on, and have some sharp corners that need
rounding off.  One of the things they feature is windsh style testing
and an ability to do libio/libstdc++ testing.  If you want to help fix
them up and ensure they work well and try them out, let me know.  The
target shell will go away one of these years (in favor of the host
based windsh), so these patches will be good to get in at some point.