Re: [PATCH] Allow MIPS call-saved-{4-6}.c tests to correctly run for micromips

["Maciej W. Rozycki" <macro at linux-mips dot org>]

On Wed, 14 Jan 2015, Richard Sandiford wrote:

> >> The aim of mips.exp is avoid skipping tests whereever possible.  If
> >> someone runs the testsuite with -mips16 and we have a -micromips test,
> >> it's better to remove -mips16 for that test than to skip the test entirely.
> >
> >  OK, good to know, thanks; that works for compilation tests only though.  
> > For execution tests however what if target hardware used is incompatible 
> > or there is no suitable C library (multilib configuration) available for 
> > the option requested?  E.g. any hardware supporting MIPS16 won't run 
> > microMIPS code, n64 tests won't link if there's no n64 multilib, etc.
> 
> In those cases it just does a compile-only test, again on the basis that
> it's better than skipping the test entirely.  See the big comment at the
> beginning of mips.exp if you're interested in the specific details of
> how this works and what is supported.

 Very good indeed then, and thanks for clearing my uncertainty!  I have 
actually been quite impressed by the robustness of the mips.exp harness, 
and missed its power while chasing test suite failures for another 
architecture where the lack of attention to multilib defaults led to silly 
option combinations causing inevitable errors.

> >> TBH this seems a bit much.  On the one hand it's more testing than you'd
> >> get for almost any other target, but on the other it leaves out important
> >> differences like MIPS I vs MIPS II vs MIPS 32, MIPS III vs MIPS IV vs MIPS64,
> >> r1 vs. r2 vs. r6, Octeon vs. Loongson vs. vanilla, DSP vs. no DSP, etc.
> >
> >  I disagree, I listed what I consider the base set of configurations for 
> > the MIPS target, spanning the major target variations:
> >
> > - MIPS/MIPS16/microMIPS can be treated almost as distinct processor 
> >   architectures, the instruction sets have much common in spirit, but 
> >   there are enough pitfalls and traps,
> >
> > - n32 covers a substantially different calling convention plus (for Linux) 
> >   SVR4 PIC code that normally isn't used for executables with o32 these 
> >   days,
> >
> > - n64 covers all that n32 does plus a 64-bit target.
> >
> > I realise ELF testing may be difficult for some people due to the hassle 
> > with setting up the runtime, so to skip an ELF target can be justified; 
> > otherwise I think it makes sense to run such testing for at least one 
> > configuration from the list above for a good measure.  As is running some 
> > of them with the big and some of them with the little endianness.
> >
> >  You've got a point with architecture levels or processor models.  I think 
> > r6 should be treated as a distinct architecture and tested as the fourth 
> > variant along MIPS/MIPS16/microMIPS, but such a test environment may not 
> > yet be available to many.  The rest I'm afraid will mostly matter for 
> > changes made to the middle end rather than the MIPS backend, in which case 
> > chances are MIPS testing will not be run at all.  A test bot (similar to 
> > JBG's build bot, but extended to run testing too) can help in this case; I 
> > don't know if anyone runs one.
> >
> >  As to DSP, MSA, hard-float, soft-float, 2008-NaN, etc., I'd only expect 
> > to run appropriate testing (i.e. with `-mdsp', etc.) across the 
> > configurations named above whenever relevant code is changed; some of this 
> > stuff is irrelevant or unavailable for some of the configurations above 
> > (e.g. n64 DSP, IIRC), or may have no influence (e.g. the NaN encoding), in 
> > which case it may be justified to skip them.
> 
> But soft vs. hard float in particular is a significant difference in
> terms of the ABI.  Especially when it comes to MIPS16 interworking
> (but even apart from that).

 And indeed I'd expect such testing to be done for changes involving FP!

> >  Sure, people will undoubtedly use different default options, I'm sure 
> > folks at Cavium will compile for Octeon rather than the base architecture 
> > for example.  Other people may have DSP enabled.  Etc., etc...  That IMHO 
> > does not preclude testing across more than just a single configuration.
> 
> Yeah, but that's just the way it goes.  By trying to get everyone to
> test with the options that matter to you, you're reducing the amount of
> work you have to do when tracking regressions on those targets, but you're
> saying that people who care about Octeon or the opposite floatness have
> to go through the process you describe as "tedious and time consuming".

 Well, when the submitted does not do enough testing then the maintainer 
has to do that instead, assuming that he finds a given change useful and 
wants to accept it.  And the maintainer's processing power is limited and 
will eventually run out.  The more testing other people do, the more 
resources the maintainer has left to do his job.

> And you don't avoid that process anyway, since people making changes
> to target-independent parts of GCC are just as likely to introduce
> a regression as those making changes to MIPS-only code.  If testing
> is cheap and takes only a small number of hours, and if you want to
> make it less tedious to track down a regression, continuous testing
> would give you a narrow window for each regression.

 Yes, this is one way to monitor regressions triggered by generic changes.

> Submitters should be free to test on what matters to them rather than
> have to test a canned set of multilibs on specific configurations.

 Well, as I say, this really boils down to who is going to do testing.  
I'll be more willing to accept a patch right away that has been broadly 
tested to one that has only received minimal testing.  But that of course 
depends on how broad the scope of the change is.

  Maciej