Re: [PATCH 01/15] Selftest framework (unittests v4)

[David Malcolm <dmalcolm at redhat dot com>]

On Wed, 2015-11-25 at 11:55 +0100, Bernd Schmidt wrote:
> On 11/25/2015 03:26 AM, David Malcolm wrote:
> > Consider the case where an assumption that the host is little-endian
> > assumption creeps into one of the bitmap functions.  Some time later,
> > another developer updates their working copy from svn on a big-endian
> > host and finds that lots of things are broken.  What's the ideal
> > behavior?
> 
> Internal compiler error in test_bitmaps, IMO. That's the quickest way to 
> get to the right place in the debugger.
> 
> >> In the end I think I lean towards run everything with automatic
> >> registration/discovery.  But I still have state worries.  Or to put it
> >> another way, given a test of tests, we should be able to run them in an
> >> arbitrary order with no changes in the expected output or pass/fail results.
> >
> > That would be the ideal - though do we require randomization, or merely
> > hold it up as an ideal?  As it happens, I believe function-tests.c has
> > an ordering dependency (an rtl initialization assert, iirc), which
> > sorting them papered over.
> 
> What do you hope to gain with randomization? IMO if there are 
> dependencies, we should be able to specify priority levels, which could 
> also help running lower-level tests first.

I don't particularly want randomization, I was just wondering if others
wanted them, given that it's one of the features that test frameworks
tend to provide.

I do want some level of determinism over test ordering, for the sake of
everyone's sanity.  It's probably simplest to either hardcode the order,
or have priority levels.  I favor the former (and right now am leaning
towards a very explicit no-magic approach with no auto-registration,
given the linker issues I've been seeing with auto-registration).