testing consistency (fwd)

[Joe Buck]

[ Jason, you'll get this twice, I goofed ]

crash@cygnus.co.jp (Jason Molenda) writes:

> The two problems I ran in to after coming up with the idea were (a) not
> much free time :), and (b) I don't understand relational databases very
> well.  I've got the basics for SQL down now, I believe, but I am just
> stumped as to how the tables should be arranged.

Well, I just read an article on SQL in Linux Journal so now I'm an
expert. :-)  (The article talks about SQL/Web interfaces based on free
software, might be interesting to some).

I guess the way to go about it is to think about what queries you want
to support and at least make the common cases easy.

If we have a table in which each test run forms a record, then we might
want to extract:

* all tests of snapshot 970904
* all tests on sparc platforms
* all cross-compiler builds
* all Linux tests on any linux platform
* all tests with more than N unexpected failures

There would be one table that would have a single entry for each testsuite
run (e.g. the gcc tests would be an entry, the g++ tests would be a
different entry).  It might be as follows:

* Email address of submitter
* Snapshot or release version
* Name of test suite (gcc,g++,libstdc++,libio)
* 1st field of canonical version (e.g. sparc)
* 2nd field (e.g. sun)
* 3rd+4th field (e.g. solaris2.5.1, linux-gnu)
* assembler used ("vendor" or binutils version)
* linker used ("vendor" or binutils version)
* libc version (blank if irrelevant)
* number of expected passes
* number of expected failures
* number of unexpected passes
* number of unexpected failures
* number of unsupported tests

I include the email address of the submitter so we can resolve any
problems (e.g. two submitters get different results for the same test).

We then want an additional table that makes separate entries for each
individual FAIL, and for each individual XPASS.  The first 9 columns
would be as above, then we would have

* test name
* test status (XPASS or FAIL)

We can then ask which tests fail, or which tests fail for the sparc,
etc.