Re: crash in 1.0.3a fortran compilation

[Craig Burley <burley at gnu dot org>]

>That's fine. Is there any reason that there shouldn't be a second
>test-suite which compiles gobs of code in an attempt to ferret out
>goofs. There are plenty of people like me with tons of machines who'd
>be happy to run the "fat suite". There's a lot more to testing than
>compiling small, easily understandable programs.

Well, I mentioned that idea in my earlier email, and totally agree.
A second tier of large-scale tests would be great, especially if
it (and maybe even the first tier) included some means for testing
and thus comparing performance (memory and CPU usage) somehow.

But, see, now, you've drawn me out, in a sense.  I have a really
bare-bones tool that represents the beginnings of this sort of
thing, which I've developed from a simple-minded shell script I've
used for years to look for "gotchas" in my g77/gcc work.

This tool is designed almost exclusively for compiler-internals
people, because it rests on the principle of "golden files" and
the deltas between those files and newly generated files that
represent the outputs of the tests.

And, a "test" in this case is hard-wired to be running the `f771'
compiler module, its "outputs" are stdout (assembler code) and
stderr (diagnostics), and its "inputs" are all the *.f files it
can find in the current directory.  Running it produces both a
completely new set of outputs plus diffs against the golden outputs.

(I *said* it was bare-bones.)

Starting a few months ago I made it a bit more elegant in that
it takes options to goldenize outputs; to "tarnish" them (remove
the latest ones); to report status and create "stamp" files that
contain said status as part of its normal operations; to support
--help; to update the data base of outputs after adding new tests;
and so on.  But it's still very primitive.

I've wanted to share this tool with the egcs/gcc community, because
it's a great way to spot bugs like where you think you've diagnosed
a bug in user code but have overreached and started diagnosing good
code as well.

But there's lots more I'd like to do to it before releasing it,
like having it take a configuration file so the mappings of inputs,
outputs, and test facilities are arbitrary (instead of hard-wiring
*.f, stderr and stdout, and f771), so the combinations of options to
pass the test facility is more easily managed (instead of just passing
one set on the command line), having it live in a directory that's
simply got a few links to directories full of tests and sort all
that out appropriately, etc.

In short, this thing's been useful to me, but it's the kind of hack
that'd probably not be really useful to others in its current form.
Instead of being written in bash (for GNU systems, e.g. it uses
`mkdir -p'), the enhanced version probably needs to be written in
Perl or Guile or Expect or Tcl (*none* of which I know), and so on.

And, it's a far cry from the kind of comprehensive test facility
you're looking for, since it isn't really designed to compile, then
link, then run, and so on, though perhaps in a better form it would
handle that sort of thing.  What it can do, though, is catch some
problems like g77 or gcc crashing on a well-known package when it
didn't before.  (Also, grep'ing the stderr outputs for possible
crashes is quite easy, something I occasionally do when I'm in the
mood to track down a typically-minor bug.)

If anyone thinks they could make use of this script, especially if
they're proficient enough with this sort of thing to consider
working on "generalizing" it sufficiently for widespread use, drop
me a note and I'll put it up for ftp.  It's so ugly (not the code
so much as the design) that it's a bit embarrassing to do so
without somebody asking.  :)

        tq vm, (burley)