Re: Library testsuite

[Brooks Moses <brooks dot moses at codesourcery dot com>]

In writing back to Tobias and FX, I realized that I've got some fairly 
concrete thoughts on this, and it's probably much more efficient for me 
just to write them all out at once rather than piecemeal in replies that 
are apt to inadvertantly look like criticism of others' suggestions.

So, when I've been thinking of a "proper" fortran-intrinsic testsuite, 
here's what I envision:

For a given intrinsic, we have:

* A test for argument-type checks.  This includes these possible errors:
	* Missing required arguments
	* Too many arguments.
	* Incorrect argument types.
	* Incorrect argument kinds, if applicable.
	* Incorrect argument dimensions/lengths, if applicable.
  Note that this will in some cases require us to deal with restrictions
  that say that one argument must the the same type, or the same type
  and kind, as another.

  However, this will certainly not require the combinatorial explosion
  of all possible wrong-argument combinations.  Probably all we need
  is a set of all of the off-by-one-thing errors; incorrect kind or
  type on one argument, or one too few, or one too many.

* A runtime check for correct values for constant-foldable expressions.
  This includes checks with all of the possible argument types and
  kinds, or at least a sufficient sample if the combinatorial explosion
  is large.

  This will probably include several different sets of input values,
  each set having a range of types and kinds to be checked over.

* A runtime check for correct values from non-constant-foldable
  expressions, identical to the above.  Ideally, these two sets shall
  be automatically generated from the same initial source, to ensure
  that both variants are tested for the same results.

* A runtime check for errors on out-of-range values in non-constant-
  foldable expressions, if applicable, again including checks for all
  of the possible argument types and kinds.

* A check for errors on out-of-range values in constant-foldable
  expressions.  This is somewhat tricky, because we have to know
  whether to expect a compile-time or runtime error.

I'd like to have these automatically generated, based on supplying the 
following data:

* A list of the types and kinds that are permitted for each argument,
  and the intent of the argument for subroutines.  This should allow
  multiple lists for each intrinsic, to deal with any cases more
  complicated than "arg3 is the same type and kind as arg2".

* For each of these lists, a series of array ranks and dimensions to
  be iterated over, along with a boolean indicating whether the given
  combination is allowed or not.

* For each of the "valid" dimension lists, a series of sets of input
  and expected output values or expected error message phrase, and
  for each set a boolean flag indicating whether a compile-time check
  should be made as well as a runtime one.

I'm also realizing, having written all of this out, that it would be 
substantially more complicated to implement than I thought, so I'm 
certainly not expecting this out of a first version!  It would be nice 
to leave room for expansion in these sorts of directions, though, and I 
think the data set description may be useful even as a start.

- Brooks