Re: tests for compiler (non-)limits

[Geoffrey Keating <geoffk at apple dot com>]

On 24/04/2007, at 10:32 AM, Mike Stump wrote:

> On Apr 23, 2007, at 8:02 PM, Geoffrey Keating wrote:
> > * 63 significant initial characters in an internal identifier or a
> >  macro name: limits-idinternal.c (1,000,000)
> >
> > * 31 significant initial characters in an external identifier:
> >  limits-idexternal.c (10,000)
>
> This is way too small.  We blew past 20-30k well more than a decade  
> ago.  When I was testing this back then, I was testing out 1,000,000  
> to be sure there weren't any limits left and ensure enough headroom  
> for the next decade of development.  I'd bump to at least 1,000,000.

I agree 10,000 is too small, so I bumped to 100,000; it didn't make  
the test significantly slower.

> Other than that, looks reasonable.
>
> > Tested by running 'make check-gcc' for just these files.
>
> I'd be interested in the time it takes to test...

I decided I should find out.  The results are very interesting.  None  
of these tests takes longer than 10 seconds of real time at -O0; the  
expensive ones are:

- parentheses in a declaration, limits-declparen
- pointer nesting, limits-pointer
- structure nesting, limits-structnest
- string literal, limits-stringlit

These are about 9 seconds, and all the rest are less than 3 seconds.   
The last one is a little slow because of I/O time (and I don't want to  
make it smaller because I think it's perfectly reasonable to have a  
1Mb string); the other three are computation.  gcc-4.0 as shipped on  
Darwin takes less than two seconds on all of these testcases at -O0  
except limits-stringlit.c.

But, that's not where the time goes in this set of tests on trunk at  
the moment.  At -O3, most of the testcases are the same speed or  
faster, but three test cases take a long time:

- function arguments, limits-fnargs.c, 23 seconds
- block nesting, limits-blocks.c, 87 seconds
- structure nesting, limits-structnest.c, 31 seconds

The last one is especially weird because there is no code at all in  
that testcase, it just declares a variable named 'x'.  By comparison,  
the times with the shipping gcc-4.0 on Darwin (with -O3) are:

limits-fnargs.c: 17 seconds (about the same since this is with disable- 
checking)
limits-blocks.c: 11 seconds (8 times faster)
limits-structnest.c: 0.7 seconds (45 times faster)

I think some of this is a very recent regression, since when I timed  
it two weeks ago it took only about 211 seconds (as measured by  
dejagnu) to run all these tests with torturing, and now it's taking  
much longer.

Attachment: smime.p7s
Description: S/MIME cryptographic signature