Re: Disabling Secondary Tests

[Lawrence Crowl <crowl at google dot com>]

On 6/9/11, Janis Johnson <janisjo@codesourcery.com> wrote:
> On 06/09/2011 01:30 PM, Lawrence Crowl wrote:
>> On 6/9/11, Janis Johnson <janisjo@codesourcery.com> wrote:
>>> On 06/08/2011 01:54 PM, Lawrence Crowl wrote:
>>>> On 6/6/11, Janis Johnson <janisjo@codesourcery.com> wrote:
>>>>> On 06/03/2011 11:14 AM, Lawrence Crowl wrote:
>>>>>> The PPH project has tests that compile two different ways, and
>>>>>> then compare the assembly.  If either of the compiles fails, the
>>>>>> comparison will fail.  We'd like to simply not run the comparison.
>>>>>>
>>>>>> We currently have:
>>>>>>
>>>>>> set have_errs [llength [grep $test "{\[ \t\]\+dg-error\[\t\]\+.*\[
>>>>>> \t\]\+}"]]
>>>>>> # Compile the file the first time for a base case.
>>>>>> dg-test -keep-output $test "$options -I." ""
>>>>>>
>>>>>> if { $have_errs } {
>>>>>>    verbose -log "regular compilation failed"
>>>>>>    fail "$nshort $options, regular compilation failed"
>>>>>>    return
>>>>>> }
>>>>>>
>>>>>> But that only stops subsequent actions when the test is known
>>>>>> a priori to have errors.  How do we detect compilation errors,
>>>>>> so as to skip the remainder of the actions?
>>>>>
>>>>> Complicated GCC tests do this by using local procs instead of
>>>>> dg-runtest
>>>>> and dg-test.  See, for example, gcc.dg/lto/lto.exp,
>>>>> gcc.dg/compat/compat.exp and gcc.dg/tree-prof/tree-prof.exp, which use
>>>>> lto.exp, compat.exp and profopt.exp from GCC's testsuite/lib.  Those
>>>>> have scenarios in which further testing is skipped after a compile or
>>>>> link fails.
>>>>
>>>> So, I ended up changing the definition of fail from "reports test
>>>> failing" to "does not produce an assembly file".  We really need
>>>> the latter for comparison, so it is the true measure.  Once I made
>>>> that change in orientation, I was able to achieve what I wanted.
>>>>
>>>> The simple part is the regular compile.
>>>>
>>>>     # Compile the file the first time for a base case.
>>>>     set dg-do-what-default compile
>>>>     dg-test -keep-output $test "$options -I." ""
>>>>
>>>>     # Quit if it did not compile successfully.
>>>>     if { ![file_on_host exists "$bname.s"] } {
>>>>         # All regular compiles should pass.
>>>>         fail "$nshort $options (regular assembly missing)"
>>>>         return
>>>>     }
>>>
>>> Don't use dg-test, use a new variant of it as is done with the lto,
>>> compat, and profopt tests.  You'll have much more control that way
>>> and can better check the success of individual steps to decide what
>>> to do next.
>>
>> I am having trouble identifying the variant.  Does it have a name,
>> or is it inline code?
>
> I meant that you should write one.  For example, gcc.dg/lto/lto.exp
> uses lto_execute which is defined in lib/lto.exp, and
> gcc.dg/compat/compat.exp uses compat-execute defined in lib/compat.exp.
>
>>>> The complicated part is the compile we are comparing against, which
>>>> required knowing whether or not a compile failure is expected.  For
>>>> that we grep for dg-xfail-if and the appropriate option.
>>>>
>>>>     # Compile a second time using the pph files.
>>>>     dg-test -keep-output $test "$options $mapflag -I." ""
>>>>
>>>>     # Quit if it did not compile successfully.
>>>>     if { ![file_on_host exists "$bname.s"] } {
>>>>         # Expect assembly to be missing when the compile is an expected
>>>> fail.
>>>>         if { ![llength [grep $test "dg-xfail-if.*-fpph-map"]] } {
>>>>             fail "$nshort $options (pph assembly missing)"
>>>>         }
>>>>         return
>>>>     }
>>>
>>> Relying on the existence of dg-xfail-if won't work when an expected
>>> compilation failure starts working, or an unexpected one arises.
>>
>> If the compilation starts working, I get an assembly file, and
>> continue to assembly comparisons.  If the compilation fails,
>> but with a different error, then the other (typically dg-bogus)
>> directives should report the unexpected failure.  In either case,
>> I think, I get proper notice.  Am I missing something?
>
> I don't fully understand the purpose of grepping for dg-xfail-if.
> You shouldn't care about that within the test, just whether the
> compile succeeded or not.  If there are messages that prevent the
> compile from succeeding, then they should be handled with
> appropriate test directives.

Each .cc file is compiled twice.  First without PPH and then
with PPH.  We compare the assembly generated as a proxy for "did
we compile correctly".  The PPH code is in development, and so
several tests fail when compiled with PPH.  We use the dg-xfail-if
to indicate when we expect the PPH compilation to fail.  The specific
failure is indicated by other directives, usually dg-bogus.
The goal is to ensure that our tests are clean, either all pass
or xfail.

> Apart from that, whether or not a dg-xfail-if directives takes
> effect depends on the target and options used for the test.

Yes, we are counting on that.  All the non-PPH compilations pass,
but only most of the PPH compilations pass.  The others get the
dg-xfail-if.

> Just in case you're not aware of it (many people aren't), the
> test directives are documented in
> http://gcc.gnu.org/onlinedocs/gccint/Directives.html#Directive.

I had found that.  One facility that seems to be missing is a
dg-bogus-if directive that enables bogus only when certain flags are
present.  As a result, we get xpass messages on the non-PPH compiles.
I am working around that with a filter on the make output.

-- 
Lawrence Crowl