Re: Disabling Secondary Tests

[Janis Johnson <janisjo at codesourcery dot com>]

On 06/09/2011 01:30 PM, Lawrence Crowl wrote:
> On 6/9/11, Janis Johnson <janisjo@codesourcery.com> wrote:
>> On 06/08/2011 01:54 PM, Lawrence Crowl wrote:
>>> On 6/6/11, Janis Johnson <janisjo@codesourcery.com> wrote:
>>>> On 06/03/2011 11:14 AM, Lawrence Crowl wrote:
>>>>> The PPH project has tests that compile two different ways, and
>>>>> then compare the assembly.  If either of the compiles fails, the
>>>>> comparison will fail.  We'd like to simply not run the comparison.
>>>>>
>>>>> We currently have:
>>>>>
>>>>> set have_errs [llength [grep $test "{\[ \t\]\+dg-error\[\t\]\+.*\[
>>>>> \t\]\+}"]]
>>>>> # Compile the file the first time for a base case.
>>>>> dg-test -keep-output $test "$options -I." ""
>>>>>
>>>>> if { $have_errs } {
>>>>>    verbose -log "regular compilation failed"
>>>>>    fail "$nshort $options, regular compilation failed"
>>>>>    return
>>>>> }
>>>>>
>>>>> But that only stops subsequent actions when the test is known
>>>>> a priori to have errors.  How do we detect compilation errors,
>>>>> so as to skip the remainder of the actions?
>>>>
>>>> Complicated GCC tests do this by using local procs instead of dg-runtest
>>>> and dg-test.  See, for example, gcc.dg/lto/lto.exp,
>>>> gcc.dg/compat/compat.exp and gcc.dg/tree-prof/tree-prof.exp, which use
>>>> lto.exp, compat.exp and profopt.exp from GCC's testsuite/lib.  Those
>>>> have scenarios in which further testing is skipped after a compile or
>>>> link fails.
>>>
>>> So, I ended up changing the definition of fail from "reports test
>>> failing" to "does not produce an assembly file".  We really need
>>> the latter for comparison, so it is the true measure.  Once I made
>>> that change in orientation, I was able to achieve what I wanted.
>>>
>>> The simple part is the regular compile.
>>>
>>>     # Compile the file the first time for a base case.
>>>     set dg-do-what-default compile
>>>     dg-test -keep-output $test "$options -I." ""
>>>
>>>     # Quit if it did not compile successfully.
>>>     if { ![file_on_host exists "$bname.s"] } {
>>>         # All regular compiles should pass.
>>>         fail "$nshort $options (regular assembly missing)"
>>>         return
>>>     }
>>
>> Don't use dg-test, use a new variant of it as is done with the lto,
>> compat, and profopt tests.  You'll have much more control that way
>> and can better check the success of individual steps to decide what
>> to do next.
> 
> I am having trouble identifying the variant.  Does it have a name,
> or is it inline code?

I meant that you should write one.  For example, gcc.dg/lto/lto.exp
uses lto_execute which is defined in lib/lto.exp, and
gcc.dg/compat/compat.exp uses compat-execute defined in lib/compat.exp.

>>> The complicated part is the compile we are comparing against, which
>>> required knowing whether or not a compile failure is expected.  For
>>> that we grep for dg-xfail-if and the appropriate option.
>>>
>>>     # Compile a second time using the pph files.
>>>     dg-test -keep-output $test "$options $mapflag -I." ""
>>>
>>>     # Quit if it did not compile successfully.
>>>     if { ![file_on_host exists "$bname.s"] } {
>>>         # Expect assembly to be missing when the compile is an expected fail.
>>>         if { ![llength [grep $test "dg-xfail-if.*-fpph-map"]] } {
>>>             fail "$nshort $options (pph assembly missing)"
>>>         }
>>>         return
>>>     }
>>
>> Relying on the existence of dg-xfail-if won't work when an expected
>> compilation failure starts working, or an unexpected one arises.
> 
> If the compilation starts working, I get an assembly file, and
> continue to assembly comparisons.  If the compilation fails,
> but with a different error, then the other (typically dg-bogus)
> directives should report the unexpected failure.  In either case,
> I think, I get proper notice.  Am I missing something?

I don't fully understand the purpose of grepping for dg-xfail-if.
You shouldn't care about that within the test, just whether the
compile succeeded or not.  If there are messages that prevent the
compile from succeeding, then they should be handled with appropriate
test directives.

Apart from that, whether or not a dg-xfail-if directives takes effect
depends on the target and options used for the test.

Just in case you're not aware of it (many people aren't), the test
directives are documented in
http://gcc.gnu.org/onlinedocs/gccint/Directives.html#Directive.

Janis