duplicate arm test results?

[Martin Sebor]

On 9/23/20 2:54 AM, Christophe Lyon wrote:
> On Wed, 23 Sep 2020 at 01:47, Martin Sebor <msebor@gmail.com> wrote:
>>
>> On 9/22/20 9:15 AM, Christophe Lyon wrote:
>>> On Tue, 22 Sep 2020 at 17:02, Martin Sebor <msebor@gmail.com> wrote:
>>>>
>>>> Hi Christophe,
>>>>
>>>> While checking recent test results I noticed many posts with results
>>>> for various flavors of arm that at high level seem like duplicates
>>>> of one another.
>>>>
>>>> For example, the batch below all have the same title, but not all
>>>> of the contents are the same.  The details (such as test failures)
>>>> on some of the pages are different.
>>>>
>>>> Can you help explain the differences?  Is there a way to avoid
>>>> the duplication?
>>>>
>>>
>>> Sure, I am aware that many results look the same...
>>>
>>>
>>> If you look at the top of the report (~line 5), you'll see:
>>> Running target myarm-sim
>>> Running target myarm-sim/-mthumb/-mcpu=cortex-m3/-mfloat-abi=soft/-march=armv7-m
>>> Running target myarm-sim/-mthumb/-mcpu=cortex-m0/-mfloat-abi=soft/-march=armv6s-m
>>> Running target myarm-sim/-mcpu=cortex-a7/-mfloat-abi=hard/-march=armv7ve+simd
>>> Running target myarm-sim/-mthumb/-mcpu=cortex-m7/-mfloat-abi=hard/-march=armv7e-m+fp.dp
>>> Running target myarm-sim/-mthumb/-mcpu=cortex-m4/-mfloat-abi=hard/-march=armv7e-m+fp
>>> Running target myarm-sim/-mthumb/-mcpu=cortex-m33/-mfloat-abi=hard/-march=armv8-m.main+fp+dsp
>>> Running target myarm-sim/-mcpu=cortex-a7/-mfloat-abi=soft/-march=armv7ve+simd
>>> Running target myarm-sim/-mthumb/-mcpu=cortex-a7/-mfloat-abi=hard/-march=armv7ve+simd
>>>
>>> For all of these, the first line of the report is:
>>> LAST_UPDATED: Tue Sep 22 09:39:18 UTC 2020 (revision
>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c)
>>> TARGET=arm-none-eabi CPU=default FPU=default MODE=default
>>>
>>> I have other combinations where I override the configure flags, eg:
>>> LAST_UPDATED: Tue Sep 22 11:25:12 UTC 2020 (revision
>>> r9-8928-gb3043e490896ea37cd0273e6e149c3eeb3298720)
>>> TARGET=arm-none-linux-gnueabihf CPU=cortex-a9 FPU=neon-fp16 MODE=thumb
>>>
>>> I tried to see if I could fit something in the subject line, but that
>>> didn't seem convenient (would be too long, and I fear modifying the
>>> awk script....)
>>
>> Without some indication of a difference in the title there's no way
>> to know what result to look at, and checking all of them isn't really
>> practical.  The duplication (and the sheer number of results) also
>> make it more difficult to find results for targets other than arm-*.
>> There are about 13,000 results for September and over 10,000 of those
>> for arm-* alone.  It's good to have data but when there's this much
>> of it, and when the only form of presentation is as a running list,
>> it's too cumbersome to work with.
>>
> 
> To help me track & report regressions, I build higher level reports like:
> https://people.linaro.org/~christophe.lyon/cross-validation/gcc/trunk/0latest/report-build-info.html
> where it's more obvious what configurations are tested.

That looks awesome!  The regression indicator looks especially
helpful.  I really wish we had an overview like this for all
results.  I've been thinking about writing a script to scrape
gcc-testresults and format an HTML table kind of like this for
years.  With that, the number of posts sent to the list wouldn't
be a problem (at least not for those using the page).  But it
would require settling on a standard format for the basic
parameters of each run.

> 
> Each line of such reports can send a message to gcc-testresults.
> 
> I can control when such emails are sent, independently for each line:
> - never
> - for daily bump
> - for each validation
> 
> So, I can easily reduce the amount of emails (by disabling them for
> some configurations),
> but that won't make the subject more informative.
> I included the short revision (rXX-YYYY) in the title to make it clearer.
> 
> The number of configurations has grown over time because we regularly
> found regressions
> in configurations not tested previously.
> 
> I can probably easily add the values of --with-cpu, --with-fpu,
> --with-mode and RUNTESTFLAGS
> as part of the [<branch> revision rXX-YYYY-ZZZZZ] string in the title,
> would that help?
> I fear that's going to make very long subject lines.
> 
> It would probably be cleaner to update test_summary such that it adds
> more info as part of $host
> (as in "... testsuite on $host"), so that it grabs useful configure
> parameters and runtestflags, however
> this would be more controversial.

Until a way to present summaries is available, would grouping
the results of multiple runs in the same "basic configuration"
(for some definition of basic) in the same post work for you?

Martin

> 
> Christophe
> 
>> Martin
>>
>>>
>>> I think HJ generates several "running targets" in the same log, I run
>>> them separately to benefit from the compute farm I have access to.
>>>
>>> Christophe
>>>
>>>> Thanks
>>>> Martin
>>>>
>>>> Results for 11.0.0 20200922 (experimental) [master revision
>>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>>>> arm-none-eabi   Christophe LYON
>>>>
>>>>        Results for 11.0.0 20200922 (experimental) [master revision
>>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>>>> arm-none-eabi   Christophe LYON
>>>>        Results for 11.0.0 20200922 (experimental) [master revision
>>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>>>> arm-none-eabi   Christophe LYON
>>>>        Results for 11.0.0 20200922 (experimental) [master revision
>>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>>>> arm-none-eabi   Christophe LYON
>>>>        Results for 11.0.0 20200922 (experimental) [master revision
>>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>>>> arm-none-eabi   Christophe LYON
>>>>        Results for 11.0.0 20200922 (experimental) [master revision
>>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>>>> arm-none-eabi   Christophe LYON
>>>>        Results for 11.0.0 20200922 (experimental) [master revision
>>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>>>> arm-none-eabi   Christophe LYON
>>>>        Results for 11.0.0 20200922 (experimental) [master revision
>>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>>>> arm-none-eabi   Christophe LYON
>>>>        Results for 11.0.0 20200922 (experimental) [master revision
>>>> r11-3343-g44135373fcdbe4019c5524ec3dff8e93d9ef113c] (GCC) testsuite on
>>>> arm-none-eabi   Christophe LYON
>>