Expensive selftests

[Martin Sebor]

On 8/17/21 12:40 AM, Thomas Schwinge wrote:
> Hi!
> 
> On 2021-08-16T14:10:00-0600, Martin Sebor <msebor@gmail.com> wrote:
>> On 8/16/21 6:44 AM, Thomas Schwinge wrote:
>>> [...], to document the current behavior, I propose to
>>> "Add more self-tests for 'hash_map' with Value type with non-trivial
>>> constructor/destructor", see attached.  OK to push to master branch?
>>> (Also cherry-pick into release branches, eventually?)
> 
> (Attached again, for easy reference.)
> 
>> Adding more tests sounds like an excellent idea.  I'm not sure about
>> the idea of adding loopy selftests that iterate as many times as in
>> the patch (looks like 1234 times two?)
> 
> Correct, and I agree it's a sensible concern, generally.
> 
> The current 1234 times two iterations is really arbitrary (should
> document that in the test case), just so that we trigger a few hash table
> expansions.
> 
> For 'selftest-c', we've got originally:
> 
>      -fself-test: 74775 pass(es) in 0.309299 seconds
>      -fself-test: 74775 pass(es) in 0.366041 seconds
>      -fself-test: 74775 pass(es) in 0.356663 seconds
>      -fself-test: 74775 pass(es) in 0.355009 seconds
>      -fself-test: 74775 pass(es) in 0.367575 seconds
>      -fself-test: 74775 pass(es) in 0.320406 seconds
> 
> ..., and with my changes we've got:
> 
>      -fself-test: 94519 pass(es) in 0.327755 seconds
>      -fself-test: 94519 pass(es) in 0.369522 seconds
>      -fself-test: 94519 pass(es) in 0.355531 seconds
>      -fself-test: 94519 pass(es) in 0.362179 seconds
>      -fself-test: 94519 pass(es) in 0.363176 seconds
>      -fself-test: 94519 pass(es) in 0.318930 seconds
> 
> So it really seems to be all in the noise?
> 
> Yet:
> 
>> Selftests run each time GCC
>> builds (i.e., even during day to day development).  It seems to me
>> that it might be better to run such selftests only as part of
>> the bootstrap process.
> 
> I'd rather have thought about a '--param self-test-expensive' (or
> similar), and then invoke the selftests via a new
> 'gcc/testsuite/selftests/expensive.exp' (or similar).
> 
> Or, adapt 'gcc/testsuite/gcc.dg/plugin/expensive_selftests_plugin.c',
> that is, invoke them via the GCC plugin mechanism, which also seems to be
> easy enough?
> 
> I don't have a strong opinion about where/when these tests get run, so
> will happily take any suggestions.

I think the right design is to move all these basic building blocks
(at a minimum, all containers, but ultimately even higher level
general-purpose APIs) into a standalone library with its own unit
tests run independently of GCC.

I'm fine with adding these tests if no one else is concerned about
the overhead, especially with a lower number of iterations like
Richard suggests (as long as it still exercises the expansion,
of course).

Thanks
Martin

> 
> 
> Grüße
>   Thomas
> 
> 
> -----------------
> Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstraße 201, 80634 München; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Thomas Heurung, Frank Thürauf; Sitz der Gesellschaft: München; Registergericht München, HRB 106955
>