*_data_map_rand.cc timeout on mips

[Ami Tavory]

On 6/21/06, Martin Michlmayr <tbm@cyrius.com> wrote:

>  Some *_data_map_rand.cc testcases that got added on 2006-06-14 produce
> a timeout during compilation on mips.  While mips is not generally a
> fast architecture, this timeout occurred on one of the fastest MIPS
> CPUs on the market.  For what it's worth, compiling trie_data_map_rand.cc
> took slightly less than 6 minutes whereas the timeout is 5 minutes.
>
> Is there a way to specify in the testsuite that some testcases may be
> slow and that the timeout should be increased for them?  If so, the
>  testcases listed below are good candidates.
>
>
> FAIL: ext/pb_ds/regression/hash_data_map_rand.cc (test for excess errors)
> WARNING: ext/pb_ds/regression/hash_data_map_rand.cc compilation failed to produce executable
> WARNING: program timed out.
> FAIL: ext/pb_ds/regression/hash_no_data_map_rand.cc (test for excess errors)
> WARNING: ext/pb_ds/regression/hash_no_data_map_rand.cc compilation failed to produce executable
> WARNING: program timed out.
> FAIL: ext/pb_ds/regression/tree_data_map_rand.cc (test for excess errors)
> WARNING: ext/pb_ds/regression/tree_data_map_rand.cc compilation failed to produce executable
> WARNING: program timed out.
> FAIL: ext/pb_ds/regression/tree_no_data_map_rand.cc (test for excess errors)
> WARNING: ext/pb_ds/regression/tree_no_data_map_rand.cc compilation failed to produce executable
> WARNING: program timed out.
> FAIL: ext/pb_ds/regression/trie_data_map_rand.cc (test for excess errors)
> WARNING: ext/pb_ds/regression/trie_data_map_rand.cc compilation failed to produce executable
> WARNING: program timed out.
> FAIL: ext/pb_ds/regression/trie_no_data_map_rand.cc (test for excess errors)
> WARNING: program timed out.
> FAIL: ext/pb_ds/regression/trie_no_data_map_rand.cc (test for excess errors)
> WARNING: ext/pb_ds/regression/trie_no_data_map_rand.cc compilation failed to produce executable
>
> --
> Martin Michlmayr
> http://www.cyrius.com/
>


   Hello,

  I'd like to suggest a way to fix this. First are possible reasons
why the tests compile so slowly:
1. Each test tests a large number of containers. For example, whereas
there is a single std::tr1::unordered_map hash-based container, then
given that in pb_ds each policy-combination gives a completely
different type of container (which needs to be tested), there are
something like 10 hash-based containers effectively. Most containers
also support more methods ( e.g., the trees and tries).
2. The tests themselves do a lot. Each test randomly generates
operations, performs them on the tested container and a reference
container, randomly decides when  to throw exceptions, checks that the
tested container is equivalent to the reference container, and checks
for proper deallocation. In some sense these are the strongest
possible tests (at least of which I could think), since they check for
regular validity as well as exception safety under completely random
operation sequences. This is multiplied by (1); that is, the code is
generated for a large number of containers.
3. It's possible that policy-based code is somewhat slow to compile to
begin with.

  Overall, I don't see a way to reduce the product (1) * (2) * (3) -
the number of containers is minimal (containers which didn't show any
advantages were already deleted), regression tests should check for
everything, and 3 is up to the compiler (if there's a way to write the
same code that makes for lighter compilation - I'm not aware of it).
  It is, however, possible to distribute the tests differently. At the
moment, each *_*_map_rand.cc uses typelists that expand to test a
number of containers. Say a typelist can expand to test a chaining
hash-based container with policy combination (a, b, c), then with
policy (a, b, d), ..., then a probing hash-based container with policy
combination (a, b, c, d), and so forth.
  Instead, it's possible to do one of the following:
1. Make more *_*_map_rand.cc files, each testing a specific container
and policy combination. This would not reduce the product (1) * (2) *
(3) (because the total compilation time would be the same), but it
would mean that each compilation would be shorter. I think there's a
better solution.
2. Randomized tests are strong in the sense that if there is a
problem, the chances that it will go undetected decreases over time.
Why not make smaller tests then (as in 1), but decide randomly which
ones to compile and run each time. In fact, just like the typelists
now expand to cover all possible policy combinations at compile time,
I can write the same code in python to generate a random subset of
tests at runtime (that is, the *_*map_rand.cc contents would be
generated on the fly). It's possible to object that not all platforms
would test the same things at the same time, but since the tests are
randomized - they're not doing so anyway.

  Overall I'd go for option 2, but I don't know if this fits in with
the way things are done. Anyway, comments would be appreciated, and
once something's decided - I'd be happy to do what I can to facilitate
it.

  Bye,

  Ami