Re: GCC Buildbot

[Joseph Myers <joseph at codesourcery dot com>]

On Thu, 21 Sep 2017, Markus Trippelsdorf wrote:

> And it has the basic problem of all automatic testing: that in the long
> run everyone simply ignores it.

Hence, see my comments about the value of having someone who monitors the 
results and files bugs / notifies patch authors / fixes issues.

It doesn't need to be the same person who runs the bot.  It doesn't need 
to be the same person for every architecture.  But the results need to be 
monitored and issues raised.

That's what I do with my glibc bots.  A lot of the time they run cleanly, 
but sometimes when they show failures it does take a significant amount of 
work to understand them and fix or inform appropriate people.  (A lot of 
fixes were also involved in getting those bots to a near-clean baseline 
state.)

I once did it, a long time ago, for some GCC bots (on HP-UX, and I think 
i686-pc-linux-gnu as well, as I recall), but that project ended and I 
stopped running the HP-UX bots and largely stopped monitoring the 
i686-pc-linux-gnu one.  My experience indicates that GCC bots would 
require a lot more monitoring than glibc ones, especially if testing lots 
of unusual configurations.

I think it would be straightforward to adapt build-many-glibcs.py to 
operate as a GCC bot running the compilation parts of the GCC testsuite 
for all or almost all supported architectures (and a bit more complicated 
to make it track regressions at the level of individual test failures), 
but I don't know how long an all-architectures compilation test run would 
take, and whether there would be people to monitor all-architectures test 
results is another matter.  (build-many-glibcs.py is useful in glibc 
development even apart from the bots, to allow people to do some 
all-architectures testing of global changes.)

> The same thing is true for the regression mailing list
> https://gcc.gnu.org/ml/gcc-regression/current/.
> It is obvious that nobody pays any attention to it, e.g. PGO bootstrap
> is broken for several months on x86_64 and i686 bootstrap is broken for
> a long time, too.

I don't know if he currently monitors it, but HJ has certainly filed bugs 
for regressions found by his bots in the past.

-- 
Joseph S. Myers
joseph@codesourcery.com