GCC Buildbot Update

[Paulo Matos <pmatos@linki.tools>]

Hello,

Apologies for the delay on the update. It was my plan to do an update on
a monthly basis but it slipped by a couple of weeks.

The current status is:

*Workers:*

- x86_64

2 workers from CF (gcc16 and gcc20) up and running;
1 worker from my farm (jupiter-F26) up and running;

2 broken CF (gcc75 and gcc76) - the reason for the brokenness is that
the machines work well but all outgoing ports except the git port is
open (9418 if not mistaken). This means that not only we cannot svn co
gcc but we can't connect a worker to the master through port 9918. I
have contacted the cf admin but the reply was that nothing can be done
as they don't really own the machine. They seemed to have relayed the
request to the machine owners.

- aarch64

I got an email suggesting I add some aarch64 workers so I did:
4 workers from CF (gcc113, gcc114, gcc115 and gcc116);

*Builds:*

As before we have the full build and the incremental build. Both enabled
for x86_64 and aarch64, except they are currently failing for aarch64
(more on that later).

The full build is triggered on Daily bump commit and the incremental
build is triggered for each commit.

The problem with this setup is that the incremental builder takes too
long to run the tests. Around 1h30m on CF machines for x86_64.

Segher Boessenkool sent me a patch to disable guality and prettyprinters
which coupled with --disable-gomp at configure time was supposed to make
things much faster. I have added this as the Fast builder, except this
is failing during the test runs:
unable to alloc 389376 bytes
/bin/bash: line 21: 32472 Aborted                 `if [ -f
${srcdir}/../dejagnu/runtest ] ; then echo ${srcdir}/../dejagnu/runtest
; else echo runtest; fi` --tool gcc
/bin/bash: fork: Cannot allocate memory
make[3]: [check-parallel-gcc] Error 254 (ignored)
make[3]: execvp: /bin/bash: Cannot allocate memory
make[3]: [check-parallel-gcc_1] Error 127 (ignored)
make[3]: execvp: /bin/bash: Cannot allocate memory
make[3]: [check-parallel-gcc_1] Error 127 (ignored)
make[3]: execvp: /bin/bash: Cannot allocate memory
make[3]: *** [check-parallel-gcc_1] Error 127


However, something interesting is happening here since the munin
interface for gcc16 doesn't show the machine running out of memory:
https://cfarm.tetaneutral.net/munin/gccfarm/gcc16/memory.html
(something confirmed by the cf admins)

The aarch64 build is failing as mentioned earlier. If you check the logs:
https://gcc-buildbot.linki.tools/#/builders/5/builds/10
the problem seems to be the assembler issuing:
Assembler messages:
Error: unknown architecture `armv8.1-a'
Error: unrecognized option -march=armv8.1-a


If I go to the machines and check the versions I get:
pmatos@gcc115:~/gcc-8-20171203_BUILD$ as --version
GNU assembler (GNU Binutils for Ubuntu) 2.24
Copyright 2013 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or later.
This program has absolutely no warranty.
This assembler was configured for a target of `aarch64-linux-gnu'.

pmatos@gcc115:~/gcc-8-20171203_BUILD$ gcc --version
gcc (Ubuntu/Linaro 4.8.4-2ubuntu1~14.04.3) 4.8.4
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

pmatos@gcc115:~/gcc-8-20171203_BUILD$ as -march=armv8.1-a
Assembler messages:
Error: unknown architecture `armv8.1-a'

Error: unrecognized option -march=armv8.1-a

However, if I run the a compiler build manually with just:

$ configure --disable-multilib
$ nice -n 19 make -j4 all

This compiles just fine. So I am at the moment attempting to investigate
what might cause the difference between what buildbot does and what I do
through ssh.

*Reporters:*

There is a single reporter which is a irc bot currently silent.

*Regression analysis:*

This is one of the most important issues to tackle and I have a solution
in a branch regression-testing :
https://github.com/LinkiTools/gcc-buildbot/tree/regression-testing

using jamais-vu from David Malcolm to analyze the regressions.
It needs some more testing and I should be able to get it working still
this year.

*LNT:*

I had mentioned I wanted to setup an interface which would allow for
easy visibility of test failures, time taken to build/test, etc.
Initially I thought a stack of influx+grafana would be a good idea, but
was pointed out to using LNT as presented by James Greenhalgh in the GNU
Cauldron. I have setup LNT (soon to be available under
https://gcc-lnt.linki.tools) and contacted James to learn more about the
setup. As it turns out James is just using it for benchmarking results
and out-of-the-box only seems to support the LLVM testing infrastructure
so getting GCC results in there might take a bit more of scripting and
plumbing.

I will probably take the same route and set it up first for the
benchmarking results and then try to get the gcc test results there as well.

*TODO:*

So on my todo list for the next iteration I have:

- analysis of out-of-memory issues in CF for Fast builders;
- analysis of aarch64 build failure;
- merge regression testing verification branch into master and deploy
into production;
  - this should then trigger the irc bot reporter for regressions;
- open up LNT for benchmarking and add benchmarking job for x64_64 using
csibe (as an initial proof of concept);

If you have any wishes, questions, want to offer some fast machines to
have workers on or if you know what's wrong with the Fast builder or the
aarch64 machines, please let me know.

I hope to send another update in about a months time.

Kind regards,
-- 
Paulo Matos