Bug 84215 - Random results in go/libgo tests
Summary: Random results in go/libgo tests
Status: NEW
Alias: None
Product: gcc
Classification: Unclassified
Component: go (show other bugs)
Version: 8.0
: P3 normal
Target Milestone: ---
Assignee: Ian Lance Taylor
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-02-05 14:40 UTC by Christophe Lyon
Modified: 2018-03-13 14:23 UTC (History)
4 users (show)

See Also:
Host:
Target:
Build:
Known to work:
Known to fail:
Last reconfirmed: 2018-02-06 00:00:00


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Christophe Lyon 2018-02-05 14:40:05 UTC
Hi,

We have noticed what appear to be random results in go and libgo tests on arm and aarch64. Since we don't plan to actually work on go, I post here the list of such tests to share information in case others have noticed similar things.

On arm (AArch32):
libgo:database/sql
libgo:os
libgo:os/signal
libgo:context

On AArch64:
libgo:os/signal
libgo:runtime/pprof
libgo:sync

We have also observed similar failures on x86_32 and x86_64, but much less frequently.

These tests are running natively.
Comment 1 Ian Lance Taylor 2018-02-05 15:38:26 UTC
What do you mean by "random results?"  Can you post some output?  Thanks.
Comment 2 Christophe Lyon 2018-02-06 10:00:50 UTC
Well, we have a nightly Jenkins job that builds GCC (from scratch) and runs 'make check' on ARM, AArch64, and x86 hosts, and from time to time we see regression reports on the mentioned tests.

And the next run generally says those tests now pass.

For instance:
Test Run By tcwg-buildslave on Sat Feb  3 08:20:17 UTC 2018
Native configuration is armv8l-unknown-linux-gnueabihf

                === libgo tests ===
[....]
PASS: os/exec
--- FAIL: TestAtomicStop (5.61s)
        testing.go:560: iteration 0: exit status "signal: Interrupt": output: 
        testing.go:560: iteration 1: exit status "signal: Interrupt": output: 
        testing.go:560: iteration 3: exit status "signal: Interrupt": output: 
        testing.go:560: iteration 5: exit status "signal: Interrupt": output: 
        testing.go:560: iteration 7: exit status "signal: Interrupt": output: lost signal on tries: 2
        testing.go:560: iteration 7: lost signal
        testing.go:560: iteration 9: exit status "signal: Interrupt": output: 
FAIL
FAIL: os/signal
PASS
PASS: os/user


For libgo:context, there might be a real compiler bug though:
go1: internal compiler error: Segmentation fault
0x5ae6ab crash_signal
        /home/tcwg-buildslave/workspace/tcwg-buildfarm/tcwg-apm_32-build/snapshots/gcc.git~master/gcc/toplev.c:325
0x1ea8e9 Block::determine_types()
        /home/tcwg-buildslave/workspace/tcwg-buildfarm/tcwg-apm_32-build/snapshots/gcc.git~master/gcc/go/gofrontend/gogo.cc:6062
0x1ea90d Block::determine_types()
        /home/tcwg-buildslave/workspace/tcwg-buildfarm/tcwg-apm_32-build/snapshots/gcc.git~master/gcc/go/gofrontend/gogo.cc:6071
0x1ea90d Block::determine_types()
        /home/tcwg-buildslave/workspace/tcwg-buildfarm/tcwg-apm_32-build/snapshots/gcc.git~master/gcc/go/gofrontend/gogo.cc:6071
0x1ea90d Block::determine_types()
        /home/tcwg-buildslave/workspace/tcwg-buildfarm/tcwg-apm_32-build/snapshots/gcc.git~master/gcc/go/gofrontend/gogo.cc:6071
0x1ea90d Block::determine_types()
        /home/tcwg-buildslave/workspace/tcwg-buildfarm/tcwg-apm_32-build/snapshots/gcc.git~master/gcc/go/gofrontend/gogo.cc:6071
0x1ea90d Block::determine_types()
        /home/tcwg-buildslave/workspace/tcwg-buildfarm/tcwg-apm_32-build/snapshots/gcc.git~master/gcc/go/gofrontend/gogo.cc:6071
0x1ecd41 Gogo::determine_types()
        /home/tcwg-buildslave/workspace/tcwg-buildfarm/tcwg-apm_32-build/snapshots/gcc.git~master/gcc/go/gofrontend/gogo.cc:5148
0x1e81ad go_parse_input_files(char const**, unsigned int, bool, bool)
        /home/tcwg-buildslave/workspace/tcwg-buildfarm/tcwg-apm_32-build/snapshots/gcc.git~master/gcc/go/gofrontend/go.cc:133
0x1e5115 go_langhook_parse_file
        /home/tcwg-buildslave/workspace/tcwg-buildfarm/tcwg-apm_32-build/snapshots/gcc.git~master/gcc/go/go-lang.c:329
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See <https://gcc.gnu.org/bugs/> for instructions.
FAIL: context
but the next build is still running, I don't know yet if it passes.
Comment 3 Christophe Lyon 2018-02-06 14:07:15 UTC
> FAIL: context
> but the next build is still running, I don't know yet if it passes.

Next build completed, and:
PASS: context

Did something change wrt this test since yesterday?
Comment 4 H.J. Lu 2018-02-06 14:12:04 UTC
FWIW, I have seen these on i686 and x86-64.  They are purely random.
Comment 5 Christophe Lyon 2018-02-08 11:58:19 UTC
Today on ARM:
--- FAIL: TestExtraFiles (1.19s)
        testing.go:560: TestExtraFiles: Something already leaked - closed fd 63
        testing.go:560: Run: exit status 1; stdout "leaked parent file. fd = 58; want 57\n", stderr ""
FAIL
FAIL: os/exec
Comment 6 ian@gcc.gnu.org 2018-02-12 18:50:47 UTC
Author: ian
Date: Mon Feb 12 18:50:16 2018
New Revision: 257599

URL: https://gcc.gnu.org/viewcvs?rev=257599&root=gcc&view=rev
Log:
	PR go/84215
    runtime, sync/atomic: use write barrier for atomic pointer functions
    
    This copies atomic_pointer.go from 1.10rc2.  It was omitted during the
    transition of the runtime from C to Go, and I forgot about it.
    
    This may help with https://gcc.gnu.org/PR84215.
    
    Reviewed-on: https://go-review.googlesource.com/93197

Added:
    trunk/libgo/go/runtime/atomic_pointer.go
Modified:
    trunk/gcc/go/gofrontend/MERGE
    trunk/libgo/go/runtime/stubs.go
    trunk/libgo/go/sync/atomic/atomic.c
Comment 7 Ian Lance Taylor 2018-02-12 18:57:01 UTC
I just committed a patch that may affect some of these failures.  Please let me know if you notice any change.  Thanks.
Comment 8 Christophe Lyon 2018-02-16 15:45:35 UTC
Still seeing at least os/signal having random pass/fail.
Comment 9 H.J. Lu 2018-02-16 22:24:11 UTC
On i686 with many cores, r257758 gave:

...
goroutine 24690 [GC worker (idle)]:
runtime.mcall
        /export/gnu/import/git/sources/gcc/libgo/runtime/proc.c:342
runtime.gopark
        /export/build/gnu/gcc-32bit/build-i686-linux/i686-linux/libgo/gotest330678/test/proc.go:321
runtime.gcBgMarkWorker
        /export/build/gnu/gcc-32bit/build-i686-linux/i686-linux/libgo/gotest330678/test/mgc.go:1776
runtime.kickoff
        /export/build/gnu/gcc-32bit/build-i686-linux/i686-linux/libgo/gotest330678/test/proc.go:1161
created by runtime.gcBgMarkStartWorkers
        /export/build/gnu/gcc-32bit/build-i686-linux/i686-linux/libgo/gotest330678/test/mgc.go:1722 +92

goroutine 23558 [running]:
        goroutine running on other thread; stack unavailable
created by testing.T.Run
        /export/gnu/import/git/sources/gcc/libgo/go/testing/testing.go:824 +740

goroutine 24113 [chan send]:
runtime.mcall
        /export/gnu/import/git/sources/gcc/libgo/runtime/proc.c:342
runtime.gopark
        /export/build/gnu/gcc-32bit/build-i686-linux/i686-linux/libgo/gotest330678/test/proc.go:321
runtime.goparkunlock
        /export/build/gnu/gcc-32bit/build-i686-linux/i686-linux/libgo/gotest330678/test/proc.go:327
runtime.chansend
        /export/build/gnu/gcc-32bit/build-i686-linux/i686-linux/libgo/gotest330678/test/chan.go:249
runtime.chansend1
        /export/build/gnu/gcc-32bit/build-i686-linux/i686-linux/libgo/gotest330678/test/chan.go:135
testing.tRunner..func1
        /export/gnu/import/git/sources/gcc/libgo/go/testing/testing.go:728
runtime.deferreturn
        /export/build/gnu/gcc-32bit/build-i686-linux/i686-linux/libgo/gotest330678/test/panic.go:205
testing.tRunner
        /export/gnu/import/git/sources/gcc/libgo/go/testing/testing.go:783
runtime.kickoff
        /export/build/gnu/gcc-32bit/build-i686-linux/i686-linux/libgo/gotest330678/test/proc.go:1161
created by testing.T.Run
        /export/gnu/import/git/sources/gcc/libgo/go/testing/testing.go:824 +740

eax    0x0
ebx    0x2
ecx    0xb93e1afc
edx    0x0
edi    0x0
esi    0x8
ebp    0xb93e1afc
esp    0xb93e1ae0
eip    0xf7f42db9
eflags 0x282
cs     0x23
fs     0x0
gs     0x63
FAIL: runtime

goroutine 4122 [IO wait]:
internal_poll.runtime_pollWait
        /export/gnu/import/git/sources/gcc/libgo/go/runtime/netpoll.go:176
internal_poll.pollDesc.wait
        /export/gnu/import/git/sources/gcc/libgo/go/internal/poll/fd_poll_runtime.go:85
internal_poll.pollDesc.waitRead
        /export/gnu/import/git/sources/gcc/libgo/go/internal/poll/fd_poll_runtime.go:90
internal_poll.FD.Accept
        /export/gnu/import/git/sources/gcc/libgo/go/internal/poll/fd_unix.go:372
net.netFD.accept
        /export/gnu/import/git/sources/gcc/libgo/go/net/fd_unix.go:238
net.TCPListener.accept
        /export/gnu/import/git/sources/gcc/libgo/go/net/tcpsock_posix.go:136
net.TCPListener.Accept
        /export/gnu/import/git/sources/gcc/libgo/go/net/tcpsock.go:259
net_http.Server.Serve
        /export/build/gnu/gcc-32bit/build-i686-linux/i686-linux/libgo/gotest249618/test/server.go:2770
httptest.func1
        /export/gnu/import/git/sources/gcc/libgo/go/net/http/httptest/server.go:280
created by net_http_httptest.Server.goServe
        /export/gnu/import/git/sources/gcc/libgo/go/net/http/httptest/server.go:278 +198

goroutine 4535 [semacquire]:
syscall.Accept4
        /export/gnu/import/git/sources/gcc/libgo/go/syscall/libcall_linux.go:176
poll.accept
        /export/gnu/import/git/sources/gcc/libgo/go/internal/poll/sock_cloexec.go:17
internal_poll.FD.Accept
        /export/gnu/import/git/sources/gcc/libgo/go/internal/poll/fd_unix.go:365
net.netFD.accept
        /export/gnu/import/git/sources/gcc/libgo/go/net/fd_unix.go:238
net.TCPListener.accept
        /export/gnu/import/git/sources/gcc/libgo/go/net/tcpsock_posix.go:136
net.TCPListener.Accept
        /export/gnu/import/git/sources/gcc/libgo/go/net/tcpsock.go:259
crypto_tls.listener.Accept
        /export/gnu/import/git/sources/gcc/libgo/go/crypto/tls/tls.go:52
net_http.Server.Serve
        /export/build/gnu/gcc-32bit/build-i686-linux/i686-linux/libgo/gotest249618/test/server.go:2770
httptest.func1
        /export/gnu/import/git/sources/gcc/libgo/go/net/http/httptest/server.go:280
created by net_http_httptest.Server.goServe
        /export/gnu/import/git/sources/gcc/libgo/go/net/http/httptest/server.go:278 +198

eax    0x0
ebx    0x2
ecx    0xce0f1fbc
edx    0x0
edi    0x0
esi    0x8
ebp    0xce0f1fbc
esp    0xce0f1fa0
eip    0xf7f3edb9
eflags 0x286
cs     0x23
fs     0x0
gs     0x63
FAIL: net/http

/tmp/cc2A9bsi.s: Assembler messages:
/tmp/cc2A9bsi.s:7954: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:7965: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:7976: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:7977: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:7978: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:7979: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:7993: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8004: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8015: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8016: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8017: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8018: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8032: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8043: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8054: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8055: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8056: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8057: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8071: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8082: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8093: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8094: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8095: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8096: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8110: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8121: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8132: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8133: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8134: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8135: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8149: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8160: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8171: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8172: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8173: Error: leb128 operand is an undefined symbol: .LVU443
/tmp/cc2A9bsi.s:8174: Error: leb128 operand is an undefined symbol: .LVU443


FAIL: golang_org/x/crypto/chacha20poly1305/internal/chacha20

FAIL: golang_org/x/crypto/curve25519
FAIL: golang_org/x/net/lex/httplex

checksyms: found unexpected symbol "__libc_start_main@@GLIBC_2.0"
FAIL: checksyms

spawn [open ...]^M
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=2 addr=4140130304 pc=4152026464]

runtime stack:
doscanstack1
        /export/gnu/import/git/sources/gcc/libgo/runtime/stack.c:77
runtime_mstart
        /export/gnu/import/git/sources/gcc/libgo/runtime/proc.c:527

goroutine 33 [GC worker (idle)]:
created by runtime.gcStart
        /export/gnu/import/git/sources/gcc/libgo/go/runtime/mgc.go:1299 +412

goroutine 1 [GC assist marking (scan)]:
panic
        /export/gnu/import/git/sources/gcc/libgo/go/runtime/panic.go:462
runtime_panicstring
        /export/gnu/import/git/sources/gcc/libgo/runtime/panic.c:38
main.main..func1
        ./index0-out.go:140
main.test
        ./index0-out.go:112
main.main
        ./index0-out.go:140
FAIL: ./index0-out.go execution,  -O0 -g -fno-var-tracking-assignments
Comment 10 Christophe Lyon 2018-03-13 08:32:15 UTC
Yesterday I noticed:
* on arm:
FAIL: libgo:database/sql
* on aarch64:
FAIL: libgo:runtime/pprof
Comment 11 Ian Lance Taylor 2018-03-13 12:51:21 UTC
Please include all the output from each failure, if possible.  Thanks.
Comment 12 Christophe Lyon 2018-03-13 14:23:12 UTC
It's not very informative.

On arm:
PASS: crypto/x509
timed out in gotest
/home/tcwg-buildslave/workspace/tcwg-buildfarm/tcwg-apm_32-build/snapshots/gcc.git~master/libgo/testsuite/gotest: line 624: 15642 Killed                  ./a.out -test.short -test.timeout=${timeout}s "$@"
FAIL: database/sql
PASS
PASS: database/sql/driver

On aarch64:
PASS: runtime/internal/sys
/home/tcwg-buildslave/workspace/tcwg-buildfarm/tcwg-apm_64-build/snapshots/gcc.git~master/libgo/testsuite/gotest: line 624: 28388 Segmentation fault      (core dumped) ./a.out -test.short -test.timeout=${timeout}s "$@"
FAIL: runtime/pprof
PASS
PASS: runtime/pprof/internal/profile

I included the PASS lines only to show the actual context around the FAIL lines.