I've reinstalled my workstation with Fedora 29 x86_64 recently, and since that both my 32-bit bootstraps hanged in libphobos testing (while 64-bit ones were ok). In both cases, I saw a stuck process: 25678 pts/6 Sl+ 0:00 /home/jakub/src/gcc/obj37/i686-pc-linux-gnu/libphobos/src/.libs/lt-unittest std.net.curl 25679 pts/6 Z+ 0:00 [cat] <defunct> When I've tried to attach to it, one thread is waiting in pthread_join #0 0xf7fc3049 in __kernel_vsyscall () #1 0xf2cf1aa6 in __pthread_timedjoin_ex () from /lib/libpthread.so.0 #2 0xf2cf1898 in pthread_join () from /lib/libpthread.so.0 #3 0xf2ea1004 in core.thread.Thread.join(bool) (rethrow=true, this=0xf29ef180) at ../../../../libphobos/libdruntime/core/thread.d:776 #4 thread_joinAll () at ../../../../libphobos/libdruntime/core/thread.d:2369 #5 0xf2ebfa5c in rt_term () at ../../../../libphobos/libdruntime/rt/dmain2.d:215 #6 0xf2ebfac9 in runAll (this=0xffd5b97c) at ../../../../libphobos/libdruntime/rt/dmain2.d:490 #7 0xf2ebf6f1 in tryExec (this=0xffd5b97c, dg=...) at ../../../../libphobos/libdruntime/rt/dmain2.d:461 #8 0xf2ebf8ee in _d_run_main (argc=2, argv=0xffd5ba74, mainFunc=0x8049810 <D main>) at ../../../../libphobos/libdruntime/rt/dmain2.d:494 #9 0x08049281 in main (argc=2, argv=0xffd5ba74) at /home/jakub/src/gcc/libphobos/libdruntime/__entrypoint.di:44 #10 0xf2b34c09 in __libc_start_main () from /lib/libc.so.6 #11 0x08049326 in _start () while the other one is stuck in: #0 0xf7fc3049 in __kernel_vsyscall () #1 0xf2cfaf11 in accept () from /lib/libpthread.so.0 #2 0xf7233398 in std.socket.Socket.accept() (this=0xf29f0010) at ../../../../libphobos/src/std/socket.d:2876 #3 0xf6b64e47 in std.net.curl.TestServer.loop(shared(std.socket.TcpSocket)) (listener=0xf29f0010) at ../../../../libphobos/src/std/net/curl.d:205 #4 0xf6b81daa in std.concurrency.exec (this=0xf29f0060) at /home/jakub/src/gcc/libphobos/src/std/concurrency.d:506 #5 0xf2e9e547 in core.thread.Thread.run() (this=0xf29ef180) at ../../../../libphobos/libdruntime/core/thread.d:1469 #6 core.thread.Thread.run() (this=0xf29ef180) at ../../../../libphobos/libdruntime/core/thread.d:1461 #7 thread_entryPoint (arg=<optimized out>) at ../../../../libphobos/libdruntime/core/thread.d:400 #8 0xf2cf05de in start_thread () from /lib/libpthread.so.0 #9 0xf2c0697a in clone () from /lib/libc.so.6 When I try to run it by hand it showed a clue, but hanged anyway: .libs/lt-unittest std.net.curl ****** FAIL release32 std.net.curl std.net.curl.CurlException@../../../../libphobos/src/std/net/curl.d(4188): Failed to load curl, tried "libcurl.so", "libcurl.so.4", "libcurl-gnutls.so.4", "libcurl-nss.so.4", "libcurl.so.3". ---------------- /home/jakub/src/gcc/libphobos/src/std/exception.d:420 pure @safe void std.exception.bailOut!(std.net.curl.CurlException).bailOut(immutable(char)[], uint, const(char[])) [0xf6bb1f08] /home/jakub/src/gcc/libphobos/src/std/exception.d:388 pure @safe bool std.exception.enforce!(std.net.curl.CurlException, bool).enforce(bool, lazy const(char)[], immutable(char)[], uint) [0xf6b737e5] ../../../../libphobos/src/std/net/curl.d:4188 void* std.net.curl.CurlAPI.loadAPI() [0xf6b6bc40] ../../../../libphobos/src/std/net/curl.d:4135 __dgliteral1 [0xf6b6bab2] /home/jakub/src/gcc/libphobos/src/std/concurrency.d:2422 __dgliteral2 [0xf6bdd299] /home/jakub/src/gcc/libphobos/src/std/concurrency.d:2493 ref void* std.concurrency.initOnce!(std.net.curl.CurlAPI._handle).initOnce(lazy void*, core.sync.mutex.Mutex) [0xf6bdd212] /home/jakub/src/gcc/libphobos/src/std/concurrency.d:2422 ref void* std.concurrency.initOnce!(std.net.curl.CurlAPI._handle).initOnce(lazy void*) [0xf6b73a1b] ../../../../libphobos/src/std/net/curl.d:4135 ref @property std.net.curl.CurlAPI.API std.net.curl.CurlAPI.instance() [0xf6b6ba89] ../../../../libphobos/src/std/net/curl.d:4245 ref @property std.net.curl.CurlAPI.API std.net.curl.Curl.curl() [0xf6b629d6] ../../../../libphobos/src/std/net/curl.d:4268 void std.net.curl.Curl.initialize() [0xf6b64d95] ../../../../libphobos/src/std/net/curl.d:2547 void std.net.curl.HTTP.initialize() [0xf6b6430f] ../../../../libphobos/src/std/net/curl.d:2520 std.net.curl.HTTP std.net.curl.HTTP.opCall() [0xf6b5e8d2] ../../../../libphobos/src/std/net/curl.d:416 void std.net.curl.download!(std.net.curl.AutoProtocol).download(const(char)[], immutable(char)[], std.net.curl.AutoProtocol) [0xf6b6e764] ../../../../libphobos/src/std/net/curl.d:433 void std.net.curl.__unittestL420_2() [0xf6b5e292] /home/jakub/src/gcc/obj37/i686-pc-linux-gnu/libphobos/src/<no_file>:1 void std.net.curl.__modtest() [0xf6b6d5d9] ../../../../libphobos/libdruntime/../testsuite/test_runner.d:58 void test_runner.doTest(object.ModuleInfo*, ref bool) [0x80495eb] ../../../../libphobos/libdruntime/../testsuite/test_runner.d:30 bool test_runner.testModules() [0x8049736] ../../../../libphobos/libdruntime/core/runtime.d:570 runModuleUnitTests [0xf2e92b44] ../../../../libphobos/libdruntime/rt/dmain2.d:485 runAll [0xf2eb8aec] ../../../../libphobos/libdruntime/rt/dmain2.d:461 tryExec [0xf2eb86f0] ../../../../libphobos/libdruntime/rt/dmain2.d:494 _d_run_main [0xf2eb88ed] /home/jakub/src/gcc/libphobos/libdruntime/__entrypoint.di:44 main [0x8049280] ??:? __libc_start_main [0xf2b2ec08] ??:? _start [0x8049325] ??:? ???[0xffffffff] Indeed, I had just 64-bit libcurl.so.4 installed, not 32-bit libcurl.so.4 and installing the latter makes the command run from the command line succeed. The reason I'm filing this is that there are multiple issues: 1) what I'm worried about most is that the timeouts for tests don't work, it happens that some test gets stuck, but it should be killed after 5 minutes or for how long the default timeout is set and the set should FAIL in that case or something similar 2) if Curl fails to initialize, the test shouldn't get stuck 3) and, if libcurl isn't available, I think it would be better to skip the test as UNSUPPORTED, i.e. add some effective-target that tests if libcurl is available and if it fails, don't even try to run the test
I think I saw a duplicate somewhere - this is using real-time threads pinned to a single CPU but yield()ing in a spinning loop expecting to make progress.
(In reply to Richard Biener from comment #1) > I think I saw a duplicate somewhere - this is using real-time threads pinned > to a single CPU but yield()ing in a spinning loop expecting to make progress. I checked the multilib issue, and couldn't see it. Had a look on D's bugtracker, and found this related issue: https://issues.dlang.org/show_bug.cgi?id=18519 Raised a pull request against upstream phobos that fixes the test itself: https://github.com/dlang/phobos/pull/6824 Though I haven't yet checked running in dejagnu that the desirable behaviour transfers over well.
(In reply to Jakub Jelinek from comment #0) > The reason I'm filing this is that there are multiple issues: > > 1) what I'm worried about most is that the timeouts for tests don't work, it > happens that some test gets stuck, but it should be killed after 5 minutes > or for how long the default timeout is set and the set should FAIL in that > case or something similar > I couldn't reproduce, maybe I'm running the 32bit testsuite differently though? Running ../../../../../libphobos/testsuite/libphobos.unittests/unittests.exp ... WARNING: program timed out. FAIL: libphobos.unittests/phobos/static/std.net.curl WARNING: program timed out. FAIL: libphobos.unittests/phobos/shared/std.net.curl > 2) if Curl fails to initialize, the test shouldn't get stuck > Patch I've sent upstream fixes this. > 3) and, if libcurl isn't available, I think it would be better to skip the > test as UNSUPPORTED, i.e. add some effective-target that tests if libcurl is > available and if it fails, don't even try to run the test This would be a proc local to libphobos I guess? There's nothing using this module outside of the libphobos testsuite.
(In reply to Iain Buclaw from comment #3) > (In reply to Jakub Jelinek from comment #0) > > The reason I'm filing this is that there are multiple issues: > > > > 1) what I'm worried about most is that the timeouts for tests don't work, it > > happens that some test gets stuck, but it should be killed after 5 minutes > > or for how long the default timeout is set and the set should FAIL in that > > case or something similar > > > > I couldn't reproduce, maybe I'm running the 32bit testsuite differently > though? > > Running ../../../../../libphobos/testsuite/libphobos.unittests/unittests.exp > ... > WARNING: program timed out. > FAIL: libphobos.unittests/phobos/static/std.net.curl > WARNING: program timed out. > FAIL: libphobos.unittests/phobos/shared/std.net.curl My i686-linux bootstraps are done through a couple of executable scripts in ~/hbin directory on x86_64-linux: for i in ~/hbin/*; do echo ===$i===; cat $i; done ===/home/jakub/hbin/as=== #!/bin/sh exec /usr/bin/as --32 "$@" ===/home/jakub/hbin/g++=== #!/bin/sh exec /usr/bin/g++ -m32 "$@" ===/home/jakub/hbin/gcc=== #!/bin/sh exec /usr/bin/gcc -m32 "$@" ===/home/jakub/hbin/ld=== #!/bin/sh case "$*" in --version) cat <<\EOF GNU ld version 2.20.52.0.1-10.fc17 20100131 Copyright 2012 Free Software Foundation, Inc. This program is free software; you may redistribute it under the terms of the GNU General Public License version 3 or (at your option) a later version. This program has absolutely no warranty. EOF exit 0;; esac exec /usr/bin/ld -m elf_i386 -L /usr/lib/ "$@" and then PATH=~/hbin:$PATH i386 ../configure --enable-languages=default,obj-c++,lto,go,brig,d --enable-checking=yes,rtl,extra \ && PATH=~/hbin:$PATH i386 make -j32 bootstrap > LOG 2>&1 \ && PATH=~/hbin:$PATH i386 make -j32 -k check > LOGC 2>&1; \ ../contrib/test_summary > LOGT 2>&1 and that got stuck forever if libcurl.i?86 wasn't installed (only libcurl.x86_64 and libcurl-devel.x86_64), but works fine now that I also have libcurl.i?86 installed. > > 3) and, if libcurl isn't available, I think it would be better to skip the > > test as UNSUPPORTED, i.e. add some effective-target that tests if libcurl is > > available and if it fails, don't even try to run the test > > This would be a proc local to libphobos I guess? There's nothing using this > module outside of the libphobos testsuite. If the test doesn't fail if it fails to load libcurl (and doesn't get stuck), it is fine too. If it fails if it is missing, yeah, something like a tcl procedure somewhere in libphobos/testsuite/lib/*.exp would do. As libphobos itself doesn't need libcurl-devel, it is just dlopen, so I think you want to test that #include <dlfcn.h> int main () { void *h = dlopen ("libcurl.so.4"); // etc., whatever the library will try to dlopen ... } will succeed dlopening it.
(In reply to Jakub Jelinek from comment #4) > > If the test doesn't fail if it fails to load libcurl (and doesn't get > stuck), it is fine too. > If it fails if it is missing, yeah, something like a tcl procedure somewhere > in libphobos/testsuite/lib/*.exp would do. > As libphobos itself doesn't need libcurl-devel, it is just dlopen, so I > think you want to test that > #include <dlfcn.h> > > int > main () > { > void *h = dlopen ("libcurl.so.4"); // etc., whatever the library will try > to dlopen > ... > } > will succeed dlopening it. The names it will try before giving up: --- version (OSX) static immutable names = ["libcurl.4.dylib"]; else version (Posix) { static immutable names = ["libcurl.so", "libcurl.so.4", "libcurl-gnutls.so.4", "libcurl-nss.so.4", "libcurl.so.3"]; } else version (Windows) static immutable names = ["libcurl.dll", "curl.dll"]; --- I think a link test would be good enough, as iff curl exists, but it has a different name to any of the above, then the std.net.curl module needs fixing.
(In reply to Jakub Jelinek from comment #4) > > My i686-linux bootstraps are done through a couple of executable scripts in > ~/hbin directory on x86_64-linux: > for i in ~/hbin/*; do echo ===$i===; cat $i; done > ===/home/jakub/hbin/as=== > #!/bin/sh > exec /usr/bin/as --32 "$@" > ===/home/jakub/hbin/g++=== > #!/bin/sh > exec /usr/bin/g++ -m32 "$@" > ===/home/jakub/hbin/gcc=== > #!/bin/sh > exec /usr/bin/gcc -m32 "$@" > ===/home/jakub/hbin/ld=== > #!/bin/sh > case "$*" in > --version) cat <<\EOF > GNU ld version 2.20.52.0.1-10.fc17 20100131 > Copyright 2012 Free Software Foundation, Inc. > This program is free software; you may redistribute it under the terms of > the GNU General Public License version 3 or (at your option) a later version. > This program has absolutely no warranty. > EOF > exit 0;; > esac > exec /usr/bin/ld -m elf_i386 -L /usr/lib/ "$@" > > and then > PATH=~/hbin:$PATH i386 ../configure > --enable-languages=default,obj-c++,lto,go,brig,d > --enable-checking=yes,rtl,extra \ > && PATH=~/hbin:$PATH i386 make -j32 bootstrap > LOG 2>&1 \ > && PATH=~/hbin:$PATH i386 make -j32 -k check > LOGC 2>&1; \ > ../contrib/test_summary > LOGT 2>&1 > and that got stuck forever if libcurl.i?86 wasn't installed (only > libcurl.x86_64 and libcurl-devel.x86_64), but works > fine now that I also have libcurl.i?86 installed. > I can't say I followed this to the letter, but I could not reproduce. RUNTESTFLAGS="unittests.exp" LIBRARY_PATH=/usr/lib32 PATH=~/hbin:$PATH \ i386 make check -j8; \ ../contrib/test_summary Naturally, the libphobos testsuite took over 660 seconds to finish. But still it eventually ended. Going to deal with [2] and [3] anyway to prevent hitting the timeout in the first place anyway.
(In reply to Jakub Jelinek from comment #0) > > 2) if Curl fails to initialize, the test shouldn't get stuck > This part has been done in r268746.
Author: ibuclaw Date: Wed Apr 24 14:17:34 2019 New Revision: 270545 URL: https://gcc.gnu.org/viewcvs?rev=270545&root=gcc&view=rev Log: libphobos: Skip curl tests if libcurl is not installed on the target. libphobos/ChangeLog: 2019-04-24 Iain Buclaw <ibuclaw@gdcproject.org> PR d/88654 * testsuite/lib/libphobos.exp (libphobos-dg-test): Check libphobos_skipped_test_p before running test. (libphobos-dg-prune): New proc. (libphobos_init): Set libphobos_skip_tests. (libphobos_skipped_test_p): New proc. (check_effective_target_libcurl_available): New proc. * testsuite/libphobos.phobos/phobos.exp: Skip curl tests if library not found. * testsuite/libphobos.phobos_shared/phobos_shared.exp: Likewise. Modified: trunk/libphobos/ChangeLog trunk/libphobos/testsuite/lib/libphobos.exp trunk/libphobos/testsuite/libphobos.phobos/phobos.exp trunk/libphobos/testsuite/libphobos.phobos_shared/phobos_shared.exp
(In reply to Jakub Jelinek from comment #0) > 3) and, if libcurl isn't available, I think it would be better to skip the > test as UNSUPPORTED, i.e. add some effective-target that tests if libcurl is > available and if it fails, don't even try to run the test This has been done in r270545, nothing else left do here as far as I can see.