pdns reportedly now crashes like the following after known good r271393 and with known bad r272147 #0 0x00005555559ab4f0 in std::_Rb_tree<vState, std::pair<vState const, std::atomic<unsigned long> >, std::_Select1st<std::pair<vState const, std::atomic<unsigned long> > >, std::less<vState>, std::allocator<std::pair<vState const, std::atomic<unsigned long> > > >::_M_get_insert_unique_pos(vState const&) [clone .constprop.0] () #1 0x000055555568bf8f in std::_Rb_tree<vState, std::pair<vState const, std::atomic<unsigned long> >, std::_Select1st<std::pair<vState const, std::atomic<unsigned long> > >, std::less<vState>, std::allocator<std::pair<vState const, std::atomic<unsigned long> > > >::_M_get_insert_hint_unique_pos (this=0x555555a7cbc8 <g_stats+456>, __k=@0x7ffff0003880: Insecure, __position=...) at /usr/include/c++/9/bits/stl_tree.h:2233 #2 std::_Rb_tree<vState, std::pair<vState const, std::atomic<unsigned long> >, std::_Select1st<std::pair<vState const, std::atomic<unsigned long> > >, std::less<vState>, std::allocator<std::pair<vState const, std::atomic<unsigned long> > > >::_M_emplace_hint_unique<std::piecewise_construct_t const&, std::tuple<vState&&>, std::tuple<> >(std::_Rb_tree_const_iterator<std::pair<vState const, std::atomic<unsigned long> > >, std::piecewise_construct_t const&, std::tuple<vState&&>&&, std::tuple<>&&) [clone .constprop.0] (__pos=..., this=0x555555a7cbc8 <g_stats+456>) at /usr/include/c++/9/bits/stl_tree.h:2459 #3 0x000055555569a70c in std::map<vState, std::atomic<unsigned long>, std::less<vState>, std::allocator<std::pair<vState const, std::atomic<unsigned long> > > >::operator[](vState&&) [clone .constprop.0] ( __k=@0x7ffff68bdef0: Insecure, this=<optimized out>) at /usr/include/c++/9/bits/stl_map.h:518 #4 0x0000555555740cff in registerAllStats () at rec_channel_rec.cc:1033 #5 0x000055555577743c in recursorThread (n=<optimized out>, worker=<optimized out>) at pdns_recursor.cc:3437 #6 0x00007ffff7362e40 in std::execute_native_thread_routine (__p=0x555555c91050) at ../../../../../libstdc++-v3/src/c++11/thread.cc:80 #7 0x00007ffff7112faa in start_thread (arg=<optimized out>) at pthread_create.c:486 #8 0x00007ffff7eed73f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 /etc/pdns/recursor.conf: ``` forward-zones=cubes.nordisch.org=172.16.16.1 16.172.in-addr.arpa=172.16.16.1 allow-from=127.0.0.0/8, 10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12 export-etc-hosts dnssec=validate local-address=0.0.0.0,:: local-port=1154 setgid=pdns setuid=pdns ``` it should be reproducible with gdb --args /usr/sbin/pdns_recursor --daemon=no --write-pid=no --disable-syslog --log-timestamp=no
Possible culplrit could be the PR85965 backport changes?
That mostly just adds static_assert checks, which can't change anything at runtime.
(In reply to Jonathan Wakely from comment #2) > That mostly just adds static_assert checks, which can't change anything at > runtime. It changes how we get to keys/values besides moving static asserts but I know nothing about this code so cannot assess whether the change does anything. At least it's the only change in the revision range that looks remotely related to the backtrace (maybe the tuples change as well). But I haven't yet tried to reproduce either.
In the end the issue may have been caused by building with LTO.
I'll debug that.
Hm, I can't reproduce that on my openSUSE TW which should have the problematic package: $ zypper info pdns-recursor ... Version : 4.2.0-1.1 $ /usr/sbin/pdns_recursor --daemon=no --write-pid=no --disable-syslog --log-timestamp=no --socket-dir=/tmp/ --config-dir=/etc/pdns/ Jul 23 12:32:37 Asked to run with pdns-distributes-queries set but no distributor threads, raising to 1 PowerDNS Recursor 4.2.0 (C) 2001-2019 PowerDNS.COM BV Using 64-bits mode. Built using gcc 9.1.1 20190611 [gcc-9-branch revision 272147]. PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2. If using IPv6, please raise sysctl net.ipv6.route.max_size, currently set to 4096 which is < 16384 NOT using IPv6 for outgoing queries - set 'query-local-address6=::' to enable Only allowing queries from: 127.0.0.0/8, 10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12 Will not send queries to: 127.0.0.0/8, 10.0.0.0/8, 100.64.0.0/10, 169.254.0.0/16, 192.168.0.0/16, 172.16.0.0/12, ::1/128, fc00::/7, fe80::/10, 0.0.0.0/8, 192.0.0.0/24, 192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24, 240.0.0.0/4, ::/96, ::ffff:0:0/96, 100::/64, 2001:db8::/32, 0.0.0.0, :: PowerDNS Recursor itself will distribute queries over threads Redirecting queries for zone 'cubes.nordisch.org' to: 172.16.16.1:53 Redirecting queries for zone '16.172.in-addr.arpa' to: 172.16.16.1:53 Inserting forward zone 'localhost' based on hosts file Inserting reverse zone '1.0.0.127.in-addr.arpa' based on hosts file Inserting forward zone 'i586' based on hosts file Inserting reverse zone '157.122.168.192.in-addr.arpa' based on hosts file Inserting forward zone 'i686' based on hosts file Inserting reverse zone '243.122.168.192.in-addr.arpa' based on hosts file Inserting rfc 1918 private space zones Will not overwrite zone '16.172.in-addr.arpa' already loaded Listening for UDP queries on 0.0.0.0:1154 Listening for UDP queries on [::]:1154 Enabled TCP data-ready filter for (slight) DoS protection Listening for TCP queries on 0.0.0.0:1154 Listening for TCP queries on [::]:1154 Raised soft limit on number of filedescriptors to 4121 to match max-mthreads and threads settings Set effective group id to 441 Set effective user id to 446 Launching 1 distributor threads Launching 2 worker threads Done priming cache with root hints Done priming cache with root hints Done priming cache with root hints Done priming cache with root hints Enabled 'epoll' multiplexer and I also build one from source files (w/ -flto and -O2): $ ./pdns_recursor --daemon=no --write-pid=no --disable-syslog --log-timestamp=no --socket-dir=/tmp/ --config-dir=/etc/pdns/ Jul 23 12:32:56 Asked to run with pdns-distributes-queries set but no distributor threads, raising to 1 PowerDNS Recursor 0.0.17287.0.master.g34f3230a08 (C) 2001-2019 PowerDNS.COM BV Using 64-bits mode. Built using gcc 9.1.1 20190703 [gcc-9-branch revision 273008] on Jul 23 2019 12:27:05 by marxin@marxinbox.suse.cz. PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2. If using IPv6, please raise sysctl net.ipv6.route.max_size, currently set to 4096 which is < 16384 NOT using IPv6 for outgoing queries - set 'query-local-address6=::' to enable Only allowing queries from: 127.0.0.0/8, 10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12 Will not send queries to: 127.0.0.0/8, 10.0.0.0/8, 100.64.0.0/10, 169.254.0.0/16, 192.168.0.0/16, 172.16.0.0/12, ::1/128, fc00::/7, fe80::/10, 0.0.0.0/8, 192.0.0.0/24, 192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24, 240.0.0.0/4, ::/96, ::ffff:0:0/96, 100::/64, 2001:db8::/32, 0.0.0.0, :: PowerDNS Recursor itself will distribute queries over threads Redirecting queries for zone 'cubes.nordisch.org' to: 172.16.16.1:53 Redirecting queries for zone '16.172.in-addr.arpa' to: 172.16.16.1:53 Inserting forward zone 'localhost' based on hosts file Inserting reverse zone '1.0.0.127.in-addr.arpa' based on hosts file Inserting forward zone 'i586' based on hosts file Inserting reverse zone '157.122.168.192.in-addr.arpa' based on hosts file Inserting forward zone 'i686' based on hosts file Inserting reverse zone '243.122.168.192.in-addr.arpa' based on hosts file Inserting rfc 1918 private space zones Will not overwrite zone '16.172.in-addr.arpa' already loaded Listening for UDP queries on 0.0.0.0:1154 Listening for UDP queries on [::]:1154 Enabled TCP data-ready filter for (slight) DoS protection Listening for TCP queries on 0.0.0.0:1154 Listening for TCP queries on [::]:1154 Raised soft limit on number of filedescriptors to 4121 to match max-mthreads and threads settings Set effective group id to 441 Set effective user id to 446 Launching 1 distributor threads Launching 2 worker threads Done priming cache with root hints Done priming cache with root hints Done priming cache with root hints Done priming cache with root hints Enabled 'epoll' multiplexer
Michael Schroeder wrote: As said I only had this issue when using pdns-recursor 4.1.13 on Tumbleweed, but not with 4.2.0. Since 4.2.0 should reach Tumbleweed really soon I don't know whether it's worth the effort. Ciao, Michael.