SIGSEGV during VNET shutdown

[erikarn]

uinet shutting down from signal handler (signal 1)
Shutting down all uinet instances...
netmap3 (netmap0:3): Stopping rx thread
netmap3 (netmap0:3): Stopping tx thread
netmap3 (netmap0:3): Interface threads stopped
Freed UMA keg was not empty (18 items). Lost 2 pages of memory.
Freed UMA keg was not empty (168 items). Lost 1 pages of memory.
Freed UMA keg was not empty (466 items). Lost 129 pages of memory.
Freed UMA keg was not empty (529 items). Lost 408 pages of memory.
hhook_vnet_uninit: hhook_head type=1, id=1 cleanup required
hhook_vnet_uninit: hhook_head type=1, id=0 cleanup required
netmap2 (netmap0:2): Stopping rx thread
netmap2 (netmap0:2): Stopping tx thread

load: 3.28 cmd: blockd 23808 [suspended] 2857.94r 6699.35u 2496.73s 33% 318072k
load: 3.28 cmd: blockd 23808 [suspended] 2858.55r 6699.35u 2496.88s 30% 335036k
load: 3.28 cmd: blockd 23808 [suspended] 2858.70r 6699.35u 2496.92s 30% 339512k
load: 3.28 cmd: blockd 23808 [suspended] 2858.98r 6699.35u 2496.98s 30% 347204k
load: 3.28 cmd: blockd 23808 [suspended] 2859.16r 6699.35u 2497.04s 27% 352280k
load: 3.28 cmd: blockd 23808 [suspended] 2859.45r 6699.35u 2497.09s 27% 357268k
load: 3.28 cmd: blockd 23808 [suspended] 2859.61r 6699.35u 2497.12s 27% 357812k
Segmentation fault (core dumped)
root@test3:/home/adrian/git/gitlab/ahc/blockd #
root@test3:/home/adrian/git/gitlab/ahc/blockd #
root@test3:/home/adrian/git/gitlab/ahc/blockd # gdb78 ./blockd
GNU gdb (GDB) 7.8 [GDB v7.8 for FreeBSD]
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd10.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./blockd...done.
(gdb) quit
root@test3:/home/adrian/git/gitlab/ahc/blockd # gdb78 ./blockd ./blockd.core
GNU gdb (GDB) 7.8 [GDB v7.8 for FreeBSD]
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd10.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./blockd...done.
[New process 100773]
[New process 100802]
[New process 100801]
[New process 100800]
[New process 100799]
[New process 100798]
[New process 100797]
[New process 100796]
[New process 100795]
[New process 100794]
[New process 100793]
[New process 100792]
[New process 100791]
[New process 100788]
[New process 100787]
[New process 100786]
[New process 100785]
[New process 100784]
[New process 100783]
[New process 100782]
[New process 100781]
[New process 100780]
[New process 100779]
[New process 100778]
[New process 100777]
[New process 100776]
[New process 100775]
[New process 100774]
[New process 100233]
[New Thread 802c10800 (LWP 100802)]
[New Thread 802c10400 (LWP 100801)]
[New Thread 802c10000 (LWP 100800)]
[New Thread 802c0fc00 (LWP 100799)]
[New Thread 802c0f800 (LWP 100798)]
[New Thread 802c0f400 (LWP 100797)]
[New Thread 802c0f000 (LWP 100796)]
[New Thread 802c0ec00 (LWP 100795)]
[New Thread 802c0e800 (LWP 100794)]
[New Thread 802c0e400 (LWP 100793)]
[New Thread 802c0e000 (LWP 100792)]
[New Thread 802c0dc00 (LWP 100791)]
[New Thread 802c0d000 (LWP 100788)]
[New Thread 802c0cc00 (LWP 100787)]
[New Thread 802c0c800 (LWP 100786)]
[New Thread 802c0c400 (LWP 100785)]
[New Thread 802c0c000 (LWP 100784)]
[New Thread 802c0bc00 (LWP 100783)]
[New Thread 802c0b800 (LWP 100782)]
[New Thread 802c09400 (LWP 100781)]
[New Thread 802c09000 (LWP 100780)]
[New Thread 802c08400 (LWP 100779)]
[New Thread 802c08000 (LWP 100778)]
[New Thread 802c07c00 (LWP 100777)]
[New Thread 802c07800 (LWP 100776)]
[New Thread 802c07400 (LWP 100775)]
[New Thread 802c07000 (LWP 100774)]
[New Thread 802c06c00 (LWP 100773)]
[New Thread 802c06400 (LWP 100233)]
Core was generated by `blockd'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00000008013a5f9a in pthread_mutex_lock () from /lib/libthr.so.3

(gdb) bt
#0 0x00000008013a5f9a in pthread_mutex_lock () from /lib/libthr.so.3
#1 0x000000000049ab99 in tcp_timer_keep (xtp=0x81d9a5830) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/tcp_timer.c:294
#2 0x0000000000422ac0 in softclock (arg=0x7184a0 <cc_cpu>) at uinet_kern_timeout.c:428
#3 0x000000000041ed91 in uinet_hardclock () at uinet_kern_clock.c:56
#4 0x0000000000423446 in timer_intr (arg=) at uinet_kern_timeout.c:844
#5 0x00000000004a5ee2 in pthread_start_routine (arg=0x802c3d3e0) at uinet_host_interface.c:580
#6 0x00000008013a04f5 in ?? () from /lib/libthr.so.3
#7 0x0000000000000000 in ?? ()

Backtrace stopped: Cannot access memory at address 0x7fffffbfe000
(gdb) info threads
Id Target Id Frame
58 Thread 802c06400 (LWP 100233) 0x00000008013ab8cc in ?? () from /lib/libthr.so.3

• 57 Thread 802c06c00 (LWP 100773) 0x00000008013a5f9a in pthread_mutex_lock () from /lib/libthr.so.3
  56 Thread 802c07000 (LWP 100774) 0x00000008013ab8cc in ?? () from /lib/libthr.so.3
  55 Thread 802c07400 (LWP 100775) 0x00000008013ab8cc in ?? () from /lib/libthr.so.3
  54 Thread 802c07800 (LWP 100776) 0x00000008013ab8cc in ?? () from /lib/libthr.so.3
  53 Thread 802c07c00 (LWP 100777) 0x00000008013ab8cc in ?? () from /lib/libthr.so.3
  52 Thread 802c08000 (LWP 100778) 0x00000008013ab8cc in ?? () from /lib/libthr.so.3
  51 Thread 802c08400 (LWP 100779) 0x00000008013ab8cc in ?? () from /lib/libthr.so.3
  50 Thread 802c09000 (LWP 100780) 0x00000008013ab8cc in ?? () from /lib/libthr.so.3
  49 Thread 802c09400 (LWP 100781) 0x000000080226a238 in read () from /lib/libc.so.7
  48 Thread 802c0b800 (LWP 100782) 0x00000008021f2d98 in accept () from /lib/libc.so.7
  47 Thread 802c0bc00 (LWP 100783) 0x00000008013ab8cc in ?? () from /lib/libthr.so.3
  46 Thread 802c0c000 (LWP 100784) 0x000000080226a1da in ioctl () from /lib/libc.so.7
  45 Thread 802c0c400 (LWP 100785) 0x00000008013ab8cc in ?? () from /lib/libthr.so.3
  44 Thread 802c0c800 (LWP 100786) 0x00000008021fa97a in poll () from /lib/libc.so.7
  43 Thread 802c0cc00 (LWP 100787) 0x00000008013ab8cc in ?? () from /lib/libthr.so.3
  42 Thread 802c0d000 (LWP 100788) 0x00000008021d842e in free () from /lib/libc.so.7
  41 Thread 802c0dc00 (LWP 100791) 0x00000008013ab8cc in ?? () from /lib/libthr.so.3
  40 Thread 802c0e000 (LWP 100792) 0x0000000802192048 in _sigwait () from /lib/libc.so.7
  39 Thread 802c0e400 (LWP 100793) 0x00000008021f2d98 in accept () from /lib/libc.so.7
  38 Thread 802c0e800 (LWP 100794) 0x000000080226a238 in read () from /lib/libc.so.7
  37 Thread 802c0ec00 (LWP 100795) 0x0000000802247efa in nanosleep () from /lib/libc.so.7
  36 Thread 802c0f000 (LWP 100796) 0x000000080226a23a in read () from /lib/libc.so.7
  35 Thread 802c0f400 (LWP 100797) 0x0000000802247efa in nanosleep () from /lib/libc.so.7
  34 Thread 802c0f800 (LWP 100798) 0x000000080224968a in kevent () from /lib/libc.so.7
  33 Thread 802c0fc00 (LWP 100799) 0x0000000802247efa in nanosleep () from /lib/libc.so.7
  32 Thread 802c10000 (LWP 100800) 0x000000080224968a in kevent () from /lib/libc.so.7
  31 Thread 802c10400 (LWP 100801) 0x0000000802247efa in nanosleep () from /lib/libc.so.7
  30 Thread 802c10800 (LWP 100802) 0x000000080224968a in kevent () from /lib/libc.so.7
• 1 Thread 802c06c00 (LWP 100773) 0x00000008013a5f9a in pthread_mutex_lock () from /lib/libthr.so.3
  (gdb) frame 0
  #0 0x00000008013a5f9a in pthread_mutex_lock () from /lib/libthr.so.3
  (gdb) frame 1
  Master adrian #1 0x000000000049ab99 in tcp_timer_keep (xtp=0x81d9a5830) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/tcp_timer.c:294
  294 INP_INFO_WLOCK(&V_tcbinfo);
  (gdb) bt
  #0 0x00000008013a5f9a in pthread_mutex_lock () from /lib/libthr.so.3
  Master adrian #1 0x000000000049ab99 in tcp_timer_keep (xtp=0x81d9a5830) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/tcp_timer.c:294
  Passive receive adrian #2 0x0000000000422ac0 in softclock (arg=0x7184a0 <cc_cpu>) at uinet_kern_timeout.c:428
  Not able to compile in Ubuntu #3 0x000000000041ed91 in uinet_hardclock () at uinet_kern_clock.c:56
  Performance Measurement #4 0x0000000000423446 in timer_intr (arg=) at uinet_kern_timeout.c:844
  Question for MultiProcess architecture integration #5 0x00000000004a5ee2 in pthread_start_routine (arg=0x802c3d3e0) at uinet_host_interface.c:580
  ether_sprintf() doesn't work #6 0x00000008013a04f5 in ?? () from /lib/libthr.so.3
  API for accept4 #7 0x0000000000000000 in ?? ()
  Backtrace stopped: Cannot access memory at address 0x7fffffbfe000
  (gdb)

.. it looks like the timer thread hasn't been stopped but the VNET state has been freed.