tests sometimes end with SIGSEGV 11 - invalid write of size 8 or invalid read of size 4

[drahnr]

valgrind trace

valgrind /media/pineapple/projects/spearow/coaster-nn/target/debug/deps/coaster_nn-b3366491e579a60f                                    17:39:20   1 
==13182== Memcheck, a memory error detector
==13182== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==13182== Using Valgrind-3.12.0 and LibVEX; rerun with -h for copyright info
==13182== Command: /media/pineapple/projects/spearow/coaster-nn/target/debug/deps/coaster_nn-b3366491e579a60f
==13182== 

running 86 tests
==13182== Warning: noted but unhandled ioctl 0x30000001 with no size/direction hints.
==13182==    This could cause spurious value errors to appear.
==13182==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==13182== Warning: noted but unhandled ioctl 0x27 with no size/direction hints.
==13182==    This could cause spurious value errors to appear.
==13182==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==13182== Warning: noted but unhandled ioctl 0x7ff with no size/direction hints.
==13182==    This could cause spurious value errors to appear.
==13182==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==13182== Warning: noted but unhandled ioctl 0x25 with no size/direction hints.
==13182==    This could cause spurious value errors to appear.
==13182==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==13182== Warning: noted but unhandled ioctl 0x17 with no size/direction hints.
==13182==    This could cause spurious value errors to appear.
==13182==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==13182== Warning: set address range perms: large range [0x200000000, 0x700000000) (noaccess)
==13182== Warning: set address range perms: large range [0x900000000, 0xb00000000) (noaccess)
==13182== Warning: noted but unhandled ioctl 0x19 with no size/direction hints.
==13182==    This could cause spurious value errors to appear.
==13182==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==13182== Warning: noted but unhandled ioctl 0x21 with no size/direction hints.
==13182==    This could cause spurious value errors to appear.
==13182==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==13182== Warning: noted but unhandled ioctl 0x1b with no size/direction hints.
==13182==    This could cause spurious value errors to appear.
==13182==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==13182== Warning: noted but unhandled ioctl 0x42 with no size/direction hints.
==13182==    This could cause spurious value errors to appear.
==13182==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==13182== Warning: noted but unhandled ioctl 0x22 with no size/direction hints.
==13182==    This could cause spurious value errors to appear.
==13182==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
test tests::activation::cuda::relu_grad_f32 ... ok
==13182== Thread 4 tests::activation::cuda::relu_grad_f32:
==13182== Invalid read of size 4
==13182==    at 0xF424DE0: pthread_mutex_lock (in /usr/lib64/libpthread-2.24.so)
==13182==    by 0x24331D: je_malloc_mutex_lock (mutex.h:101)
==13182==    by 0x24331D: je_tcache_bin_flush_small (tcache.c:119)
==13182==    by 0x244A42: tcache_destroy (tcache.c:371)
==13182==    by 0x244D81: je_tcache_cleanup (tcache.c:410)
==13182==    by 0x2456B4: je_tsd_cleanup (tsd.c:82)
==13182==    by 0xF4214E8: __nptl_deallocate_tsd.part.4 (in /usr/lib64/libpthread-2.24.so)
==13182==    by 0xF4228E7: start_thread (in /usr/lib64/libpthread-2.24.so)
==13182==    by 0xF957E7E: clone (in /usr/lib64/libc-2.24.so)
==13182==  Address 0xb88 is not stack'd, malloc'd or (recently) free'd
==13182== 
==13182== Can't extend stack to 0x4030138 during signal delivery for thread 4:
==13182==   no stack segment
==13182== 
==13182== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==13182==  Access not within mapped region at address 0x4030138
==13182==    at 0xF424DE0: pthread_mutex_lock (in /usr/lib64/libpthread-2.24.so)
==13182==    by 0x24331D: je_malloc_mutex_lock (mutex.h:101)
==13182==    by 0x24331D: je_tcache_bin_flush_small (tcache.c:119)
==13182==    by 0x244A42: tcache_destroy (tcache.c:371)
==13182==    by 0x244D81: je_tcache_cleanup (tcache.c:410)
==13182==    by 0x2456B4: je_tsd_cleanup (tsd.c:82)
==13182==    by 0xF4214E8: __nptl_deallocate_tsd.part.4 (in /usr/lib64/libpthread-2.24.so)
==13182==    by 0xF4228E7: start_thread (in /usr/lib64/libpthread-2.24.so)
==13182==    by 0xF957E7E: clone (in /usr/lib64/libc-2.24.so)
==13182==  If you believe this happened as a result of a stack
==13182==  overflow in your program's main thread (unlikely but
==13182==  possible), you can try to increase the size of the
==13182==  main thread stack using the --main-stacksize= flag.
==13182==  The main thread stack size used in this run was 8388608.
==13182== Invalid write of size 8
==13182==    at 0x4A286C0: _vgnU_freeres (vg_preloaded.c:59)
==13182==  Address 0x4030ff8 is on thread 4's stack
==13182== 
==13182== 
==13182== Process terminating with default action of signal 11 (SIGSEGV)
==13182==  Access not within mapped region at address 0x4030FF8
==13182==    at 0x4A286C0: _vgnU_freeres (vg_preloaded.c:59)
==13182==  If you believe this happened as a result of a stack
==13182==  overflow in your program's main thread (unlikely but
==13182==  possible), you can try to increase the size of the
==13182==  main thread stack using the --main-stacksize= flag.
==13182== 
==13182== HEAP SUMMARY:
==13182==     in use at exit: 699,926,716 bytes in 747,368 blocks
==13182==   total heap usage: 1,758,065 allocs, 1,010,697 frees, 1,492,755,407 bytes allocated
==13182== 
==13182== LEAK SUMMARY:
==13182==    definitely lost: 0 bytes in 0 blocks
==13182==    indirectly lost: 0 bytes in 0 blocks
==13182==      possibly lost: 6,082,204 bytes in 22,711 blocks
==13182==    still reachable: 693,844,512 bytes in 724,657 blocks
==13182==         suppressed: 0 bytes in 0 blocks
==13182== Rerun with --leak-check=full to see details of leaked memory
==13182== 
==13182== For counts of detected and suppressed errors, rerun with: -v
==13182== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)
[1]    13182 segmentation fault (core dumped)  valgrind

[drahnr]

Could be related to: rust-lang/rust-bindgen#726 - probably not

[drahnr]

Or maybe: rust-lang/rust-bindgen#550 since

test generated::bindgen_test_layout_short3 ... ok
test generated::bindgen_test_layout_max_align_t ... FAILED
test generated::bindgen_test_layout_surfaceReference ... ok

[Anton-4]

I ran the tests 10 times without failure, valgrind also doesn't report any errors.
So I guess this can be closed?

[drahnr]

What GPU / driver / cudnn do you have?

It could be that this is obsolete, since I regenerated the cudnn bindings of cudnn in rcudnn a few weeks ago.

[Anton-4]

• 780ti
• cudnn 6.0.21
• cuda 8 rev2

[drahnr]

To be verified with

• gtx460 and gtx1050ti
• cudnn 7.0.3.11
• cuda 8.0.61