xpmem stuff #1

jsquyres · 2018-03-09T22:47:32Z

You were pretty close, but it probably would have been pretty hard to find the use of FI_CHECK_PREFIX_DIR.

Also, the other two key points were:

Do not set CPPFLAGS (and friends) directly, but rather pass in the prefix/libdir to FI_CHECK_PACKAGE, and let it figure out the xpmem-specific CPPFLAGS (and friends).
Use the xpmem-specific CPPFLAGS (and friends) in prov/shm/Makefile.include

I also included some extra crispy bonus commits (which should also go upstream, even though they weren't technically part of fixing the xpmem stuff). Check out the commit messages of each of the commits.

Enjoy.

Explain what $2 is for. Signed-off-by: Jeff Squyres <jsquyres@cisco.com>

Otherwise, you can end up in a situation where a user specifies `--with-xpmem`, configure is unable to find xpmem support, but it continues on and ends successfully (with a $status of 0), leading the user to think that xpmem support will be included in the build. Signed-off-by: Jeff Squyres <jsquyres@cisco.com>

You'll end up squashing this in to your other commit(s) anyway, so I'll just describe here. * Call FI_PREFIX_DIR on the $enable_xpmem value (if it was a directory). This sets $xpmem_PREFIX and $xpmem_LIBDIR, which we use later. * Don't manually set CPPFLAGS and LDFLAGS -- let FI_CHECK_PACKAGE figure all that out and give us xpmem-specific CPPFLAGS, LDFLAGS, and LIBS. Then AC_SUBST those values and use them appropriately in the prov/shm/Makefile.include so that they are used when compiling only the prov/shm sources (when possible). * We do this (instead of loading CPPFLAGS/LDFLAGS/LIBS) so that we can try to limit where the xpmem flags need to be used. If you load CPPFLAGS (and friends), then those flags are *always* used for *all* compilations/links. Signed-off-by: Jeff Squyres <jsquyres@cisco.com>

aingerson · 2018-03-09T22:52:27Z

aingerson · 2018-03-09T22:56:31Z

You are the best. Thank you so much!!

I'm not entirely sure if it is fixes the issue our QA is seeing (as they get err_entry.err=-104 - a wrong negative value), but with error injection I could easily trigger a use-after-free with the root from this function (with err_entry.err=104, though, so I still don't know where the wrong error sign came from). In my error injection reproducer ofi_send_socket() fails sometimes, which then triggers free of cm_ctx without removing the fd and cm_ctx from polling. Next poll round will then access cm_ctx and trigger a use-after-free. client_send_connreq tx_cm_data ofi_send_socket -> fails goto err ... err: free(cm_ctx) ASAN reports READ of size 4 at 0x6120000106c8 thread T4 (rpc_poll-0) #0 0x7f77005e0f21 in process_cm_ctx prov/tcp/src/tcpx_conn_mgr.c:482 #1 0x7f77005e15ef in tcpx_conn_mgr_run prov/tcp/src/tcpx_conn_mgr.c:535 #2 0x7f77005fc429 in tcpx_eq_read prov/tcp/src/tcpx_eq.c:48 #3 0x4926dd in fi_eq_read /home/bschubert/local/rhel7/libfabric/include/rdma/fi_eq.h:352 0x6120000106c8 is located 8 bytes inside of 280-byte region [0x6120000106c0,0x6120000107d8) freed by thread T4 (rpc_poll-0) here: #0 0x7f77015915e7 in __interceptor_free #1 0x7f77005e083b in client_send_connreq prov/tcp/src/tcpx_conn_mgr.c:422 #2 0x7f77005e0f7e in process_cm_ctx prov/tcp/src/tcpx_conn_mgr.c:487 #3 0x7f77005e15ef in tcpx_conn_mgr_run prov/tcp/src/tcpx_conn_mgr.c:535 #4 0x7f77005fc429 in tcpx_eq_read prov/tcp/src/tcpx_eq.c:48 previously allocated by thread T5 (rpc_conn_mgr) here: #0 0x7f7701591b7e in __interceptor_calloc #1 0x7f77005edb5c in tcpx_ep_connect prov/tcp/src/tcpx_ep.c:103 #2 0x478b2f in fi_connect /home/bschubert/local/rhel7/libfabric/include/rdma/fi_cm.h:98 Signed-off-by: Bernd Schubert <bschubert@ddn.com>

Problem reported by Address Sanitizer: ================================================================= ==25220==ERROR: AddressSanitizer: heap-use-after-free on address 0x6270000072e0 at pc 0x00010b926a3c bp 0x700001bd1c30 sp 0x700001bd1c28 READ of size 4 at 0x6270000072e0 thread T4 #0 0x10b926a3b in sock_conn_listener_thread (libfabric.1.dylib:x86_64+0xdca3b) #1 0x7fff7e2d5660 in _pthread_body (libsystem_pthread.dylib:x86_64+0x3660) #2 0x7fff7e2d550c in _pthread_start (libsystem_pthread.dylib:x86_64+0x350c) #3 0x7fff7e2d4bf8 in thread_start (libsystem_pthread.dylib:x86_64+0x2bf8) 0x6270000072e0 is located 480 bytes inside of 12944-byte region [0x627000007100,0x62700000a390) freed by thread T0 here: #0 0x10baf1a9d in wrap_free (libclang_rt.asan_osx_dynamic.dylib:x86_64+0x56a9d) #1 0x10b9016bf in sock_ep_close (libfabric.1.dylib:x86_64+0xb76bf) #2 0x10b7f4a8f in fi_close fabric.h:593 #3 0x10b7f4209 in main shared_ctx.c:649 #4 0x7fff7dfbd014 in start (libdyld.dylib:x86_64+0x1014) previously allocated by thread T0 here: #0 0x10baf1e27 in wrap_calloc (libclang_rt.asan_osx_dynamic.dylib:x86_64+0x56e27) #1 0x10b906df4 in sock_alloc_endpoint (libfabric.1.dylib:x86_64+0xbcdf4) #2 0x10b8f7fdb in sock_msg_ep (libfabric.1.dylib:x86_64+0xadfdb) #3 0x10b7f7c93 in fi_endpoint fi_endpoint.h:164 #4 0x10b7f5e40 in server_connect shared_ctx.c:471 #5 0x10b7f49ba in run shared_ctx.c:573 #6 0x10b7f411b in main shared_ctx.c:647 #7 0x7fff7dfbd014 in start (libdyld.dylib:x86_64+0x1014) Thread T4 created by T0 here: #0 0x10bae999d in wrap_pthread_create (libclang_rt.asan_osx_dynamic.dylib:x86_64+0x4e99d) #1 0x10b925f9b in sock_conn_start_listener_thread (libfabric.1.dylib:x86_64+0xdbf9b) #2 0x10b8e7eb2 in sock_domain (libfabric.1.dylib:x86_64+0x9deb2) #3 0x10b7f87d3 in fi_domain fi_domain.h:306 #4 0x10b7f5c9f in server_connect shared_ctx.c:460 #5 0x10b7f49ba in run shared_ctx.c:573 #6 0x10b7f411b in main shared_ctx.c:647 #7 0x7fff7dfbd014 in start (libdyld.dylib:x86_64+0x1014) The issue shows up more frequently on OS X, which emulates epoll. However, I believe the problem could occur on any platform. In sock_ep_close, we remove the socket from the epoll fd, then free the endpoint. However, if the listener thread has received an event on the socket, but has not yet started processing it, then a race can occur. The listener thread could have returned from ofi_epoll_wait, but suspended trying to acquire the signal_lock. The signal_lock is acquired from sock_ep_close, where ofi_epoll_del is called, then released. The endpoint is then freed. The listener thread can now acquire the signal_lock, where it will attempt to access the freed endpoint data. To avoid the race, we add a change boolean to the listener. That boolean is only changed while holding the signal_lock. When a socket is removed from the epollfd, we mark the listener state as 'changed'. The listener thread checks the changed state prior to processing any events. If set, it clears the state, and calls ofi_epoll_wait again to get a new set of events to process. Note that this works for epoll set to level-triggered (poll semantics). Sockets that reported events will report those same events when wait is called a second time. Sockets which were removed from the epoll set would have their events removed, as they are no longer being monitored. This fix is applied both to the listener thread and cm thread. Signed-off-by: Sean Hefty <sean.hefty@intel.com>

I'm not entirely sure if it is fixes the issue our QA is seeing (as they get err_entry.err=-104 - a wrong negative value), but with error injection I could easily trigger a use-after-free with the root from this function (with err_entry.err=104, though, so I still don't know where the wrong error sign came from). In my error injection reproducer ofi_send_socket() fails sometimes, which then triggers free of cm_ctx without removing the fd and cm_ctx from polling. Next poll round will then access cm_ctx and trigger a use-after-free. client_send_connreq tx_cm_data ofi_send_socket -> fails goto err ... err: free(cm_ctx) ASAN reports READ of size 4 at 0x6120000106c8 thread T4 (rpc_poll-0) #0 0x7f77005e0f21 in process_cm_ctx prov/tcp/src/tcpx_conn_mgr.c:482 #1 0x7f77005e15ef in tcpx_conn_mgr_run prov/tcp/src/tcpx_conn_mgr.c:535 #2 0x7f77005fc429 in tcpx_eq_read prov/tcp/src/tcpx_eq.c:48 #3 0x4926dd in fi_eq_read /home/bschubert/local/rhel7/libfabric/include/rdma/fi_eq.h:352 0x6120000106c8 is located 8 bytes inside of 280-byte region [0x6120000106c0,0x6120000107d8) freed by thread T4 (rpc_poll-0) here: #0 0x7f77015915e7 in __interceptor_free #1 0x7f77005e083b in client_send_connreq prov/tcp/src/tcpx_conn_mgr.c:422 #2 0x7f77005e0f7e in process_cm_ctx prov/tcp/src/tcpx_conn_mgr.c:487 #3 0x7f77005e15ef in tcpx_conn_mgr_run prov/tcp/src/tcpx_conn_mgr.c:535 #4 0x7f77005fc429 in tcpx_eq_read prov/tcp/src/tcpx_eq.c:48 previously allocated by thread T5 (rpc_conn_mgr) here: #0 0x7f7701591b7e in __interceptor_calloc #1 0x7f77005edb5c in tcpx_ep_connect prov/tcp/src/tcpx_ep.c:103 #2 0x478b2f in fi_connect /home/bschubert/local/rhel7/libfabric/include/rdma/fi_cm.h:98 Signed-off-by: Bernd Schubert <bschubert@ddn.com>

ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fff4c61e7e0 at pc 0x14f2cb7ae0b9 bp 0x7fff4c61e650 sp 0x7fff4c61ddd8 WRITE of size 17 at 0x7fff4c61e7e0 thread T0 #0 0x14f2cb7ae0b8 (/lib64/libasan.so.5+0xb40b8) #1 0x14f2cb7aedd2 in vsscanf (/lib64/libasan.so.5+0xb4dd2) #2 0x14f2cb7aeede in __interceptor_sscanf (/lib64/libasan.so.5+0xb4ede) #3 0x14f2cb230766 in ofi_addr_format src/common.c:401 #4 0x14f2cb233238 in ofi_str_toaddr src/common.c:780 #5 0x14f2cb314332 in vrb_handle_ib_ud_addr prov/verbs/src/verbs_info.c:1670 #6 0x14f2cb314332 in vrb_get_match_infos prov/verbs/src/verbs_info.c:1787 #7 0x14f2cb314332 in vrb_getinfo prov/verbs/src/verbs_info.c:1841 #8 0x14f2cb21fc28 in fi_getinfo_ src/fabric.c:1010 #9 0x14f2cb25fcc0 in ofi_get_core_info prov/util/src/util_attr.c:298 #10 0x14f2cb269b20 in ofix_getinfo prov/util/src/util_attr.c:321 #11 0x14f2cb3e29fd in rxd_getinfo prov/rxd/src/rxd_init.c:122 #12 0x14f2cb21fc28 in fi_getinfo_ src/fabric.c:1010 #13 0x407150 in ft_getinfo common/shared.c:794 #14 0x414917 in ft_init_fabric common/shared.c:1042 #15 0x402f40 in run functional/bw.c:155 #16 0x402f40 in main functional/bw.c:252 #17 0x14f2ca1b28e2 in __libc_start_main (/lib64/libc.so.6+0x238e2) #18 0x401d1d in _start (/root/libfabric/fabtests/functional/fi_bw+0x401d1d) Address 0x7fff4c61e7e0 is located in stack of thread T0 at offset 48 in frame #0 0x14f2cb2306f3 in ofi_addr_format src/common.c:397 This frame has 1 object(s): [32, 48) 'fmt' <== Memory access at offset 48 overflows this variable HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext (longjmp and C++ exceptions *are* supported) SUMMARY: AddressSanitizer: stack-buffer-overflow (/lib64/libasan.so.5+0xb40b8) Shadow bytes around the buggy address: 0x1000698bbca0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1000698bbcb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1000698bbcc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1000698bbcd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1000698bbce0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x1000698bbcf0: 00 00 00 00 00 00 f1 f1 f1 f1 00 00[f2]f2 f3 f3 0x1000698bbd00: f3 f3 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 0x1000698bbd10: f1 f1 00 f2 f2 f2 f2 f2 f2 f2 00 f2 f2 f2 f2 f2 0x1000698bbd20: f2 f2 00 f2 f2 f2 f2 f2 f2 f2 00 f2 f2 f2 f2 f2 0x1000698bbd30: f2 f2 00 00 00 00 00 06 f2 f2 f2 f2 f2 f2 00 00 0x1000698bbd40: 00 00 00 06 f2 f2 f2 f2 f2 f2 00 00 00 00 00 00 Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Fixes: 5d31276 ("common: Redo address string conversions") Signed-off-by: Honggang Li <honli@redhat.com>

Utility providers have to call fi_getinfo again to get core providers resulting in deceptive and confusing log lines where a core provider might return FI_ENODATA for a utility provider but FI_SUCCESS for the app. Extra log levels were added that say Begin/End ofi_get_core_info to make this clearer but these debug-only (not info) logs can get lost among the hundreds of lines of output. To make it easier to distinguish between log lines with and without a core provider, specifically during fi_getinfo, add a log_prefix to the log output which clarifies that the log line was outputed as part of the layered fi_getinfo call For example, the following log line sees changes as such: libfabric:53685:1643663041:verbs:fabric:vrb_get_matching_info():1514<info> checking domain: #1 mlx5_0 libfabric:53685:1643663041:ofi_rxm:verbs:fabric:vrb_get_matching_info():1514<info> checking domain: #1 mlx5_0 Signed-off-by: aingerson <alexia.ingerson@intel.com>

If a posted receive matches with a saved receive, we may need to increment the rx counter. Set the rx counter increment callback to match that of the posted receive. This fixes an assert in xnet_cntr_inc() accessing a NULL cntr_inc function pointer. Program received signal SIGABRT, Aborted. 0x0000155552d4d37f in raise () from /lib64/libc.so.6 #0 0x0000155552d4d37f in raise () from /lib64/libc.so.6 #1 0x0000155552d37db5 in abort () from /lib64/libc.so.6 #2 0x0000155552d37c89 in __assert_fail_base.cold.0 () from /lib64/libc.so.6 #3 0x0000155552d45a76 in __assert_fail () from /lib64/libc.so.6 #4 0x00001555522967f9 in xnet_cntr_inc (ep=0x6e4c70, xfer_entry=0x6f7a30) at prov/tcp/src/xnet_cq.c:347 #5 0x0000155552296836 in xnet_report_cntr_success (ep=0x6e4c70, cq=0x6ca930, xfer_entry=0x6f7a30) at prov/tcp/src/xnet_cq.c:354 #6 0x000015555229970d in xnet_complete_saved (saved_entry=0x6f7a30) at prov/tcp/src/xnet_progress.c:153 #7 0x0000155552299961 in xnet_recv_saved (saved_entry=0x6f7a30, rx_entry=0x6f7840) at prov/tcp/src/xnet_progress.c:188 #8 0x00001555522946f8 in xnet_srx_tag (srx=0x6dd1c0, recv_entry=0x6f7840) at prov/tcp/src/xnet_srx.c:445 #9 0x0000155552294bb1 in xnet_srx_trecv (ep_fid=0x6dd1c0, buf=0x6990c4, len=4, desc=0x0, src_addr=0, tag=21474836494, ignore=3458764513820540928, context=0x7ffffffeb180) at prov/tcp/src/xnet_srx.c:558 #10 0x000015555228f60e in fi_trecv (ep=0x6dd1c0, buf=0x6990c4, len=4, desc=0x0, src_addr=0, tag=21474836494, ignore=3458764513820540928, context=0x7ffffffeb180) at ./include/rdma/fi_tagged.h:91 #11 0x00001555522900a7 in xnet_rdm_trecv (ep_fid=0x6d9fe0, buf=0x6990c4, len=4, desc=0x0, src_addr=0, tag=21474836494, ignore=3458764513820540928, context=0x7ffffffeb180) at prov/tcp/src/xnet_rdm.c:212 Signed-off-by: Sean Hefty <sean.hefty@intel.com>

jsquyres added 3 commits March 9, 2018 14:31

fi_provider.m4: add missing comment

7f3b53f

Explain what $2 is for. Signed-off-by: Jeff Squyres <jsquyres@cisco.com>

aingerson merged commit bf12056 into aingerson:debug Mar 9, 2018

jsquyres deleted the pr/good-stuff-for-alexia branch March 9, 2018 23:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

xpmem stuff #1

xpmem stuff #1

jsquyres commented Mar 9, 2018

aingerson commented Mar 9, 2018

aingerson commented Mar 9, 2018

xpmem stuff #1

xpmem stuff #1

Conversation

jsquyres commented Mar 9, 2018

aingerson commented Mar 9, 2018

aingerson commented Mar 9, 2018