motr kut test failure

[yanqingfu]

unit test passed, but kut test has some problems
reboot the vm still face the same issue

Time: 2333.70 sec, Mem: 64 GiB, Leaked: 28 MiB, Asserts: 49994761
Unit tests status: SUCCESS
END Iteration: 1 out of 1

utime 737.141124 stime 227.491366 maxrss 1452776 nvcsw 2813412 nivcsw 12727
minflt 2769436 majflt 628 inblock 88528 oublock 42068544
rchar 37141238 wchar 33796789 syscr 13581 syscw 3824
read_bytes 45490176 write_bytes 21555871744 cancelled_write_bytes 641359872
[root@ssc-vm-g3-rhev4-1122 cortx-motr]# scripts/m0 run-kut
----- run_kut -----
Running kernel UT (this may take a while) ...
Nov 23 02:01:09 ssc-vm-g3-rhev4-1122 kernel: Motr m0lnet deregistered
Nov 23 02:02:22 ssc-vm-g3-rhev4-1122 kernel: Motr trace immediate mask: 0x0
Nov 23 02:02:22 ssc-vm-g3-rhev4-1122 kernel: motr: trace level: fatal+
Nov 23 02:02:22 ssc-vm-g3-rhev4-1122 kernel: motr: trace print context: short
Nov 23 02:02:22 ssc-vm-g3-rhev4-1122 kernel: motr: trace header address: 0xffffadd281602000
Nov 23 02:02:22 ssc-vm-g3-rhev4-1122 kernel: motr: trace buffer address: 0xffffadd281612000
Nov 23 02:02:22 ssc-vm-g3-rhev4-1122 kernel: Motr m0lnet registered with minor 57
Nov 23 02:02:22 ssc-vm-g3-rhev4-1122 kernel: motr: max CPUs for parity calcs: 2
Nov 23 02:02:22 ssc-vm-g3-rhev4-1122 kernel: Motr Kernel Unit Test
Nov 23 02:02:22 ssc-vm-g3-rhev4-1122 kernel: klibm0-ut
Nov 23 02:02:22 ssc-vm-g3-rhev4-1122 kernel: . bitmap 0.00 sec 80 B
Nov 23 02:02:22 ssc-vm-g3-rhev4-1122 kernel: . bitmap-onwire 0.00 sec 48 B
Nov 23 02:02:22 ssc-vm-g3-rhev4-1122 kernel: . memory 0.00 sec 136 B
Nov 23 02:02:22 ssc-vm-g3-rhev4-1122 kernel: . bob 0.00 sec 0 B
Nov 23 02:02:22 ssc-vm-g3-rhev4-1122 kernel: . buf 0.00 sec 448 B
motr[04974]: e370 ERROR [lib/uuid.c:68:parse_hex] <! rc=-22
motr[04974]: e370 ERROR [lib/uuid.c:68:parse_hex] <! rc=-22
motr[04974]: e3e0 ERROR [lib/uuid.c:96:m0_uuid_parse] <! rc=-22
motr[04974]: e440 ERROR [lib/uuid.c:145:m0_node_uuid_init] Unable to parse node UUID string
subsystem libm0 init failed: rc = -22
motr[04974]: e450 FATAL [lib/assert.c:50:m0_panic] panic: (rc == 0) at main() (net/lnet/ut/lut_main.c:345) [git: 2.0.0-527-28-ga0edefb] /var/motr/m0kut-4946/m0trace.4974
Motr panic: (rc == 0) at main() net/lnet/ut/lut_main.c:345 (errno: 0) (last failed: none) [git: 2.0.0-527-28-ga0edefb] pid: 4974 /var/motr/m0kut-4946/m0trace.4974
/var/cortx/cortx-motr/motr/.libs/libmotr.so.2(m0_arch_backtrace+0x20)[0x7f5a5353c950]
/var/cortx/cortx-motr/motr/.libs/libmotr.so.2(m0_arch_panic+0xe6)[0x7f5a5353cb06]
/var/cortx/cortx-motr/motr/.libs/libmotr.so.2(+0x380b34)[0x7f5a5352ab34]
/var/cortx/cortx-motr/net/lnet/ut/.libs/lt-m0kut-helper(_start+0x0)[0x40134b]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f5a468ba555]
/var/cortx/cortx-motr/net/lnet/ut/.libs/lt-m0kut-helper[0x401374]
/var/cortx/cortx-motr/ut/m0kut: line 137: 4974 Aborted (core dumped) $top_srcdir/net/lnet/ut/m0kut-helper
[root@ssc-vm-g3-rhev4-1122 cortx-motr]# Nov 23 02:02:23 ssc-vm-g3-rhev4-1122 kernel: . chan 0.85 sec 241 KiB
Nov 23 02:02:23 ssc-vm-g3-rhev4-1122 kernel: . cookie 0.10 sec 16 B
Nov 23 02:02:23 ssc-vm-g3-rhev4-1122 kernel: . finject 0.00 sec 0 B
Nov 23 02:02:23 ssc-vm-g3-rhev4-1122 kernel: . hash 0.00 sec 1 KiB
Nov 23 02:02:23 ssc-vm-g3-rhev4-1122 kernel: . list 0.00 sec 0 B
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: . locality [Nikita] 0.49 sec 3 MiB
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: ------------[ cut here ]------------
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: WARNING: CPU: 0 PID: 4965 at mm/page_alloc.c:3220 __alloc_pages_slowpath+0x6f/0x729
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: . loc-chores [Nikita]
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: Modules linked in:
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: m0ut(POE) loop ksocklnd(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache sunrpc nfit libnvdimm iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr virtio_rng virtio_balloon sg parport_pc i2c_piix4 parport ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi virtio_net virtio_scsi net_failover failover virtio_console qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul ttm crct10dif_common crc32c_intel ata_piix drm serio_raw virtio_pci virtio_ring floppy libata virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: m0tr]
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: CPU: 0 PID: 4965 Comm: m0_loc_thread Kdump: loaded Tainted: P OE ------------ 3.10.0-1160.el7.x86_64 #1
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: Hardware name: Red Hat RHEV Hypervisor, BIOS 1.11.0-2.el7 04/01/2014
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: Call Trace:
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: [] dump_stack+0x19/0x1b
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: [] __warn+0xd8/0x100
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: [] warn_slowpath_null+0x1d/0x20
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: [] __alloc_pages_slowpath+0x6f/0x729
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: [] ? update_rq_clock.part.77+0x4c/0x150
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: [] ? dequeue_entity+0x11c/0x5c0
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: [] __alloc_pages_nodemask+0x436/0x450
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: [] alloc_pages_current+0x98/0x110
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: [] kmalloc_order+0x18/0x40
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: [] kmalloc_order_trace+0x26/0xa0
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: [] __kmalloc+0x211/0x230
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: [] m0_arch_alloc+0x13/0x20 [m0ut]
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: [] m0_alloc+0x5b/0x140 [m0ut]
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: [] ? m0_locality_get+0x47/0x80 [m0ut]
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: [] chore_add_cb+0xa5/0x160 [m0ut]
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: [] m0_sm_asts_run+0xeb/0x1a0 [m0ut]
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: [] ? add+0xd3/0x110 [m0ut]
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: [] loc_handler_thread+0x137/0x4a0 [m0ut]
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: [] ? m0_arch_cookie_global_fini+0x10/0x10 [m0ut]
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: [] m0_thread_trampoline+0x68/0xb0 [m0ut]
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: [] kthread_trampoline+0x38/0xd0 [m0ut]
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: [] kthread+0xd1/0xe0
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: [] ? insert_kthread_work+0x40/0x40
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: [] ret_from_fork_nospec_begin+0x21/0x21
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: [] ? insert_kthread_work+0x40/0x40
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: ---[ end trace 527a037614add140 ]---
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: 0.08 sec 95 KiB
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: . lockers 0.00 sec 0 B
Nov 23 02:02:24 ssc-vm-g3-rhev4-1122 kernel: . tlist 0.00 sec 0 B
Nov 23 02:02:25 ssc-vm-g3-rhev4-1122 kernel: . mutex 1.27 sec 77 KiB
Nov 23 02:02:25 ssc-vm-g3-rhev4-1122 kernel: . queue 0.00 sec 0 B
Nov 23 02:02:25 ssc-vm-g3-rhev4-1122 kernel: . refs 0.00 sec 16 B
Nov 23 02:02:26 ssc-vm-g3-rhev4-1122 kernel: . rwlock 1.00 sec 4 MiB
Nov 23 02:02:26 ssc-vm-g3-rhev4-1122 kernel: . thread 0.16 sec 45 KiB
Nov 23 02:02:27 ssc-vm-g3-rhev4-1122 kernel: . time 0.83 sec 49 KiB
Nov 23 02:02:27 ssc-vm-g3-rhev4-1122 kernel: . trace 0.19 sec 0 B
Nov 23 02:02:28 ssc-vm-g3-rhev4-1122 kernel: . uuid 0.01 sec 0 B
Nov 23 02:02:29 ssc-vm-g3-rhev4-1122 kernel: . varr 1.78 sec 5 MiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . vec 0.64 sec 15 MiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . zerovec 0.02 sec 384 B
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . fold [Nikita] 0.00 sec 0 B
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . [ time: 7.53 sec, mem: 29 MiB, leaked: 9 MiB ]
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: addb2-base
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . init-fini 0.00 sec 16 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . push-pop 0.00 sec 20 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . push0-pop 0.00 sec 20 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . push5-pop 0.00 sec 20 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . push^N-pop^N 0.00 sec 20 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . add 0.00 sec 20 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . add-var 0.00 sec 20 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . full 0.00 sec 20 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . stop-idle 0.00 sec 65 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . sensor-depth 0.02 sec 20 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . [ time: 0.05 sec, mem: 241 KiB, leaked: 0 B ]
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: addb2-consumer
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . empty 0.00 sec 16 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . empty-push-pop 0.00 sec 20 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . data 0.00 sec 20 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . data-label 0.00 sec 20 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . sensor 0.00 sec 20 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . sensor-N 0.00 sec 20 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . id-philter 0.00 sec 20 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . global-philter 0.00 sec 0 B
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . [ time: 0.00 sec, mem: 136 KiB, leaked: 0 B ]
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: addb2-histogram
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . init-fini 0.00 sec 0 B
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . history-bucket 0.00 sec 0 B
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . [ time: 0.00 sec, mem: 0 B, leaked: 0 B ]
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: addb2-sys
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . init-fini [Nikita] 0.00 sec 512 B
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . mach-1 [Nikita] 0.00 sec 16 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . mach-toomany [Nikita] 0.00 sec 16 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . mach-cache [Nikita] 0.00 sec 16 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . mach-cache-N [Nikita] 0.00 sec 2 MiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . noqueue-add [Nikita] 0.01 sec 82 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . queue-add [Nikita] 0.00 sec 82 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . sm-add [Nikita] 0.00 sec 28 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . [ time: 0.03 sec, mem: 2 MiB, leaked: 4 KiB ]
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: di-ut
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . di-init 0.00 sec 66 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . di-cksum-test 0.00 sec 0 B
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . di-ref-tag-test 0.00 sec 0 B
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . di-test 0.00 sec 0 B
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . di-none-test 0.00 sec 0 B
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . di-fini 0.00 sec 0 B
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . [ time: 0.01 sec, mem: 66 KiB, leaked: 0 B ]
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: file-io-ut
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . basic_data_structures_test 0.00 sec 15 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . helper_routines_test 0.00 sec 0 B
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . parity_group_ops_test 0.00 sec 5 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . nw_xfer_ops_test 0.00 sec 38 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . target_ioreq_ops_test 0.00 sec 22 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . dgmode_readio_test 0.00 sec 57 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . fsync_test 0.00 sec 38 KiB
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . [ time: 0.08 sec, mem: 1 MiB, leaked: 140 KiB ]
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: be-ut
Nov 23 02:02:30 ssc-vm-g3-rhev4-1122 kernel: . list 0.00 sec 52 KiB
Nov 23 02:02:37 ssc-vm-g3-rhev4-1122 kernel: . btree-create_destroy 6.35 sec 3 MiB
Nov 23 02:02:43 ssc-vm-g3-rhev4-1122 kernel: . btree-create_truncate 6.11 sec 2 MiB
Nov 23 02:02:43 ssc-vm-g3-rhev4-1122 kernel: . seg_dict 0.01 sec 1 KiB
Nov 23 02:02:43 ssc-vm-g3-rhev4-1122 kernel: . emap 0.00 sec 298 KiB
Nov 23 02:02:43 ssc-vm-g3-rhev4-1122 kernel: . [ time: 12.49 sec, mem: 6 MiB, leaked: 10 KiB ]
Nov 23 02:02:43 ssc-vm-g3-rhev4-1122 kernel: buffer_pool_ut
Nov 23 02:02:43 ssc-vm-g3-rhev4-1122 kernel: . buffer_pool_init 0.00 sec 25 KiB
Nov 23 02:02:43 ssc-vm-g3-rhev4-1122 kernel: . buffer_pool_get_put 0.00 sec 0 B
Nov 23 02:02:43 ssc-vm-g3-rhev4-1122 kernel: . buffer_pool_get_put_colour 0.00 sec 0 B
Nov 23 02:02:43 ssc-vm-g3-rhev4-1122 kernel: . buffer_pool_grow 0.00 sec 2 KiB
Nov 23 02:02:43 ssc-vm-g3-rhev4-1122 kernel: . buffer_pool_prune 0.00 sec 0 B
Nov 23 02:02:43 ssc-vm-g3-rhev4-1122 kernel: . buffer_pool_get_put_multiple 0.01 sec 2 KiB
Nov 23 02:02:43 ssc-vm-g3-rhev4-1122 kernel: . buffer_pool_fini 0.00 sec 0 B
Nov 23 02:02:43 ssc-vm-g3-rhev4-1122 kernel: . [ time: 0.01 sec, mem: 30 KiB, leaked: 0 B ]
Nov 23 02:02:43 ssc-vm-g3-rhev4-1122 kernel: bulk-client-ut
Nov 23 02:02:59 ssc-vm-g3-rhev4-1122 kernel: . bulkclient_test 16.02 sec 35 KiB
Nov 23 02:02:59 ssc-vm-g3-rhev4-1122 kernel: . [ time: 16.02 sec, mem: 35 KiB, leaked: 16 KiB ]
Nov 23 02:02:59 ssc-vm-g3-rhev4-1122 kernel: net-bulk-if
Nov 23 02:02:59 ssc-vm-g3-rhev4-1122 kernel: . net_bulk_if 0.04 sec 50 KiB
Nov 23 02:02:59 ssc-vm-g3-rhev4-1122 kernel: . [ time: 0.05 sec, mem: 50 KiB, leaked: 0 B ]
Nov 23 02:02:59 ssc-vm-g3-rhev4-1122 kernel: net-bulk-mem
Nov 23 02:02:59 ssc-vm-g3-rhev4-1122 kernel: . net_bulk_mem_buf_copy_test 0.00 sec 1 KiB
Nov 23 02:02:59 ssc-vm-g3-rhev4-1122 kernel: . net_bulk_mem_tm_test 0.00 sec 384 B
Nov 23 02:02:59 ssc-vm-g3-rhev4-1122 kernel: . net_bulk_mem_ep 0.00 sec 768 B
Nov 23 02:02:59 ssc-vm-g3-rhev4-1122 kernel: . net_bulk_mem_failure_tests 0.00 sec 4 KiB
Nov 23 02:03:06 ssc-vm-g3-rhev4-1122 kernel: . net_bulk_mem_ping_tests 7.06 sec 214 KiB
Nov 23 02:03:06 ssc-vm-g3-rhev4-1122 kernel: . [ time: 7.08 sec, mem: 220 KiB, leaked: 8 KiB ]
Nov 23 02:03:06 ssc-vm-g3-rhev4-1122 kernel: net-lnet
Nov 23 02:03:06 ssc-vm-g3-rhev4-1122 kernel: . net_lnet_buf_shape (K) 0.00 sec 272 B
Nov 23 02:03:06 ssc-vm-g3-rhev4-1122 kernel: . net_lnet_buf_reg (K) 0.00 sec 1 MiB
Nov 23 02:03:06 ssc-vm-g3-rhev4-1122 kernel: . net_lnet_ep_addr (K) 0.00 sec 32 B
Nov 23 02:03:06 ssc-vm-g3-rhev4-1122 kernel: . net_lnet_enc_dec (K) 0.00 sec 0 B
Nov 23 02:03:08 ssc-vm-g3-rhev4-1122 kernel: . net_lnet_msg (K) 2.02 sec 18 KiB
Nov 23 02:03:10 ssc-vm-g3-rhev4-1122 kernel: . net_lnet_bulk (K) 2.01 sec 9 KiB

Message from syslogd@ssc-vm-g3-rhev4-1122 at Nov 23 02:03:15 ...
kernel:Motr panic: Unit-test assertion failed: ok at ktest_dev() /var/cortx/cortx-motr/net/lnet/ut/linux_kernel/klnet_ut.c:2179 (last failed: none) [git: 2.0.0-527-28-ga0edefb]
Nov 23 02:03:15 ssc-vm-g3-rhev4-1122 kernel: . net_lnet_device motr[04973]: fc10 FATAL [lib/assert.c:50:m0_panic] panic: Unit-test assertion failed: ok at ktest_dev() (/var/cortx/cortx-motr/net/lnet/ut/linux_kernel/klnet_ut.c:2179) [git: 2.0.0-527-28-ga0edefb]
Nov 23 02:03:15 ssc-vm-g3-rhev4-1122 kernel: Motr panic: Unit-test assertion failed: ok at ktest_dev() /var/cortx/cortx-motr/net/lnet/ut/linux_kernel/klnet_ut.c:2179 (last failed: none) [git: 2.0.0-527-28-ga0edefb]
Nov 23 02:03:15 ssc-vm-g3-rhev4-1122 kernel: CPU: 0 PID: 4973 Comm: m0kut Kdump: loaded Tainted: P W OE ------------ 3.10.0-1160.el7.x86_64 #1
Nov 23 02:03:15 ssc-vm-g3-rhev4-1122 kernel: Hardware name: Red Hat RHEV Hypervisor, BIOS 1.11.0-2.el7 04/01/2014
Nov 23 02:03:15 ssc-vm-g3-rhev4-1122 kernel: Call Trace:
Nov 23 02:03:15 ssc-vm-g3-rhev4-1122 kernel: [] dump_stack+0x19/0x1b
Nov 23 02:03:15 ssc-vm-g3-rhev4-1122 kernel: [] m0_arch_panic+0x66/0xb0 [m0ut]
Nov 23 02:03:15 ssc-vm-g3-rhev4-1122 kernel: [] m0_panic+0xe9/0xf0 [m0ut]
Nov 23 02:03:15 ssc-vm-g3-rhev4-1122 kernel: [] m0_ut_assertimpl+0xb7/0xc0 [m0ut]
Nov 23 02:03:15 ssc-vm-g3-rhev4-1122 kernel: [] ktest_dev+0x19c/0x830 [m0ut]
Nov 23 02:03:15 ssc-vm-g3-rhev4-1122 kernel: [] m0_ut_run+0x240/0x650 [m0ut]
Nov 23 02:03:15 ssc-vm-g3-rhev4-1122 kernel: [] ? printk+0x60/0x77
Nov 23 02:03:15 ssc-vm-g3-rhev4-1122 kernel: [] ? m0_arch_cookie_global_fini+0x10/0x10 [m0ut]
Nov 23 02:03:15 ssc-vm-g3-rhev4-1122 kernel: [] run_kernel_ut+0x1c/0x20 [m0ut]
Nov 23 02:03:15 ssc-vm-g3-rhev4-1122 kernel: [] m0_thread_trampoline+0x68/0xb0 [m0ut]
Nov 23 02:03:15 ssc-vm-g3-rhev4-1122 kernel: [] kthread_trampoline+0x38/0xd0 [m0ut]
Nov 23 02:03:15 ssc-vm-g3-rhev4-1122 kernel: [] kthread+0xd1/0xe0
Nov 23 02:03:15 ssc-vm-g3-rhev4-1122 kernel: [] ? insert_kthread_work+0x40/0x40
Nov 23 02:03:15 ssc-vm-g3-rhev4-1122 kernel: [] ret_from_fork_nospec_begin+0x21/0x21
Nov 23 02:03:15 ssc-vm-g3-rhev4-1122 kernel: [] ? insert_kthread_work+0x40/0x40

[cortx-admin]

For the convenience of the Seagate development team, this issue has been mirrored in a private Seagate Jira Server: https://jts.seagate.com/browse/EOS-26580. Note that community members will not be able to access that Jira server but that is not a problem since all activity in that Jira mirror will be copied into this GitHub issue.

[r-wambui]

Hi @yanqingfu I'm working on reproducing this issue.

[stale]

This issue/pull request has been marked as needs attention as it has been left pending without new activity for 4 days. Tagging @nkommuri @mehjoshi @huanghua78 for appropriate assignment. Sorry for the delay & Thank you for contributing to CORTX. We will get back to you as soon as possible.

[r-wambui]

@yanqingfu I did manage to run the kut tests without any issues, however I'm having some panic errors on the unit tests

After running the first time

dtm0_r [Sergey] motr[02189]:  b6b0  FATAL  [lib/assert.c:50:m0_panic]  panic: (svc != ((void *)0)) at drlink_fom_init() (dtm0/drlink.c:141)  [git: 2.0.0-527-35-g3d7b2da] /var/motr/m0ut/m0trace.2189
Motr panic: (svc != ((void *)0)) at drlink_fom_init() dtm0/drlink.c:141 (errno: 4) (last failed: none) [git: 2.0.0-527-35-g3d7b2da] pid: 2189  /var/motr/m0ut/m0trace.2189
/root/cortx-motr/motr/.libs/libmotr.so.2(m0_arch_backtrace+0x20)[0x7f719254df20]
/root/cortx-motr/motr/.libs/libmotr.so.2(m0_arch_panic+0xe6)[0x7f719254e0d6]
/root/cortx-motr/motr/.libs/libmotr.so.2(+0x389104)[0x7f719253c104]
/root/cortx-motr/motr/.libs/libmotr.so.2(m0_dtm0_req_post+0x38d)[0x7f71924fc9ad]
/root/cortx-motr/ut/.libs/libmotr-ut.so.0(+0x20e00d)[0x7f7193c3100d]
/root/cortx-motr/ut/.libs/libmotr-ut.so.0(+0x20e0a6)[0x7f7193c310a6]
/root/cortx-motr/ut/.libs/libmotr-ut.so.0(m0_ut_run+0x263)[0x7f7193cabc23]
/root/cortx-motr/ut/.libs/lt-m0ut(main+0x11c6)[0x4046b6]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f7190546555]
/root/cortx-motr/ut/.libs/lt-m0ut[0x40482e]
/root/cortx-motr/utils/m0run: line 425:  2189 Aborted

After running the second time

buffer_pool_ut
  buffer_pool_init  motr[102873]:  8040  FATAL  [lib/assert.c:50:m0_panic]  panic: (rc == 0) at test_init() (net/ut/buffer_pool_ut.c:60)  [git: 2.0.0-527-38-g43a75c5] /var/motr/m0ut/m0trace.102873
Motr panic: (rc == 0) at test_init() net/ut/buffer_pool_ut.c:60 (errno: 2) (last failed: none) [git: 2.0.0-527-38-g43a75c5] pid: 102873  /var/motr/m0ut/m0trace.102873
/root/cortx-motr/motr/.libs/libmotr.so.2(m0_arch_backtrace+0x20)[0x7fe74ace8870]
/root/cortx-motr/motr/.libs/libmotr.so.2(m0_arch_panic+0xe6)[0x7fe74ace8a26]
/root/cortx-motr/motr/.libs/libmotr.so.2(+0x381a54)[0x7fe74acd6a54]
/root/cortx-motr/ut/.libs/libmotr-ut.so.0(+0x2423c3)[0x7fe74c3f13c3]
/root/cortx-motr/ut/.libs/libmotr-ut.so.0(m0_ut_run+0x263)[0x7fe74c437ce3]
/root/cortx-motr/ut/.libs/lt-m0ut(main+0x11c9)[0x4046a9]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fe748fac555]
/root/cortx-motr/ut/.libs/lt-m0ut[0x404821]
/root/cortx-motr/utils/m0run: line 425: 102873 Aborted  

Could you please confirm if you are still facing the issues on kut test?

[yanqingfu]

yes, I am still facing the kut test issue and also the ut test issue you indicated.

[root@ssc-vm-g3-rhev4-1122 cortx-motr]# scripts/m0 run-ut -t idx-dix-mt
----- run_ut -t idx-dix-mt -----
START Iteration: 1 out of 1
idx-dix-mt [Anatoliy]
fom [Anatoliy] 33.53 sec 293 MiB
lsf [Anatoliy] 0.01 sec 96 KiB
lsfc [Vikram] 0.92 sec 14 MiB
dtm0 [Anatoliy] 0.14 sec 3 MiB
dtm0_putdel [Ivan] 3.57 sec 59 MiB
dtm0_e_then_s [Ivan] 3.00 sec 51 MiB
dtm0_c [Ivan] 7.71 sec 51 MiB
dtm0_r [Sergey] motr[01544]: c240 FATAL [lib/assert.c:50:m0_panic] panic: (svc != ((void *)0)) at drlink_fom_init() (dtm0/drlink.c:141) [git: 2.0.0-527-42-gab22d23] /var/motr/m0ut/m0trace.1544
Motr panic: (svc != ((void *)0)) at drlink_fom_init() dtm0/drlink.c:141 (errno: 4) (last failed: none) [git: 2.0.0-527-42-gab22d23] pid: 1544 /var/motr/m0ut/m0trace.1544
/var/cortx/cortx-motr/motr/.libs/libmotr.so.2(m0_arch_backtrace+0x20)[0x7f477a6e0910]
/var/cortx/cortx-motr/motr/.libs/libmotr.so.2(m0_arch_panic+0xe6)[0x7f477a6e0ac6]
/var/cortx/cortx-motr/motr/.libs/libmotr.so.2(+0x381af4)[0x7f477a6ceaf4]
/var/cortx/cortx-motr/motr/.libs/libmotr.so.2(m0_dtm0_req_post+0x38d)[0x7f477a68f39d]
/var/cortx/cortx-motr/ut/.libs/libmotr-ut.so.0(+0x20e0ad)[0x7f477bdb50ad]
/var/cortx/cortx-motr/ut/.libs/libmotr-ut.so.0(+0x20e146)[0x7f477bdb5146]
/var/cortx/cortx-motr/ut/.libs/libmotr-ut.so.0(m0_ut_run+0x263)[0x7f477be2fd83]
/var/cortx/cortx-motr/ut/.libs/lt-m0ut(main+0x11c9)[0x4046c9]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f47789a4555]
/var/cortx/cortx-motr/ut/.libs/lt-m0ut[0x404841]
/var/cortx/cortx-motr/utils/m0run: line 425: 1544 Aborted (core dumped) $(srcdir_path_of $binary) "$@"

[r-wambui]

@yanqingfu I'm pinging @chandradharraval for assistance.

Chandradhar, could you please look into this issue, why we're facing some errors while running the tests?

[chandradharraval]

@r-wambui - This is known issue and the same is fixed via #1285

[r-wambui]

@yanqingfu the issue has been resolved with #1285 could you please re-test, just a reminder ensure you have the latest codebase.

[cortx-admin]

Chandradhar Raval commented in Jira Server:

This is known issue and the same is duplicate of EOS-26618. Marking the same Resolved

[~935494] to verify the same with release which includes [https://github.com//pull/1285] fix and close the same

[yanqingfu]

kut test still has the same issue

[chandradharraval]

@yanqingfu , @r-wambui
PR for EOS-26618 #1285 is not merged yet. You can test this issue once the same is merged

[r-wambui]

I'm reopening this to test and confirm if it works.

[chandradharraval]

@r-wambui - #1285 is merged now, You can go ahead and test this issue

[cortx-admin]

Nagakishore Kommuri commented in Jira Server:

Verified by reporter.

[cortx-admin]

Mehul Manharlal Joshi commented in Jira Server:

This was closed erroneously. It is not a duplicate. Hence reopening it.

[chandradharraval]

@r-wambui , @yanqingfu - Are we still seeing this issue?

[cortx-admin]

Chandradhar Raval commented in Jira Server:

Marking this issue closed as the same UT is no longer failing

[johnbent]

@r-wambui and @yanqingfu , this was closed because we think this problem has been resolved. Please do re-open if you are still seeing this error.

[yanqingfu]

Sure. Right now, I am using the libfabric config and the kut test is disabled