Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error in `qemu-system-arm': malloc(): memory corruption #12

Open
amboar opened this issue Sep 11, 2017 · 3 comments
Open

Error in `qemu-system-arm': malloc(): memory corruption #12

amboar opened this issue Sep 11, 2017 · 3 comments

Comments

@amboar
Copy link
Member

amboar commented Sep 11, 2017

This was from a few commits on top of qemu/qemu@fd479c6. The extra commits added PMBus support to QEMU, along with a MAX31785 model. I couldn't find the core

*** Error in `qemu-system-arm': malloc(): memory corruption: 0x000056080a4e05b0 ***                                                                                                         [251/1927]
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x7908b)[0x7f3b8e57a08b]
/lib/x86_64-linux-gnu/libc.so.6(+0x84ace)[0x7f3b8e585ace]
/lib/x86_64-linux-gnu/libc.so.6(__libc_calloc+0x27b)[0x7f3b8e5887cb]
/lib64/ld-linux-x86-64.so.2(_dl_allocate_tls+0x2b)[0x7f3ba780dbab]
/lib/x86_64-linux-gnu/libpthread.so.0(pthread_create+0x8ec)[0x7f3b8e8d029c]
qemu-system-arm(+0x6286d5)[0x5608083436d5]
qemu-system-arm(+0x623567)[0x56080833e567]
qemu-system-arm(+0x62358d)[0x56080833e58d]
qemu-system-arm(+0x622d0e)[0x56080833dd0e]
qemu-system-arm(+0x625b90)[0x560808340b90]
qemu-system-arm(+0x622bee)[0x56080833dbee]
/lib/x86_64-linux-gnu/libglib-2.0.so.0(g_main_context_dispatch+0x2a7)[0x7f3b90179377]
qemu-system-arm(+0x624e13)[0x56080833fe13]
qemu-system-arm(main+0x46c7)[0x560807f4e107]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f3b8e5213f1]
qemu-system-arm(+0x23705a)[0x560807f5205a]
======= Memory map: ========
560807d1b000-56080856d000 r-xp 00000000 fd:01 3411821                    /home/andrew/src/qemu/aspeed/arm-softmmu/qemu-system-arm
56080876c000-560808949000 r--p 00851000 fd:01 3411821                    /home/andrew/src/qemu/aspeed/arm-softmmu/qemu-system-arm
560808949000-5608089e0000 rw-p 00a2e000 fd:01 3411821                    /home/andrew/src/qemu/aspeed/arm-softmmu/qemu-system-arm
5608089e0000-560808e3a000 rw-p 00000000 00:00 0
560809a8a000-56080a52c000 rw-p 00000000 00:00 0                          [heap]
7f3b2c000000-7f3b2c021000 rw-p 00000000 00:00 0
7f3b2c021000-7f3b30000000 ---p 00000000 00:00 0
7f3b332b6000-7f3b33eb8000 rw-p 00000000 00:00 0
7f3b33eb8000-7f3b34000000 rw-p 00000000 00:00 0
7f3b34000000-7f3b34022000 rw-p 00000000 00:00 0
7f3b34022000-7f3b38000000 ---p 00000000 00:00 0
7f3b383c4000-7f3b383c5000 ---p 00000000 00:00 0
7f3b383c5000-7f3b384c5000 rw-p 00000000 00:00 0
7f3b384c5000-7f3b384c6000 ---p 00000000 00:00 0
7f3b384c6000-7f3b385c6000 rw-p 00000000 00:00 0
7f3b385c6000-7f3b385c7000 ---p 00000000 00:00 0
7f3b385c7000-7f3b386c7000 rw-p 00000000 00:00 0
7f3b386c7000-7f3b386c8000 ---p 00000000 00:00 0
7f3b386c8000-7f3b387c8000 rw-p 00000000 00:00 0
7f3b387c8000-7f3b387c9000 ---p 00000000 00:00 0
7f3b387c9000-7f3b388c9000 rw-p 00000000 00:00 0
7f3b388c9000-7f3b388ca000 ---p 00000000 00:00 0
7f3b388ca000-7f3b389ca000 rw-p 00000000 00:00 0
7f3b389ca000-7f3b389cb000 ---p 00000000 00:00 0
7f3b389cb000-7f3b38acb000 rw-p 00000000 00:00 0
7f3b38acb000-7f3b38acc000 ---p 00000000 00:00 0
7f3b38acc000-7f3b38bcc000 rw-p 00000000 00:00 0
7f3b38bcc000-7f3b38bcd000 ---p 00000000 00:00 0
7f3b38bcd000-7f3b38ccd000 rw-p 00000000 00:00 0
7f3b38ccd000-7f3b38cce000 ---p 00000000 00:00 0
7f3b38cce000-7f3b38dce000 rw-p 00000000 00:00 0
7f3b38dce000-7f3b38dcf000 ---p 00000000 00:00 0
7f3b38dcf000-7f3b38ecf000 rw-p 00000000 00:00 0
7f3b38ecf000-7f3b38ed0000 ---p 00000000 00:00 0                                                                                                                                             [199/1927]
7f3b38ed0000-7f3b38fd0000 rw-p 00000000 00:00 0
7f3b38fd0000-7f3b38fd1000 ---p 00000000 00:00 0
7f3b38fd1000-7f3b390d1000 rw-p 00000000 00:00 0
7f3b390d1000-7f3b390d2000 ---p 00000000 00:00 0
7f3b390d2000-7f3b391d2000 rw-p 00000000 00:00 0
7f3b391d2000-7f3b391d3000 ---p 00000000 00:00 0
7f3b391d3000-7f3b392d3000 rw-p 00000000 00:00 0
7f3b392d3000-7f3b392d4000 ---p 00000000 00:00 0
7f3b392d4000-7f3b393d4000 rw-p 00000000 00:00 0
7f3b393d4000-7f3b393d5000 ---p 00000000 00:00 0
7f3b393d5000-7f3b394d5000 rw-p 00000000 00:00 0
7f3b394d5000-7f3b394d6000 ---p 00000000 00:00 0
7f3b394d6000-7f3b395d6000 rw-p 00000000 00:00 0
7f3b395d6000-7f3b395d7000 ---p 00000000 00:00 0
7f3b395d7000-7f3b396d7000 rw-p 00000000 00:00 0
7f3b396d7000-7f3b396d8000 ---p 00000000 00:00 0
7f3b396d8000-7f3b397d8000 rw-p 00000000 00:00 0
7f3b397d8000-7f3b397d9000 ---p 00000000 00:00 0
7f3b397d9000-7f3b398d9000 rw-p 00000000 00:00 0
7f3b398d9000-7f3b398da000 ---p 00000000 00:00 0
7f3b398da000-7f3b399da000 rw-p 00000000 00:00 0
7f3b399da000-7f3b399db000 ---p 00000000 00:00 0
7f3b399db000-7f3b39adb000 rw-p 00000000 00:00 0
7f3b39adb000-7f3b39adc000 ---p 00000000 00:00 0
7f3b39adc000-7f3b39bdc000 rw-p 00000000 00:00 0
7f3b39bdc000-7f3b39bdd000 ---p 00000000 00:00 0
7f3b39bdd000-7f3b39cdd000 rw-p 00000000 00:00 0
7f3b39cdd000-7f3b39cde000 ---p 00000000 00:00 0
7f3b39cde000-7f3b39dde000 rw-p 00000000 00:00 0
7f3b39dde000-7f3b39ddf000 ---p 00000000 00:00 0
7f3b39ddf000-7f3b39edf000 rw-p 00000000 00:00 0
7f3b39edf000-7f3b39ee0000 ---p 00000000 00:00 0
7f3b39ee0000-7f3b39fe0000 rw-p 00000000 00:00 0
7f3b39fe0000-7f3b39fe1000 ---p 00000000 00:00 0
7f3b39fe1000-7f3b3a0e1000 rw-p 00000000 00:00 0
7f3b3a0e1000-7f3b3a0e2000 ---p 00000000 00:00 0
7f3b3a0e2000-7f3b3a1e2000 rw-p 00000000 00:00 0
7f3b3a1e2000-7f3b3a1e3000 ---p 00000000 00:00 0
7f3b3a1e3000-7f3b3a2e3000 rw-p 00000000 00:00 0
7f3b3a2e3000-7f3b3a2e4000 ---p 00000000 00:00 0
7f3b3a2e4000-7f3b3a3e4000 rw-p 00000000 00:00 0
7f3b3a3e4000-7f3b3a3e5000 ---p 00000000 00:00 0
7f3b3a3e5000-7f3b3a4e5000 rw-p 00000000 00:00 0
7f3b3a4e5000-7f3b3a4e6000 ---p 00000000 00:00 0
7f3b3a4e6000-7f3b3a5e6000 rw-p 00000000 00:00 0
7f3b3a5e6000-7f3b3a5e7000 ---p 00000000 00:00 0
7f3b3a5e7000-7f3b3a6e7000 rw-p 00000000 00:00 0
7f3b3a6e7000-7f3b3a6e8000 ---p 00000000 00:00 0
7f3b3a6e8000-7f3b3a7e8000 rw-p 00000000 00:00 0
7f3b3a7e8000-7f3b3a7e9000 ---p 00000000 00:00 0
7f3b3a7e9000-7f3b3a8e9000 rw-p 00000000 00:00 0
7f3b3a8e9000-7f3b3a8ea000 ---p 00000000 00:00 0
7f3b3a8ea000-7f3b3a9ea000 rw-p 00000000 00:00 0
7f3b3a9ea000-7f3b3a9eb000 ---p 00000000 00:00 0
7f3b3a9eb000-7f3b3aaeb000 rw-p 00000000 00:00 0
7f3b3aaeb000-7f3b3aaec000 ---p 00000000 00:00 0
7f3b3aaec000-7f3b3abec000 rw-p 00000000 00:00 0
7f3b3abec000-7f3b3abed000 ---p 00000000 00:00 0
7f3b3abed000-7f3b3aced000 rw-p 00000000 00:00 0
7f3b3aced000-7f3b3acee000 ---p 00000000 00:00 0
7f3b3acee000-7f3b3adee000 rw-p 00000000 00:00 0
7f3b3adee000-7f3b3adef000 ---p 00000000 00:00 0
7f3b3adef000-7f3b3aeef000 rw-p 00000000 00:00 0
7f3b3aeef000-7f3b3aef0000 ---p 00000000 00:00 0
7f3b3aef0000-7f3b3aff0000 rw-p 00000000 00:00 0
7f3b3aff0000-7f3b3aff1000 ---p 00000000 00:00 0
7f3b3aff1000-7f3b3b0f1000 rw-p 00000000 00:00 0
7f3b3b0f1000-7f3b3b0f2000 ---p 00000000 00:00 0
7f3b3b0f2000-7f3b3b1f2000 rw-p 00000000 00:00 0
7f3b3b1f2000-7f3b3b1f3000 ---p 00000000 00:00 0
7f3b3b1f3000-7f3b3b2f3000 rw-p 00000000 00:00 0
7f3b3b2f3000-7f3b3b2f4000 ---p 00000000 00:00 0
7f3b3b2f4000-7f3b3b3f4000 rw-p 00000000 00:00 0
7f3b3b3f4000-7f3b3b3f5000 ---p 00000000 00:00 0
7f3b3b3f5000-7f3b3b4f5000 rw-p 00000000 00:00 0
7f3b3b4f5000-7f3b3b4f6000 ---p 00000000 00:00 0
7f3b3b4f6000-7f3b3b5f6000 rw-p 00000000 00:00 0
7f3b3b5f6000-7f3b3b5f7000 ---p 00000000 00:00 0
7f3b3b5f7000-7f3b3b6f7000 rw-p 00000000 00:00 0
7f3b3b6f7000-7f3b3b6f8000 ---p 00000000 00:00 0
7f3b3b6f8000-7f3b3b7f8000 rw-p 00000000 00:00 0
7f3b3b7f8000-7f3b3b7f9000 ---p 00000000 00:00 0
7f3b3b7f9000-7f3b3b8f9000 rw-p 00000000 00:00 0
7f3b3b8f9000-7f3b3b8fa000 ---p 00000000 00:00 0
7f3b3b8fa000-7f3b3b9fa000 rw-p 00000000 00:00 0
7f3b3b9fa000-7f3b3b9fb000 ---p 00000000 00:00 0
7f3b3b9fb000-7f3b3bafb000 rw-p 00000000 00:00 0
7f3b3bafb000-7f3b3bafc000 ---p 00000000 00:00 0
7f3b3bafc000-7f3b3bbfc000 rw-p 00000000 00:00 0
7f3b3bbfc000-7f3b3bbfd000 ---p 00000000 00:00 0
7f3b3bbfd000-7f3b3bcfd000 rw-p 00000000 00:00 0
7f3b3bcfd000-7f3b3bcfe000 ---p 00000000 00:00 0
7f3b3bcfe000-7f3b3bdfe000 rw-p 00000000 00:00 0
7f3b3bdfe000-7f3b3bdff000 ---p 00000000 00:00 0
7f3b3bdff000-7f3b3beff000 rw-p 00000000 00:00 0
7f3b3beff000-7f3b3bf00000 ---p 00000000 00:00 0
7f3b3bf00000-7f3b3c000000 rw-p 00000000 00:00 0
7f3b3c000000-7f3b3c1e5000 rw-p 00000000 00:00 0
7f3b3c1e5000-7f3b40000000 ---p 00000000 00:00 0
7f3b400fd000-7f3b400fe000 ---p 00000000 00:00 0
7f3b400fe000-7f3b401fe000 rw-p 00000000 00:00 0
7f3b401fe000-7f3b401ff000 ---p 00000000 00:00 0
7f3b401fe000-7f3b401ff000 ---p 00000000 00:00 0                                                                                                                                              [97/1927]
7f3b401ff000-7f3b402ff000 rw-p 00000000 00:00 0
7f3b402ff000-7f3b40300000 ---p 00000000 00:00 0
7f3b40300000-7f3b40400000 rw-p 00000000 00:00 0
7f3b40400000-7f3b48400000 rw-p 00000000 00:00 0
7f3b48400000-7f3b48401000 ---p 00000000 00:00 0
7f3b484f8000-7f3b484f9000 ---p 00000000 00:00 0
7f3b484f9000-7f3b525fd000 rw-p 00000000 00:00 0
7f3b525fd000-7f3b525fe000 ---p 00000000 00:00 0
7f3b525fe000-7f3b54e00000 rw-p 00000000 00:00 0
7f3b54e00000-7f3b74e00000 rw-p 00000000 00:00 0
7f3b74e00000-7f3b74e01000 ---p 00000000 00:00 0
7f3b74e7d000-7f3b74e7e000 ---p 00000000 00:00 0
7f3b74e7e000-7f3b75000000 rw-p 00000000 00:00 0
7f3b75000000-7f3b75009000 rw-p 00000000 00:00 0
7f3b75009000-7f3b7500a000 ---p 00000000 00:00 0
7f3b7502b000-7f3b7502c000 ---p 00000000 00:00 0
7f3b7502c000-7f3b7582c000 rw-p 00000000 00:00 0
7f3b7587e000-7f3b7d87e000 rwxp 00000000 00:00 0
7f3b7d87e000-7f3b7d87f000 ---p 00000000 00:00 0
7f3b7d87f000-7f3b7d880000 ---p 00000000 00:00 0
7f3b7d880000-7f3b7e080000 rw-p 00000000 00:00 0
7f3b7e080000-7f3b7e087000 r-xp 00000000 fd:01 8945565                    /usr/lib/x86_64-linux-gnu/liburcu-bp.so.4.1.0
7f3b7e087000-7f3b7e286000 ---p 00007000 fd:01 8945565                    /usr/lib/x86_64-linux-gnu/liburcu-bp.so.4.1.0
7f3b7e286000-7f3b7e287000 r--p 00006000 fd:01 8945565                    /usr/lib/x86_64-linux-gnu/liburcu-bp.so.4.1.0
7f3b7e287000-7f3b7e288000 rw-p 00007000 fd:01 8945565                    /usr/lib/x86_64-linux-gnu/liburcu-bp.so.4.1.0
7f3b7e288000-7f3b7e292000 r-xp 00000000 fd:01 8945560                    /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0.0.0
7f3b7e292000-7f3b7e492000 ---p 0000a000 fd:01 8945560                    /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0.0.0
7f3b7e492000-7f3b7e493000 r--p 0000a000 fd:01 8945560                    /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0.0.0
7f3b7e493000-7f3b7e494000 rw-p 0000b000 fd:01 8945560                    /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0.0.0
7f3b7e494000-7f3b7e4a4000 rw-p 00000000 00:00 0
7f3b7e4a4000-7f3b7e4b7000 r-xp 00000000 fd:01 12189883                   /lib/x86_64-linux-gnu/libgpg-error.so.0.21.0
7f3b7e4b7000-7f3b7e6b6000 ---p 00013000 fd:01 12189883                   /lib/x86_64-linux-gnu/libgpg-error.so.0.21.0
7f3b7e6b6000-7f3b7e6b7000 r--p 00012000 fd:01 12189883                   /lib/x86_64-linux-gnu/libgpg-error.so.0.21.0
7f3b7e6b7000-7f3b7e6b8000 rw-p 00013000 fd:01 12189883                   /lib/x86_64-linux-gnu/libgpg-error.so.0.21.0
7f3b7e6b8000-7f3b7e745000 r-xp 00000000 fd:01 8922196                    /usr/lib/x86_64-linux-gnu/libvorbisenc.so.2.0.11
7f3b7e745000-7f3b7e944000 ---p 0008d000 fd:01 8922196                    /usr/lib/x86_64-linux-gnu/libvorbisenc.so.2.0.11
7f3b7e944000-7f3b7e960000 r--p 0008c000 fd:01 8922196                    /usr/lib/x86_64-linux-gnu/libvorbisenc.so.2.0.11
7f3b7e960000-7f3b7e961000 rw-p 000a8000 fd:01 8922196                    /usr/lib/x86_64-linux-gnu/libvorbisenc.so.2.0.11
7f3b7e961000-7f3b7e98b000 r-xp 00000000 fd:01 8922377                    /usr/lib/x86_64-linux-gnu/libvorbis.so.0.4.8
7f3b7e98b000-7f3b7eb8a000 ---p 0002a000 fd:01 8922377                    /usr/lib/x86_64-linux-gnu/libvorbis.so.0.4.8
7f3b7eb8a000-7f3b7eb8b000 r--p 00029000 fd:01 8922377                    /usr/lib/x86_64-linux-gnu/libvorbis.so.0.4.8
7f3b7eb8b000-7f3b7eb8c000 rw-p 0002a000 fd:01 8922377                    /usr/lib/x86_64-linux-gnu/libvorbis.so.0.4.8
7f3b7eb8c000-7f3b7eb93000 r-xp 00000000 fd:01 8922143                    /usr/lib/x86_64-linux-gnu/libogg.so.0.8.2
7f3b7eb93000-7f3b7ed93000 ---p 00007000 fd:01 8922143                    /usr/lib/x86_64-linux-gnu/libogg.so.0.8.2
7f3b7ed93000-7f3b7ed94000 r--p 00007000 fd:01 8922143                    /usr/lib/x86_64-linux-gnu/libogg.so.0.8.2
7f3b7ed94000-7f3b7ed95000 rw-p 00008000 fd:01 8922143                    /usr/lib/x86_64-linux-gnu/libogg.so.0.8.2
7f3b7ed95000-7f3b7ee0a000 r-xp 00000000 fd:01 8930627                    /usr/lib/x86_64-linux-gnu/libFLAC.so.8.3.0
7f3b7ee0a000-7f3b7f00a000 ---p 00075000 fd:01 8930627                    /usr/lib/x86_64-linux-gnu/libFLAC.so.8.3.0
7f3b7f00a000-7f3b7f00b000 r--p 00075000 fd:01 8930627                    /usr/lib/x86_64-linux-gnu/libFLAC.so.8.3.0
7f3b7f00b000-7f3b7f00c000 rw-p 00076000 fd:01 8930627                    /usr/lib/x86_64-linux-gnu/libFLAC.so.8.3.0
7f3b7f00c000-7f3b7f022000 r-xp 00000000 fd:01 12189932                   /lib/x86_64-linux-gnu/libnsl-2.24.so
7f3b7f022000-7f3b7f221000 ---p 00016000 fd:01 12189932                   /lib/x86_64-linux-gnu/libnsl-2.24.so                                                                                [45/1927]
7f3b7f221000-7f3b7f222000 r--p 00015000 fd:01 12189932                   /lib/x86_64-linux-gnu/libnsl-2.24.so
7f3b7f222000-7f3b7f223000 rw-p 00016000 fd:01 12189932                   /lib/x86_64-linux-gnu/libnsl-2.24.so
7f3b7f223000-7f3b7f225000 rw-p 00000000 00:00 0
7f3b7f225000-7f3b7f32c000 r-xp 00000000 fd:01 12189720                   /lib/x86_64-linux-gnu/libgcrypt.so.20.1.6
7f3b7f32c000-7f3b7f52b000 ---p 00107000 fd:01 12189720                   /lib/x86_64-linux-gnu/libgcrypt.so.20.1.6
7f3b7f52b000-7f3b7f52d000 r--p 00106000 fd:01 12189720                   /lib/x86_64-linux-gnu/libgcrypt.so.20.1.6
7f3b7f52d000-7f3b7f534000 rw-p 00108000 fd:01 12189720                   /lib/x86_64-linux-gnu/libgcrypt.so.20.1.6
7f3b7f534000-7f3b7f54b000 r-xp 00000000 fd:01 8915454                    /usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1
7f3b7f54b000-7f3b7f74a000 ---p 00017000 fd:01 8915454                    /usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1
7f3b7f74a000-7f3b7f74b000 r--p 00016000 fd:01 8915454                    /usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1
7f3b7f74b000-7f3b7f74c000 rw-p 00017000 fd:01 8915454                    /usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1
7f3b7f74c000-7f3b7f770000 r-xp 00000000 fd:01 12190986                   /lib/x86_64-linux-gnu/liblzma.so.5.2.2
7f3b7f770000-7f3b7f970000 ---p 00024000 fd:01 12190986                   /lib/x86_64-linux-gnu/liblzma.so.5.2.2
7f3b7f970000-7f3b7f971000 r--p 00024000 fd:01 12190986                   /lib/x86_64-linux-gnu/liblzma.so.5.2.2
7f3b7f971000-7f3b7f972000 rw-p 00025000 fd:01 12190986                   /lib/x86_64-linux-gnu/liblzma.so.5.2.2
7f3b7f972000-7f3b7f97b000 r-xp 00000000 fd:01 12189922                   /lib/x86_64-linux-gnu/libcrypt-2.24.so
7f3b7f97b000-7f3b7fb7a000 ---p 00009000 fd:01 12189922                   /lib/x86_64-linux-gnu/libcrypt-2.24.so
7f3b7fb7a000-7f3b7fb7b000 r--p 00008000 fd:01 12189922                   /lib/x86_64-linux-gnu/libcrypt-2.24.so
7f3b7fb7b000-7f3b7fb7c000 rw-p 00009000 fd:01 12189922                   /lib/x86_64-linux-gnu/libcrypt-2.24.so
7f3b7fb7c000-7f3b7fbaa000 rw-p 00000000 00:00 0
7f3b7fbaa000-7f3b7fcab000 r-xp 00000000 fd:01 8913447                    /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
7f3b7fcab000-7f3b7feab000 ---p 00101000 fd:01 8913447                    /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
7f3b7feab000-7f3b7feae000 r--p 00101000 fd:01 8913447                    /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
7f3b7feae000-7f3b7feb0000 rw-p 00104000 fd:01 8913447                    /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
7f3b7feb0000-7f3b7feb1000 rw-p 00000000 00:00 0
7f3b7feb1000-7f3b7fef7000 r-xp 00000000 fd:01 8919252                    /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
7f3b7fef7000-7f3b800f7000 ---p 00046000 fd:01 8919252                    /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
7f3b800f7000-7f3b800f9000 r--p 00046000 fd:01 8919252                    /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
7f3b800f9000-7f3b800fb000 rw-p 00048000 fd:01 8919252                    /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
7f3b800fb000-7f3b800fc000 rw-p 00000000 00:00 0
7f3b800fc000-7f3b8010a000 r-xp 00000000 fd:01 8916834                    /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
7f3b8010a000-7f3b80309000 ---p 0000e000 fd:01 8916834                    /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
7f3b80309000-7f3b8030a000 r--p 0000d000 fd:01 8916834                    /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
7f3b8030a000-7f3b8030b000 rw-p 0000e000 fd:01 8916834                    /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
7f3b8030b000-7f3b80332000 r-xp 00000000 fd:01 8917595                    /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
7f3b80332000-7f3b80532000 ---p 00027000 fd:01 8917595                    /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
7f3b80532000-7f3b80533000 r--p 00027000 fd:01 8917595                    /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
7f3b80533000-7f3b80534000 rw-p 00028000 fd:01 8917595                    /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
7f3b80534000-7f3b80559000 r-xp 00000000 fd:01 8926121                    /usr/lib/x86_64-linux-gnu/libkj-0.5.3.so
7f3b80559000-7f3b80759000 ---p 00025000 fd:01 8926121                    /usr/lib/x86_64-linux-gnu/libkj-0.5.3.so
7f3b80759000-7f3b8075a000 r--p 00025000 fd:01 8926121                    /usr/lib/x86_64-linux-gnu/libkj-0.5.3.so
7f3b8075a000-7f3b8075b000 rw-p 00026000 fd:01 8926121                    /usr/lib/x86_64-linux-gnu/libkj-0.5.3.so
7f3b8075b000-7f3b80772000 r-xp 00000000 fd:01 8926123                    /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.62.0
7f3b80772000-7f3b80972000 ---p 00017000 fd:01 8926123                    /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.62.0
7f3b80972000-7f3b80973000 r--p 00017000 fd:01 8926123                    /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.62.0
7f3b80973000-7f3b80974000 rw-p 00018000 fd:01 8926123                    /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.62.0
7f3b80974000-7f3b8097a000 r-xp 00000000 fd:01 8915503                    /usr/lib/x86_64-linux-gnu/libdatrie.so.1.3.3
7f3b8097a000-7f3b80b79000 ---p 00006000 fd:01 8915503                    /usr/lib/x86_64-linux-gnu/libdatrie.so.1.3.3
7f3b80b79000-7f3b80b7a000 r--p 00005000 fd:01 8915503                    /usr/lib/x86_64-linux-gnu/libdatrie.so.1.3.3
7f3b80b7a000-7f3b80b7b000 rw-p 00006000 fd:01 8915503                    /usr/lib/x86_64-linux-gnu/libdatrie.so.1.3.3
7f3b80b7b000-7f3b80b9f000 r-xp 00000000 fd:01 8918167                    /usr/lib/x86_64-linux-gnu/libgraphite2.so.3.0.1
7f3b80b9f000-7f3b80d9e000 ---p 00024000 fd:01 8918167                    /usr/lib/x86_64-linux-gnu/libgraphite2.so.3.0.1
7f3b80d9e000-7f3b80da0000 r--p 00023000 fd:01 8918167                    /usr/lib/x86_64-linux-gnu/libgraphite2.so.3.0.1
7f3b80da0000-7f3b80da1000 rw-p 00025000 fd:01 8918167                    /usr/lib/x86_64-linux-gnu/libgraphite2.so.3.0.1
7f3b80da1000-7f3b80da6000 r-xp 00000000 fd:01 8920607                    /usr/lib/x86_64-linux-gnu/libasyncns.so.0.3.1
7f3b80da6000-7f3b80fa5000 ---p 00005000 fd:01 8920607                    /usr/lib/x86_64-linux-gnu/libasyncns.so.0.3.1
7f3b80fa5000-7f3b80fa6000 r--p 00004000 fd:01 8920607                    /usr/lib/x86_64-linux-gnu/libasyncns.so.0.3.1
7f3b80fa6000-7f3b80fa7000 rw-p 00005000 fd:01 8920607                    /usr/lib/x86_64-linux-gnu/libasyncns.so.0.3.1
7f3b80fa7000-7f3b8101a000 r-xp 00000000 fd:01 8921349                    /usr/lib/x86_64-linux-gnu/libsndfile.so.1.0.27
7f3b8101a000-7f3b81219000 ---p 00073000 fd:01 8921349                    /usr/lib/x86_64-linux-gnu/libsndfile.so.1.0.27
7f3b81219000-7f3b8121c000 r--p 00072000 fd:01 8921349                    /usr/lib/x86_64-linux-gnu/libsndfile.so.1.0.27
7f3b8121c000-7f3b8121d000 rw-p 00075000 fd:01 8921349                    /usr/lib/x86_64-linux-gnu/libsndfile.so.1.0.27
7f3b8121d000-7f3b8121f000 rw-p 00000000 00:00 0
7f3b8121f000-7f3b81227000 r-xp 00000000 fd:01 12192318                   /lib/x86_64-linux-gnu/libwrap.so.0.7.6
7f3b81227000-7f3b81427000 ---p 00008000 fd:01 12192318                   /lib/x86_64-linux-gnu/libwrap.so.0.7.6
7f3b81427000-7f3b81428000 r--p 00008000 fd:01 12192318                   /lib/x86_64-linux-gnu/libwrap.so.0.7.6
7f3b81428000-7f3b81429000 rw-p 00009000 fd:01 12192318                   /lib/x86_64-linux-gnu/libwrap.so.0.7.6
7f3b81429000-7f3b8143e000 r-xp 00000000 fd:01 8913789                    /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
7f3b8143e000-7f3b8163d000 ---p 00015000 fd:01 8913789                    /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
7f3b8163d000-7f3b8163e000 r--p 00014000 fd:01 8913789                    /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
7f3b8163e000-7f3b8163f000 rw-p 00015000 fd:01 8913789                    /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
7f3b8163f000-7f3b81673000 r-xp 00000000 fd:01 8917461                    /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
7f3b81673000-7f3b81873000 ---p 00034000 fd:01 8917461                    /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
7f3b81873000-7f3b81874000 r--p 00034000 fd:01 8917461                    /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
7f3b81874000-7f3b81875000 rw-p 00035000 fd:01 8917461                    /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
7f3b81875000-7f3b81876000 rw-p 00000000 00:00 0
7f3b81876000-7f3b81915000 r-xp 00000000 fd:01 8915052                    /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
7f3b81915000-7f3b81b15000 ---p 0009f000 fd:01 8915052                    /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
7f3b81b15000-7f3b81b16000 r--p 0009f000 fd:01 8915052                    /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
7f3b81b16000-7f3b81b19000 rw-p 000a0000 fd:01 8915052                    /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
7f3b81b19000-7f3b81ba0000 r-xp 00000000 fd:01 8919257                    /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
7f3b81ba0000-7f3b81d9f000 ---p 00087000 fd:01 8919257                    /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
7f3b81d9f000-7f3b81da2000 r--p 00086000 fd:01 8919257                    /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
7f3b81da2000-7f3b81da5000 rw-p 00089000 fd:01 8919257                    /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
7f3b81da5000-7f3b81da6000 rw-p 00000000 00:00 0
7f3b81da6000-7f3b81dae000 r-xp 00000000 fd:01 8923743                    /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
7f3b81dae000-7f3b81fad000 ---p 00008000 fd:01 8923743                    /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
7f3b81fad000-7f3b81fae000 r--p 00007000 fd:01 8923743                    /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
7f3b81fae000-7f3b81faf000 rw-p 00008000 fd:01 8923743                    /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
7f3b81faf000-7f3b81fb2000 r-xp 00000000 fd:01 12189766                   /lib/x86_64-linux-gnu/libkeyutils.so.1.5
7f3b81fb2000-7f3b821b1000 ---p 00003000 fd:01 12189766                   /lib/x86_64-linux-gnu/libkeyutils.so.1.5
7f3b821b1000-7f3b821b2000 r--p 00002000 fd:01 12189766                   /lib/x86_64-linux-gnu/libkeyutils.so.1.5
7f3b821b2000-7f3b821b3000 rw-p 00003000 fd:01 12189766                   /lib/x86_64-linux-gnu/libkeyutils.so.1.5
7f3b821b3000-7f3b82200000 r-xp 00000000 fd:01 8923592                    /usr/lib/x86_64-linux-gnu/libprotobuf-lite.so.10.0.0
7f3b82200000-7f3b823ff000 ---p 0004d000 fd:01 8923592                    /usr/lib/x86_64-linux-gnu/libprotobuf-lite.so.10.0.0Aborted (core dumped)
@legoater
Copy link

Do you have some scenario to reproduce the corruption ? if so, may be you could use valgrind to track it.

@amboar
Copy link
Member Author

amboar commented Sep 11, 2017

I think so. I ran the configuration again and got a similar result. I then tried with v2.10.0 and the failure went away, so I'm possibly doing something wrong with the patches I have on top. It was a bit of a yak-shaving exercise so I haven't tried to reproduce it again after moving to v2.10.0.

@amboar
Copy link
Member Author

amboar commented Sep 11, 2017

Also I have a core file contrary to the description in the post. I discovered the need for ulimit -c unlimited in my .bashrc.

amboar pushed a commit that referenced this issue Oct 23, 2018
@jemmy858585
migration: implement io_set_aio_fd_handler function for RDMA QIOChannel
4d9f675 
if qio_channel_rdma_readv return QIO_CHANNEL_ERR_BLOCK, the destination qemu
crash.

The backtrace is:
(gdb) bt
    #0  0x0000000000000000 in ?? ()
    #1  0x00000000008db50e in qio_channel_set_aio_fd_handler (ioc=0x38111e0, ctx=0x3726080,
        io_read=0x8db841 <qio_channel_restart_read>, io_write=0x0, opaque=0x38111e0) at io/channel.c:
    #2  0x00000000008db952 in qio_channel_set_aio_fd_handlers (ioc=0x38111e0) at io/channel.c:438
    #3  0x00000000008dbab4 in qio_channel_yield (ioc=0x38111e0, condition=G_IO_IN) at io/channel.c:47
    #4  0x00000000007a870b in channel_get_buffer (opaque=0x38111e0, buf=0x440c038 "", pos=0, size=327
        at migration/qemu-file-channel.c:83
    #5  0x00000000007a70f6 in qemu_fill_buffer (f=0x440c000) at migration/qemu-file.c:299
    #6  0x00000000007a79d0 in qemu_peek_byte (f=0x440c000, offset=0) at migration/qemu-file.c:562
    #7  0x00000000007a7a22 in qemu_get_byte (f=0x440c000) at migration/qemu-file.c:575
    #8  0x00000000007a7c78 in qemu_get_be32 (f=0x440c000) at migration/qemu-file.c:655
    #9  0x00000000007a0508 in qemu_loadvm_state (f=0x440c000) at migration/savevm.c:2126
    #10 0x0000000000794141 in process_incoming_migration_co (opaque=0x0) at migration/migration.c:366
    #11 0x000000000095c598 in coroutine_trampoline (i0=84033984, i1=0) at util/coroutine-ucontext.c:1
    #12 0x00007f9c0db56d40 in ?? () from /lib64/libc.so.6
    #13 0x00007f96fe858760 in ?? ()
    #14 0x0000000000000000 in ?? ()

RDMA QIOChannel not implement io_set_aio_fd_handler. so
qio_channel_set_aio_fd_handler will access NULL pointer.

Signed-off-by: Lidong Chen <lidongchen@tencent.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
amboar pushed a commit that referenced this issue Oct 23, 2018
@jemmy858585
migration: invoke qio_channel_yield only when qemu_in_coroutine()
5d5f4d8 
when qio_channel_read return QIO_CHANNEL_ERR_BLOCK, the source qemu crash.

The backtrace is:
    (gdb) bt
    #0  0x00007fb20aba91d7 in raise () from /lib64/libc.so.6
    #1  0x00007fb20abaa8c8 in abort () from /lib64/libc.so.6
    #2  0x00007fb20aba2146 in __assert_fail_base () from /lib64/libc.so.6
    #3  0x00007fb20aba21f2 in __assert_fail () from /lib64/libc.so.6
    #4  0x00000000008dba2d in qio_channel_yield (ioc=0x22f9e20, condition=G_IO_IN) at io/channel.c:460
    #5  0x00000000007a870b in channel_get_buffer (opaque=0x22f9e20, buf=0x3d54038 "", pos=0, size=32768)
        at migration/qemu-file-channel.c:83
    #6  0x00000000007a70f6 in qemu_fill_buffer (f=0x3d54000) at migration/qemu-file.c:299
    #7  0x00000000007a79d0 in qemu_peek_byte (f=0x3d54000, offset=0) at migration/qemu-file.c:562
    #8  0x00000000007a7a22 in qemu_get_byte (f=0x3d54000) at migration/qemu-file.c:575
    #9  0x00000000007a7c46 in qemu_get_be16 (f=0x3d54000) at migration/qemu-file.c:647
    #10 0x0000000000796db7 in source_return_path_thread (opaque=0x2242280) at migration/migration.c:1794
    #11 0x00000000009428fa in qemu_thread_start (args=0x3e58420) at util/qemu-thread-posix.c:504
    #12 0x00007fb20af3ddc5 in start_thread () from /lib64/libpthread.so.0
    #13 0x00007fb20ac6b74d in clone () from /lib64/libc.so.6

This patch fixed by invoke qio_channel_yield only when qemu_in_coroutine().

Signed-off-by: Lidong Chen <lidongchen@tencent.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
amboar pushed a commit that referenced this issue Oct 23, 2018
@jemmy858585
migration: implement the shutdown for RDMA QIOChannel
54db882 
Because RDMA QIOChannel not implement shutdown function,
If the to_dst_file was set error, the return path thread
will wait forever. and the migration thread will wait
return path thread exit.

the backtrace of return path thread is:

(gdb) bt
    #0  0x00007f372a76bb0f in ppoll () from /lib64/libc.so.6
    #1  0x000000000071dc24 in qemu_poll_ns (fds=0x7ef7091d0580, nfds=2, timeout=100000000)
        at qemu-timer.c:325
    #2  0x00000000006b2fba in qemu_rdma_wait_comp_channel (rdma=0xd424000)
        at migration/rdma.c:1501
    #3  0x00000000006b3191 in qemu_rdma_block_for_wrid (rdma=0xd424000, wrid_requested=4000,
        byte_len=0x7ef7091d0640) at migration/rdma.c:1580
    #4  0x00000000006b3638 in qemu_rdma_exchange_get_response (rdma=0xd424000,
        head=0x7ef7091d0720, expecting=3, idx=0) at migration/rdma.c:1726
    #5  0x00000000006b3ad6 in qemu_rdma_exchange_recv (rdma=0xd424000, head=0x7ef7091d0720,
        expecting=3) at migration/rdma.c:1903
    #6  0x00000000006b5d03 in qemu_rdma_get_buffer (opaque=0x6a57dc0, buf=0x5c80030 "", pos=8,
        size=32768) at migration/rdma.c:2714
    #7  0x00000000006a9635 in qemu_fill_buffer (f=0x5c80000) at migration/qemu-file.c:232
    #8  0x00000000006a9ecd in qemu_peek_byte (f=0x5c80000, offset=0)
        at migration/qemu-file.c:502
    #9  0x00000000006a9f1f in qemu_get_byte (f=0x5c80000) at migration/qemu-file.c:515
    #10 0x00000000006aa162 in qemu_get_be16 (f=0x5c80000) at migration/qemu-file.c:591
    #11 0x00000000006a46d3 in source_return_path_thread (
        opaque=0xd826a0 <current_migration.37100>) at migration/migration.c:1331
    #12 0x00007f372aa49e25 in start_thread () from /lib64/libpthread.so.0
    #13 0x00007f372a77635d in clone () from /lib64/libc.so.6

the backtrace of migration thread is:

(gdb) bt
    #0  0x00007f372aa4af57 in pthread_join () from /lib64/libpthread.so.0
    #1  0x00000000007d5711 in qemu_thread_join (thread=0xd826f8 <current_migration.37100+88>)
        at util/qemu-thread-posix.c:504
    #2  0x00000000006a4bc5 in await_return_path_close_on_source (
        ms=0xd826a0 <current_migration.37100>) at migration/migration.c:1460
    #3  0x00000000006a53e4 in migration_completion (s=0xd826a0 <current_migration.37100>,
        current_active_state=4, old_vm_running=0x7ef7089cf976, start_time=0x7ef7089cf980)
        at migration/migration.c:1695
    #4  0x00000000006a5c54 in migration_thread (opaque=0xd826a0 <current_migration.37100>)
        at migration/migration.c:1837
    #5  0x00007f372aa49e25 in start_thread () from /lib64/libpthread.so.0
    #6  0x00007f372a77635d in clone () from /lib64/libc.so.6

Signed-off-by: Lidong Chen <lidongchen@tencent.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
amboar pushed a commit that referenced this issue Oct 23, 2018
@elmarco @bonzini
megasas: fix sglist leak
48285ba 
tests/cdrom-test -p /x86_64/cdrom/boot/megasas

Produces the following ASAN leak.

==25700==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7f06f8faac48 in malloc (/lib64/libasan.so.5+0xeec48)
    #1 0x7f06f87a73c5 in g_malloc (/lib64/libglib-2.0.so.0+0x523c5)
    #2 0x55a729f17738 in pci_dma_sglist_init /home/elmarco/src/qq/include/hw/pci/pci.h:818
    #3 0x55a729f2a706 in megasas_map_dcmd /home/elmarco/src/qq/hw/scsi/megasas.c:698
    #4 0x55a729f39421 in megasas_handle_dcmd /home/elmarco/src/qq/hw/scsi/megasas.c:1574
    #5 0x55a729f3f70d in megasas_handle_frame /home/elmarco/src/qq/hw/scsi/megasas.c:1955
    #6 0x55a729f40939 in megasas_mmio_write /home/elmarco/src/qq/hw/scsi/megasas.c:2119
    #7 0x55a729f41102 in megasas_port_write /home/elmarco/src/qq/hw/scsi/megasas.c:2170
    #8 0x55a729220e60 in memory_region_write_accessor /home/elmarco/src/qq/memory.c:527
    #9 0x55a7292212b3 in access_with_adjusted_size /home/elmarco/src/qq/memory.c:594
    #10 0x55a72922cf70 in memory_region_dispatch_write /home/elmarco/src/qq/memory.c:1473
    #11 0x55a7290f5907 in flatview_write_continue /home/elmarco/src/qq/exec.c:3255
    #12 0x55a7290f5ceb in flatview_write /home/elmarco/src/qq/exec.c:3294
    #13 0x55a7290f6457 in address_space_write /home/elmarco/src/qq/exec.c:3384
    #14 0x55a7290f64a8 in address_space_rw /home/elmarco/src/qq/exec.c:3395
    #15 0x55a72929ecb0 in kvm_handle_io /home/elmarco/src/qq/accel/kvm/kvm-all.c:1729
    #16 0x55a7292a0db5 in kvm_cpu_exec /home/elmarco/src/qq/accel/kvm/kvm-all.c:1969
    #17 0x55a7291c4212 in qemu_kvm_cpu_thread_fn /home/elmarco/src/qq/cpus.c:1215
    #18 0x55a72a966a6c in qemu_thread_start /home/elmarco/src/qq/util/qemu-thread-posix.c:504
    #19 0x7f06ed486593 in start_thread (/lib64/libpthread.so.0+0x7593)

Move the qemu_sglist_destroy() from megasas_complete_command() to
megasas_unmap_frame(), so map/unmap are balanced.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180814141247.32336-1-marcandre.lureau@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
amboar pushed a commit that referenced this issue Oct 23, 2018
@elmarco @dagrh
hmp: fix migrate status timer leak
d34a10a 
Spotted by ASAN doing some manual testing:

Direct leak of 48 byte(s) in 1 object(s) allocated from:
    #0 0x7f5fcdc75e50 in calloc (/lib64/libasan.so.5+0xeee50)
    #1 0x7f5fcd47241d in g_malloc0 (/lib64/libglib-2.0.so.0+0x5241d)
    #2 0x55f989be92ce in timer_new /home/elmarco/src/qq/include/qemu/timer.h:561
    #3 0x55f989be92ff in timer_new_ms /home/elmarco/src/qq/include/qemu/timer.h:630
    #4 0x55f989c0219d in hmp_migrate /home/elmarco/src/qq/hmp.c:2038
    #5 0x55f98955927b in handle_hmp_command /home/elmarco/src/qq/monitor.c:3498
    #6 0x55f98955fb8c in monitor_command_cb /home/elmarco/src/qq/monitor.c:4371
    #7 0x55f98ad40f11 in readline_handle_byte /home/elmarco/src/qq/util/readline.c:393
    #8 0x55f98955fa4f in monitor_read /home/elmarco/src/qq/monitor.c:4354
    #9 0x55f98aae30d7 in qemu_chr_be_write_impl /home/elmarco/src/qq/chardev/char.c:175
    #10 0x55f98aae317a in qemu_chr_be_write /home/elmarco/src/qq/chardev/char.c:187
    #11 0x55f98aae940c in fd_chr_read /home/elmarco/src/qq/chardev/char-fd.c:66
    #12 0x55f98ab63018 in qio_channel_fd_source_dispatch /home/elmarco/src/qq/io/channel-watch.c:84
    #13 0x7f5fcd46c8ac in g_main_dispatch gmain.c:3177

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180901134652.25884-1-marcandre.lureau@redhat.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
amboar pushed a commit that referenced this issue Oct 23, 2018
@slp @kevmw
block/linux-aio: acquire AioContext before qemu_laio_process_completions
e091f0e 
In qemu_laio_process_completions_and_submit, the AioContext is acquired
before the ioq_submit iteration and after qemu_laio_process_completions,
but the latter is not thread safe either.

This change avoids a number of random crashes when the Main Thread and
an IO Thread collide processing completions for the same AioContext.
This is an example of such crash:

 - The IO Thread is trying to acquire the AioContext at aio_co_enter,
   which evidences that it didn't lock it before:

Thread 3 (Thread 0x7fdfd8bd8700 (LWP 36743)):
 #0  0x00007fdfe0dd542d in __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
 #1  0x00007fdfe0dd0de6 in _L_lock_870 () at /lib64/libpthread.so.0
 #2  0x00007fdfe0dd0cdf in __GI___pthread_mutex_lock (mutex=mutex@entry=0x5631fde0e6c0)
    at ../nptl/pthread_mutex_lock.c:114
 #3  0x00005631fc0603a7 in qemu_mutex_lock_impl (mutex=0x5631fde0e6c0, file=0x5631fc23520f "util/async.c", line=511) at util/qemu-thread-posix.c:66
 #4  0x00005631fc05b558 in aio_co_enter (ctx=0x5631fde0e660, co=0x7fdfcc0c2b40) at util/async.c:493
 #5  0x00005631fc05b5ac in aio_co_wake (co=<optimized out>) at util/async.c:478
 #6  0x00005631fbfc51ad in qemu_laio_process_completion (laiocb=<optimized out>) at block/linux-aio.c:104
 #7  0x00005631fbfc523c in qemu_laio_process_completions (s=s@entry=0x7fdfc0297670)
    at block/linux-aio.c:222
 #8  0x00005631fbfc5499 in qemu_laio_process_completions_and_submit (s=0x7fdfc0297670)
    at block/linux-aio.c:237
 #9  0x00005631fc05d978 in aio_dispatch_handlers (ctx=ctx@entry=0x5631fde0e660) at util/aio-posix.c:406
 #10 0x00005631fc05e3ea in aio_poll (ctx=0x5631fde0e660, blocking=blocking@entry=true)
    at util/aio-posix.c:693
 #11 0x00005631fbd7ad96 in iothread_run (opaque=0x5631fde0e1c0) at iothread.c:64
 #12 0x00007fdfe0dcee25 in start_thread (arg=0x7fdfd8bd8700) at pthread_create.c:308
 #13 0x00007fdfe0afc34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

 - The Main Thread is also processing completions from the same
   AioContext, and crashes due to failed assertion at util/iov.c:78:

Thread 1 (Thread 0x7fdfeb5eac80 (LWP 36740)):
 #0  0x00007fdfe0a391f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
 #1  0x00007fdfe0a3a8e8 in __GI_abort () at abort.c:90
 #2  0x00007fdfe0a32266 in __assert_fail_base (fmt=0x7fdfe0b84e68 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x5631fc238ccb "offset == 0", file=file@entry=0x5631fc23698e "util/iov.c", line=line@entry=78, function=function@entry=0x5631fc236adc <__PRETTY_FUNCTION__.15220> "iov_memset")
    at assert.c:92
 #3  0x00007fdfe0a32312 in __GI___assert_fail (assertion=assertion@entry=0x5631fc238ccb "offset == 0", file=file@entry=0x5631fc23698e "util/iov.c", line=line@entry=78, function=function@entry=0x5631fc236adc <__PRETTY_FUNCTION__.15220> "iov_memset") at assert.c:101
 #4  0x00005631fc065287 in iov_memset (iov=<optimized out>, iov_cnt=<optimized out>, offset=<optimized out>, offset@entry=65536, fillc=fillc@entry=0, bytes=15515191315812405248) at util/iov.c:78
 #5  0x00005631fc065a63 in qemu_iovec_memset (qiov=<optimized out>, offset=offset@entry=65536, fillc=fillc@entry=0, bytes=<optimized out>) at util/iov.c:410
 #6  0x00005631fbfc5178 in qemu_laio_process_completion (laiocb=0x7fdd920df630) at block/linux-aio.c:88
 #7  0x00005631fbfc523c in qemu_laio_process_completions (s=s@entry=0x7fdfc0297670)
    at block/linux-aio.c:222
 #8  0x00005631fbfc5499 in qemu_laio_process_completions_and_submit (s=0x7fdfc0297670)
    at block/linux-aio.c:237
 #9  0x00005631fbfc54ed in qemu_laio_poll_cb (opaque=<optimized out>) at block/linux-aio.c:272
 #10 0x00005631fc05d85e in run_poll_handlers_once (ctx=ctx@entry=0x5631fde0e660) at util/aio-posix.c:497
 #11 0x00005631fc05e2ca in aio_poll (blocking=false, ctx=0x5631fde0e660) at util/aio-posix.c:574
 #12 0x00005631fc05e2ca in aio_poll (ctx=0x5631fde0e660, blocking=blocking@entry=false)
    at util/aio-posix.c:604
 #13 0x00005631fbfcb8a3 in bdrv_do_drained_begin (ignore_parent=<optimized out>, recursive=<optimized out>, bs=<optimized out>) at block/io.c:273
 #14 0x00005631fbfcb8a3 in bdrv_do_drained_begin (bs=0x5631fe8b6200, recursive=<optimized out>, parent=0x0, ignore_bds_parents=<optimized out>, poll=<optimized out>) at block/io.c:390
 #15 0x00005631fbfbcd2e in blk_drain (blk=0x5631fe83ac80) at block/block-backend.c:1590
 #16 0x00005631fbfbe138 in blk_remove_bs (blk=blk@entry=0x5631fe83ac80) at block/block-backend.c:774
 #17 0x00005631fbfbe3d6 in blk_unref (blk=0x5631fe83ac80) at block/block-backend.c:401
 #18 0x00005631fbfbe3d6 in blk_unref (blk=0x5631fe83ac80) at block/block-backend.c:449
 #19 0x00005631fbfc9a69 in commit_complete (job=0x5631fe8b94b0, opaque=0x7fdfcc1bb080)
    at block/commit.c:92
 #20 0x00005631fbf7d662 in job_defer_to_main_loop_bh (opaque=0x7fdfcc1b4560) at job.c:973
 #21 0x00005631fc05ad41 in aio_bh_poll (bh=0x7fdfcc01ad90) at util/async.c:90
 #22 0x00005631fc05ad41 in aio_bh_poll (ctx=ctx@entry=0x5631fddffdb0) at util/async.c:118
 #23 0x00005631fc05e210 in aio_dispatch (ctx=0x5631fddffdb0) at util/aio-posix.c:436
 #24 0x00005631fc05ac1e in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>) at util/async.c:261
 #25 0x00007fdfeaae44c9 in g_main_context_dispatch (context=0x5631fde00140) at gmain.c:3201
 #26 0x00007fdfeaae44c9 in g_main_context_dispatch (context=context@entry=0x5631fde00140) at gmain.c:3854
 #27 0x00005631fc05d503 in main_loop_wait () at util/main-loop.c:215
 #28 0x00005631fc05d503 in main_loop_wait (timeout=<optimized out>) at util/main-loop.c:238
 #29 0x00005631fc05d503 in main_loop_wait (nonblocking=nonblocking@entry=0) at util/main-loop.c:497
 #30 0x00005631fbd81412 in main_loop () at vl.c:1866
 #31 0x00005631fbc18ff3 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>)
    at vl.c:4647

 - A closer examination shows that s->io_q.in_flight appears to have
   gone backwards:

(gdb) frame 7
 #7  0x00005631fbfc523c in qemu_laio_process_completions (s=s@entry=0x7fdfc0297670)
    at block/linux-aio.c:222
222	            qemu_laio_process_completion(laiocb);
(gdb) p s
$2 = (LinuxAioState *) 0x7fdfc0297670
(gdb) p *s
$3 = {aio_context = 0x5631fde0e660, ctx = 0x7fdfeb43b000, e = {rfd = 33, wfd = 33}, io_q = {plugged = 0,
    in_queue = 0, in_flight = 4294967280, blocked = false, pending = {sqh_first = 0x0,
      sqh_last = 0x7fdfc0297698}}, completion_bh = 0x7fdfc0280ef0, event_idx = 21, event_max = 241}
(gdb) p/x s->io_q.in_flight
$4 = 0xfffffff0

Signed-off-by: Sergio Lopez <slp@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
amboar pushed a commit that referenced this issue Oct 23, 2018
@kevmw
test-replication: Lock AioContext around blk_unref()
90042c6 
Recently, the test case has started failing because some job related
functions want to drop the AioContext lock even though it hasn't been
taken:

    (gdb) bt
    #0  0x00007f51c067c9fb in raise () from /lib64/libc.so.6
    #1  0x00007f51c067e77d in abort () from /lib64/libc.so.6
    #2  0x0000558c9d5dde7b in error_exit (err=<optimized out>, msg=msg@entry=0x558c9d6fe120 <__func__.18373> "qemu_mutex_unlock_impl") at util/qemu-thread-posix.c:36
    #3  0x0000558c9d6b5263 in qemu_mutex_unlock_impl (mutex=mutex@entry=0x558c9f3999a0, file=file@entry=0x558c9d6fd36f "util/async.c", line=line@entry=516) at util/qemu-thread-posix.c:96
    #4  0x0000558c9d6b0565 in aio_context_release (ctx=ctx@entry=0x558c9f399940) at util/async.c:516
    #5  0x0000558c9d5eb3da in job_completed_txn_abort (job=0x558c9f68e640) at job.c:738
    #6  0x0000558c9d5eb227 in job_finish_sync (job=0x558c9f68e640, finish=finish@entry=0x558c9d5eb8d0 <job_cancel_err>, errp=errp@entry=0x0) at job.c:986
    #7  0x0000558c9d5eb8ee in job_cancel_sync (job=<optimized out>) at job.c:941
    #8  0x0000558c9d64d853 in replication_close (bs=<optimized out>) at block/replication.c:148
    #9  0x0000558c9d5e5c9f in bdrv_close (bs=0x558c9f41b020) at block.c:3420
    #10 bdrv_delete (bs=0x558c9f41b020) at block.c:3629
    #11 bdrv_unref (bs=0x558c9f41b020) at block.c:4685
    #12 0x0000558c9d62a3f3 in blk_remove_bs (blk=blk@entry=0x558c9f42a7c0) at block/block-backend.c:783
    #13 0x0000558c9d62a667 in blk_delete (blk=0x558c9f42a7c0) at block/block-backend.c:402
    #14 blk_unref (blk=0x558c9f42a7c0) at block/block-backend.c:457
    #15 0x0000558c9d5dfcea in test_secondary_stop () at tests/test-replication.c:478
    #16 0x00007f51c1f13178 in g_test_run_suite_internal () from /lib64/libglib-2.0.so.0
    #17 0x00007f51c1f1337b in g_test_run_suite_internal () from /lib64/libglib-2.0.so.0
    #18 0x00007f51c1f1337b in g_test_run_suite_internal () from /lib64/libglib-2.0.so.0
    #19 0x00007f51c1f13552 in g_test_run_suite () from /lib64/libglib-2.0.so.0
    #20 0x00007f51c1f13571 in g_test_run () from /lib64/libglib-2.0.so.0
    #21 0x0000558c9d5de31f in main (argc=<optimized out>, argv=<optimized out>) at tests/test-replication.c:581

It is yet unclear whether this should really be considered a bug in the
test case or whether blk_unref() should work for callers that haven't
taken the AioContext lock, but in order to fix the build tests quickly,
just take the AioContext lock around blk_unref().

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
amboar pushed a commit that referenced this issue Oct 23, 2018
@elmarco
tests/check-qjson: fix a leak
8b8fa99 
Spotted by ASAN:
=================================================================
==11893==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 1120 byte(s) in 28 object(s) allocated from:
    #0 0x7fd0515b0c48 in malloc (/lib64/libasan.so.5+0xeec48)
    #1 0x7fd050ffa3c5 in g_malloc (/lib64/libglib-2.0.so.0+0x523c5)
    #2 0x559e708b56a4 in qstring_from_str /home/elmarco/src/qq/qobject/qstring.c:66
    #3 0x559e708b4fe0 in qstring_new /home/elmarco/src/qq/qobject/qstring.c:23
    #4 0x559e708bda7d in parse_string /home/elmarco/src/qq/qobject/json-parser.c:143
    #5 0x559e708c1009 in parse_literal /home/elmarco/src/qq/qobject/json-parser.c:484
    #6 0x559e708c1627 in parse_value /home/elmarco/src/qq/qobject/json-parser.c:547
    #7 0x559e708c1c67 in json_parser_parse /home/elmarco/src/qq/qobject/json-parser.c:573
    #8 0x559e708bc0ff in json_message_process_token /home/elmarco/src/qq/qobject/json-streamer.c:92
    #9 0x559e708d1655 in json_lexer_feed_char /home/elmarco/src/qq/qobject/json-lexer.c:292
    #10 0x559e708d1fe1 in json_lexer_feed /home/elmarco/src/qq/qobject/json-lexer.c:339
    #11 0x559e708bc856 in json_message_parser_feed /home/elmarco/src/qq/qobject/json-streamer.c:121
    #12 0x559e708b8b4b in qobject_from_jsonv /home/elmarco/src/qq/qobject/qjson.c:69
    #13 0x559e708b8d02 in qobject_from_json /home/elmarco/src/qq/qobject/qjson.c:83
    #14 0x559e708a74ae in from_json_str /home/elmarco/src/qq/tests/check-qjson.c:30
    #15 0x559e708a9f83 in utf8_string /home/elmarco/src/qq/tests/check-qjson.c:781
    #16 0x7fd05101bc49 in test_case_run gtestutils.c:2255
    #17 0x7fd05101bc49 in g_test_run_suite_internal gtestutils.c:2339

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180901211917.10372-1-marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
amboar pushed a commit that referenced this issue Jan 14, 2019
@kevmw
mirror: fix dead-lock
d12ade5 
Let start from the beginning:

Commit b9e413d (in 2.9)
"block: explicitly acquire aiocontext in aio callbacks that need it"
added pairs of aio_context_acquire/release to mirror_write_complete and
mirror_read_complete, when they were aio callbacks for blk_aio_* calls.

Then, commit 2e1990b (in 3.0) "block/mirror: Convert to coroutines"
dropped these blk_aio_* calls, than mirror_write_complete and
mirror_read_complete are not callbacks more, and don't need additional
aiocontext acquiring. Furthermore, mirror_read_complete calls
blk_co_pwritev inside these pair of aio_context_acquire/release, which
leads to the following dead-lock with mirror:

 (gdb) info thr
   Id   Target Id         Frame
   3    Thread (LWP 145412) "qemu-system-x86" syscall ()
   2    Thread (LWP 145416) "qemu-system-x86" __lll_lock_wait ()
 * 1    Thread (LWP 145411) "qemu-system-x86" __lll_lock_wait ()

 (gdb) bt
 #0  __lll_lock_wait ()
 #1  _L_lock_812 ()
 #2  __GI___pthread_mutex_lock
 #3  qemu_mutex_lock_impl (mutex=0x561032dce420 <qemu_global_mutex>,
     file=0x5610327d8654 "util/main-loop.c", line=236) at
     util/qemu-thread-posix.c:66
 #4  qemu_mutex_lock_iothread_impl
 #5  os_host_main_loop_wait (timeout=480116000) at util/main-loop.c:236
 #6  main_loop_wait (nonblocking=0) at util/main-loop.c:497
 #7  main_loop () at vl.c:1892
 #8  main

Printing contents of qemu_global_mutex, I see that "__owner = 145416",
so, thr1 is main loop, and now it wants BQL, which is owned by thr2.

 (gdb) thr 2
 (gdb) bt
 #0  __lll_lock_wait ()
 #1  _L_lock_870 ()
 #2  __GI___pthread_mutex_lock
 #3  qemu_mutex_lock_impl (mutex=0x561034d25dc0, ...
 #4  aio_context_acquire (ctx=0x561034d25d60)
 #5  dma_blk_cb
 #6  dma_blk_io
 #7  dma_blk_read
 #8  ide_dma_cb
 #9  bmdma_cmd_writeb
 #10 bmdma_write
 #11 memory_region_write_accessor
 #12 access_with_adjusted_size
 #15 flatview_write
 #16 address_space_write
 #17 address_space_rw
 #18 kvm_handle_io
 #19 kvm_cpu_exec
 #20 qemu_kvm_cpu_thread_fn
 #21 qemu_thread_start
 #22 start_thread
 #23 clone ()

Printing mutex in fr 2, I see "__owner = 145411", so thr2 wants aio
context mutex, which is owned by thr1. Classic dead-lock.

Then, let's check that aio context is hold by mirror coroutine: just
print coroutine stack of first tracked request in mirror job target:

 (gdb) [...]
 (gdb) qemu coroutine 0x561035dd0860
 #0  qemu_coroutine_switch
 #1  qemu_coroutine_yield
 #2  qemu_co_mutex_lock_slowpath
 #3  qemu_co_mutex_lock
 #4  qcow2_co_pwritev
 #5  bdrv_driver_pwritev
 #6  bdrv_aligned_pwritev
 #7  bdrv_co_pwritev
 #8  blk_co_pwritev
 #9  mirror_read_complete () at block/mirror.c:232
 #10 mirror_co_read () at block/mirror.c:370
 #11 coroutine_trampoline
 #12 __start_context

Yes it is mirror_read_complete calling blk_co_pwritev after acquiring
aio context.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@amboar @legoater