JSC Crash in operationLinkDirectCall

[aazamansari]

#0 WTFCrash libWPEWebKit.so.0.0.20161117 Source/WTF/wtf/Assertions.cpp:324 (0x8)
#1 operationLinkDirectCall libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/jit/JITOperations.cpp:953 (0x4)
#2 @0x5ab86150
#3 deref libWPEWebKit.so.0.0.20161117 Source/WTF/wtf/ThreadSafeRefCounted.h:36 (0x8)
#4 _fini libWPEWebKit.so.0.0.20161117
#5 execute libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/jit/JITCode.cpp:81 (0xc)
#6 executeCall libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/interpreter/Interpreter.cpp:952 (0x10)
#7 call libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/runtime/CallData.cpp:39 (0x0)
#8 boundThisNoArgsFunctionCall libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/runtime/JSBoundFunction.cpp:54 (0x14)
#9 libWPEWebKit.so.0.0.20161117@0x69858c libWPEWebKit.so.0.0.20161117
#10 executeCall libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/interpreter/Interpreter.cpp:955 (0xc)
#11 call libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/runtime/CallData.cpp:39 (0x0)
#12 profiledCall libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/runtime/CallData.cpp:59 (0x0)
#13 JSObjectCallAsFunction libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/API/JSObjectRef.cpp:541 (0x38)
#14 libCUSTOMPlayer.so.0.0.0@0x7be70 libCUSTOMPlayer.so.0.0.0
#15 libCUSTOMPlayer.so.0.0.0@0x6bc58 libCUSTOMPlayer.so.0.0.0
#16 libCUSTOMPlayer.so.0.0.0@0x740978 libCUSTOMPlayer.so.0.0.0
#17 libCUSTOMPlayer.so.0.0.0@0x6cf40 libCUSTOMPlayer.so.0.0.0
#18 libCUSTOMPlayer.so.0.0.0@0x72768 libCUSTOMPlayer.so.0.0.0
#19 libCUSTOMPlayer.so.0.0.0@0x1f8228 libCUSTOMPlayer.so.0.0.0
#20 libCUSTOMPlayer.so.0.0.0@0x1f8400 libCUSTOMPlayer.so.0.0.0
#21 libCUSTOMPlayer.so.0.0.0@0x1f8750 libCUSTOMPlayer.so.0.0.0
#22 WPECallbackManager::eventPosted()::{lambda(void*)#1}::_FUN(void*) libComcastInjectedBundle.so
#23 g_timeout_dispatch libglib-2.0.so.0.4800.1 glib-2.0/1_2.48.1-r0/glib-2.48.1/glib/gmain.c:4577 (0x4)
#24 g_main_context_dispatch libglib-2.0.so.0.4800.1 glib-2.0/1_2.48.1-r0/glib-2.48.1/glib/gmain.c:3154 (0x0)
#25 g_main_context_iterate libglib-2.0.so.0.4800.1 glib-2.0/1_2.48.1-r0/glib-2.48.1/glib/gmain.c:3840 (0x4)
#26 g_main_loop_run libglib-2.0.so.0.4800.1 glib-2.0/1_2.48.1-r0/glib-2.48.1/glib/gmain.c:4034 (0xc)
#27 run libWPEWebKit.so.0.0.20161117 Source/WTF/wtf/glib/RunLoopGLib.cpp:97 (0x4)
#28 ChildProcessMain<WebKit::WebProcess, WebKit::WebProcessMain> libWPEWebKit.so.0.0.20161117 Source/WebKit2/Shared/unix/ChildProcessMain.h:61 (0x4)
#29 main WPEWebProcess Source/WebKit2/WebProcess/EntryPoint/unix/WebProcessMain.cpp:52 (0x8)
#30 libc-2.19.so@0x194a4 libc-2.19.so
#31 libgcc_s.so.1@0x2e18 libgcc_s.so.1
#32 ld-2.19.so@0xd80 ld-2.19.so

[guijemont]

Thank you for the stack trace. Can you provide more information on the conditions in which this happened? What hardware, what version of WPE, how you compiled it, how you were running it and what code/webpage you were running at the time? Anything that could help us reproduce the bug would be helpful.
Also, it looks like you hit an assertion: did any message print regarding this assertion? What was it?

Thanks!

[aazamansari]

We use 5c0c3fd from stable branch. Below are the settings that we used:

export WPE_DISK_CACHE_SIZE=10m
export WPE_RAM_SIZE=192m
export WPE_POLL_MAX_MEMORY='WPEWebProcess:150M,*Process:50M'

This happens on a mips platform. I compile the WPE with -O2 option.

We don't see any asserting logs.

[guijemont]

What is the scenario to reproduce? What webpage, what were you doing?

[aazamansari]

The issue happens when playing live video. It is internal link for linear video(live video) which is not accessible from outside.

[aazamansari]

I see crash in same call without the custom player library that we have.

#0 WTFCrash libWPEWebKit.so.0.0.20161117 Source/WTF/wtf/Assertions.cpp:324 (0x8)
#1 operationLinkDirectCall libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/jit/JITOperations.cpp:953 (0x4)
#2 @0x587fef4c
#3 jsEventType libWPEWebKit.so.0.0.20161117 Source/WebCore/bindings/js/JSDOMConvert.h:110 (0x4)
#4 @0xfffffff3
#5 isCurrentThreadBusy libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/heap/Heap.cpp:1977 (0x0)
#6 executeCall libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/interpreter/Interpreter.cpp:952 (0x10)
#7 call libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/runtime/CallData.cpp:39 (0x0)
#8 boundThisNoArgsFunctionCall libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/runtime/JSBoundFunction.cpp:54 (0x28)
#9 libWPEWebKit.so.0.0.20161117@0x8fa39c libWPEWebKit.so.0.0.20161117
#10 _fini libWPEWebKit.so.0.0.20161117
#11 libstdc++.so.6.0.20@0x4e8fc libstdc++.so.6.0.20
#12 profiledCall libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/runtime/CallData.cpp:65 (0x30)
#13 _fini libWPEWebKit.so.0.0.20161117
#14 _fini libWPEWebKit.so.0.0.20161117
#15 create libWPEWebKit.so.0.0.20161117 Source/WebCore/platform/graphics/texmap/coordinated/CoordinatedSurface.cpp:43 (0x0)
#16 @0xfffffff8
#17 fireEventListeners libWPEWebKit.so.0.0.20161117 Source/WebCore/dom/EventTarget.cpp:200 (0x1c)
#18 handleLocalEvents libWPEWebKit.so.0.0.20161117 Source/WebCore/dom/EventContext.cpp:54 (0x8)
#19 dispatchEvent libWPEWebKit.so.0.0.20161117 Source/WebCore/dom/EventDispatcher.cpp:85 (0x8)
#20 dispatchOneEvent libWPEWebKit.so.0.0.20161117 Source/WebCore/dom/GenericEventQueue.cpp:70 (0x8)
#21 dispatchOneTask libWPEWebKit.so.0.0.20161117 Source/WTF/wtf/Function.h:50 (0x8)
#22 sharedTimerFired libWPEWebKit.so.0.0.20161117 Source/WebCore/platform/GenericTaskQueue.cpp:66 (0x4)
#23 sharedTimerFiredInternal libWPEWebKit.so.0.0.20161117 Source/WebCore/platform/ThreadTimers.cpp:121 (0x0)
#24 _FUN libWPEWebKit.so.0.0.20161117 Source/WTF/wtf/glib/RunLoopGLib.cpp:165 (0x0)
#25 g_main_context_dispatch libglib-2.0.so.0.4800.1 glib-2.48.1/glib/gmain.c:3154 (0x0)
#26 g_main_context_iterate libglib-2.0.so.0.4800.1 glib-2.48.1/glib/gmain.c:3840 (0x4)
#27 g_main_loop_run libglib-2.0.so.0.4800.1 glib-2.48.1/glib/gmain.c:4034 (0x8)
#28 run libWPEWebKit.so.0.0.20161117 Source/WTF/wtf/glib/RunLoopGLib.cpp:97 (0x4)
#29 ChildProcessMain<WebKit::WebProcess, WebKit::WebProcessMain> libWPEWebKit.so.0.0.20161117 Source/WebKit2/Shared/unix/ChildProcessMain.h:61 (0x4)
#30 main WPEWebProcess Source/WebKit2/WebProcess/EntryPoint/unix/WebProcessMain.cpp:52 (0x8)
#31 __libc_start_main libc-2.19.so eglibc-2.19/libc/csu/libc-start.c:285 (0xc)
#32 __start WPEWebProcess

[albertd]

@aazamansari is this reproducible on the wpe-2017 branch?

[guijemont]

@aazamansari are you able to reproduce with a debug build, or with an -O0 (or maybe -O1) build with debug symbols? With the build+core files that you provided, I haven't been able to see the values of local variables, which makes this very hard to investigate.

[Gavriliuk]

The issue is still being reprodused in the release versions, hundreds crashes every day:
https://crashportal.stb.r53.xcal.tv/minidumpReport/list?signature=operationLinkDirectCall

[wouterlucas]

@Gavriliuk and we've been trying to reproduce this on a vanilla Broadcom STB with WPE for hours by different people, different devices without success using refsw 16.x and 17.x.

Without having a reliable way for us to reproduce we can't really help you.

[Gavriliuk]

@wouterlucas
Could I help you by providing stacktraces, logs etc. from the crashportal?

[guijemont]

@Gavriliuk a core file from a debug build (and associated build files and source code/revision) could be helpful.

[Gavriliuk]

Hi @guijemont, unfortunately this doesn't look possible

1. webkit produces a lot of compilation errors when compiling in the debug mode
2. even if I fix or hide these errors locally I can't reproduce the bug with my local build

We observe these crashes in the field, I can attach some minidump reports with stacktraces from the recent builds:
https://crashportal.stb.r53.xcal.tv/minidumpReport/show/118427400
https://crashportal.stb.r53.xcal.tv/minidumpReport/show/118472731
https://crashportal.stb.r53.xcal.tv/minidumpReport/show/118587423

118427400.txt
118472731.txt
118587423.txt