JSC Crash in operationLinkDirectCall

[aazamansari]

#0 WTFCrash libWPEWebKit.so.0.0.20161117 Source/WTF/wtf/Assertions.cpp:324 (0x8)
#1 operationLinkDirectCall libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/jit/JITOperations.cpp:953 (0x4)
#2 @0x5ab86150
#3 deref libWPEWebKit.so.0.0.20161117 Source/WTF/wtf/ThreadSafeRefCounted.h:36 (0x8)
#4 _fini libWPEWebKit.so.0.0.20161117
#5 execute libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/jit/JITCode.cpp:81 (0xc)
#6 executeCall libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/interpreter/Interpreter.cpp:952 (0x10)
#7 call libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/runtime/CallData.cpp:39 (0x0)
#8 boundThisNoArgsFunctionCall libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/runtime/JSBoundFunction.cpp:54 (0x14)
#9 libWPEWebKit.so.0.0.20161117@0x69858c libWPEWebKit.so.0.0.20161117
#10 executeCall libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/interpreter/Interpreter.cpp:955 (0xc)
#11 call libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/runtime/CallData.cpp:39 (0x0)
#12 profiledCall libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/runtime/CallData.cpp:59 (0x0)
#13 JSObjectCallAsFunction libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/API/JSObjectRef.cpp:541 (0x38)
#14 libCUSTOMPlayer.so.0.0.0@0x7be70 libCUSTOMPlayer.so.0.0.0
#15 libCUSTOMPlayer.so.0.0.0@0x6bc58 libCUSTOMPlayer.so.0.0.0
#16 libCUSTOMPlayer.so.0.0.0@0x740978 libCUSTOMPlayer.so.0.0.0
#17 libCUSTOMPlayer.so.0.0.0@0x6cf40 libCUSTOMPlayer.so.0.0.0
#18 libCUSTOMPlayer.so.0.0.0@0x72768 libCUSTOMPlayer.so.0.0.0
#19 libCUSTOMPlayer.so.0.0.0@0x1f8228 libCUSTOMPlayer.so.0.0.0
#20 libCUSTOMPlayer.so.0.0.0@0x1f8400 libCUSTOMPlayer.so.0.0.0
#21 libCUSTOMPlayer.so.0.0.0@0x1f8750 libCUSTOMPlayer.so.0.0.0
#22 WPECallbackManager::eventPosted()::{lambda(void*)#1}::_FUN(void*) libComcastInjectedBundle.so
#23 g_timeout_dispatch libglib-2.0.so.0.4800.1 glib-2.0/1_2.48.1-r0/glib-2.48.1/glib/gmain.c:4577 (0x4)
#24 g_main_context_dispatch libglib-2.0.so.0.4800.1 glib-2.0/1_2.48.1-r0/glib-2.48.1/glib/gmain.c:3154 (0x0)
#25 g_main_context_iterate libglib-2.0.so.0.4800.1 glib-2.0/1_2.48.1-r0/glib-2.48.1/glib/gmain.c:3840 (0x4)
#26 g_main_loop_run libglib-2.0.so.0.4800.1 glib-2.0/1_2.48.1-r0/glib-2.48.1/glib/gmain.c:4034 (0xc)
#27 run libWPEWebKit.so.0.0.20161117 Source/WTF/wtf/glib/RunLoopGLib.cpp:97 (0x4)
#28 ChildProcessMain<WebKit::WebProcess, WebKit::WebProcessMain> libWPEWebKit.so.0.0.20161117 Source/WebKit2/Shared/unix/ChildProcessMain.h:61 (0x4)
#29 main WPEWebProcess Source/WebKit2/WebProcess/EntryPoint/unix/WebProcessMain.cpp:52 (0x8)
#30 libc-2.19.so@0x194a4 libc-2.19.so
#31 libgcc_s.so.1@0x2e18 libgcc_s.so.1
#32 ld-2.19.so@0xd80 ld-2.19.so

[guijemont]

Thank you for the stack trace. Can you provide more information on the conditions in which this happened? What hardware, what version of WPE, how you compiled it, how you were running it and what code/webpage you were running at the time? Anything that could help us reproduce the bug would be helpful.
Also, it looks like you hit an assertion: did any message print regarding this assertion? What was it?

Thanks!

[aazamansari]

We use 5c0c3fd from stable branch. Below are the settings that we used:

export WPE_DISK_CACHE_SIZE=10m
export WPE_RAM_SIZE=192m
export WPE_POLL_MAX_MEMORY='WPEWebProcess:150M,*Process:50M'

This happens on a mips platform. I compile the WPE with -O2 option.

We don't see any asserting logs.

[guijemont]

What is the scenario to reproduce? What webpage, what were you doing?

[aazamansari]

The issue happens when playing live video. It is internal link for linear video(live video) which is not accessible from outside.

[aazamansari]

I see crash in same call without the custom player library that we have.

#0 WTFCrash libWPEWebKit.so.0.0.20161117 Source/WTF/wtf/Assertions.cpp:324 (0x8)
#1 operationLinkDirectCall libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/jit/JITOperations.cpp:953 (0x4)
#2 @0x587fef4c
#3 jsEventType libWPEWebKit.so.0.0.20161117 Source/WebCore/bindings/js/JSDOMConvert.h:110 (0x4)
#4 @0xfffffff3
#5 isCurrentThreadBusy libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/heap/Heap.cpp:1977 (0x0)
#6 executeCall libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/interpreter/Interpreter.cpp:952 (0x10)
#7 call libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/runtime/CallData.cpp:39 (0x0)
#8 boundThisNoArgsFunctionCall libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/runtime/JSBoundFunction.cpp:54 (0x28)
#9 libWPEWebKit.so.0.0.20161117@0x8fa39c libWPEWebKit.so.0.0.20161117
#10 _fini libWPEWebKit.so.0.0.20161117
#11 libstdc++.so.6.0.20@0x4e8fc libstdc++.so.6.0.20
#12 profiledCall libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/runtime/CallData.cpp:65 (0x30)
#13 _fini libWPEWebKit.so.0.0.20161117
#14 _fini libWPEWebKit.so.0.0.20161117
#15 create libWPEWebKit.so.0.0.20161117 Source/WebCore/platform/graphics/texmap/coordinated/CoordinatedSurface.cpp:43 (0x0)
#16 @0xfffffff8
#17 fireEventListeners libWPEWebKit.so.0.0.20161117 Source/WebCore/dom/EventTarget.cpp:200 (0x1c)
#18 handleLocalEvents libWPEWebKit.so.0.0.20161117 Source/WebCore/dom/EventContext.cpp:54 (0x8)
#19 dispatchEvent libWPEWebKit.so.0.0.20161117 Source/WebCore/dom/EventDispatcher.cpp:85 (0x8)
#20 dispatchOneEvent libWPEWebKit.so.0.0.20161117 Source/WebCore/dom/GenericEventQueue.cpp:70 (0x8)
#21 dispatchOneTask libWPEWebKit.so.0.0.20161117 Source/WTF/wtf/Function.h:50 (0x8)
#22 sharedTimerFired libWPEWebKit.so.0.0.20161117 Source/WebCore/platform/GenericTaskQueue.cpp:66 (0x4)
#23 sharedTimerFiredInternal libWPEWebKit.so.0.0.20161117 Source/WebCore/platform/ThreadTimers.cpp:121 (0x0)
#24 _FUN libWPEWebKit.so.0.0.20161117 Source/WTF/wtf/glib/RunLoopGLib.cpp:165 (0x0)
#25 g_main_context_dispatch libglib-2.0.so.0.4800.1 glib-2.48.1/glib/gmain.c:3154 (0x0)
#26 g_main_context_iterate libglib-2.0.so.0.4800.1 glib-2.48.1/glib/gmain.c:3840 (0x4)
#27 g_main_loop_run libglib-2.0.so.0.4800.1 glib-2.48.1/glib/gmain.c:4034 (0x8)
#28 run libWPEWebKit.so.0.0.20161117 Source/WTF/wtf/glib/RunLoopGLib.cpp:97 (0x4)
#29 ChildProcessMain<WebKit::WebProcess, WebKit::WebProcessMain> libWPEWebKit.so.0.0.20161117 Source/WebKit2/Shared/unix/ChildProcessMain.h:61 (0x4)
#30 main WPEWebProcess Source/WebKit2/WebProcess/EntryPoint/unix/WebProcessMain.cpp:52 (0x8)
#31 __libc_start_main libc-2.19.so eglibc-2.19/libc/csu/libc-start.c:285 (0xc)
#32 __start WPEWebProcess

[albertd]

@aazamansari is this reproducible on the wpe-2017 branch?

[guijemont]

@aazamansari are you able to reproduce with a debug build, or with an -O0 (or maybe -O1) build with debug symbols? With the build+core files that you provided, I haven't been able to see the values of local variables, which makes this very hard to investigate.

[Gavriliuk]

The issue is still being reprodused in the release versions, hundreds crashes every day:
https://crashportal.stb.r53.xcal.tv/minidumpReport/list?signature=operationLinkDirectCall

[wouterlucas]

@Gavriliuk and we've been trying to reproduce this on a vanilla Broadcom STB with WPE for hours by different people, different devices without success using refsw 16.x and 17.x.

Without having a reliable way for us to reproduce we can't really help you.

[Gavriliuk]

@wouterlucas
Could I help you by providing stacktraces, logs etc. from the crashportal?

[guijemont]

@Gavriliuk a core file from a debug build (and associated build files and source code/revision) could be helpful.

[Gavriliuk]

Hi @guijemont, unfortunately this doesn't look possible

1. webkit produces a lot of compilation errors when compiling in the debug mode
2. even if I fix or hide these errors locally I can't reproduce the bug with my local build

We observe these crashes in the field, I can attach some minidump reports with stacktraces from the recent builds:
https://crashportal.stb.r53.xcal.tv/minidumpReport/show/118427400
https://crashportal.stb.r53.xcal.tv/minidumpReport/show/118472731
https://crashportal.stb.r53.xcal.tv/minidumpReport/show/118587423

118427400.txt
118472731.txt
118587423.txt

[ncuillery]

At Youi we have the exact same error (WTFCrash caused by operationLinkDirectCall) on our apps.

We use the 2.19.1 tarball (https://webkitgtk.org/releases/webkitgtk-2.19.1.tar.xz) that we build ourself.

We have been able to narrow down the issue to the following:
It only happens on:

• Galaxy S7 and S7 Edge
• Android 7.0 only (doesn't crash in Android 8.0)
• Arch arm64-v8a only (the same app build using armeabi-v7a and deployed on a device mentioned above won't crash)

@Gavriliuk Does it matches the reports you saw in your dashboard? (it is not public)

[Gavriliuk]

To @ncuillery
Thank you Nicolas
We observe this kind of crashes on TV boxes
(OS Linux, various platforms and architectures)
I hope your comment will be helpful for Metro team

[albertd]

@Gavriliuk do we have a public app available to validate?

[Gavriliuk]

@albertd, I actually can't know what do you have :-)
But I guess we ourself don't have a such app

This crash happens on our TV STBs - not very rarely, but randomly, no steps to reproduce
E.g., 15805 crashes were registered from testing boxes during the recent 30 days
I'd attached some example stacktraces above

Thanks
Alex

[Gavriliuk]

It seems like the crash is being happened during the XMLHttpRequest sends the asynchronous request or processes its response

[nrajan002c]

Hi @wouterlucas @albertd ,

Again we started analyzing this issue since this crash frequency is higher in field.
Originally the SIGUSR1 is being sent inside operationLinkDirectCall api.
(i.e. RELEASE_ASSERT(callLinkInfo->isDirect());
code: https://github.com/WebPlatformForEmbedded/WPEWebKit/blob/wpe-20170728/Source/JavaScriptCore/jit/JITOperations.cpp#L999)

operationLinkDirectCall will be called only when the calltype is Direct*.
But before this api is executed, someone else is calling setCallType and changed the callLinkInfo object's callType. So that assert is happened.

Can we modify the code like this to avoid the crash,
void JIT_OPERATION operationLinkDirectCall(ExecState* exec, CallLinkInfo* callLinkInfo, JSFunction* callee) { ..... //RELEASE_ASSERT(callLinkInfo->isDirect()); if(!callLinkInfo->isDirect()) return;

Could you please tell the consequence of this change.

[guijemont]

Can we modify the code like this to avoid the crash,
void JIT_OPERATION operationLinkDirectCall(ExecState* exec, CallLinkInfo* callLinkInfo, JSFunction* callee) { ..... //RELEASE_ASSERT(callLinkInfo->isDirect()); if(!callLinkInfo->isDirect()) return;

Could you please tell the consequence of this change.
I would need to check more in detail, but this assert is very likely here for a reason, and I would expect that removing it would just move the bug somewhere else.

Interestingly, this week I've been playing with running jsc's tests with qemu, and I found a crash that looks a lot like this one (which wouldn't reproduce on the mips hardware I have). I'll keep on investigating that since now I have a way to reproduce, and I will update this bug once I understand more of what's happening.

[nrajan002c]

Thanks @guijemont for the update.
We are seeing this problem in mips platform also. Since this occurs randomly in field, We don't have the concrete reproducing steps. Please let me know if you want to try anything in STB.

[Gavriliuk]

This problem is often observed when load average is too high

It seems the operation block is already destroyed when its call is executed

It leads to SIGSEGV in many cases, and sometimes SIGUSR1 from WTFCrash because of RELEASE_ASSERT on the line 999:

void JIT_OPERATION operationLinkDirectCall(ExecState* exec, CallLinkInfo* callLinkInfo, JSFunction* callee)
{
    VM* vm = &exec->vm();
    auto throwScope = DECLARE_THROW_SCOPE(*vm);

    CodeSpecializationKind kind = callLinkInfo->specializationKind();
    NativeCallFrameTracer tracer(vm, exec);

/// Line 999:
    RELEASE_ASSERT(callLinkInfo->isDirect());

[Balajims88]

@wouterlucas @woutermeek I am believing this #650 fix this issue.
so could you please review and merge this #650

#650

CC @aazamansari

[guijemont]

#650 has been merged in 80c2847. Please confirm whether this solves the issue, but from the description of the fix on the upstream bug, I expect it to fix this.

[guijemont]

@Balajims88 @aazamansari have you been able to confirm that the issue is fixed?

[kkanag314]

Hi @guijemont

The JSC crash in operationLinkDirectCall is still not resolved.

[ 10 Thread 0 (crashed)
11 0 libWPEWebKit.so.0.0.20170728!WTFCrash [Assertions.cpp : 270 + 0x8]
12 1 libWPEWebKit.so.0.0.20170728!operationLinkDirectCall [JITOperations.cpp : 999 + 0x4]
13 2 0x71107d54]

The crash has been identified in the boxes like SamsungXG2V2. Exact steps to reproduce is not available as they are occurring random and intermittently.

[Balajims88]

Hi @guijemont ,
Issue is still reproducible. so the above fix is not fixing properly.
so could you please take a look?

[guijemont]

Hi @guijemont ,
Issue is still reproducible. so the above fix is not fixing properly.
so could you please take a look?

Hi @Balajims88! Are you able to provide a way for us to reproduce the issue? This would be the one thing that would allow us to figure things out relatively quickly.

[Balajims88]

@guijemont Sorry, I have a no steps to reproduce this issue.

[pkumbh631]

Hi @guijemont,
Please find the below steps to reproduce the issue,
Step 1# Push 4.2p12s1 to pace XG2V2.
Step 2# Play DASH IPVOD and Peacock Playbacks.
Step 3# Monitor for webprocess crash. Signature : operationLinkDirectCall.

Thank you...

[guijemont]

Hi @guijemont,
Please find the below steps to reproduce the issue,
Step 1# Push 4.2p12s1 to pace XG2V2.

Can you tell me what revision of https://github.com/WebPlatformForEmbedded/WPEWebKit matches "4.2p12s1" ?

I don't have access to "XG2V2", but I can try to reproduce on another device of the same architecture. Is that an arm or mips device?

Step 2# Play DASH IPVOD and Peacock Playbacks.

How can I access that?

Step 3# Monitor for webprocess crash. Signature : operationLinkDirectCall.

Does the crash happen every time? And how quickly does it happen?

[pkumbh631]

The crash usually happens upon launching a web-app. Three most common crash URLs are:

https://xfinity.ccast.api.amazonvideo.com/lrc-vending/html5/index.html?deviceTyp
https://www.youtube.com/tv?launch=menu&lmt=0&us_privacy=1-N-
https://comcast.play.hbomax.com/?launchpoint=&query=&assetId=&assetType=&entityI

The release assert gets fired that checks if it is a Direct Call :

WPEWebKit/Source/JavaScriptCore/jit/JITOperations.cpp

Line 1041 in 941e508

RELEASE_ASSERT(!callLinkInfo->isDirect());

`void JIT_OPERATION operationLinkDirectCall(ExecState* exec, CallLinkInfo* callLinkInfo, JSFunction* callee)
{
VM* vm = &exec->vm();
auto throwScope = DECLARE_THROW_SCOPE(*vm);

CodeSpecializationKind kind = callLinkInfo->specializationKind();
NativeCallFrameTracer tracer(vm, exec);

RELEASE_ASSERT(callLinkInfo->isDirect()) // <== HERE `

It looks like operationLinkDirectCall is getting called for an unexpected CallLinkInfo call type, possibly as a result of some kind of data corruption.
This assert gets fired on MIPS-based devices only, so it can be a MIPS specific error or something related to limited memory of those device types.

[glibnovodran]

@guijemont ,
The crash seems to happen on both wpe-20170728 and wpe-2.22
More crashes tend to happen to wpe-20170728, but this is probably because more devices run the builds based upon it.
The problem seems to be MIPS-specific, but can be a consequence of limited memory amount (a GC-related issue?)
Presently wpe-20170728 crashes and wpe-2.22 crashes triggered by the same assertion.

wpe-20170728

1 WTFCrash
‎/usr/src/debug/wpe-webkit/0.4.4+gitAUTOINC+5f899bc2e0-r0/git/Source/WTF/wtf/Assertions.cpp:270
2 operationLinkDirectCall
‎/usr/src/debug/wpe-webkit/0.4.4+gitAUTOINC+5f899bc2e0-r0/git/Source/JavaScriptCore/jit/JITOperations.cpp:999
...

WPEWebKit/Source/JavaScriptCore/jit/JITOperations.cpp

Line 999 in a25b9a7

RELEASE_ASSERT(callLinkInfo->isDirect());

wpe-2.22
1 WTFCrash
‎/usr/src/debug/wpe-webkit/2.22+gitAUTOINC+686cd2f7df-r0/git/Source/WTF/wtf/Assertions.cpp:255
2 operationLinkDirectCall
‎/usr/src/debug/wpe-webkit/2.22+gitAUTOINC+686cd2f7df-r0/git/Source/JavaScriptCore/jit/JITOperations.cpp:1112
...

2.22+gitAUTOINC+686cd2f7df-r0/git/Source/JavaScriptCore/jit/JITOperations.cpp:1112

WPEWebKit/Source/JavaScriptCore/jit/JITOperations.cpp

Line 1112 in 87cbd6c

RELEASE_ASSERT(callLinkInfo->isDirect());

[glibnovodran]

@guijemont, my understanding is @woutermeek has received or will receive soon a more detailed information from Comcast as well as the MIPS-based device to reproduce with.

[mbhatt627]

There are random crash happens on JS Core( WPE -2.22).
These crashes are seen only on MIPS devices.
Crash happens on operationLinkDirectCall()
1- RELEASE_ASSERT(callLinkInfo->isDirect()); -> This crash is more , in a day 23K ~ 25K crashes
2- RELEASE_ASSERT(callLinkInfo->executable()); -> This less compare to 1. in a day crashes 400~500

There is no reproduction scenario.

WTFCrash_call_stack_isDirect_1112_1.txt
WTFCrash_call_stack_isDirect_1112.txt

Attached few callstack from crash.

[mbhatt627]

Most of the times process's vm.peak exceeds vm.size as well vm.rss.peak exceeds vm.rss.size.
Ex:

vm.rss.peak 239,296
vm.rss.size 211,668
vm.shared.size 104,340
vm.stack.size 136
vm.swap.size 1,308
vm.vma.peak 778,876
vm.vma.size 774,088

[mbhatt627]

@woutermeek , GCC version on Dunfell build is gcc version 9.3.0, where we see increse in this crash.

[mbhatt627]

@guijemont , Do you have any test case of JSC which can reproduce this crash ?

[guijemont]

@guijemont , Do you have any test case of JSC which can reproduce this crash ?

Not yet, though there is still hope that I could find one among the failures I see with qemu.

[modeveci]

Inactive ticket for long time!

Closing the ticket; if you think it is still relevant and/or valid for the latest version/s. Please do not hesitate to re-open!