Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

problems with GC memory reclamation on wpe 2.38 #1363

Open
tomasz-karczewski-red opened this issue Jul 9, 2024 · 27 comments
Open

problems with GC memory reclamation on wpe 2.38 #1363

tomasz-karczewski-red opened this issue Jul 9, 2024 · 27 comments
Assignees
Labels

Comments

@tomasz-karczewski-red
Copy link

Hello,

On our platform (LGI boxes, ARM / Broadcom soc, e,g BCM72180, and other Broadcom platforms we have) we are experiencing the issue where sometimes wpe GC seems to not able to reclaim the memory after exiting some of the heavier web apps. Some example are Sky apps:

https://stv.prd.sky.ch/store/
https://stv.prd.sky.ch/show/

but similar effect could be observed e.g. with Apple TV+ (https://atve.tv.apple.com/94819831-5404-4438-810e-afb648d6a826/tvw_0e7f0489b969451a9f79941a0d18fad1/). These apps normally needs more than 200MB to work, and we have a container cgroup limit of 550MB, so when the memory is not reclaimed quickly enough, wpe gets killed.

Often when these applications exit, I can see that even after GC Full Collection (seen with JSC_logGC=1) the memory is still not reclaimed. Like here - GC was run, spent quite a lot of time, visited a lot of memory, but still wasn't able to release much ('Full sweep: 142348kb => 141164kb')

Jul 09 10:00:19 E0B7B1-APLSTB-300037462300 wpe.sh[13290]: [GC<0x8aff5050>: START M 142134kb => FullCollection, v=0kb (C:0 M:0 P1:0) o=0 b=15866 i#1:N<CsMsrShDMsm(0)> 1+0 v=109974kb (C:55860 M:0 P1:54113) o=11 b=15866 i#2:N 366+0 v=109990kb (C:55872 M:0 P1:54118) o=13 b=15866 i#3:P 0+0 v=109990kb (C:55872 M:0 P1:54118) o=13 b=15866 i#4:P<WsOJwMsrShCsMsm(0)DDomoCb> => 120237kb, p=2353.949000ms (max 2353.949000), cycle 2353.843000ms END]
Jul 09 10:00:19 E0B7B1-APLSTB-300037462300 wpe.sh[13290]: GC END!
Jul 09 10:00:19 E0B7B1-APLSTB-300037462300 wpe.sh[13290]: [GC<0x8aff5050>: finalize 49.610000ms]
1720512019 209
Jul 09 10:00:19 E0B7B1-APLSTB-300037462300 wpe.sh[13290]: [GC<0x8aff5050>: Full sweep: 142348kb => 141164kb, 78.841000ms]

In our setup, each time we leave any app, we return to metro '#boot' (https://widgets.metrological.com/lightning/liberty/2e3c4fc22f0d35e3eb7fdb47eb7d4658#boot). This 'application' is pretty small, and normally uses maybe 20MB of ram. But even when we wait for some time and then browse to some other webapp - the memory often is not released, keeps on piling up, and so we run OOM after several iterations.

I've tried with some upstream changes, like this one:

#1285

Also tried some additional modifications to GC process, like here:

LibertyGlobal@ab127e3

(for example I've modified Source/bmalloc/bmalloc/AvailableMemory.cpp to check for WPE_RAM_SIZE when determining available memory; we are running wpe in lxc container, so there are different limits that what sysinfo returns I believe. but this doesn't seem to fix the problem by itself; also - I've enabled some bmalloc verbose flags & from what I've seen it doesn't seem that bmalloc is to be blamed here)

I was also checking out many JSC options, like:

JSC_forceRAMSize=576716800
JSC_mediumHeapGrowthFactor=1.1
JSC_smallHeapGrowthFactor=1.1
JSC_largeHeapGrowthFactor=1.1
JSC_smallHeapRAMFraction=0.25
JSC_mediumHeapRAMFraction=0.5
JSC_customFullGCCallbackBailThreshold=1.0
JSC_maximumMutatorUtilization=0.6
JSC_minimumGCPauseMS=1
JSC_useStochasticMutatorScheduler=false
JSC_gcIncrementScale=1
JSC_criticalGCMemoryThreshold=0.5
JSC_forceDidDeferGCWork=true
JSC_useGlobalGC=true

Some of these sometimes seem to help for a while, but then - the app runs into the same problem.

I've also added some logic (on thunder plugin side) to force GC in regular intervals after the application returns to #boot app (via webkit_web_context_garbage_collect_javascript_objects that ends up calling WebProcess::garbageCollectJavaScriptObjects); this generally helps, but even this is not 100% reliable. Even if this sometimes makes the situation better, at times I still run into the scenario when the app memory is not reclaimed, even when GC is invoked manually from the inspector (via $vm.gc(), if enabled via JSC_enableDollarVM=true). This behavior seems a bit random - for example it might start working fine after the reboot. I've checked wpe 2.22, and didn't observe these issues; memory is reclaimed pretty quickly when I leave the app.

Looking into web inspector / Timelines / memory tab, I see that most of this 'unreclaimable' memory is still counted as 'JavaScript' - in line with what I can see in the GC run results.

So for example, on our platform, the memory graph for 2.22 versus 2.38 compares like this:

for 2.22: see attached wpe-2.22-graph.png

wpe-2 22-graph

for 2.38: see attached wpe-2.38-graph,png

wpe-2 38-graph

So while 2.22 was running the test scenario just fine, 2.38 OOMed pretty quickly.

I've tried similar test on wpe 2.38 with different platform, the 'Video Accelerator' (https://rdkcentral.com/rdk-video-accelerator). Here, there are no containers, but when I check the WPEWebProcess Rss memory after exiting these apps, I see something like this: (checked after each iteration of 'enter the app - exit to #boot')

RssAnon: 14580 kB
RssAnon: 14580 kB
RssAnon: 211136 kB
RssAnon: 215216 kB
RssAnon: 67124 kB
RssAnon: 228000 kB
RssAnon: 228052 kB
RssAnon: 236304 kB
RssAnon: 381584 kB
RssAnon: 234400 kB
RssAnon: 234400 kB
RssAnon: 235620 kB

So sometimes the memory gets cleaned up, sometimes - keeps on piling up. With our 550MB memory limit, 381584 kB in idle state might be enough to run into OOM when we try to start eg. Sky Store app.

Do you have some clues what could cause the difference in GC behaviour in 2.38, maybe there are some other config options we are not aware of?

@justinmichaud
Copy link

Hi Tomasz,

Sorry for the delay. I have been struggling to get a WPE build working on a rpi remotely all last week, but my pi finally arrived in the mail today so I should be able to reproduce this tomorrow.

I have tried unsuccessfully to reproduce this on desktop with PSON (process swap on navigation) disabled, but I will try a few more configurations.

I am curious though; why does wpe navigate in the same process? This is not an area that I am at all familiar with. My intuition is that this would be faster, and also reduce memory fragmentation, without too much memory overhead.

@magomez magomez assigned justinmichaud and unassigned magomez Jul 22, 2024
@tomasz-karczewski-red
Copy link
Author

hello @justinmichaud,
Sorry for the late update, I was on the vacation. Regarding PSON - yes, we do not seem to be using that; I'll investigate why & if that is possible to test in our setup.

@tomasz-karczewski-red
Copy link
Author

tomasz-karczewski-red commented Jul 24, 2024

I was able to get a heap snapshot ('gcdebugging') type, it's here:

snapshot-leak-gddebugging.json.z01.txt

snapshot-leak-gddebugging.json.zip.txt

(Had to split the zip file - too big for github - and add '.txt' suffixes to upload - github doesn't accept 'z01' files)

Once unzipped, it is possible to open that file in web inspector; it is visible that the heap still contains the objects referencing both pages that I've visited ('https://stv.prd.sky.ch/store/' and 'https://widgets.metrological.com/lightning/liberty/2e3c4fc22f0d35e3eb7fdb47eb7d4658#boot).

Also, curiously, web inspector shows the Window objects that seem to be connected to both these pages (and 3 more):

obj_graph

I've reformatted the GC roots information from this heap, it is here:

snapshot-leak-gddebugging.rootsinfo.json

Again, the heap seems to contain GC roots that reference both old & new pages.

@tomasz-karczewski-red
Copy link
Author

also, regarding PSON - this seems to be disabled by default on wpe:

configuration.setProcessSwapsOnNavigation(false);

@tomasz-karczewski-red
Copy link
Author

One more update, regarding our platform - it might not be obvious from the description, but this is 32 bit arm (armv7l)

cat /proc/cpuinfo
processor : 0
model name : ARMv7 Processor rev 0 (v7l)
BogoMIPS : 54.00
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x42
CPU architecture: 7
CPU variant : 0x0
CPU part : 0x100
CPU revision : 0

processor : 1
model name : ARMv7 Processor rev 0 (v7l)
BogoMIPS : 54.00
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x42
CPU architecture: 7
CPU variant : 0x0
CPU part : 0x100
CPU revision : 0

processor : 2
model name : ARMv7 Processor rev 0 (v7l)
BogoMIPS : 54.00
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x42
CPU architecture: 7
CPU variant : 0x0
CPU part : 0x100
CPU revision : 0

processor : 3
model name : ARMv7 Processor rev 0 (v7l)
BogoMIPS : 54.00
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x42
CPU architecture: 7
CPU variant : 0x0
CPU part : 0x100
CPU revision : 0

Hardware : Broadcom STB (Flattened Device Tree)
Revision : 0000
Serial : 0000000000000000

@justinmichaud
Copy link

Hey! No worries, I just got back from vacation myself. I am still working on this, but getting an armv7 setup working is proving to be challenging.

In the meantime, I have been trying to do some investigation on the desktop port. I would like to be able to log where the roots are coming from without WebInspector, because it can keep some things alive. I am also trying to find a way to manually trigger the memory pressure handler.

Does this issue reproduce on 64-bit builds for you?

@tomasz-karczewski-red
Copy link
Author

hello,

to log where the roots are coming from without WebInspector

fwiw, I achieved something like this by directly calling GCController::singleton().dumpHeap() via WebProcess::garbageCollectJavaScriptObjects, which can be invoked from wpe thunder plugin in our integration. I also needed to modify HeapSnapshotBuilder.cpp to dump directly to the file, otherwise it will quickly require a lot of memory (json StringBuffer resizes exponentially)

Does this issue reproduce on 64-bit builds

I haven't tested that on 64 bit builds. I could probably try on my desktop with x86_64, but what exact setup would you suggest? wpe 2.38 with the cog (https://github.com/Igalia/cog) application?

@justinmichaud
Copy link

Don't worry, I thought maybe you may have had a 64-bit device handy running the same WPE build and software stack handy, since that would reduce a number of variables. I'll try to reproduce this on desktop too.

@tomasz-karczewski-red
Copy link
Author

Hello,

I've been doing some investigation of this issue in parallel, and though I'd share what I've seen so far. Since the web inspector is rather underwhelming when it comes to analysing the heap dumps, I've prepared a small tool myself (see nodereader-networkx.py attached). And did following test, typical of our application flow:

  1. start wpe
  2. navigate to https://widgets.metrological.com/lightning/liberty/2e3c4fc22f0d35e3eb7fdb47eb7d4658#boot
  3. navigate to https://tv.playsuisse.ch/, finish page loading
  4. navigate back to https://widgets.metrological.com/lightning/liberty/2e3c4fc22f0d35e3eb7fdb47eb7d4658#boot
  5. full GC forced (4 times)
  6. navigate to https://stv.prd.sky.ch/store/, finish page loading
  7. navigate back to https://widgets.metrological.com/lightning/liberty/2e3c4fc22f0d35e3eb7fdb47eb7d4658#boot
  8. full GC forced (4 times)

And after that, I took a heap (gcdebugging type) snapshot.

With the help of this python tool (using 'node_by_class Window command') I can see that there are multiple Window objects; the most recent one still has proper 'Label' with the latest url; the old ones have 'about:blank' labels (but still exist, and via some links, they can be connected to these previously visited apps). Something like this:

Enter node id: node_by_class Window
NodeID: 4524649 Size: 2377 ClassName: 'Window' Flags: 0 Label: 'about:blank' Cell: 0x77184000 Wrapped: 0x0 ROOT: None Reason: None Reachibility: None
NodeID: 2509240 Size: 2377 ClassName: 'Window' Flags: 0 Label: 'https://widgets.metrological.com/lightning/liberty/2e3c4fc22f0d35e3eb7fdb47eb7d4658#boot' Cell: 0x771862c0 Wrapped: 0x0 ROOT: None Reason: None Reachibility: None
NodeID: 2460729 Size: 2428 ClassName: 'Window' Flags: 0 Label: 'https://widgets.metrological.com/lightning/liberty/2e3c4fc22f0d35e3eb7fdb47eb7d4658#boot' Cell: 0x77186b70 Wrapped: 0x0 ROOT: None Reason: None Reachibility: None
NodeID: 2514486 Size: 2377 ClassName: 'Window' Flags: 0 Label: 'https://widgets.metrological.com/lightning/liberty/2e3c4fc22f0d35e3eb7fdb47eb7d4658#boot' Cell: 0x77187420 Wrapped: 0x0 ROOT: None Reason: None Reachibility: None
NodeID: 2645150 Size: 2387 ClassName: 'Window' Flags: 0 Label: 'about:blank' Cell: 0x78e1f238 Wrapped: 0x0 ROOT: None Reason: None Reachibility: None
NodeID: 4526513 Size: 2377 ClassName: 'Window' Flags: 0 Label: 'about:blank' Cell: 0x78e5d038 Wrapped: 0x0 ROOT: None Reason: None Reachibility: None
NodeID: 2642737 Size: 2442 ClassName: 'Window' Flags: 0 Label: 'about:blank' Cell: 0x78ec3038 Wrapped: 0x0 ROOT: None Reason: None Reachibility: None
NodeID: 2575714 Size: 2387 ClassName: 'Window' Flags: 0 Label: 'about:blank' Cell: 0x7e812038 Wrapped: 0x0 ROOT: None Reason: None Reachibility: None
NodeID: 2474901 Size: 35 ClassName: 'Window' Flags: 0 Label: '' Cell: 0x822b7280 Wrapped: 0x0 ROOT: None Reason: None Reachibility: None
NodeID: 4526306 Size: 17 ClassName: 'Window' Flags: 0 Label: '' Cell: 0x84bc1f60 Wrapped: 0x0 ROOT: None Reason: None Reachibility: None
NodeID: 4497882 Size: 17 ClassName: 'Window' Flags: 0 Label: '' Cell: 0x84bc2120 Wrapped: 0x0 ROOT: None Reason: None Reachibility: None
NodeID: 4527500 Size: 17 ClassName: 'Window' Flags: 0 Label: '' Cell: 0x84bc22e0 Wrapped: 0x0 ROOT: None Reason: None Reachibility: None
NodeID: 2514187 Size: 17 ClassName: 'Window' Flags: 0 Label: '' Cell: 0x84bc2990 Wrapped: 0x0 ROOT: None Reason: None Reachibility: None
NodeID: 2474910 Size: 17 ClassName: 'Window' Flags: 0 Label: '' Cell: 0x84bc2b50 Wrapped: 0x0 ROOT: None Reason: None Reachibility: None
NodeID: 2518349 Size: 17 ClassName: 'Window' Flags: 0 Label: '' Cell: 0x84bc2d10 Wrapped: 0x0 ROOT: None Reason: None Reachibility: None
NodeID: 2655961 Size: 17 ClassName: 'Window' Flags: 0 Label: '' Cell: 0x84bc33c0 Wrapped: 0x0 ROOT: None Reason: None Reachibility: None
NodeID: 2584279 Size: 17 ClassName: 'Window' Flags: 0 Label: '' Cell: 0x84bc38c0 Wrapped: 0x0 ROOT: None Reason: None Reachibility: None
NodeID: 2640999 Size: 17 ClassName: 'Window' Flags: 0 Label: '' Cell: 0x84bc3ed0 Wrapped: 0x0 ROOT: None Reason: None Reachibility: None
NodeID: 2461886 Size: 2467 ClassName: 'Window' Flags: 0 Label: 'about:blank' Cell: 0xab878938 Wrapped: 0x0 ROOT: None Reason: None Reachibility: None
Enter node id: 2645150
NodeID: 2645150 Size: 2387 ClassName: 'Window' Flags: 0 Label: 'about:blank' Cell: 0x78e1f238 Wrapped: 0x0 ROOT: None Reason: None Reachibility: None
Enter node id: 2460729
NodeID: 2460729 Size: 2428 ClassName: 'Window' Flags: 0 Label: 'https://widgets.metrological.com/lightning/liberty/2e3c4fc22f0d35e3eb7fdb47eb7d4658#boot' Cell: 0x77186b70 Wrapped: 0x0 ROOT: None Reason: None Reachibility: None

Taking a look at the root objects that are holding these Window objects ('closest_root_paths $window_object_node_id' command), it seems that these old (not the latest) Window object are mostly held by root paths that look like this:

(4523988 class 'TextTrack' label 'url https://tv.playsuisse.ch/Landing' cell 0x79675c98(0xab95d3f8) ROOT) -> [0/na] -> (4523988 class 'Structure' label 'TextTrack' cell 0x7a0b62b0(0x0)) -> [0/na] -> (4523988 class 'Window' label 'about:blank' cell 0x7e812038(0x0))

(4523988 class 'CSSStyleDeclaration' label 'url https://tv.playsuisse.ch/Landing' cell 0x7a0d45e0(0xab938294) ROOT) -> [0/na] -> (4523988 class 'Structure' label 'CSSStyleDeclaration' cell 0x7a0d8c30(0x0)) -> [0/na] -> (4523988 class 'Window' label 'about:blank' cell 0x7e812038(0x0))

  • so generally 'class X -> Structure of class X -> Window' path. This Window objects seem to correspond to 'global object' (eg. looking at the code in JSWindowProxy::setWindow - 'JSDOMGlobalObject *window = JSDOMWindow::create ... ASSERT(window->globalObject() == window') )

I've looked into Source/JavaScriptCore/runtime/Structure.cpp / Structure::visitChildrenImpl; during GC it is indeed visiting m_globalObject, guess that's where the GC link comes from. I've actually tried to comment this part out, like:

template
void Structure::visitChildrenImpl(JSCell* cell, Visitor& visitor)
...
#if 0
visitor.append(thisObject->m_globalObject);
#endif

  • and then these 'class X -> structure X -> Window' paths disappear, but the Window objects are still not GC'd and instead I get longer paths via Function object, generally similar to:

(3793819 class 'CSSStyleDeclaration' label 'url https://stv.prd.sky.ch/store/' cell 0x764c7520(0xab9144c4) ROOT) -> [0/na] -> (1934867 class 'Structure' label 'CSSStyleDeclaration' cell 0x82263520(0x0)) -> [0/na] -> (1922666 class 'CSSStyleDeclaration' label '' cell 0x84bc3c30(0x0)) -> [1/setProperty] -> (1926559 class 'Function' label 'setProperty' cell 0x7e2b4be0(0x0)) -> [0/na] -> (1913906 class 'Window' label 'about:blank' cell 0xab8d8038(0x0))

so, now it's like 'class X -> structure of class X -> class X again (but some different cell/object now?) -> some Function (like, 'setProperty' here) -> Window'

Don't know how to get rid of this 'Function' link. It seems that this global object / Window instances are held locally, while this global object actually changes when you navigate (right?)

Attaching this new gc debugging snapshot (5-snapshot-then-exited-sstore.json), it should be possible to look into it with this helper tool I've attached.
5-snapshot-then-exited-sstore.json.zip.txt
5-snapshot-then-exited-sstore.json.z01.txt
5-snapshot-then-exited-sstore.json.z02.txt
nodereader-networkx.py.txt

@justinmichaud
Copy link

(4523988 class 'TextTrack' label 'url https://tv.playsuisse.ch/Landing' cell 0x79675c98(0xab95d3f8) ROOT)

That looks suspicious! Why is this a root? I'll investigate, but this seems like the culprit.

Just an update: I fixed a bug in WebInspector so linux users can actually load heap dumps now. This python tool is super helpful, thanks!

Separately, I created a way to manually trigger the memory pressure handler, which does free up some of the memory. I think this might be related to a different issue we were experiencing with MotionMark.

@justinmichaud
Copy link

Weak references now display the correct root marking reason. Weak references shouldn't be marked unless they are still reachable, so I am trying to see why these weak references are reachable. It is unfortunate that they make it harder to see where the "true" root is coming from.

I have made a little script that exports the heap snapshot to graphml, and I am opening it in Gephi. I will see if I can see any interesting structure

@justinmichaud
Copy link

heapSnapshot.patch.txt

@justinmichaud
Copy link

Update: I see that there are a bunch of Handle<>s keeping the window object alive. I am adding some logging to see where these are allocated from.

My guess at the moment is that there is somewhere in our generated dom bindings code that is keeping these alive when they shouldn't.

@justinmichaud
Copy link

I have removed as many Strong<>'s as I can and logged the rest. It seems like ScriptController is keeping alive the JS Window objects via a Strong. My hypothesis is that a C++ reference cycle is keeping Frame alive, which is the owner of ScriptController.

@justinmichaud
Copy link

justinmichaud commented Aug 5, 2024

Hey @tomasz-karczewski-red:

Can you see if this fixes your issue? I will check on my rpi tomorrow, I am just suggesting it here in case you can get to it first (timezones):

diff --git a/Source/WebCore/inspector/InspectorInstrumentation.cpp b/Source/WebCore/inspector/InspectorInstrumentation.cpp
index da310c6df500..921022d9af0e 100644
--- a/Source/WebCore/inspector/InspectorInstrumentation.cpp
+++ b/Source/WebCore/inspector/InspectorInstrumentation.cpp
@@ -646,6 +646,8 @@ void InspectorInstrumentation::didReceiveResourceResponseImpl(InstrumentingAgent
     if (LIKELY(!instrumentingAgents.inspectorEnvironment().developerExtrasEnabled()))
         return;
 
+    return;
+
     if (auto* networkAgent = instrumentingAgents.enabledNetworkAgent())
         networkAgent->didReceiveResponse(identifier, loader, response, resourceLoader);
     if (auto* consoleAgent = instrumentingAgents.webConsoleAgent())

For completeness, I am running with the following options and I can no longer reproduce the issue I was seeing on ToT desktop. That doesn't necessarily mean anything about WPE.

JSC_useSourceProviderCache=0 JSC_useCodeCache=0 WEBKIT_DISABLE_SANDBOX_THIS_IS_DANGEROUS=1 MALLOC=0 WE_B_PROCESS_CMD_PREFIX="/usr/bin/valgrind --tool=massif " Tools/Scripts/run-minibrowser --gtk --debug https://stv.prd.sky.ch/show/

I saw the following trace:

1   0x7fad0632d74c JSC::debugReportLiveStrong(void*)
2   0x7fad0642757f JSC::Strong<JSC::JSGlobalObject, (JSC::ShouldStrongDestructorGrabLock)0>::Strong()
3   0x7fad0650bfef Inspector::JSGlobalObjectInspectorController::JSGlobalObjectInspectorController(JSC::JSGlobalObject&)
4   0x7fad06c1551e std::__detail::_MakeUniq<Inspector::JSGlobalObjectInspectorController>::__single_object std::make_unique<Inspector::JSGlobalObjectInspectorController, JSC::JSGlobalObject&>(JSC::JSGlobalObject&)
5   0x7fad06beab16 decltype(auto) WTF::makeUnique<Inspector::JSGlobalObjectInspectorController, JSC::JSGlobalObject&>(JSC::JSGlobalObject&)
6   0x7fad06bf8fb6 JSC::JSGlobalObject::init(JSC::VM&)
7   0x7fad06c0965b JSC::JSGlobalObject::finishCreation(JSC::VM&, JSC::JSObject*)
8   0x7fad1668581e WebCore::JSDOMGlobalObject::finishCreation(JSC::VM&, JSC::JSObject*)
9   0x7fad166a926d WebCore::JSDOMWindowBase::finishCreation(JSC::VM&, WebCore::JSWindowProxy*)
10  0x7fad142c0741 WebCore::JSDOMWindow::finishCreation(JSC::VM&, WebCore::JSWindowProxy*)
11  0x7fad167123e8 WebCore::JSDOMWindow::create(JSC::VM&, JSC::Structure*, WTF::Ref<WebCore::DOMWindow, WTF::RawPtrTraits<WebCore::DOMWindow>, WTF::DefaultRefDerefTraits<WebCore::DOMWindow> >&&, WebCore::JSWindowProxy*)
12  0x7fad16712bfc WebCore::JSWindowProxy::setWindow(WebCore::DOMWindow&)
13  0x7fad16712835 WebCore::JSWindowProxy::finishCreation(JSC::VM&, WebCore::DOMWindow&)
14  0x7fad16712914 WebCore::JSWindowProxy::create(JSC::VM&, WebCore::DOMWindow&, WebCore::DOMWrapperWorld&)
15  0x7fad168148e7 WebCore::WindowProxy::createJSWindowProxy(WebCore::DOMWrapperWorld&)
16  0x7fad16814b86 WebCore::WindowProxy::createJSWindowProxyWithInitializedScript(WebCore::DOMWrapperWorld&)
17  0x7fad166a5846 WebCore::WindowProxy::jsWindowProxy(WebCore::DOMWrapperWorld&)
18  0x7fad1677125c WebCore::ScriptController::jsWindowProxy(WebCore::DOMWrapperWorld&)
19  0x7fad1676f84c WebCore::ScriptController::evaluateInWorld(WebCore::ScriptSourceCode const&, WebCore::DOMWrapperWorld&)
20  0x7fad167723fe WebCore::ScriptController::executeScriptInWorld(WebCore::DOMWrapperWorld&, WebCore::RunJavaScriptParameters&&)
21  0x7fad16773da5 WebCore::ScriptController::executeAsynchronousUserAgentScriptInWorld(WebCore::DOMWrapperWorld&, WebCore::RunJavaScriptParameters&&, WTF::CompletionHandler<void (std::experimental::fundamentals_v3::expected<JSC::JSValue, WebCore::ExceptionDetails>)>&&)
22  0x7fad1397ef89 WebKit::WebPage::runJavaScript(WebKit::WebFrame*, WebCore::RunJavaScriptParameters&&, WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&)
23  0x7fad1397f5f9 WebKit::WebPage::runJavaScriptInFrameInScriptWorld(WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> const&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&)
24  0x7fad1243e5fd auto IPC::callMemberFunction<WebKit::WebPage, WebKit::WebPage, void (WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> const&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&), std::tuple<WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >, void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>(WebKit::WebPage*, void (WebKit::WebPage::*)(WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> const&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&), std::tuple<WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >&&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&)::{lambda((auto:1&&)...)#1}::operator()<WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >(WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >&&, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String>&&) const
25  0x7fad1247a33f void std::__invoke_impl<void, IPC::callMemberFunction<WebKit::WebPage, WebKit::WebPage, void (WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> const&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&), std::tuple<WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >, void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>(WebKit::WebPage*, void (WebKit::WebPage::*)(WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> const&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&), std::tuple<WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >&&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&)::{lambda((auto:1&&)...)#1}, WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >(std::__invoke_other, IPC::callMemberFunction<WebKit::WebPage, WebKit::WebPage, void (WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> const&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&), std::tuple<WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >, void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>(WebKit::WebPage*, void (WebKit::WebPage::*)(WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> const&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&), std::tuple<WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >&&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&)::{lambda((auto:1&&)...)#1}&&, WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >&&, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String>&&)
26  0x7fad1245c3de std::__invoke_result<IPC::callMemberFunction<WebKit::WebPage, WebKit::WebPage, void (WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> const&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&), std::tuple<WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >, void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>(WebKit::WebPage*, void (WebKit::WebPage::*)(WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> const&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&), std::tuple<WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >&&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&)::{lambda((auto:1&&)...)#1}, WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >::type std::__invoke<IPC::callMemberFunction<WebKit::WebPage, WebKit::WebPage, void (WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> const&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&), std::tuple<WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >, void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>(WebKit::WebPage*, void (WebKit::WebPage::*)(WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> const&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&), std::tuple<WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >&&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&)::{lambda((auto:1&&)...)#1}, WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >(IPC::callMemberFunction<WebKit::WebPage, WebKit::WebPage, void (WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> const&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&), std::tuple<WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >, void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>(WebKit::WebPage*, void (WebKit::WebPage::*)(WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> const&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&), std::tuple<WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >&&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&)::{lambda((auto:1&&)...)#1}&&, WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >&&, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String>&&)
27  0x7fad1243e68c decltype(auto) std::__apply_impl<IPC::callMemberFunction<WebKit::WebPage, WebKit::WebPage, void (WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> const&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&), std::tuple<WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >, void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>(WebKit::WebPage*, void (WebKit::WebPage::*)(WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> const&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&), std::tuple<WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >&&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&)::{lambda((auto:1&&)...)#1}, std::tuple<WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >, 0ul, 1ul, 2ul>(IPC::callMemberFunction<WebKit::WebPage, WebKit::WebPage, void (WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> const&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&), std::tuple<WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >, void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>(WebKit::WebPage*, void (WebKit::WebPage::*)(WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> const&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&), std::tuple<WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >&&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&)::{lambda((auto:1&&)...)#1}&&, std::tuple<WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >&&, std::integer_sequence<unsigned long, 0ul, 1ul, 2ul>)
28  0x7fad1243e6d3 decltype(auto) std::apply<IPC::callMemberFunction<WebKit::WebPage, WebKit::WebPage, void (WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> const&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&), std::tuple<WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >, void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>(WebKit::WebPage*, void (WebKit::WebPage::*)(WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> const&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&), std::tuple<WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >&&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&)::{lambda((auto:1&&)...)#1}, std::tuple<WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> > >(IPC::callMemberFunction<WebKit::WebPage, WebKit::WebPage, void (WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> const&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&), std::tuple<WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >, void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>(WebKit::WebPage*, void (WebKit::WebPage::*)(WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> const&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&), std::tuple<WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >&&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&)::{lambda((auto:1&&)...)#1}&&, std::tuple<WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >&&)
29  0x7fad1243e748 void IPC::callMemberFunction<WebKit::WebPage, WebKit::WebPage, void (WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> const&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&), std::tuple<WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >, void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>(WebKit::WebPage*, void (WebKit::WebPage::*)(WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> const&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&), std::tuple<WebCore::RunJavaScriptParameters, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> >&&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&)
30  0x7fad12423df7 void IPC::handleMessageAsync<Messages::WebPage::RunJavaScriptInFrameInScriptWorld, WebKit::WebPage, WebKit::WebPage, void (WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> const&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&)>(IPC::Connection&, IPC::Decoder&, WebKit::WebPage*, void (WebKit::WebPage::*)(WebCore::RunJavaScriptParameters&&, std::optional<WebCore::ProcessQualified<WTF::ObjectIdentifierGeneric<WebCore::FrameIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long> > >, std::pair<WTF::ObjectIdentifierGeneric<WebKit::ContentWorldIdentifierType, WTF::ObjectIdentifierMainThreadAccessTraits<unsigned long>, unsigned long>, WTF::String> const&, WTF::CompletionHandler<void (std::span<unsigned char const, 18446744073709551615ul>, std::optional<WebCore::ExceptionDetails> const&)>&&))
31  0x7fad12417d46 WebKit::WebPage::didReceiveWebPageMessage(IPC::Connection&, IPC::Decoder&)

@tomasz-karczewski-red
Copy link
Author

Hello,

I've tried the InspectorInstrumentation.cpp change; something like:

--- a/Source/WebCore/inspector/InspectorInstrumentation.cpp
+++ b/Source/WebCore/inspector/InspectorInstrumentation.cpp
@@ -615,6 +615,8 @@ void InspectorInstrumentation::didReceiveResourceResponseImpl(InstrumentingAgent
{
if (LIKELY(!instrumentingAgents.inspectorEnvironment().developerExtrasEnabled()))
return;

  • fprintf(stderr, "early return from InspectorInstrumentation::didReceiveResourceResponseImpl\n");

  • return;

    if (auto* networkAgent = instrumentingAgents.enabledNetworkAgent())
    networkAgent->didReceiveResponse(identifier, loader, response, resourceLoader);

with following envs:

JSC_useSourceProviderCache=0
JSC_useCodeCache=0
JSC_dumpOptions=0

The change seems to be applied, I can see the added logs - but unfortunately it doesn't seem to improve the memory situation; nothing more seems to be released on full GC. I have not applied the previous patch you've supplied (heapSnapshot.patch.txt) - it would require some rebasing for me, it seems to be prepared over 2.42 version (looking at Heap.cpp differences). Should I try with these changes applied first?

Another thing, I think I've found a reliable workaround on our platform - you can see the change here:

LibertyGlobal@44bb731

44bb7312475166426ede9a046e353bbd9e9ef314.patch.txt

So, what the change does is disabling Conservative Scan on-demand. And I do it only when manually forcing GC (from our thunder plugin, via webkit_web_context_garbage_collect_javascript_objects call, which goes to WebProcess::garbageCollectJavaScriptObjects). I only do it in 'safe point', when wpe have navigated to 'idle/#boot' webpage. And this has the effect of releasing the memory momentarily, just like on wpe 2.22 - the heap is mostly clean. I've tested this workaround on 2 of our platforms, and it seems to work 100% of the time on both (from what I've tested so far). My current understanding is:

  • conservative scan in essence goes through the heap & tries to interpret everything as a pointers to other heap objects, to detect native pointers

  • our box is 32 bit, so the address space is not that big. The app (sky store) itself is like 200MB, so it takes around 5% of this address space. So it is not that hard to actually hit that area. I've also checked, and there is not much variance in memory virtual space allocations between application restarts on our box. The allocated ranges are quite similar; this would explain quite high reproducibility of the problem. I sometimes had test runs when this problem didn't happen - this could amount to 'lucky' address space layouts.

  • in sky store app we have for example this aws waf-related file:

https://3df51d5ca14f.edge.sdk.awswaf.com/3df51d5ca14f/4104523408d1/challenge.js

which contains a long Uint32Array of 32 bit pseudo-random values. If conservative scan tries to interpret that as pointers, it is almost 100% sure that some of them will point to some allocated heap objects (like, with 5% mem occupied by the app data, probability of not hitting this area on random - (95/100)^128 - is already around 0.1%).

If that hypotesis is right, would we be able to do anything else about it? This seems to be what conservative root scan should work like, after all. Probably this is much smaller issue on 64 bit ... And what do you think about such a workaround? (again, this would need to be done only in 'safe places', and the interface would be something different that the current atomic_bool ...)

@justinmichaud
Copy link

This was my fear. That being said, I am not so sure that this is the case we are encountering yet.

In JSC, we only conservatively scan the stack + registers. It is very easy for something to find its way in there accidentally, but in that case it would show up clearly in the heap snapshot.

Skipping Weaks, the heap mark reason should be ConservativeRoots if this was the case. My patch also fixes the heap root mark reason being NONE, but only for Strongs. So I am not so sure that this is the case.

In the meantime, if you can try only disabling parts of the conservative collection, that might help narrow it down. Make sure the VM top callframe is being set correctly too, in case we are scanning too much.

For a workaround you can actually ship (in addition to providing a performance benefit), you can enable PSON. Even if conservative scanning is the cause, this is the only sound workaround that I can think of. If anything blocks you from doing that, we can try to fix it.

@magomez
Copy link

magomez commented Aug 6, 2024

PSON is disabled on WebPlatformForEmbedded because of the extra memory usage of having several web processes alive simultaneously. WPE is usually running on an environment where memory is scarce, and those extra MB are very valuable.

@tomasz-karczewski-red
Copy link
Author

Hello @justinmichaud ,
Thanks - I'll check the conservative scanner in more details. One questions though:

Make sure the VM top callframe is being set correctly too

  • how would I do it?
    Regarding PSON; I've done some initial evaluation, and it generally seemed to work, but it would necessitate some architectural changes in our app integration (eg. because the wpe wayland window is recreated; we do not expect that). That's why I'd prefer to avoid that for now.

@justinmichaud
Copy link

@magomez if there is enough memory to jit, there should be enough to use pson. We should figure out what issues are blocking enablement. Even if we discover conservative scanning is not the cause of this leak, it totally may cause memory issues in the future that are far greater than the cost of the prewarmed process. We can disable the back-forward cache.

The workaround above is very incorrect and will lead to crashes. This is the only workaround that can ship.

@tomasz-karczewski-red
Copy link
Author

so, seems this GC problem can be narrowed down to:

conservativeRoots.add(currentThreadState.stackTop, currentThreadState.stackOrigin, jitStubRoutines, codeBlocks);

in MachineThreads::gatherFromCurrentThread; disabling that part seems enough to prevent weird GC behavior. From some other debugs I've added, seems the stack span being considered is eg. like 'begin: 0xbe84c680 end: 0xbe84e000', 6528 bytes. Some values that are considered look like pointers, some not:

ConservativeRoots::genericAddPointer p 0xb634b29c
...
ConservativeRoots::genericAddPointer p 0xaf4e12d0
ConservativeRoots::genericAddPointer p 0xaf4fb1fc
...
ConservativeRoots::genericAddPointer p 0xfffffffc
...
ConservativeRoots::genericAddPointer p 0x1
...
ConservativeRoots::genericAddPointer p 0x15250
... etc.

(there is also this removeArrayPtrTag invocation in genericAddPointer, but it doesn't seem to modify the pointers on our arch)

@tomasz-karczewski-red
Copy link
Author

tomasz-karczewski-red commented Aug 7, 2024

I also took a coredump around MachineThreads::gatherFromCurrentThread; from what I see:

  1. The range / (part of) current thread stack scanned by CS will be:

currentThreadState.stackTop: 0xbeacf680 currentThreadState.stackOrigin: 0xbead1000.

  1. this corresponds to the stack as it was at frame Basic Weseros support added. #24 ('JSC::callWithCurrentThreadState'), around where the stack state 'snapshot' was taken:

(gdb) f 24
#24 JSC::callWithCurrentThreadState(WTF::ScopedLambda<void (JSC::CurrentThreadState&)> const&) ()
at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/MachineStackMarker.cpp:231
231 /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/MachineStackMarker.cpp: No such file or directory.
(gdb) info frame
Stack level 24, frame at 0xbeacf6c0:
pc = 0xb3d87b9e in JSC::callWithCurrentThreadState(WTF::ScopedLambda<void (JSC::CurrentThreadState&)> const&)
(/usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/MachineStackMarker.cpp:231); saved pc = 0xb3d77176
called by frame at 0xbeacf6e0, caller of frame at 0xbeacf6c0
source language c++.
Arglist at 0xbeacf680, args:
Locals at 0xbeacf680, Previous frame's sp is 0xbeacf6c0
Saved registers:
r4 at 0xbeacf6b0, r5 at 0xbeacf6b4, r6 at 0xbeacf6b8, lr at 0xbeacf6bc

  1. the whole stack trace is like this:

(gdb) bt
#0 __libc_do_syscall () at libc-do-syscall.S:49
#1 0xb3117490 in __libc_signal_restore_set (set=0xbeaccdd0) at ../sysdeps/unix/sysv/linux/internal-signals.h:86
#2 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:48
#3 0xb310a7a2 in __GI_abort () at abort.c:79
#4 0xb3d8767e in JSC::MachineThreads::gatherFromCurrentThread () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/MachineStackMarker.cpp:50
#5 0xb3d87a3a in JSC::MachineThreads::gatherConservativeRoots () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/MachineStackMarker.cpp:212
#6 0xb3d76240 in JSC::Heap::gatherStackRoots () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/Heap.cpp:789
#7 operator()JSC::SlotVisitor () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/Heap.cpp:2785
#8 0xb3d8c15e in JSC::MarkingConstraint::execute () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/MarkingConstraint.cpp:58
#9 JSC::MarkingConstraintSolver::runExecutionThread(JSC::SlotVisitor&, JSC::MarkingConstraintSolver::SchedulerPreference, WTF::ScopedLambda<std::optional ()>) ()
at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/MarkingConstraintSolver.cpp:237
#10 0xb3d8c8f8 in operator() () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/MarkingConstraintSolver.cpp:67
#11 run () at WTF/Headers/wtf/SharedTask.h:91
#12 0xb3d62d32 in JSC::Heap::runTaskInParallel(WTF::RefPtr<WTF::SharedTask<void (JSC::SlotVisitor&)>, WTF::RawPtrTraits<WTF::SharedTask<void (JSC::SlotVisitor&)> >, WTF::DefaultRefDerefTraits<WTF::SharedTask<void (JSC::SlotVisitor&)> > >) () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/Heap.cpp:3133
#13 0xb3d8c57c in runFunctionInParallel<JSC::MarkingConstraintSolver::execute(JSC::MarkingConstraintSolver::SchedulerPreference, WTF::ScopedLambda<std::optional()>)::<lambda(JSC::SlotVisitor&)> > () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/Heap.h:547
#14 JSC::MarkingConstraintSolver::execute(JSC::MarkingConstraintSolver::SchedulerPreference, WTF::ScopedLambda<std::optional ()>) ()
at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/MarkingConstraintSolver.cpp:66
#15 0xb3d8c74e in JSC::MarkingConstraintSolver::drain () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/MarkingConstraintSolver.cpp:97
#16 0xb3d8d00a in JSC::MarkingConstraintSet::executeConvergenceImpl () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/MarkingConstraintSet.cpp:109
#17 0xb3d8d048 in JSC::MarkingConstraintSet::executeConvergence () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/MarkingConstraintSet.cpp:83
#18 0xb3d76bb4 in JSC::Heap::runFixpointPhase () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/Heap.cpp:1469
#19 0xb3d77092 in JSC::Heap::runCurrentPhase () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/Heap.cpp:1299
#20 JSC::Heap::runCurrentPhase () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/Heap.cpp:1267
#21 0xb3d788d2 in operator() () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/Heap.cpp:1923
#22 implFunction () at WTF/Headers/wtf/ScopedLambda.h:106
#23 0xb3d87b9e in WTF::ScopedLambda<void (JSC::CurrentThreadState&)>::operator()JSC::CurrentThreadState&(JSC::CurrentThreadState&) const () at WTF/Headers/wtf/ScopedLambda.h:58
#24 JSC::callWithCurrentThreadState(WTF::ScopedLambda<void (JSC::CurrentThreadState&)> const&) ()
at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/MachineStackMarker.cpp:231
#25 0xb3d77176 in JSC::Heap::collectInMutatorThread () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/Heap.cpp:1935
#26 0xb3d771d8 in JSC::Heap::stopIfNecessarySlow () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/Heap.cpp:1904
#27 JSC::Heap::stopIfNecessarySlow () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/Heap.cpp:1885
#28 0xb3d77ee2 in waitForCollector<JSC::Heap::waitForCollection(JSC::Heap::Ticket)::<lambda(const WTF::AbstractLocker&)> >(void) ()
at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/Heap.cpp:1961
#29 0xb3d7803e in JSC::Heap::waitForCollection () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/Heap.cpp:2200
#30 JSC::Heap::collectSync () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/Heap.cpp:1218
#31 JSC::Heap::collectSync () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/Heap.cpp:1210
#32 0xb3d78184 in JSC::Heap::collectNow () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/Heap.cpp:1166
#33 JSC::Heap::collectNow () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/JavaScriptCore/heap/Heap.cpp:1153
#34 0xb4bc4372 in WebCore::GCController::garbageCollectNow () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/WebCore/bindings/js/GCController.cpp:96
#35 0xb385331c in WebKit::WebProcess::garbageCollectJavaScriptObjects () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/WebKit/WebProcess/WebProcess.cpp:1050
#36 0xb3532826 in IPC::callMemberFunctionImpl<WebKit::WebProcess, void (WebKit::WebProcess::)(), std::tuple<>>(WebKit::WebProcess, void (WebKit::WebProcess::)(), std::tuple<>&&, std::integer_sequence) () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/WebKit/Platform/IPC/HandleMessage.h:131
#37 IPC::callMemberFunction<WebKit::WebProcess, void (WebKit::WebProcess::
)(), std::tuple<>, std::integer_sequence >(std::tuple<>&&, WebKit::WebProcess*, void (WebKit::WebProcess::)()) () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/WebKit/Platform/IPC/HandleMessage.h:137
#38 IPC::handleMessage<Messages::WebProcess::GarbageCollectJavaScriptObjects, WebKit::WebProcess, void (WebKit::WebProcess::
)()> ()
at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/WebKit/Platform/IPC/HandleMessage.h:259
#39 WebKit::WebProcess::didReceiveWebProcessMessage () at DerivedSources/WebKit/WebProcessMessageReceiver.cpp:336
#40 0xb36c927a in IPC::Connection::dispatchMessage () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/WebKit/Platform/IPC/Connection.cpp:1105
#41 0xb36ca222 in IPC::Connection::dispatchMessage () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/WebKit/Platform/IPC/Connection.cpp:1150
#42 0xb36cb572 in IPC::Connection::dispatchOneIncomingMessage () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/WebKit/Platform/IPC/Connection.cpp:1219
#43 0xb42806a8 in WTF::Function<void ()>::operator()() const () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/WTF/wtf/Function.h:82
#44 WTF::RunLoop::performWork () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/WTF/wtf/RunLoop.cpp:134
#45 0xb42c3e30 in operator() () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/WTF/wtf/glib/RunLoopGLib.cpp:80
#46 _FUN () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/WTF/wtf/glib/RunLoopGLib.cpp:82
#47 0xb42c47f8 in operator() () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/WTF/wtf/glib/RunLoopGLib.cpp:53
#48 _FUN () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/WTF/wtf/glib/RunLoopGLib.cpp:56
#49 0xb2ca79e2 in g_main_dispatch (context=0x15250) at ../glib-2.62.6/glib/gmain.c:3216
#50 g_main_context_dispatch (context=context@entry=0x15250) at ../glib-2.62.6/glib/gmain.c:3881
#51 0xb2ca7b92 in g_main_context_iterate (context=0x15250, block=block@entry=1, dispatch=dispatch@entry=1, self=) at ../glib-2.62.6/glib/gmain.c:3954
#52 0xb2ca7eec in g_main_loop_run (loop=0x2e628) at ../glib-2.62.6/glib/gmain.c:4148
#53 0xb42c4b6e in WTF::RunLoop::run () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/WTF/wtf/glib/RunLoopGLib.cpp:108
#54 0xb3901086 in WebKit::AuxiliaryProcessMainBase<WebKit::WebProcess, true>::run () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/WebKit/Shared/AuxiliaryProcessMain.h:71
#55 WebKit::AuxiliaryProcessMainBase<WebKit::WebProcess, true>::run () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/WebKit/Shared/AuxiliaryProcessMain.h:58
#56 WebKit::AuxiliaryProcessMainWebKit::WebProcessMainWPE () at /usr/src/debug/wpe-webkit/1_2.38+git-r0/git/Source/WebKit/Shared/AuxiliaryProcessMain.h:97
#57 0xb310aa66 in __libc_start_main (main=0x0, argc=0, argv=0xb6fdbfa4, init=, fini=0x8635, rtld_fini=0xb6fbf171 <_dl_fini>, stack_end=0xbead0134) at libc-start.c:308
#58 0x0000851c in ?? ()

  1. currentThreadState.stackOrigin is 0xbead1000, and the last frame I got - for __libc_start_main - I see:

(gdb) f 57
#57 0xb310aa66 in __libc_start_main (main=0x0, argc=0, argv=0xb6fdbfa4, init=, fini=0x8635, rtld_fini=0xb6fbf171 <_dl_fini>, stack_end=0xbead0134) at libc-start.c:308
308 libc-start.c: No such file or directory.
(gdb) info frame
Stack level 57, frame at 0xbead0128:
pc = 0xb310aa66 in __libc_start_main (libc-start.c:308); saved pc = 0x851c
called by frame at 0xbead0128, caller of frame at 0xbeacffe0
source language c.
Arglist at 0xbeacffe0, args: main=0x0, argc=0, argv=0xb6fdbfa4, init=, fini=0x8635, rtld_fini=0xb6fbf171 <_dl_fini>, stack_end=0xbead0134
Locals at 0xbeacffe0, Previous frame's sp is 0xbead0128
Saved registers:
r4 at 0xbead0108, r5 at 0xbead010c, r6 at 0xbead0110, r7 at 0xbead0114, r8 at 0xbead0118, r9 at 0xbead011c, r10 at 0xbead0120, lr at 0xbead0124

so the frame is at 0xbead0128 while currentThreadState.stackOrigin is 0xbead1000, quite a bit higher - might that be the issue of 'scanning too much'?

In any case, seems to me that for this particular case - when GC is invoked via WebKit::WebProcess::garbageCollectJavaScriptObjects (called via VM::whenIdle), from IPC message - the part of the stack that is being scanned for roots (frames #24 - #58) can only provide false positives, as it happens to us. I mean, in this case - seems there is no user script processing related code, only some internal GC logic, IPC and glib/gloop. And there are like >1600 potential 'addresses' to scan, that with 32 bit address space and relatively big javascript heap makes us hold the memory forever ...

@tomasz-karczewski-red
Copy link
Author

Hello @justinmichaud,

One more debugging result that might be interesting: I've tried to limit the CurrentThreadState stack range that MachineThreads::gatherFromCurrentThread is considering; simply something like this:

Source/WebKit/WebProcess/WebProcess.cpp:

extern std::atomic<void*> GCJSO_STACK;
(...)
void WebProcess::garbageCollectJavaScriptObjects()
{
int a = 42;
GCJSO_STACK.store(&a);
{
JSLockHolder lock(commonVM());
commonVM().shrinkFootprintWhenIdle();
GCJSO_STACK.store(nullptr);
}
(...)

And then in Source/JavaScriptCore/heap/MachineStackMarker.cpp, MachineThreads::gatherFromCurrentThread, if this 'GCJSO_STACK' falls in the range [currentThreadState.stackTop, currentThreadState.stackOrigin] - stackOrigin is corrected to 'GCJSO_STACK' value. This makes this range much smaller:

(debug) currentThreadState.stackTop: 0xbec01660 GCJSO_STACK: 0xbec017ec currentThreadState.stackOrigin: 0xbec03000; correcting stackOrigin to GCJSO_STACK

And from what I've tested so far, this seems enough to get the correct behavior. So suppose indeed, as you suggested - VM top callframe not being set correctly might be the root cause of the problem here.

@justinmichaud
Copy link

Nice, this is a super promising lead!

Sorry for the slow progress over the past couple days, I have been taking some PTO. I have been attempting to upstream my debugging tools, but in the process, I broke ToT again. I have been trying to understand why, since the heap snapshots you provided seem to suggest that some Strong<> is responsible for keeping alive these window objects.

It is sounding like the ToT issue may be unrelated, which is not surprising. I will investiagte how VMTopCallframe is being set, this sounds very suspicious.

For reference: WebKit/WebKit@b52c0f4 seems related.

I will follow up when I have more information, thank you for your thorough debugging!

@justinmichaud
Copy link

WebKit/WebKit@31ae64f too

tomasz-karczewski-red added a commit to LibertyGlobal/WPEWebKit that referenced this issue Aug 8, 2024
Turns out that conservative GC root scanning is
scanning too deep into the stack & this results
in GC being prevented (for more details see the
ticket & upstream issue:
WebPlatformForEmbedded#1363

This workaround introduces 'stack guards' that prevent
this in particular cases, that is - if there is gloop
main loop routine in the stack (see RunLoopGLib), the
GC root search will not go deeper that this frame. This
is enough to make GC work when invoked synchronously,
via VM::shrinkFootprintWhenIdle; therefore for the
workaround to work, wpe plugin needs to call that
(another part of the workaround is implemented in
WebKitBrowser plugin)
tomasz-karczewski-red added a commit to LibertyGlobal/WPEWebKit that referenced this issue Aug 12, 2024
* ARRISEOS-45892 limit conservative scan range

Turns out that conservative GC root scanning is
scanning too deep into the stack & this results
in GC being prevented (for more details see the
ticket & upstream issue:
WebPlatformForEmbedded#1363

This workaround introduces 'stack guards' that prevent
this in particular cases, that is - if there is gloop
main loop routine in the stack (see RunLoopGLib), the
GC root search will not go deeper that this frame. This
is enough to make GC work when invoked synchronously,
via VM::shrinkFootprintWhenIdle; therefore for the
workaround to work, wpe plugin needs to call that
(another part of the workaround is implemented in
WebKitBrowser plugin)

* ARRISEOS-45892 limit conservative scan range

Small correction: immediate garbageCollectNow is
still useful, since under heavy load shrinkFootprintWhenIdle
might be executed quite late.
@justinmichaud
Copy link

I have finally gotten a fully working RPI setup with buildroot, debugging info, and cogctl allowing JS input.

I made a hack that also fixes the memory leak on ToT by making some things weak references. Of course this can't ship, but it helps to narrow it down.

This also fixes the issue on the pi:

# cog --enable-write-console-messages-to-stdout=1 https://example.com


# cogctl -y open https://stv.prd.sky.ch/show/
# cat /proc/$(ps -ef | awk '$3=="/usr/libexec/wpe-webkit-1.1/WPEWebProcess" {print $1}')/status | grep "VmRSS"
VmRSS:	  244844 kB
# cat /proc/$(ps -ef | awk '$3=="/usr/libexec/wpe-webkit-1.1/WPEWebProcess" {print $1}')/status | grep "VmRSS"
VmRSS:	  232916 kB
# cat /proc/$(ps -ef | awk '$3=="/usr/libexec/wpe-webkit-1.1/WPEWebProcess" {print $1}')/status | grep "VmRSS"
VmRSS:	  232912 kB
# cogctl -y open https://example.com
# cat /proc/$(ps -ef | awk '$3=="/usr/libexec/wpe-webkit-1.1/WPEWebProcess" {print $1}')/status | grep "VmRSS"
VmRSS:	  208428 kB
# cogctl -y open "javascript:\$vm.gc()|\$vm.triggerMemoryPressure()"
# cat /proc/$(ps -ef | awk '$3=="/usr/libexec/wpe-webkit-1.1/WPEWebProcess" {print $1}')/status | grep "VmRSS"
VmRSS:	   70800 kB
# 

I think it is important to find a good explanation for this. Since I observe the same behaviour on 64-bit, it makes me wonder why scanning too deeply could be the cause. I'll follow up with more results.

amol-virnodkar-infosys pushed a commit to LibertyGlobal/WPEWebKit that referenced this issue Aug 21, 2024
* ARRISEOS-45892 limit conservative scan range

Turns out that conservative GC root scanning is
scanning too deep into the stack & this results
in GC being prevented (for more details see the
ticket & upstream issue:
WebPlatformForEmbedded#1363

This workaround introduces 'stack guards' that prevent
this in particular cases, that is - if there is gloop
main loop routine in the stack (see RunLoopGLib), the
GC root search will not go deeper that this frame. This
is enough to make GC work when invoked synchronously,
via VM::shrinkFootprintWhenIdle; therefore for the
workaround to work, wpe plugin needs to call that
(another part of the workaround is implemented in
WebKitBrowser plugin)

* ARRISEOS-45892 limit conservative scan range

Small correction: immediate garbageCollectNow is
still useful, since under heavy load shrinkFootprintWhenIdle
might be executed quite late.
varadharajan-v pushed a commit to LibertyGlobal/WPEWebKit that referenced this issue Aug 21, 2024
* ARRISEOS-45892 limit conservative scan range

Turns out that conservative GC root scanning is
scanning too deep into the stack & this results
in GC being prevented (for more details see the
ticket & upstream issue:
WebPlatformForEmbedded#1363

This workaround introduces 'stack guards' that prevent
this in particular cases, that is - if there is gloop
main loop routine in the stack (see RunLoopGLib), the
GC root search will not go deeper that this frame. This
is enough to make GC work when invoked synchronously,
via VM::shrinkFootprintWhenIdle; therefore for the
workaround to work, wpe plugin needs to call that
(another part of the workaround is implemented in
WebKitBrowser plugin)

* ARRISEOS-45892 limit conservative scan range

Small correction: immediate garbageCollectNow is
still useful, since under heavy load shrinkFootprintWhenIdle
might be executed quite late.
tomasz-karczewski-red added a commit to LibertyGlobal/WPEWebKit that referenced this issue Oct 15, 2024
* ARRISEOS-45892 limit conservative scan range

Turns out that conservative GC root scanning is
scanning too deep into the stack & this results
in GC being prevented (for more details see the
ticket & upstream issue:
WebPlatformForEmbedded#1363

This workaround introduces 'stack guards' that prevent
this in particular cases, that is - if there is gloop
main loop routine in the stack (see RunLoopGLib), the
GC root search will not go deeper that this frame. This
is enough to make GC work when invoked synchronously,
via VM::shrinkFootprintWhenIdle; therefore for the
workaround to work, wpe plugin needs to call that
(another part of the workaround is implemented in
WebKitBrowser plugin)

* ARRISEOS-45892 limit conservative scan range

Small correction: immediate garbageCollectNow is
still useful, since under heavy load shrinkFootprintWhenIdle
might be executed quite late.
tomasz-karczewski-red added a commit to LibertyGlobal/WPEWebKit that referenced this issue Oct 15, 2024
Turns out that conservative GC root scanning is going too deep into the stack
and this may prevent the collection of objects - see:

WebPlatformForEmbedded#1363

The change introduces 'stack guards' that prevent this in particular cases,
that is - if there is gloop main loop routine in the stack the GC root stack search
will not go deeper that this frame. This avoids some 'false positives' conservative roots
that can make the app hold a lot of memory.
@eocanha
Copy link
Member

eocanha commented Oct 24, 2024

I've tried with some upstream changes, like this one:
#1285

Appart from any other remediations you might try, please try also 22b825d, coming from #1400.

tomasz-karczewski-red added a commit to LibertyGlobal/WPEWebKit that referenced this issue Oct 28, 2024
Turns out that conservative GC root scanning is going too deep into the stack
and this may prevent the collection of objects - see:

WebPlatformForEmbedded#1363

The change introduces 'stack guards' that prevent this in particular cases,
that is - if there is gloop main loop routine in the stack the GC root stack search
will not go deeper that this frame. This avoids some 'false positives' conservative roots
that can make the app hold a lot of memory.
suresh-khurdiya-infosys pushed a commit to LibertyGlobal/WPEWebKit that referenced this issue Nov 28, 2024
* ARRISEOS-45892 limit conservative scan range

Turns out that conservative GC root scanning is
scanning too deep into the stack & this results
in GC being prevented (for more details see the
ticket & upstream issue:
WebPlatformForEmbedded#1363

This workaround introduces 'stack guards' that prevent
this in particular cases, that is - if there is gloop
main loop routine in the stack (see RunLoopGLib), the
GC root search will not go deeper that this frame. This
is enough to make GC work when invoked synchronously,
via VM::shrinkFootprintWhenIdle; therefore for the
workaround to work, wpe plugin needs to call that
(another part of the workaround is implemented in
WebKitBrowser plugin)

* ARRISEOS-45892 limit conservative scan range

Small correction: immediate garbageCollectNow is
still useful, since under heavy load shrinkFootprintWhenIdle
might be executed quite late.
suresh-khurdiya-infosys pushed a commit to LibertyGlobal/WPEWebKit that referenced this issue Dec 10, 2024
* ARRISEOS-45892 limit conservative scan range

Turns out that conservative GC root scanning is
scanning too deep into the stack & this results
in GC being prevented (for more details see the
ticket & upstream issue:
WebPlatformForEmbedded#1363

This workaround introduces 'stack guards' that prevent
this in particular cases, that is - if there is gloop
main loop routine in the stack (see RunLoopGLib), the
GC root search will not go deeper that this frame. This
is enough to make GC work when invoked synchronously,
via VM::shrinkFootprintWhenIdle; therefore for the
workaround to work, wpe plugin needs to call that
(another part of the workaround is implemented in
WebKitBrowser plugin)

* ARRISEOS-45892 limit conservative scan range

Small correction: immediate garbageCollectNow is
still useful, since under heavy load shrinkFootprintWhenIdle
might be executed quite late.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

No branches or pull requests

5 participants