Extension host terminated unexpectedly for Golang project #109

CxRP · 2021-07-16T11:08:12Z

Describe the bug
Opening a Golang project in Visual Code with jFrog extension enabled causes notification message of "Extension host terminated unexpectedly".
Restarting the Extension Host makes no difference.
Opening Developer Tools and looking at the console there are the following messages:

 WARN UNRESPONSIVE extension host, 'JFrog.jfrog-vscode-extension' took 95% of 5622.04ms, saved PROFILE here: 'file:///c%3A/Users/{username}/AppData/Local/Temp/exthost-482aef.cpuprofile'

Array(2)
0: id: "gc", percentage: 5, total: 295823
1: id: "JFrog.jfrog-vscode-extension", percentage: 95, total: 5325888

and

<--- Last few GCs --->
[20552:0000028200000000]    98829 ms: Scavenge (reduce) 3617.3 (3949.9) -> 3616.7 (3949.9) MB, 4.9 / 0.0 ms  (average mu = 0.803, current mu = 0.599) allocation failure
[20552:0000028200000000]    99409 ms: Scavenge (reduce) 3617.4 (3949.9) -> 3616.8 (3949.9) MB, 8.2 / 0.0 ms  (average mu = 0.803, current mu = 0.599) allocation failure
<--- JS stacktrace --->
FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory
<--- Last few GCs --->
[20552:0000028200000000]    98829 ms: Scavenge (reduce) 3617.3 (3949.9) -> 3616.7 (3949.9) MB, 4.9 / 0.0 ms  (average mu = 0.803, current mu = 0.599) allocation failure
[20552:0000028200000000]    99409 ms: Scavenge (reduce) 3617.4 (3949.9) -> 3616.8 (3949.9) MB, 8.2 / 0.0 ms  (average mu = 0.803, current mu = 0.599) allocation failure
<--- JS stacktrace --->
FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory

and

ERR Extension host terminated unexpectedly. The following extensions were running:
vscode.microsoft-authentication, 
vscode.debug-auto-launch,
vscode.git,
JFrog.jfrog-vscode-extension,
golang.go,
ms-vscode.js-debug,
vscode.github,
vscode.github-authentication,
vscode.emmet, vscode.merge-conflict,
ms-vscode-remote.remote-wsl-recommender,
vscode.testing-editor-contributions,
dbaeumer.vscode-eslint,
eamodio.gitlens,
ms-vscode-remote.remote-wsl

To Reproduce
Open Golang project of ~188 .go files and ~27 third party imports referenced in the go.mod. The go.sum file is populated and valid.

Expected behavior

Screenshots

Versions

JFrog VS-Code extension version: 1.8.0
JFrog VS-Code extension operating system: Windows 10 21H1
JFrog Xray Version: JFrog Cloud
VS-Code version: 1.58.2
Golang version: 1.16.6

Additional context
The extension works fine with a very small Golang project of 5 files.
Running on i7 1.99GHz laptop with 40GB RAM

The text was updated successfully, but these errors were encountered:

CxRP · 2021-07-16T16:15:43Z

Looking in the exthost-482aef.cpuprofile file, there are:

490 nodes
40,753 samples
40,753 timeDeltas

CxRP · 2021-07-19T13:14:49Z

Another example:

<--- Last few GCs --->
[29088:0000103300000000]   104661 ms: Scavenge (reduce) 3616.3 (3949.3) -> 3615.6 (3949.3) MB, 7.9 / 0.0 ms  (average mu = 0.823, current mu = 0.629) allocation failure
[29088:0000103300000000]   104678 ms: Scavenge (reduce) 3616.7 (3949.5) -> 3616.1 (3949.5) MB, 7.6 / 0.0 ms  (average mu = 0.823, current mu = 0.629) allocation failure
[29088:0000103300000000]   104691 ms: Scavenge (reduce) 3616.9 (3949.5) -> 3616.2 (3950.0) MB, 5.1 / 0.0 ms  (average mu = 0.823, current mu = 0.629) allocation failure
<--- JS stacktrace --->
FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory
1: 00007FF654FEDC26 node::Buffer::New+47318
2: 00007FF654FEDE25 node::OnFatalError+453
3: 00007FF656AED883 v8::Object::SlowGetInternalField+2035
4: 00007FF656AED491 v8::Object::SlowGetInternalField+1025
5: 00007FF655E52F83 v8::V8::ToLocalEmpty+8595
6: 00007FF6557E9A98 v8::Value::ToString+106312
7: 00007FF6557E95F0 v8::Value::ToString+105120
8: 00007FF652F1E2F5 v8::internal::OSROptimizedCodeCache::ClearEntry+3477
9: 00007FF652F1DFF5 v8::internal::OSROptimizedCodeCache::ClearEntry+2709
10: 00007FF652F1DC8D v8::internal::OSROptimizedCodeCache::ClearEntry+1837
11: 00007FF6550A34A4 uv_sleep+616532
12: 00007FF6550A37C2 uv_sleep+617330
13: 00007FF652F58295 uv_os_getpid+25925
14: 00007FF652F5753E uv_os_getpid+2251
15: 00007FF652F55F2B uv_os_getpid+16859
16: 00007FF652F53A2D uv_os_getpid+738
917: 00007FF652F53024 uv_os_getpid+4820
18: 00007FF652F4B8F8 v8::Extension::dependency_count+19400
19: 00007FF652F48130 v8::Extension::dependency_count+5120
20: 00007FF65564A6F4 Cr_z_deflate+76244
21: 00007FF6557E4E61 v8::Value::ToString+86801
22: 00007FF6557E4C83 v8::Value::ToString+86323
23: 00007FF65342D5D7 v8::internal::OSROptimizedCodeCache::InitializeEntry+21415
24: 00007FF655AE511C v8::ObjectTemplate::New+35900
25: 00007FF655AE4B5B v8::ObjectTemplate::New+34427
26: 00007FF655AE4A88 v8::ObjectTemplate::New+34216
27: 00007FF654DDB69C v8_inspector::protocol::Binary::operator=+464732
28: 0000103300154A38

485 nodes
37,977 samples
37,977 timeDeltas

yahavi · 2021-07-25T09:37:34Z

@CxRP,
Thanks for reporting this issue!
We are currently working to reduce the memory consumption of the extension.
We'll keep you updated.

yahavi · 2021-08-05T07:10:00Z

@CxRP,
JFrog VS-Code extension 1.8.1 is released. This version includes the changes in #111.
Feel free to upgrade.

There are more improvements planned for the next version (1.8.2 or 1.9.0). Some of them are -

Reduce the amount of memory that the extension consumes by storing the issues and licenses in the file system.
Improve the Go projects scanning algorithm.

We'd appreciate your feedback on that.

CxRP · 2021-08-06T10:24:14Z

@yahavi

I am still getting the same issue.

WARN UNRESPONSIVE extension host, 'JFrog.jfrog-vscode-extension' took 95% of 5715.404ms, saved PROFILE here: 'file:///c%3A/Users/{username}/AppData/Local/Temp/exthost-0c7416.cpuprofile'

509 nodes
35,906 samples
35,906 timeDeltas

"startTime": 149345623243
"endTime": 149351338647

Versions

JFrog VS-Code extension version: 1.8.1
JFrog VS-Code extension operating system: Windows 10 21H1
JFrog Xray Version: JFrog Cloud
VS-Code version: 1.59.0
Golang version: 1.16.6

yahavi · 2021-08-15T13:13:18Z

@CxRP,
JFrog VS-Code extension 1.8.3 is released. In this version, we reduced the loading time and extension size by 90%, and reduced the memory consumption of the extension.

Could you please check and let us know if the issue still reproduces in 1.8.3?
Thanks!

CxRP · 2021-08-16T10:09:53Z

@yahavi

I am still getting the same issue.

WARN UNRESPONSIVE extension host, 'JFrog.jfrog-vscode-extension' took 93% of 5534.349ms, saved PROFILE here: 'file:///c%3A/Users/{username}/AppData/Local/Temp/exthost-036db1.cpuprofile'

522 nodes
36,717 samples
36,717 timeDeltas

"startTime": 420839732254
"endTime": 420845266603

Versions

JFrog VS-Code extension version: 1.8.3
JFrog VS-Code extension operating system: Windows 10 21H1
JFrog Xray Version: JFrog Cloud
VS-Code version: 1.59.0
Golang version: 1.16.7

yahavi · 2021-08-16T12:46:11Z

@CxRP -
With the new version, do you experience anything else other than the above warning message? Does anything get stuck or crash for you?
Do you still see the JavaScript heap out of memory error?
Thanks!

CxRP · 2021-08-16T15:10:35Z

here's the current errors from the Developer Tools Console:

<--- Last few GCs --->
[23408:000072DE00000000]    88100 ms: Scavenge (reduce) 3644.5 (3950.0) -> 3643.8 (3950.0) MB, 3.7 / 0.0 ms  (average mu = 0.937, current mu = 0.896) allocation failure
[23408:000072DE00000000]    88116 ms: Scavenge (reduce) 3644.8 (3950.0) -> 3644.1 (3950.3) MB, 4.0 / 0.0 ms  (average mu = 0.937, current mu = 0.896) allocation failure
[23408:000072DE00000000]    88129 ms: Scavenge (reduce) 3645.2 (3950.5) -> 3644.6 (3950.5) MB, 4.2 / 0.0 ms  (average mu = 0.937, current mu = 0.896) allocation failure

<--- JS stacktrace --->
FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory

<--- Last few GCs --->
[23408:000072DE00000000]    88100 ms: Scavenge (reduce) 3644.5 (3950.0) -> 3643.8 (3950.0) MB, 3.7 / 0.0 ms  (average mu = 0.937, current mu = 0.896) allocation failure
[23408:000072DE00000000]    88116 ms: Scavenge (reduce) 3644.8 (3950.0) -> 3644.1 (3950.3) MB, 4.0 / 0.0 ms  (average mu = 0.937, current mu = 0.896) allocation failure
[23408:000072DE00000000]    88129 ms: Scavenge (reduce) 3645.2 (3950.5) -> 3644.6 (3950.5) MB, 4.2 / 0.0 ms  (average mu = 0.937, current mu = 0.896) allocation failure

<--- JS stacktrace --->
FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory

Here is the output in the jFrog window before crash:

[DEBUG - 16:03:47] Resolved JFrog platform URL: https://XXX.jfrog.io/
[DEBUG - 16:03:47] Resolved Xray URL: https://XXX.jfrog.io/xray
[DEBUG - 16:03:47] Resolved Artifactory URL: https://XXX.jfrog.io/artifactory
[INFO - 16:03:47] Starting quick scan
[INFO - 16:03:47] Locating go.mod files in workspace "master".

Screenshots

CxRP · 2021-08-18T08:47:27Z

@yahavi yahavi
FYI: I tested the MinIO repo and this has the same issues

yahavi · 2021-08-22T14:25:36Z

Thanks for the input, @CxRP.
We think we found the issue -
To build the dependency tree, we use the go mod graph command. This command list all packages transitively.
Not all listed packages are in use, for example:
A:1 -> B:1 -> ... -> A:2
In that case, Go may not build your package with A:2. Therefore there is no point In showing it in the Graph.

We found out that there are some root Go packages that have a very long dependency chain. In following packages list the parents are above and the children are below:

golang.org/x/oauth2:0.0.0-20210819190943-2bc19b11175f (direct dependency)
cloud.google.com/go:0.65.0                            (child of golang.org/x/oauth2:0.0.0-20210819190943-2bc19b11175f) 
cloud.google.com/go/storage:1.10.0                    (child of cloud.google.com/go:0.65.0)
cloud.google.com/go/bigquery:1.8.0                    (...)
cloud.google.com/go:0.57.0
cloud.google.com/go/storage:1.6.0
cloud.google.com/go:0.53.0
cloud.google.com/go/storage:1.5.0
cloud.google.com/go/pubsub:1.1.0
cloud.google.com/go/storage:1.0.0
cloud.google.com/go:0.46.3
cloud.google.com/go/pubsub:1.0.1
cloud.google.com/go:0.45.1
cloud.google.com/go/bigquery:1.0.1
google.golang.org/api:0.8.0
cloud.google.com/go:0.38.0

In the above example, the dependency tree has 16 levels. However, after running go list -m all, we found that cloud.google.com/go:0.57.0 is not in use at all. Therefore, this tree could be much smaller:

golang.org/x/oauth2:0.0.0-20210819190943-2bc19b11175f (root parent)
cloud.google.com/go:0.65.0
cloud.google.com/go/storage:1.10.0
cloud.google.com/go/bigquery:1.8.0

The solution is to make intersection between go mod graph and go list -m all.
We will start working on a fix soon and will keep you updated on that.

yahavi · 2021-08-31T06:56:38Z

@CxRP,
JFrog VS-Code extension 1.8.4 is released.
This version includes the suggested changes.
We'd appreciate your feedback on that.

CxRP · 2021-08-31T09:40:49Z

@yahavi thank you very much. this all seems to be working now.
no more crashes and results are coming back in the plug-in window :-)

yahavi · 2021-08-31T11:46:38Z

@CxRP, thanks for your feedback!

CxRP added the bug Something isn't working label Jul 16, 2021

yahavi self-assigned this Jul 25, 2021

yahavi mentioned this issue Aug 1, 2021

Bugfix: unresponsive extension #111

Merged

2 tasks

yahavi closed this as completed in #111 Aug 4, 2021

This was referenced Aug 8, 2021

Bundle plugin with Webpack #115

Merged

Don't hold components to scan after scan finished #116

Merged

yahavi reopened this Aug 15, 2021

yahavi mentioned this issue Aug 27, 2021

Filter out unused dependencies in Go #119

Merged

2 tasks

yahavi closed this as completed in #119 Aug 31, 2021

yahavi mentioned this issue Sep 30, 2021

Improve Go scan algorithm jfrog/ide-plugins-common#45

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extension host terminated unexpectedly for Golang project #109

Extension host terminated unexpectedly for Golang project #109

CxRP commented Jul 16, 2021 •

edited

Loading

CxRP commented Jul 16, 2021

CxRP commented Jul 19, 2021

yahavi commented Jul 25, 2021

yahavi commented Aug 5, 2021

CxRP commented Aug 6, 2021

yahavi commented Aug 15, 2021

CxRP commented Aug 16, 2021

yahavi commented Aug 16, 2021

CxRP commented Aug 16, 2021

CxRP commented Aug 18, 2021

yahavi commented Aug 22, 2021 •

edited

Loading

yahavi commented Aug 31, 2021

CxRP commented Aug 31, 2021

yahavi commented Aug 31, 2021

Extension host terminated unexpectedly for Golang project #109

Extension host terminated unexpectedly for Golang project #109

Comments

CxRP commented Jul 16, 2021 • edited Loading

CxRP commented Jul 16, 2021

CxRP commented Jul 19, 2021

yahavi commented Jul 25, 2021

yahavi commented Aug 5, 2021

CxRP commented Aug 6, 2021

yahavi commented Aug 15, 2021

CxRP commented Aug 16, 2021

yahavi commented Aug 16, 2021

CxRP commented Aug 16, 2021

CxRP commented Aug 18, 2021

yahavi commented Aug 22, 2021 • edited Loading

yahavi commented Aug 31, 2021

CxRP commented Aug 31, 2021

yahavi commented Aug 31, 2021

CxRP commented Jul 16, 2021 •

edited

Loading

yahavi commented Aug 22, 2021 •

edited

Loading