[Bug]: When opening Ballerina projects with large ballerina files the VS Code plugin crashes

[ayeshLK]

Description

$subject

Following error log was found in the VS Code logs:

[Error - 5:44:40 PM] Stopping server failed
Error: Stopping the server timed out
    at /home/ayesh/.vscode/extensions/wso2.ballerina-4.7.2/dist/extension.js:2:1065110
[Error - 5:44:40 PM] Stopping server failed
Error: Stopping the server timed out
    at /home/ayesh/.vscode/extensions/wso2.ballerina-4.7.2/dist/extension.js:2:1065110
[Error - 6:03:11 PM] Client Ballerina LS Client: connection to server is erroring. Shutting down server.
[Error - 6:03:11 PM] Client Ballerina LS Client: connection to server is erroring. Shutting down server.
[Error - 6:13:32 PM] Client Ballerina LS Client: connection to server is erroring. Shutting down server.
[Error - 6:13:32 PM] Client Ballerina LS Client: connection to server is erroring. Shutting down server.
[Error - 6:13:32 PM] Client Ballerina LS Client: connection to server is erroring. Shutting down server.
[Error - 6:13:32 PM] Client Ballerina LS Client: connection to server is erroring. Shutting down server.
[Error - 6:13:32 PM] Client Ballerina LS Client: connection to server is erroring. Shutting down server.
[Error - 6:13:51 PM] Client Ballerina LS Client: connection to server is erroring. Shutting down server.
[Error - 6:13:51 PM] Client Ballerina LS Client: connection to server is erroring. Shutting down server.
[Error - 6:13:51 PM] Client Ballerina LS Client: connection to server is erroring. Shutting down server.
[Error - 6:13:51 PM] Client Ballerina LS Client: connection to server is erroring. Shutting down server.
[Error - 6:13:51 PM] Client Ballerina LS Client: connection to server is erroring. Shutting down server.
[Error - 6:13:51 PM] Client Ballerina LS Client: connection to server is erroring. Shutting down server.
ballerina: Oh no, something really went wrong. Bad. Sad.

We appreciate it if you can report the code that broke Ballerina in
https://github.com/ballerina-platform/ballerina-lang/issues with the
log you get below and your sample code.

We thank you for helping make us better.

[Error - 6:14:11 PM] Client Ballerina LS Client: connection to server is erroring. Shutting down server.
[2024-07-12 18:14:11,133] SEVERE {b7a.log.crash} - Java heap space 
java.lang.OutOfMemoryError: Java heap space
 
[Error - 6:14:11 PM] Client Ballerina LS Client: connection to server is erroring. Shutting down server.
[Error - 6:14:11 PM] Client Ballerina LS Client: connection to server is erroring. Shutting down server.
[Error - 6:14:11 PM] Client Ballerina LS Client: connection to server is erroring. Shutting down server.
[Error - 6:14:11 PM] Client Ballerina LS Client: connection to server is erroring. Shutting down server.
[Error - 6:14:11 PM] Client Ballerina LS Client: connection to server is erroring. Shutting down server.
[Error - 6:14:11 PM] Client Ballerina LS Client: connection to server is erroring. Shutting down server.
[Error - 6:14:11 PM] Client Ballerina LS Client: connection to server is erroring. Shutting down server.
[Error - 6:14:11 PM] Client Ballerina LS Client: connection to server is erroring. Shutting down server.
[Error - 6:14:11 PM] Client Ballerina LS Client: connection to server is erroring. Shutting down server.
[Error - 6:14:11 PM] Client Ballerina LS Client: connection to server is erroring. Shutting down server.
[Error - 6:14:11 PM] Client Ballerina LS Client: connection to server is erroring. Shutting down server.
[Error - 6:14:11 PM] Client Ballerina LS Client: connection to server is erroring. Shutting down server.
[Error - 6:14:51 PM] Client Ballerina LS Client: connection to server is erroring. Shutting down server.

Following heapdump was found in the project [1]

[1] - https://drive.google.com/file/d/18dKWEe9Yzu63VqpplLQK-aH1lf102Lcp/view?usp=sharing

Steps to Reproduce

Try opening the client.bal in the following project using VS Code.

stripe_new.zip

Affected Version(s)

No response

OS, DB, other environment details and versions

No response

Related area

-> Compilation

Related issue(s) (optional)

No response

Suggested label(s) (optional)

No response

Suggested assignee(s) (optional)

No response

[nipunayf]

Related to #42612

[ayeshLK]

In addition to the above when I import the stripe package into another ballerina project (which only has few lines of code), and try to open it via VS Code the Ballerina VS Code plugin crashes with a similar behaviour. IMO this is a critical issue which we should fix as soon as possible.

[ayeshLK]

When divide the types.bal into multiple files [1] the LS OOM was fixed. But, when I try to locally publish this connector and use it in a Ballerina project the LS OOM still happens. The heap-dump can be found here [2]

I can see the following error notification in VS Code.

[1] - ballerina-platform/module-ballerinax-stripe#4
[2] - https://drive.google.com/file/d/1Fj5KrrA_u9w-AHygHCZDGoANlImzjNtx/view?usp=sharing

[ayeshLK]

@KavinduZoysa currently I am blocked in this issue and unable to resolve this [1] internal support ticket. Do we have any progress on this or do you guys have a workaround ?

[1] - https://github.com/wso2-enterprise/internal-support-ballerina/issues/719

[KavinduZoysa]

@ayeshLK, still we are trying to find the root cause for this issue. Sometimes we see a GSON array with the size of 300MB-500MB and sometimes, we observe the heap is mostly allocated for the project context. Currently, we are investigating both aspects to find the exact root cause. We will update on the progress.

As a workaround, we can divide the types.bal file into multiple files launch vs code. But if we import the same library in a different project, we are getting OOM.

[mindula]

In addition to what @KavinduZoysa has mentioned, we increased the memory to 2GB, and we didn't encounter the crash anymore.

[mindula]

I tried importing the Stripe module after building it locally and did not encounter any OOM issues. We are still investigating the OOM issue with large files and will provide updates on our progress.

[mindula]

When we analysed the heap dump, we identified an ArrayList with 2,927 elements (matching the number of types in the types.bal file) as the main suspect.

After examining the JsonObjects, we observed that they contained key value pairs representing the syntax tree, as shown in the below image.

After debugging the code, we discovered that these JsonObjects were created in the Diagram.util class. After commenting out the relevant lines and returning an empty json object instead, we no longer encountered the OOM issue, and all Language Server functionalities worked as expected.

It appears that the Diagram.util class represents every token with a JsonObject. @kanushka @hevayo

[kanushka]

After debugging the code, we discovered that these JsonObjects were created in the Diagram.util class. After commenting out the relevant lines and returning an empty json object instead, we no longer encountered the OOM issue, and all Language Server functionalities worked as expected.

It appears that the Diagram.util class represents every token with a JsonObject. @kanushka @hevayo

This JSON syntax tree is used to generate the Ballerina low-code diagram. I think we can deprecate this generation since the new Ballerina extension will use another LS API to generate the low-code diagram.

[kanushka]

This JSON syntax tree is used to generate the Ballerina low-code diagram. I think we can deprecate this generation since the new Ballerina extension will use another LS API to generate the low-code diagram.

I'm sorry, we cannot deprecate this generation since we are using this functionality in a few more places.

The initial language service shutdown issue will occur due to code lens generation logic. Ballerina extension uses the getSyntaxTree API and traverses visitors to find possible code lenses. Since the file is too bulky, getSyntaxTree will result in an out-of-memory condition.

If we can come up with a new API to handle the code lens requirement, we will be able to remove the initial call to the getSyntaxTree API.

However, we still need the syntax tree to work with the data mapper, service designer, and statement editor.