Investigate reported performance of JetBrains

[loujaybee]

We've had several reports of the JetBrains experience feeling slow, but don't currently have good data on the situation. It seems that the Gateway implementation experience degrades quite significantly when on a low internet connection, we should look into this issue to figure out whether it's something we can improve on our Gitpod infrastructure, or whether it's something to report back to JetBrains.

We also have the opportunity to leverage API's for the new performance indicator in Gateway

[loujaybee]

@iQQBot - Can you add more details here about the performance issues you were having with JetBrains? Any logs, data, insight about your current bandwidth, machine specs etc would all be useful when reporting back to JetBrains.

[iQQBot]

I think it now about Gitpod, Jetbrains Gateway itself is not a great experience, I tried to use a wired network to connect to my other server on my LAN, but the experience was still poor

It's hard for me to say what the detailed performance metrics differences are (ping is good ~0-10ms, cpu and memory usage is not high), but the experience does feel much worse compared to the local Jetbranins IDE, so it may be an overall problem, I'm currently using the 2021.3.2 version of Gateway

[felladrin]

@easyCZ mentioned he didn't felt any performance issues with Goland, so I opened Gitpod repo on Goland and IDEA, to compare the resource usage (using EAP version of those IDEs).
But the resources are pretty similar, and yet dev experience with Goland is smoother.
So I guess it's just Java/Kotlin language server that is slow.

[felladrin]

I can say that high latency makes the experience worse. The difference between having ping 60 and 350+ is noticeable.

[felladrin]

Another two tests we can do:

• Compare performance on IntelliJ IDEA when opening a large repo (Gitpod) vs a small one (Spring PetClinic).
• Compare two relatively same-size repos where one is written in Java and the other is written in Kotlin, to check if Kotlin language server is the bottleneck.

[iQQBot]

Another try! I use my self-hosted Gitpod with low latency, it works like local

[felladrin]

Another try! I use my self-hosted Gitpod with low latency, it works like local

Does it mean that the problems are due to high latency? Or Gitpod's server resources?

[buggtb]

I've been seeing some issues, but mine are a little different. My performance is okay because I stuck in a sed line to my dotfiles to fix it :) :

sed -i 's/Xmx2048m/Xmx4096m/g' /ide-desktop/backend/plugins/remote-dev-server/bin/launcher.sh

but I do have issues on occasion with key strokes just bring dropped which is increadibly irritating when you're having to go back and retype stuff extra slow to make it work. It doesn't have to have a high ping number or anything, it was happening yesterday, all the pings looked good but it was so sluggish. I'd happily take "slow to render" over "can't be bothered to even register the fact that you typed something" ;)

If I see it again I'll do my best to dump some logs, but I guess if its not accepting keystrokes there isn't much to show cause it doesn't know its doing anything wrong

[akosyakov]

In #9796 I'm adding an env var JETBRAINS_BACKEND_XMX. I wonder whether something like that can work or we would rather allow to configure it for each product individually via .gitpod.yml? Something like:

jetbrains:
  // default for all if not specified 2048m
  xmx: "2048m" 
  intellij:
    // override for intellij
    xmx: "4096m" 
  goland:
    // override for goland
    xmx: "1024m" 

or we have both env var INTELLIJ_XMX, GOLAND_XMX env vars which are taking precedence over .gitpod.yml?

[yaohui-wyh]

In #9796 I'm adding an env var JETBRAINS_BACKEND_XMX. I wonder whether something like that can work or we would rather allow to configure it for each product individually via .gitpod.yml? Something like:

jetbrains:
  // default for all if not specified 2048m
  xmx: "2048m" 
  intellij:
    // override for intellij
    xmx: "4096m" 
  goland:
    // override for goland
    xmx: "1024m" 

or we have both env var INTELLIJ_XMX, GOLAND_XMX env vars which are taking precedence over .gitpod.yml?

Specify JVM max heap usage with -XX:MaxRAMPercentage vm options might be a better solution for containerized env?

Some potential problems with using -Xmx

1. If the Gitpod workspace memory limit is less than the -Xmx size, it would cause intelliJ backend server OOM and workspace crash
2. -Xmx4g might still not enough for some large project, and the fixed -Xmx parameter limit the heap size even if there's free memory available

---

For a Gitpod workspace with 12g memory limit, a -XX:MaxRAMPercentage=50.0 would allow the ide server use up to 6g max heap size. So -XX:MaxRAMPercentage might be a better VM settings for compatibility (with different workspace limit configuration) and ide server performance

gitpod /workspace/spring-petclinic (master) $ cat /sys/fs/cgroup/memory/memory.limit_in_bytes 
12884901888
gitpod /workspace/spring-petclinic (master) $ unset JAVA_TOOL_OPTIONS && java -XX:MaxRAMPercentage=50 -XshowSettings:vm -version
VM settings:
    Max. Heap Size (Estimated): 6.00G
    Using VM: OpenJDK 64-Bit Server VM

openjdk version "11.0.13" 2021-10-19 LTS
OpenJDK Runtime Environment Zulu11.52+13-CA (build 11.0.13+8-LTS)
OpenJDK 64-Bit Server VM Zulu11.52+13-CA (build 11.0.13+8-LTS, mixed mode)

[yaohui-wyh]

Some POC around modifying intellij backend server vmoptions in a Gitpod workspace:

1. modify the vmoptions:
   sed -i "s/-Xmx2048m/-XX:MaxRAMPercentage=50.0/g" /ide-desktop/backend/bin/idea64.vmoptions
2. restart the backend server by killing /ide-desktop/startup.sh (and supervisor would restart the script when the readinessProbe failed)
   ps auxf /* find the pid */ && kill -9 <pid>
3. start the JB client by gateway and check the "Connection details" info (max heap size is configured as expected):

[akosyakov]

I'm not sure about MaxRAMPercentage. cgroup limit is not really max memory which user can get, i.e. on free plan 20 users compete for 64Gb with limit of 12Gb in cgroup, on average they can get 3.2Gb, but MaxRAMPercentage of 50% will overcommit Java memory for no good reason from the start.

[yaohui-wyh]

I'm not sure about MaxRAMPercentage. cgroup limit is not really max memory which user can get, i.e. on free plan 20 users compete for 64Gb with limit of 12Gb in cgroup, on average they can get 3.2Gb, but MaxRAMPercentage of 50% will overcommit Java memory for no good reason from the start.

Yeah I understand. IMHO I just wanted to suggest -XX:MaxRAMPercentage as a better option comparing to a fixed size -Xmx<some-large-number> for a containerized env.

Actually the adjustment to -Xmx2g option by JetBrains IDE server scripts could be enough for most cases. Since Desktop IDE default -Xmx is only 750m (src file) while desktop IDE has much more memory cost like UI rendering.

For some self-hosted solution (such as if the oversold level for memory is 10users/64G), maybe users can try setting -XX:MaxRAMPercentage (preferred) or enlarge -Xmx

[akosyakov]

from @aledbf:

is there any plan to check/verify (or something) Java environment variables to avoid "invalid" values?
This is an example that sets -Xmx to 13GB. That will work, but if you have more things running, it could throw an OOM error and also terminate the workspace
https://twitter.com/nilshartmann/status/1537149455392284673

[loujaybee]

Closing this issue now, it's served it's purpose and has spawned many more specific performance issues to be looked into.

A brief summary of (some of) those issues here:

Better options for workspace resources:

• Epic: Optimised workspace class for Java / JetBrains IDEs #11152
• Epic: High-performance workspace classes on SaaS #10861

This is dependent upon:

• Epic: Usage-Based Pricing #9036
• Epic: Two Workspace Classes on SaaS #8261

Some performance issues are down to Gitpod region support e.g.

• Epic: more regional workspace clusters #7489
• Gitpod in Mumbai region #6139
• Gitpod Singapore (sg) region #5534

Fully supporting prebuilds + JetBrains should also make a difference here:

• Epic: JetBrains Prebuilds for Maven #6740

For debugging, we've introduced:

• Expose perf details for the current workspace #9521 (gp top)
• Integrate with JetBrains backend control center #10580

Also, please see epic for IntelliJ IDEA GA:

• Epic: JetBrains (IntelliJ IDEA) - GA #7956