improve startup performance for language server

[martinlippert]

This is a summary issue item for discussing ideas, share measurements, and implement improvements to speed up the startup of the language server, primarily the spring-boot-language-server.

[martinlippert]

First thing that I noticed is this:
spring-projects/spring-boot#16655

As a result, we need to find a way to ship the language servers in an exploded format for the various clients and start from an exploded version of the fat jar.

[martinlippert]

Here are first results from @dsyer looking at this:

07:50:19.567 [LanguageServerApp-lifecycle] INFO o.s.i.v.c.l.LanguageServerRunner - Server ready to start after 1099 ms

That's with lazy flag on and with proxyBeanMethods=false on all the @Configuration classes (I only found a handful).
So that's about 2x faster than the first attempt.
Also with -XX:TieredStopAtLevel=1 -noverify (same as yesterday)
It's possible that it actually doesn't work with those settings (I did notice a BootJavaConfig that is a @Component and an InitializingBean - if nothing depends on that there is nothing to cause it to initialize).
But that can probably be worked around

[martinlippert]

That's with lazy flag on and with proxyBeanMethods=false on all the @Configuration classes (I only found a handful).

The proxyBeanMethods seems very reasonable for us. The lazy init is probably not super useful for us, since we not only care about the pure JVM and boot startup, but the startup time to get the server initialized and responding to the first request. And for that, I think we need most (if not all) of the beans.

Also with -XX:TieredStopAtLevel=1 -noverify (same as yesterday)

I am not sure about the -XX:TieredStopAtLevel=1. Doesn't it affect the performance in the long run, since the C2 compiler isn't been taken into account anymore? Would be interesting to measure the difference, since some of the language server operations are quite CPU intense (like parsing java source), so I would guess that the C2 compiler would have some positive effects on this (in the longer run). But maybe that doesn't matter a lot. Needs a good measurement session... :-)

[kdvolder]

@dsyer wrote:

It's possible that it actually doesn't work with those settings (I did notice a BootJavaConfig that is a @component and an InitializingBean - if nothing depends on that there is nothing to cause it to initialize).
But that can probably be worked around

I do beleave I have used (or abused?) Initializing bean's in a few places because I needed to trigger some code that needs to be executed after all injections are done. Those beans serve no other purpose than just to get the initialization code to run. So yes, I'm pretty sure that some things will be broken if those bits don't get executed.

[dsyer]

Ideally, you would want to mimic the servlets/MVC handlers: defer initialization until the first request comes in. Since I'm not 100% sure how that happens in this app it's hard to know exactly where to put the extra abstraction. There must be some background threads that handle the I/O - those would be better places to initalize, rather than the main thread, I guess?

[dsyer]

Is it possible to switch on C2 (tiered complication or whatever it is) at runtime? Then we could start as fast as possible and then slide into a more efficient runtime once all the expensive class loading is out of the way.

Unrelated: probably it would be a good idea to use CDS (maybe squeeze out another 10-15%). Problem is that is JVM and platform dependent. Maybe we could ship a docker image instead of a jar file?

[kdvolder]

defer initialization until the first request comes in.

The first request that comes in would be actually part of the initialization process. This is a request from the LSP client to initialize the server. Until that request is handled, the server is not really 'initialized'. The client will only start sending 'real' requests after that. The 'initilialize' request will also come immediately after the server is launched. So deferring things until that happens is probably not that helpful.

The 'startup time' that we really care about is really upto the point where the initialization sequence, including the handling of the initialize request is completed.

That being said, from what I've seen the handling of the initialize request isn't the big time-sink. It only takes about 100ms. The bulk of the time seems to be spent creating / wiring / initializing beans.

There must be some background threads that handle the I/O - those would be better places to initalize, rather than the main thread, I guess?

There's no threads dedicated to IO specifically. There's a thread dedicated to reconciling. And a thread for handling most other requests. There's also a thread that lsp4j uses to dispatch request I think. I am sorry to say it's actually a bit of a mess and hard to keep track of how it really works.

Should also point out, this not too similar to a web-server since its really not meant to serve multiple clients/sessions. And there's not really that much to be gained by parallelizing things in multiple threads. Mostly... the server is expected to handle the requests exactly in the order received (because it matters for tracking the state of documents across document change events, for example).

[martinlippert]

As a first step I did:

• changed the bundle shape of the spring-boot-language-server to dir
• include the spring-boot-language-server in exploded format in the bundle
• startup the language server as regular java app (no boot launcher involved anymore) directly from the bundle language server directory
• set -noverify as JVM arg

We need to double check that this works across the different platforms, so @kdvolder please double check a distro build on linux, @aclement please double check the lates distro build on Windows - if possible. Best is to turn on the console log output in the preferences first, then open the first Java editor to see if the language server startups nicely.

Next performance optimizations to follow. I would be eager to learn more the proxyBeanMethods=false settings on the @Configuration classes as well as functional bean definitions. @dsyer are you continue to look into this as an experiment? would be happy to share some code (on a branch or PR, whatever you prefer).

[dsyer]

From what I can see you can just add the proxyBeanMethods=false flag to all (4) configuration classes (2 autoconfigs, one @SpringBootApplication and one SpringSymbolIndexerConfig). Seems to work for me anyway - but double check that there are no internal calls of @Bean methods (I didn't see any).

Functional beans is more work, and might not help much unless Spring Boot is functional too (e.g. by using the spring-init jars). I'll continue to look into it. Currently it isn't working, but maybe now that Kris has removed the duplicate bean definition it might make some progress.

[martinlippert]

@dsyer I don't see a big difference when adding the proxyBeanMethods=false to the two mentioned classes... :-(

[dsyer]

Not a surprise. But there are actually 4 config files - I guess you skipped the autoconfigs in the commons library?

[martinlippert]

@dsyer I added this to the other two classes, so the proxyBeanMethods=false setting is in across the place now. Still not a dramatic improvement in startup time from this, but every bit counts.

@dsyer I am really curious to see what the effect of switching to functional bean definitions will be.

[dsyer]

On master:

11:53:37.457 [LanguageServerApp-lifecycle] INFO  o.s.i.v.c.l.LanguageServerRunner - Server ready to start after 1361 ms

Functional beans:

11:45:51.088 [LanguageServerApp-lifecycle] INFO  o.s.i.v.c.l.LanguageServerRunner - Server ready to start after 1186 ms

Some improvement. Probably not worth the extra effort?

[dsyer]

I also found that excluding spring-boot-starter-logging and switching to slf4j-jdk14 saved up to 200ms. So that might be worth a try. The flame graph looks pretty clean apart from that all we can do is try to load fewer classes