Refactor the GDScript Language Server to improve maintainability

[HolonProduction]

Note

I'll be using LSP as short form of Language Server or LSP Server in this proposal, it isn't really correct but I find it more clear than LS, especially since Godot does not contain an LSP Client.

Describe the project you are working on

The GDScript Language Server

Describe the problem or limitation you are having in your project

The Godot LSP is not fully spec compliant, for example we are allowing multiple connections to the server, which the specification forbids¹. This makes the implementation of client capability tracking very challenging and prevents us from using modern LSP features, for example snippet insert mode.

Furthermore the LSP contains a lot of "glue" code with a lot of linkage between different endpoints, leading to hard to fix issues like seen in godotengine/godot#82948.

All LSP data types and their serialization are implemented by hand, making it hard to keep them up to date and to apply fixes to the serialization. E.g. the issue I fixed in godotengine/godot#96725 might still come back at us with other data types that make the same issue when serializing.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

The following covers some ideas of mine to rework our LSP implementation to be more closely aligned to the spec, to reduce maintenance burden in the future, and to make it possible to fix some outstanding issues that are not solvable in a good way at the moment.

1. Break apart LSP and GDScript:
   The LSP implementation will become an own module, it will contain all the LSP data types, serialization code and networking code. This would make it possible to reuse LSP data types in the future, for example to provide a server for GDShader. Parts of this module will consist of generated code. Since LSP version 3.17 there is a machine readable description of the specification, which in a first step will be used to generate the data types and serialization code². This will make maintenance of this part much easier and ensure consistency in serialization behavior.

2. Break apart the Language Server and Editor:
   At the moment, the LSP is running inside the Godot Editor and external editors connect to it. This approach makes it basically impossible to limit connections to 1 per server, since it would make working with the LSP a pain. To solve this the LSP will be transitioned to a more spec compliant approach³. It will be a separate process managed by the external editor. Godot will gain an --gdscript-lsp command line argument which will start a headless instance that is running the LSP. When this argument is passed, the initialization of Godot will be suspended until the initialization request for the LSP is received, this is needed to make the location of res:// configurable by the external editor, this would fix Editor tries to open files from totally unrelated project godot#92248.

3. Limit connections to 1 and implement client capabilities (this will unlock usage of modern LSP features)

4. Generate server stubs:
   The LSP meta model can also be used to generate a server implementation with the needed virtual methods to fill in. This will remove the dependency on the JSONRPC module, which from my understanding is the reason for the strangely exposed state of the LSP (See Cannot access the GDScriptLanguageProtocol in GDScript godot#86132). The current implementation requires the LSP implementation to be bound to Godot's Object system, and it was apparently not sufficiently marked as internal here. We now have a type GDScriptLanguageProtocol and a global variable GDScriptLanguageProtocol which is of type JSONRPC, I suspect this constellation for being responsible for the issue. With this proposal the type will be fully internal, the Global variable will be marked as deprecated. I don't see it as desireable to expose the LSP interface to scripting since it isn't compatible with the idea of one connection per server. If users request certain features we should look into making them accessible via the ScriptLanguage interface.

5. Move as much code as possible from the LSP to gdscript_editor, and reduce internal coupling between endpoints
   This will include deprecating and removing the "smart resolve" feature (first actionable step would be turning it off by default). The gdscript_editor implementations of features are good/smart enough and we should aim for consistency between internal and external editors.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

I'll just link to the last diagram of #10977 here. I'd like the LSP to be a binding layer that we keep as thin and maintainable as possible.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No

Is there a reason why this should be core and not an add-on in the asset library?

It might be possible to implement a server in this fashion as separate project, but it would kind of destroy the idea of consistency between internal and external editors.

Footnotes

1. See the last paragraph of the Language Server Protocol overview ↩

2. I have been prototyping with this a bit, the main problems still to solve at the moment: recursive data types require some special solution; I've been using std::variant for or types, since Godot's Variant doesn't really fit the job; The spec is based on typescripts type system so I'll still need a solid strategy to handle nullability ↩

3. https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#lifeCycleMessages ↩

[sockeye-d]

Instead of moving all the LSP code to the GDScript analyzer and making the LSP interface with that (this is what you're suggesting right?), is it possible to consolidate everything in be LSP module and make the script editor connect to the secondary process? That way, it could also connect to things like the C# server or other servers for GDExtension, then the built-in editor will work with all languages, provided you have the servers installed.

For C# there would be a modal like the export templates one to install it, for other languages you'd need to install them manually but then you can manage the connections in the same window as C#. This would also remove any parity issues between the gds language server and the built-in editor, because they're the same thing.

Is this bad for platform compatibility? Will this be impossible because Android doesn't support spawning multiple processes or stdin/out?

[HolonProduction]

Many of the most notable features the LSP offers are already implemented in gdscript_editor and used from the LSP. Making the Godot Editor use the LSP would require a huge amount of work to port those over. Breaking the LSP structures out of the GDScript module would be a step towards an LSP Client, but implementing that would be a huge amount of work as well, and it isn't needed to improve LSP maintenance and reduce spec compliance issues, which is the goal of this proposal.

For implementing an LSP Client see #2215. I don't think it has much todo with this proposal, an builtin LSP client could work perfectly fine with my proposed LSP changes. But those changes would also work perfectly fine with the current builtin editor.