Switch Java infrastructure to JDT LS

[gorkem]

Following the proof of concept work in #5730, we have come up with a list of gaps that we need to fill in order to keep the current level of Java support in Che with a jdt.ls-based back end.

Endgame

Go criteria

• Performance Target: Che in Che
  Currently (with a fixed 1Gig memory for jdt.ls), The Che in Che scenario takes a very long time and fails at import in the end. There are two approaches to fix this problem:
  • test increasing memory for jdt.ls to find out if performance is ok with enough memory. Current work includes changes to allow to increase memory in a preference.
  • Harvest moderately low hanging fruits in m2e to improve scalability
• Quality target: Make Selenium tests work
  There is a suite of selenium tests that covers Java IDE stuff quite well. We need to update it to work with jdt.ls make it pass. Product management to decide whether any differences from "classic" behaviour we find is ok.

• Enable Running Selenium Tests on jdt.ls Branch in CI #8413
• Analyze Selenium Test Failures for jdt.ls #8414
• Adapt selenium tests for JDT LS #8466
• Address Selenium Test Merge Blockers #9591 Adress Selenium Test Blockers

• Testing
  We need to decide what kind of testing we need to do to make us confident about jdt.ls in Che.

Current top concerns

1. M2E scalability & stability
2. Progress reporting

PoC cleanup

• Implement language server per workspace.
  Normally, the Che LSP implementation starts one language server instance per project (vs. one instance for the whole workspace). This probably does not scale in the case of jdt.ls. We could manage a single LS instance for the whole workspace and have the per project instances act as proxies for the single LS.

• jdt.ls Use one LS per workspace #6368

• Extensive testing/fixing. I haven't spent any time on getting things correct in the Poc. When I was convinced that something worked in principle, I moved to the next topic.

• Productise jdt.ls plugin #6369

• Preferences for jdt.ls
  jdt.ls user preferences can be set for jdt.ls through the LSP. Che currently has not support for this.

• Implement User Preferences for jdt.ls #6733

Set up jdt.ls extension project

• Create jdt.ls extension #6670

Necessary bugfixes

• Open Symbol is Broken in jdt.ls Branch #8436
• Cleanup unused Java support #8334
• Cleanup unused Java support Part 2 #8666
• Need to remember LSP diagnostics for ALL files #6773
• Switch IDE LSP stuff to use URIs instead of paths #7852
• Fix job watching in "reimport projects" #8302 Fix job watching in "reimport projects"
• External Libraries not available before opeining a Java File #7794
• Encoding issue with jdt.ls javadoc #7674
• #8049
• A problem with creation of Plain java project #8061
• NPE when try to open implementation from external library #8162
• Quick documentation is broken when use java language server #8467
• No pom's diagnostics after reopening #8470
• Method parameters hints #8532
• Use custom options for formating #8572
• Dto Generation is broken for Formatter Options #8557
• Make it possible to close Proposals window via Esc #8628

References/Definitions

The current "usages" view has has two main differences to the LSP references and defintions views.

1. Snippets
   An excerpt of the source code around the usage is shown for each occurrence.
2. The usages are not shown in a flat list, but in a tree structure.

We will address this gap in two steps:

1. Implement the current usages view through the LSP references/definitions. Snippets for each usage will be extracted by a separate service in the workspace.

• Use jdt.ls for "usages" view #6682

2. Improve the generic LSP views provide the same features as the "usages" view: optional snippets and a plugin to parse the reference URL's to provide tree structure.

• Improve "references" and "declarations" LSP view #6683 (optional)

Implementations

There is no support for this in the LSP. Move the current code into jdt.ls che extension + make available as custom command.

• Port implementors to jdt.ls extension #6736

Refactoring/move

There is no support for this in the LSP. Take the code from Che and put it into the jdt.ls extension. Move to jdt.ls when convenient.
see eclipse-jdtls/eclipse.jdt.ls#363

• Implement Move Refactoring #6748

Refactoring/rename

jdt.ls supports this LSP call. However, the resulting workspace edit command does not support renaming or moving files, unlike the current rename support (see microsoft/language-server-protocol#272). Our preferred solution would be to get the LSP bug fixed. @gorkem is talking to MS on this topic. Otherwise we'd have to have a protocol extension (custom command) from jdt.ls that allows "full" rename refactorings.

• Rename Refactoring #8939

Organize Imports

There is no support for this in LSP. This is implemented in jdt.ls as a code action, but we'd need a protocol extension to call it explicitly.

• Expose a command to organize imports eclipse-jdtls/eclipse.jdt.ls#292
• Implement "organize imports" command #7881

Show external libraries

The current Java support shows "external libraries" like the jre libraries in the explorer. LSP has no support for this, so we'd have to extend jdt.ls with a suitable protocol. This would be a good opportunity to show maven dependencies and such in the libraries.

• Move support for "libraries" to Che jdt.ls extension #6751
  There is also Expose Java Classpath eclipse-jdtls/eclipse.jdt.ls#48, but it's probably easier to get this up an running in our own extension.

Configure classpath

Che has UI to manipulate the classpath for "simple" Java projects. It seems that the classpath is managed in a file in /.che/classpath that looks a lot like a JDT classpath file.
We will implement a custom command that talks to the jdt IProject and adds libraries/source folders.
Migration of current "Simple Java" project could be done via jdt.ls extension point or we could detect the case in Che and offer conversion.

• Add Support for "Simple Java" projects for jdt.ls #6913

Navigate File Structure

Che Java support shows a structured view of the syntactic structure of Java files ("Navigate file structure").

LSP allows to fetch the document symbols as a flat list. While symbol information can contain the name of the container, in a case like the above, a unique tree structure cannot be deduced from that information for the two inner classes named "bork" (because their parents are named the same).
So in order to correctly implement this, we must rely on the containment of source ranges on the symbols.
Also, "Navigate Structure" allows to show all inherited members, as well. There is no protocol for this in LSP, so we'd need to add an extension to jdt.ls.

• Extend textDocumentSymbol to return inherited symbols eclipse-jdtls/eclipse.jdt.ls#366
• Use LSP for file structure view. #6752

The idea is to implement two commands in jdt.ls, one with inherited members and one without.

Maven Plugin

While removing jdt stuff from Che, I found out that maven project type has things which rely on a maven server+jdt running in the maven server plugin:

Report progress on maven

The maven plugin reports progress on updating and resolving projects. Right now, the Java LS does not report any meaningful progress (It has a non-standard protocol extension for reporting status, though). Note that while inconvenient, lack of this feature would not be a loss of functionality.
@fbricon not quite sure how to report progress for multiple tasks.

• Provide Progress Notification eclipse-jdtls/eclipse.jdt.ls#404
• Progress UI for Language Servers #8412

Get effective pom

This should be available in jdt.ls and can be exposed via a protocol extension. Make custom command in jdt.ls Che extension & call m2e API to get the effective pom.xml.

• Implement "get effective pom" #6907

Error reporting for pom.xml

According to @fbricon, jdt.ls sends diagnostics for pom.xml. We can remove this support from the maven plugin.

• Check diagnostics in Maven plugin vs. jdt.ls #6908

Reimport Maven Projects

This sounds a lot like "update project" in M2E. Is this necessary?
Reimport project deletes the project dependencies from the local .m2 repo, as well. Need to port to a custom command and do the correct thing.

• Replace "reimport maven project" for m2e #6910

Java Debugger Plugin

• JavaDebuggerUtils converts from debugger Location to fully qualified class name
• JavaDebuggerUtils converts from jdi location to debugger location
  Put code into a jdt.ls extension custom command:

• Move Java Debugger Location Conversion to jdt.ls extension #6914

Testing plugins

plugin-testing-classpath

• search for test classes and methods within a project, package, compilation unit. Subclasses for JUnit and TestNG (TestNgRunner, JUnit4TestRunner) determine Test status by checking annotations.
• build runtime class path for a java project (including jars, output locations + recursively used projects)
  Again, move the code into jdt.ls Che extension & provide custom commands.

• Replace JDT search & AST use with jdt.ls implementation #6659

Simple Java Projects

Support for adding source folders and jars from the workspace. We would need to add support for this in jdt.ls. As far as I know, jdt.ls alread DOES handle preexisting Eclipse/Java projects, so the addition would only involve the creation/manipulation of the project classpath/setup
Find out whether we can tell jdt.ls to use the classpath file in ".che/classpath". If we can, we can reuse that location. Otherwise, need to implement some migration code (use a importer extension in jdt.ls). Need to move the code that initializes the java project (AbstractJavaInitHandler subclass) to the jdt.ls extension

• Add Support for "Simple Java" projects for jdt.ls #6913
• jdt.ls has to recognize existed Plain Java projects #8045

[gorkem]

/cc @fbricon

[tsmaeder]

Depends on: #6575

[fbricon]

Are Implementations (subtypes/supertypes) and Navigate File Structure any different? Seems to me the server simply needs to expose a type hierarchy API. It might look like different UIs on the client side but they look like variations of the same use case on the backend.

[bdshadow]

@fbricon yes, they're different. Navigate File Structure can thought about as an Outline view in Eclipse. And Implementations (subtypes/supertypes) is indeed a work of a type hierarchy API (Type Hierarchy view in Eclipse).

[gorkem]

@tsmaeder Created eclipse-lsp4j/lsp4j#163 for LSP4J that brings the proposed LSP changes for file level operations.