SADL cannot highlight a SADL file in Eclipse 2022-03 with Xtext 2.26.0 #907
Comments
|
I think it is because of changed dependency to lsp4j 0.12.0, which now uses lsp 3.16. |
|
@dhuebner , I'm having a hard time finding what has changed in the "new Xtext 2.26 API" that is relevant to this problem. I'm not finding anything about how to replace ISemanticHighlightingStyleToTokenMapper. Can you point me to something to help me identify what has changed and how to use the new API? |
|
@crapo |
|
@crapo |
|
@dhuebner . this is very helpful. Thanks for sharing!!! |
|
Hi Andy @crapo, I am Paul Meng from GE Research. I am now part of Kareem's team. John Interrante @tuxji and I are working on integrating SADL/Eclipse with OSATE and came across this issue. The problem is that OSATE uses xText 2.27 and SADL is using xText 2.25.0, which causes the incompatible issue. So I wonder if you are planning to resolve this anytme soon? |
|
Hi Paul @baoluomeng , as you can see from the comments on this issue, I started looking into it when @tuxji reported it last March. However, I was unable at that time to get a clear idea of what had changed and how to accomplish the SADL code changes to go to a new API with confidence. Over much of the life of SADL, we have brought in consultants who were Xtext experts to aid in making the most of Xtext's capabilities. Initially we used itemis and once Sven Efftinge left itemis and started TypeFox, we used TypeFox. This had enormous benefits--it allowed us to focus on the semantic modeling aspects while getting the best out of Xtext at a much lower cost than if we had tried to leverage Xtext on our own. I would recommend the same approach to this problem. I'd be happy to work with an Xtext expert consultant to make sure that the modeling capability of SADL is preserved and maybe enhanced and that the modifications necessary to go to the new semantic highlighting API are robust. Akos Kitta at TypeFox has deep knowledge of how Xtext is used in SADL. I don't know if there is any chance of contracting for some of his time, but I believe it would be by far the best approach. Please let me know if this is a possible direction. |
|
Hi Andy and Paul! I'd like to inform you about changes at TypeFox in the recent years. Sven and Akos are no longer at TypeFox – a part of the team moved to the new company Gitpod when it was founded in 2020. I am now leading TypeFox together with @danieldietrich; see also this blog post. We still have Xtext experts, though, @dhuebner being one of them. So continuing support for SADL is generally possible. But I'd also like to highlight an alternative: Langium (langium.org). We started that project in 2021 as a successor of Xtext in order to get back to a high development speed and become compatible with the web platform. The main difference is that Langium is fully built on TypeScript; it can run both in desktop IDEs / backends (Node.js) and web apps / frontends. If you'd like to discuss more in depth: miro.spoenemann@typefox.io |
|
I took another look at this issue by downloading the latest Eclipse for DSL, 2022-12 and running the Xtext examples. It doesn't appear that any of them implement semantic highlighting or any explicit highlighting functionality. Nor do the tutorials give examples. The example that @dhuebner referenced above, see TypeFox/yang-lsp@4428d19, is certainly helpful, but doesn't give the same level of understanding and confidence that a good tutorial might provide. The information at https://www.eclipse.org/Xtext/documentation/340_lsp_support.html has a link for Semantic Tokens (aka Semantic Highlighting), but once again the high-level discussion of how semantic highlighting is now implemented, and how that relates to the prior implementation, is, as nearly as I can tell, entirely missing. @spoenemann , as far as langium use is concerned, I can imagine that this would be a better technology and make Web SADL and Eclipse SADL more compatible, but I can only imagine that moving SADL and its extension languages and IDEs to an entirely new platform, with a different parser, would be a major undertaking. The architecture of SADL was modified by @kittaakos to support both Web SADL and Eclipse SADL, but at the beginning of efforts to support both in the same Xtext-based implementation. I suspect this work makes the modification to support the new semantic highlighting API a little more complex. @baoluomeng , I assume that you are following this conversation... |
|
I was traveling. Yes Andy, I am following this conversation. @crapo |
|
I have updated SADL to Tycho 3.0.4, Java 17, and Eclipse 2022-12 in PR #1063. Please test the latest build artifacts and let me know if any important SADL functionality has been affected or not. |
|
I'm curious to know if anyone has a long-term plan for semantic highlighting in SADL. SADL without semantic highlighting seems to me to be much less appealing. The Xtext move to LSP and the introduction of the replacement platform Langium poses significant challenges. The long-term support of Xtext is in question as documented in this Xtext issue: eclipse-xtext/xtext#1721. I do see that some tools for moving applications from Xtext to Langium have been developed since this issue was first created: see https://www.typefox.io/blog/xtext-to-langium. @tuxji , @baoluomeng , can you share information about the current level of usage of SADL and commitment to maintaining a semantic highlighting capability in SADL? |
|
I don't want to spend any more time than necessary updating SADL to work with the latest Java, Xtext, and Eclipse. SADL still does some syntactic coloring (highlighting) of the text when you edit a SADL file in Eclipse. To me, the SADL file still seems to look the same in Eclipse as it did before. I am hoping that whatever additional functionality semantic highlighting provides (I'm not familiar with what semantic highlighting does) will not be missed that much, but if we get feedback and comments from the RACK development team that SADL is much less appealing without the semantic highlighting, I can try to emulate the same two commits that Dennis (@dhuebner) made to TypeFox/yang-lsp. Let's try the current changes and get some feedback from people who currently use SADL like the RACK development team. |
|
With semantic highlighting (e.g., prior to your latest removals, a simple model looks like this: Using the build artifacts you reference above, the same model looks like this; In other words, semantic highlighting is what gives color coding to classes, properties of various types, instances, variables, and a few other things. |
|
Yes, I now see the effect of losing the semantic highlighting functionality and the loss of the color coding does seem significant. Thanks for showing me the difference. |
|
@tuxji , I don't know if you've gotten around to thinking about semantic highlighting re-implementation in SADL, but I've been doing a little reading on language servers and want to just highlight (no pun intended) a couple of things.
What are you thinking is the best approach? I'm thinking that we should consider implementing a language server for SADL in Java and it can then be used by both Eclipse-based SADL and WebSADL, which can be updated if desired to use the latest version of VSCode. |
|
There is a language server I believe, used by WebSADL, implemented by Akos, and in Typescript I think. That was before LSP supported semantic highlighting. That's why I think maybe re-implementing in Java might be the way to go. |
|
@tuxji 
|
|
After repeated failed attempts, I have finally successfully implemented the example in the Xtext LSP-Support document. I submitted a PR on that document to fill in a little of the missing information. This demonstrates using Eclipse LSP4J and LSP4E. I think that the existing SADL code base has everything necessary for using LSP for code completion, semantic highlighting, quick fix, folding, etc. It's a question of refactoring everything to access the correct pieces of implemented capability through the language server. There is also the question of the client. The LSP-Support example uses the Eclipse Generic Editor. Presumably we could continue to use an Eclipse Xtext editor. Or we could use Theia for both the fat client and the Web client. As I understand it, if one did that there is little effort to support VSCode as the client as well. @dhuebner , @tuxji , does this all sound accurate? @tuxji , have you done any work in this area since we last discussed it? |
|
Hi @crapo , I have kept some browser tabs around ready for when I come back to this area, but I have not had time to do any work with LSP support in SADL yet. However, I will have time in December to help add LSP support to SADL, so let's start working together on that effort. SADL uses the Xtext editor for editing SADL files, so let's do just enough refactoring and programming to support LSP using the Xtext editor. Let's not concern ourselves with Theia and VS Code right now. |
|
Hi @tuxji . I have been trying to get my head around Language Servers in Eclipse in general and Xtext in specific. Among other things, I worked on the example at https://eclipse.dev/Xtext/documentation/340_lsp_support.html. This is missing a critical piece of information--what to do with the TextMate json content. I finally figured out that one sames it as a file (...tm.json) and then adds that to the TextMate Grammar in Preferences. I submitted a PR which was approved but hasn't been incorporated. You can see the relevant screenshot here. I finally got this example working as expected and have then tried to expand upon it to have a custom LS instead of the generic one, but so far haven't quite got that to work. I guess I don't understand how the binding works so that I can bind the custom LS so it gets used. This link also seems to be very relevant. There are lots of pieces already in the SADL code base. In particular, in the ...sadl.ide project:
I've looked at lots of other stuff, but haven't found anything else that seems directly useful. Please let me know if the solution to any of these problems is obvious to youi. |
|
I googled "Unknown extension point" and someone said to double check that the plugin(s) defining the extension point are dependencies in your plugin.xml's accompanying MANIFEST.MF. I ran "git pull" in my sadl checkout hoping to see your changes in a branch so I could troubleshoot further, but you haven't pushed them to the repo yet. My only other advice (besides pushing your changes) is to look at some of the projects mentioned in tm4e's README.md as using tm4e in case one can serve as a good role model/example to emulate. Also, https://idiomaticsoft.com/post/2022-04-25-lsp-xtext-tutorial-intro/ may be worth a look. |
|
@tuxji , let's pursue this first with a minimal example. I've tried to move the result of the example at https://eclipse.dev/Xtext/documentation/340_lsp_support.html to a plug-in environment but have some issues. I've pushed this minimal example to LS-Exploration branch as a set of nested projects parallel to sadl3 in the github repo. This set of project builds with cmd-line Maven without error. However, I can't get the Eclipse environment to resolve LSP4J references. Possibly related, I don't understand why I don't have any Maven Dependencies listed in the Project Explorer. The quick fix on the unresolved references is to add to the target platform, but when I pick that and then pick LSP4E to add, nothing happens. If you can help me get these issue sorted out, then we can work on follow-on issues:
Then I can work on customizing the LS. This will give us a working example to guide the implementation in SADL. |
|
@crapo , I read https://eclipse.dev/Xtext/documentation/340_lsp_support.html and I do not see a link to download source code; rather, I see instructions to write source files with some copy and paste. Is that how you created the source files in the new org.example.domainmodel.parent directory, by following these steps manually in Eclipse? In particular, I want to know where the Maven pom files came from (did Eclipse generate them when you created the projects)? Ultimately, I'm looking to either compare the source files in org.example.domainmodel.parent against a set of original source files or follow the same steps you followed while I make improvements along the way. |
|
@tuxji , the instructions in 340_lsp_support.html do not create Eclipse plug-ins but rather just add an Xtext grammar and the default LS to the current Eclipse environment. (See the first screenshot in that documentation.) The instructions send you off to another tutorial to follow to get the contents of the grammar. I'm not aware of any source available for the project. The pom files are created by the new Xtext project wizard I guess. Following these instructions, I did get everything working in the original Eclipse environment. In other words, you create a .dmodel file right in the development eclipse, you don't run a new Eclipse with developed plug-ins. However, for our minimal example we want the new Xtext project wizard to create plug-ins so I followed these directions as closely as I could but chose "Eclipse plug-in" in the Advanced Xtext Configuration dialog shown in the first screenshot. I then copied over the several classes from the working project created when I followed the directions exactly, and some classes that I created to provide a customized LanguageServer, namely:
As well as the TextMate Grammar file:
The poms were, once again, created by the new project wizard, Maven being chosen as the preferred build system. I also added LSP4E by through the Help -> Install New Software..., using https://download.eclipse.org/lsp4e/releases/latest/ as the installation site. I checked the first three options but also tried with all four options checked. Note: I found and fixed some typos in names in writing this comment, so I suggest pulling changes from github in order to keep things consistent. |
|
@tuxji , just to confirm that we're on the same page, I'm hoping that you can help get the architecture understood and working. Once that is in place I should be able to use my knowledge of the codebase for semantic highlighting, content assist, renaming, etc, to get all necessary functionality moved over to the LS. |
|
OK, I'll make some time next week to help get a LS project working. |
Xtext 2.26.0 appears to have changed or removed some highlighting API used by SADL. When SADL is installed in Eclipse 2021-12 with Xtext 2.25.0, it can open and highlight a SADL file. When SADL is installed in Eclipse 2022-03 with Xtext 2.26.0, it encounters exceptions when trying to open and highlight a SADL file.
The text was updated successfully, but these errors were encountered: