GraphQL Language Service IntelliJ plugin

[asiandrummer]

Hi @jimkyndemeyer and @martijnwalraven - I've been talking to you guys individually, but you guys have experiences/working examples of the IntelliJ plugin for GraphQL Language Service, which is amazing btw :D I just wanted to initiate the discussion for us to collaborate together in creating a IntelliJ plugin using this GraphQL Language Service and the server!

For the actual details to making it happen I have less preferences - I thought that we could either work together on already-existing @jimkyndemeyer's IntelliJ plugin and improve that, or start from the scratch depending on how the plugin development is structured and etc. You guys obviously have a better idea how this goes, but I'm here to help as much as I can.

Also, note that by us working together to implement this, we can make any changes we want in this graphql-language-service repo to fit your needs when this is still a private repo. Let me know how we can proceed!

[jimkyndemeyer]

Sounds good to me.

I should be able to do a write-up of the IntelliJ plugin and the currently bundled language service during the weekend. I'll summarize current challenges, pain points, etc. This might help us identify and prioritize how to move forward.

[martijnwalraven]

That would be great! One of the things I think we'll need to discuss is whether we want to depend on the language service for tokenization/parsing.

I've been working on a Grammar Kit parser and tokenizer to avoid having to rely on a service call for basic features like syntax highlighting and brace matching, but I can see the benefit of not introducing another grammar. On the other hand, that is what most other clients of the language service will need to do, because a lot of environments force you to use their own grammar definition language.

[jimkyndemeyer]

I took a look at the two existing repositories in terms of largest classes and overall LOC.

# IntelliJ Plugin LOC:

JSGraphQLLanguageUIProjectService.java 684
JSGraphQLSchemaLanguageProjectService.java 557
JSGraphQLQueryContextHighlightVisitor.java 569
JSGraphQLDocumentationProvider.java 338
JSGraphQLParser.java 326
JSGraphQLNodeLanguageServiceInstance.java 334
JSGraphQLConfigurationProvider.java 272
JSGraphQLStructureViewTreeElement.java 280
JSGraphQLBlockWrapper.java 248
JSGraphQLLanguageToolWindowManager.java 247
JSGraphQLLexer.java 230
JSGraphQLInjectedLanguageBlockBuilder.java 209
JSGraphQLCompletionContributor.java 211
JSGraphQLAnnotator.java 181
JSGraphQLConfigEditorNotificationProvider.java 180
JSGraphQLBlock.java 171
JSGraphQLNamedPsiElement.java 172
JSGraphQLNodeLanguageServiceClient.java 148
JSGraphQLNodeInterpreterEditorNotificationProvider.java 133
JSGraphQLApolloLokkaEditorNotificationProvider.java 134
JSGraphQLLanguageInjectionUtil.java 131
JSGraphQLSyntaxHighlighter.java 116
JSGraphQLNamedTypePsiElement.java 129
JSGraphQLColorSettingsPage.java 104
...
total: 10895 (8065 source code)

# Language Service LOC:

languageservice.js 555
relay-templates 462
project.js 186
total: 1203

Like Martin mentioned, one of the central things to decide on is whether to use the language service for parsing and tokenization.

Based on my current understanding, the language service can't avoid these tasks internally when it has to provide hints, linting, validation, and so forth.

I'd like to see parsing and tokenization provided by the languages service for the following reasons:

• IDE developers can focus on IDE-specific integration, while having access to the latest GraphQL parser (e.g. the JS reference impl.)
• GraphQL tooling across different IDEs will appear more consistent and unified
• 99% of the time, the GraphQL in the editor buffer doesn't produce a valid AST. Placing a parser with error recovery in the language service can shield the IDE developers from a lot of complexity.

But the biggest reason for centralizing parsing in the language service is due to how client frameworks need to template GraphQL. To widest known examples include Relay and Apollo. For example:

Relay.QL`
   fragment on User {
     ${UserComponent.getFragment('user')}
   }
`;

In this example, the GraphQL spec requires a name after the fragment keyword, and even if the parser didn't choke on the ${...} placeholder, the validator would complain that the fragment has to select at least one field. There are many other examples of false positives in the validator when templating is involved. What I attempted in the current language service was to temporarily insert the name, transform the placeholders into line comments and __typename fields, before passing it to the GraphQL reference implementation. While it does the job, it's far from elegant or sustainable.

Also, outside the Relay.QL tagged template, there's JavaScript code, so the buffer can't be used "as is". In this use case the IntelliJ plugin replaces all outside code with white-spaces (while preserving line breaks). That way any linting errors returned will be correctly positioned. Again, each IDE developer has to deal with this unless it becomes a part of the language service.

[martijnwalraven]

I agree it would be great to be able to leave tokenization and parsing to the language service. There will be a bit of communication overhead, but being able to cache results in between requests might make up for that.

The problem is that not every editing environment is as flexible as IntelliJ. So in many cases, editors/IDEs rely on their own declarative grammar format and can't easily use an external language service. In Atom for example, language plugins are based on the old TextMate format. And even Visual Studio Code's new language service protocol doesn't include parsing, and still depends on TextMate grammars for syntax highlighting.

(One benefit of these grammar formats is that they are usually explicitly designed to deal with the problem of incomplete or invalid syntax. That is also true of Grammar-Kit).

So even though I would welcome adding this to the language service, I'm afraid the use will be limited to IntelliJ for now. So we'll have to decide whether that is worth it. Alternatively, we could use the Grammar-Kit grammar I've been working on.

As for the problem of template strings, we may be able to reuse parts of eslint-plugin-graphql.

[jimkyndemeyer]

@martijnwalraven Yeah, I've been quite happy with Grammar-Kit in JetBrains/js-graphql-intellij-plugin#15 for the GraphQL Endpoint language (it's schema shorthand with the addition of imports, auto-implementation of interfaces, and annotations to attach resolvers and mutators). There's no doubt that this is the "proper" way to build a parser for use inside IntelliJ.

How complete is the grammar you're working on, and can we access it some where?

What do you guys think is the best way to organize the code in terms of github repositories and organizations? I'd be happy to add you guys to js-graphql-intellij-plugin, but maybe it would make more sense to move it into an organization?

[martijnwalraven]

@jimkyndemeyer: The grammar is complete, following the spec, but I still need to fine tune the tokens and PSI. I'd be happy to spend some more time on it and put the project in a repo.

As for the organization, I could create a repo under apollostack, where we also host other general GraphQL utilities that are not tied to Apollo (like the eslint plugin).

[asiandrummer]

@jimkyndemeyer @martijnwalraven great discussion btw - sorry I've been away for a while. I was working on improving the Nuclide implementation of the language service; will sync the updated language service code soon.

I'd like to see parsing and tokenization provided by the languages service for the following reasons:

In fact, as you probably are aware, there is already some tools for online-parsing GraphQL queries provided with this library. It's basically what powered codemirror-graphql library to parse and style each GraphQL syntactical tokens, and I would consider this one as a "reference implementation" of an online-parser for GraphQL, probably specific to be used for tokenization/grammar and style declarations. I'm not sure if the Grammar-Kit implementation refers to the Grammar Rules from this library - I'd recommend for it as it gets updated frequently when spec and graphql-js change.

The plan is to create a micro library for the parser-related code and share across wherever we need it, but probably not for IntelliJ plugin as it's Java-based, unless there are ways to pull in JavaScript code and use them. As per having the language service provide the parsing, @martijnwalraven briefly mentioned it, but communicating with the language service server to achieve grammar-related tasks might be a bit too expensive. In addition to the graceful error handling and etc, IDE-specific parsing/tokenizations have the benefit of being able to locally perform the tasks without delays, and that's probably what we want - i.e. providing syntax highlighted query string that gets updated for every keystroke. Because of this I'd vote for having an IDE-specific grammar parser like how @rmosolgo and @gandm implement language-graphql.

My long-term goal is, such as how I'd love to have ecosystem of GraphQL language service implementations across common IDEs, I'd also be psyched to have many different GraphQL parser libraries for common IDEs. Would this be okay?

I'd be happy to spend some more time on it and put the project in a repo.

I'm sure we'd be totally okay checking out what you have so far in a private repo just like this one ;)

What do you guys think is the best way to organize the code in terms of github repositories and organizations?

I don't mind having anything anywhere, and am probably not thinking about this matter enough, or more than I really should have been :D I'm seeing three possible repositories that can happen:

• graphql-language-service
• graphql-intellij-plugin
• graphql-intellij-grammar (or something around the parsing/tokenization)

I don't really know how these things are organized, but maybe we could host the repositories wherever we'd like for now, and transfer to an organization later on if we feel the need for it? For graphql-language-service I think we could have it as it is for now - wherever you'd like the IntelliJ plugin (and maybe parser) to reside in is really up to you guys. We can either reuse Jim's repository or have it in Apollo's github organization for now. Later in the future, I can totally see something in lines of "GraphQL Toolling" organization happening, where we can host all GraphQL-related toolings in an open-source format, indifferent to the company/individuals owning it.

[martijnwalraven]

I've been traveling, but I should be able to get to this tomorrow and share the work in progress for the IntelliJ grammar. We can take it from there and see how we can build on that.

[martijnwalraven]

Sorry this took a little longer, but I wanted to clean up the code a bit first.

I've just pushed an initial version of the Grammar-Kit based IntelliJ plugin to intellij-graphql. The repo is public, but this is definitely work in progress and meant as food for discussion.

My main concern was to generate a clean PSI, and experiment with that to drive some basic editor features. It is by no means as complete as @jimkyndemeyer's plugin, but it does syntax highlighting and code indentation. Some elements implement PsiNameIdentifierOwner, which could be the basis for referencing.

Syntax highlighting is based both on the lexical tokens, and on a second level of annotation that uses the PSI generated from the parser:

[jimkyndemeyer]

@martijnwalraven The grammar and PSI looks good to me.

A bit of news, and the reason for my delayed answer: A lot of moving parts have fallen into place, so I will be building a new house this year. This means that I will have very limited time to contribute to this collaborative effort, given that it was primarily taken out of my spare time.

I will still try to contribute as much as humanly possible. If you should want to integrate this new grammar and PSI into a branch of js-graphql-intellij-plugin, I'd be happy to add you to the repo. Otherwise, feel free to use what you see fit from the code base as a starting point.

Best,
Jim.

[martijnwalraven]

@jimkyndemeyer: Wow, that sounds like a great project! Good luck with building a new house.

Thanks for allowing us to reuse code from js-graphql-intellij-plugin. I think it would be most useful for me to start from the new codebase, because I'd like to understand exactly what goes in there, and we may not need everything that was there before now that we have a standalone grammar and PSI classes.

I'm not sure how much time I'll be able to spend on this project either, because my primary focus is on our iOS GraphQL client. But I'd like to get a basic version of the plugin out soon, even if it doesn't use the language service yet.

[asiandrummer]

Hey guys - I also apologize for my delayed answer. I've been on vacation (and am still on it!) in Korea :D

@jimkyndemeyer awesome news - thanks for letting us know. We'll still hang around here I presume ;)

@martijnwalraven about iOS GraphQL client, wasn't there a Xcode version of GraphQL language service that we wanted to build/already have started on?

[martijnwalraven]

@asiandrummer: It's not really a language service, but I built a grammar plugin for Xcode that adds syntax highlighting for GraphQL. The current version of Xcode isn't very extensible unfortunately, so for true integration with a language service, code completion, etc. we'll probably have to rely on integrating with an external tool for now.

[asiandrummer]

@martijnwalraven I see. For the update on my side, I'll probably release Nuclide integration to the public in a week or two (still a bug-catching phase), and will go public with this repo as well :D I'd like to try carving out some time to help you with IntelliJ plugin - I think I can start with your Grammar-Kit or whatever your current progress is with the integration.

[asiandrummer]

@martijnwalraven and @jimkyndemeyer - I've decided to work on the language service integration on IntelliJ. I read through the code in the existing js-graphql-intellij-plugin, and thought that it's probably a better idea to work on the integration on top of this repo instead of creating yet another repository for this :) I wanted to consult you guys before starting on it (and potentially be added to the repo for contribution).

Also, my plan suggests we switch to using graphql-language-service instead of js-graphql-language-service - I'm going to work on having a feature parity between these two repos, but wanted to see if @jimkyndemeyer would be okay with it. If you have other suggested approach, please let me know!

My plan in a high level is:

• Study @martijnwalraven's Grammar-Kit and potentially integrate them into js-graphql-intellij-plugin.
• Study js-graphql-language-service repo and potentially build support for the methods used in that library in graphql-language-service, or substitute with what's already available from this library.
• Build/modify integration to use graphql-language-service's stream for the communication mean.

What do you guys think?

[martijnwalraven]

Sounds great! I think it makes sense to build on the Grammar-Kit PSI classes, because those integrate well with IntelliJ and support fast incremental parsing. You can then annotate them with semantic information from the language service, use the service for autocompletion, etc.

There may also be interest to help with this project from the Apollo Android contributors. I mentioned it on the #android channel of the Apollo slack, but having a better idea of the direction and roadmap would help get people to contribute.

[jimkyndemeyer]

Sounds good to me. I'll add you both to the repositories.

Like Martin said, I also think the first step is to replace the current PSI classes with what Martin has been working on. Just note that there is a lot of functionality that depends on the current model, including folding, formatting (both top level and for language injection), find usages, structure view, go to declaration and so forth. Also, the plugin displays the loaded schema using shorthand notation, so the Grammar Kit PSI needs to include this if it doesn't already.

I'm working on getting v1.5.0 of the plugin out this weekend, which includes graphql-js 0.9.2. This release also includes a variation on the shorthand notation called GraphQL Endpoint Language which my team has been using internally. We also used Grammar Kit, and have been really pleased with the performance.

[martijnwalraven]

Great! I also invited both of you to https://github.com/apollostack/intellij-graphql.

I also still think we may want to take a look at Visual Studio Code's new language service protocol to see if it makes sense to build on that.

[mickaelistria]

I also still think we may want to take a look at Visual Studio Code's new language service protocol to see if it makes sense to build on that.

Definitely! Supporting this protocol allows good integration in multiple IDEs with very minimal effort.

[darrinps]

Don't know if I'll be able to add much with respect to coding (although I could try if needed), I'd like to be able to try the tool and provide feedback if that would be beneficial in helping this along.

[wincent]

Supporting this protocol allows good integration in multiple IDEs with very minimal effort.

Yeah, @asiandrummer already took a stab at implementing this in the service (see here and other commits on master).

[asiandrummer]

@darrinps that would be fantastic! This is still in a technical preview (and does need a lot of improvements before we get out of it), so please feel free to try out and give unsolicited feedback!

[asiandrummer]

Opened JetBrains/js-graphql-intellij-plugin#62 to keep this conversation going :D