Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

internal/lsp: Provide (opt-in) custom semantic tokens & modifier #833

Merged
merged 3 commits into from
Mar 21, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/language-clients.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@ Clients specifically should **not** send `*.tf.json`, `*.tfvars.json` nor
Packer HCL config nor any other HCL config files as the server is not
equipped to handle these file types.

## Syntax Highlighting

Read more about how we recommend Terraform files to be highlighted in [syntax-highlighting.md](./syntax-highlighting.md).

### Internal parser

The server expects clients to use standard text synchronization LSP methods
Expand Down
80 changes: 80 additions & 0 deletions docs/syntax-highlighting.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# Syntax Highlighting

Highlighting syntax is one of the key features expected of any editor. Editors typically have a few different solutions to choose from. Below is our view on how we expect editors to highlight Terraform code while using this language server.

## Static Grammar

Highlighting Terraform language syntax via static grammar (such as TextMate) _accurately_ may be challenging but brings more immediate value to the end user, since starting language server may take time. Also not all language clients may implement semantic token based highlighting.

HashiCorp maintains a set of grammars in https://github.com/hashicorp/syntax and we encourage you to use the available Terraform grammar as the *primary* way of highlighting the Terraform language.

## Semantic Tokens

[LSP (Language Server Protocol) 3.16](https://microsoft.github.io/language-server-protocol/specifications/specification-3-16/) introduced language server-driven highlighting. This language server is better equipped to provide more contextual and accurate highlighting as it can parse the whole AST, unlike a TextMate grammar operating on a regex-basis.

LSP 3.17 does support use cases where semantic highlighting is the only way to highlight a file (through [`augmentsSyntaxTokens` client capability](https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#semanticTokensClientCapabilities)). However in the context of the Terraform language we recommend semantic highlighting to be used as in *addition* to a static grammar - i.e. this server does _not_ support `augmentsSyntaxTokens: false` mode and is not expected to be used in isolation to highlight configuration.

There are two main use cases we're targeting with semantic tokens.

### Improving Accuracy

Regex-based grammars (like TextMate) operate on line-basis, which makes it difficult to accurately highlight certain parts of the syntax, for example nested blocks occuring in the Terraform language (as below).

```hcl
terraform {
required_providers {

}
}
```

Language server can use the AST and other important context (such as Terraform version or provider schema) to fully understand the whole configuration and provide more accurate highlighting.

### Custom Theme Support

Many _default_ IDE themes are intended as general-purpose themes, highlighting token types, modifiers and scopes mappable to most languages. We recognize that theme authors would benefit from token types & modifiers which more accurately reflect the Terraform language.

LSP spec doesn't _explicitly_ encourage defining custom token types or modifiers, however the default token types and modifiers which are part of the spec are not well suited to express all the different constructs of a DSL (Domain Specific Language), such as Terraform language. With that in mind we use the LSP client/server capability negotiation mechanism to provide the following custom token types & modifiers with fallback to the predefined ones.

#### Token Types

Primary token types are preferred if deemed supported by client per `SemanticTokensClientCapabilities.TokenTypes`, fallbacks are also only reported if client claim support (using the same capability).

Fallback types are chosen based on meaningful semantic mapping and default themes in VSCode.

| Primary | Fallback |
| ------- | -------- |
| `hcl-blockType` | `type` |
| `hcl-blockLabel` | `enumMember` |
| `hcl-attrName` | `property` |
| `hcl-bool` | `keyword` |
| `hcl-number` | `number` |
| `hcl-string` | `string` |
| `hcl-objectKey` | `parameter` |
| `hcl-mapKey` | `parameter` |
| `hcl-keyword` | `variable` |
| `hcl-traversalStep` | `variable` |
| `hcl-typeCapsule` | `function` |
| `hcl-typePrimitive` | `keyword` |

#### Token Modifiers

Modifiers which do not have fallback are not reported at all if not received within `SemanticTokensClientCapabilities.TokenModifiers` (just like fallback modifier that isn't supported).

| Primary | Fallback |
| ------- | -------- |
| `hcl-dependent` | `defaultLibrary` |
| `terraform-data` | |
| `terraform-locals` | |
| `terraform-module` | |
| `terraform-output` | |
| `terraform-provider` | |
| `terraform-resource` | |
| `terraform-provisioner` | |
| `terraform-connection` | |
| `terraform-variable` | |
| `terraform-terraform` | |
| `terraform-backend` | |
| `terraform-name` | |
| `terraform-type` | |
| `terraform-requiredProviders` | |
4 changes: 2 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,12 @@ require (
github.com/hashicorp/go-uuid v1.0.2
github.com/hashicorp/go-version v1.4.0
github.com/hashicorp/hc-install v0.3.1
github.com/hashicorp/hcl-lang v0.0.0-20220314150337-d770b425fb22
github.com/hashicorp/hcl-lang v0.0.0-20220316204834-49ffde67ce68
github.com/hashicorp/hcl/v2 v2.11.1
github.com/hashicorp/terraform-exec v0.16.0
github.com/hashicorp/terraform-json v0.13.0
github.com/hashicorp/terraform-registry-address v0.0.0-20210816115301-cb2034eba045
github.com/hashicorp/terraform-schema v0.0.0-20220225085753-faadc57bd40a
github.com/hashicorp/terraform-schema v0.0.0-20220316204916-c6585b866d6d
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/mh-cbon/go-fmt-fail v0.0.0-20160815164508-67765b3fbcb5
github.com/mitchellh/cli v1.1.2
Expand Down
11 changes: 4 additions & 7 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -295,10 +295,8 @@ github.com/hashicorp/hc-install v0.3.1 h1:VIjllE6KyAI1A244G8kTaHXy+TL5/XYzvrtFi8
github.com/hashicorp/hc-install v0.3.1/go.mod h1:3LCdWcCDS1gaHC9mhHCGbkYfoY6vdsKohGjugbZdZak=
github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=
github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
github.com/hashicorp/hcl-lang v0.0.0-20211118124824-da3a292c5d7a/go.mod h1:0W3+VP07azoS+fCX5hWk1KxwHnqf1s9J7oBg2cFXm1c=
github.com/hashicorp/hcl-lang v0.0.0-20220314150337-d770b425fb22 h1:u++Zu5hfJPSNuHh7cV1QfTItINnEGRMdOvT9KjaDQUQ=
github.com/hashicorp/hcl-lang v0.0.0-20220314150337-d770b425fb22/go.mod h1:vyszbX6YNHCKIaVUhbh3LIZljxwYOtgWCIkhT5zKfjc=
github.com/hashicorp/hcl/v2 v2.10.1/go.mod h1:FwWsfWEjyV/CMj8s/gqAuiviY72rJ1/oayI9WftqcKg=
github.com/hashicorp/hcl-lang v0.0.0-20220316204834-49ffde67ce68 h1:CdUL7gJYGdJheCfAmCWNE65wimdo9YWJSqB/+NtfWPc=
github.com/hashicorp/hcl-lang v0.0.0-20220316204834-49ffde67ce68/go.mod h1:oQgcOV8OizFyZfZh3FbQSsQtvtTv8hD23MLAxfn3E+E=
github.com/hashicorp/hcl/v2 v2.11.1 h1:yTyWcXcm9XB0TEkyU/JCRU6rYy4K+mgLtzn2wlrJbcc=
github.com/hashicorp/hcl/v2 v2.11.1/go.mod h1:FwWsfWEjyV/CMj8s/gqAuiviY72rJ1/oayI9WftqcKg=
github.com/hashicorp/logutils v1.0.0/go.mod h1:QIAnNjmIWmVIIkWDTG1z5v++HQmx9WQRO+LraFDTW64=
Expand All @@ -315,8 +313,8 @@ github.com/hashicorp/terraform-json v0.13.0/go.mod h1:y5OdLBCT+rxbwnpxZs9kGL7R9E
github.com/hashicorp/terraform-registry-address v0.0.0-20210412075316-9b2996cce896/go.mod h1:bzBPnUIkI0RxauU8Dqo+2KrZZ28Cf48s8V6IHt3p4co=
github.com/hashicorp/terraform-registry-address v0.0.0-20210816115301-cb2034eba045 h1:R/I8ofvXuPcTNoc//N4ruvaHGZcShI/VuU2iXo875Lo=
github.com/hashicorp/terraform-registry-address v0.0.0-20210816115301-cb2034eba045/go.mod h1:anRyJbe12BZscpFgaeGu9gH12qfdBP094LYFtuAFzd4=
github.com/hashicorp/terraform-schema v0.0.0-20220225085753-faadc57bd40a h1:zmKoQsY/7OzNElKxg8Y+GGWzWCnQsE0JBdPfY7tMqPo=
github.com/hashicorp/terraform-schema v0.0.0-20220225085753-faadc57bd40a/go.mod h1:Y6ag6iaW+d2PwoWSLFrt+azKn4CryA+7bKUlqxD9ogQ=
github.com/hashicorp/terraform-schema v0.0.0-20220316204916-c6585b866d6d h1:XLelo71INyUNDHQAWnGwsfAA5Ccj9LqtbaejBUzYKhc=
github.com/hashicorp/terraform-schema v0.0.0-20220316204916-c6585b866d6d/go.mod h1:i0M64K9OfxlLRuFOThK1KRi9+20Y9XbyWpgPaEycbec=
github.com/hashicorp/terraform-svchost v0.0.0-20200729002733-f050f53b9734 h1:HKLsbzeOsfXmKNpr3GiT18XAblV0BjCbzL8KQAMZGa0=
github.com/hashicorp/terraform-svchost v0.0.0-20200729002733-f050f53b9734/go.mod h1:kNDNcF7sN4DocDLBkQYz73HGKwN1ANB1blq4lIYLYvg=
github.com/huandu/xstrings v1.3.2 h1:L18LIDzqlW6xN2rEkpdV8+oL/IXWJ1APd+vsdYy4Wdw=
Expand Down Expand Up @@ -811,7 +809,6 @@ golang.org/x/tools v0.1.4/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
golang.org/x/tools v0.1.7/go.mod h1:LGqMHiF4EqQNHR1JncWGqT5BVaXmza+X+BDGol+dOxo=
golang.org/x/tools v0.1.8/go.mod h1:nABZi5QlRsZVlzPpHl034qft6wpY4eDcsTt5AaioBiU=
golang.org/x/tools v0.1.9/go.mod h1:nABZi5QlRsZVlzPpHl034qft6wpY4eDcsTt5AaioBiU=
golang.org/x/tools v0.1.10 h1:QjFRCZxdOhBJ/UNgnBZLbNV13DlbnK0quyivTnXJM20=
golang.org/x/tools v0.1.10/go.mod h1:Uh6Zz+xoGYZom868N8YTex3t7RhtHDBrE8Gzo9bV56E=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
Expand Down
57 changes: 57 additions & 0 deletions internal/lsp/semantic_tokens.go
Original file line number Diff line number Diff line change
@@ -1,9 +1,66 @@
package lsp

import (
"github.com/hashicorp/hcl-lang/lang"
"github.com/hashicorp/terraform-ls/internal/lsp/semtok"
lsp "github.com/hashicorp/terraform-ls/internal/protocol"
tfschema "github.com/hashicorp/terraform-schema/schema"
)

// Registering types which are actually in use
var (
serverTokenTypes = semtok.TokenTypes{
semtok.TokenTypeEnumMember,
semtok.TokenTypeFunction,
semtok.TokenTypeKeyword,
semtok.TokenTypeNumber,
semtok.TokenTypeParameter,
semtok.TokenTypeProperty,
semtok.TokenTypeString,
semtok.TokenTypeType,
semtok.TokenTypeVariable,
}
serverTokenModifiers = semtok.TokenModifiers{
semtok.TokenModifierDefaultLibrary,
}
)

func init() {
for _, tokType := range lang.SupportedSemanticTokenTypes {
serverTokenTypes = append(serverTokenTypes, semtok.TokenType(tokType))
}
serverTokenModifiers = append(serverTokenModifiers, semtok.TokenModifier(lang.TokenModifierDependent))
for _, tokModifier := range tfschema.SemanticTokenModifiers {
serverTokenModifiers = append(serverTokenModifiers, semtok.TokenModifier(tokModifier))
}
}

func TokenTypesLegend(clientSupported []string) semtok.TokenTypes {
legend := make(semtok.TokenTypes, 0)

// Filter only supported token types
for _, tokenType := range serverTokenTypes {
if sliceContains(clientSupported, string(tokenType)) {
legend = append(legend, semtok.TokenType(tokenType))
}
}

return legend
}

func TokenModifiersLegend(clientSupported []string) semtok.TokenModifiers {
legend := make(semtok.TokenModifiers, 0)

// Filter only supported token modifiers
for _, modifier := range serverTokenModifiers {
if sliceContains(clientSupported, string(modifier)) {
legend = append(legend, semtok.TokenModifier(modifier))
}
}

return legend
}

type SemanticTokensClientCapabilities struct {
lsp.SemanticTokensClientCapabilities
}
Expand Down
15 changes: 15 additions & 0 deletions internal/lsp/semtok/lsp_token_modifiers.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
package semtok

var (
// Modifiers predefined in LSP spec
TokenModifierDeclaration TokenModifier = "declaration"
TokenModifierDefinition TokenModifier = "definition"
TokenModifierReadonly TokenModifier = "readonly"
TokenModifierStatic TokenModifier = "static"
TokenModifierDeprecated TokenModifier = "deprecated"
TokenModifierAbstract TokenModifier = "abstract"
TokenModifierAsync TokenModifier = "async"
TokenModifierModification TokenModifier = "modification"
TokenModifierDocumentation TokenModifier = "documentation"
TokenModifierDefaultLibrary TokenModifier = "defaultLibrary"
)
27 changes: 27 additions & 0 deletions internal/lsp/semtok/lsp_token_types.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
package semtok

const (
// Types predefined in LSP spec
TokenTypeClass TokenType = "class"
TokenTypeComment TokenType = "comment"
TokenTypeEnum TokenType = "enum"
TokenTypeEnumMember TokenType = "enumMember"
TokenTypeEvent TokenType = "event"
TokenTypeFunction TokenType = "function"
TokenTypeInterface TokenType = "interface"
TokenTypeKeyword TokenType = "keyword"
TokenTypeMacro TokenType = "macro"
TokenTypeMethod TokenType = "method"
TokenTypeModifier TokenType = "modifier"
TokenTypeNamespace TokenType = "namespace"
TokenTypeNumber TokenType = "number"
TokenTypeOperator TokenType = "operator"
TokenTypeParameter TokenType = "parameter"
TokenTypeProperty TokenType = "property"
TokenTypeRegexp TokenType = "regexp"
TokenTypeString TokenType = "string"
TokenTypeStruct TokenType = "struct"
TokenTypeType TokenType = "type"
TokenTypeTypeParameter TokenType = "typeParameter"
TokenTypeVariable TokenType = "variable"
)
37 changes: 37 additions & 0 deletions internal/lsp/semtok/token_modifier.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
package semtok

import "math"

type TokenModifier string
type TokenModifiers []TokenModifier

func (tm TokenModifiers) AsStrings() []string {
modifiers := make([]string, len(tm))

for i, tokenModifier := range tm {
modifiers[i] = string(tokenModifier)
}

return modifiers
}

func (tm TokenModifiers) BitMask(declaredModifiers TokenModifiers) int {
bitMask := 0b0

for i, modifier := range tm {
if isDeclared(modifier, declaredModifiers) {
bitMask |= int(math.Pow(2, float64(i)))
}
}

return bitMask
}

func isDeclared(mod TokenModifier, declaredModifiers TokenModifiers) bool {
for _, dm := range declaredModifiers {
if mod == dm {
return true
}
}
return false
}
23 changes: 23 additions & 0 deletions internal/lsp/semtok/token_types.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
package semtok

type TokenType string
type TokenTypes []TokenType

func (tt TokenTypes) AsStrings() []string {
types := make([]string, len(tt))

for i, tokenType := range tt {
types[i] = string(tokenType)
}

return types
}

func (tt TokenTypes) Index(tokenType TokenType) int {
for i, t := range tt {
if t == tokenType {
return i
}
}
return -1
}
Loading