Improve language extensibility

[dbanck]

Background

The language server currently supports two different languages, Terraform (terraform) and Terraform variable definition (terraform-vars). Support for both is tightly coupled, making it difficult to add support for other Terraform specific languages. To support Terraform tests and stacks, we need to add new "dialects" of the Terraform language and effectively support more languages (terraform-test, terraform-mock, terraform-stack, ...).

Current state

We currently store everything about a Terraform module at a specific path inside a big Module struct in MemDB. That includes:

• Directory Path
• Module Manifest (parsed .terraform/modules/modules.json)
• Installed Terraform Version (parsed terraform version -json output)
• Installed Providers (parsed .terraform.lock.hcl and .terraform/ contents)
• Collected Reference Targets
• Collected Reference Origins
• Collected Variable Definition Reference Origins
• Parsed Module Files
• Parsed Variable Definition Files
• Diagnostics for Module Files
• Diagnostics for Variable Definition Files
• Metadata
  • Required Terraform Version
  • Required Providers & Provider Versions

There are multiple factors impacting the completion/hover/etc. data that we return for each file which don't concern the configuration itself, e.g.

• provider schema
• Terraform version

Importantly, these can change at runtime. We currently use a system of memdb watchers to monitor for specific changes and trigger notifications to the client to e.g. re-render semantic tokens, re-publish diagnostics, etc. All of this logic is currently coupled with the modules described above.

Jobs Dependencies

Jobs are currently dependent on each other, such that e.g. decoding *.tfvars entries depend on decoded variable blocks in *.tf, or decoding module inputs depends on the called modules source to be decoded, or at least its variable blocks. So there are dependencies both within the same language and across languages.

Current Challenges

• Tight coupling between module files (*.tf) and variable definition files (*.tfvars)
• Tight coupling between module configuration and other data outside of configuration
  • provider lock file
  • module manifest
  • terraform version
  • provider requirements & versions
• Upcoming support for new languages would further expand the big module struct
• Indexing every directory of the workspace at the start is slow for large workspaces
• Amount of work we do on each keystroke (send [duplicate] telemetry, republish diagnostics etc.)
• Walker and workspace/didChangeWatchedFiles handler have tight coupling with specific file name patterns

Proposal

• Split monolithic state into multiple entities #1626
• Create a generic abstraction over all state entities #1627
• Reduce the amount of initial work when walking a workspace #1629
• Only schedule jobs relevant to the files when indexing a directory #1630

[dbanck]

Closed via #1667

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.