Store documents in a memdb-backed table

[radeksimko]

This is a first part of refactoring per #719

---

Why

Per LSP contract the language server is basically expected to maintain the up-to-date copy of the document sent by client via text synchronization methods (e.g. didOpen or didChange). Previously we did this as part of the filesystem package where we kept it within a map:

terraform-ls/internal/filesystem/filesystem.go

Lines 20 to 21 in b13641c

	docMeta map[string]*documentMetadata
	docMetaMu *sync.RWMutex

The filesystem package also served two use cases, which ended up not being as related to each as we originally thought:

• unified FS interface to pass e.g. to HCL parser so that it can parse either (virtual) document or the underlying file on the OS FS easily - previously solved largely via spf13/afero
• direct access to the document metadata via read and write methods

While this was a reasonable implementation originally, I believe we have slowly outgrown it with our needs.

Much of the important state which LS maintains is held in memdb and so it makes sense to move this data to memdb as well. We can benefit from the "data locality" as we don't need to be importing another package like filesystem if we only need access to documents (which is majority of the RPC layer). Secondly we can query and update data across memdb tables more easily. Lastly it makes the architecture a bit easier to understand when data/state is kept in one place, stored the same way.

New document package

A new package is dedicated to:

• Document type describing the document itself (exactly how it is stored in memdb)
• Decoupled concept of "handles" which was originally spread across lsp, filesystem packages and elsewhere. This makes it a single place which deals with conversions between file, document and directory URIs as needed in various places.
• logic related to calculation of changes between two versions of a document

Having a dedicated (smaller) package makes it easier to import throughout the codebase without running into import cycles.

New documents memdb table

type Document struct {
	Dir      DirHandle
	Filename string

	ModTime    time.Time
	LanguageID string
	Version    int

	// Text contains the document body stored as bytes.
	// It originally comes as string from the client via LSP
	// but bytes are accepted by HCL and io/fs APIs, hence preferred.
	Text []byte

	// Lines contains Text separated into lines to enable byte offset
	// computation for any position-based operations within HCL, such as
	// completion, hover, semantic token based highlighting, etc.
	// and to aid in calculating diff when formatting document.
	// LSP positions contain just line+column but hcl.Pos requires offset.
	Lines source.Lines
}

Previously the filesystem package maintained a isOpen flag next to a document indicating whether the document is open but that flag was never used because we only ever tracked open documents and removed them when they get closed - see

terraform-ls/internal/filesystem/filesystem.go

Lines 171 to 187 in b13641c

	func (fs *fsystem) CloseAndRemoveDocument(dh DocumentHandler) error {
	isOpen, err := fs.isDocumentOpen(dh)
	if err != nil {
	return err
	}
	if !isOpen {
	return &DocumentNotOpenErr{dh}
	}
	err = fs.memFs.Remove(dh.FullPath())
	if err != nil {
	return err
	}
	return fs.removeDocumentMetadata(dh)
	}

It therefore made sense to just avoid that flag.

Removal of spf13/afero

After decoupling documents into memdb and re-plumbing DocumentStore methods back to the filesystem FS interface I realized that there's actually relatively little benefit we would now get from the external library and decided to just reimplement the small part that we actually need - which is translation between document.Document and fs.FileInfo or fs.DirEntry which ended in internal/filesystem/document.go, internal/filesystem/inmem.go and internal/filesystem/os_fs.go - altogether relatively few LOC.

Unfortunately we still depend on afero indirectly through mockery which we use in tests:

$ go mod why github.com/spf13/afero

# github.com/spf13/afero
github.com/hashicorp/terraform-ls/tools
github.com/vektra/mockery/v2
github.com/vektra/mockery/v2/cmd
github.com/spf13/viper
github.com/spf13/afero

but this is "dev" dependency only.

[dbanck]

Fantastic work and presentation! Thank you for taking the time to make the review as easy as possible.

I was able to follow through with all of your changes, and the language server still works ;)

I've added a few questions to understand memdb handling transactions better.

[radeksimko]

The refactored uri package now accounts for the edge case we discovered with Windows-style paths C:\ & VS Code as per microsoft/vscode#75027 and contains some tests for that as well.

I also moved some of the existing logic we had in place for case-insensitive comparison of URIs there and added some comments after researching the problem.

There is a few problems which this PR does not address, but we have issues to track them separately:

• Support UNC paths on Windows #74
• Better guide WSL users opening UNC \\wsl$\ paths #656
• internal/filesystem: Account for case-insensitive filesystems #796

@dbanck Do you mind reviewing the newly added commits? I intentionally left the old ones intact, so it should be easier to filter out.

I re-tested the PR in a Windows VM on VMware Fusion but I'd also appreciate any additional testing on any other Windows machine.

[dbanck]

Thank you for answering all my questions and adding more tests!

Your refactorings around the handle/URI logic look solid, and the comments are helpful. I would never have guessed that VS Code is making paths so difficult.

Next, I'll test the changes on Windows.

[dbanck]

🎉 🎉 🎉

Everything works fine for me on Windows, too.

[github-actions]

This functionality has been released in v0.26.0 of the language server.
If you use the official Terraform VS Code extension, it will prompt you to upgrade to this version automatically upon next launch or within the next 24 hours.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.