Use window/logMessage instead of STDERR for log messages

[radeksimko]

Background

Currently the server uses STDERR as the default channel for logs. This can be overridden via -log-file flag of the terraform-ls serve subcommand, to send logs to a file.

The default behaviour is however undesirable for a couple of reasons:

• it makes trawling through the logs difficult as it's all just plaintext, so we have to resort to Ctrl/Cmd+F, grep and similar techniques
• some clients, such as NeoVim (understandably) treat STDERR as "errors", which then causes users to think there's "lots of errors" when these are just log messages
  • Nvim LSP log full of errors #1234
  • Indexing seems to block completion #1124
  • https://discuss.hashicorp.com/t/terraform-lsp-log-going-crazy/47715

Using STDERR originally was mostly a pragmatic "shortcut" we took in the early versions of LS development, which felt fine given that window/logMessage is not considered mandatory for servers to implement per the spec and STDOUT is already used for LSP communication.

Concerns

There were a few concerns we didn't know how to best address (which also further supported the STDERR route at the time):

• where do we send logs related to anything that happens prior to server setup, i.e. before initialization is finished and window/logMessage can be sent?
  • This is still an ongoing concern, but such situations and need for logging seem to be rare enough that we could only log errors and still send those to STDERR or something like that.
• how do we decide on the log level inside the LS codebase?
  • This could be addressed either by adopting a library such as go-hclog or possibly the upcoming experimental slog package with some kind of custom adapters for LSP
• how do we debug/log the LSP traffic itself without doubling the number of requests, i.e. without logging every e.g. textDocument/completion via window/logMessage?
  • This is still an ongoing concern, mostly dependent on various clients. The good news is that there are some relatively mature clients out there now which can do such logging on the client side.
• Should we consider $/logTrace for anything at all? Where do we draw the line between debugging messages and server execution tracing?

We will likely have to log human-readable messages to LSP (as it accepts just string), but in the future this may change, so we should be able to take advantage of that eventually: microsoft/vscode#185904

Proposal

Explore solutions for the above ^ concerns and look into adopting window/logMessage

https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#window_logMessage