Allow ignoring some directories (or specifying a "root directory") for language server

[ian-h-chamberlain]

Is your feature request related to a problem? Please describe.
Yes, when opening .robot files in my org's large monorepo, I see a ton of messages in the output like this (file paths abbreviated):

lsp: 2021-07-15 11:09:27 UTC pid: 60672 - Thread-3 - CRITICAL - robocorp_ls_core.workspace
Directory tree more than 20 levels deep: /Users/ichamberlain/Documents/workspace/...

lsp: 2021-07-15 11:09:27 UTC pid: 60672 - Thread-3 - CRITICAL - robocorp_ls_core.workspace
Directory tree more than 20 levels deep: /Users/ichamberlain/Documents/workspace/...

These paths are behind symlinks and are in places that the robot language server really does not need to be traversing, since there are no robot libraries or resource files in those directories.

This also seems to lead to Undefined keyword errors, although the root cause of that may be different and I am still investigating.

Describe the solution you'd like
Ideally, add a configuration option which can be treated like VSCode's files.exclude, .gitignore, or similar. This would allow me to exclude specific directories from being traversed.

Describe alternatives you've considered
From #238 it seems like the file watcher API being used might not have the ability to add ignored/excluded directories, so perhaps an alternative (in my use-case) would be to specify a "root" directory for the server to walk. In our repo, there is a top-level robot directory under which all of our .robot files live, so this would solve the case for us. I imagine this is slightly less flexible in general, however.

[fabioz]

As a note, in the mode where this is reported (which is in poll mode) it should be possible to filter it (I just haven't made it customizable yet).

But note that this is also mostly a warning (when it gets to 20 levels deep it stops recursing that directory structure and notifies about it, but it shouldn't be the source of errors -- unless you actually had .robot or .resource files which should be found there, but from what you said, this isn't the case).

This is to say that I'll probably make that customizable in the poll mode, but this shouldn't be the cause for undefined keywords...

[ian-h-chamberlain]

Yeah, after revisiting this it was totally unrelated to the missing keywords, I resolved that eventually. I still think this would be a nice enhancement to avoid traversing large chunks of the filesystem unnecessarily, though.

[ian-h-chamberlain]

@fabioz I ran into this again, and I'm wondering if I misremembered the issue I had seen previously. In this case I don't see red squigglies or anything, but completion, "go to definition", etc. do not work, and I think it's happening because the server is spending so much time traversing the extra directories unnecessarily. This log in particular (stuck in among all the Directory tree more than 20 levels deep messages) is part of why I think it might be an issue:

lsp: 2021-07-22 09:15:03 UTC pid: 50260 - MainThread - EXCEPTION - robotframework_ls.server_manager
Error (None) starting robotframework server api (still running). Base exception: Request timed-out (30): {'jsonrpc': '2.0', 'id': 0, 'method': 'initialize', 'params': {'processId': 50260, 'rootUri': 'file:///Users/ichamberlain/Documents/workspace', 'workspaceFolders': [{'uri': 'file:///Users/ichamberlain/Documents/workspace', 'name': 'workspace'}]}}.

Traceback (most recent call last):
  File "/Users/ichamberlain/.vscode/extensions/robocorp.robotframework-lsp-0.19.0/src/robotframework_ls/server_manager.py", line 261, in get_robotframework_api_client
    api.initialize(
  File "/Users/ichamberlain/.vscode/extensions/robocorp.robotframework-lsp-0.19.0/src/robotframework_ls/server_api/client.py", line 43, in initialize
    return self.request(
  File "/Users/ichamberlain/.vscode/extensions/robocorp.robotframework-lsp-0.19.0/src/robotframework_ls/vendored/robocorp_ls_core/client_base.py", line 345, in request
    raise TimeoutError("Request timed-out (%s): %s" % (timeout, contents))
TimeoutError: Request timed-out (30): {'jsonrpc': '2.0', 'id': 0, 'method': 'initialize', 'params': {'processId': 50260, 'rootUri': 'file:///Users/ichamberlain/Documents/workspace', 'workspaceFolders': [{'uri': 'file:///Users/ichamberlain/Documents/workspace', 'name': 'workspace'}]}}

I see about 10 of these in my log over the last 2-3 minutes since I opened a .robot file, which leads me to believe that the server might be overloaded trying to walk these deep directory structures even though it really doesn't need to.

[fabioz]

I've just implemented this.

See: https://github.com/robocorp/robotframework-lsp/blob/master/robotframework-ls/docs/faq.md#how-to-change-the-file-watch-mode for details on how to set it.

(it'll be available for the next release, which should be 1-2 weeks away, but you can get a pre-release by downloading the .vsix from https://github.com/robocorp/robotframework-lsp/actions/runs/1075887279).

[ian-h-chamberlain]

@fabioz Thanks for working on this! I installed the vsix to try it out, but it looks like my glob patterns are not working quite as I would expect (slightly modified to scrub org-specific files but I think it's pretty representative):

Relevant configuration:

{
	"robot.python.env": {
		"ROBOTFRAMEWORK_LS_WATCH_IMPL": "fsnotify",
		"ROBOTFRAMEWORK_LS_IGNORE_DIRS":
			"[\"**/bazel-*\", \"**/my_org\", \"**/.bazel_out\"]"
	},
	"robot.language-server.args": [
		"-vvv"
	]
}

Log output:

lsp: 2021-07-28 14:40:35 UTC pid: 61202 - Thread-2 - DEBUG - robocorp_ls_core.load_ignored_dirs
Directory tracked for changes: /Users/ichamberlain/Documents/workspace/bazel-workspace/.bazel_out/execroot/my_org/.bazel_out/external/bazel_gazelle_go_repository_cache/pkg/mod/cache/download/github.com/subosito

lsp: 2021-07-28 14:40:35 UTC pid: 61202 - Thread-2 - DEBUG - robocorp_ls_core.load_ignored_dirs
Directory tracked for changes: /Users/ichamberlain/Documents/workspace/bazel-workspace/.bazel_out/execroot/my_org/.bazel_out/external/bazel_gazelle_go_repository_cache/pkg/mod/cache/download/github.com/subosito/gotenv

lsp: 2021-07-28 14:40:35 UTC pid: 61202 - Thread-2 - DEBUG - robocorp_ls_core.load_ignored_dirs
Directory tracked for changes: /Users/ichamberlain/Documents/workspace/bazel-workspace/.bazel_out/execroot/my_org/.bazel_out/external/bazel_gazelle_go_repository_cache/pkg/mod/cache/download/github.com/subosito/gotenv/@v

lsp: 2021-07-28 14:40:35 UTC pid: 61202 - Thread-2 - DEBUG - robocorp_ls_core.load_ignored_dirs
Directory tracked for changes: /Users/ichamberlain/Documents/workspace/bazel-workspace/.bazel_out/execroot/my_org/.bazel_out/external/bazel_gazelle_go_repository_cache/pkg/mod/cache/download/github.com/prometheus

Some other examples:

lsp: 2021-07-28 14:45:04 UTC pid: 65197 - Thread-2 - DEBUG - robocorp_ls_core.load_ignored_dirs
Directory tracked for changes: /Users/ichamberlain/Documents/workspace/bazel-out/darwin-fastbuild/bin/python/some_application/test/unit_test.runfiles/my_org/external/pypi__36__packaging_20_4

lsp: 2021-07-28 14:45:04 UTC pid: 65197 - Thread-2 - DEBUG - robocorp_ls_core.load_ignored_dirs
Directory tracked for changes: /Users/ichamberlain/Documents/workspace/bazel-out/darwin-fastbuild/bin/python/some_application/test/unit_test.runfiles/my_org/external/pypi__36__packaging_20_4/packaging

lsp: 2021-07-28 14:45:04 UTC pid: 65197 - Thread-2 - DEBUG - robocorp_ls_core.load_ignored_dirs
Directory tracked for changes: /Users/ichamberlain/Documents/workspace/bazel-out/darwin-fastbuild/bin/python/some_application/test/unit_test.runfiles/my_org/external/pypi__36__packaging_20_4/packaging-20.4.dist-info

Is there something I'm missing with the way these globs work? They seem to be defined roughly the same as e.g. __pycache__ or .git, which I do see working as expected in the logs.

It also seems like some of my rules do work as expected, i.e. if I create a "**/CMakeFiles" rule, I see this:

lsp: 2021-07-28 14:47:26 UTC pid: 66721 - Thread-3 - DEBUG - robocorp_ls_core.load_ignored_dirs
Directory untracked for changes: /Users/ichamberlain/Documents/workspace/build/integrationTest/lib/test/CMakeFiles

lsp: 2021-07-28 14:47:26 UTC pid: 66721 - Thread-3 - DEBUG - robocorp_ls_core.load_ignored_dirs
Directory untracked for changes: /Users/ichamberlain/Documents/workspace/build/integrationTest/lib/CMakeFiles

Any ideas? Thanks in advance, this looks very promising to make the extension usable for me!

[fabioz]

Can you try to set those patterns with /** in the end?

i.e.: something as:

**/.bazel_out/**

(if it was recursing it'd match it without the /**, but in this case the entry seems to be already inside of it, which is the case where it's working)

Also, right now you should set those environment variables in the system (or in the shell that starts vscode -- there are 2 places that'll read it and one will pick that up and the other won't).

[ian-h-chamberlain]

I tried setting the env var outside of code (as well as in settings.json, although is just setting it in the system environment enough?) using /** and I'm still seeing the same kind of output.

Something I noticed just now is that my workspace does contain an infinite symlink loop, which is perhaps part of the problem? I realize this is a bit unusual and lots of tools don't handle it well, which is part of why I wanted to ignore it altogether, lol.

Example:

lsp: 2021-07-29 09:20:27 UTC pid: 87496 - Thread-3 - DEBUG - robocorp_ls_core.load_ignored_dirs
Directory tracked for changes: /Users/ichamberlain/Documents/workspace/bazel_out/execroot/my_org/bazel_out/execroot/my_org/bazel_out/execroot/my_org/bazel_out/execroot/my_org/bazel_out/execroot/my_org/python/some_lib

There is a symlink /Users/ichamberlain/Documents/workspace/bazel_out/execroot/my_org/bazel_out@ -> /Users/ichamberlain/Documents/workspace/bazel_out which seems to be the problematic one.

I added both **/.bazel_out/** and **/bazel_out/** to my ignore list, plus **/my_org/**.
It seems like even though it should have stopped traversing after reaching the first bazel_out, it continues to follow the symlink recursively looking for more files?

[fabioz]

Can you get the latest version from: https://github.com/robocorp/robotframework-lsp/actions/runs/1078944316 (I added some logging to help diagnose it) and provide the full logs (following: https://github.com/robocorp/robotframework-lsp/blob/master/robotframework-ls/docs/reporting_issues.md)?