rg takes up all memory (>10GB)

[albertz]

• VSCode Version:
  Version: 1.44.0
  Commit: 2aae1f2
  Date: 2020-04-08T08:23:56.137Z
  Electron: 7.1.11
  Chrome: 78.0.3904.130
  Node.js: 12.8.1
  V8: 7.8.279.23-electron.0
  OS: Darwin x64 17.7.0

• OS Version: Desktop is MacOSX, SSH server is Linux.

Steps to Reproduce:

Start VSCode with SSH extension.
See ps axu on the Linux SSH server, and search for the rg process by VSCode:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
zeyer    14685  117 53.7 8244360 8231756 ?     Sl   11:12  13:09 /u/zeyer/.vscode-server/bin/0ba0ca52957102ca3527cf479571617f0de6ed50/node_modules/vscode-ripgrep/bin/rg --files --hidden --case-sensitive -g **/*.ipynb -g !**/.git -g !**/.svn -g !**/.hg -g !**/CVS -g !**/.DS_Store -g !**/__pycache__ -g !**/*.pyc --no-ignore --follow --no-config --no-ignore-global
zeyer    20113 87.3 12.8 1978088 1965448 ?     Sl   11:30   4:42 /u/zeyer/.vscode-server/bin/2aae1f26c72891c399f860409176fe435a154b13/node_modules/vscode-ripgrep/bin/rg --files --hidden --case-sensitive -g **/*.ipynb -g !**/.git -g !**/.svn -g !**/.hg -g !**/CVS -g !**/.DS_Store -g !**/__pycache__ -g !**/*.pyc --no-ignore --follow --no-config --no-ignore-global

The cwd of that process is a VSCode setup, which is a very big directory. Many of the files are behind some symlinks, but I partly want to follow symlinks. (I'm not sure how to selectively disable to not follow some of the symlinks.)

Does this issue occur when all extensions are disabled?:
I cannot try without SSH extension currently (but this should not be relevant).
However, after disabling the Python extension (ms-python.python), this problem does not seem to occur anymore. However, I want to use that extension.

I think the issue is that the directory is just very big.

I happened already multiple times that the Linux server became unresponsive because all memory was occupied, and luckily the Linux OOM killer killed rg and the system recovered a while later. This is what I see in dmesg -T:

[Tue Apr 14 11:14:15 2020] [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
...
[Tue Apr 14 11:14:15 2020] [27303]  2574 27303  2550868  2546981 20484096        0             0 rg
...
[Tue Apr 14 11:14:15 2020] Out of memory: Kill process 27303 (rg) score 528 or sacrifice child
[Tue Apr 14 11:14:15 2020] Killed process 27303 (rg) total-vm:10203472kB, anon-rss:10187924kB, file-rss:0kB, shmem-rss:0kB
[Tue Apr 14 11:14:15 2020] oom_reaper: reaped process 27303 (rg), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

In any case, this should never happen (crashing my whole system because OOM), even if the user opens a immensely huge directory (or let's say an infinite big virtual FUSE directory).

My current workaround is to have this in the .vscode/settings.json:

{
    // ...

    "search.exclude": {
        "**/data-*": true,  // there are huge directories behind this
        "logs-*": true,
        // and some more...
    },
}

[chrmarti]

/causedByExtension

This is caused by the Python extension. Please file a new issue for it: https://github.com/microsoft/vscode-python/issues/new/choose

Specifically this line of code seems to trigger the long search: https://github.com/microsoft/vscode-python/blob/bd8d32bccbf9b459981db18c837ee68414335cf4/src/client/datascience/interactive-ipynb/nativeEditorProvider.ts#L91

[albertz]

@chrmarti From the discussion here, there might be a bug in VS Code core after all. workspace.findFiles is used here, and in the documentation it says that the default excludes should apply. But what are the default excludes? search.exclude? But they are ignored here. So that would be a bug. Or the documentation for findFiles is wrong.

[chrmarti]

Indeed, that's #48674 then. Thanks for following up.