rg takes up all memory (>10GB)

[albertz]

Environment data

• VS Code Version:
  Version: 1.44.0
  Commit: 2aae1f26c72891c399f860409176fe435a154b13
  Date: 2020-04-08T08:23:56.137Z
  Electron: 7.1.11
  Chrome: 78.0.3904.130
  Node.js: 12.8.1
  V8: 7.8.279.23-electron.0
  OS: Darwin x64 17.7.0
• Extension version (available under the Extensions sidebar): 2020.3.71659
• OS and version: Desktop is MacOSX 10.13, SSH server is Linux (Ubuntu 16).
• Python version (& distribution if applicable, e.g. Anaconda): CPython 3.6.3
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): None
• Relevant/affected Python packages and their versions: None
• Relevant/affected Python-related VS Code extensions and their versions: ms-python.python
• Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info How to update the language server to the latest stable version #3977): Jedi is enabled.
• Value of the python.languageServer setting: Microsoft

Via.

Expected behaviour

It should run without crashing the system due to running out-of-memory (OOM).

Actual behaviour

The rg process takes a lot of memory (>10GB). In the extreme case, it crashes the whole system.

Steps to reproduce:

(Start VSCode with SSH extension. This is probably not relevant. This is just needed in my case because the project directory is on the Linux server.)

Open a very big project directory (containing several 100GB, many thousand files, slow to access via NFS).

See ps axu on the Linux SSH server, and search for the rg process by VSCode:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
zeyer    14685  117 53.7 8244360 8231756 ?     Sl   11:12  13:09 /u/zeyer/.vscode-server/bin/0ba0ca52957102ca3527cf479571617f0de6ed50/node_modules/vscode-ripgrep/bin/rg --files --hidden --case-sensitive -g **/*.ipynb -g !**/.git -g !**/.svn -g !**/.hg -g !**/CVS -g !**/.DS_Store -g !**/__pycache__ -g !**/*.pyc --no-ignore --follow --no-config --no-ignore-global
zeyer    20113 87.3 12.8 1978088 1965448 ?     Sl   11:30   4:42 /u/zeyer/.vscode-server/bin/2aae1f26c72891c399f860409176fe435a154b13/node_modules/vscode-ripgrep/bin/rg --files --hidden --case-sensitive -g **/*.ipynb -g !**/.git -g !**/.svn -g !**/.hg -g !**/CVS -g !**/.DS_Store -g !**/__pycache__ -g !**/*.pyc --no-ignore --follow --no-config --no-ignore-global

The cwd of that process is a VSCode setup, which is the very big project directory. Many of the files are behind some symlinks, but I partly want to follow symlinks. (I'm not sure how to selectively disable to not follow some of the symlinks.)

I think the issue is that the directory is just very big.

I happened already multiple times that the Linux server became unresponsive because all memory was occupied, and luckily the Linux OOM killer killed rg and the system recovered a while later. This is what I see in dmesg -T:

[Tue Apr 14 11:14:15 2020] [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
...
[Tue Apr 14 11:14:15 2020] [27303]  2574 27303  2550868  2546981 20484096        0             0 rg
...
[Tue Apr 14 11:14:15 2020] Out of memory: Kill process 27303 (rg) score 528 or sacrifice child
[Tue Apr 14 11:14:15 2020] Killed process 27303 (rg) total-vm:10203472kB, anon-rss:10187924kB, file-rss:0kB, shmem-rss:0kB
[Tue Apr 14 11:14:15 2020] oom_reaper: reaped process 27303 (rg), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

In any case, this should never happen (crashing my whole system because OOM), even if the user opens a immensely huge directory (or let's say an infinite big virtual FUSE directory).

I tried to have this in the .vscode/settings.json but this does not seem to have an effect:

{
    // ...

    "search.exclude": {
        "**/data-*": true,  // there are huge directories behind this
        "logs-*": true,
        // and some more...
    },
}

Logs

Nothing relevant.

[karthiknadig]

@albertz This looks like it should be filed on https://github.com/microsoft/vscode-ripgrep . This has nothing to do with the Python Extension. If you disagree please let us know.

[albertz]

@karthiknadig Ok, I will do. However, one thing: The workaround I tried with "search.exclude" does not work. Maybe there is another exclude option? From the code (nativeEditorProvider.ts) I'm not sure what exclude will be relevant. I think this would be anyway good to have. Or maybe I just did sth wrong.

[karthiknadig]

/cc @rchiodo Since the question is about nativeEditorProvider.ts.

[rchiodo]

Not sure why nativeEditorProvider.ts is relevant? Is that causing your memory issue? Our search?

[albertz]

@rchiodo Yes exactly. The linked code line exactly causes the memory issue. Also, "search.exclude" does not seem to have an effect on this.

[rchiodo]

We're calling a VS code API there. This one:

/**
		 * Find files across all [workspace folders](#workspace.workspaceFolders) in the workspace.
		 *
		 * @sample `findFiles('**​/*.js', '**​/node_modules/**', 10)`
		 * @param include A [glob pattern](#GlobPattern) that defines the files to search for. The glob pattern
		 * will be matched against the file paths of resulting matches relative to their workspace. Use a [relative pattern](#RelativePattern)
		 * to restrict the search results to a [workspace folder](#WorkspaceFolder).
		 * @param exclude  A [glob pattern](#GlobPattern) that defines files and folders to exclude. The glob pattern
		 * will be matched against the file paths of resulting matches relative to their workspace. When `undefined` only default excludes will
		 * apply, when `null` no excludes will apply.
		 * @param maxResults An upper-bound for the result.
		 * @param token A token that can be used to signal cancellation to the underlying search engine.
		 * @return A thenable that resolves to an array of resource identifiers. Will return no results if no
		 * [workspace folders](#workspace.workspaceFolders) are opened.
		 */
		export function findFiles(include: GlobPattern, exclude?: GlobPattern | null, maxResults?: number, token?: CancellationToken): Thenable<Uri[]>;

So it looks like it should be using the default excludes. Looking at the code in VS code, it looks like it might be a bug in how they handle it.

[albertz]

What are the default excludes? search.exclude?

[rchiodo]

I don't think there are any. That's the bug. You'd have to ask VS code to know for sure.

[albertz]

As a follow-up for the findFiles and default includes: It sounds like it uses the option "files.exclude". See the related issue here.