Integrated Terminal Lags Intermittently

[davidzech]

• VSCode Version: 1.48.2(macOS Stable)
• OS Version: macOS Big Sur 11.0 Beta (20A5354i) (Public Beta Release 2)

Steps to Reproduce:

1. Open Terminal
2. Spam random keys until lag is experienced Press enter, then continuously type (anything) for about 2 seconds.
3. Experience a very long stutter

Does this issue occur when all extensions are disabled?: Yes

This occurs on a public beta build of macOS 11. I've confirmed the same issue on two different MacBook Pro's on the same OS version, and same VS Code version.

Attached is a performance profile. You can see the spike in the integrated terminal response time at around 23000ms - 24000ms

Profile-20200826T124154.json.zip

[davidzech]

It seems like

vscode/src/vs/workbench/contrib/terminal/node/terminalProcess.ts

Line 276 in f74e473

exec('lsof -OPl -p ' + this._ptyProcess.pid + ' | grep cwd', (error, stdout, stderr) => {

is taking a long time.

[davidzech]

Benchmarking this call, it takes on average 800ms. It doesn't make sense, but this seems to be a Node.js x macOS beta type of issue.

[festiveelephantseal]

I have been having this issue as well with the insiders build of vscode

[Tyriar]

@deepak1556 do you know why an exec call would be blocking the thread?

[davidzech]

Some further notes for reference:

Monkey patching

vscode/src/vs/workbench/contrib/terminal/browser/terminalInstance.ts

Line 1194 in a4befc3

this._updateProcessCwd();

to be

async() => { this._updateProcessCwd(); }

Seems to fix the terminal from being blocked. It's still a bandaid over the underlying problem, which is child_process.exec() taking a long time to execute on macOS Big Sur Beta.

Edit: I was wrong, still exhibits the behavior, weird.

[matinzd]

+1
I have this issue either and it happened after updating to macOS 11 (Big Sur) beta. I am currently on beta 9 and the issue still exists.

UPDATE:
I have the lag even if I am typing or running commands and its so annoying.

Version: 1.50.0
Commit: 93c2f0fbf16c5a4b10e4d5f89737d9c2c25488a3
Date: 2020-10-07T06:00:00.397Z (6 days ago)
Electron: 9.2.1
Chrome: 83.0.4103.122
Node.js: 12.14.1
V8: 8.3.110.13-electron.0
OS: Darwin x64 20.1.0

[tolgaerdonmez]

+1
I have this issue either and it happened after updating to macOS 11 (Big Sur) beta. I am currently on beta 9 and the issue still exists.

Version: 1.50.0
Commit: 93c2f0fbf16c5a4b10e4d5f89737d9c2c25488a3
Date: 2020-10-07T06:00:00.397Z (6 days ago)
Electron: 9.2.1
Chrome: 83.0.4103.122
Node.js: 12.14.1
V8: 8.3.110.13-electron.0
OS: Darwin x64 20.1.0

It also occurred with me after updating to Big Sur Beta, I think it is about the beta.

[davidzech]

Some further insights:

I have done some debugging with lldb, and have narrowed it down to fork() being the bottleneck here.

Edit:
You can observe calls with
sudo dtruss -p <window_pid> -t fork -e

[Vardiak]

@davidzech Don't you think this might be a bug from the macOS kernel then?

[Adam2Marsh]

+1 for having this issue; only remember seeing this issue after upgrading to Big Sur. Happy to run some debugging if anyone has any ideas.

Thanks.

[davidzech]

@Adam2Marsh Have you built locally? To my surprise, I ran my tests on master and tag 1.50.1 and I do not experience this lag at all, yet is persistent in both Stable and Insider public releases.

An useful bisection perhaps.

See

Edit 2: VSCodium also doesn't exhibit this behavior for me.

[KangDroid]

@davidzech
+1 Agree, I built Code-OSS[master branch] with macOS 11.0.1[build = 20B5012d]
and I see no problem with input-lag with integrated terminal. Only for local build.
Public version of vscode still has that problem tho.

[deepak1556]

Thanks for narrowing down the issue @davidzech , we do use a custom electron build for the stable and insiders while we use https://electronjs.org/ releases for the OSS build. But I don't see any patches around the child_process code path.

Can you run the test on following electron builds, you can extract and run the electron binary, toggle developer tools from the View menu

• Custom electron build v9.3.3
• OSS electron build v9.3.3

[marcello3d]

FYI the root cause has already been tracked down and being fixed in libuv/electron, see:

• Integrated Terminal Lags Intermittently  #105446 (comment) (a comment on this issue)
• perf: patch libuv to use posix_spawn on macOS electron/electron#27026
• darwin: Use posix_spawn to spawn subprocesses in macOS libuv/libuv#3064

[pdesantis]

To piggyback on what @marcello3d said, this was affecting our own Electron app, so we built a custom version of Electron using that libuv patch and it solved the issue for us. Hopefully this will be merged upstream soon, but in the meantime, other developers can try building Electron with that patch.

[deepak1556]

Thanks @marcello3d @pdesantis for the work on fix.

Just to update, whenever upstream releases a new version with the patch vscode will update to it. Also we didn't build the patch internally because it was too late in the current January iteration to get enough testing before stable release 1.53 that will happen next week. We will get this fix in to our internal builds next February iteration for early testing if there is no release from upstream.

I see that the issue is getting side tracked again, will lock until further updates.

[deepak1556]

The libuv fix has been backported internally and is available in latest insiders, please verify if this issue is fixed. Thanks!

[deepak1556]

\closedWith ff85144

[github-actions]

This bug has been fixed in to the latest release of VS Code Insiders!

@davidzech, you can help us out by confirming things are working as expected in the latest Insiders release. If things look good, please leave a comment with the text /verified to let us know. If not, please ensure you're on version 52838cf of Insiders (today's or later - you can use Help: About in the command palette to check), and leave a comment letting us know what isn't working as expected.

Happy Coding!

[davidzech]

/verified

Insiders Build 52838cf:

Current Stable Build 8490d3d:

[m1nkeh]

I got an update of the non-insiders today, and no issues that I could see either, weird.. v1.53

[darrenjennings]

oh thank God this was driving me crazy, thank you for fixing 🙏🏻

[ay0o]

so, what's the correct way of undoing codesign --remove-signature /Applications/Visual\ Studio\ Code.app/Contents/Frameworks/Code\ Helper\ \(Renderer\).app.

Guess we need to sign the app again. But with which key?

[MaciekBaron]

@ay0o the update should have installed a new signed binary

[superandrew]

is this distributed to everyone or just in a beta channel?

[gquittet]

I've received the update today and I can feel the improvement !
Thank you ! You rocks ! 💪

[leafac]

I’m running version 1.53.2 and I still see a significant lag when opening new terminal panes. I ran the codesign command and it fixed the issue (I’ve been doing that for every new version…).

I think the issue hasn’t been solved for me. Am I doing something wrong?

[flash-gordon]

@leafac the fix is available in the insiders builds only as of now. Switch to insiders builds or stick to the codesign workaround for the time being.

[leafac]

@flash-gordon: Ah, I had misunderstood the previous comments and I thought the fix was generally available. Thanks for the clarification 👍

[calrsom]

Heads up, the fix was reverted upstream: libuv/libuv#3050 (comment)

I'm not sure what that means for VSCode or if this issue should be reopened? I guess maybe it depends on how Electron handles this.

[deepak1556]

FYI, the fix is reverted in electron but I have added workarounds for the failing test cases mentioned in the libuv repo while it is being investigated, so I have not reverted the fix internally for vscode.