Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Many Different Extensions Frequently Crash All Others on Startup with Reoccurring "Extension host terminated unexpectedly" – Extension Sandboxing / Isolation, Missing Error Log Fixes and Host Auto-Restart Needed #79782

Open
PowerWeb5 opened this issue Aug 26, 2019 · 5 comments
Assignees

Comments

@PowerWeb5
Copy link

PowerWeb5 commented Aug 26, 2019

There have been far too may reports (800+ now) of the VSCode Extension Host crashing with "Extension host terminated unexpectedly" like shown in the below screenshot:

image

When this happens, one extension ends up crashing all other extensions and even crashing core VSCode services as well as apparently preventing logging of the errors, and then requires users to manually restart all extensions.

These issues affect a wide range of popular extensions, including core Microsoft ones like Live Share Audio, as well as frequent third party ones like Live Chat (karigari.chat) and Live Server. And there are 800+ reports of the critical "Extension host terminated unexpectedly" errors, and this issue becoming far more frequent of late.

It's not even possible to determine the problematic extension (so that you can disable it and submit a bug report) – even if you were to startup 170 times with a different extension disabled each time, as, for some reason, the original extension error and even the extension host crash itself, as well as what extension was starting to be activated or updated, often fail to ever get logged when such crashes occur.

This has become more frequent and has been reoccurring for extensions even after fixing it, breaking again even when no extension updates were released. It has been causing VS Code to crash on startup much of the time for many months now without any way to identify and disable the extension causing it. Then, after finally disabling one that fixed for a while, another extension started causing the same thing all over again. And, on each VSCode startup, repeated manual user intervention is often required to resolve what could be auto-repaired.

Therefore, it seems that VSCode changes are needed to minimize these issues (vs many repeated workarounds for each extension) – both short-term and long-term – as outlined further below.

The Case for Extension Sandboxing

I believe these widespread reoccurring critical issues also indicate the need for Extension Sandboxing (like proposed in #52116). Among other benefits, that would prevent extensions from crashing any (or at least not all) other extensions as well as core VSCode services, make it possible to determine which extensions are crashing, and also open the door to security benefits with extension permission management. I've

Repeated Critical Crashes on Startup I've Encountered For Months with Multiple Different Extensions

I have had to manually restart extension host nearly every time I started VSCode for months now. Then, finally, after identifying and disabling the problematic extension (only possible by reading through may bug reports to determine most likely causes), the issue started happening again, with another extension (equally difficult to determine which one).

And that was for an extension where the issue had been fixed several times already, with no update released since the last version fixing it, yet the issue started occurring again (possibly due to VS Code or other extensions changing). Considering that as well as how it seems to almost be randomly occurring, always on startup (eg. as if timing related or while other extensions are being updated), then it may be
that something in VSCode needs to change to minimize these issues.

Related Bug Report with Log Files

You can find details regarding my own most experiences with VS Code host crashes for the most recent extension, Live Server, at ritwickdey/vscode-live-server#552.

There, I've provided zips of log files when crashes do and do not occur (as needed for comparison due to lack of detailed logging when crashes occur).

I've also referenced to a number of related issues in that report, as well as further below.

That does not include my earlier months of VS Code crashing on startup with other extensions, such as what I'd eventually identified (despite lack of any indication in log files) as being caused by Live Share Chat, which I'd fixed just a week or so before encountering with Live Server as well.

Suggested VSCode Fixes / Changes (to Minimize Host Crashes, Allow Identifying Problem Extensions, and Prevent Crashing All Extensions)

  • Extensions should not be frequently crashing other extensions (especially not crashing ALL of them)
  • Extensions should not be crashing core VS Code services like the shared Extension Host (and potentially even logging services, etc.)
  • VS Code Extension Host should be auto-restarted when that occurs within seconds of startup, where there is no risk of data loss (especially since has been 100% successful so far when doing so)
  • VS Code should also identify the extension causing the error, when possible, report to the user, log it, and prompt the user to disable it.
  • Fixes for issues preventing both the crash and original errors from ever getting logged should be fixed.
  • Extensions shouldn't be crashing the
  • VSCode or Extension Host changes should be made to minimize such frequent and reoccurring host crashes
  • Extension Sandboxing / Isolation should be considered, for both Reliability (so as to avoid not crashing all extensions and core services) at the very least, make it possible to determine which extension is the one crashing, and even open the door to security benefits (with permission management being the norm for browser, etc. extensions these days)
  • Sandboxing would also opening the door to security benefits with extension permission management (as is common)

Identifying and Addressing the Root Cause vs. Repeated Per-Extension Temporary Workarounds

It would be helpful if someone could investigate what, if any, common denominator there may be between the different extension host crash causing extension issue (such as use of binary dependencies or how extensions are updated or activated) or such as the various extensions starting with "Live" in their name (if not a coincide with them being the most frequent offenders).

The root cause should be addressed – with changes made to VSCode (or the extension host) to minimize them instead of requiring extension developers to have to keep (clearly unreliably, temporarily) implementing workarounds for each extension. Whether changes to how extensions are

800+ Outstanding Issue Reports and Many (Even Official) Extensions with Frequent, Reoccurring Extension Host Crashes

Many others are reporting the same, such as my recent report (ritwickdey/vscode-live-server#285) as well as #72476, #75267, microsoft/live-share#1510, ritwickdey/vscode-live-server#285, Live Share Chat review, and 250+ other "Extension host terminated" bug reports just in the VSCode repo, 133 open issues under chrmarti/testissues and 850+ "Extension host terminated unexpectedly" bug reports across all GitHub VSCode extension repos

Other recent reports in VSCode repo:

#77099, #79177, #78524, #78196, #77917, #77780, #78809, #78331, #78196, #77768, #77547, #77505

Other issues related to extension crashing affecting other extensions:
#73041, #24441, #76178

Many Different Extensions Frequently Crash ALL Other Extensions with Issue Reoccurring or Unable to Be Fixed

Just a few of the highly popular, and even Microsoft official VSCode extensions which this occurs for includes:

Live Share Audio, Live Server, Live Share Chat (karigari.chat), Live Share, Live SASS Compiler, Typed Test, Java Language Server, Peacock, and Mocha Sidebar

Not only are these issues widespread across a large number of popular extensions, but they are reoccurring. The issue keeps coming back for the same extensions not long after each fix is released.

Extension Host Crash Reappears After Fixing even when Extension Weren't Updated Since

With Live Server, as well as other extensions, such extension host crashing on VSCode startup has come back every time it gets (temporarily) fixed.

In that issue I'd just reported for Live Server, for example, the crashing started occurring again for the very same extension version (5.6.1) which was released specifically to fix the issue and which had been confirmed (for a little while) as having done so, as seen in
vscode-live-server#431 the issue is back not months after the last update which was released specifically to address that issue.

Accordingly, it seems like changes to VSCode itself (or possibly frequently arising conflicts between extensions may be the cause) may be part of the issue preventing reliable fixes or avoidance of such crashes.

And before that was report "Extension host terminated unexpectedly is back" (ritwickdey/vscode-live-server#399) about how the issue had come back after having been fixed a number of times before even that.

Crashes ALL Extensions and Core VSCode Services

Worse yet, such crashes don't just affect the problematic extension, they crash ALL extensions. Even worse, they crash a core VSCode service, the Extension Host, and require the user to manually restart the host to reload all extensions.

Especially considering that, it seems like some kind of core VSCode or Extension Host changes may be needed to reduce, avoid or at least isolate the effect of the errors which are currently causing frequent reoccurring Extension Host crashes.

Some of the Popular Extensions Most Frequently Causing Extension Host Crashes

This issue is well known to often occur with Live Share Chat (karigari.chat) like reported in #77099 and mentioned in dozens of other VSCode issues. And it remains an open known issue, again, which is still unable to be addressed, like seen at vsls-contrib#410.

Even Microsoft's own Live Share Audio extension (as well as Live Share itself) frequently causes Extension Host crashes. And there is no plan to fix it any time soon, according to its developers.

Possible Causes of Frequent Extension Host Crashes (eg. Timing or Dependencies)?

Can anyone clarify what the common causes of such crashes are? I've seen reference to use of binary dependencies as a common case. Is that so? Why would such crashes start occurring again, with the same version where it was fixed previously, despite no new updates to such fixed extensions?

Whatever the cause, this seems widespread enough and critical enough that I would suggest something be done to minimize the frequency of such Extension host crashes, if possible.

Does anyone have any ideas on what might be able to be done to help reduce the frequency of Extension Host crashes?

If the most recent fixes for Live Share Chat (which was causing months of issues prior to Live Server) are any indication (where "Add Live Share as an extension dependency to fix timing issues during activation" is mentioned as the primary fix in v0.21.0 released on 2019-08-19), then interdependency between extensions and timing related issues might be one type of common cause in Extension Host crashes, especially considering how the crashes don't occur on every startup (just seemingly randomly ones).

I'd also seen extension a number of "Deleted from disk" entries in sharedprocess.log (one of the few non-empty log files) as well as "Starting to clean up unused language packs.", which I mention in case the timing of such extension updates/cleanups might be a factor.

Extension Sandboxing / Isolation Needed for Reliability (as well as Security) (+ Link to Detailed Proposal/Ways to Implement with Electron)

In any case, this is a good example of why Extension Sandboxing / Isolation is needed.

I had previously proposed in #52116 several options for implementing Extension Sandboxing / Isolation for reliability and security (as well as Chrome/Firefox-like Permission Management). There are several projects, like I'd summarized there, which could be utilized or which have successfully implemented such extension sandboxing, even with Electron-based apps like VSCode and Electron options/APIs now available, which could be used as a model or incorporated directly.

Need to Auto Restart Host If Occurs Immediately Upon Startup

Also, at the very least, I believe that there should be an option to automatically recover.

Ideally this can occur after logging the crash itself, trying to identify which extension may have caused the crash and logging and reporting that to the user, and then flushing the logs (without waiting for any user interaction).

Then the Extension Host should be automatically restarted when Extension Host crashes on startup.

This is important and reasonable to do because, in each of the many dozens of crashes I've encountered, this crash occurs within seconds of VSCode startup, and restarting always fixes the issue successfully (which my never encountering another host crash afterward during that session).

Therefore, IMO, the extension host should always be automatically restarted if it crashes within a configurable number of seconds (which could default to 5 or 10) such apparently expected. This seems reasonable handling for what is apparently far from exceptional and repeated issues, especially when automated recovery/restarting without any impact on the user can occur and even more so if this might be related to auto extension downloading/updating/install which may be occurring prior to the crash.

The user should be notified still, but there should be an option to hide such future notifications (at least showing such an option if another crash notification occurs within X days or Y number of startups since the last one) to avoid nagging the user. Especially when, at one point, I had to see annoying crash notifications and manually restart as prompted nearly every time I launched VS Code, without any indication of what extension caused or way for me to stop this occurring for months on end.

Need to Report the Extension Causing the Crash & Prompt to Disable

Also, VSCode should try to determine which extension caused (or set of extensions may have likely caused) the crash. If a specific extension can be identified as the cause, that should be shown to the user together with the host crash notification, and with a suggestion (at least for repeat offender extensions) and in-notification button (together with the Show Developer Tools and Restart Extension Host buttons) or 1-click disabling of that extension. Most often so far, the extensions causing such crashes are ones I can live without.

Currently Impossible to Identify Extension Causing Crash (Due to Logging Issues and Lack of Isolation)

However, the most important problem of all, is that it, as it stands now, it is nearly impossible to identify which extension is causing the problem.

It's not feasible for users to test startups by disabling extensions one-by-one to determine which extension or combination of extensions is causing the issue. In fact, that becomes virtually impossible when you consider that such crashes on startups don't happen every time (just every day or two in many cases). And even more so, when it might be a conflict between multiple extensions (as I've seen reference to in other issue reports) or there are multiple extensions causing such crashes at the same time (which I have also encountered).

Extension Error and Even Host Crash Itself Fail to Get Logged or Flushed

Even after finding and reading through (as well as searching) every single log file under Roaming/Code/Logs for the session (including the annoying number of empty folders to search through which are created on each startup, which I would suggest be created as-needed instead) there was absolutely no logged error. Even the activation of the extension which ended up seeming to be the cause of the crashes didn't end up in the logs or flushed to disk. And restarting the extension host or showing Developer Tools ended up obscuring the log entries and can occur 20 minutes after the actual crash (as was the case for me).

However, the only time I saw an actual error logged related to the host crash was after Restarting the Extension Host and showing Developer Tools, in which case it either failed to show anything from before that or it obscured it with many more entries. Even then, it was just a generic extension host crash error, without any indication of the original cause (neither extension or the error that caused it).

When trying to look for the relevant errors to identify which extension may have caused them and therefore which to try to disable to avoid these crashes, it was always the case that almost all log files were empty (extensions.log, tasks.log, essentially main.log for most part, and nearly all files in the few created "output_" subfolders). No errors appeared (except for a few clearly unrelated warnings which occurred even in non-crash cases). Only by comparing to non-crash startup logs could I get an idea of what might have caused the issue, and only by looking at what log entries were missing such as for extension activations that occurred immediately after where the log file was cutoff.

Logging Fixes Needed So Crash or Extension Error Is Actually Logged

As the host crash error itself doesn't end up in the log files in many cases under Roaming/Code/Logs or visibly elsewhere (other than the notification box shown to the user) the following might be helpful:

  • Flushing logs more frequently and/or after crash occurs
  • Log host crash error sooner (before waiting for user to possibly restart the host)
  • Ensure original log entries are shown even after restarting the host and then showing developer tools
  • Changes to logging so not affected by extension host crashes
  • Log "Starting ..." for more operations (and flushing to disk) like "Downloading/Deleting/Extracting extension/update started" vs. just after "Downloaded/Deleted" etc. so know which extension may have caused the crash

My Platform and Configuration

  • Visual Studio Code v1.37.1 (user setup)
  • Windows 10 x64 Pro (Windows NT x64 10.0.18362)
  • Startup Command line args: "C:\Users\Dan\AppData\Local\Programs\Microsoft VS Code\Code.exe" "C:{My Project Path}\README.md"
  • Electron: 4.27
  • Chrome: 69.0.3497.128
  • Node.js: 10.11.0
  • V8: 6.9.427.31-electron.0
  • Live Server (ritwickdey.liveserver) extension v5.6.1
@vscodebot
Copy link

vscodebot bot commented Aug 26, 2019

(Experimental duplicate detection)
Thanks for submitting this issue. Please also check if it is already covered by an existing one, like:

@Gruntfuggly
Copy link

I know my extension causes the host to crash, but I have no way of find our where or how. 😞

Please address this somehow vscode team!

@PowerWeb5
Copy link
Author

@RMacfarlane and @kieferrm, please see the crash report which I just submitted, #80165 for additional details and logs likely useful for debugging this.

That provides detailed logs that may help in debugging this issue, with logs both for sessions with and without this issue occurring, despite same steps to reproduce (needed since doesn't seem to log any extension-specific errors related just to the crash case), as well as logs for session preceding the crash (in case related to extension updating/cleanup) and log files saved from Developer Tools (which show some errors, which otherwise don't get saved to log folders on disk).

@Narvey
Copy link

Narvey commented Feb 19, 2020

Why is this not a higher priority?

I recommend the sandboxing solution (#52116) Without it, VS Code looks like a liability to the IT departments that might consider approving it! (think about trying to determine the risks of a wide variety of addons, and whether they will be stealing company secrets like proprietary code)

@vintprox
Copy link

On kinda related note, Chat Session from VS Live Share is randomly closing after 2 minutes of inactivity. I don't know how to catch it, but it just happened a lot today, when I tried it out with my student.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants