The Windows Subsystem for Linux instance has terminated.

[shadycuz]

Version

Microsoft Windows [Version 10.0.19043.1266]

WSL Version

• WSL 2
• WSL 1

Kernel Version

shadycuz-> wsl.exe --status
Default Distribution: Ubuntu
Default Version: 2

Windows Subsystem for Linux was last updated on 7/22/2021
WSL automatic updates are on.

Kernel version: 5.10.16

Distro Version

Ubuntu

shadycuz-> lsb_release -r
Release:        20.04

Other Software

Docker Desktop (Windows) 3.5.1 (66090)
Visual Studio Code 1.61.0

Repro Steps

This is an intermittent issue that happens to me everyday but under different circumstances. The common denominator seems to be the use of vscode.

While using vscode in the wsl backend, the editor will suddenly disconnect. This is around the time that Vmmem has exhausted the 6GB of memory I have given to wsl, even though I can't seem to see what is using the 6GB of ram. In wsl it shows very little usage. In docker desktop I will see 2/3GB if I have a running container. It seems like their is a memory leak, since I will be coding for 2 or 3 hours before the crash but not opening any new processes. Except maybe browser tabs outside of wsl.

Once wsl has crashed I can not get back into it or use docker for windows.

C:\Users\work>wsl
The Windows Subsystem for Linux instance has terminated.

C:\Users\work>wsl -l -v
  NAME                   STATE           VERSION
* Ubuntu                 Running         2
  docker-desktop         Running         2
  docker-desktop-data    Running         2

but I can shut it down and restart it

C:\Users\work>wsl --shutdown

C:\Users\work>wsl
[/mnt/c/Users/work]
shadycuz->

Expected Behavior

It would be nice if it gave some info about why it has shutdown so I could fix/prevent it. It would also be nice to have some detailed logs somewhere like in the WSL container or in windows? Maybe the event viewer?

Actual Behavior

wsl crashes without explanation for why/what caused it.

Diagnostic Logs

I will try and provide next time it crashes.

Update

I was working inside a container using the remote-container extension for about an hour and all seemed fine. I then used the workspace search feature in vscode to search for Class I wanted to rename in multiple files. The ram usage for my docker container grew by over 1GB and the 1 or 2GB of free ram I had in WSL using free -h -t went from around 1.5Gi available to 36Mi.

I had first noticed that the search was taking forever when it's usually instant. Then the vscode window disconnected.

Note: In the search it says 1 result but that is not correct.

The container that vs-code was running then disconnected with exit code 137 which is from OOM kill. I had a second wsl window open running watch free -h -t and it froze.

My wsl2 config:

[wsl2]
memory=6GB # Limits VM memory in WSL 2 to 4 GB
processors=3 # Makes the WSL 2 VM use two virtual processo

I wonder if making the swap bigger could prevent these crashes. I could also give more memory?

[Johnz86]

Me and my team have similar issue. Wsl crashes multiple times a day. I need to restart wsl and docker to work again. We are using intellij solutions, Webstorm and Phpstorm. We have standard more wsl and memory.

[wsl2]
memory=8GB   # Limits VM memory in WSL up to 8GB
processors=4 # Makes the WSL use 4 virtual processors

I tried to increase memory for wsl, it does not resolve our issues.

[Lkopo]

@benhillis
@keith-horton

or anyone else from team. Can some of you please look into the issue? It's occurring so often and issue is here for over year without any action. Please stop ignoring it. Thank you!

[keith-horton]

Hi guys. Sorry this is crashing for you. Do you generally see this with Docker also being deployed? we're trying to find what might be causing this.

[OneBlue]

/logs

[ghost]

Hello! Could you please provide more logs to help us better diagnose your issue?

To collect WSL logs, download and execute collect-wsl-logs.ps1 in an administrative powershell prompt:

Invoke-WebRequest -UseBasicParsing "https://raw.githubusercontent.com/microsoft/WSL/master/diagnostics/collect-wsl-logs.ps1" -OutFile collect-wsl-logs.ps1
Set-ExecutionPolicy Bypass -Scope Process -Force
.\collect-wsl-logs.ps1

The scipt will output the path of the log file once done.

Once completed please upload the output files to this Github issue.

Click here for more info on logging

Thank you!

[Lkopo]

@keith-horton
Yes, we use Docker Desktop connected with WSL2. But I also heard a colleague who uses Rancher alternative and is facing the same issue.

@OneBlue
Thanks! Will try it and paste when the WSL crashes again. Stay tuned.

Executing second command:

Set-ExecutionPolicy Bypass -Scope Process -Force

outputs:

Set-ExecutionPolicy : Windows PowerShell updated your execution policy successfully, but the setting is overridden by a
 policy defined at a more specific scope.  Due to the override, your shell will retain its current effective execution
policy of RemoteSigned. Type "Get-ExecutionPolicy -List" to view your execution policy settings. For more information p
lease see "Get-Help Set-ExecutionPolicy".
At line:1 char:1
+ Set-ExecutionPolicy Bypass -Scope Process -Force
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : PermissionDenied: (:) [Set-ExecutionPolicy], SecurityException
    + FullyQualifiedErrorId : ExecutionPolicyOverride,Microsoft.PowerShell.Commands.SetExecutionPolicyCommand

unsure whether it might affect the log output, but I am afraid I won't be able to change that as policies are managed by company.

[Johnz86]

WslLogs-2022-11-30_09-58-22.zip
Thisi is what I could see in console.

Directory: C:\Users\z0034zpz\wsl.logs


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----        11/30/2022   9:58 AM                WslLogs-2022-11-30_09-58-22
The operation completed successfully.
The operation completed successfully.
The operation completed successfully.

        The profiles are already running.

        Profile Id: WSL.Verbose.File

        Error code: 0xc5583001
Log collection is running. Please reproduce the problem and press any key to save the logs.
Saving logs...

        There are no trace profiles running.

        Error code: 0xc5583000

[Johnz86]

There notifications about wsl backend stopping in windows notifications panel.

[Lkopo]

@OneBlue

Like on purpose, yesterday afternoon I weren't able to catch the crash. Today morning it finally crashed, attaching the logs
WslLogs-2022-12-01_08-57-37.zip

Est. time of crash 9:03

[Lkopo]

Another crash log retrieved:

WslLogs-2022-12-01_09-20-42.zip

Est. time of crash 13:03

[shadycuz]

I have since moved to 64GB of ram and don't have problems anymore. 32GB might also work.

[Lkopo]

This might sound as a workaround but not a solution. We have 32GB rams, but cannot use all the ram for WSL only. IDEs requires a lot as well for performance, docker too and so...

[Lkopo]

@keith-horton
@OneBlue

Hi guys, any update please?

[OneBlue]

Thank you for the logs @Lkopo.

It looks like the WSL service (lxssmanager) is either being restarted, or crashing.

Can you please follow the '9) Collect WSL logs with Feedback hub
' section and share the resulting URL ?

With that we should be able to see service crashes

[Lkopo]

Hi @OneBlue

It seems like my corporate laptop does not allow Feedback Hub. Is there any other way to catch service crashes?

[Lkopo]

@OneBlue

This is what I see in WinDbg:

(6fd4.2198): Security check failure or stack buffer overrun - code c0000409 (first/second chance not available)
Subcode: 0x7 FAST_FAIL_FATAL_APP_EXIT 
For analysis of this file, run !analyze -v
ucrtbase!abort+0x4e:
00007ffb`26fd286e cd29            int     29h

*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************


KEY_VALUES_STRING: 1

    Key  : Analysis.CPU.mSec
    Value: 640

    Key  : Analysis.DebugAnalysisManager
    Value: Create

    Key  : Analysis.Elapsed.mSec
    Value: 2758

    Key  : Analysis.IO.Other.Mb
    Value: 73

    Key  : Analysis.IO.Read.Mb
    Value: 1

    Key  : Analysis.IO.Write.Mb
    Value: 102

    Key  : Analysis.Init.CPU.mSec
    Value: 1921

    Key  : Analysis.Init.Elapsed.mSec
    Value: 639149

    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 141

    Key  : FailFast.Name
    Value: FATAL_APP_EXIT

    Key  : FailFast.Type
    Value: 7

    Key  : Timeline.OS.Boot.DeltaSec
    Value: 182167

    Key  : Timeline.Process.Start.DeltaSec
    Value: 1

    Key  : WER.OS.Branch
    Value: vb_release

    Key  : WER.OS.Timestamp
    Value: 2019-12-06T14:06:00Z

    Key  : WER.OS.Version
    Value: 10.0.19041.1

    Key  : WER.Process.Version
    Value: 10.0.19041.1566


FILE_IN_CAB:  wsl.exe.28628.dmp

NTGLOBALFLAG:  0

PROCESS_BAM_CURRENT_THROTTLED: 0

PROCESS_BAM_PREVIOUS_THROTTLED: 0

APPLICATION_VERIFIER_FLAGS:  0

CONTEXT:  (.ecxr)
rax=0000000000000001 rbx=00007ffb26fd1f80 rcx=0000000000000007
rdx=000000000000000f rsi=00007ff7d164b7c0 rdi=0000000000000000
rip=00007ffb26fd286e rsp=000000230027b760 rbp=000000230027fce0
 r8=0000000000000001  r9=000000230027b708 r10=0000000000000012
r11=0000200880000020 r12=ffffffffffffffff r13=00007ffb26ec69a8
r14=0000000000000001 r15=000000230027b9b0
iopl=0         nv up ei pl nz na pe nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202
ucrtbase!abort+0x4e:
00007ffb`26fd286e cd29            int     29h
Resetting default scope

EXCEPTION_RECORD:  (.exr -1)
ExceptionAddress: 00007ffb26fd286e (ucrtbase!abort+0x000000000000004e)
   ExceptionCode: c0000409 (Security check failure or stack buffer overrun)
  ExceptionFlags: 00000001
NumberParameters: 1
   Parameter[0]: 0000000000000007
Subcode: 0x7 FAST_FAIL_FATAL_APP_EXIT 

PROCESS_NAME:  wsl.exe

ERROR_CODE: (NTSTATUS) 0xc0000409 - The system detected an overrun of a stack-based buffer in this application. This overrun could potentially allow a malicious user to gain control of this application.

EXCEPTION_CODE_STR:  c0000409

EXCEPTION_PARAMETER1:  0000000000000007

FAULTING_THREAD:  00002198

STACK_TEXT:  
00000023`0027b760 00007ffb`26fd1f9f     : 00007ffb`00000003 00007ffb`00000003 00000000`0000021a 00000000`00000000 : ucrtbase!abort+0x4e
00000023`0027b790 00007ffb`26fbe540     : 00007ffb`26fd1f80 00007ff7`d164b7c0 00000020`00001000 00000000`01000000 : ucrtbase!terminate+0x1f
00000023`0027b7c0 00007ff7`d164b7f7     : 00000000`0001e000 00000000`00002000 00000000`00000000 00000000`ffffffff : ucrtbase!__crt_state_management::wrapped_invoke<void (__cdecl*)(void),void>+0x18
00000023`0027b7f0 00007ffb`26d402e7     : 00000000`00000000 00007ff7`d164b7c0 00000000`00000000 00007ffb`288fb2c6 : wsl!_scrt_unhandled_exception_filter+0x37
00000023`0027b820 00007ffb`294f5530     : 00000023`0027ba60 00007ffb`29597658 00000000`00000000 00000023`0027b9f8 : KERNELBASE!UnhandledExceptionFilter+0x1e7
00000023`0027b940 00007ffb`294dc876     : 00007ffb`295c4a24 00007ffb`29450000 00000023`0027ba60 00007ffb`29480e7b : ntdll!RtlUserThreadStart$filt$0+0xa2
00000023`0027b980 00007ffb`294f241f     : 00000000`00000000 00000023`0027bf60 00000023`0027c9c0 00000000`00000000 : ntdll!_C_specific_handler+0x96
00000023`0027b9f0 00007ffb`294a14a4     : 00000000`00000000 00000023`0027bf60 00000023`0027c9c0 00000000`00000001 : ntdll!RtlpExecuteHandlerForException+0xf
00000023`0027ba20 00007ffb`294a11f5     : 00000000`00000000 00000023`0027c860 00000000`00000000 00000023`0027c170 : ntdll!RtlDispatchException+0x244
00000023`0027c130 00007ffb`26c5cd29     : 000001af`1c46cd80 00007ffb`229e0b10 00000023`0027f2e0 000000c0`00000000 : ntdll!RtlRaiseException+0x185
00000023`0027c9a0 00007ffb`26f9da1d     : 00000023`0027ccb0 00000000`00000049 00007ffb`229d66f0 00000000`0000002c : KERNELBASE!RaiseException+0x69
00000023`0027ca80 00007ffb`229a3295     : 00007ffb`229a0000 00000000`00000000 00000023`0027e200 00007ffb`229a4789 : ucrtbase!CxxThrowException+0xad
00000023`0027caf0 00007ffb`229a31f6     : 00000000`00000000 00000023`0027e200 00000023`0027f2e0 00007ffb`229b6114 : wslclient!wil::details::ThrowResultExceptionInternal+0x25
00000023`0027cbf0 00007ffb`229a3934     : 00000000`00000000 00007ffb`288fe5ba 000001af`1c456f98 00000000`00000000 : wslclient!wil::ThrowResultException+0x16
00000023`0027cc20 00007ffb`229a398c     : 00000000`00000000 00007ffb`288ea53c 000001af`1c4596f0 00000023`0027f2e0 : wslclient!wil::details::ReportFailure+0x174
00000023`0027e190 00007ffb`229aa142     : 00000000`00000000 00007ffb`229cefd2 00007ffb`229d6330 00000000`00000000 : wslclient!wil::details::ReportFailure_Hr+0x44
00000023`0027e1f0 00007ffb`229b644c     : 000001af`1c4596f0 00000000`00000000 00000023`0027e3f0 00000000`00000000 : wslclient!wil::details::in1diag3::Throw_Hr+0x26
00000023`0027e240 00007ffb`229b6114     : 00000023`0027e344 00000000`00000000 00000023`0027e358 00000000`00000000 : wslclient!Microsoft::WRL::Details::MakeAndInitialize<WslClient,IUnknown>+0x128
00000023`0027e2b0 00007ffb`28955854     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : wslclient!Microsoft::WRL::SimpleClassFactory<WslClient,0>::CreateInstance+0x54
00000023`0027e2f0 00007ffb`289019f0     : 00000023`0027ed88 00000000`00000000 00000000`00000070 00000000`00000000 : combase!CServerContextActivator::CreateInstance+0x1d4
00000023`0027e470 00007ffb`28955fdc     : 00000000`00000000 00007ffb`26c967f8 00000023`0027f740 00000000`00000000 : combase!ActivationPropertiesIn::DelegateCreateInstance+0x90
00000023`0027e500 00007ffb`28992018     : 00000000`00000000 00000023`0027eb50 00000000`00000000 00007ffb`28bc1e90 : combase!CApartmentActivator::CreateInstance+0x9c
00000023`0027e5b0 00007ffb`28996e80     : 00000023`00000000 00000000`00000000 00000023`0027f648 48b1241d`615a13be : combase!CProcessActivator::CCICallback+0x58
00000023`0027e600 00007ffb`2898f481     : 00007ffb`28bc1e90 00000023`0027f648 00000000`00000000 00000023`0027e780 : combase!CProcessActivator::AttemptActivation+0x40
00000023`0027e650 00007ffb`28992250     : 00007ffb`28bc1e90 00000023`28b4c418 00000000`00000000 00000023`00000001 : combase!CProcessActivator::ActivateByContext+0x91
00000023`0027e6e0 00007ffb`289019f0     : 00000023`0027ed88 00000023`0027e790 00000023`0027eb50 ffffffdc`ffd814b0 : combase!CProcessActivator::CreateInstance+0x80
00000023`0027e730 00007ffb`288fdc5f     : 00000023`0027ed88 00000023`0027ed88 00000023`0027e7f0 00000023`0027ed88 : combase!ActivationPropertiesIn::DelegateCreateInstance+0x90
00000023`0027e7c0 00007ffb`289019f0     : 00000023`0027ed88 ffffffdc`ffd814b0 00000000`00000000 00000000`80004005 : combase!CClientContextActivator::CreateInstance+0x17f
00000023`0027ea70 00007ffb`288ebd3a     : 00000000`00000000 00000023`0027f5d0 00000000`00000001 00000000`00000000 : combase!ActivationPropertiesIn::DelegateCreateInstance+0x90
00000023`0027eb00 00007ffb`288ea7e0     : 00000000`00000000 00007ffb`26c7ff2b 00000000`00000000 00000000`00000000 : combase!ICoCreateInstanceEx+0x90a
00000023`0027f9f0 00007ffb`288ea53c     : 00000000`00000000 00007ffb`26c7b0eb 00000000`00000002 00000000`00000000 : combase!CComActivator::DoCreateInstance+0x240
00000023`0027fb50 00007ff7`d16487f0     : 000001af`1c4329f0 00000000`00000000 00000000`00000000 00000000`00020000 : combase!CoCreateInstance+0x10c
00000023`0027fbf0 00007ff7`d164aefc     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : wsl!wmain+0x150
00000023`0027fc70 00007ffb`282174b4     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : wsl!__scrt_common_main_seh+0x10c
00000023`0027fcb0 00007ffb`294a26a1     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
00000023`0027fce0 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21


STACK_COMMAND:  ~0s ; .cxr ; kb

SYMBOL_NAME:  ucrtbase!abort+4e

MODULE_NAME: ucrtbase

IMAGE_NAME:  ucrtbase.dll

FAILURE_BUCKET_ID:  FAIL_FAST_FATAL_APP_EXIT_c0000409_ucrtbase.dll!abort

OS_VERSION:  10.0.19041.1

BUILDLAB_STR:  vb_release

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

IMAGE_VERSION:  10.0.19041.789

FAILURE_ID_HASH:  {e31753ac-c98a-8055-3663-47e707543d20}

Followup:     MachineOwner
---------

Seems it's some buffer overflow issue.

[OneBlue]

Thank you @Lkopo, that's very interesting.

Can you write a dump file and share it with us (.dump /f) ?

[OneBlue]

Oh wait, looking at the stack, I think I know what it is. It's an issue that should be fixed with this update. Let me know if installing the update resolves the issue.

[Lkopo]

Thank you, will let you know in next day(s). We expect new updates tomorrow to be available for us.

[Lkopo]

Hi @OneBlue,

adding the following update. So the suggested update seems to be not automatically available for us as it's a Preview version, but after installing December update https://support.microsoft.com/en-us/topic/december-13-2022-kb5021233-os-builds-19042-2364-19043-2364-19044-2364-and-19045-2364-44e774aa-60c4-4e38-b7e7-c886d210db3b, WSL terminal suggested to update WSL via wsl.exe --update. I did that yesterday, and since then I didn't experience any crashes. It might be too soon, so I'll keep you posted whether the issue is still present.

On the contrary, everytime I start up my computer, I receieve following error message for "RemoteApp" which I have to accept at least 3-4 times to disappear.

This started happen directly after wsl.exe --update command finishes.

[OneBlue]

Interesting, that's not expected.

@hideyukn88: Do you know what could cause this ?

[hideyukn88]

@Lkopo, it's most likely your organization deployed some group policy? See microsoft/wslg#841, thanks!

[Lkopo]

@hideyukn88 you are right, I have AuthenticationLevel set to 0x2. As I understood in the issue discussion, this will be fixed?

[ghost]

This issue has been automatically closed since it has not had any author activity for the past 7 days. If you're still experiencing this issue please re-open it.

Thank you!

[manshul014]

Hey I'm facing the same issue