Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ServiceHub dying because of uncaught cancellation token. #31845

Closed
CyrusNajmabadi opened this issue Dec 16, 2018 · 10 comments
Closed

ServiceHub dying because of uncaught cancellation token. #31845

CyrusNajmabadi opened this issue Dec 16, 2018 · 10 comments
Assignees
@sharwell
Labels
Area-IDE Bug Resolution-Not Reproducible The described behavior could not be reproduced by developers
Milestone
Backlog

Comments

@CyrusNajmabadi
Copy link
Member

CyrusNajmabadi commented Dec 16, 2018 •
edited by sharwell
Loading

Callstack:

Call stack 
Application: ServiceHub.RoslynCodeAnalysisService32.exe
Framework Version: v4.0.30319
Description: The application requested process termination through System.Environment.FailFast(string message).
Message: System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at Microsoft.CodeAnalysis.Remote.Extensions.<InvokeAsync>d__3`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.CodeAnalysis.Remote.ServiceHubServiceBase.<RunServiceAsync>d__25`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.CodeAnalysis.Remote.SnapshotService.JsonRpcAssetSource.<RequestAssetsAsync>d__2.MoveNext()
Stack:
   at System.Environment.FailFast(System.String, System.Exception)
   at Microsoft.CodeAnalysis.FailFast.OnFatalException(System.Exception)
   at Microsoft.CodeAnalysis.Remote.SnapshotService+JsonRpcAssetSource.ReportUnlessCanceled(System.Exception, System.Threading.CancellationToken)
   at Microsoft.CodeAnalysis.Remote.SnapshotService+JsonRpcAssetSource+<RequestAssetsAsync>d__2.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
   at Microsoft.CodeAnalysis.Remote.SnapshotService+JsonRpcAssetSource+<RequestAssetsAsync>d__2.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.InvokeMoveNext(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task.FinishStageThree()
   at System.Threading.Tasks.Task.CancellationCleanupLogic()
   at System.Threading.Tasks.Task`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].TrySetCanceled(System.Threading.CancellationToken, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].SetException(System.Exception)
   at Microsoft.CodeAnalysis.Remote.ServiceHubServiceBase+<RunServiceAsync>d__25`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.InvokeMoveNext(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task.FinishStageThree()
   at System.Threading.Tasks.Task.CancellationCleanupLogic()
   at System.Threading.Tasks.Task`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].TrySetCanceled(System.Threading.CancellationToken, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].SetException(System.Exception)
   at Microsoft.CodeAnalysis.Remote.Extensions+<InvokeAsync>d__3`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.InvokeMoveNext(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task.FinishStageThree()
   at System.Threading.Tasks.Task.CancellationCleanupLogic()
   at System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].TrySetCanceled(System.Threading.CancellationToken, System.Object)
   at System.Threading.Tasks.TaskFactory`1[[System.Threading.Tasks.VoidTaskResult, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].FromAsyncCoreLogic(System.IAsyncResult, System.Func`2<System.IAsyncResult,System.Threading.Tasks.VoidTaskResult>, System.Action`1<System.IAsyncResult>, System.Threading.Tasks.Task`1<System.Threading.Tasks.VoidTaskResult>, Boolean)
   at System.Threading.Tasks.TaskFactory`1+<>c__DisplayClass35_0[[System.Threading.Tasks.VoidTaskResult, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].<FromAsyncImpl>b__0(System.IAsyncResult)
   at System.IO.Pipes.NamedPipeServerStream.AsyncWaitForConnectionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
   at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
@CyrusNajmabadi
Copy link
Member Author

Tagging @jinujoseph @heejaechang

@jinujoseph jinujoseph added this to the 16.0.P2 milestone Dec 17, 2018
@CyrusNajmabadi
Copy link
Member Author

Hit this a couple more times.

@heejaechang
Copy link
Contributor

this means

at System.Environment.FailFast(System.String, System.Exception)
at Microsoft.CodeAnalysis.FailFast.OnFatalException(System.Exception)
at Microsoft.CodeAnalysis.Remote.SnapshotService+JsonRpcAssetSource.ReportUnlessCanceled(System.Exception, System.Threading.CancellationToken)
at Microsoft.CodeAnalysis.Remote.SnapshotService+JsonRpcAssetSource+d__2.MoveNext()

backend connection between VS and OOP is for some reason failed. cancellation exception is just side effect of the failure since JsonRpc throws cancellation exception when connection is closed or timed out.

can you share logs in %temp%\servicehub\logs ?

@CyrusNajmabadi
Copy link
Member Author

Sure. Here you go!
servicehub.zip

@jinujoseph jinujoseph modified the milestones: 16.0.P2, 16.0.P3 Jan 16, 2019
@heejaechang
Copy link
Contributor

the log seems doesn't contain log about the issue above. might be already cleaned up by OS if you have auto temp directory clean up on.

@heejaechang heejaechang added the Need More Info The issue needs more information to proceed. label Jan 24, 2019
@jinujoseph jinujoseph modified the milestones: 16.0.P3, 16.1.P1 Jan 24, 2019
@jinujoseph jinujoseph modified the milestones: 16.1.P1, 16.1, Backlog Apr 24, 2019
@sharwell
Copy link
Member

@heejaechang Also showed up here: https://developercommunity.visualstudio.com/content/problem/610986/frequent-process-used-by-visual-studio-has-encount.html

@jinujoseph jinujoseph modified the milestones: Backlog, 16.2 Jun 21, 2019
@heejaechang
Copy link
Contributor

heejaechang commented Jun 22, 2019 •
edited
Loading

not enough info to find out. one of report in feedback shows Out of memory exception causing all kind of exception to happen. but I don't think that is the cause. (one of feedback has servicehug logs included)

hopefully, change the feedback team will put to automatically put all servicehub logs in feedback ticket will make investigation easier from now on.

@jinujoseph jinujoseph modified the milestones: 16.2, 16.3 Jun 24, 2019
@MarkZuber
Copy link

@heejaechang - I submitted https://developercommunity.visualstudio.com/content/problem/610986/frequent-process-used-by-visual-studio-has-encount.html which was resolved as dupe to this issue. I can repro this failure consistently if you'd like to setup a teams meeting and share my screen. I'm over in bldg 27.

@jinujoseph jinujoseph assigned sharwell and unassigned heejaechang Oct 9, 2019
@jinujoseph jinujoseph modified the milestones: 16.3, 16.4 Oct 9, 2019
@jinujoseph jinujoseph removed this from the 16.4 milestone Dec 11, 2019
@jinujoseph jinujoseph modified the milestones: Backlog, 16.5 Dec 11, 2019
@jinujoseph jinujoseph modified the milestones: 16.5, Backlog Feb 21, 2020
@CyrusNajmabadi
Copy link
Member Author

Haven't seen this in a while. Closing out as likely changed through the many improvements that have happened here.

@sharwell sharwell added Resolution-Not Reproducible The described behavior could not be reproduced by developers and removed Need More Info The issue needs more information to proceed. labels May 27, 2020
@sharwell
Copy link
Member

This was likely introduced by #25376 (15.7) and eventually fixed by #33595 (16.1).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sharwell sharwell

Labels
Area-IDE Bug Resolution-Not Reproducible The described behavior could not be reproduced by developers
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
5 participants
@heejaechang @sharwell @CyrusNajmabadi @MarkZuber @jinujoseph