Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unregister onExit event handler on disposing interactive host #24058

Merged
merged 19 commits into from
Jan 26, 2018
Merged
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
using System.Diagnostics;
using System.IO;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.CodeAnalysis.ErrorReporting;
using Roslyn.Utilities;

Expand All @@ -16,12 +17,13 @@ internal sealed class RemoteService
public readonly Process Process;
public readonly Service Service;
private readonly int _processId;
private SemaphoreSlim _disposeSemaphore = new SemaphoreSlim(initialCount: 1);

// output pumping threads (stream output from stdout/stderr of the host process to the output/errorOutput writers)
private Thread _readOutputThread; // nulled on dispose
private Thread _readErrorOutputThread; // nulled on dispose
private InteractiveHost _host; // nulled on dispose
private bool _disposing; // set to true on dispose
private volatile ProcessExitHandlerStatus _processExitHandlerStatus; // set to Handled on dispose

internal RemoteService(InteractiveHost host, Process process, int processId, Service service)
{
Expand All @@ -30,10 +32,10 @@ internal RemoteService(InteractiveHost host, Process process, int processId, Ser
Debug.Assert(service != null);

_host = host;
_disposing = false;
this.Process = process;
_processId = processId;
this.Service = service;
_processExitHandlerStatus = ProcessExitHandlerStatus.Uninitialized;

// TODO (tomat): consider using single-thread async readers
_readOutputThread = new Thread(() => ReadOutput(error: false));
Expand All @@ -49,35 +51,36 @@ internal RemoteService(InteractiveHost host, Process process, int processId, Ser

internal void HookAutoRestartEvent()
{
const int ProcessExitHooked = 1;
const int ProcessExitHandled = 2;

int processExitHandling = 0;
using (_disposeSemaphore.DisposableWait())
{
// hook the event only once per process:
if (_processExitHandlerStatus == ProcessExitHandlerStatus.Uninitialized)
{
Process.Exited += ProcessExitedHandler;
_processExitHandlerStatus = ProcessExitHandlerStatus.Hooked;
}
}
}

async void localHandler(object _, EventArgs __)
private async void ProcessExitedHandler(object _, EventArgs __)
{
try
{
try
using (await _disposeSemaphore.DisposableWaitAsync().ConfigureAwait(false))
{
if (Interlocked.Exchange(ref processExitHandling, ProcessExitHandled) == ProcessExitHooked)
if (_processExitHandlerStatus == ProcessExitHandlerStatus.Hooked)
{
Process.Exited -= localHandler;

if (!_disposing)
{
await _host.OnProcessExited(Process).ConfigureAwait(false);
}
Process.Exited -= ProcessExitedHandler;
_processExitHandlerStatus = ProcessExitHandlerStatus.Handled;
// Should set _processExitHandlerStatus before calling OnProcessExited to avoid deadlocks.
// Calling the host should be within the lock to prevent its disposing during the execution.
await _host.OnProcessExited(Process).ConfigureAwait(false);
}
}
catch (Exception e) when (FatalError.Report(e))
{
throw ExceptionUtilities.Unreachable;
}
}

// hook the event only once per process:
if (Interlocked.Exchange(ref processExitHandling, ProcessExitHooked) == 0)
catch (Exception e) when (FatalError.Report(e))
{
Process.Exited += localHandler;
throw ExceptionUtilities.Unreachable;
}
}

Expand Down Expand Up @@ -113,37 +116,38 @@ private void ReadOutput(bool error)

internal void Dispose(bool joinThreads)
{
// set _disposing so that we don't attempt restart the host anymore:
_disposing = true;

InitiateTermination(Process, _processId);

// only tests require joining the threads, so we can wait synchronously
if (joinThreads)
// There can be a call from host initiated from OnProcessExit.
// This check on the beginning helps to avoid a reentrancy.
if (_processExitHandlerStatus == ProcessExitHandlerStatus.Hooked)
Copy link
Member

@jasonmalinowski jasonmalinowski Jan 25, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the problem here that this can deadlock because if the _host.OnProcessExited() call is happening above and that's still holding the lock, and it tries to call back into Dispose(), that will deadlock? Why can't we call _host.OnProcessExited() outside the lock?

I guess a better comment here would be appreciated. The use of volatile here is generally scary. I think it works, but that doesn't make it less scary. #Resolved

Copy link
Member

@jasonmalinowski jasonmalinowski Jan 25, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(this question is best answered by just adding more comments so the code is clearer vs. answering my question in GitHub) #Resolved

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, Jason! I added more comments to clarify the scenario.

Maybe there can be an option to avoid using volatile and removing the _host.OnProcessExited() outside of the lock. We could not see these options so far. The problem we fixed that the host is being disposed during _host.OnProcessExited() execution. So, the fix is to protect it from the dispose.


In reply to: 163994639 [](ancestors = 163994639)

{
if (_readOutputThread != null)
using (_disposeSemaphore.DisposableWait())
{
try
if (_processExitHandlerStatus == ProcessExitHandlerStatus.Hooked)
{
_readOutputThread.Join();
}
catch (ThreadStateException)
{
// thread hasn't started
Process.Exited -= ProcessExitedHandler;
_processExitHandlerStatus = ProcessExitHandlerStatus.Handled;
}
}
}

if (_readErrorOutputThread != null)
{
try
{
_readErrorOutputThread.Join();
}
catch (ThreadStateException)
{
// thread hasn't started
}
}
InitiateTermination(Process, _processId);

try
{
_readOutputThread?.Join();
}
catch (ThreadStateException)
{
// thread hasn't started
}

try
{
_readErrorOutputThread?.Join();
}
catch (ThreadStateException)
{
// thread hasn't started
}

// null the host so that we don't attempt to write to the buffer anymore:
Expand All @@ -166,6 +170,13 @@ internal static void InitiateTermination(Process process, int processId)
Debug.WriteLine("InteractiveHostProcess: can't terminate process {0}: {1}", processId, e.Message);
}
}

private enum ProcessExitHandlerStatus
{
Uninitialized = 0,
Hooked = 1,
Handled = 2
}
}
}
}
21 changes: 6 additions & 15 deletions src/Interactive/Features/Interactive/Core/InteractiveHost.cs
Original file line number Diff line number Diff line change
Expand Up @@ -89,11 +89,6 @@ internal IpcServerChannel _ServerChannel
get { return _serverChannel; }
}

internal void Dispose(bool joinThreads)
{
Dispose(joinThreads, disposing: true);
}

#endregion

private static string GenerateUniqueChannelLocalName()
Expand Down Expand Up @@ -226,25 +221,21 @@ private bool CheckAlive(Process process)

~InteractiveHost()
{
Dispose(joinThreads: false, disposing: false);
DisposeRemoteService(disposing: false);
}

public void Dispose()
{
Dispose(joinThreads: false, disposing: true);
DisposeChannel();
DisposeRemoteService(disposing: true);
GC.SuppressFinalize(this);
}

private void Dispose(bool joinThreads, bool disposing)
private void DisposeRemoteService(bool disposing)
{
if (disposing)
{
GC.SuppressFinalize(this);
DisposeChannel();
}

if (_lazyRemoteService != null)
{
_lazyRemoteService.Dispose(joinThreads);
_lazyRemoteService.Dispose(disposing);
_lazyRemoteService = null;
}
}
Expand Down
2 changes: 1 addition & 1 deletion src/Interactive/HostTest/AbstractInteractiveHostTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ internal static string GetInteractiveHostPath()
internal static void DisposeInteractiveHostProcess(InteractiveHost process)
{
IpcServerChannel serverChannel = process._ServerChannel;
process.Dispose(joinThreads: true);
process.Dispose();

var listenerThread = (Thread)s_ipcServerChannelListenerThread.GetValue(serverChannel);
listenerThread.Join();
Expand Down