Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Access violation (crashes the process) while writing EventSource events #99603

Closed
minkom-ms opened this issue Mar 12, 2024 · 7 comments · Fixed by #99606
Closed

Access violation (crashes the process) while writing EventSource events #99603

minkom-ms opened this issue Mar 12, 2024 · 7 comments · Fixed by #99606
Labels
area-System.Net.Http bug tenet-reliability Reliability/stability related issue (stress, load problems, etc.)
Milestone

Comments

@minkom-ms
Copy link

minkom-ms commented Mar 12, 2024

Describe the bug

There is a path in the logging of EventSource events in the HTTP stack which causes an Access Violation.

To Reproduce

Issue appears entirely in .NET call stacks so there are no specific steps to reproduce.

Exceptions (if any)

Application: Microsoft.Office.AugLoop.ServerCore.YarpGateway.exe
CoreCLR Version: 8.0.224.6711
.NET Version: 8.0.2
Description: The process was terminated due to an unhandled exception.
Exception Info: System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
Stack:
   at System.Buffer.Memmove(Byte ByRef, Byte ByRef, UIntPtr)
   at System.String.Ctor(Char*, Int32, Int32)
   at System.Diagnostics.Tracing.EventSource.DecodeObjects(System.Object[], System.Type[], EventData*)
   at System.Diagnostics.Tracing.EventSource.WriteToAllListeners(System.Diagnostics.Tracing.EventWrittenEventArgs, Int32, EventData*)
   at System.Diagnostics.Tracing.EventSource.WriteEventWithRelatedActivityIdCore(Int32, System.Guid*, Int32, EventData*)
   at System.Net.Http.HttpTelemetry.WriteEvent(Int32, Byte, Byte, Int64, System.String, System.String, Int32, System.String)
   at System.Net.Http.HttpConnectionBase..ctor(System.Net.Http.HttpConnectionPool, System.Net.IPEndPoint)
   at System.Net.Http.HttpConnectionPool+<ConstructHttp11ConnectionAsync>d__108.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.Net.Http.HttpConnectionPool+<ConstructHttp11ConnectionAsync>d__108, System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ConstructHttp11ConnectionAsync>d__108 ByRef)
   at System.Net.Http.HttpConnectionPool+<CreateHttp11ConnectionAsync>d__105.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Http.HttpConnectionPool+<CreateHttp11ConnectionAsync>d__105, System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Net.Http.HttpConnectionPool+<ConnectAsync>d__103.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.ValueTuple`3[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Http.HttpConnectionPool+<ConnectAsync>d__103, System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Net.Http.ConnectHelper+<EstablishSslConnectionAsync>d__2.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Http.ConnectHelper+<EstablishSslConnectionAsync>d__2, System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.Threading.Tasks.VoidTaskResult>, System.Threading.Tasks.VoidTaskResult)
   at System.Net.Security.SslStream+<ForceAuthenticationAsync>d__150`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream+<ForceAuthenticationAsync>d__150`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Net.Security.SslStream+<ReceiveHandshakeFrameAsync>d__151`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream+<ReceiveHandshakeFrameAsync>d__151`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
   at System.Net.Security.SslStream+<EnsureFullTlsFrameAsync>d__161`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Net.Sockets.SocketAsyncEventArgs+<>c.<.cctor>b__173_0(UInt32, UInt32, System.Threading.NativeOverlapped*)
   at System.Threading.ThreadPoolTypedWorkItemQueue`2[[System.Threading.PortableThreadPool+IOCompletionPoller+Event, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Threading.PortableThreadPool+IOCompletionPoller+Callback, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].System.Threading.IThreadPoolWorkItem.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()

Further technical details

CLR and .Net versions are in the call stack above.

@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Mar 12, 2024
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@KalleOlaviNiemitalo
Copy link

There is a bug here:

const int NumEventDatas = 7;
EventData* descrs = stackalloc EventData[NumEventDatas];
fixed (char* arg4Ptr = arg4)
fixed (char* arg5Ptr = arg5)
fixed (char* arg7Ptr = arg7)
{
descrs[0] = new EventData
{
DataPointer = (IntPtr)(&arg1),
Size = sizeof(byte)
};
descrs[1] = new EventData
{
DataPointer = (IntPtr)(&arg2),
Size = sizeof(byte)
};
descrs[2] = new EventData
{
DataPointer = (IntPtr)(&arg3),
Size = sizeof(long)
};
descrs[3] = new EventData
{
DataPointer = (IntPtr)arg4Ptr,
Size = (arg4.Length + 1) * sizeof(char)
};
descrs[4] = new EventData
{
DataPointer = (IntPtr)arg5Ptr,
Size = (arg5.Length + 1) * sizeof(char)
};
descrs[5] = new EventData
{
DataPointer = (IntPtr)(&arg6),
Size = sizeof(int)
};
descrs[6] = new EventData
{
DataPointer = (IntPtr)arg7Ptr,
Size = (arg7.Length + 1) * sizeof(char)
};
}
WriteEventCore(eventId, NumEventDatas, descrs);

The WriteEventCore(eventId, NumEventDatas, descrs) call is not within the fixed block. This allows the garbage collector to move the strings and invalidate the pointers that were stored in descrs[3].DataPointer, descrs[4].DataPointer, and descrs[6].DataPointer. I don't know whether this is what actually causes the crash, but it could very well be.

@KalleOlaviNiemitalo
Copy link

Repro by internal reflection

using System;
using System.Diagnostics.Tracing;
using System.Net;
using System.Net.Http;
using System.Reflection;
using System.Threading;
using System.Threading.Tasks;

class Program
{
    static void Main()
    {
        Type httpTelemetryType = Type.GetType("System.Net.Http.HttpTelemetry, System.Net.Http", throwOnError: true)!;
        FieldInfo logField = httpTelemetryType.GetField("Log") ?? throw new Exception("HttpTelemetry.Log field does not exist");
        var httpTelemetry = (EventSource)logField.GetValue(null) ?? throw new Exception("HttpTelemetry.Log is null");

        // public void Http11ConnectionEstablished(long connectionId, string scheme, string host, int port, IPEndPoint? remoteEndPoint)
        Action<long, string, string, int, IPEndPoint?> http11ConnectionEstablished
            = httpTelemetryType.GetMethod("Http11ConnectionEstablished")
            .CreateDelegate<Action<long, string, string, int, IPEndPoint?>>(httpTelemetry);

        // public void Http11ConnectionClosed(long connectionId)
        Action<long> http11ConnectionClosed
            = httpTelemetryType.GetMethod("Http11ConnectionClosed")
            .CreateDelegate<Action<long>>(httpTelemetry);

        var listener = new DummyListener();
        listener.EnableEvents(httpTelemetry, EventLevel.Verbose);

        var gcInducerThread = new Thread(() =>
        {
            while (true)
            {
                Thread.Sleep(TimeSpan.FromSeconds(1));
                Console.Write("GC...");
                GC.Collect();
                Console.WriteLine("finished.");
            }
        });
        gcInducerThread.IsBackground = true;
        gcInducerThread.Start();

        Console.WriteLine("Running...");
        Parallel.For(1, 1_000_000_000, (long connectionId) =>
        {
            var remoteEndPoint = new IPEndPoint(address: connectionId, port: 80);
            http11ConnectionEstablished(connectionId, "http", $"host{connectionId}", remoteEndPoint.Port, remoteEndPoint);
            http11ConnectionClosed(connectionId);
        });
    }
}

class DummyListener : EventListener
{
}

@baronfel baronfel transferred this issue from dotnet/sdk Mar 12, 2024
@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Mar 12, 2024
@KalleOlaviNiemitalo
Copy link

I guess area-System.Net.Http.

@vcsjones vcsjones added area-System.Net.Http and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Mar 12, 2024
@MihaZupan MihaZupan added the bug label Mar 12, 2024
@MihaZupan MihaZupan added this to the 9.0.0 milestone Mar 12, 2024
@MihaZupan MihaZupan added tenet-reliability Reliability/stability related issue (stress, load problems, etc.) and removed untriaged New issue has not been triaged by the area owner labels Mar 12, 2024
@MihaZupan
Copy link
Member

Looks like we introduced this one in 8.0: #88853
Thanks for calling out the fixed scope issue @KalleOlaviNiemitalo

@MihaZupan
Copy link
Member

Reopening to track backport consideration

@MihaZupan
Copy link
Member

MihaZupan commented Mar 21, 2024

Fixed in main (9.0) in PR #99606, and in 8.0.5 in PR #99607.

@github-actions github-actions bot locked and limited conversation to collaborators Apr 21, 2024
@karelz karelz modified the milestones: 9.0.0, 8.0.x Jun 24, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-System.Net.Http bug tenet-reliability Reliability/stability related issue (stress, load problems, etc.)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants