"Unexpected timeout from process 1. Process will no longer be monitored" While Egressing Trace in Kubernetes

[kkeirstead]

Description

When egressing a trace in Kubernetes using .NET Monitor 8, the following messages show up in the logs:

2023-03-09T19:43:27.5549626Zwarn: Microsoft.Diagnostics.Tools.Monitor.ServerEndpointInfoSource[52] Unexpected timeout from process 1. Process will no longer be monitored. 2023-03-09T19:43:27.5581934Zinfo: Microsoft.Diagnostics.Tools.Monitor.CollectionRules.CollectionRuleService[44] => TargetProcessId:1 TargetRuntimeInstanceCookie:f29dfb3c42de421eb8cdbb65fec4b124 Stopping collection rules. 2023-03-09T19:43:27.5588928Zinfo: Microsoft.Diagnostics.Tools.Monitor.CollectionRules.CollectionRuleService[45] => TargetProcessId:1 TargetRuntimeInstanceCookie:f29dfb3c42de421eb8cdbb65fec4b124 All collection rules have stopped.

Checking the /processes route while egressing shows that there are no processes present - after completing the egress operation, if there aren't collection rules, the /processes route once again shows the default process (with a persistent UUID). If there are collection rules, these will fail to resume, and the /processes route will not show the default process again.

It's also worth noting that the egress is successful, and that dotnet-monitor continues to work as expected after this issue.

Configuration

DM - image: mcr.microsoft.com/dotnet/monitor:8.0-preview
Target - image: mcr.microsoft.com/dotnet/samples:aspnetapp
ConnectionMode - Listen
API Operation - curl.exe -v "http://localhost:52323/trace?pid=1&durationSeconds=5&egressProvider=monitorBlob"

Regression?

This issue appears to be a regression - the .NET Monitor 7.0.x image in Kubernetes with the exact same configuration does not have this issue.

The .NET Monitor 7.1 image does have this issue.

Other information

[jander-msft]

Let's check if the 7.1 image exhibits the same issue; if it does, it's likely due to some change in the NETCore.Client library from dotnet/diagnostics.

[kkeirstead]

Let's check if the 7.1 image exhibits the same issue; if it does, it's likely due to some change in the NETCore.Client library from dotnet/diagnostics.

Confirming that the 7.1 image does have this issue.

[jander-msft]

This has been fixed in the 8.0 Preview 3 and 7.1.1 releases.