Failed USB connection via port 54050, error 61, in tvOS arm64 Release AllSubsets_Mono

[carlossanlop]

• Affected PR: [7.0] Fix intellisense xml file selection from 'net' or 'dotnet-plat-ext'  #82554
• Affected queue: Build tvOS arm64 Release AllSubsets_Mono
• Job result: https://dev.azure.com/dnceng-public/public/public%20Team/_build/results?buildId=182544&view=logs&j=f4520fb1-1559-5885-1d9c-3cb3f6a85e23&t=6c7a8cfe-f92e-569a-eef9-b2ad3e13056d&l=57
• Artifacts: https://dev.azure.com/dnceng-public/public/_build/results?buildId=182544&view=ms.vss-test-web.build-test-results-tab&runId=3552693&resultId=100001&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab
• console.log output file repeatedly shows the USB connection error below: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-82554-merge-8bea006df9294c8cbf/System.Runtime.Tests/1/console.76e18c2e.log?helixlogtype=result

[13:58:26] dbug: [TCP tunnel] Xamarin.Hosting: Failed to connect to port 54050 (8915) on device (error: 61)
[13:58:26] dbug: [TCP tunnel] Xamarin.Hosting: Attempting USB tunnel between the port 54050 on the device and the port 54050 (8915) on the mac: 61

It's a bit confusing, since there's a net.dot.System.Runtime.Tests.log output file that shows some succeeded System.Runtime test executions.

Anyway, the USB failure seems to be unique, so I'm adding the "known issue" json text to track it.
Edit. Removed the match string because the error message was matching un-related failures.

{
  "ErrorMessage": "",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323893
Error message validated: Xamarin.Hosting: Failed to connect to port
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:28:49 PM UTC

[ghost]

Tagging subscribers to 'os-tvos': @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

Issue Details

---

• Affected PR: [7.0] Fix intellisense xml file selection from 'net' or 'dotnet-plat-ext'  #82554
• Affected queue: Build tvOS arm64 Release AllSubsets_Mono
• Job result: https://dev.azure.com/dnceng-public/public/public%20Team/_build/results?buildId=182544&view=logs&j=f4520fb1-1559-5885-1d9c-3cb3f6a85e23&t=6c7a8cfe-f92e-569a-eef9-b2ad3e13056d&l=57
• Artifacts: https://dev.azure.com/dnceng-public/public/_build/results?buildId=182544&view=ms.vss-test-web.build-test-results-tab&runId=3552693&resultId=100001&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab
• console.log output file repeatedly shows the USB connection error below: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-82554-merge-8bea006df9294c8cbf/System.Runtime.Tests/1/console.76e18c2e.log?helixlogtype=result

[13:58:26] dbug: [TCP tunnel] Xamarin.Hosting: Failed to connect to port 54050 (8915) on device (error: 61)
[13:58:26] dbug: [TCP tunnel] Xamarin.Hosting: Attempting USB tunnel between the port 54050 on the device and the port 54050 (8915) on the mac: 61

It's a bit confusing, since there's a net.dot.System.Runtime.Tests.log output file that shows some succeeded System.Runtime test executions.

Anyway, the USB failure seems to be unique, so I'm adding the "known issue" json text to track it.

{
  "ErrorMessage": "Xamarin.Hosting: Failed to connect to port 54050 (8915) on device (error: 61)",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Author:	carlossanlop
Assignees:	-
Labels:	arch-arm64, blocking-clean-ci, area-VM-meta-mono, runtime-mono, os-tvos, Known Build Error
Milestone:	7.0.x

[lewing]

dotnet/arcade#11683

[steveisok]

Might be better to keep this open since it's tagged with a known build error label. I don't think we need to have blocking clean ci attached to it.

[carlossanlop]

I originally logged this issue for tvOS, but I'm also seeing it in iOS arm64. Here is a fresh example:

• PR: Implement stream-based ZipFile ExtractToDirectory and CreateFromDirectory method overloads #85491
• Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=288704&view=results
• Job: ios-arm64 Release AllSubsets_Mono
• Job results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=288704&view=logs&j=7f49df26-8126-5de3-bf2f-6ac6bde01830&t=c62b6a8c-bf71-5057-7f45-9c355fe0b802
• Log file: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-85491-merge-e34e065cea094eb8b3/System.Runtime.Tests.Attempt.3/1/console.43a8ef2d.log?helixlogtype=result

[22:47:19] dbug: [TCP tunnel] Xamarin.Hosting: Attempting USB tunnel between the port 52498 on the device and the port 52498 (4813) on the mac: 61
[22:47:19] dbug: [TCP tunnel] Xamarin.Hosting: Failed to connect to port 52498 (4813) on device (error: 61)

[kotlarmilos]

https://helix.dot.net/api/jobs/4cbf9a6d-6389-43f9-9e2c-16540105525e/workitems?api-version=2019-06-17

Both tests pass based on the logs, but one of them is unable to establish a connection via TCP, while the other establishes a connection every time. This failure seems to be permanent and could be a good starting point for identifying the issue.

When I attempted to reproduce the error locally, I encountered the same issue. However, I have not opened any ports, which might be a possible cause. I assume that the CI machines have ports 49152-65535 (source https://support.apple.com/en-us/HT202944).

I wonder if certain ports are busy or if there might be a firewall misconfiguration.

@premun Any ideas why the error persists when running the functional test?

/cc @akoeplinger @steveisok

[premun]

@kotlarmilos if this was firewall misconfiguration, I think we should see the same set of machines failing. I don't think this is the case: https://dataexplorer.azure.com/clusters/engsrvprod/databases/engineeringdata?query=H4sIAAAAAAAAAx3MuwoCMRAF0N6vGLbabQI+ECy2EgsLBSvrMbmQoJmESaKr+PEu9odzTXo/VsSy+NLLQ0GHKdR9cqBxpN2KWBxdGhrOHEFBqO9SmcxybTi67cZwzg/Up0kZ0g3zUlqMrOEDsqlJ7Qe6venE1gf5HzNRiINSDrCetf4ANsm6/4QAAAA=

Seems like this is not tied to specific machines?

@premun Any ideas why the error persists when running the functional test?

Do I understand it correctly that you have 100% repro with a specific app?

[kotlarmilos]

Host: osx-arm64
Device: iPhone iOS 16.5.1
Xcode: 14.3.1 (14E300c)

1. Install xharness locally (I don't believe it has anything with the repro, that is just my local setup):

# Using bash on Linux/MacOS
curl -L https://aka.ms/get-xharness | bash -

2. Download the bundle and place it in a test directory.
   tracing_eventpipe_tcp_issue.zip

3. Run the following script from the local directory:

./xharness apple test --app "<LOCAL_DIRECTORY>/tracing_eventpipe_tcp_issue.app" --target "ios-device" --output-directory "<LOCAL_DIRECTORY>/output" --set-env=MONO_APPLE_APP_ENTRY_POINT_LIB_NAME=testdir-buffersize/buffersize.dll --set-env=MONO_APPLE_APP_ASSEMBLY_LOAD_PREFIX=testdir-buffersize --xcode /Applications/Xcode.app -v --launch-timeout 00:05:00

The test should start executing and may fail (due to other reasons), but the TCP connection is not established. The app fails locally with

fail: Application test run crashed
      TCP Tunnel Connection Failed to connect to TCP port

[kotlarmilos]

For the apple test command, XHarness expects the application to contain a TestRunner which is a library you can find in this repository. This library executes unit tests similarly how you would execute them on other platforms. However, the TestRunner from this repository contains more mechanisms that help to work around some issues (mostly in Apple platforms).

The runtime test did not utilize the TestRunner, and it was expected to fail. After further investigation and monitoring of the CI, it appears that the TCP failures might be related to the app. If such a failure occurs, ensure that the app functions correctly or consider using apple run commands.

[ericstj]

Removed [7.0] from the title since this appears to be happening across branches.

[kotlarmilos]

@ivanpovazan If you try to run the library tests using the apple run command, please share your results. I suspect the issue is either app-specific or related to the AppleTestRunner. We haven't observed this problem in runtime tests that utilize the apple run command.

[ivanpovazan]

@kotlarmilos and myself are looking into this.

[matouskozak]

Removing the Xamarin.Hosting: Failed to connect to port match string because it's too general and it's matching un-related failures. I'll keep the issue fyi: @ivanpovazan @kotlarmilos

[jeffschwMSFT]

removing 'blocking-clean-ci' as it has not failed in 30 days

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0