Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dotnet restore (from nuget) doesn't complete on M1 mac #12329

Closed
tpitman opened this issue Dec 19, 2022 · 47 comments
Closed

dotnet restore (from nuget) doesn't complete on M1 mac #12329

tpitman opened this issue Dec 19, 2022 · 47 comments
Assignees
Labels
Functionality:Restore Platform:Docker All NuGet on docker scenarios Platform:Xplat Resolution:External This issue appears to be External to nuget Type:Bug

Comments

@tpitman
Copy link

tpitman commented Dec 19, 2022

NuGet Product Used

dotnet.exe

Product Version

dotnet: 7.0.1, arm64 - msbuild: 16.10.1. nuget. Not sure, this gets executed inside of a docker-compose linux build.

Worked before?

Worked in net6

Impact

I'm unable to use this version

Repro Steps & Context

  1. Create a asp.net project on M1 Mac
  2. Add docker support that includes docker compose support
  3. Go to command line in the solution folder where the docker compose file exists
  4. Run the command: docker-compose build
  5. Wait for any of the tasks that are: RUN dotnet restore "[project file].csproj
  6. Notice that on an Intel mac this takes just a few seconds
  7. Notice that on an M1 mac this continues and never finishes (I let it run for 15 minutes and it never completed)

Verbose Logs

No response

@erdembayar
Copy link
Contributor

Thank you for filing this issue.
Could you able to test if restore works with simple solution without docker?
Maybe you can create a simple solution with
dotnet new console
dotnet add package newtonsoft.json

If above work, then try with simple bare minimum asp.net project.

@erdembayar erdembayar added Functionality:Restore Platform:Docker All NuGet on docker scenarios Platform:Xplat and removed Triage:Untriaged Platform:Docker All NuGet on docker scenarios labels Dec 20, 2022
@tpitman
Copy link
Author

tpitman commented Dec 20, 2022

Restore seems to work fine in all cases when not building for docker. I believe it is when trying to build for x86 docker images that the issue occurs.

I created a new asp.net 7 web app.

I did right click on the project and added docker support.

I then went to a terminal and did docker-compose build. This works.

Then I add the following line to the docker-compose.yml file:

platform: linux/amd64

Then I do docker-compose build again and it has the issue.

I need that line because Azure requires x86 linux containers, so I have to force the output image to be amd64. If I don't then the resulting image is for arm and won't run on Azure.

Here is the full docker-compose file:

version: '3.7'

services:
nugetdotnetrestoretest:
platform: linux/amd64
image: ${DOCKER_REGISTRY-}nugetdotnetrestoretest
build:
context: .
dockerfile: NugetDotnetRestoreTest/Dockerfile

@erdembayar
Copy link
Contributor

Could we be able to share us NugetDotnetRestoreTest code after removing sensitive information?

@tpitman
Copy link
Author

tpitman commented Dec 20, 2022

It is probably just as easy for you to create a new one. I didn't change anything in the project except adding that line in the docker-compose.yml file.

It might even validate that it happens for everyone.

If you are not able to then I will go through all the files and remove any personal information and upload it.

Thank you for helping out.

@erdembayar erdembayar added the WaitingForClientTeam Customer replied, needs attention from client team. Do not apply this label manually. label Dec 22, 2022
@erdembayar
Copy link
Contributor

Docker is not our expertise, we might miss out something from repro. If you could provide us repro code, then it would be easier for us to diagnose.
By the way does this repro on non-M1 MAC? Don't have M1 MAC ready for test.

@ghost ghost added WaitingForCustomer Applied when a NuGet triage person needs more info from the OP and removed WaitingForClientTeam Customer replied, needs attention from client team. Do not apply this label manually. labels Dec 22, 2022
@tpitman
Copy link
Author

tpitman commented Dec 22, 2022

Ok. I uploaded the example: https://github.com/tpitman/NugetDotnetRestoreTest

It will work fine on an Intel Mac.

You will need to do the docker-compose build on an M1 Mac.

The issue is completely related to trying to build on M1 mac to Linux container that targets Intel processors.

@ghost ghost added WaitingForClientTeam Customer replied, needs attention from client team. Do not apply this label manually. and removed WaitingForCustomer Applied when a NuGet triage person needs more info from the OP labels Dec 22, 2022
@erdembayar
Copy link
Contributor

Sorry, currently many folks are OOF due to holiday. We need to wait another 1-2 weeks before we can do anything about it.
Thank you.

@ghost ghost added WaitingForCustomer Applied when a NuGet triage person needs more info from the OP and removed WaitingForClientTeam Customer replied, needs attention from client team. Do not apply this label manually. labels Dec 22, 2022
@tpitman
Copy link
Author

tpitman commented Dec 22, 2022

I understand. I can get by by using my Intel mac. Means having to carry around 2, but hopefully you guys will get it figured out soon after the holidays.

@ghost ghost added WaitingForClientTeam Customer replied, needs attention from client team. Do not apply this label manually. and removed WaitingForCustomer Applied when a NuGet triage person needs more info from the OP labels Dec 22, 2022
@erdembayar erdembayar self-assigned this Dec 22, 2022
@erdembayar erdembayar removed the WaitingForClientTeam Customer replied, needs attention from client team. Do not apply this label manually. label Dec 22, 2022
@tobsai
Copy link

tobsai commented Dec 25, 2022

I am seeing the same issue on M1.

@zivkan
Copy link
Member

zivkan commented Dec 27, 2022

Nobody has copied the error message in this issue. I don't have any Mac, let alone a M1 Mac. Is it the same error as #12227? Although it seems like that error might be due to qemu, rather than ARM more generally.

@tpitman
Copy link
Author

tpitman commented Dec 27, 2022

There isn't an error that I have seen. The dotnet restore step just keeps going. I let it go for 10 minutes before giving up.

It is possible that if I let it go long enough maybe it would give an error.

I am also not sure whether they are related because I don't think I am using buildx. I am just running docker-compose build and don't see it reference buildx anywhere.

@zivkan
Copy link
Member

zivkan commented Dec 27, 2022

Can you use dotnet-dump to get a (managed?) memory dump of the process, so we can try to see where it got stuck? Although the direct downloads for dotnet-dump don't have an ARM64 version. This also assumes it's something in managed code causing problems, not the runtime itself.

If dotnet-dump doesn't work on ARM64 Macs, then I can ask some dotnet runtime people for advice on investigating, although there's a good chance nobody will be around until January.

@tpitman
Copy link
Author

tpitman commented Jan 19, 2023

This is a docker container coming directly from mcr.microsoft.com/dotnet/sdk:7.0. There doesn't seem to be any other users.

This works fine when building for ARM on the M1 or when building for Intel on and Intel.

@ghost ghost added WaitingForClientTeam Customer replied, needs attention from client team. Do not apply this label manually. and removed WaitingForCustomer Applied when a NuGet triage person needs more info from the OP labels Jan 19, 2023
@tpitman
Copy link
Author

tpitman commented Jan 19, 2023

I have done some more testing.

dotnet nuget --version works without making any changes to the HOME env variable.

dotnet nuget restore still hangs even after setting HOME to something other than /root

@ZeroSkill1
Copy link

Note that when I found this "fix", I was using SDK version 7.0.102.

@tpitman
Copy link
Author

tpitman commented Jan 19, 2023

Note that when I found this "fix", I was using SDK version 7.0.102.

I too saw a post about this being fixed in 7.0.102, but my testing has not confirmed that. I added code to the example project that output the version using dotnet --version, it showed 7.0.102, but still hung up on the dotnet restore command.

If someone else can take the example project mentioned above and have it work on an M1 mac and give details, I would love to see it.

@joergjo
Copy link

joergjo commented Jan 19, 2023

I retract my previous statement - both your sample project and mine hang as before. Not sure how I fooled myself, but anyway... back to square one.

@tpitman
Copy link
Author

tpitman commented Jan 19, 2023

I retract my previous statement - both your sample project and mine hang as before. Not sure how I fooled myself, but anyway... back to square one.

Sorry to hear that. I was really hoping it was just something I was doing different. Thank you for helping....

@kartheekp-ms
Copy link
Contributor

kartheekp-ms commented Jan 19, 2023

@joergjo - Can you please capture dump of the running process (using dotnet-dump command) on docker environment? I am only interested in the call stack that is causing restore to wait indefinitely.

@ghost ghost added WaitingForCustomer Applied when a NuGet triage person needs more info from the OP and removed WaitingForClientTeam Customer replied, needs attention from client team. Do not apply this label manually. labels Jan 19, 2023
@kartheekp-ms
Copy link
Contributor

dotnet nuget --version works without making any changes to the HOME env variable.

@tpitman - Good to know. In your sample repo, I see that .csproj file don't have any package reference. https://github.com/tpitman/NugetDotnetRestoreTest/blob/main/NugetDotnetRestoreTest/NugetDotnetRestoreTest.csproj

Would it be possible to add a package reference to a package (for example Newtonsoft.Json from nuget.org) and see if restore succeeds?

dotnet add package Newtonsoft.Json --version 13.0.2

or update csproje file as shown below.

<ItemGroup>
   <PackageReference Include="Newtonsoft.Json" Version="13.0.2" />
</ItemGroup>

@tpitman
Copy link
Author

tpitman commented Jan 21, 2023

That did not make any difference that I could tell. It still didn't proceed to do anything.

Good idea, though :-).

@ghost ghost added WaitingForClientTeam Customer replied, needs attention from client team. Do not apply this label manually. and removed WaitingForCustomer Applied when a NuGet triage person needs more info from the OP labels Jan 21, 2023
@jstedfast
Copy link

jstedfast commented Jan 24, 2023

I just tried this (hopefully I did it correctly) and I think I also hit the hang:

[jestedfa@localhost NugetDotnetRestoreTest]$ docker-compose build                  
[+] Building 86.9s (11/17)                                                                                              
 => [internal] load build definition from Dockerfile                                                               0.0s
 => => transferring dockerfile: 869B                                                                               0.0s
 => [internal] load .dockerignore                                                                                  0.0s
 => => transferring context: 2B                                                                                    0.0s
 => [internal] load metadata for mcr.microsoft.com/dotnet/sdk:7.0.102                                              0.2s
 => [internal] load metadata for mcr.microsoft.com/dotnet/aspnet:7.0                                               0.2s
 => [build 1/7] FROM mcr.microsoft.com/dotnet/sdk:7.0.102@sha256:028d360c0f409f13daf8689edfe680dba9d4b651eb8a6f12  0.0s
 => [base 1/2] FROM mcr.microsoft.com/dotnet/aspnet:7.0@sha256:debec27f5579ddee99c2fd5616ee18def873aa83ad68e7b563  0.0s
 => [internal] load build context                                                                                  0.0s
 => => transferring context: 848.98kB                                                                              0.0s
 => CACHED [base 2/2] WORKDIR /app                                                                                 0.0s
 => CACHED [final 1/2] WORKDIR /app                                                                                0.0s
 => CACHED [build 2/7] WORKDIR /src                                                                                0.0s
 => CACHED [build 3/7] COPY [NugetDotnetRestoreTest/NugetDotnetRestoreTest.csproj, NugetDotnetRestoreTest/]        0.0s
 => [build 4/7] RUN dotnet restore "NugetDotnetRestoreTest/NugetDotnetRestoreTest.csproj"

I am on a MacBook Pro M1 machine with the "Apple Chip" version of Docker installed (by default, when you go to the Download section of Docker's website, it offers you the x64 version - you have to specifically click the "Apple Chip" link below the big "Download" button).

I also modified the Docker file to use dotnet restore -v diag to see if anything got logged, but nothing:

 => [build 4/7] RUN dotnet restore -v diag "NugetDotnetRestoreTest/NugetDotnetRestoreTest.csproj"                 57.2s

What I can say is that if I use a bogus verbosity level, the dotnet restore command does error out:

 => ERROR [build 4/7] RUN dotnet restore -v bogus "NugetDotnetRestoreTest/NugetDotnetRestoreTest.csproj"           3.7s
------                                                                                                                  
 > [build 4/7] RUN dotnet restore -v bogus "NugetDotnetRestoreTest/NugetDotnetRestoreTest.csproj":                      
#0 3.289 System.InvalidOperationException: Cannot parse argument 'bogus' for option '-v' as expected type 'Microsoft.DotNet.Cli.VerbosityOptions'.                                                                                              
#0 3.289    at System.CommandLine.Binding.ArgumentConverter.GetValueOrDefault[T](ArgumentConversionResult result)       
#0 3.289    at System.CommandLine.Parsing.OptionResult.GetValueOrDefault[T]()                                           
#0 3.290    at System.CommandLine.Parsing.SymbolResult.GetValueForOption[T](Option`1 option)
#0 3.290    at Microsoft.DotNet.Cli.ParseResultExtensions.SafelyGetValueForOption[T](ParseResult parseResult, Option`1 optionToGet)
#0 3.290    at Microsoft.DotNet.Cli.Telemetry.TelemetryFilter.LogVerbosityForAllTopLevelCommand(ICollection`1 result, ParseResult parseResult, String topLevelCommandName, Dictionary`2 measurements)
#0 3.290    at Microsoft.DotNet.Cli.Telemetry.TelemetryFilter.Filter(Object objectToFilter)
#0 3.290    at Microsoft.DotNet.Cli.Utils.TelemetryEventEntry.SendFiltered(Object o)
#0 3.290    at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, TimeSpan startupTime, ITelemetry telemetryClient)
#0 3.290    at Microsoft.DotNet.Cli.Program.Main(String[] args)
------
failed to solve: executor failed running [/bin/sh -c dotnet restore -v bogus "NugetDotnetRestoreTest/NugetDotnetRestoreTest.csproj"]: exit code: 1

Not sure if that information is helpful or not, but maybe it helps determine if the issue is the dotnet program itself hanging at startup vs the dotnet program hanging trying to restore a project.

@kartheekp-ms
Copy link
Contributor

kartheekp-ms commented Jan 24, 2023

I also modified the Docker file to use dotnet restore -v diag to see if anything got logged, but nothing:
=> [build 4/7] RUN dotnet restore -v diag "NugetDotnetRestoreTest/NugetDotnetRestoreTest.csproj"

@jstedfast - Thank you for taking the time to reproduce this issue. Can you please follow the steps suggested in dotnet/dotnet-docker#4285 (reply in thread) and dotnet/dotnet-docker#4285 (reply in thread) and provide us with the call stack that is causing this issue?

@ghost ghost added WaitingForCustomer Applied when a NuGet triage person needs more info from the OP and removed WaitingForClientTeam Customer replied, needs attention from client team. Do not apply this label manually. labels Jan 24, 2023
@jstedfast
Copy link

I couldn't figure out how to get dotnet-dump working for this yesterday and have to get back to other tasks, but if someone can provide me with a clearer set of instructions, I can give it another try.

@erdembayar
Copy link
Contributor

erdembayar commented Jan 26, 2023

I couldn't figure out how to get dotnet-dump working for this yesterday and have to get back to other tasks, but if someone can provide me with a clearer set of instructions, I can give it another try.

Sorry for getting back this issue late. I was busy with other stuff, also don't have MAC M1 machine, catching up today.
I just tested that dotnet-dump doesn't work on MAC M1 machine, it only works on intel base MAC.
If look at the doc then it only supports intel x64.

@tpitman
I can repro it. This problem is regression with sdk7.0 from sdk6.0, everything works as expected on sdk 6.0. So, if possible, you can temporarily unblock yourself by targeting sdk6.0 image in your Dockerfile until actual solution is available. Sorry, if it's not an option for you. I'll post you more updates with my findings soon, looks like docker image issue considering restore/build works outside docker image on M1 machine.

@erdembayar
Copy link
Contributor

erdembayar commented Jan 26, 2023

@tpitman @jstedfast

After playing for while with dotnet restore with docker I randomly hit memory violation error message in log, but still not able to get call stack.
Finally got quite consistent repro with stack trace with following Dockerfile. Here dotnet new list doesn't involve any nuget operation, so it's not NuGet problem. This one is either docker image or sdk/runtime problem, you can see memory violation/corruption error in stack trace.

FROM mcr.microsoft.com/dotnet/sdk:7.0 as base
FROM mcr.microsoft.com/dotnet/sdk:7.0 as build
WORKDIR /src
RUN `dotnet new list`
  1. Here is stack trace dotnet new list:
DEBU[0000] using default config store "/Users/tester/.docker/buildx" 
[+] Building 4.8s (7/10)                                                                                     
 => [internal] load build definition from Dockerfile                                                    0.0s
 => => transferring dockerfile: 32B                                                                     0.0s
 => [internal] load .dockerignore                                                                       0.0s
 => => transferring context: 2B                                                                         0.0s
 => [internal] load metadata for mcr.microsoft.com/dotnet/sdk:7.0                                       0.0s
 => [build 1/6] FROM mcr.microsoft.com/dotnet/sdk:7.0@sha256:028d360c0f409f13daf8689edfe680dba9d4b651e  0.0s
 => [internal] load build context                                                                       0.0s
 => => transferring context: 2.08kB                                                                     0.0s
 => CACHED [build 2/6] WORKDIR /src                                                                     0.0s
 => ERROR [build 3/6] RUN dotnet new list                                                               4.7s
------                                                                                                       
 > [build 3/6] RUN dotnet new list:                                                                          
#0 4.412 Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.                                                             
#0 4.418    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1[[System.Int32, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetStateMachineBox[[System.Net.Security.SslStream+<ReadAsyncInternal>d__159`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ReadAsyncInternal>d__159`1<System.Net.Security.AsyncReadWriteAdapter> ByRef, StateMachineBox<Int32> ByRef)
#0 4.418    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1[[System.Int32, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].AwaitUnsafeOnCompleted[[System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1+ConfiguredValueTaskAwaiter[[System.Int32, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream+<ReadAsyncInternal>d__159`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](ConfiguredValueTaskAwaiter<Int32> ByRef, <ReadAsyncInternal>d__159`1<System.Net.Security.AsyncReadWriteAdapter> ByRef, StateMachineBox<Int32> ByRef)
#0 4.420    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1[[System.Int32, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].AwaitUnsafeOnCompleted[[System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1+ConfiguredValueTaskAwaiter[[System.Int32, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream+<ReadAsyncInternal>d__159`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](ConfiguredValueTaskAwaiter<Int32> ByRef, <ReadAsyncInternal>d__159`1<System.Net.Security.AsyncReadWriteAdapter> ByRef)
#0 4.420    at System.Net.Security.SslStream+<ReadAsyncInternal>d__159`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
#0 4.420    at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.Net.Security.SslStream+<ReadAsyncInternal>d__159`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ReadAsyncInternal>d__159`1<System.Net.Security.AsyncReadWriteAdapter> ByRef)
#0 4.420    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1[[System.Int32, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[System.Net.Security.SslStream+<ReadAsyncInternal>d__159`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ReadAsyncInternal>d__159`1<System.Net.Security.AsyncReadWriteAdapter> ByRef)
#0 4.420    at System.Net.Security.SslStream.ReadAsyncInternal[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](System.Memory`1<Byte>, System.Threading.CancellationToken)
#0 4.420    at System.Net.Http.HttpConnection.PrepareForReuse(Boolean)
#0 4.420    at System.Net.Http.HttpConnectionPool.TryGetPooledHttp11Connection(System.Net.Http.HttpRequestMessage, Boolean, System.Net.Http.HttpConnection ByRef, HttpConnectionWaiter`1<System.Net.Http.HttpConnection> ByRef)
#0 4.420    at System.Net.Http.HttpConnectionPool+<SendWithVersionDetectionAndRetryAsync>d__84.MoveNext()
#0 4.420    at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.Net.Http.HttpConnectionPool+<SendWithVersionDetectionAndRetryAsync>d__84, System.Net.Http, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<SendWithVersionDetectionAndRetryAsync>d__84 ByRef)
#0 4.420    at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[System.Net.Http.HttpConnectionPool+<SendWithVersionDetectionAndRetryAsync>d__84, System.Net.Http, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<SendWithVersionDetectionAndRetryAsync>d__84 ByRef)
#0 4.420    at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(System.Net.Http.HttpRequestMessage, Boolean, Boolean, System.Threading.CancellationToken)
#0 4.420    at System.Net.Http.HttpConnectionPool.SendAsync(System.Net.Http.HttpRequestMessage, Boolean, Boolean, System.Threading.CancellationToken)
#0 4.420    at System.Net.Http.HttpConnectionPoolManager.SendAsyncCore(System.Net.Http.HttpRequestMessage, System.Uri, Boolean, Boolean, Boolean, System.Threading.CancellationToken)
#0 4.420    at System.Net.Http.HttpConnectionPoolManager.SendAsync(System.Net.Http.HttpRequestMessage, Boolean, Boolean, System.Threading.CancellationToken)
#0 4.420    at System.Net.Http.HttpMessageHandlerStage.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken)
#0 4.421    at System.Net.Http.DiagnosticsHandler.SendAsync(System.Net.Http.HttpRequestMessage, Boolean, System.Threading.CancellationToken)
#0 4.421    at System.Net.Http.HttpMessageHandlerStage.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken)
#0 4.421    at System.Net.Http.SocketsHttpHandler.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken)
#0 4.421    at Microsoft.ApplicationInsights.Channel.RedirectHttpHandler+<SendAsync>d__4.MoveNext()
#0 4.421    at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.ApplicationInsights.Channel.RedirectHttpHandler+<SendAsync>d__4, Microsoft.ApplicationInsights, Version=2.20.0.103, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<SendAsync>d__4 ByRef)
#0 4.421    at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Microsoft.ApplicationInsights.Channel.RedirectHttpHandler+<SendAsync>d__4, Microsoft.ApplicationInsights, Version=2.20.0.103, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<SendAsync>d__4 ByRef)
#0 4.421    at Microsoft.ApplicationInsights.Channel.RedirectHttpHandler.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken)
#0 4.421    at System.Net.Http.HttpClient+<<SendAsync>g__Core|83_0>d.MoveNext()
#0 4.421    at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.Net.Http.HttpClient+<<SendAsync>g__Core|83_0>d, System.Net.Http, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<<SendAsync>g__Core|83_0>d ByRef)
#0 4.421    at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[System.Net.Http.HttpClient+<<SendAsync>g__Core|83_0>d, System.Net.Http, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<<SendAsync>g__Core|83_0>d ByRef)
#0 4.421    at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(System.Net.Http.HttpRequestMessage, System.Net.Http.HttpCompletionOption, System.Threading.CancellationTokenSource, Boolean, System.Threading.CancellationTokenSource, System.Threading.CancellationToken)
#0 4.421    at System.Net.Http.HttpClient.SendAsync(System.Net.Http.HttpRequestMessage, System.Net.Http.HttpCompletionOption, System.Threading.CancellationToken)
#0 4.421    at Microsoft.ApplicationInsights.Channel.Transmission+<SendAsync>d__53.MoveNext()
#0 4.421    at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.ApplicationInsights.Channel.Transmission+<SendAsync>d__53, Microsoft.ApplicationInsights, Version=2.20.0.103, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<SendAsync>d__53 ByRef)
#0 4.421    at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Microsoft.ApplicationInsights.Channel.Transmission+<SendAsync>d__53, Microsoft.ApplicationInsights, Version=2.20.0.103, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<SendAsync>d__53 ByRef)
#0 4.421    at Microsoft.ApplicationInsights.Channel.Transmission.SendAsync()
#0 4.421    at Microsoft.DotNet.Cli.Telemetry.PersistenceChannel.Sender.Send(Microsoft.DotNet.Cli.Telemetry.PersistenceChannel.StorageTransmission, System.TimeSpan ByRef)
#0 4.421    at Microsoft.DotNet.Cli.Telemetry.PersistenceChannel.Sender.SendLoop()
#0 4.421    at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
#0 4.421    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
#0 4.425 qemu: uncaught target signal 6 (Aborted) - core dumped
#0 4.717 Aborted
------
DEBU[0000] serving grpc connection                      
DEBU[0000] stopping session                              span="load buildkit capabilities"
DEBU[0000] serving grpc connection                      
DEBU[0005] stopping session                             
failed to solve: executor failed running [/bin/sh -c dotnet new list]: exit code: 134
  1. Different stacktrace for same dotnet new list, I got this from inside container. Below messages suggest it's sdk image problem, probably Docker is not pulling in correct platform image. I did try linux/arm64/v8 platform with docker-composer but it found, not sure if it by design or not. There are many issues regarding qemu: uncaught target signal 11 (Segmentation fault), they suggest it's either qemu or docker image problem: https://stackoverflow.com/a/68916382, qemu: uncaught target signal 11 (Segmentation fault) - core dumped when running docker-compose up on Apple Silicon docker/for-mac#5123
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested.
 
>dotnet new list
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault
  1. Here is stack trace for dotnet new console,
Stack trace details
Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at Newtonsoft.Json.JsonWriter.AutoComplete(Newtonsoft.Json.JsonToken)
   at Newtonsoft.Json.JsonTextWriter.WriteValue(Int64)
   at Newtonsoft.Json.Linq.JValue.WriteTo(Newtonsoft.Json.JsonWriter, Newtonsoft.Json.JsonConverter[])
   at Newtonsoft.Json.Linq.JToken.ToString(Newtonsoft.Json.Formatting, Newtonsoft.Json.JsonConverter[])
   at Microsoft.TemplateEngine.JExtensions.ToJTokenStringDictionary(Newtonsoft.Json.Linq.JToken, System.StringComparer, System.String)
   at Microsoft.TemplateEngine.Orchestrator.RunnableProjects.ConfigModel.GeneratedSymbol..ctor(System.String, Newtonsoft.Json.Linq.JObject)
   at Microsoft.TemplateEngine.Orchestrator.RunnableProjects.ConfigModel.SymbolModelConverter.GetModelForObject(System.String, Newtonsoft.Json.Linq.JObject, Microsoft.Extensions.Logging.ILogger, System.String)
   at Microsoft.TemplateEngine.Orchestrator.RunnableProjects.ConfigModel.TemplateConfigModel..ctor(Newtonsoft.Json.Linq.JObject, Microsoft.Extensions.Logging.ILogger, Microsoft.TemplateEngine.Orchestrator.RunnableProjects.ISimpleConfigModifiers, System.String)
   at Microsoft.TemplateEngine.Orchestrator.RunnableProjects.RunnableProjectConfig..ctor(Microsoft.TemplateEngine.Abstractions.IEngineEnvironmentSettings, Microsoft.TemplateEngine.Abstractions.IGenerator, Microsoft.TemplateEngine.Abstractions.Mount.IFile, Microsoft.TemplateEngine.Abstractions.Mount.IFile, Microsoft.TemplateEngine.Abstractions.Mount.IFile, System.String)
   at Microsoft.TemplateEngine.Orchestrator.RunnableProjects.RunnableProjectGenerator.Microsoft.TemplateEngine.Abstractions.IGenerator.GetTemplatesAndLangpacksFromDir(Microsoft.TemplateEngine.Abstractions.Mount.IMountPoint, System.Collections.Generic.IList`1<Microsoft.TemplateEngine.Abstractions.ILocalizationLocator> ByRef)
   at Microsoft.TemplateEngine.Edge.Settings.Scanner.ScanMountPointForTemplatesAndLangpacks(MountPointScanSource)
   at Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager+<>c__DisplayClass22_0.<UpdateTemplateCacheAsync>b__2(Int32)
   at System.Threading.Tasks.Parallel+<>c__DisplayClass19_0`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<ForWorker>b__1(System.Threading.Tasks.RangeWorker ByRef, Int32, Boolean ByRef)
   at System.Threading.Tasks.TaskReplicator+Replica.Execute()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
   at System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(System.Threading.Tasks.Task, Boolean)
   at System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean)
   at System.Threading.Tasks.Task.InternalRunSynchronously(System.Threading.Tasks.TaskScheduler, Boolean)
   at System.Threading.Tasks.TaskReplicator.Run[[System.Threading.Tasks.RangeWorker, System.Threading.Tasks.Parallel, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](ReplicatableUserAction`1<System.Threading.Tasks.RangeWorker>, System.Threading.Tasks.ParallelOptions, Boolean)
   at System.Threading.Tasks.Parallel.ForWorker[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](Int32, Int32, System.Threading.Tasks.ParallelOptions, System.Action`1<Int32>, System.Action`2<Int32,System.Threading.Tasks.ParallelLoopState>, System.Func`4<Int32,System.Threading.Tasks.ParallelLoopState,System.__Canon,System.__Canon>, System.Func`1<System.__Canon>, System.Action`1<System.__Canon>)
   at System.Threading.Tasks.Parallel.For(Int32, Int32, System.Action`1<Int32>)
   at Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager+<UpdateTemplateCacheAsync>d__22.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager+<UpdateTemplateCacheAsync>d__22, Microsoft.TemplateEngine.Edge, Version=7.0.102.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].ExecutionContextCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager+<UpdateTemplateCacheAsync>d__22, Microsoft.TemplateEngine.Edge, Version=7.0.102.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].MoveNext(System.Threading.Thread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager+<UpdateTemplateCacheAsync>d__22, Microsoft.TemplateEngine.Edge, Version=7.0.102.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.__Canon)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.__Canon>, System.__Canon)
   at Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager+<GetTemplatePackagesAsync>d__13.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager+<GetTemplatePackagesAsync>d__13, Microsoft.TemplateEngine.Edge, Version=7.0.102.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].ExecutionContextCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager+<GetTemplatePackagesAsync>d__13, Microsoft.TemplateEngine.Edge, Version=7.0.102.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].MoveNext(System.Threading.Thread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager+<GetTemplatePackagesAsync>d__13, Microsoft.TemplateEngine.Edge, Version=7.0.102.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.__Canon)
   at System.Threading.Tasks.UnwrapPromise`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetFromTask(System.Threading.Tasks.Task, Boolean)
   at System.Threading.Tasks.UnwrapPromise`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].ProcessInnerTask(System.Threading.Tasks.Task)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Threading.Tasks.Task.FinishSlow(Boolean)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()
qemu: uncaught target signal 6 (Aborted) - core dumped
Aborted
  1. For sake of completeness here is successful docker build log with sdk6 on same M1 MAC machine, it's before regression experience.
Succesful run details
DEBU[0000] using default config store "/Users/tester/.docker/buildx" 
[+] Building 32.2s (17/18)                                                                                            
 => [internal] load build definition from Dockerfile                                                             0.0s
 => => transferring dockerfile: 838B                                                                             0.0s
 => [internal] load .dockerignore                                                                                0.0s
 => => transferring context: 2B                                                                                  0.0s
 => [internal] load metadata for mcr.microsoft.com/dotnet/sdk:6.0                                               32.1s
 => CACHED [build 1/9] FROM mcr.microsoft.com/dotnet/sdk:6.0@sha256:408f1593d25f04e5d1c97115da40760ca032505c97d  0.0s
 => [internal] load build context                                                                                0.0s
 => => transferring context: 5.38kB                                                                              0.0s
 => [build 2/9] RUN dotnet --info                                                                                1.1s
 => [build 3/9] RUN dotnet --version                                                                             1.1s
 => [build 4/9] WORKDIR /src                                                                                     0.0s 
 => [build 5/9] COPY [ericktest/ericktest.csproj, ericktest/]                                                    0.0s 
 => [build 6/9] COPY [ericktest/Program.cs, ericktest/]                                                          0.0s 
 => [build 7/9] COPY . .                                                                                         0.0s 
 => [build 8/9] WORKDIR /src/ericktest                                                                           0.0s
 => [build 9/9] RUN dotnet build "ericktest.csproj" -c Release -o /app/build                                    20.0s
 => [publish 1/1] RUN dotnet publish "ericktest.csproj" -c Release -o /app/publish /p:UseAppHost=false           9.8s
 => CACHED [base 2/2] WORKDIR /app                                                                               0.0s 
 => CACHED [final 1/2] WORKDIR /app                                                                              0.0s 
 => CACHED [final 2/2] COPY --from=publish /app/publish .                                                        0.0s 
 => exporting to image                                                                                           0.0s 
 => => exporting layers                                                                                          0.0s
 => => writing image sha256:aaab5afe6491bc002ca867aef6859a214e2fb83979399576e9b482ad5076a2cd                     0.0s
 => => naming to docker.io/library/ericktest                                                                     0.0s
DEBU[0000] serving grpc connection                      
DEBU[0000] stopping session                              span="load buildkit capabilities"
DEBU[0000] serving grpc connection                      
DEBU[0032] stopping session     

@aortiz-msft
I believe this is not NuGet problem because same problem repro with dotnet new list command inside docker container.

@rainersigwald
This one could be connected to #12227, please let me know if you need more details from me.

Update 1/26/2023:
Here is my docker repro erickTest.zip
It looks like an issue in the runtime: System.Net.Security.AsyncReadWriteAdapter

@rainersigwald
Copy link

@erdembayar I think this can be closed since it's pretty well tracked by #12227, dotnet/runtime#13648, and https://gitlab.com/qemu-project/qemu/-/issues/249.

@erdembayar
Copy link
Contributor

Based on @rainersigwald comment we can close it, it's tracked by other issues.

@ghost ghost removed the WaitingForCustomer Applied when a NuGet triage person needs more info from the OP label Feb 7, 2023
@erdembayar erdembayar added the Resolution:External This issue appears to be External to nuget label Feb 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Functionality:Restore Platform:Docker All NuGet on docker scenarios Platform:Xplat Resolution:External This issue appears to be External to nuget Type:Bug
Projects
None yet
Development

No branches or pull requests