-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
.NET8 Crash on use HttpClientHandler.Credentials #97966
Comments
was tested on x86 and Arm MacOS, everywhere it crash when using .net 8 |
Please take a look, thank you! |
I can reproduce this against net8.0, but not Native stack:
The managed stack
|
Tagging subscribers to this area: @dotnet/ncl, @bartonjs, @vcsjones Issue DetailsDescriptionOur company uses Microsoft Exchange email, to communicate with it we have a separate service that is authenticated using HttpClientHandler.Credentials. After updating .NET to version 8, I received application crashes on any request. After finding out the reasons, I was able to get to the minimal environment that repeats this error. The application almost always crashes. And it seems this only happens on MacOS runtime. Error:
Reproduction StepsI have prepared a repository that allows you to reproduce the error https://github.com/dumkin/net8-credentials-crash Also you may run code
Expected behaviorHttpClient auth and worked correctly Actual behaviorApplication crashed Regression?Yes, it worked on .NET 7 Known WorkaroundsNo Configuration
Other informationNo response
|
/cc @filipnavara in case something jumps out at you. |
On .NET 8 it's possible to opt-in to the managed NTLM/SPNEGO implementation by adding this property to the .csproj file: |
did the repro work for you @filipnavara? I could not reproduce it so far on my (Intel) MacBook. I simply get 401 back. |
And perhaps if you do have repro @vcsjones, could you try it with 6.0 or 7.0? I'm wondering if this is 8.0 regression or of the issue always existed. |
@wfurt |
yes, I did run it several times @dumkin. I can let it sit in loop .... and I can possibly also get hands on arm Mac. |
I have similar results as @dumkin.
Since it may be sensitive to macOS environment:
This is on an M1.
It seems that way. |
@vcsjones Thanks for the summary. I will investigate tomorrow morning. Update: I can reproduce this. |
This is starting to ring some bells. macOS implementation of runtime/src/libraries/System.Net.Security/src/System/Net/NegotiateAuthenticationPal.Unix.cs Lines 624 to 627 in 512bcaf
FWIW Apple updated the code last September and added more of the early frees here - apple-oss-distributions/Heimdal@07a3113#diff-c60d25db88cb547db736ca8f6a23712a4ce2bf2be392eb265878d5b7b2f020f7 - which could explain why it's only happening on some versions of macOS. |
We have a new Apple bug here. Yay! In the call to https://github.com/apple-oss-distributions/Heimdal/blob/48f86d0ceef220f75b16f0fc8266b53d50129c38/lib/gssapi/digest/init_sec_context.c#L324 When the authentication inevitably fails, the
and When we later call Submitted to Apple as FB13600619 |
Finally, to answer why it doesn't crash on .NET 6/7... It actually does - if you run it in a loop and force handles to be finalized with for (;;)
{
try { await client.GetStringAsync(""); }
catch (System.Net.Http.HttpRequestException) { GC.Collect(); }
} We were just leaking the handles and depending on finalization instead of releasing them deterministically. |
To follow this ticket |
I'm having the same problem when using the |
Unless I am mistaken, it seems we cannot do anything about this bug from .NET and it needs to be fixed in the GSSAPI codebase. @filipnavara am I right?, Also, is there a link on which the status of the issue you filed can be tracked? |
Unfortunately Apple Feedback is private. I will share any update as soon as I receive it but there has been no response from Apple so far. I don’t think a workaround is possible on .NET side aside from the aforementioned UseManagedNtlm switch which bypasses the whole Apple SPNEGO implementation. |
I checked the code (https://github.com/apple-oss-distributions/Heimdal/blob/48f86d0ceef220f75b16f0fc8266b53d50129c38/lib/gssapi/spnego/compat.c#L247-L351; callers and callees) and I didn't find any public API to do so. |
Since there is no easy workaround and the crash is pretty nasty, I'm wondering if we should port the any thoughts on this @karelz ??? |
This currently causes issues on macOS and iOS (#99892). Please make UseManagedNtlm = True the default in 8.0 LTS. It took us ages to trace this back to Ntlm credentials, most people might not even be able to trace this back since it causes random crashes. |
Folks, if you are impacted by this issue, can you please add upvote on top post? It will be easier to track number of people impacted. Thanks! |
We have seen the crashes on .NET 7 and .NET 8 in our app. The problem is that it's not easily traceable to the root cause. I expect the number of the people to be affected to be high. Just my $.02. |
How big would be the change to introduce the switch ion 8.0 @filipnavara? I assume not that big as we already do it for Android???? We discussed the possibility of servicing with @karelz. Changing the default may be difficult to push through. (but we can try) |
The runtime part of the switch is already there, we just miss the SDK MSBuild part (dotnet/sdk#34903). Possibly we would need to backport some of the Managed NTLM fixes in dotnet/runtime which were usually small and targeted. |
I meant original post at the very top by dumkin |
FWIW I tried to burn one of my paid support requests for code-level support. Today I received a reply that there's no workaround, the feedback is still under investigation, and I got the support request credited back to my account. |
We've reached 20 upvotes! 🎉 |
I didn't receive any update from Apple but the bug seems to be fixed in macOS 15.0 Beta 1 (24A5264n). The small repro code I sent them no longer crashes. |
It also fixed on macOS 14.5 in latest update. |
Wow you're right! macOS 14.5 came out on May 13'th 2024 so it's already been working for a month haha. I guess everybody thought we just had to wait on the fix to be implemented in the .NET 8 sdk. |
thanks @filipnavara for taking it up to Apple. Should we close this as resolved? Perhaps @dumkin can also verify as the original reporter. |
The related problem, where I get the same error during |
what is your stacktrace @Scyarah? DO you have any simplified repro? |
Unfortunately not that easily, because it is reliant on having a NuGet packagesource other than the standard one and using PackageSourceCredentials for this. I currently need this for my workflow, because we are using the artifact storage of a local azure devops instance for our NuGet package. But with a workaround to the set the version with a global.json to 7.x I can currently still use it. Sorry, for not being able to provide more. |
Then I think you should try to reproduce it under debugger and/or get dump. I think it will be difficult to investigate otherwise @Scyarah |
FWIW I have the same issue. Creating a new project via If there's any way I can provide information about it, please point me in the right direction. |
@lbelloq are you still encountering this issue? you should be able to run something like this
and then using |
Because I just encountered the error again I ran
The backtrace
Not sure if this information from lldb is actually helpful. I'm not experienced with debugging binaries. My system information
|
@Scyarah The backtrace seems to be from the same problem as before, the double free via |
I’ll try to do more tests across various macOS versions next week. |
I have the same problem with macOS 14.6 and macOS 14.6.1, working with .net maui and nugets. |
I have updated to the .NET 9 SDK (9.0.100-rc.2.24474.11 ) and now it seems like it is working again. |
.NET 9 uses the managed NTLM and SPNEGO implementation on macOS by default so it would not hit the issue. |
Description
Our company uses Microsoft Exchange email, to communicate with it we have a separate service that is authenticated using HttpClientHandler.Credentials. After updating .NET to version 8, I received application crashes on any request. After finding out the reasons, I was able to get to the minimal environment that repeats this error. The application almost always crashes. And it seems this only happens on MacOS runtime.
Error:
Reproduction Steps
I have prepared a repository that allows you to reproduce the error
https://github.com/dumkin/net8-credentials-crash
Also you may run code
Expected behavior
HttpClient auth and worked correctly
Actual behavior
Application crashed
Regression?
Yes, it worked on .NET 7
Known Workarounds
No
Configuration
Other information
No response
The text was updated successfully, but these errors were encountered: