Crash on WindowsCryptographicException on net60 GA #61417

theolivenbaum · 2021-11-10T14:09:56Z

Description

System.AggregateException: One or more errors occurred. (Unknown error (0xc1000008))
---> Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: Unknown error (0xc1000008)
at Internal.Cryptography.HashProviderCng.AppendHashData(ReadOnlySpan`1 source)
at Internal.Cryptography.HashProvider.AppendHashData(Byte[] data, Int32 offset, Int32 count)
at System.Security.Cryptography.SHA256.Implementation.HashCore(Byte[] array, Int32 ibStart, Int32 cbSize)
at System.Security.Cryptography.HashAlgorithm.ComputeHash(Byte[] buffer)
at Host.CLI.Program.<>c__DisplayClass9_2.<b__3>d.MoveNext() in C:\work\curiosity\curiosity-ai-website\tools\host\Program.cs:line 280
--- End of stack trace from previous location ---
at Host.CLI.Program.BuildAsync(String path, String buildPath, String templatePath, String translationPath) in C:\work\curiosity\curiosity-ai-website\tools\host\Program.cs:line 296
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait()
at Host.CLI.Delayed.g__CheckBacklog|5_1() in C:\work\curiosity\curiosity-ai-website\tools\host\Delayed.cs:line 71
at Host.CLI.Delayed.<>c.b__5_0(Object _) in C:\work\curiosity\curiosity-ai-website\tools\host\Delayed.cs:line 49
at System.Threading.TimerQueueTimer.<>c.<.cctor>b__27_0(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.TimerQueueTimer.CallCallback(Boolean isThreadPool)
at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
at System.Threading.TimerQueue.FireNextTimers()
at System.Threading.TimerQueue.AppDomainTimerCallback(Int32 id)

Reproduction Steps

The app in question is a simple web host, accessing the page via the edge browser.

Expected behavior

No crash

Actual behavior

Process crashes hard with an uncaught exception

Regression?

Yes. The issue never happen till I installed the net60 GA SDK installed this morning.

Known Workarounds

No response

Configuration

Window x64

dotnet --list-sdks

2.1.526 [C:\Program Files\dotnet\sdk]
2.1.617 [C:\Program Files\dotnet\sdk]
2.1.701 [C:\Program Files\dotnet\sdk]
2.1.818 [C:\Program Files\dotnet\sdk]
2.2.101 [C:\Program Files\dotnet\sdk]
2.2.301 [C:\Program Files\dotnet\sdk]
3.0.100-preview8-013656 [C:\Program Files\dotnet\sdk]
3.0.100 [C:\Program Files\dotnet\sdk]
3.1.200 [C:\Program Files\dotnet\sdk]
3.1.301 [C:\Program Files\dotnet\sdk]
3.1.415 [C:\Program Files\dotnet\sdk]
5.0.104 [C:\Program Files\dotnet\sdk]
5.0.209 [C:\Program Files\dotnet\sdk]
5.0.303 [C:\Program Files\dotnet\sdk]
5.0.403 [C:\Program Files\dotnet\sdk]
6.0.100-rc.1.21458.32 [C:\Program Files\dotnet\sdk]
6.0.100 [C:\Program Files\dotnet\sdk]  <---- App is running on this now

Other information

No response

The text was updated successfully, but these errors were encountered:

ghost · 2021-11-10T14:10:03Z

Tagging subscribers to this area: @bartonjs, @vcsjones, @krwq, @GrabYourPitchforks
See info in area-owners.md if you want to be subscribed.

Issue Details

Description

System.AggregateException: One or more errors occurred. (Unknown error (0xc1000008))
---> Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: Unknown error (0xc1000008)
at Internal.Cryptography.HashProviderCng.AppendHashData(ReadOnlySpan`1 source)
at Internal.Cryptography.HashProvider.AppendHashData(Byte[] data, Int32 offset, Int32 count)
at System.Security.Cryptography.SHA256.Implementation.HashCore(Byte[] array, Int32 ibStart, Int32 cbSize)
at System.Security.Cryptography.HashAlgorithm.ComputeHash(Byte[] buffer)
at Host.CLI.Program.<>c__DisplayClass9_2.<b__3>d.MoveNext() in C:\work\curiosity\curiosity-ai-website\tools\host\Program.cs:line 280
--- End of stack trace from previous location ---
at Host.CLI.Program.BuildAsync(String path, String buildPath, String templatePath, String translationPath) in C:\work\curiosity\curiosity-ai-website\tools\host\Program.cs:line 296
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait()
at Host.CLI.Delayed.g__CheckBacklog|5_1() in C:\work\curiosity\curiosity-ai-website\tools\host\Delayed.cs:line 71
at Host.CLI.Delayed.<>c.b__5_0(Object _) in C:\work\curiosity\curiosity-ai-website\tools\host\Delayed.cs:line 49
at System.Threading.TimerQueueTimer.<>c.<.cctor>b__27_0(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.TimerQueueTimer.CallCallback(Boolean isThreadPool)
at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
at System.Threading.TimerQueue.FireNextTimers()
at System.Threading.TimerQueue.AppDomainTimerCallback(Int32 id)

Reproduction Steps

The app in question is a simple web host, accessing the page via the edge browser.

Expected behavior

No crash

Actual behavior

Process crashes hard with an uncaught exception

Regression?

Yes. The issue never happen till I installed the net60 GA SDK installed this morning.

Known Workarounds

No response

Configuration

Window x64

dotnet --list-sdks

2.1.526 [C:\Program Files\dotnet\sdk]
2.1.617 [C:\Program Files\dotnet\sdk]
2.1.701 [C:\Program Files\dotnet\sdk]
2.1.818 [C:\Program Files\dotnet\sdk]
2.2.101 [C:\Program Files\dotnet\sdk]
2.2.301 [C:\Program Files\dotnet\sdk]
3.0.100-preview8-013656 [C:\Program Files\dotnet\sdk]
3.0.100 [C:\Program Files\dotnet\sdk]
3.1.200 [C:\Program Files\dotnet\sdk]
3.1.301 [C:\Program Files\dotnet\sdk]
3.1.415 [C:\Program Files\dotnet\sdk]
5.0.104 [C:\Program Files\dotnet\sdk]
5.0.209 [C:\Program Files\dotnet\sdk]
5.0.303 [C:\Program Files\dotnet\sdk]
5.0.403 [C:\Program Files\dotnet\sdk]
6.0.100-rc.1.21458.32 [C:\Program Files\dotnet\sdk]
6.0.100 [C:\Program Files\dotnet\sdk]  <---- App is running on this now

Other information

No response

Author:	theolivenbaum
Assignees:	-
Labels:	`area-System.Security`, `untriaged`
Milestone:	-

theolivenbaum · 2021-11-10T14:12:33Z

If it helps, here is a crash dump

vcsjones · 2021-11-10T16:25:34Z

A few questions that might help tracking down the issue:

Does this happen repeatedly, or was it a one-off crash? What about on multiple machines?
Is it specific to your application, or does attempting to hash something in a brand-new application cause an exception as well? e.g. can you run this in a console application?
```
using System;
using System.Security.Cryptography;

using SHA256 sha = SHA256.Create();
Console.WriteLine(Convert.ToHexString(sha.ComputeHash(new byte[32])));
```
If it crashes in your application, but not the sample above, is it possible for you reduce your application code to sample code that does reproduce the issue?
What version of Windows is this? (you can run winver to get the exact version).

jaylagorio · 2021-11-26T22:00:20Z

@vcsjones It looks like I'm running into the same thing, also after an update to net60 after the app worked fine in net50. It's intermittent and unpredictable, and in my case it's in a 3rd-party library but with a similar upper portion of the stack trace:

at Internal.Cryptography.HashProviderCng.AppendHashData(ReadOnlySpan`1 source)
   at Internal.Cryptography.HashProvider.AppendHashData(Byte[] data, Int32 offset, Int32 count)
   at System.Security.Cryptography.HashAlgorithm.ComputeHash(Byte[] buffer)
   at Kraken.Net.KrakenAuthenticationProvider.AddAuthenticationToHeaders(String uri, HttpMethod method, Dictionary`2 parameters, Boolean signed, HttpMethodParameterPosition parameterPosition, ArrayParametersSerialization arraySerialization)

Windows 10 v21H2 (OS Build 19044.1348)

vcsjones · 2021-11-26T22:36:00Z

A few guesses, since I don't have a lot to go on.

Assuming it is this project: https://github.com/JKorf/Kraken.Net/
And it's throwing at this line: https://github.com/JKorf/Kraken.Net/blob/0802f64af669bd77209218517063303750aadc79/Kraken.Net/KrakenAuthenticationProvider.cs#L59

I suspect it is might be because the HMACSHA512 encryptor field is being accessed from multiple threads. Instances of HMACSHA512 are not thread safe.

I can reproduce this exception by using an HMACSHA512 instance between threads:

using System;
using System.Threading;
using System.Security.Cryptography;

using HMACSHA512 hmac = new HMACSHA512();
Thread thread1 = new Thread(static obj => {
    while (true)
    {
        ((HMACSHA512)obj).ComputeHash(new byte[1]);
    }
});
Thread thread2 = new Thread(static obj => {
    while (true)
    {
        ((HMACSHA512)obj).ComputeHash(new byte[1]);
    }
});

thread1.Start(hmac);
thread2.Start(hmac);
thread1.Join();
thread2.Join();

That will quickly give me the same issue as described in the original post.

At this point I can only suggest that you confirm with the library author that no instance of a hash or HMAC algorithm is being used by two or more threads, concurrently.

If that is indeed the case, I would recommend creating a new instance of the hash / HMAC algorithm per-thread, or just always as-needed. Alternatively, using the HMACXYZ.HashData one-shot static methods that were introduced in .NET 5 for digests, and .NET 6 for HMACs.

theolivenbaum · 2021-11-26T23:00:26Z

Hi @vcsjones thanks for the suggestion, I'll investigate if this is the case on my side. It should have been a single thread accessing it, but I'll double check it

ghost · 2021-12-02T22:30:55Z

This issue has been marked needs more info since it may be missing important information. Please refer to our contribution guidelines for tips on how to report issues effectively.

alanmcgovern · 2021-12-04T11:03:12Z

@vcsjones I have a repro case for a similar, if not identical, issue:

This breaks:

var buffer = new byte[1024];
var hasher = SHA1.Create ();
hasher.TransformBlock (buffer, 0, buffer.Length, buffer, 0);
//hasher.TransformFinalBlock (Array.Empty<byte> (), 0, 0);
hasher.Initialize ();
hasher.TransformBlock (buffer, 0, buffer.Length, buffer, 0);
hasher.TransformFinalBlock (Array.Empty<byte> (), 0, 0);
GC.KeepAlive (hasher.Hash);
hasher.Initialize ();
hasher.TransformBlock (buffer, 0, buffer.Length, buffer, 0);

If you want things to break in .NET6 just leave the 4th line commented. If you want it to not break, uncomment line 4.

If you re-initialize a hasher after calling TransformBlock and before calling TransformFinalBlock, you corrupt the hasher and it fails.

EDIT: Thanks to @EvgeniyZ-ru for bringing this issue to my attention, and working with me to provide the information I needed to distill a testcase :)

alanmcgovern · 2021-12-04T11:45:15Z

I took a few minutes and reduced it a bit further:

var buffer = new byte[1024];
var hasher = SHA1.Create ();
hasher.TransformBlock (buffer, 0, buffer.Length, buffer, 0);
hasher.Initialize ();
hasher.TransformFinalBlock (Array.Empty<byte> (), 0, 0);
hasher.Initialize ();
hasher.TransformFinalBlock (Array.Empty<byte> (), 0, 0);

If you call Initialize on a hash algorithm, after you've invoked TransformBlock but before you've invoked TransformFinalBlock, the instance corrupts and mysteriously fails to transform blocks later on. Works around dotnet/runtime#61417 Addresses #483

vcsjones · 2021-12-04T15:48:54Z

Thanks for the test case, I can reproduce it now. I will look at this now.

vcsjones · 2021-12-04T16:17:17Z

I have a PR open to address the issue uncovered by @alanmcgovern's test case.

This won't fix @jaylagorio's reported instance - sharing an instance between threads will still lead to the same issue of the CNG object getting in to an incorrect state.

This particular issue is new in .NET 6 because prior to .NET 6, the Initialize was a no-op. See #51172 for additional background.

@theolivenbaum can you confirm if this is the issue you were running in to as well? Are you using Initialize anywhere in your code?

vcsjones · 2021-12-04T17:46:13Z

I just noticed on @theolivenbaum's stack System.AggregateException.

This to me implies a possibility that the hash object is being used concurrently from different Tasks.

It's not definitive evidence, but a possibility.

jaylagorio · 2021-12-06T00:21:41Z

@vcsjones Thanks so much for looking into this and thanks even more for submitting a PR to the library I'm using! Fingers crossed for a new release soon.

theolivenbaum · 2021-12-07T09:33:16Z

Not sure if related @vcsjones, but I'm also using SHA256.ComputeHash(...), and it throws randomly this:

SafeHandle cannot be null. (Parameter 'pHandle')
   at System.ThrowHelper.ThrowArgumentNullException(ExceptionArgument argument, ExceptionResource resource)
   at System.StubHelpers.StubHelpers.SafeHandleAddRef(SafeHandle pHandle, Boolean& success)
   at Interop.BCrypt.BCryptHashData(SafeBCryptHashHandle hHash, Byte& pbInput, Int32 cbInput, Int32 dwFlags)
   at Internal.Cryptography.HashProviderCng.AppendHashData(ReadOnlySpan`1 source)
   at Internal.Cryptography.HashProvider.AppendHashData(Byte[] data, Int32 offset, Int32 count)
   at System.Security.Cryptography.SHA256.Implementation.HashCore(Byte[] array, Int32 ibStart, Int32 cbSize)
   at System.Security.Cryptography.HashAlgorithm.ComputeHash(Byte[] buffer)
   at Host.CLI.Program.<>c__DisplayClass9_2.<<BuildAsync>b__3>d.MoveNext() in C:\tldo\website\tools\host\Program.cs:line 280
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Host.CLI.Program.<BuildAsync>d__9.MoveNext() in .\host\Program.cs:line 296

theolivenbaum · 2021-12-07T09:36:59Z

The code is approximately this:

var files = new []{ "path to file", ... }; //list of files
await Task.WhenAll(files.Select(async f => await DoSomethingWithFile(f, Sha256.ComputeHash(await File.ReadAllBytesAsync(f)))));

theolivenbaum · 2021-12-07T09:40:28Z

It also throws this on subsequent calls:

Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: Unknown error (0xc1000008)
   at Internal.Cryptography.HashProviderCng.AppendHashData(ReadOnlySpan`1 source)
   at Internal.Cryptography.HashProvider.AppendHashData(Byte[] data, Int32 offset, Int32 count)
   at System.Security.Cryptography.HashAlgorithm.ComputeHash(Byte[] buffer)

vcsjones · 2021-12-07T12:58:59Z

The code is approximately this:

var files = new []{ "path to file", ... }; //list of files
await Task.WhenAll(files.Select(async f => await DoSomethingWithFile(f, Sha256.ComputeHash(await File.ReadAllBytesAsync(f)))));

It looks like you are using an instance of SHA256 concurrently. You have a property, field, or local instance named Sha256 that is concurrently (Task.WhenAll) computing the file hashes.

If you are targeting net5.0 or higher, you can change it to this:

var files = new []{ "path to file", ... }; //list of files
await Task.WhenAll(files.Select(async f => await DoSomethingWithFile(f, SHA256.HashData(await File.ReadAllBytesAsync(f)))));

HashData was introduced in .NET 5. It is a static method and it is thread safe.

theolivenbaum · 2021-12-07T13:27:17Z

True, I thought I was already using the static method 🤦‍♂️

vcsjones · 2021-12-15T19:18:10Z

To round things out here:

The issue reported by @theolivenbaum and @jaylagorio was due to misusing an instance between threads. @theolivenbaum did switching to the static one-shot alleviate your exceptions?
The issue reported by @alanmcgovern was fixed, and has been backported to .NET 6. I believe the fix will be in the 6.0.2 release.

theolivenbaum · 2021-12-15T19:34:07Z

Hi @vcsjones, I've not observed the issue anymore! Nice to hear it uncovered another issue nevertheless! Thanks!

dotnet-issue-labeler bot added area-System.Security untriaged New issue has not been triaged by the area owner labels Nov 10, 2021

jaylagorio mentioned this issue Nov 27, 2021

Multithreading Support JKorf/Kraken.Net#56

Closed

bartonjs added needs more info and removed untriaged New issue has not been triaged by the area owner labels Dec 2, 2021

EvgeniyZ-ru-zz mentioned this issue Dec 3, 2021

Error when hashing files on .NET6 alanmcgovern/monotorrent#483

Closed

vcsjones self-assigned this Dec 4, 2021

vcsjones mentioned this issue Dec 4, 2021

Fix re-initialization of hash objects on CNG. #62397

Merged

vcsjones removed the needs more info label Dec 4, 2021

vcsjones mentioned this issue Dec 5, 2021

[release/6.0] Fix re-initialization of hash objects on CNG. #62403

Merged

bartonjs closed this as completed Jan 7, 2022

benmccallum mentioned this issue Jan 31, 2022

Possible thread safety issue with RequestValidator.cs causing System.Security.Cryptography.CryptographicException twilio/twilio-csharp#466

Closed

ghost locked as resolved and limited conversation to collaborators Feb 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crash on WindowsCryptographicException on net60 GA #61417

Crash on WindowsCryptographicException on net60 GA #61417

theolivenbaum commented Nov 10, 2021

ghost commented Nov 10, 2021

Description

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

theolivenbaum commented Nov 10, 2021

vcsjones commented Nov 10, 2021 •

edited

Loading

jaylagorio commented Nov 26, 2021

vcsjones commented Nov 26, 2021

theolivenbaum commented Nov 26, 2021

ghost commented Dec 2, 2021

alanmcgovern commented Dec 4, 2021 •

edited

Loading

alanmcgovern commented Dec 4, 2021 •

edited

Loading

vcsjones commented Dec 4, 2021

vcsjones commented Dec 4, 2021 •

edited

Loading

vcsjones commented Dec 4, 2021 •

edited

Loading

jaylagorio commented Dec 6, 2021

theolivenbaum commented Dec 7, 2021

theolivenbaum commented Dec 7, 2021

theolivenbaum commented Dec 7, 2021

vcsjones commented Dec 7, 2021 •

edited

Loading

theolivenbaum commented Dec 7, 2021

vcsjones commented Dec 15, 2021

theolivenbaum commented Dec 15, 2021

Crash on WindowsCryptographicException on net60 GA #61417

Crash on WindowsCryptographicException on net60 GA #61417

Comments

theolivenbaum commented Nov 10, 2021

Description

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

ghost commented Nov 10, 2021

Description

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

theolivenbaum commented Nov 10, 2021

vcsjones commented Nov 10, 2021 • edited Loading

jaylagorio commented Nov 26, 2021

vcsjones commented Nov 26, 2021

theolivenbaum commented Nov 26, 2021

ghost commented Dec 2, 2021

alanmcgovern commented Dec 4, 2021 • edited Loading

alanmcgovern commented Dec 4, 2021 • edited Loading

vcsjones commented Dec 4, 2021

vcsjones commented Dec 4, 2021 • edited Loading

vcsjones commented Dec 4, 2021 • edited Loading

jaylagorio commented Dec 6, 2021

theolivenbaum commented Dec 7, 2021

theolivenbaum commented Dec 7, 2021

theolivenbaum commented Dec 7, 2021

vcsjones commented Dec 7, 2021 • edited Loading

theolivenbaum commented Dec 7, 2021

vcsjones commented Dec 15, 2021

theolivenbaum commented Dec 15, 2021

vcsjones commented Nov 10, 2021 •

edited

Loading

alanmcgovern commented Dec 4, 2021 •

edited

Loading

alanmcgovern commented Dec 4, 2021 •

edited

Loading

vcsjones commented Dec 4, 2021 •

edited

Loading

vcsjones commented Dec 4, 2021 •

edited

Loading

vcsjones commented Dec 7, 2021 •

edited

Loading