CosmosClient causes Blazor WASM to deadlock on net7

[y2kbugger]

Describe the bug
When newing up a CosmosClient (either as DI services or in async callback on a page) the entire browser tab will deadlock.

This does not occur when I retarget the app to .net6

Seems similar to #1429 and I have tried various combinations of injecting a separate HttpClient via () => new HttpClient() and LimitToEndpoint = true

To Reproduce
Steps to reproduce the behavior. If you can include code snippets or links to repositories containing a repro of the issue that can helps us in detecting the scenario it would speed up the resolution.

Console.WriteLine("Creating client");
var c = new CosmosClient(
    CosmosEndpoint, CosmosKey,
    new CosmosClientOptions()
        {
            ApplicationName = "Test",
            ConnectionMode = ConnectionMode.Gateway,
        }
    );
Console.WriteLine("Created client");
Console.WriteLine(c.ToString());

Expected behavior
No hang

Actual behavior
Browser tab hangs completely, can't even right-click/inspect. If I open the developer console before calling the code, e.g. via a button, no error messages are found in console.

Also it is strange to me that the callback continues after the Client is Created but we are just deadlocked.

Creating client
Created client
Microsoft.Azure.Cosmos.CosmosClient

Environment summary
SDK Version: .net7
OS Version (e.g. Windows, Linux, MacOSX): windows

Additional context

In his blog Jermery Likness states

https://blog.jeremylikness.com/blog/azure-ad-secured-serverless-cosmosdb-from-blazor-webassembly/

Now all of the foundational pieces are in place. To retrieve the data from the Blazor WebAssembly app, I created a special client. It is designed to return a DbContext, but you can easily modify it to return a CosmosClient instance instead if you wish to use the SDK directly instead of via EF Core.

Which is what I'm trying to do.

This is my full csproj file

<Project Sdk="Microsoft.NET.Sdk.BlazorWebAssembly">
  <PropertyGroup>
    <RootNamespace>MyApp</RootNamespace>
    <TargetFramework>net7.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <LangVersion>11</LangVersion>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="ExcelDataReader" Version="3.6.0" />
    <PackageReference Include="ExcelDataReader.DataSet" Version="3.6.0" />
    <PackageReference Include="Microsoft.AspNetCore.Components.WebAssembly" Version="7.0.0" />
    <PackageReference Include="Microsoft.AspNetCore.Components.WebAssembly.DevServer" Version="7.0.0" PrivateAssets="all" />
    <PackageReference Include="Radzen.Blazor" Version="4.4.1" />
    <PackageReference Include="System.Text.Encoding.CodePages" Version="7.0.0" />
    <PackageReference Include="Microsoft.Azure.Cosmos" Version="3.31.2" />
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\SharedModels\SharedModels.csproj" />
  </ItemGroup>

</Project>

[y2kbugger]

related? dotnet/aspnetcore#45233

[ealsur]

@y2kbugger If the process is deadlocked, then a profiler should be able to show the place where the deadlock is happening.

The fact that this only happens on NET7 and it works fine on NET6 means there is some change in the implementation that is causing it, because the SDK is not changing in-between and it targets NETStandard2.0, there is no NET6 or NET7 specific code that it uses, it relies entirely on the implementation of the NETStandard2.0 interface.

If you can find where the deadlock is happening, we can help, but based on the output of the console in your test, it is not the client creation (otherwise it would not print "Created client").

[y2kbugger]

I've made an issue with the aspnetcore team as well as a minimal reproduction repository.

https://github.com/y2kbugger/ReproduceDeadlockIssueBlazorWasmNet7

Run the project with

dotnet run

Goto the counter page and put in the key

TxzSgDHnH3yz8WsQ0bdoElUPgrPz98Bk0OCGTMG70fEz77dAeCLEVjKmqW71JIgEVvqrN0be4RtQACDbiZyfLQ==

and click the button.

It's a read-only key for an empty throwaway DB, but they are really serious about a fully working bug reproduction or else they will close issue.

[y2kbugger]

Sorry didn't mean to close....

[y2kbugger]

I noticed they have some workarounds for previous deadlock issues that deal occur then there is no debugger attached. https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos/src/CosmosClient.cs#L144
Maybe something changed between net6 and net7 either in runtime or aspnetcore that broke the workaround?

[ealsur]

That line is not relevant. What that line does is remove the Default Trace Listener if it's attached. The Default Trace Listener is a Trace Listener that generates lock contention under heavy load and it's inefficient. The Default Trace Listener would not generate a lock like the one you are describing.

Default Trace Listener is mostly enabled in NET FX + IIS environments.

When it is enabled, and the machine is under heavy load, if the SDK is generating Traces, it will generate high CPU consumption due to the lock it internally has, that is why it is advised to remove it.

[y2kbugger]

Any idea how to profile/track down a deadlock that occurs within a WASM environment?

[ealsur]

Honestly no, I would reach out to the WASM team for that, I am no expert on profiling WASM. Maybe asking on the other issue you are tracking.

[wrharper-AASP]

Personally, I wouldn't recommend using CosmosClient in a Blazor environment. Instead, create a function app REST API that the Blazor environment uses. This helps keep things locked down. I know this is an alternative to the issue you are reporting, but it could be helpful.

[danglingneuron]

.Net 7: It creates client and then goes to sleep on await FeedIterator.ReadNextAsync()
.Net 6: throws exception that System.Security.Cryptography is not supported on Blazor WASM

[ghost]

y2kbugger this issue requires more information for the team to be able to help. In case this information is available, please add it and re-open the Issue.

[y2kbugger]

Personally, I wouldn't recommend using CosmosClient in a Blazor environment. Instead, create a function app REST API that the Blazor environment uses. This helps keep things locked down. I know this is an alternative to the issue you are reporting, but it could be helpful.

Microsoft doesn't agree. Accessing directly from the browser is part of their use case as shown by the CORS documentation:

Once you set the CORS rules for your Azure Cosmos account, a properly authenticated request made to the service from a browser client will be evaluated to determine whether it is allowed according to the rules you have specified.
Also they have created an entire JS client for this.

[wrharper-AASP]

Websites can be reverse engineered. That is why I use a REST layer. Also, you would have to expose a key vault for user-based interactions. Nothing can be exposed this way.

[y2kbugger]

Websites can be reverse engineered. That is why I use a REST layer. Also, you would have to expose a key vault for user-based interactions. Nothing can be exposed this way.

Obviously, but not everyone is building internet facing apps for mass adoption. Some people are replacing vba macros built by interns and which are used by 3 people for a specific task and doesn't contain any trade secrets.

My point is that people have different use cases and "I don't do it that way for my needs" isn't a helpful answer to broken software.