[BUG] Azure Blob Storage Client SDK No Longer Supports Globalization Invariant Mode for Account Key Authentication

[wsugarman]

Library name and version

Azure.Storage.Blobs 12.21.0

Describe the bug

After attempting to update Azure.Storage.Blobs to the latest version 12.21.0 from 12.20.0, I noticed that my PR pipeline was failing. Upon inspection, I see that the application fails trying to load the type StorageSharedKeyPipelinePolicy:

System.InvalidOperationException: Failed to compare two elements in the array.  ---> System.TypeInitializationException: The type initializer for 'Azure.Storage.StorageSharedKeyPipelinePolicy' threw an exception.  ---> System.Globalization.CultureNotFoundException: Only the invariant culture is supported in globalization-invariant mode. See https://aka.ms/GlobalizationInvariantMode for more information. (Parameter 'name') en-US is an invalid culture identifier.
    at System.Globalization.CultureInfo..ctor(String name, Boolean useUserOverride)
    at Azure.Storage.StorageSharedKeyPipelinePolicy..cctor()    --- End of inner exception stack trace ---
    at System.Collections.Generic.ArraySortHelper`1.IntroSort(Span`1 keys, Int32 depthLimit, Comparison`1 comparer)
    at System.Collections.Generic.ArraySortHelper`1.Sort(Span`1 keys, Comparison`1 comparer)    --- End of inner exception stack trace ---
    at System.Collections.Generic.ArraySortHelper`1.Sort(Span`1 keys, Comparison`1 comparer)
    at Azure.Storage.StorageSharedKeyPipelinePolicy.BuildCanonicalizedHeaders(StringBuilder stringBuilder, HttpMessage message)
    at Azure.Storage.StorageSharedKeyPipelinePolicy.BuildStringToSign(HttpMessage message)
    at Azure.Storage.StorageSharedKeyPipelinePolicy.OnSendingRequest(HttpMessage message)
    at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
    at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
    at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.InnerProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
    at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
    at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
    at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
    at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.InnerProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
    at Azure.Storage.Blobs.BlobRestClient.DownloadAsync(String snapshot, String versionId, Nullable`1 timeout, String range, String leaseId, Nullable`1 rangeGetContentMD5, Nullable`1 rangeGetContentCRC64, String encryptionKey, String encryptionKeySha256, Nullable`1 encryptionAlgorithm, Nullable`1 ifModifiedSince, Nullable`1 ifUnmodifiedSince, String ifMatch, String ifNoneMatch, String ifTags, CancellationToken cancellationToken)
    at Azure.Storage.Blobs.Specialized.BlobBaseClient.StartDownloadAsync(HttpRange range, BlobRequestConditions conditions, DownloadTransferValidationOptions validationOptions, Int64 startOffset, Boolean async, CancellationToken cancellationToken)
    at Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadStreamingInternal(HttpRange range, BlobRequestConditions conditions, DownloadTransferValidationOptions transferValidationOverride, IProgress`1 progressHandler, String operationName, Boolean async, CancellationToken cancellationToken)
    at Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadStreamingDirect(HttpRange range, BlobRequestConditions conditions, DownloadTransferValidationOptions transferValidationOverride, IProgress`1 progressHandler, String operationName, Boolean async, CancellationToken cancellationToken)
    at Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadContentInternal(BlobRequestConditions conditions, IProgress`1 progressHandler, HttpRange range, DownloadTransferValidationOptions transferValidationOverride, Boolean async, CancellationToken cancellationToken)
    at Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadContentAsync(BlobRequestConditions conditions, CancellationToken cancellationToken)
    at Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadContentAsync(CancellationToken cancellationToken)
    at Keda.Scaler.DurableTask.AzureStorage.TaskHub.AzureStorageTaskHubClient.GetMonitorAsync(AzureStorageAccountInfo accountInfo, String taskHub, CancellationToken cancellationToken) in /scaler/src/TaskHub/AzureStorageTaskHubClient.cs:line 82
    at Keda.Scaler.DurableTask.AzureStorage.Web.DurableTaskAzureStorageScalerService.GetMetrics(GetMetricsRequest request, ServerCallContext context) in /scaler/src/Web/DurableTaskAzureStorageScalerService.cs:line 90
    at Grpc.Shared.Server.UnaryServerMethodInvoker`3.ResolvedInterceptorInvoker(TRequest resolvedRequest, ServerCallContext resolvedContext)
    at Grpc.Shared.Server.UnaryServerMethodInvoker`3.ResolvedInterceptorInvoker(TRequest resolvedRequest, ServerCallContext resolvedContext)
    at Keda.Scaler.DurableTask.AzureStorage.Interceptors.ExceptionInterceptor.UnaryServerHandler[TRequest,TResponse](TRequest request, ServerCallContext context, UnaryServerMethod`2 continuation)

This regression at first glance appears to be related to this PR: https://github.com/Azure/azure-sdk-for-net/pull/43131/files#diff-2ca4e8d05dd448866d56400a1db672172908ed1d538cb52e4d177690ba9df139

I run my app within a dockerfile seen here. It uses the Mariner2.0 aspnet base image: mcr.microsoft.com/dotnet/aspnet:8.0.7-cbl-mariner2.0-distroless-amd64@sha256:b5e3a7388a4f1ac750c502e9ce559a8c45e8ffa86162f95373aac57bc8c7b70d. I suspect this package update will be a problem for anyone running on Mariner2.0 images by default as I can see it includes the environment variable "DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=true".

Expected behavior

The type StorageSharedKeyPipelinePolicy should continue to load successfully when Globalization Invariant Mode is enabled. This bug will affect applications that use AccountKey (like Azurite) for authentication.

Actual behavior

The type StorageSharedKeyPipelinePolicy fails to load when connecting to Azurite when Globalization Invariant Mode is enabled.

Reproduction Steps

Running the snippet below with Azurite running should result in an InvalidOperationException. Be sure to enable "Globalization Invariant Mode." For example, the csproj can include the build property: <InvariantGlobalization>true</InvariantGlobalization>

using Azure.Storage.Blobs;

BlobServiceClient client = new("UseDevelopmentStorage=true");
BlobContainerClient container = client.GetBlobContainerClient("example");
await container.CreateIfNotExistsAsync();

Environment

No response

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

[aligneddev]

I ran into this as well.
For me it worked when running from Visual Studio in Windows, but after I deployed to k8s via our FROM mcr.microsoft.com/dotnet/aspnet:8.0-alpine AS base dockerfile I get a CrashLoopBackoff

 fail: Microsoft.Extensions.Hosting.Internal.Host[11]
      Hosting failed to start
      System.AggregateException: One or more errors occurred. (Failed to compare two elements in the array.)
       ---> System.InvalidOperationException: Failed to compare two elements in the array.
       ---> System.TypeInitializationException: The type initializer for 'Azure.Storage.StorageSharedKeyPipelinePolicy' threw an exception.
       ---> System.Globalization.CultureNotFoundException: Only the invariant culture is supported in globalization-invariant mode. See https://aka.ms/GlobalizationInvariantMode for more information. (Parameter 'name')
      en-US is an invalid culture identifier.
         at System.Globalization.CultureInfo..ctor(String name, Boolean useUserOverride)
         at Azure.Storage.StorageSharedKeyPipelinePolicy..cctor()
         --- End of inner exception stack trace ---
         at Azure.Storage.StorageSharedKeyPipelinePolicy.<>c.<BuildCanonicalizedHeaders>b__6_0(HttpHeader x, HttpHeader y)
         at System.Collections.Generic.ArraySortHelper`1.InsertionSort(Span`1 keys, Comparison`1 comparer)
         at System.Collections.Generic.ArraySortHelper`1.IntroSort(Span`1 keys, Int32 depthLimit, Comparison`1 comparer)
         at System.Collections.Generic.ArraySortHelper`1.IntrospectiveSort(Span`1 keys, Comparison`1 comparer)
         at System.Collections.Generic.ArraySortHelper`1.Sort(Span`1 keys, Comparison`1 comparer)
         --- End of inner exception stack trace ---
         at System.Collections.Generic.ArraySortHelper`1.Sort(Span`1 keys, Comparison`1 comparer)
         at System.Collections.Generic.List`1.Sort(Comparison`1 comparison)
         at Azure.Storage.StorageSharedKeyPipelinePolicy.BuildCanonicalizedHeaders(StringBuilder stringBuilder, HttpMessage message)
         at Azure.Storage.StorageSharedKeyPipelinePolicy.BuildStringToSign(HttpMessage message)
         at Azure.Storage.StorageSharedKeyPipelinePolicy.OnSendingRequest(HttpMessage message)
         at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
         at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
         at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.InnerProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
         at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
         at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
         at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
         at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.InnerProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
         at Azure.Core.HttpPipelineExtensions.ProcessMessageAsync(HttpPipeline pipeline, HttpMessage message, RequestContext requestContext, CancellationToken cancellationToken)
         at Azure.Storage.Blobs.BlobRestClient.GetPropertiesAsync(String snapshot, String versionId, Nullable`1 timeout, String leaseId, String encryptionKey, String encryptionKeySha256, String encryptionAlgorithm, String ifTags, RequestConditions requestConditions, RequestContext context)
         at Azure.Storage.Blobs.Specialized.BlobBaseClient.GetPropertiesInternal(BlobRequestConditions conditions, Boolean async, RequestContext context, String operationName)
         at Azure.Storage.Blobs.Specialized.BlobBaseClient.GetPropertiesAsync(BlobRequestConditions conditions, CancellationToken cancellationToken)
         at Azure.Messaging.EventHubs.Primitives.BlobCheckpointStoreInternal.GetCheckpointAsync(String fullyQualifiedNamespace, String eventHubName, String consumerGroup, String partitionId, CancellationToken cancellationToken)
         at Azure.Messaging.EventHubs.EventProcessorClient.GetCheckpointAsync(String partitionId, CancellationToken cancellationToken)
         at Azure.Messaging.EventHubs.Primitives.EventProcessor`1.ValidateStorageConnectionAsync(CancellationToken cancellationToken)
         at Azure.Messaging.EventHubs.Primitives.EventProcessor`1.ValidateProcessingPreconditions(CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at Azure.Messaging.EventHubs.Primitives.EventProcessor`1.StartProcessingInternalAsync(Boolean async, CancellationToken cancellationToken)
         at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(Task task)
         at Azure.Messaging.EventHubs.Primitives.EventProcessor`1.StartProcessing(CancellationToken cancellationToken)
         at Azure.Messaging.EventHubs.EventProcessorClient.<>n__2(CancellationToken cancellationToken)
         at Azure.Messaging.EventHubs.EventProcessorClient.StartProcessingInternalAsync(Boolean async, CancellationToken cancellationToken)
         at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(Task task)
         at Azure.Messaging.EventHubs.EventProcessorClient.StartProcessing(CancellationToken cancellationToken)
         at Ours.Api.AppConfigurationEventHubService.InitEventHubProcessing(EventHubConfigurationSettings configuration) in /src/Ours.Api/AppConfigurationEventHubService.cs:line 65
         at Ours.Api.AppConfigurationEventHubService..ctor(IOptions`1 appSettings, IConfigurationRefresherProvider configurationRefresherProvider, ILogger`1 logger) in /src/Ours.Api/AppConfigurationEventHubService.cs:line 27
         at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
         at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
         at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
         at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
         at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
         at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
         at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
         at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateServiceAccessor(ServiceIdentifier serviceIdentifier)
         at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
         at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(ServiceIdentifier serviceIdentifier, ServiceProviderEngineScope serviceProviderEngineScope)
         at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
         at Ours.Api.Startup.Configure(IApplicationBuilder app, IWebHostEnvironment env, IApiVersionDescriptionProvider provider) in /src/Ours.Api/Startup.cs:line 225
         at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
         at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
         at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
         at Microsoft.AspNetCore.Hosting.ConfigureBuilder.Invoke(Object instance, IApplicationBuilder builder)
         at Microsoft.Extensions.DependencyInjection.AutoRegisterMiddleware.<>c__DisplayClass4_0.<Configure>b__0(IApplicationBuilder app)
         at Microsoft.ApplicationInsights.AspNetCore.ApplicationInsightsStartupFilter.<>c__DisplayClass2_0.<Configure>b__0(IApplicationBuilder app)
         at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
         at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__15_1(IHostedService service, CancellationToken token)
         at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)
Unhandled exception. System.AggregateException: One or more errors occurred. (Failed to compare two elements in the array.)
 ---> System.InvalidOperationException: Failed to compare two elements in the array.
 ---> System.TypeInitializationException: The type initializer for 'Azure.Storage.StorageSharedKeyPipelinePolicy' threw an exception.
 ---> System.Globalization.CultureNotFoundException: Only the invariant culture is supported in globalization-invariant mode. See https://aka.ms/GlobalizationInvariantMode for more information. (Parameter 'name')
en-US is an invalid culture identifier.
   at System.Globalization.CultureInfo..ctor(String name, Boolean useUserOverride)
   at Azure.Storage.StorageSharedKeyPipelinePolicy..cctor()

InitEventHubProcessing truncated is

        var storageClient = new BlobContainerClient(configuration.StorageClientConnectionString(),
            configuration.StorageContainerName);

        _eventProcessorClient = new EventProcessorClient(storageClient, configuration.ConsumerGroup, configuration.ConnectionString(),
            configuration.ServiceBusName);

        _eventProcessorClient.ProcessEventAsync += ProcessEventHandler;
        _eventProcessorClient.ProcessErrorAsync += ProcessErrorHandler;
        _eventProcessorClient.StartProcessing();
    }

[goncalo-oliveira]

Yeah... just ran into this issue as well. Locally it runs fine because it's not running in invariant mode, but running on a Kubernetes cluster it throws this same exception. The error can be reproduced locally by setting the environment variable DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1.

[UladzK]

I also got the same issue after upgrading Azure.Storage.Blobs.Batch to version 12.18.0. the package implicitly uses Azure.Storages.Blobs >=12.21.0. This issue was reproduced when running app in the k8s cluster.

I was also able to reproduce the issue locally using Azurit after applying <InvariantGlobalization>true</InvariantGlobalization> attribute to my csproj.

[OnurGumus]

I also hit the same problem with Azure.Storage.Queues.QueueClient.ReceiveMessagesAsync Here's my stacktrace

System.InvalidOperationException: Failed to compare two elements in the array.
 ---> System.TypeInitializationException: The type initializer for 'Azure.Storage.StorageSharedKeyPipelinePolicy' threw an exception.
 ---> System.Globalization.CultureNotFoundException: Only the invariant culture is supported in globalization-invariant mode. See https://aka.ms/GlobalizationInvariantMode for more information. (Parameter 'name')
en-US is an invalid culture identifier.
   at System.Globalization.CultureInfo..ctor(String name, Boolean useUserOverride)
   at Azure.Storage.StorageSharedKeyPipelinePolicy..cctor()
   --- End of inner exception stack trace ---
   at Azure.Storage.StorageSharedKeyPipelinePolicy.<>c.<BuildCanonicalizedHeaders>b__6_0(HttpHeader x, HttpHeader y)
   at System.Collections.Generic.ArraySortHelper`1.InsertionSort(Span`1 keys, Comparison`1 comparer)
   at System.Collections.Generic.ArraySortHelper`1.IntroSort(Span`1 keys, Int32 depthLimit, Comparison`1 comparer)
   at System.Collections.Generic.ArraySortHelper`1.IntrospectiveSort(Span`1 keys, Comparison`1 comparer)
   at System.Collections.Generic.ArraySortHelper`1.Sort(Span`1 keys, Comparison`1 comparer)
   --- End of inner exception stack trace ---
   at System.Collections.Generic.ArraySortHelper`1.Sort(Span`1 keys, Comparison`1 comparer)
   at System.Collections.Generic.List`1.Sort(Comparison`1 comparison)
   at Azure.Storage.StorageSharedKeyPipelinePolicy.BuildCanonicalizedHeaders(StringBuilder stringBuilder, HttpMessage message)
   at Azure.Storage.StorageSharedKeyPipelinePolicy.BuildStringToSign(HttpMessage message)
   at Azure.Storage.StorageSharedKeyPipelinePolicy.OnSendingRequest(HttpMessage message)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.InnerProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.InnerProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Storage.Queues.MessagesRestClient.DequeueAsync(Nullable`1 numberOfMessages, Nullable`1 visibilitytimeout, Nullable`1 timeout, CancellationToken cancellationToken)
   at Azure.Storage.Queues.QueueClient.ReceiveMessagesInternal(Nullable`1 maxMessages, Nullable`1 visibilityTimeout, String operationName, Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.Queues.QueueClient.ReceiveMessagesAsync(Nullable`1 maxMessages, Nullable`1 visibilityTimeout, CancellationToken cancellationToken)
   at Azure.Storage.Queues.QueueClient.ReceiveMessagesAsync(CancellationToken cancellationToken)

[amnguye]

Thanks for bringing this to our attention.

We're currently working on a fix to resolve this issue as soon as possible. Will update the issue once we have a fix in and when the next release will come out.

[amnguye]

Hi we released a patch release. Please upgrade accordingly

https://www.nuget.org/packages/Azure.Storage.Common/12.20.1
https://www.nuget.org/packages/Azure.Storage.Blobs/12.21.1
https://www.nuget.org/packages/Azure.Storage.Files.DataLake/12.19.1
https://www.nuget.org/packages/Azure.Storage.Queues/12.19.1
https://www.nuget.org/packages/Azure.Storage.Files.Shares/12.19.1

[wsugarman]

Thank you very much!!