Skip to content

Microsoft.Extensions.Http.Polly: Deadlock when retrying a "503 - Service unavailable" #28384

Closed
@marcpaulv

Description

@marcpaulv

Description:

When the Application Pool of an IIS hosted website stops for whatever reason (in my case i stopped it manually), IIS returns a 503 - Service unavailable status code and adds a Connection: close response header.

When retrying such a request with the retry policies (the below is with WaitAndRetryAsync), the first 2 retries are returning the same status (503) but the third retry is causing the application to stop responding (most probably a deadlock happens).

Steps to reproduce

  1. Copy the below code into a .Net framework 4.7.2 console application
  2. Import the following Nuget packages
  <PackageReference Include="Microsoft.Extensions.DependencyInjection">
      <Version>2.1.1</Version>
    </PackageReference>
    <PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions">
      <Version>2.1.1</Version>
    </PackageReference>
    <PackageReference Include="Microsoft.Extensions.Http">
      <Version>2.1.1</Version>
    </PackageReference>
    <PackageReference Include="Microsoft.Extensions.Http.Polly">
      <Version>2.1.1</Version>
    </PackageReference>
    <PackageReference Include="Polly">
      <Version>7.0.3</Version>
    </PackageReference>
  1. Replace URLPlaceholder with a URL pointing to an IIS hosted API hosted on the network (not on localhost)
  2. After receiving a couple of 200 OK requests, manually Stop the Application pool of the IIS hosted API
  3. After receiving a couple of 503 Service Unavailable responses, manually Start the Application pool of the IIS hosted API

Expected behavior

The application should start receiving 200 OK responses.

Actual behaviour

The application hangs after the first 2 retries of the first 503 Service unavailable response.

Note that if the WaitAndRetryAsync policy addition is commented out, the application behaves as expected (without the retries of course).

Code to reproduce the problem

using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Http;
using Polly;
using Polly.Timeout;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Net.Sockets;
using System.Text;
using System.Threading;
using System.Threading.Tasks;

namespace ConsoleApp2
{
    class Program
    {
        async static Task Main(string[] args)
        {
            IServiceCollection services = new ServiceCollection();
            services.AddHttpClient("test")
              // comment out the below policy for a correct behavior
               .AddTransientHttpErrorPolicy(builder => builder.WaitAndRetryAsync(new[]{
               TimeSpan.FromSeconds(1),
               TimeSpan.FromSeconds(2),
               TimeSpan.FromSeconds(4),
               }));
  
              // removing the logging handlers as a work around for https://github.com/aspnet/Extensions/issues/563
            ServiceDescriptor loggingHandler = services.FirstOrDefault(e => e.ServiceType == typeof(IHttpMessageHandlerBuilderFilter));
            if (loggingHandler != null)
            {
                services.Remove(loggingHandler);
            }

            IHttpClientFactory factory = services.BuildServiceProvider().GetService<IHttpClientFactory>();

            HttpClient client = factory.CreateClient("test");

            while (true)
            {
                HttpResponseMessage response = null;
                try
                {
                    response = await client.GetAsync("URLplaceholder").ConfigureAwait(false);
                }
                catch (Exception e)
                {
                    // logging
                }

                Thread.Sleep(5000);
            }
        }
    }
}

Observations

  1. With only one retry for the request, the behavior is as expected.
  2. The problem does not seem to be in the Polly library, as the same request executed through a ExecutyPolicyAsync (with the .AddTransientHttpErrorPolicy commented out) is working as expected:
response = await Policy
                .HandleResult<HttpResponseMessage>(r => r.StatusCode == HttpStatusCode.ServiceUnavailable)
                .WaitAndRetryAsync(
                new[]{
                 TimeSpan.FromSeconds(1),
                 TimeSpan.FromSeconds(2),
                 TimeSpan.FromSeconds(4),
                 }).ExecuteAsync(() => client.GetAsync(url)).ConfigureAwait(false);

Metadata

Metadata

Assignees

Labels

area-networkingIncludes servers, yarp, json patch, bedrock, websockets, http client factory, and http abstractionsfeature-httpclientfactoryIncludes: HttpClientFactory (some bugs also in Extensions repo)investigate

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions