Closed
Description
Description:
When the Application Pool of an IIS hosted website stops for whatever reason (in my case i stopped it manually), IIS returns a 503 - Service unavailable status code and adds a Connection: close response header.
When retrying such a request with the retry policies (the below is with WaitAndRetryAsync
), the first 2 retries are returning the same status (503) but the third retry is causing the application to stop responding (most probably a deadlock happens).
Steps to reproduce
- Copy the below code into a .Net framework 4.7.2 console application
- Import the following Nuget packages
<PackageReference Include="Microsoft.Extensions.DependencyInjection">
<Version>2.1.1</Version>
</PackageReference>
<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions">
<Version>2.1.1</Version>
</PackageReference>
<PackageReference Include="Microsoft.Extensions.Http">
<Version>2.1.1</Version>
</PackageReference>
<PackageReference Include="Microsoft.Extensions.Http.Polly">
<Version>2.1.1</Version>
</PackageReference>
<PackageReference Include="Polly">
<Version>7.0.3</Version>
</PackageReference>
- Replace URLPlaceholder with a URL pointing to an IIS hosted API hosted on the network (not on localhost)
- After receiving a couple of 200 OK requests, manually Stop the Application pool of the IIS hosted API
- After receiving a couple of 503 Service Unavailable responses, manually Start the Application pool of the IIS hosted API
Expected behavior
The application should start receiving 200 OK responses.
Actual behaviour
The application hangs after the first 2 retries of the first 503 Service unavailable response.
Note that if the WaitAndRetryAsync
policy addition is commented out, the application behaves as expected (without the retries of course).
Code to reproduce the problem
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Http;
using Polly;
using Polly.Timeout;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Net.Sockets;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
namespace ConsoleApp2
{
class Program
{
async static Task Main(string[] args)
{
IServiceCollection services = new ServiceCollection();
services.AddHttpClient("test")
// comment out the below policy for a correct behavior
.AddTransientHttpErrorPolicy(builder => builder.WaitAndRetryAsync(new[]{
TimeSpan.FromSeconds(1),
TimeSpan.FromSeconds(2),
TimeSpan.FromSeconds(4),
}));
// removing the logging handlers as a work around for https://github.com/aspnet/Extensions/issues/563
ServiceDescriptor loggingHandler = services.FirstOrDefault(e => e.ServiceType == typeof(IHttpMessageHandlerBuilderFilter));
if (loggingHandler != null)
{
services.Remove(loggingHandler);
}
IHttpClientFactory factory = services.BuildServiceProvider().GetService<IHttpClientFactory>();
HttpClient client = factory.CreateClient("test");
while (true)
{
HttpResponseMessage response = null;
try
{
response = await client.GetAsync("URLplaceholder").ConfigureAwait(false);
}
catch (Exception e)
{
// logging
}
Thread.Sleep(5000);
}
}
}
}
Observations
- With only one retry for the request, the behavior is as expected.
- The problem does not seem to be in the Polly library, as the same request executed through a
ExecutyPolicyAsync
(with the.AddTransientHttpErrorPolicy
commented out) is working as expected:
response = await Policy
.HandleResult<HttpResponseMessage>(r => r.StatusCode == HttpStatusCode.ServiceUnavailable)
.WaitAndRetryAsync(
new[]{
TimeSpan.FromSeconds(1),
TimeSpan.FromSeconds(2),
TimeSpan.FromSeconds(4),
}).ExecuteAsync(() => client.GetAsync(url)).ConfigureAwait(false);