BackgroundService blocked the execution of whole host

[vova-lantsov-dev]

Describe the bug

When I run the specific foreach cycle in BackgroundService-derived class, it blocks the whole host from starting. Even second hosted service doesn't start. When I comment foreach cycle, everything works as expected.

To Reproduce

TargetFramework: netcoreapp2.1
Version: 2.1.12

Use following hosted service: NotificationRunner.cs
And singleton: NotificationService.cs

Expected behavior

BackgroundService.ExecuteAsync should work in background without blocking even if it has blocking code. As you can see in NotificationService class, cancellation of enumerator is based on IApplicationLifetime.ApplicationStopping, but anyway it shouldn't affect the host startup because BackgroundService is expected to run in background :)

Screenshots

When foreach cycle exists

Then execution is blocked on this cycle

But when foreach cycle is commented

Then execution continues as expected

(But why twice?)

Additional context

.NET Core SDK (reflecting any global.json):
 Version:   3.0.100-preview7-012821
 Commit:    6348f1068a

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.17763
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\3.0.100-preview7-012821\

Host (useful for support):
  Version: 3.0.0-preview7-27912-14
  Commit:  4da6ee6450

.NET Core SDKs installed:
  2.1.700 [C:\Program Files\dotnet\sdk]
  2.1.701 [C:\Program Files\dotnet\sdk]
  2.1.801 [C:\Program Files\dotnet\sdk]
  2.2.300 [C:\Program Files\dotnet\sdk]
  2.2.301 [C:\Program Files\dotnet\sdk]
  2.2.401 [C:\Program Files\dotnet\sdk]
  3.0.100-preview7-012821 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0-preview7.19365.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0-preview7-27912-14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.0.0-preview7-27912-14 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

[vova-lantsov-dev]

Fixed by adding await Task.Yield() at the top of method

[dcarr42]

Yeah the method shouldn't be blocking. Wrapping in a Task.Run will also dispatch.

[rynowak]

Reopening this so we can look at it. I'm glad that you solved your problem but we should try to solve this so others don't hit it.

[rynowak]

@anurse @Tratcher - thoughts?

It looks like if your run a bunch of CPU-bound code in ExecuteAsync then it will stall startup until that work reaches the first actual async point.

See: https://github.com/aspnet/Extensions/blob/master/src/Hosting/Abstractions/src/BackgroundService.cs#L33

In health checks I ended up inserting a Task.Yield to avoid this. We'd get an additional state machine in this case, 1 per service, but it seems like a worthwhile tradeoff.

[vova-lantsov-dev]

Async method executes synchronously till first await is reached (it means BackgroundService.ExecuteAsync won't return till await is called). So this problem is not related to ASP.NET Core, but to async compilation.

[vova-lantsov-dev]

But it would be great if you can solve this case

[Tratcher]

It's a trade off. There may be reasonable initialization code the service needs to run before letting the app continue. It's within the service's control how long it blocks.

[kostya9]

I have also encountered this issue and solved it in the same way as @vova-lantsov-dev did. However, my main concern is that this behavior is not straightforward, would be nice if this was more predictable or documented somewhere

[analogrelay]

I think it's reasonable to dispatch ExecuteAsync to the thread-pool in StartAsync. If the user wants to perform initialization that must complete before letting the app continue, they can override StartAsync itself, right?

public class MyService: BackgroundService
{
	public override async Task StartAsync()
	{
		await InitializeAsync();
		await base.StartAsync();
	}

	public override async Task ExecuteAsync()
	{
		// Do background stuff...
	}
}

It's called BackgroundService. It seems odd that you can block the foreground in the default case. If we're not happy with overriding StartAsync we could add a new InitializeAsync that is called and awaited during StartAsync.

This would be a breaking change however, since existing services may depend on the initial synchronous work blocking the startup process.

[davidfowl]

Sorry I never did this change but enough people have hit it now I think we should just do it. https://github.com/aspnet/Extensions/tree/davidfowl/background-service

[pholly]

I just ran into this and was about to leave feedback on the docs. Glad I checked here first. The Worker template works because it awaits a Task.Delay. Change that to a Thread.Sleep, remove async keyword, and return a CompletedTask from ExecuteAsync and it can be easily reproduced. Execution is not returned to the caller so StartAsync never finishes and the host never finishes initialization so cancellationtoken does not work and any other HostedServices registered would never start.

[pholly]

@davidfowl I just looked at the BackgroundService changes in your branch - would it make more sense for the Host to Task.Run each hosted service instead of depending on the hosted service to do it?

[davidfowl]

I'm not a fan as it's wasteful, but if that many people run into it it might be worth doing. I'd honestly rather teach people that these things aren't running on dedicated threads so blocking them isn't the way to go.

[analogrelay]

Triage summary: The work here is to consider dispatching BackgroundService.ExecuteAsync.

[drmcclelland]

It was very helpful to learn from the updated docs that "No further services are started until ExecuteAsync becomes asynchronous, such as by calling await." docs

Is there a possibility of this being addressed in .NET Core 3.1?

[ghost]

This is how I resolved it in my case [as detailed in 17674 referenced above]:

I started from scratch, choosing to implement my own version of it using IHostedService. In StartAsync, I start a single-fire System.Threading Timer with a period of 10s [10 is a value I came up with after a few runs of the setup to see how much time things took to startup]. The Timer's callback method actually fires the rest of the service (the stuff that is in the ExecuteAsync portion of the BackgroundService.

This works quite well in my setup and actually sped up the app start time [compared to even using the Task.Yield() workaround as mentioned above].

[alexrp]

For what it's worth:

The argument that you can just override StartAsync to accomplish the same thing is basically true but there is a pretty notable drawback: Now, state that you used to be able to initialize and use locally in ExecuteAsync has to be promoted to mutable class state. This gets particularly ugly when NRTs are enabled.

This:

class MyService : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        var state = ...;

        await Task.Yield();

        while (true)
        {
            Work(state);

            await Task.Delay(..., ct);
        }
    }
}

Turns into this:

class MyService : BackgroundService
{
    T? _state;

    public override Task StartAsync(CancellationToken ct)
    {
        _state = ...;

        return base.StartAsync(ct);
    }

    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        while (true)
        {
            Work(_state!);

            await Task.Delay(..., ct);
        }
    }
}

This seems like a significant regression in code clarity to me.

There's also an argument to be made that, if the current behavior of ExecuteAsync is misleading, then so too is the StartAsync override in the example above.

The name BackgroundService is definitely misleading given the current behavior. But I also think that the current behavior is actually useful - I rely on it in two of my projects for sane synchronous initialization ordering. So, I hope that if this change does go through, an alternative class will be provided that retains the current behavior.

[mhintzke]

@davidfowl is your commit here still a true WIP or are you redacting your idea to go this route? I found the documentation here that references this issue a little confusing as it initially made me think that even using await at the top of ExecuteAsync() would block the other host intialization when it actually doesn't (due to the fact that its actually the issuer's NotificationService enumeration that was blocking on his BlockingCollection)

I believe any confusion would be mitigated by doing what you initially proposed in that commit and having the abstraction dispatch our executing Task, although I also see the benefits of teaching the community about how to properly use it as-is as well.

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. Please help me learn by adding exactly one area label.

[HelloKitty]

I noticed that hangs can now occur in later versions of ASP Core if you do a Task.Delay. Previous versions of ASP Core were not causing this issue but I don't have exact version information. Calling await Task.Delay(JobConfig.IntervalMilliseconds, cancellationToken); within public async Task StartAsync(CancellationToken cancellationToken) In ASP Core 3.1 now, unlike a previous version, blocks startup for some reason.

[vova-lantsov-dev]

@HelloKitty IHostedService.StartAsync always blocks the execution of the whole host. This is expected behaviour. Try to use BackgroundService.ExecuteAsync for non-blocking delayed tasks.

[HelloKitty]

@vova-lantsov-dev It appears so, but I do not think this was the case in a previous version of ASP Core. Maybe 2.1. This ran fine without hanging a year ago, anyway I have adopted BackgroundService.ExecuteAsync and now delay in there. Seems to work.

[drewkill32]

This causes an issue when unit testing. If I mock a service to return a CompletedTask instead of an awaited task StartAsync never returns.

 [TestClass]
public class TestClass
{
        [TestMethod, TestCategory("UnitTest")]
        public async Task Background_Service_Should_Return_FromStartAsync()
        {
            
            var sut = new FakeService();

            await sut.StartAsync(Token);

            //code never reaches here because ExecuteAsync never awaits a task
            Assert.IsTrue(true);
        }
}
public class FakeService : BackgroundService
{
        protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            while (!stoppingToken.IsCancellationRequested)
            {
                await Task.CompletedTask;
            }
        }
}

[rjgotten]

While implementing BackgroundService it worked without any Task.Yield(), but today it doesn't. Does anyone know how that is possible?

Look at the implementation of BackgroundService.

StartAsync is a synchronous (!!) method which returns either the Task taken as result from ExecuteAsync, if said task IsCompleted; and otherwise returns a pre-completed Task.CompletedTask. It will take responsbility to stop the execute task in StopAsync by triggering the cancellation token, regardless.

Because StartAsync is synchronous, the code will block until the actual ExecuteAsync implementation, if it is an async method, does its first await. That first await is where the asynchronous state machine compiler magic effectively kicks in and the async method returns its Task in a yet-to-be-completed state.

[MartinDemberger]

Another problem with this behaviour is with exceptions.

When the service throws a exception in the first line the process is stopped even if HostOptions.BackgroundServiceExceptionBehavior is set to BackgroundServiceExceptionBehavior.Ignore.

   public class ServiceWithError : BackgroundService
    {
        //protected override Task ExecuteAsync(CancellationToken stoppingToken) => Task.Run(() => throw new System.NotImplementedException());
        protected override Task ExecuteAsync(CancellationToken stoppingToken) => throw new System.NotImplementedException();
    }

[eerhardt]

This issue hasn't been planned for 7.0 (see #64015). Moving to Future.

[suchoss]

Async method executes synchronously till first await is reached (it means BackgroundService.ExecuteAsync won't return till await is called). So this problem is not related to ASP.NET Core, but to async compilation.

I would like to point out that the "executes synchronously till first await is reached" is not true in my case.

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        using var timer = new PeriodicTimer(TimeSpan.FromHours(24));
        
        // If I do not add following line, the code stops for very long time. More than 1 hour.
        await Task.Yield();  
        
        do
        {
            logger.Debug($"Timer triggered.");
            var count = Interlocked.Increment(ref executionCount);
            logger.Debug($"Trigger counter:{count}.");
            
            // this await wont return control back to main program
            // it is still blocking
            await RunLoop(stoppingToken);
            
        } while (await timer.WaitForNextTickAsync(stoppingToken));
    }

    private async Task RunLoop(CancellationToken stoppingToken)
    {
        foreach (var indexType in Enum.GetValues<AutocompleteIndexType>())
        {
            logger.Debug($"Creating {indexType:G} index.");
            try
            {
                // this await wont return control back to main program
                // it is still blocking - probably because 90 % of following calls are "synchronous" inside
                await IndexFactory.CreateCacheIndex(indexType);
            }
            catch (Exception e)
            {
                logger.Error($"Error occured during {indexType:G} index creation.", e);
            }
        }
        
        logger.Debug("Running delete of old indexes.");
        try
        {
            IndexFactory.DeleteOld();

        }
        catch (Exception e)
        {
            logger.Error($"Problem occured when deleting old files.", e);
        }
    }

But await Task.Yield(); at the beginning works.

[Leandro-Albano]

I just ran into this issue, and in my case it seems that await Task.Yield(); doesn’t help. In my case, it also seem to be a problem just on Debug mode. I have the following:

public class ConsumerBackgroundService : BackgroundService
    {
        protected override async Task ExecuteAsync(CancellationToken cancellationToken)
        {
            await Task.Yield();
            while (!cancellationToken.IsCancellationRequested)
            {
            }
        }

        public override Task StopAsync(CancellationToken cancellationToken) => base.StopAsync(cancellationToken);

    }

[Leandro-Albano]

So, I tried the above code, and apparently the issue only happens on Mac, on both vscode and vs for Mac.

[thomrad]

So, I tried the above code, and apparently the issue only happens on Mac, on both vscode and vs for Mac.

Yes, can confirm this message. Debug build in Rider on Mac blocks, Run in Rider on Mac works.

[montella1507]

We have simple Backround service like this:

`

public class RefresherService : BackgroundService
{


    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        try
        {

            while (!stoppingToken.IsCancellationRequested)
            {
                try
                {

                  await Do();
                }
                catch (Exception ex)
                {
                  
                }
                finally
                {
               
                    await Task.Delay(TimeSpan.FromMinutes(1), stoppingToken);
                }
            }
        }
        catch (Exception ex) when (ex is TaskCanceledException || ex is OperationCanceledException)
        {
       
        }
        catch (Exception ex)
        {
       
        }
    }

}`

And it B L O C K S the startup of the server for 1 minute.... This is not how should have the BackgroundService and ExecuteAsync behave...

I have found article about this strange behavior here:
https://blog.stephencleary.com/2020/05/backgroundservice-gotcha-startup.html

but.. somehow this important issue died?

[Davilink]

Maybe we should have 2 class of background service:
BlockingBackgroundService and NonBlockingBackgroudService
BlockingBackgroundService would use the same current implement of backgroundService and the NonBlockingBackgroudService would spawn internally spawn a Task.Run() just like demonstrated in this commit : dotnet/extensions@8d0b128#diff-a2481caefbc8935408d8a841d3040f716cac00cbd846ef84c9892ce4046c59d9R37

[radwansalah]

I have the same issue, but I noticed that this issue happens only when I run the project using the command line using "dotnet run", but when using the visual studio code or visual studio, it runs successfully, and the background service is working alongside the API, I test the API using postman and it works.
Can anyone tell me how to run it using the command line successfully? For example, If I have a missing option something like that.

[christallire]

I've run into this too.
@eerhardt could we make this happen in 8.0? this could be a relatively simple fix compared to the time spent scratching their heads :)

[Xor-el]

I think this should make it in NET 8.0 as indicated here
#86511
We have gotten two new properties added that should resolve this.

https://learn.microsoft.com/dotnet/api/microsoft.extensions.hosting.hostoptions.servicesstartconcurrently?view=dotnet-plat-ext-8.0

https://learn.microsoft.com/dotnet/api/microsoft.extensions.hosting.hostoptions.servicesstopconcurrently?view=dotnet-plat-ext-8.0

[StephenCleary]

The IHostedLifecycleService and ServicesStartConcurrently changes in .NET 8.0 do not address this issue. The host (as currently written) still starts the services by calling StartAsync directly. So a synchronous implementation of StartAsync will still block host startup.

[lonix1]

Sad that even in v8 this won't be fixed.

For anyone who lands here, see Stephen Cleary's blog series which provides workarounds for these issues.

[naumenkoff]

.NET 8 has been released, and unfortunately, Microsoft has turned a blind eye to this issue.
For those who have encountered it, it would be wise not to use BackgroundService. Instead, you can create your implementation based on IHostedService or the new IHostedLifecycleService.

The situation is amusing - to start a synchronous service, you need to await something, for example, Task.Yield, resulting in the generation of a state machine and a loss of at least 0.05% performance at this stage.
Alternatively, you can use await Task.Run and lose at least 0.10% simply because of the presence of a state machine to start the service elegantly with just one addition to the DI container.

[montella1507]

Yeah.. MS is way more focusing on implementing YET next type of UI library, just to deprecate and kill it 1 year later, rather then solve these issues..

[davidfowl]

I’m very confused why Task.Run isn’t a viable workaround?

[naumenkoff]

I’m very confused why Task.Run isn’t a viable workaround?

You can do that until the presence of a state machine becomes a hindrance to you, and the TaskScheduler works perfectly.

[davidfowl]

You can do that until the presence of a state machine becomes a hindrance to you

When does that happen?

[StephenCleary]

Task.Run is a perfectly viable workaround. But I'd still like to see it baked into BackgroundService because it's not currently a pit of success.

Most developers don't think about async methods as starting synchronously. There's a number of SO questions that are variants of "I started with a background service template and it works fine until I try to remove the Task.Delay" or "it stops working as soon as I try to [synchronously] read from a queue of work". For those of us with a good understanding of async and how the host starts background services, the problem is obvious; but that's not most developers.

[JohannSig]

This sounds like an unnecessary "rite of passage" pitfall for anyone deciding to stroll into BackgroundService country. Myself twice now (2020 + 2024) 😆

[jesslilly]

Nothing here was working for me...

🔴 As it turns out you cannot test this with breakpoints!

Maybe that seems logical to some people, but it did not dawn on me until I spent an hour, tried every single solution and workaround in this issue, and finally said, to myself, "Hmmm, maybe I'll try logging instead of using a breakpoint".

Give me a thumbs up if this was helpful. And give me a pie in the face otherwise! 🤣

[BernhardPollerspoeck]

... Most developers don't think about async methods as starting synchronously. There's a number of SO questions that are variants of "I started with a background service template and it works fine until I try to remove the Task.Delay" or "it stops working as soon as I try to [synchronously] read from a queue of work". For those of us with a good understanding of async and how the host starts background services, the problem is obvious; but that's not most developers.

this sounds more like a edjucation/knownledge or documentation issue then. Just slapping async/await onto everything is not "working with tasks".
Maybe a analyzer would help a lot here to clarify and guide?

[christallire]

Everything should've been built in a foolproof way.

[lonix1]

Hit this issue again. It takes a long time to diagnose the issue and arrive here for various workarounds. If one doesn't have guru-level async knowledge, it's an insidious time waster.

Please please please consider this for v9. 🙏