[API Proposal]: Add API to enable BackgroundService startup to avoid blocking host startup

[vova-lantsov-dev]

Background and motivation

IHostedService.StartAsync executes inline (and sequentially) when IHost.Start/StartAsync is called. With a BackgroundService, this causes lots of confusion since running code that isn't async in ExecuteAsync or StartAsync also runs inline. The workaround today is to call Task.Run on your own code or to use Task.Yield(), but this is not obvious.

API Proposal

public class BackgroundService : IHostedService
{
+    public virtual bool StartAsynchronously { get; } = true;
}

NOTE: StartAsynchronously would be true by default.

API Usage

class MyBackgroundService : BackgroundService
{
    // It's true by default, this is just for illustration.
    public override bool StartAsynchronously => true;

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        using var timer = new PeriodicTimer(TimeSpan.FromSeconds(5));

        while (await timer.WaitForNextTickAsync(stoppingToken))
        {
            Console.WriteLine("Hello from background service!");
        }
    }
}

Alternative Designs

An optional constructor argument.

Original Issue

BackgroundService blocked the execution of whole host

Describe the bug

When I run the specific foreach cycle in BackgroundService-derived class, it blocks the whole host from starting. Even second hosted service doesn't start. When I comment foreach cycle, everything works as expected.

To Reproduce

TargetFramework: netcoreapp2.1
Version: 2.1.12

Use following hosted service: NotificationRunner.cs
And singleton: NotificationService.cs

Expected behavior

BackgroundService.ExecuteAsync should work in background without blocking even if it has blocking code. As you can see in NotificationService class, cancellation of enumerator is based on IApplicationLifetime.ApplicationStopping, but anyway it shouldn't affect the host startup because BackgroundService is expected to run in background :)

Screenshots

When foreach cycle exists

Then execution is blocked on this cycle

But when foreach cycle is commented

Then execution continues as expected

(But why twice?)

Additional context

.NET Core SDK (reflecting any global.json):
 Version:   3.0.100-preview7-012821
 Commit:    6348f1068a

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.17763
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\3.0.100-preview7-012821\

Host (useful for support):
  Version: 3.0.0-preview7-27912-14
  Commit:  4da6ee6450

.NET Core SDKs installed:
  2.1.700 [C:\Program Files\dotnet\sdk]
  2.1.701 [C:\Program Files\dotnet\sdk]
  2.1.801 [C:\Program Files\dotnet\sdk]
  2.2.300 [C:\Program Files\dotnet\sdk]
  2.2.301 [C:\Program Files\dotnet\sdk]
  2.2.401 [C:\Program Files\dotnet\sdk]
  3.0.100-preview7-012821 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0-preview7.19365.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0-preview7-27912-14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.0.0-preview7-27912-14 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

[vova-lantsov-dev]

Fixed by adding await Task.Yield() at the top of method

[dcarr42]

Yeah the method shouldn't be blocking. Wrapping in a Task.Run will also dispatch.

[rynowak]

Reopening this so we can look at it. I'm glad that you solved your problem but we should try to solve this so others don't hit it.

[rynowak]

@anurse @Tratcher - thoughts?

It looks like if your run a bunch of CPU-bound code in ExecuteAsync then it will stall startup until that work reaches the first actual async point.

See: https://github.com/aspnet/Extensions/blob/master/src/Hosting/Abstractions/src/BackgroundService.cs#L33

In health checks I ended up inserting a Task.Yield to avoid this. We'd get an additional state machine in this case, 1 per service, but it seems like a worthwhile tradeoff.

[vova-lantsov-dev]

Async method executes synchronously till first await is reached (it means BackgroundService.ExecuteAsync won't return till await is called). So this problem is not related to ASP.NET Core, but to async compilation.

[vova-lantsov-dev]

But it would be great if you can solve this case

[Tratcher]

It's a trade off. There may be reasonable initialization code the service needs to run before letting the app continue. It's within the service's control how long it blocks.

[kostya9]

I have also encountered this issue and solved it in the same way as @vova-lantsov-dev did. However, my main concern is that this behavior is not straightforward, would be nice if this was more predictable or documented somewhere

[analogrelay]

I think it's reasonable to dispatch ExecuteAsync to the thread-pool in StartAsync. If the user wants to perform initialization that must complete before letting the app continue, they can override StartAsync itself, right?

public class MyService: BackgroundService
{
	public override async Task StartAsync()
	{
		await InitializeAsync();
		await base.StartAsync();
	}

	public override async Task ExecuteAsync()
	{
		// Do background stuff...
	}
}

It's called BackgroundService. It seems odd that you can block the foreground in the default case. If we're not happy with overriding StartAsync we could add a new InitializeAsync that is called and awaited during StartAsync.

This would be a breaking change however, since existing services may depend on the initial synchronous work blocking the startup process.

[davidfowl]

Sorry I never did this change but enough people have hit it now I think we should just do it. https://github.com/aspnet/Extensions/tree/davidfowl/background-service

[pholly]

I just ran into this and was about to leave feedback on the docs. Glad I checked here first. The Worker template works because it awaits a Task.Delay. Change that to a Thread.Sleep, remove async keyword, and return a CompletedTask from ExecuteAsync and it can be easily reproduced. Execution is not returned to the caller so StartAsync never finishes and the host never finishes initialization so cancellationtoken does not work and any other HostedServices registered would never start.

[pholly]

@davidfowl I just looked at the BackgroundService changes in your branch - would it make more sense for the Host to Task.Run each hosted service instead of depending on the hosted service to do it?

[davidfowl]

I'm not a fan as it's wasteful, but if that many people run into it it might be worth doing. I'd honestly rather teach people that these things aren't running on dedicated threads so blocking them isn't the way to go.

[analogrelay]

Triage summary: The work here is to consider dispatching BackgroundService.ExecuteAsync.

[drmcclelland]

It was very helpful to learn from the updated docs that "No further services are started until ExecuteAsync becomes asynchronous, such as by calling await." docs

Is there a possibility of this being addressed in .NET Core 3.1?

[ghost]

This is how I resolved it in my case [as detailed in 17674 referenced above]:

I started from scratch, choosing to implement my own version of it using IHostedService. In StartAsync, I start a single-fire System.Threading Timer with a period of 10s [10 is a value I came up with after a few runs of the setup to see how much time things took to startup]. The Timer's callback method actually fires the rest of the service (the stuff that is in the ExecuteAsync portion of the BackgroundService.

This works quite well in my setup and actually sped up the app start time [compared to even using the Task.Yield() workaround as mentioned above].

[alexrp]

For what it's worth:

The argument that you can just override StartAsync to accomplish the same thing is basically true but there is a pretty notable drawback: Now, state that you used to be able to initialize and use locally in ExecuteAsync has to be promoted to mutable class state. This gets particularly ugly when NRTs are enabled.

This:

class MyService : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        var state = ...;

        await Task.Yield();

        while (true)
        {
            Work(state);

            await Task.Delay(..., ct);
        }
    }
}

Turns into this:

class MyService : BackgroundService
{
    T? _state;

    public override Task StartAsync(CancellationToken ct)
    {
        _state = ...;

        return base.StartAsync(ct);
    }

    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        while (true)
        {
            Work(_state!);

            await Task.Delay(..., ct);
        }
    }
}

This seems like a significant regression in code clarity to me.

There's also an argument to be made that, if the current behavior of ExecuteAsync is misleading, then so too is the StartAsync override in the example above.

The name BackgroundService is definitely misleading given the current behavior. But I also think that the current behavior is actually useful - I rely on it in two of my projects for sane synchronous initialization ordering. So, I hope that if this change does go through, an alternative class will be provided that retains the current behavior.

[mhintzke]

@davidfowl is your commit here still a true WIP or are you redacting your idea to go this route? I found the documentation here that references this issue a little confusing as it initially made me think that even using await at the top of ExecuteAsync() would block the other host intialization when it actually doesn't (due to the fact that its actually the issuer's NotificationService enumeration that was blocking on his BlockingCollection)

I believe any confusion would be mitigated by doing what you initially proposed in that commit and having the abstraction dispatch our executing Task, although I also see the benefits of teaching the community about how to properly use it as-is as well.

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. Please help me learn by adding exactly one area label.

[HelloKitty]

I noticed that hangs can now occur in later versions of ASP Core if you do a Task.Delay. Previous versions of ASP Core were not causing this issue but I don't have exact version information. Calling await Task.Delay(JobConfig.IntervalMilliseconds, cancellationToken); within public async Task StartAsync(CancellationToken cancellationToken) In ASP Core 3.1 now, unlike a previous version, blocks startup for some reason.

[vova-lantsov-dev]

@HelloKitty IHostedService.StartAsync always blocks the execution of the whole host. This is expected behaviour. Try to use BackgroundService.ExecuteAsync for non-blocking delayed tasks.

[HelloKitty]

@vova-lantsov-dev It appears so, but I do not think this was the case in a previous version of ASP Core. Maybe 2.1. This ran fine without hanging a year ago, anyway I have adopted BackgroundService.ExecuteAsync and now delay in there. Seems to work.

[drewkill32]

This causes an issue when unit testing. If I mock a service to return a CompletedTask instead of an awaited task StartAsync never returns.

 [TestClass]
public class TestClass
{
        [TestMethod, TestCategory("UnitTest")]
        public async Task Background_Service_Should_Return_FromStartAsync()
        {
            
            var sut = new FakeService();

            await sut.StartAsync(Token);

            //code never reaches here because ExecuteAsync never awaits a task
            Assert.IsTrue(true);
        }
}
public class FakeService : BackgroundService
{
        protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            while (!stoppingToken.IsCancellationRequested)
            {
                await Task.CompletedTask;
            }
        }
}

[christallire]

I've run into this too.
@eerhardt could we make this happen in 8.0? this could be a relatively simple fix compared to the time spent scratching their heads :)

[Xor-el]

I think this should make it in NET 8.0 as indicated here
#86511
We have gotten two new properties added that should resolve this.

https://learn.microsoft.com/dotnet/api/microsoft.extensions.hosting.hostoptions.servicesstartconcurrently?view=dotnet-plat-ext-8.0

https://learn.microsoft.com/dotnet/api/microsoft.extensions.hosting.hostoptions.servicesstopconcurrently?view=dotnet-plat-ext-8.0

[StephenCleary]

The IHostedLifecycleService and ServicesStartConcurrently changes in .NET 8.0 do not address this issue. The host (as currently written) still starts the services by calling StartAsync directly. So a synchronous implementation of StartAsync will still block host startup.

[lonix1]

Sad that even in v8 this won't be fixed.

For anyone who lands here, see Stephen Cleary's blog series which provides workarounds for these issues.

[naumenkoff]

.NET 8 has been released, and unfortunately, Microsoft has turned a blind eye to this issue.
For those who have encountered it, it would be wise not to use BackgroundService. Instead, you can create your implementation based on IHostedService or the new IHostedLifecycleService.

The situation is amusing - to start a synchronous service, you need to await something, for example, Task.Yield, resulting in the generation of a state machine and a loss of at least 0.05% performance at this stage.
Alternatively, you can use await Task.Run and lose at least 0.10% simply because of the presence of a state machine to start the service elegantly with just one addition to the DI container.

[montella1507]

Yeah.. MS is way more focusing on implementing YET next type of UI library, just to deprecate and kill it 1 year later, rather then solve these issues..

[davidfowl]

I’m very confused why Task.Run isn’t a viable workaround?

[naumenkoff]

I’m very confused why Task.Run isn’t a viable workaround?

You can do that until the presence of a state machine becomes a hindrance to you, and the TaskScheduler works perfectly.

[davidfowl]

You can do that until the presence of a state machine becomes a hindrance to you

When does that happen?

[StephenCleary]

Task.Run is a perfectly viable workaround. But I'd still like to see it baked into BackgroundService because it's not currently a pit of success.

Most developers don't think about async methods as starting synchronously. There's a number of SO questions that are variants of "I started with a background service template and it works fine until I try to remove the Task.Delay" or "it stops working as soon as I try to [synchronously] read from a queue of work". For those of us with a good understanding of async and how the host starts background services, the problem is obvious; but that's not most developers.

[JohannSig]

This sounds like an unnecessary "rite of passage" pitfall for anyone deciding to stroll into BackgroundService country. Myself twice now (2020 + 2024) 😆

[jesslilly]

Nothing here was working for me...

🔴 As it turns out you cannot test this with breakpoints!

Maybe that seems logical to some people, but it did not dawn on me until I spent an hour, tried every single solution and workaround in this issue, and finally said, to myself, "Hmmm, maybe I'll try logging instead of using a breakpoint".

Give me a thumbs up if this was helpful. And give me a pie in the face otherwise! 🤣

[BernhardPollerspoeck]

... Most developers don't think about async methods as starting synchronously. There's a number of SO questions that are variants of "I started with a background service template and it works fine until I try to remove the Task.Delay" or "it stops working as soon as I try to [synchronously] read from a queue of work". For those of us with a good understanding of async and how the host starts background services, the problem is obvious; but that's not most developers.

this sounds more like a edjucation/knownledge or documentation issue then. Just slapping async/await onto everything is not "working with tasks".
Maybe a analyzer would help a lot here to clarify and guide?

[christallire]

Everything should've been built in a foolproof way.

[lonix1]

Hit this issue again. It takes a long time to diagnose the issue and arrive here for various workarounds. If one doesn't have guru-level async knowledge, it's an insidious time waster.

Please please please consider this for v9. 🙏

[Bertramp]

Just hit this issue again. I reviewed some PR's for some of our senior devs and assumed it had been fixed since they are very experienced. Turns out they had never used .NET background services and too just fell into the newb trap.

Something called BackgroundService simply cannot under any circumstance be allowed to not run on a background thread. Currently the default behaviour of BackgroundService just hides the fact that StartAsync simply calls ExecuteAsync. Any sane person would assume that since StartAsync is not declared abstract, it's because it handles dispatching the actual work to a background thread, and that you shouldn't mess with that implementation unless you really have to.

[davidfowl]

I'd be fine taking a breaking change in .NET 10 for this. We can have an option to change the behavior back for where it breaks people (possibly per BackgroundService so that the break can be isolated)..

Thoughts @ericstj @eerhardt ?

[ericstj]

I think that's a reasonable breaking change to take. Is a property or constructor parameter on BackgroundService sufficient for the control of this? I imagine in most cases the derived class would chaose one option or the other. Anyone want to tackle converting this to an API proposal? cc @steveharter

[BowserKingKoopa]

This is such a funny issue. On the one hand you'd think starting your "Background Service" would run in the background. On the other hand async/await runs sync until it hits something async. So you either get a weird idea of an 'eventually background service' or you make this a special case where await doesn't actually do what await does everywhere else in .net. Is this just fallout from the weird 'runs sync until async' thing. That can bite you in other circumstances too. It's weird, or maybe 'unexpected' or 'unintuitive' rather.

For my background services I'm just Task.Yielding at the start of them reflexively now so I don't even have to think about when it switches to actually async.

[davidfowl]

I think that's a reasonable breaking change to take. Is a property or constructor parameter on BackgroundService sufficient for the control of this? I imagine in most cases the derived class would chaose one option or the other. Anyone want to tackle converting this to an API proposal? cc @steveharter

Yep! I took a stab at it, and marked it ready for review. Feel free to edit the proposal or suggest better naming (or patterns).

[terrajobst]

Video

• We don't believe many people will be broken by the new behavior
  • People who are, can either implement IHostedService or do their own dependency handling
• IOW, we like the behavior breaking change but not the API.
  • We will close this issue (as it tracks the API). There will be another issue.

namespace Microsoft.Extensions.Hosting;

public class BackgroundService : IHostedService
{
    public virtual bool StartAsynchronously { get; } = true;
}

[StephenCleary]

I like and agree with just making this always on. Can you link the new issue number here when available?