startMode="AlwaysRunning" for inProcess ANCM

[shirhatti]

There was an earlier issue asking for this (aspnet/AspNetCoreModule#29) which we declined to do when ANCM was only out of process.

At the minimum, we need to ensure ANCM doesn't break with AlwaysRunning for 2,1. I suspect it might just work, but @jkotalik and @pan-wang know better.

[pan-wang]

AlwaysRunning just let WAS start w3wp.exe. Customer's true ask is about "Application_Start" or a "Startup" class in asp.Net core. I don't think asp.net core emit such notification. @davidfowl @Tratcher can you confirm it?

[asbjornu]

I'm with @pan-wang in that the requirement here might be two or even threefold. Solving the IIS worker process recycle mechanism isn't a real solution, at least not to me. Microsoft has managed to make this very complicated and hard to understand by having three different related and similar features for "keeping an IIS application warm" (which I believe is what most developers want):

1. Implementing the IProcessHostPreloadClient interface and registering it within the serviceAutoStartProviders section of IIS.
2. Configure the AutoStart feature.
3. Install and configure the applicationInitialization IIS module.

I think the first requires the second point, but I have no idea how they interact with the third. Why this has to be so difficult is impossible for me to understand, but I believe the feature request is as simple as: Please give us a way to control how our ASP.NET Core web applications are recycled and when they are, make IIS perform an HTTP request to it so the application can warm itself up.

[tuespetre]

@shirhatti @asbjornu my original request was basically this: when the application pool starts I want the app to also start. I don’t want it to wait for a request to start.

[sergioadh]

I participated in the previous thread and I was asking for what @tuespetre said as well

[shirhatti]

my original request was basically this: when the application pool starts I want the app to also start. I don’t want it to wait for a request to start.

Yup, we're on the same page here. Forgive my rather ambiguous issue title

[Looooooka]

https://blogs.msdn.microsoft.com/benjaminperkins/2014/01/07/configure-the-iis-application-initialization-module/
this works for me

[jkotalik]

I think we need to keep the current AlwaysRunning functionality the same as before. If we want to support "when the application pool starts I want the app to also start," we should design a different feature. For example, @BillHiebert wanted to use "AlwaysRunning" to allow VS to debug attach to w3wp.exe before the first request comes in.

[shirhatti]

I disagree. The semantic meaning of the feature is not just to start w3wp.exe, but rather start your application as well. This is what happens with a system.web app today.
Just because we got we never got around to doing it in ANCM thus far, we don't have to preserve that status quo.

Designing it is a new feature is not a viable option since we can't modify IIS schema to add another attribute to the AppPool.

[jkotalik]

Will this be any different from the app warmup module https://blogs.msdn.microsoft.com/bryang/2011/04/29/iis-application-warm-up-module/ ?

[rockerinthelocker]

This thread is somewhat confusing because IIS Application Initialization already works well with ASP.NET Core (via startMode="AlwaysRunning" and preloadEnabled="true" settings) to 'warm up' an ASP.NET Core application (i.e. via 'Configure' method). Can anyone shed some light on this? Thank you!

[sergioadh]

AlwaysRunning and preloadEnabled control the application pool but not the ASP.NET Core app, which shuts down somehow. Even though the app pool is "warmed-up" the first requests will take a long time since the ASP.NET Core app is not "warmed-up".

There were previous issues and this has been moved around but you can read them:

aspnet/AspNetCoreModule#29

aspnet/IISIntegration#161

[tuespetre]

@jkotalik @rockerinthelocker:

The difference between this proposed feature and using the warmup/initialization module is that this proposed feature would not require a 'sentinel' web request to be issued to kick off the application. Additionally, when an application running under ANCM 'crashes', ANCM will simply wait until another request is issued to start up the application again. This is fundamentally disconnected from the application pool lifecycle, and thus the initialization module.

[rockerinthelocker]

@sergioadh , Well, it's up to the application developer to implement 'warm up' tasks to be executed on application start. There is no magic the ASP.NET Core module can do for developers.

@tuespetre , If it were possible to implement such a feature, no hosting provider would install the ASP.NET Core module on shared servers. Just imagine thousands of sites hosted on a server would 'warm up' on each IIS start/reset/configuration change.

[Looooooka]

I really don't see what the problem is here. The module literally allows you to set a start url which is called when the pool is started. All you need to do is handle it and preload whatever you want.

[sergioadh]

That's ok and good but the Kestrel server goes down even though the ApplicationPool is up, so when the app pool recycles it will send the request and we can get everything up and running but then Kestrel goes down and if a request comes in it takes a while to respond.

[pan-wang]

@sergioadh I got your point now. This is doable. ANCM does listen on the process exit of Kestrel server. If the backend is down, ANCM can immediately start another one. We need a configure flag to allow user to tune the behavior. This can be feature for next release.

[rockerinthelocker]

@pan-wang @jkotalik @Tratcher @muratg @coolcsh @davidfowl @DamianEdwards @halter73 , Simply restarting an application that might be unhealthy is certainly not the proper course of action; and probably not the job of the ASP.NET Core module anyway.

However, looking at how things are handled during application pool start/recycle/stop and site (application) start/restart/stop, it seems they are not handled properly.

Here is a simple setup (requires IIS Application Initialization) to reproduce the results shown below (replace 'C:\results.txt' with a path the application has write access to; and set startMode="AlwaysRunning" on the application pool and preloadEnabled="true" on the site's default application in applicationHost.config).

Startup.cs:

public void Configure(IApplicationBuilder app, IHostingEnvironment env, IApplicationLifetime appLife)
{
	System.IO.File.AppendAllText(@"C:\results.txt", $"Start executing Configure()...{System.Environment.NewLine}");
	System.IO.File.AppendAllText(@"C:\results.txt", $"Register IApplicationLifetime.ApplicationStarted, IApplicationLifetime.ApplicationStopping, IApplicationLifetime.ApplicationStopped...{System.Environment.NewLine}");

	appLife.ApplicationStarted.Register(ApplicationStarted);
	appLife.ApplicationStopping.Register(ApplicationStopping);
	appLife.ApplicationStopped.Register(ApplicationStopped);

	app.UseMvc(routes =>
	{
		routes.MapRoute(
		name: "default",
		template: "{controller=Home}/{action=Index}/{id?}");
	});

	System.IO.File.AppendAllText(@"C:\results.txt", $"Finished executing Configure()...{System.Environment.NewLine}");
}

private void ApplicationStarted()
{
	System.IO.File.AppendAllText(@"C:\results.txt", $"Executing ApplicationStarted()...{System.Environment.NewLine}");
}
private void ApplicationStopping()
{
	System.IO.File.AppendAllText(@"C:\results.txt", $"Executing ApplicationStopping()...{System.Environment.NewLine}");
}
private void ApplicationStopped()
{
	System.IO.File.AppendAllText(@"C:\results.txt", $"Executing ApplicationStopped()...{System.Environment.NewLine}");
}

HomeController.cs

public class HomeController : Controller
{
	public IActionResult Index()
	{
		System.IO.File.AppendAllText(@"C:\results.txt", $"Executing HomeController.Index()...{System.Environment.NewLine}");

		return View();
	}
}

1. Application Pool Start

Start executing Configure()...
Register event handlers for IApplicationLifetime.ApplicationStarted, IApplicationLifetime.ApplicationStopping, IApplicationLifetime.ApplicationStopped...
Finished executing Configure()...
Executing ApplicationStarted() event handler...
Executing HomeController.Index()...

2. Application Pool Recycle

Start executing Configure()...
Register event handlers for IApplicationLifetime.ApplicationStarted, IApplicationLifetime.ApplicationStopping, IApplicationLifetime.ApplicationStopped...
Finished executing Configure()...
Executing ApplicationStarted() event handler...
Executing HomeController.Index()...
Executing ApplicationStopping() event handler...
Executing ApplicationStopped() event handler...

3. Request After Application Pool Recycle

Executing HomeController.Index()...

4. Application Pool Stop

Executing ApplicationStopping() event handler...
Executing ApplicationStopped() event handler...

5. Application Pool Start

Start executing Configure()...
Register event handlers for IApplicationLifetime.ApplicationStarted, IApplicationLifetime.ApplicationStopping, IApplicationLifetime.ApplicationStopped...
Finished executing Configure()...
Executing ApplicationStarted() event handler...
Executing HomeController.Index()...

6. Site Stop

Executing ApplicationStopping() event handler...
Executing ApplicationStopped() event handler...

7. Site Start

Nothing logged. Application is not started.

Conclusion:

1. ApplicationStopping and ApplicationStopped being raised on application pool recycle is certainly a bug.
2. Starting/restarting the site in IIS (unloads the application) should behave the same way as if the application pool was started (i.e. starting up the application).

[davidfowl]

ApplicationStopping and ApplicationStopped being raised on application pool recycle is certainly a bug.

I'm confused by this. A recycle is a stop then a start. Can you clarify what you mean here? What are you expecting?

Starting/restarting the site in IIS (unloads the application) should behave the same way as if the application pool was started (i.e. starting up the application).

Site starting in IIS AFAIK is just removing and adding bindings. There's no process activity associated with sites, just application pools (since those are the processes).

[rockerinthelocker]

@davidfowl , Sure, the proper order of 'events' logged should be

2. Application Pool Recycle

Executing ApplicationStopping() event handler...
Executing ApplicationStopped() event handler...
Start executing Configure()...
Register event handlers for IApplicationLifetime.ApplicationStarted, IApplicationLifetime.ApplicationStopping, IApplicationLifetime.ApplicationStopped...
Finished executing Configure()...
Executing ApplicationStarted() event handler...
Executing HomeController.Index()...

However, ApplicationStopping/ApplicationStopped are raised after the new application pool is started.

Actually, when stopping a site, IIS sets the 'serverAutoStart' attribute to 'false'. Anyway, ANCM unloads the application if the site is stopped (i.e. during restart) so it should reload the application if the site is started again. Because site start/stop triggers a configuration change event, ANCM may subscribe to that event.

[KLuuKer]

@davidfowl i think @rockerinthelocker is talking about (or not aware of) overlapping recycles

[davidfowl]

@rockerinthelocker can you update your logs to print the process ID prefixed on each of the lines?

[rockerinthelocker]

@davidfowl , Thanks for the heads up! Indeed, the overlapped application pool recycle caused the confusion here. So, just the issue with site start is left.

[thegoodnerd]

@rockerinthelocker I'm observing exactly the same behavior and have this far been unable to find a way around this.

[dotnetcanuck]

@rockerinthelocker @winretri I found my way here after having the exact same issues with an application at my company. We're seeing startup times (on initial requests) north of 30 seconds, with subsequent requests being processed at the speed of light.

I'm sorry, but that is completely ridiculous for a web application.

@sergioadh @Looooooka Can either of you elaborate on how I can specify a 'startup' URL for the AspNetCoreModule to handle? I can't find anything in the configuration reference.

[orionstudt]

@dotnetcanuck I don't think a Startup URL will solve the problem for you, because as @sergioadh pointed out - the Startup URL will be hit when the Application Pool starts, but then Kestrel will go down when it is idle.

My short term resolution is a separate console application that regularly pings the app to prevent Kestrel from going down.. but that is less than ideal. Hence the feature request.

[dotnetcanuck]

console application that regularly pings the app to prevent Kestrel from going down.. but that is less than ideal.

@orionstudt I'll say! We're looking at doing something similar to work around the issue for now.

Really hoping that Microsoft can implement this feature though. 😞

[dotnetcanuck]

@orionstudt How often (ballpark) would you say that Kestrel takes itself down?

Still looking at implementing a console app or health check, but in our testing, we're finding that if the API gets hit several times in quick succession, requests will suddenly start getting queued, and the application gets hung/paused for about 40 seconds until API calls begin getting responses.

Obviously, I have more investigation to do on that, and I don't mean to hijack the thread. But I guess my main thought there is, is Kestrel shutting itself down suddenly, or is there something else going on?

[rockerinthelocker]

@dotnetcanuck , The Configure method in Startup.cs is certainly a good place to run 'warm up' tasks. In order to let IIS send a user request to the application on start/recycle, set

startMode="AlwaysRunning" on the application pool,
serverAutoStart="true" on the site (set by default), and
preloadEnabled="true" on the application.

References:
startMode: https://docs.microsoft.com/en-us/iis/configuration/system.applicationhost/applicationpools/add/index
serverAutoStart: https://docs.microsoft.com/en-us/iis/configuration/system.applicationhost/sites/site/
preloadEnabled: https://docs.microsoft.com/en-us/iis/configuration/system.applicationhost/sites/site/application/

Note, however, that when publishing the application (i.e. via Visual Studio), IIS won't send the user request and you would have to recycle the application pool.

[MovGP0]

In Azure, there is the Always on option that just works and don't require any hacks. Unfortunately, it isn't as trivial for on-premise applications.

In IIS, I've set Start Mode to AlwaysRunning on the Application Pool and Preload Enabled to True in IIS. Still, the ASP.NET Core (Kestrel hosted) web application goes to sleep, killing the nightly Hangfire Background jobs with it.

So the best workaround seems to be a polling service that does regular health checks. Unfortunately, that is not an option everywhere. So you might need to have a Task Scheduler (Windows) or cron job (Linux/Unix) to implement a simple polling service.

Reference: Polling Service with Task Scheduler

As a workaround, I'm using Task Scheduler to create a task that runs every minute and polls the health check endpoint of my service.

• Open Task Scheduler
• Create Task...
• On General, enter a Name and use Change User Group... to define the user permission. Set Run whether user is logged on or not.
• On Triggers, use New... to create a new trigger.
  • Set Begin the task: to At Startup
  • Set Repeat task every to 5 minutes
  • Set for a duration of: to Indefinitely
  • Set enabled
  • OK
• On Actions, use New... to create a new action
  • Set Action: to Start a program
  • Set Program/script to powershell.exe
  • Set Add arguments (optional) to 'curl "https://mywebsite/hc"' (change the URL to whatever endpoint you need)
  • OK
• OK
• Enter the User password as needed.

Make sure that the user you choose has the proper privileges.

Set User rights for batch job

• Open gpedit.msc and navigate to Computer Configuration\Windows Settings\Security Settings\Local Policies\User Rights Assignment.
• Right-click Log on as a batch job and select Properties.
• Add User or Group...
• enter the user name in the form of DOMAIN\USERNAME and click OK
• OK

You may want to test-run the service at this point. If everything works, you can restart the computer. Check the Task History to ensure everything works properly.

[Looooooka]

Like I wrote before...always running. Plus under configuration editor(not site but the main icon set under iis manager) click on system.applicationHost/sites. You can enable autostart there.
Now if you want/need a timer just start a timer when the app goes live and use a webclient or httpnetclient to open an url every x seconds. On windows in the past you could literaly put an object inside the iis process which would stay alive when iis would do the apppool cycling every 20 minutes. But this works just fine as well since kestrel will be restarted in case errors start popping up. Been using this technique in production since forever and we know exactly when something dies/doesn't start.

[jkotalik]

@Looooooka with the release of 2.2, the module AspNetCoreModuleV2 should make startMode="AlwaysRunning" work. Please give it a try.

@carlowahlstedt we now have documentation on ANCM Inprocess in our docs. See https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/iis/?view=aspnetcore-2.2#iis-options and https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/aspnet-core-module?view=aspnetcore-2.2

@Joelbear5 startMode="AlwaysRunning" is a feature of IIS and isn't implemented by Kestrel itself. If you are using a reverse proxy, there may be a feature to keep Kestrel alive.

[carlowahlstedt]

Thanks @jkotalik. What's the difference between the web.config hostingModel="InProcess" and the csproj <AspNetCoreHostingModel>InProcess</AspNetCoreHostingModel>? Are both needed?

[jkotalik]

@carlowahlstedt sorry for the incredibly late reply. InProcess is used to populate the web.config if there isn't one present in the project. If you have a web.config in your directory, it should always be used when you publish the application.

[YuliiaBabychKdd]

any updates?
We have netcoreapp2.2, but still same issue:(

Any magic tricks or ideas how to fix? or when it will be fixed?

[binarypatrick]

We just wrote a tickle script

param($Uri)

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
Invoke-WebRequest -Uri $Uri | Select-Object -ExpandProperty StatusCode

[YuliiaBabychKdd]

@binarypatrick Thanks. Finished with same solution^^

[robbaman]

I've noticed that when you setup AlwaysRunning with a AspNet Core application (2.2) and I recycle the application (by clicking recycle in the IIS Manager) it correctly starts a new instance including the actual Kestrel-based application. This is also visible in the Windows Event Log with two events:

4-7-2019 12:04:24: Application 'D:\Sites\ShippingService' started the coreclr in-process successfully.
4-7-2019 12:04:26: Application 'MACHINE/WEBROOT/APPHOST/SHIPPINGSERVICE' has shutdown.

Note how the Shutdown follows the start of a new instance. When the application pool lifecycle is managed/interrupted by an app_offline.html however, this does not work. Instead the logs are these:

4-7-2019 11:58:30: Application 'D:\Sites\ShippingService' was recycled after detecting the app_offline file.
4-7-2019 11:58:33: Application 'MACHINE/WEBROOT/APPHOST/SHIPPINGSERVICE' has shutdown.

Notice a lack of the "started the coreclr in-process successfully" event in the logs. The Kestrel-application does indeed not start in this instance. This makes publishing a new version via Webdeploy.exe pretty hard. Once a single request comes in the site starts properly and everything works.

[unionthugface]

Sorry to reply to an old thread, but I am desperate. I have a .NET Core 2.2 app, in-process, hosted in IIS 10. I have application pool set to AlwaysRunning with Timeout=0, Preload=true; however my app still stops every 29 hours when the application pool recycles, and then does not restart itself until someone pings the URL. Back in the .NET Framework days I could write a bootstrapper class and hook into the Global.asax file/IIS ApplicationPreload settings; I haven't found any such thing for .NET Core and would so appreciate if someone could point me to one. My office is going nuts because we moved a recurring jobs handler app to .NET Core, and now it won't work because of this application pool recycle thing; every 29 hours on the dot, it stops. What am I missing? If I go back to my boss and tell him we need to write a tickle script like someone suggested, he'll be upset.

[Looooooka]

First of all just set the managed pipeline to classic and net clr mode to no managed code.
As you've noticed it's completely useless since the preload thing doesn't work as expected so loading the NET CLR is a giant waste of resources.

Instead change the Configure method in Startup.cs and register a method you wish to execute when the app is started using the ApplicationStarted functionality(you can handle the shutdown as well).
Then if you want to let's say "preload" a controller or anything so speed up requests just add some httpclient logic in that (OnStartUp) method and make a call to the website itself. Seems to work for us(since 2.0 and now on 3.1 as well).

public void Configure(IHostApplicationLifetime applicationLifetime, IApplicationBuilder app) { applicationLifetime.ApplicationStarted.Register(OnStartUp); }

also don't forget to use the UseIISIntegration and UseIIS calls!

[Joelbear5]

My team has migrated away from dotnet core due to this issue. Even setting health check pings to an API only loads that controller. I think Azure might have a way to keep APIs "warm", but IMHO dotnet core completely failed us as a web server.

[Coruscate5]

For all those still experiencing this issue - after a lot of research, I determined that "new" servers did not have any issues autostarting .NET Core apps per the docs.

Based on ProcMon and Net call traces, it looks like this is a problem with IIS - it gets confused when multiple custom modules are installed after original install.

So, the solution was (somewhat) simple - Reinstall IIS completely (then install your custom .NET core runtime modules again after). Most of your settings will be saved - I wrote a logger in my test service to make sure the service kept running as well.

Check my SO post: https://stackoverflow.com/questions/59905391/net-core-3-iis-application-initialization-doesnt-work

[guardrex]

Feedback on the IIS topic indicates that this might be the issue there. dotnet/AspNetCore.Docs#16675

[ljani]

This seems a good issue to mention that if IIS has only Windows authentication enabled, then this won't work. You will need to enable anonymous authentication as well. I was able to figure this out using Failed Requests Tracing.

[Coruscate5]

@guardrex - I'm not sure if you have any responsibility over the IIS portion here, but, it appears that IIS eventually stops respecting the warm-up even after a reinstall and re-registering of modules.

Seems like some sort of internal IIS issue (or chaining issue), because it will load & run methods from warmup.dll, but then never actually start the dotnet process specifically for .NET Core. (Re)reinstalling IIS fixes it temporarily

[guardrex]

I just work on the docs @Coruscate5. Engineering (here) would need to follow up on your report.

[Coruscate5]

@guardrex - OK, thanks - given that this particular thread seems to have gone off the rails, I'll open a new issue under the aspnetcore repo when I have some time (given the strange nature of this happening "eventually" to IIS servers hosting .NET Core projects with background tasks, I'm not sure if I'll actually get any traction)

[unionthugface]

I posted this potential solution on @Coruscate5 's thread new issue #19509 above
( #19509 (comment) ) but figured I'd post it here too:

In my case, lack of Garbage Collecting was making the problem. I traced the memory usage and it got increased by any page load. I just added this to my .csproj file:
<PropertyGroup> <ServerGarbageCollection>true</ServerGarbageCollection> <ConcurrentGarbageCollection>false</ConcurrentGarbageCollection> </PropertyGroup>
and the problem got solved.

Originally posted by @mobinseven in #11950 (comment)

I deployed this in my .NET Core 3.1 app on Friday, came back in this morning (Monday), and my app successfully shut itself down and restarted itself all weekend, no gaps in service.

tl:dr; this appears to be a problem with .NET Core garbage collection.

Would love if anyone could chime in about why this works and/or any potential side effects. I am too ignorant to make a guess, but my app is definitely working as expected with this code in place.

[MV10]

I'll be "that guy" and bump this ... is any of this targeted for any specific release?

Specifically, the IProcessHostPreloadClient option was only mentioned once way back in 2018, but it is still very different from the other options and very important because it happens without any restrictions or overhead associated with servicing a request.

We have some systems that have to cache data from very, very slow mainframe back-ends, and if they rely on requests for their initial load, those requests will time out. Those folks are now migrating their apps to ASP.NET Core (finally!) and all we have to guarantee pre-load are goofy scripts.

[MV10]

Sigh, another year goes by...

The pre-load scripts (ie. request some "warmup" URL) are a problem because nothing prevents other real, live requests from bombarding the app while it's doing the warmup thing. That's the other very important way in which IProcessHostPreloadClient is very different from everything else here.

Within the past year, many of our internal apps still stuck on .NET Framework have been implementing pre-load. I really hope it comes to ASP.NET Core soon.