core monitoring health
Monitoring: Health CheckNetFusion integrates checking the overall health of a microservice into its defined plug-in architecture. Since a microservice is composed of plug-ins and modules, they provide the ideal location for detecting the overall health of a microservice.
Modules within a plug-in can contribute in determining the overall health of the microservice by implementing the IModuleHealthCheckProvider interface. Modules implementing this interface have the CheckModuleAspectsAsync method invoked and passed an instance of the ModuleHealthCheck class. The ModuleHealthCheck instance is used to record the current health of specific aspects managed by the module. The overall health of a plug-in is represented by the PluginHealthCheck class and is based on the most severe HealthCheckStatusType recorded by it contained plug-in modules. Similarly, the overall health a microservice is represented by the CompositeAppHealthCheck class and is based on the most severe HealthCheckStatusType recorded by the plug-ins from which the composite-application was built.
This example will define an application and infrastructure plug-in modules used to determine the current health of the microservice. Health checks are commonly used when running a microservice within an component orchestrator such as Kubernetes or behind a load-balancer. When this is the case, messages are only sent to microservice instances reporting a healthy status.
The following will define an application plugin module that participates in determining the overall health of the microservice. Within the Examples.Monitorig.App project, define the following plugin module service and implementation used to track the number of pending requests.
Add the module service contract to the following directory: Examples.Monitoring.App/Plugin
using NetFusion.Core.Bootstrap.Plugins;
namespace Examples.Monitoring.App.Plugin;
public interface IPendingRequestsMonitor : IPluginModuleService
{
void SetPendingRequests(int numberPending);
}Add the module implementation to the following directory: Examples.Monitoring.App/Plugin/Modules
using System.Threading.Tasks;
using NetFusion.Core.Bootstrap.Health;
using NetFusion.Core.Bootstrap.Plugins;
namespace Examples.Monitoring.App.Plugin.Modules;
public class PendingRequestsModule : PluginModule,
IPendingRequestsMonitor,
IModuleHealthCheckProvider
{
private int _numberPending;
public void SetPendingRequests(int numberPending)
{
_numberPending = numberPending;
}
public Task CheckModuleAspectsAsync(ModuleHealthCheck healthCheck)
{
HealthAspectCheck aspect = _numberPending switch
{
>= 0 and <= 5 => HealthAspectCheck.ForHealthy("Pending-Requests", _numberPending.ToString()),
> 5 and <= 50 => HealthAspectCheck.ForDegraded("Pending-Requests", _numberPending.ToString()),
> 50 => HealthAspectCheck.ForUnhealthy("Pending-Requests", _numberPending.ToString()),
_ => HealthAspectCheck.ForUnhealthy("Pending-Requests", _numberPending.ToString())
};
healthCheck.RecordAspect(aspect);
return Task.CompletedTask;
}
}This plug-in module implements a service interface named: IPendingRequestsMonitor that will be injected into a MVC controller so the number of pending requests can be changed at runtime to vary the overall health of the module.
Add the new PendingRequestModule to the plugin by adding the following line of code to the AppPlugin class:
//...
public AppPlugin()
{
AddModule<ServiceModule>();
AddModule<PendingRequestsModule>(); // <--- Add this line
Description = "Plugin component containing the Microservice's application services.";
}
// ...The code within the CheckModuleAspectsAsync method determines if the health of the plugin is healthy, degraded, or unhealthy based on it current state. A plugin reports its current health on one or more aspects related to its implementation identified by a string identifier and value. The identifier and value are logged so it can be determined why the microservice is reporting a given health status. Each aspect is added to the passed ModuleHealthCheck reference and are used to determine the overall health of the plugin.
The following will define an infrastructure plugin module that participates in determining the overall health of the microservice. Within the Examples.HealthChecks.Infra project, define the following plugin module service and implementation used to track the number of cached items.
Add the module service contract to the following directory: Examples.Monitoring.Infra/Plugin
using NetFusion.Core.Bootstrap.Plugins;
namespace Examples.Monitoring.Infra.Plugin;
public interface ICachedItemsMonitor : IPluginModuleService
{
void SetCachedItems(int numberPending);
}Add the module implementation to the following directory: Examples.Monitoring.Infra/Plugin/Modules
using System.Threading.Tasks;
using NetFusion.Core.Bootstrap.Health;
using NetFusion.Core.Bootstrap.Plugins;
namespace Examples.Monitoring.Infra.Plugin.Modules;
public class CachedItemsModule : PluginModule,
ICachedItemsMonitor,
IModuleHealthCheckProvider
{
private int _numberCached;
public void SetCachedItems(int numberCached)
{
_numberCached = numberCached;
}
public Task CheckModuleAspectsAsync(ModuleHealthCheck healthCheck)
{
HealthAspectCheck aspect = _numberCached switch
{
>= 0 and <= 1000 => HealthAspectCheck.ForHealthy("Cached-Items", _numberCached.ToString()),
> 1000 and <= 2000 => HealthAspectCheck.ForDegraded("Cached-Items", _numberCached.ToString()),
> 2000 => HealthAspectCheck.ForUnhealthy("Cached-Items", _numberCached.ToString()),
_ => HealthAspectCheck.ForUnhealthy("Cached-Items", _numberCached.ToString())
};
healthCheck.RecordAspect(aspect);
return Task.CompletedTask;
}
}Add the new CachedItemsModule to the plugin by adding the following line of code to the InfraPlugin class:
// ...
public InfraPlugin() {
AddModule<RepositoryModule>();
AddModule<CachedItemsModule>(); // <-- Add this line
Description = "Plugin component containing the application infrastructure.";
}
// ...
Defining the above contracts are not necessary and only exist so the modules can be accessed from a MVC controller to change the underlying data on which the health checks are based.
With the plugin modules defined, a test MVC controller will be declared injecting the IPendingRequestsMonitor and ICachedItemsMonitor services. Add the following code to the ExamplesController defined within the Examples.Monitoring.WebApi/Controllers directory:
using Examples.Monitoring.App.Plugin;
using Examples.Monitoring.Infra.Plugin;
using Microsoft.AspNetCore.Mvc;
namespace Examples.Monitoring.WebApi.Controllers;
[ApiController, Route("api/[controller]")]
public class ExamplesController : ControllerBase
{
private readonly IPendingRequestsMonitor _pendingRequests;
private readonly ICachedItemsMonitor _cachedItems;
public ExamplesController(
IPendingRequestsMonitor pendingRequests,
ICachedItemsMonitor cachedItems)
{
_pendingRequests = pendingRequests;
_cachedItems = cachedItems;
}
[HttpPost("pending/requests/{numberRequests}")]
public IActionResult SetPendingRequests(int numberRequests)
{
_pendingRequests.SetPendingRequests(numberRequests);
return Ok();
}
[HttpPost("cached/items/{numberCached}")]
public IActionResult SetCachedItems(int numberCached)
{
_cachedItems.SetCachedItems(numberCached);
return Ok();
}
}At this point, we have two modules that determine the overall state of the microservice based on the data it maintains. This sample data can be modified by invoking HTTP requests to change the overall health of the module, its containing plug-in, and the composite-application as a whole.
The last step is to expose an HTTP endpoint that can be invoked to determine the overall heath of the microservice. The NetFusion.Web Nuget contains an extension method to expose an endpoint that can be called to query the current health of the microservice.
This is accomplished by adding the following to Examples.Monitoring.WebApi/Program.cs:
// ..
var app = builder.Build();
app.UseSerilogRequestLogging();
app.UseRouting();
app.MapHealthCheck(); // <-- Add this line
app.MapControllers();
// ..Complete the following to run the example microservice and send a HTTP Post request to the example controller:
cd ./Examples.Monitoring/src/Examples.Montoring.WebApi/
dotnet runStart by checking the current health of the microservice by executing the following HTTP request:
curl http://localhost:5007/mgt/health-check --verbose
Since the current values of each module used to determine its health are both considered healthy values, the overall status of the composite-application reports a HTTP 200 OK status indicating a healthy status.
Next, make the following HTTP request to place the PendingRequestsModule module in a degraded state:
curl -X POST http://localhost:5007/api/examples/pending/requests/10
curl http://localhost:5007/mgt/health-check --verbose
Now that one of the modules has a degraded status, HTTP 503 Service Unavailable status is returned indicated the microservice does not have a healthy status. Since both the degraded and unhealthy statuses return a HTTP 500 status code, the header named Health-Check is added to the response indicating the specific status.
Next, execute the following HTTP request to place the CachedItemsModule in an unhealthy state:
curl -X POST http://localhost:5007/api/examples/cached/items/2500
curl http://localhost:5007/mgt/health-check --verbose
Since one module reported a degraded status and the other an unhealthy status, the health-check returns unhealthy since it is the more sever of the two statuses.
When a health-check status returns a HTTP status other than 200 OK, the details determining the overall status of the microservice are logged. Below shows the log entry create:
For each module, the values of each aspect used to determine the overall health of the microservice, are logged giving the developer detailed information used to isolate the underlying cause.
When running a microservice within a container orchestrator such as Kubernetes, the health endpoint is called often and therefore NetFusion excludes these requests from the log if the HTTP result is a 200 OK.
Lastly, execute the following to restore both modules to a healthy state:
curl -X POST http://localhost:5007/api/examples/pending/requests/3
curl -X POST http://localhost:5007/api/examples/cached/items/250
curl http://localhost:5007/mgt/health-check --verbose