[logs] Reloading application configuration may lead to unnecessary thread creation

[CodeBlanch]

Overview

The delegate passed to OpenTelemetryLoggingExtensions.AddOpenTelemetry is registered as an options configuration (IConfigureOptions) and will be replayed on a fresh instance of OpenTelemetryLoggerOptions anytime the application IConfiguration is reloaded. If that delegate performs openTelemetryLoggerOptions.AddProcessor(new BatchLogRecordExportProcessor(...)) a background thread is created which is never signaled to shutdown. In processes which reload configuration periodically this will lead to an ever-increasing number of threads being created. Note: The additional threads are never assigned logs to export/process they will just sit asleep and periodically wake up to check if there is work to do (which there never is).

Reproduction \ example

var root = new ConfigurationBuilder().Build();

var services = new ServiceCollection();

services.AddSingleton<IConfiguration>(root);

services.AddLogging(logging => logging
    .AddConfiguration(root.GetSection("logging")) // Note: This is performed automatically when using a host
    .AddOpenTelemetry(options =>
    {
        options.AddProcessor(new BatchLogRecordExportProcessor(new OtlpLogExporter(new())));
    }));

using var serviceProvider = services.BuildServiceProvider();

var loggerFactory = serviceProvider.GetRequiredService<ILoggerFactory>();

for (int i = 0; i < 100; i++)
{
    root.Reload();
}

// 101 threads created here

Not impacted

• Delegates using the factory form of AddProcessor which accepts the sp parameter (options.AddProcessor(sp => BuildProcessor(sp)) are NOT impacted. The factory delegate is never invoked for configuration reloads. The AddOtlpExporter extension uses the factory form so it is NOT impacted.

• The experimental LoggerProviderBuilder and WithLogging APIs are NOT impacted.

   var root = new ConfigurationBuilder().Build();
  
   var services = new ServiceCollection();
  
   services.AddSingleton<IConfiguration>(root);
  
   services.AddLogging(logging => logging
       .AddConfiguration(root.GetSection("logging"))); // Note: This is performed automatically when using a host
  
   services.AddOpenTelemetry()
       .WithLogging(builder =>
       {
           builder.AddProcessor(new BatchLogRecordExportProcessor(new OtlpLogExporter(new())));
       });
  
   using var serviceProvider = services.BuildServiceProvider();
  
   var loggerFactory = serviceProvider.GetRequiredService<ILoggerFactory>();
  
   for (int i = 0; i < 100; i++)
   {
       root.Reload();
   }
  
   // Only 1 thread created here