Issue with TimeoutException During Load Testing (Duende Identity Server 7 + Postgres OperationalStore)

[olhakhariv]

Which version of Duende BFF are you using?
Duende Identity Server 7.0.7

Which version of .NET are you using?
.NET 8.0

Describe the bug
While conducting load testing with up to 50 simulated users (10 new users per second), we encounter a TimeoutException during the /connect/token request when the number of users increases

To Reproduce

1. Set up Duende Identity Server 7.0.7 with a PostgreSQL OperationalStore.
2. Simulate up to 50 users authenticating at a rate of 10 new users per second.
3. Observe the TimeoutException occurring in the /connect/token request after reaching the peak load.

Expected behavior
We expected the system to handle the requests without timing out as the load increases

Log output/exception with stacktrace

contextType
Duende.IdentityServer.EntityFramework.DbContexts.PersistedGrantDbContext

error
System.InvalidOperationException: An exception has been raised that is likely due to a transient failure.
 ---> Npgsql.NpgsqlException (0x80004005): The operation has timed out
 ---> System.TimeoutException: The operation has timed out.
   at Npgsql.ThrowHelper.ThrowNpgsqlExceptionWithInnerTimeoutException(String message)
   at Npgsql.Util.NpgsqlTimeout.Check()
   at Npgsql.Util.NpgsqlTimeout.CheckAndGetTimeLeft()
   at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|213_1(NpgsqlConnector conn, SslMode sslMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken, Boolean isFirstAttempt)
   at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
   at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
   at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenAsync(CancellationToken cancellationToken, Boolean errorsExpected)
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
   at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()

Additional context

We would like to understand if this is a typical issue when performing load testing under these conditions, or if we may have missed something in our configuration. Could you kindly provide any advice or recommendations on how we could address this problem and optimize the system for higher traffic?

Thank you for your support, and we look forward to your guidance.

[olhakhariv]

I think this can be related to the DuendeSoftware/products#1564

[RolandGuijt]

The exception is thrown in the ADO.NET data provider for PostgreSQL. Probably this isn't IdentityServer or even EF related. The database itself just can't keep up it seems.
If the rate of transactions you mentioned is needed in practice please investigate how to make the database itself perform better. For example: considering faster storage for the database files.

Should you still come to the conclusion this is related to IdentityServer, please report back so we can investigate further.

[RolandGuijt]

We also had some memory related problems with this provider lately. Turning on connection pooling helped.

.AddOperationalStore(options =>
        {
            options.ConfigureDbContext = optionsBuilder => { optionsBuilder.UseNpgsql(connectionString); };
            options.EnablePooling = true;
            OperationalStoreOptionsConfig.Configure(options);
        })

This may or may not help in your case but it's worth a try.

[olhakhariv]

Thank you for the suggestions. Unfortunately, they didn’t resolve the issue.

I’d like to highlight that this problem did not occur with .NET 6 and Duende version 6.2.3. At that time, everything worked as expected, and the queries performed quickly.

The slowdown started after upgrading, and it only affects queries related to Duende. All our other queries to the database work fine.

[olhakhariv]

The issue was on our side.
We did not use the NodaTime plugin correctly.
Here is the working version:

.AddOperationalStore(options =>
    options.ConfigureDbContext = b => b.UseNpgsql(connectionString,
        opt => opt.MigrationsAssembly(migrationsAssembly).UseNodaTime()))

Thank you for your help. I'm closing the ticket.