Exception "The operation was canceled."

[manelataresredarbor]

Which version of Duende IdentityServer are you using?

7.08

Which version of .NET are you using?

Core 8

Describe the bug

From time to time we get the exception regarding the access to the PersistedGrandStore

To Reproduce

When we have a high load of traffic of users logins

Log output/exception with stacktrace

The operation was canceled.

at System.Threading.CancellationToken.ThrowOperationCanceledException() at MySqlConnector.Core.CommandExecutor.ExecuteReaderAsync(CommandListPosition commandListPosition, ICommandPayloadCreator payloadCreator, CommandBehavior behavior, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/CommandExecutor.cs:line 15 at MySqlConnector.MySqlCommand.ExecuteNonQueryAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlCommand.cs:line 309 at MySqlConnector.MySqlConnection.BeginTransactionAsync(IsolationLevel isolationLevel, Nullable1 isReadOnly, IOBehavior ioBehavior, CancellationToken cancellationToken) in //src/MySqlConnector/MySqlConnection.cs:line 163 at MySqlConnector.MySqlConnection.BeginDbTransactionAsync(IsolationLevel isolationLevel, CancellationToken cancellationToken) in //src/MySqlConnector/MySqlConnection.cs:line 108 at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.BeginTransactionAsync(IsolationLevel isolationLevel, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.BeginTransactionAsync(CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList1 entriesToSave, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.<>c__DisplayClass30_02.<<ExecuteAsync>b__0>d.MoveNext() --- End of stack trace from previous location --- at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func4 operation, Func4 verifySucceeded, TState state, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func4 operation, Func4 verifySucceeded, TState state, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func4 operation, Func4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Duende.IdentityServer.EntityFramework.Stores.PersistedGrantStore.StoreAsync(PersistedGrant token) in /_/src/EntityFramework.Storage/Stores/PersistedGrantStore.cs:line 78 at Duende.IdentityServer.Stores.DefaultGrantStore1.StoreItemByHashedKeyAsync(String hashedKey, T item, String clientId, String subjectId, String sessionId, String description, DateTime created, Nullable1 expiration, Nullable1 consumedTime) in /_/src/IdentityServer/Stores/Default/DefaultGrantStore.cs:line 231 at Duende.IdentityServer.Stores.DefaultGrantStore1.CreateItemAsync(T item, String clientId, String subjectId, String sessionId, String description, DateTime created, Int32 lifetime) in /_/src/IdentityServer/Stores/Default/DefaultGrantStore.cs:line 177 at Duende.IdentityServer.ResponseHandling.AuthorizeResponseGenerator.CreateCodeFlowResponseAsync(ValidatedAuthorizeRequest request) in /_/src/IdentityServer/ResponseHandling/Default/AuthorizeResponseGenerator.cs:line 143 at Duende.IdentityServer.ResponseHandling.AuthorizeResponseGenerator.CreateResponseAsync(ValidatedAuthorizeRequest request) in /_/src/IdentityServer/ResponseHandling/Default/AuthorizeResponseGenerator.cs:line 100 at Duende.IdentityServer.Endpoints.AuthorizeEndpointBase.ProcessAuthorizeRequestAsync(NameValueCollection parameters, ClaimsPrincipal user, Boolean checkConsentResponse) in /_/src/IdentityServer/Endpoints/AuthorizeEndpointBase.cs:line 148 at Duende.IdentityServer.Endpoints.AuthorizeEndpointBase.ProcessAuthorizeRequestAsync(NameValueCollection parameters, ClaimsPrincipal user, Boolean checkConsentResponse) in /_/src/IdentityServer/Endpoints/AuthorizeEndpointBase.cs:line 160 at Duende.IdentityServer.Endpoints.AuthorizeCallbackEndpoint.ProcessAsync(HttpContext context) in /_/src/IdentityServer/Endpoints/AuthorizeCallbackEndpoint.cs:line 51 at Duende.IdentityServer.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IdentityServerOptions options, IEndpointRouter router, IUserSession userSession, IEventService events, IIssuerNameService issuerNameService, ISessionCoordinationService sessionCoordinationService) in /_/src/IdentityServer/Hosting/IdentityServerMiddleware.cs:line 106

Additional context

We are using EF with MySQL
connString: DefaultCommandTimeout=300;Connection Timeout=100;SslMode=none;Max Pool Size=200;Min Pool Size=10

[maartenba]

This looks like potentially the command is timing out on the database, or the connection pool is exhausted. Do you have any application logs from just before the exception to see if there is useful info in there? Does this correlate with high load on your database?