[Bug] Execution BackgroundServerProcess is still in the Failed state

[DevOnBike]

We have upgraded packages for Haangfire to these version:
HangFire Version="1.7.30"
Hangfire.Autofac Version="2.3.1"
HangFire.SqlServer Version="1.7.30"

.net 5 and app in docker

and now we see in logs these errors:
xecution BackgroundServerProcess is still in the Failed state for 1.10:39:04.7995672 due to an exception, will be retried no more than in 00:00:15

System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection) at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource1 retry, DbConnectionOptions userOptions)
at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource1 retry) at System.Data.SqlClient.SqlConnection.Open() at Hangfire.SqlServer.SqlServerStorage.CreateAndOpenConnection() at Hangfire.SqlServer.SqlServerStorage.UseConnection[T](DbConnection dedicatedConnection, Func2 func)
at Hangfire.SqlServer.SqlServerStorage.UseConnection(DbConnection dedicatedConnection, Action1 action) at Hangfire.Server.BackgroundServerProcess.CreateServer(BackgroundServerContext context) at Hangfire.Server.BackgroundServerProcess.Execute(Guid executionId, BackgroundExecution execution, CancellationToken stoppingToken, CancellationToken stoppedToken, CancellationToken shutdownToken) at Hangfire.Server.BackgroundProcessingServer.RunServer(Guid executionId, Object state) at Hangfire.Processing.BackgroundExecution.Run(Action2 callback, Object state)

[dehghanfar]

same issue

[odinserj]

Please try installing the latest version of Microsoft.Data.SqlClient package (latest stable version at the moment is 4.1.0) and use the following connection factory in your Hangfire configuration logic in order to use SqlConnection class from the newly installed package.

.UseSqlServerStorage(
    () => new Microsoft.Data.SqlClient.SqlConnection(connectionString));

Please let me know if it solves your issue and I will make changes to use this package by default in the next version.

[DevOnBike]

Please try installing the latest version of Microsoft.Data.SqlClient package (latest stable version at the moment is 4.1.0) and use the following connection factory in your Hangfire configuration logic in order to use SqlConnection class from the newly installed package.

.UseSqlServerStorage(
    () => new Microsoft.Data.SqlClient.SqlConnection(connectionString));

Please let me know if it solves your issue and I will make changes to use this package by default in the next version.

This change of code caused my developer connection to DB broke with some weird exception regarding secure connection, but I solved it by setting new flag in connection string TrustServerCertificate=True.

I can give u feedback in few days after I deploy this code to test environment.
So stay tuned :)

[odinserj]

Oh, just found this issue with 4.X versions – dotnet/SqlClient#1402 – looks like problems with encryption are planned to be fixed in 5.0.

[DevOnBike]

So new code is working on test environment for 2 days and it looks very promising - no memory leaks, low memory consuption, no errors for now. I will let this fixed code work on k8s for at least week to be sure that it is really fixed.

Next update on wednesday. So stay tuned.

[odinserj]

Oh my friend, if you knew how long I've been chasing the cause of this problem, blaming Hangfire. Don't understand how System.Data.SqlClient library began to work so poorly after years of stable work, may be it's related to SQL Azure internals. Thanks a lot for the update, will wait for your update on Wednesday.

[wjeremies]

Same issue. Any updates on it?

[DevOnBike]

Oh my friend, if you knew how long I've been chasing the cause of this problem, blaming Hangfire. Don't understand how System.Data.SqlClient library began to work so poorly after years of stable work, may be it's related to SQL Azure internals. Thanks a lot for the update, will wait for your update on Wednesday.

It looks that with given code everything is working fine on test environment. This month probably it will go to production for final test. If there will be issues I will tell you for sure.

[dehghanfar]

just deployed it to production with over 3m hits daily. will report back. hangfire with azure sql on k8.

[odinserj]

Thanks for your feedback @DevOnBike and @dehghanfar!

[dehghanfar]

I am happy to report with high level of confidence that the issue is fixed.

[odinserj]

Thanks a lot for letting me know! Unfortunately I'm not able to make Microsoft.Data.SqlClient to be the default option in 1.7.X, because of breaking changes related to authentication in its version 4.0 that requires changes to the connection string. Unfortunately 5.0 doesn't fix anything, so I'm afraid that MD.SqlClient will be the default option only in Hangfire 1.8.