When use Min Pool Size = 10 app hangs due to connection timeout

[jjavierdguezas]

Steps to reproduce

Use this project as example

MySql.Hang.Test.zip

I am using this Dockerfile:

FROM mcr.microsoft.com/dotnet/aspnet:8.0-jammy AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443

FROM mcr.microsoft.com/dotnet/sdk:8.0-jammy AS build

ARG BUILD_CONFIGURATION=Release

RUN apt update 

COPY . /source

WORKDIR /source/MySql.Hang.Test.Api

RUN dotnet restore

RUN dotnet build "MySql.Hang.Test.Api.csproj" -c $BUILD_CONFIGURATION -o /app/build

FROM build AS publish
ARG BUILD_CONFIGURATION=Release
RUN dotnet publish "MySql.Hang.Test.Api.csproj" -c $BUILD_CONFIGURATION -o /app/publish /p:UseAppHost=false

FROM base AS final

RUN ln -s /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libdl.so
RUN apt-get update && apt-get upgrade -y
RUN apt-get -y install libgdiplus libc6-dev fontconfig libfreetype6 libjpeg-turbo8 libpng16-16 libx11-6 libxcb1 libxext6 libxrender1 xfonts-75dpi xfonts-base xvfb libfontconfig libfontconfig1 libfontconfig1-dev nfs-common
RUN ln -s /usr/lib/libgdiplus.so /lib/x86_64-linux-gnu/libgdiplus.so


WORKDIR /app
COPY --from=publish /app/publish .

ENTRYPOINT ["dotnet", "MySql.Hang.Test.Api.dll"]

and this docker-compose.yml:

version: "3.3"
name: "test-app"

services:

  test-api:
    image: test-api:latest
    container_name: "test-api"
    depends_on:
      - mysql
    ports:
      - "80:80"
      - "443:443"
    environment:
      ASPNETCORE_HTTP_PORTS: 80
      # ConnectionStrings__DefaultConnection: server=mysql,3306;user=root;password=p@ssw0rd.;database=mysql_hang_db;;
      ConnectionStrings__DefaultConnection: server=mysql,3306;user=root;password=p@ssw0rd.;database=mysql_hang_db;Min Pool Size=10
      Logging__LogLevel__Microsoft.EntityFrameworkCore: Debug
      Logging__LogLevel__Microsoft.EntityFrameworkCore.Database.Connection: Debug
      Logging__LogLevel__Microsoft.EntityFrameworkCore.Database.Command: Debug
      Logging__LogLevel__Default: Debug
      Logging__LogLevel__Microsoft.AspNetCore: Debug

  mysql:
    image: mysql:8.0.35
    container_name: "test-db"
    restart: always
    ports:
      - "3306:3306"
    environment:
      MYSQL_ROOT_PASSWORD: p@ssw0rd.
      MYSQL_DATABASE: mysql_hang_db

The issue

On ubuntu 22.04 If I use a connection string like:

"ConnectionStrings": {
    "DefaultConnection": "server=127.0.0.1,3306;user=root;password=p@ssw0rd.;database=mysql_hang_test"
  }

it works but if I include Min Pool Size=10

"ConnectionStrings": {
    "DefaultConnection": "server=127.0.0.1,3306;user=root;password=p@ssw0rd.;database=mysql_hang_test;Min Pool Size=10"
  }

it doesnt. A timeout error is thrown

test-api  | fail: Microsoft.EntityFrameworkCore.Database.Connection[20004]
test-api  |       An error occurred using the connection to database 'mysql_hang_db' on server 'mysql,3306'.
test-api  | Unhandled exception. MySqlConnector.MySqlException (0x80004005): Connect Timeout expired.
test-api  |  ---> MySqlConnector.MySqlException (0x80004005): Connect Timeout expired.
test-api  |    at MySqlConnector.Core.ServerSession.OpenTcpSocketAsync(ConnectionSettings cs, ILoadBalancer loadBalancer, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 1063
test-api  |    at MySqlConnector.Core.ServerSession.ConnectAsync(ConnectionSettings cs, MySqlConnection connection, Int64 startingTimestamp, ILoadBalancer loadBalancer, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 422
test-api  |    at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, Action`4 logMessage, Int64 startingTimestamp, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 427
test-api  |    at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, Action`4 logMessage, Int64 startingTimestamp, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 432
test-api  |    at MySqlConnector.Core.ConnectionPool.CreateMinimumPooledSessions(MySqlConnection connection, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 405
test-api  |    at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int64 startingTimestamp, Int32 timeoutMilliseconds, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 33
test-api  |    at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int64 startingTimestamp, Activity activity, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 1056
test-api  |    at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int64 startingTimestamp, Activity activity, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 1090
test-api  |    at MySqlConnector.MySqlConnection.OpenAsync(Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 556
test-api  |    at MySqlConnector.MySqlConnection.Open() in /_/src/MySqlConnector/MySqlConnection.cs:line 518
test-api  |    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenDbConnection(Boolean errorsExpected)
test-api  |    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternal(Boolean errorsExpected)
test-api  |    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean errorsExpected)
test-api  |    at Pomelo.EntityFrameworkCore.MySql.Storage.Internal.MySqlRelationalConnection.Open(Boolean errorsExpected)
test-api  |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteScalar(RelationalCommandParameterObject parameterObject)
test-api  |    at Microsoft.EntityFrameworkCore.Migrations.HistoryRepository.Exists()
test-api  |    at Microsoft.EntityFrameworkCore.Migrations.Internal.Migrator.Migrate(String targetMigration)
test-api  |    at Microsoft.EntityFrameworkCore.RelationalDatabaseFacadeExtensions.Migrate(DatabaseFacade databaseFacade)
test-api  |    at MySql.Hang.Test.Data.AppDbContext..ctor(DbContextOptions`1 options) in /source/MySql.Hang.Test.Data/AppDbContext.cs:line 18
test-api  |    at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
test-api  |    at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
test-api  |    at System.Reflection.MethodBaseInvoker.InvokeWithOneArg(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
test-api  |    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
test-api  |    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
test-api  |    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite callSite, RuntimeResolverContext context)
test-api  |    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
test-api  |    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
test-api  |    at Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine.<>c__DisplayClass2_0.<RealizeService>b__0(ServiceProviderEngineScope scope)
test-api  |    at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(ServiceIdentifier serviceIdentifier, ServiceProviderEngineScope serviceProviderEngineScope)
test-api  |    at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
test-api  |    at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
test-api  |    at MySql.Hang.Test.Api.Program.InitializeDb(IServiceProvider provider) in /source/MySql.Hang.Test.Api/Program.cs:line 85
test-api  |    at MySql.Hang.Test.Api.Program.Main(String[] args) in /source/MySql.Hang.Test.Api/Program.cs:line 73
test-api  |    at MySql.Hang.Test.Api.Program.<Main>(String[] args)

Further technical details

MySQL version: mysql:8.0.35
Operating system: Ubuntu 22.04.2 LTS
Pomelo.EntityFrameworkCore.MySql version: 8.0.2
Microsoft.AspNetCore.App version: 8.0.5

Other details about my project setup:

I reduced the test scenario to an aspnet project (see attached .zip file) with a method that inserts data into the db when starting the app and a mysql all in a docker container

// ...
            await InitializeDb(((IApplicationBuilder)app).ApplicationServices);

            app.Run();
        }

        private static async Task InitializeDb(IServiceProvider provider)
        {
            using var scope = provider.GetService<IServiceScopeFactory>().CreateScope();
            var serviceProvider = scope.ServiceProvider;

            var dbContextFactory = serviceProvider.GetService<IDbContextFactory<AppDbContext>>();
            using var db = await dbContextFactory.CreateDbContextAsync();
            //using var db = serviceProvider.GetService<AppDbContext>();
            //await db.Database.MigrateAsync();


            var userInfos = new List<UserInfo>(1000);

            for (int i = 0; i < 1000; i++)
            {
                var name = $"User {i:000}";

                if (!await db.Set<UserInfo>().AnyAsync(x => x.Name == name))
                {
                    userInfos.Add(new UserInfo() { Id = Guid.NewGuid(), Name = name, CreatedOn = DateTime.Now });
                }
            }

            if (userInfos.Count > 0)
            {
                await db.AddRangeAsync(userInfos);
                await db.SaveChangesAsync();
            }

        }
    }

--
PS: I dont really know if this is related to Pomelo, mysql o what.. so any help is welcome. I test this exact docker compose on windows and it works I run the attached project from visual studio on windows and a mysql on docker desktop and it works. A coworker tested it on windows wsl and it works too. it seems related to linux/ubuntu

[mcuervoe]

I have the same issue. It looks like it is related to where the app is running.

• In Windows 11 Pro + WSL Ubuntu 22.04 + docker desktop it works without problems
• In Windows 11 Pro + VM Ubuntu 22.04 + docker, it works without problems
• In bare metal Ubuntu 22.04 + docker, it fails
• In bare metal Ubuntu 22.04, running locally without docker, it fails
• In AWS EC2 Ubuntu 22.04 or Amazon Linux 2023 + docker, it fails
• In AWS ECS Fargate it fails

The same behavior is observed with MySQL 5.7, MySQL 8, Aurora DB, and MariaDB

I also tried with MYSQL running directly on the operating system, on a docker image, AWS RDS, and I got the same behavior

This issue is blocking the adoption of the .NET solution in our project

[rachael-ross]

Has this been resolved or a solution found?

[jjavierdguezas]

Has this been resolved or a solution found?

Sadly, no. We switched to PostgreSQL 🤷🏻‍♂️

[lauxjpn]

@jjavierdguezas I am unable to reproduce the issue with your provided code. I used your zip file (with its dockerfile) and your provided compose file from this post, and the app executs InitializeDb() without issues and then continues on with app.Run().

I would suspect a user error here. Maybe the max_connections of the database server were somehow exhausted and the database docker service was in all cases only restarted (and thereby remained in that state), instead of recreated.

Since you moved to a different database server, I suspect you don't intend to follow-up on this issue. If you do, however, then please recreate your test images, ensure that it is indeed the docker service database server that the asp.net core app connects to (and not another database server that uses 3306 on the host machine) and recheck that you are still able to reproduce the issue.

(Since you provided all files and are using docker and pinned the relevant versions, the issue should be reproducible for us, though it is not.)

---

@mcuervoe Feel free to drop us a minimal reproducible example here, if you want us to investigate further.

---

@rachael-ross Do you experience similar issues?

[jjavierdguezas]

Hi @lauxjpn,

Thank you for your response. I appreciate your efforts in trying to reproduce the issue.

Since we have moved to PostgreSQL, this issue is not a priority for me at the moment, so it might take some time for me to follow up. I will look into it again and try to provide an update soon.

Thanks again for your help!