Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dynamic-datasource如果之前数据库连接错误,无法关闭,直至达到最大重试次数。 #19

Open
coreyho opened this issue Feb 27, 2019 · 5 comments

Comments

@coreyho
Copy link

coreyho commented Feb 27, 2019

如果出现数据库连接错误,比如url写错了,会实例化一个HikariDataSource(null),在重新配置新数据库设置后,将关闭上一个连接,进入terminateHikariDataSource()方法,执行poolMXBean.softEvictConnections()时抛会异常,致使不断循环直到达到最大重试次数,在terminate()方法中的if (dataSource instanceof HikariDataSource )添加一个非空检查可以避免不断重试。

@coreyho coreyho changed the title dynamic-datasource如果之前数据库连接错误,无法关闭。 dynamic-datasource如果之前数据库连接错误,无法关闭,直至达到最大重试次数。 Feb 27, 2019
@nobodyiam
Copy link
Member

我理解你的意思,不过我本地貌似无法复现这个场景,你能提供一个可以复现的步骤吗?

  1. 如果启动时url就是错的,那启动就失败了
  2. 如果启动时是正常的,后续改错了,那么新的连接也不会成功,后续也不会terminate

@coreyho
Copy link
Author

coreyho commented Mar 3, 2019

我的程序中ORM用的是Mybatis,在Apollo中配置错误的url,启动程序,会在建立mysql连接时报错,生成空的datasource,Mybatis生成空的sqlSessionFactory和sqlSessionTemplate,但程序不会停止,然后在Apollo中更改正确的配置,就出现了上述场景。

我试了下你提供的实例,启动错误的url,建立mysql连接时报错,会在hibernate处报错并退出。

@nobodyiam
Copy link
Member

这个应该和mybatis没啥关系,和datasource的创建逻辑有关,如果连接失败的话,datasource应该创建失败,可以看下你这边空的datasource是怎么出现的?

@soulmz
Copy link

soulmz commented Aug 11, 2020

@nobodyiam @coreyho
我今天碰到了类似的问题。

日志如下

2020-08-11 14:48:09,985  INFO 33838 --- [)-192.168.10.11] com.zaxxer.hikari.HikariDataSource      [110] : HikariPool-1 - Starting...
2020-08-11 14:48:11,333 ERROR 33838 --- [)-192.168.10.11] com.zaxxer.hikari.pool.HikariPool       [593] : HikariPool-1 - Exception during pool initialization.

java.sql.SQLException: Access denied for user 'root'@'192.168.20.122' (using password: YES)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3976)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3912)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:871)
	at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1714)
	at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1224)
	at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2190)
	at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2221)
	at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2016)
	at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:776)
	at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
	at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:386)
	at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138)
	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:358)
	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206)
	at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:477)
	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:560)
	at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115)
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112)
	at com.example.config.apollo.datasource.DynamicDataSource.getConnection(DynamicDataSource.java:31)
	at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:158)
	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:116)
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79)
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:324)
	at org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator.getProduct(DataSourceHealthIndicator.java:122)
	at org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator.doDataSourceHealthCheck(DataSourceHealthIndicator.java:105)
	at org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator.doHealthCheck(DataSourceHealthIndicator.java:100)
	at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:82)
	at org.springframework.boot.actuate.health.HealthIndicator.getHealth(HealthIndicator.java:37)
	at org.springframework.boot.actuate.health.HealthEndpoint.getHealth(HealthEndpoint.java:71)
	at org.springframework.boot.actuate.health.HealthEndpoint.getHealth(HealthEndpoint.java:39)
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:99)
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getAggregateHealth(HealthEndpointSupport.java:110)
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:96)
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:74)
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:61)
	at org.springframework.boot.actuate.health.HealthEndpoint.health(HealthEndpoint.java:65)
	at org.springframework.boot.actuate.health.HealthEndpoint.health(HealthEndpoint.java:55)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:282)
	at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:77)
	at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:60)
	at org.springframework.boot.actuate.endpoint.jmx.EndpointMBean.invoke(EndpointMBean.java:121)
	at org.springframework.boot.actuate.endpoint.jmx.EndpointMBean.invoke(EndpointMBean.java:96)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
	at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
	at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
	at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
	at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
	at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)
	at sun.reflect.GeneratedMethodAccessor197.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
	at sun.rmi.transport.Transport$1.run(Transport.java:200)
	at sun.rmi.transport.Transport$1.run(Transport.java:197)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

2020-08-11 14:48:11,338  WARN 33838 --- [)-192.168.10.11] o.s.b.a.jdbc.DataSourceHealthIndicator  [87] : DataSource health check failed

org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLException: Access denied for user 'root'@'192.168.20.122' (using password: YES)
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:82)
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:324)
	at org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator.getProduct(DataSourceHealthIndicator.java:122)
	at org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator.doDataSourceHealthCheck(DataSourceHealthIndicator.java:105)
	at org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator.doHealthCheck(DataSourceHealthIndicator.java:100)
	at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:82)
	at org.springframework.boot.actuate.health.HealthIndicator.getHealth(HealthIndicator.java:37)
	at org.springframework.boot.actuate.health.HealthEndpoint.getHealth(HealthEndpoint.java:71)
	at org.springframework.boot.actuate.health.HealthEndpoint.getHealth(HealthEndpoint.java:39)
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:99)
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getAggregateHealth(HealthEndpointSupport.java:110)
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:96)
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:74)
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:61)
	at org.springframework.boot.actuate.health.HealthEndpoint.health(HealthEndpoint.java:65)
	at org.springframework.boot.actuate.health.HealthEndpoint.health(HealthEndpoint.java:55)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:282)
	at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:77)
	at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:60)
	at org.springframework.boot.actuate.endpoint.jmx.EndpointMBean.invoke(EndpointMBean.java:121)
	at org.springframework.boot.actuate.endpoint.jmx.EndpointMBean.invoke(EndpointMBean.java:96)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
	at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
	at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
	at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
	at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
	at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)
	at sun.reflect.GeneratedMethodAccessor197.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
	at sun.rmi.transport.Transport$1.run(Transport.java:200)
	at sun.rmi.transport.Transport$1.run(Transport.java:197)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: Access denied for user 'root'@'192.168.20.122' (using password: YES)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3976)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3912)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:871)
	at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1714)
	at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1224)
	at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2190)
	at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2221)
	at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2016)
	at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:776)
	at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
	at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:386)
	at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138)
	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:358)
	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206)
	at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:477)
	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:560)
	at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115)
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112)
	at com.example.config.apollo.datasource.DynamicDataSource.getConnection(DynamicDataSource.java:31)
	at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:158)
	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:116)
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79)
	... 47 common frames omitted

2020-08-11 14:48:59,600  INFO 33838 --- [Apollo-Config-2] c.d.c.a.datasource.DataSourceRefresher  [53] : Refreshing data source
2020-08-11 14:48:59,887  INFO 33838 --- [Apollo-Config-2] com.zaxxer.hikari.HikariDataSource      [110] : HikariPool-2 - Starting...
2020-08-11 14:49:00,140  INFO 33838 --- [Apollo-Config-2] com.zaxxer.hikari.HikariDataSource      [123] : HikariPool-2 - Start completed.
2020-08-11 14:49:00,150  INFO 33838 --- [Apollo-Config-2] c.d.c.a.datasource.DataSourceRefresher  [62] : Finished refreshing data source
2020-08-11 14:49:00,150  INFO 33838 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [45] : Trying to terminate data source: HikariDataSource (null)
2020-08-11 14:49:00,150  WARN 33838 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [53] : Terminating data source HikariDataSource (null) failed, will retry in 5000 ms, error message: null
2020-08-11 14:49:05,155  INFO 33838 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [45] : Trying to terminate data source: HikariDataSource (null)
2020-08-11 14:49:05,156  WARN 33838 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [53] : Terminating data source HikariDataSource (null) failed, will retry in 5000 ms, error message: null
2020-08-11 14:49:10,160  INFO 33838 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [45] : Trying to terminate data source: HikariDataSource (null)
2020-08-11 14:49:10,162  WARN 33838 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [53] : Terminating data source HikariDataSource (null) failed, will retry in 5000 ms, error message: null
2020-08-11 14:49:15,165  INFO 33838 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [45] : Trying to terminate data source: HikariDataSource (null)
2020-08-11 14:49:15,165  WARN 33838 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [53] : Terminating data source HikariDataSource (null) failed, will retry in 5000 ms, error message: null
2020-08-11 14:49:20,167  INFO 33838 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [45] : Trying to terminate data source: HikariDataSource (null)
2020-08-11 14:49:20,167  WARN 33838 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [53] : Terminating data source HikariDataSource (null) failed, will retry in 5000 ms, error message: null
2020-08-11 14:49:25,173  INFO 33838 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [45] : Trying to terminate data source: HikariDataSource (null)
2020-08-11 14:49:25,173  WARN 33838 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [53] : Terminating data source HikariDataSource (null) failed, will retry in 5000 ms, error message: null
2020-08-11 14:49:30,179  INFO 33838 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [45] : Trying to terminate data source: HikariDataSource (null)
2020-08-11 14:49:30,179  WARN 33838 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [53] : Terminating data source HikariDataSource (null) failed, will retry in 5000 ms, error message: null
2020-08-11 14:49:35,183  INFO 33838 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [45] : Trying to terminate data source: HikariDataSource (null)
2020-08-11 14:49:35,184  WARN 33838 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [53] : Terminating data source HikariDataSource (null) failed, will retry in 5000 ms, error message: null
2020-08-11 14:49:40,188  INFO 33838 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [45] : Trying to terminate data source: HikariDataSource (null)
2020-08-11 14:49:40,188  WARN 33838 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [53] : Terminating data source HikariDataSource (null) failed, will retry in 5000 ms, error message: null
2020-08-11 14:49:45,193  INFO 33838 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [45] : Trying to terminate data source: HikariDataSource (null)
2020-08-11 14:49:45,193  WARN 33838 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [53] : Terminating data source HikariDataSource (null) failed, will retry in 5000 ms, error message: null
2020-08-11 14:49:50,198  INFO 33838 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [45] : Trying to terminate data source: HikariDataSource (null)
2020-08-11 14:49:50,199  WARN 33838 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [53] : Terminating data source HikariDataSource (null) failed, will retry in 5000 ms, error message: null
2020-08-11 14:49:55,203  INFO 33838 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [45] : Trying to terminate data source: HikariDataSource (null)
2020-08-11 14:49:55,204  WARN 33838 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [53] : Terminating data source HikariDataSource (null) failed, will retry in 5000 ms, error message: null
2020-08-11 14:50:00,209  INFO 33838 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [45] : Trying to terminate data source: HikariDataSource (null)
2020-08-11 14:50:33,853  WARN 33838 --- [l-2 housekeeper] com.zaxxer.hikari.pool.HikariPool       [787] : HikariPool-2 - Thread starvation or clock leap detected (housekeeper delta=1m3s605ms).
2020-08-11 14:50:46,661  WARN 33838 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [53] : Terminating data source HikariDataSource (null) failed, will retry in 5000 ms, error message: null

重现步骤:

一开始 apollo 配置的 账户密码是错误的。无法正常建立MySQL连接。

后面 改成正确的正确密码,后发现旧数据源 poolMXBean is null 因没正确的建立连接池。 无法关闭。一直重试。

/**
     * 优雅的终止 HikariCP线程池
     *
     * @see <a href="https://github.com/brettwooldridge/HikariCP/issues/742">Support graceful shutdown of connection
     * pool</a>
     */
    private boolean terminateHikariDataSource(HikariDataSource dataSource) {
        HikariPoolMXBean poolMXBean = dataSource.getHikariPoolMXBean();
        // 增加判断
        if (Objects.isNull(poolMXBean)) {
            dataSource.close();
            return true;
        }
        //evict idle connections
        poolMXBean.softEvictConnections();
        if (poolMXBean.getActiveConnections() > 0 && retryTimes < MAX_RETRY_TIMES) {
            logger.warn("Data source {} still has {} active connections, will retry in {} ms.", dataSource,
                    poolMXBean.getActiveConnections(), RETRY_DELAY_IN_MILLISECONDS);
            return false;
        }
        if (poolMXBean.getActiveConnections() > 0) {
            logger.warn("Retry times({}) >= {}, force closing data source {}, with {} active connections!", retryTimes,
                    MAX_RETRY_TIMES, dataSource, poolMXBean.getActiveConnections());
        }
        dataSource.close();
        return true;
    }

增加 Objects.isNull(poolMXBean) 判断,空则直接关闭。

2020-08-11 15:00:21,330  INFO 41627 --- [on(2)-127.0.0.1] com.zaxxer.hikari.HikariDataSource      [110] : HikariPool-1 - Starting...
2020-08-11 15:00:22,697 ERROR 41627 --- [on(2)-127.0.0.1] com.zaxxer.hikari.pool.HikariPool       [593] : HikariPool-1 - Exception during pool initialization.

java.sql.SQLException: Access denied for user 'root'@'192.168.20.122' (using password: YES)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3976)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3912)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:871)
	at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1714)
	at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1224)
	at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2190)
	at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2221)
	at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2016)
	at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:776)
	at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
	at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:386)
	at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138)
	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:358)
	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206)
	at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:477)
	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:560)
	at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115)
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112)
	at com.example.config.apollo.datasource.DynamicDataSource.getConnection(DynamicDataSource.java:31)
	at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:158)
	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:116)
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79)
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:324)
	at org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator.getProduct(DataSourceHealthIndicator.java:122)
	at org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator.doDataSourceHealthCheck(DataSourceHealthIndicator.java:105)
	at org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator.doHealthCheck(DataSourceHealthIndicator.java:100)
	at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:82)
	at org.springframework.boot.actuate.health.HealthIndicator.getHealth(HealthIndicator.java:37)
	at org.springframework.boot.actuate.health.HealthEndpoint.getHealth(HealthEndpoint.java:71)
	at org.springframework.boot.actuate.health.HealthEndpoint.getHealth(HealthEndpoint.java:39)
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:99)
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getAggregateHealth(HealthEndpointSupport.java:110)
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:96)
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:74)
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:61)
	at org.springframework.boot.actuate.health.HealthEndpoint.health(HealthEndpoint.java:65)
	at org.springframework.boot.actuate.health.HealthEndpoint.health(HealthEndpoint.java:55)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:282)
	at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:77)
	at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:60)
	at org.springframework.boot.actuate.endpoint.jmx.EndpointMBean.invoke(EndpointMBean.java:121)
	at org.springframework.boot.actuate.endpoint.jmx.EndpointMBean.invoke(EndpointMBean.java:96)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
	at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
	at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
	at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
	at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
	at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)
	at sun.reflect.GeneratedMethodAccessor196.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
	at sun.rmi.transport.Transport$1.run(Transport.java:200)
	at sun.rmi.transport.Transport$1.run(Transport.java:197)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

2020-08-11 15:00:22,702  WARN 41627 --- [on(2)-127.0.0.1] o.s.b.a.jdbc.DataSourceHealthIndicator  [87] : DataSource health check failed

org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLException: Access denied for user 'root'@'192.168.20.122' (using password: YES)
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:82)
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:324)
	at org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator.getProduct(DataSourceHealthIndicator.java:122)
	at org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator.doDataSourceHealthCheck(DataSourceHealthIndicator.java:105)
	at org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator.doHealthCheck(DataSourceHealthIndicator.java:100)
	at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:82)
	at org.springframework.boot.actuate.health.HealthIndicator.getHealth(HealthIndicator.java:37)
	at org.springframework.boot.actuate.health.HealthEndpoint.getHealth(HealthEndpoint.java:71)
	at org.springframework.boot.actuate.health.HealthEndpoint.getHealth(HealthEndpoint.java:39)
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:99)
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getAggregateHealth(HealthEndpointSupport.java:110)
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:96)
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:74)
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:61)
	at org.springframework.boot.actuate.health.HealthEndpoint.health(HealthEndpoint.java:65)
	at org.springframework.boot.actuate.health.HealthEndpoint.health(HealthEndpoint.java:55)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:282)
	at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:77)
	at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:60)
	at org.springframework.boot.actuate.endpoint.jmx.EndpointMBean.invoke(EndpointMBean.java:121)
	at org.springframework.boot.actuate.endpoint.jmx.EndpointMBean.invoke(EndpointMBean.java:96)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
	at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
	at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
	at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
	at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
	at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)
	at sun.reflect.GeneratedMethodAccessor196.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
	at sun.rmi.transport.Transport$1.run(Transport.java:200)
	at sun.rmi.transport.Transport$1.run(Transport.java:197)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: Access denied for user 'root'@'192.168.20.122' (using password: YES)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3976)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3912)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:871)
	at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1714)
	at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1224)
	at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2190)
	at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2221)
	at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2016)
	at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:776)
	at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
	at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:386)
	at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138)
	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:358)
	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206)
	at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:477)
	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:560)
	at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115)
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112)
	at com.example.config.apollo.datasource.DynamicDataSource.getConnection(DynamicDataSource.java:31)
	at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:158)
	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:116)
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79)
	... 47 common frames omitted

2020-08-11 15:00:31,712  INFO 41627 --- [Apollo-Config-2] c.d.c.a.datasource.DataSourceRefresher  [53] : Refreshing data source
2020-08-11 15:00:32,007  INFO 41627 --- [Apollo-Config-2] com.zaxxer.hikari.HikariDataSource      [110] : HikariPool-2 - Starting...
2020-08-11 15:00:32,277  INFO 41627 --- [Apollo-Config-2] com.zaxxer.hikari.HikariDataSource      [123] : HikariPool-2 - Start completed.
2020-08-11 15:00:32,302  INFO 41627 --- [Apollo-Config-2] c.d.c.a.datasource.DataSourceRefresher  [62] : Finished refreshing data source
2020-08-11 15:00:32,302  INFO 41627 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [46] : Trying to terminate data source: HikariDataSource (null)
2020-08-11 15:00:48,470  INFO 41627 --- [pool-5-thread-1] c.d.c.a.d.DataSourceTerminationTask     [39] : Data source HikariDataSource (null) terminated successfully!

这样正常的关闭了。

@nobodyiam
Copy link
Member

@soulmz 多谢指出,6735097

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants