[Dubbo-6567] fix consumer warning "safe guard client , should not be called ,must have a bug." #6959

wikiwikiwiki · 2020-11-30T08:00:16Z

issue #6567 里面提到了这个问题，刚好也碰到了，修复一下。

具体触发条件为：

启动一个普通consumer和一个provider，consumer可以正常调用。
关闭provider进程，观察注册中心，一直等到provider注册的url消失。
重启provider进程，consumer再次调用，报"safe guard client , should not be called ,must have a bug."并且之后每5000次调用会触发一次。

这个问题的根源是LazyConnectExchangeClient在两种情况下被使用，这两种情况需要表现的行为不一样，但是代码一样导致。
这两种情况一个是consumer主动设置为延迟连接，一个是consumer和provider断开连接时进行防御式编程，如果在连接关闭后还有请求过来就用这个延迟初始化的client处理一下。

具体到上面的这个例子：
第一步consumer正常调用时，DubboProtocol里面referenceClientMap字段保存了consumer连接provider的ReferenceCountExchangeClient。
第二步关闭provider进程一直到provider注册url消失时，referenceClientMap字段里面的引用还在，但是ReferenceCountExchangeClient的replaceWithLazyClient方法被调用，内部的ExchangeClient换成了LazyConnectExchangeClient。
第三步provider进程重启，consumer再次调用时，会重新获取到这个已经关闭的ReferenceCountExchangeClient。判断代码为

referenceCountExchangeClient == null || referenceCountExchangeClient.isClosed()

而referenceCountExchangeClient.isClosed()会调用到LazyConnectExchangeClient的isClosed方法

if (client != null) {
    return client.isClosed();
} else {
    return false;
}

此时还没有延迟初始化，会直接返回false，导致后续的调用都会使用这个client，每5000报告一次warning。

这个bug应该是在 #5532 这个pr被合并后触发，原本LazyConnectExchangeClient在没有client时isClosed()方法会返回true，会在上面例子第三步时给清理掉。而在没有client时，isClosed()方法返回false，应该是针对consumer主动设置为延迟连接的情况；而consumer和provider断开连接时建立的那个LazyConnectExchangeClient的isClosed()方法应该始终返回true。

具体修复方式为模仿replaceWithLazyClient方法里的方式，在场景不同时添加参数。
通过增加一个LAZY_CLOSE_STATE_KEY，强制设置isClosed方法修改的返回值，isClosed()方法内容改为

        if (closeState != null) {
            return closeState;
        }

        if (client == null) {
            return false;
        } else {
            return client.isClosed();
        }

…must have a bug."

…tion should always be true

wow-blacksheep · 2020-12-03T05:04:21Z

你好，请问我在不升级dubbo的情况下，有没有好的方式解决这个问题

wikiwikiwiki · 2020-12-04T02:37:22Z

你好，请问我在不升级dubbo的情况下，有没有好的方式解决这个问题

好的方式只有改代码发一遍替换jar包

AlbumenJ · 2021-04-12T08:29:25Z

@wikiwikiwiki Hi, thanks for your contribution.

Please merge the latest master branch to resolve confilcts.

# Conflicts: # dubbo-rpc/dubbo-rpc-dubbo/src/main/java/org/apache/dubbo/rpc/protocol/dubbo/LazyConnectExchangeClient.java # dubbo-rpc/dubbo-rpc-dubbo/src/main/java/org/apache/dubbo/rpc/protocol/dubbo/ReferenceCountExchangeClient.java

…ust have a bug."

wikiwikiwiki · 2021-04-28T05:57:51Z

@wikiwikiwiki Hi, thanks for your contribution.

Please merge the latest master branch to resolve confilcts.

already resolved

AlbumenJ · 2021-04-30T03:03:17Z

pls fix ut

xiaoheng1 · 2021-05-23T07:47:26Z

relation to #7737.

AlbumenJ · 2021-05-23T13:00:50Z

Related with @7737

Please feel free to reopen if you have any question.

startjava · 2021-06-01T05:30:37Z

@AlbumenJ 什么时候放出FIX这个BUG的新版本？

startjava · 2021-06-01T05:31:22Z

另外顺便问一下，现在阿里官方是不是没有人维护DUBBO了，感觉解决问题的都是热心网友，官方不再参与了，是一个弃儿。

AlbumenJ · 2021-06-03T09:15:07Z

@AlbumenJ 什么时候放出FIX这个BUG的新版本？

本月会发 2.7.12 版本

sh1yu · 2021-06-16T07:54:29Z

@AlbumenJ 什么时候放出FIX这个BUG的新版本？

本月会发 2.7.12 版本

2.7.12版本已经发布，但是release log上并无该问题的修复说明。请问该问题已经修复了吗？

AlbumenJ · 2021-06-21T02:14:23Z

@AlbumenJ 什么时候放出FIX这个BUG的新版本？

本月会发 2.7.12 版本

2.7.12版本已经发布，但是release log上并无该问题的修复说明。请问该问题已经修复了吗？

masquee · 2021-07-08T02:24:20Z

safe guard client , should not be called ,must have a bug.

这个英文有点水平...

pirateskipper · 2021-08-24T01:33:46Z

...o-rpc-dubbo/src/main/java/org/apache/dubbo/rpc/protocol/dubbo/LazyConnectExchangeClient.java

    private volatile ExchangeClient client;
    private AtomicLong warningcount = new AtomicLong(0);

    public LazyConnectExchangeClient(URL url, ExchangeHandler requestHandler) {
        // lazy connect, need set send.reconnect = true, to avoid channel bad status.
        this.url = url.addParameter(SEND_RECONNECT_KEY, Boolean.TRUE.toString());
        this.requestHandler = requestHandler;
-        this.initialState = url.getParameter(LAZY_CONNECT_INITIAL_STATE_KEY, DEFAULT_LAZY_CONNECT_INITIAL_STATE);
+        this.initialConnectState = url.getParameter(LAZY_CONNECT_INITIAL_STATE_KEY, DEFAULT_LAZY_CONNECT_INITIAL_STATE);
+        if (url.hasParameter(LAZY_CLOSE_STATE_KEY)) {


this.closeState = null; if (url.hasParameter(LAZY_CLOSE_STATE_KEY)) { this.closeState = Boolean.parseBoolean(url.getParameter(LAZY_CLOSE_STATE_KEY)); }

pirateskipper · 2021-08-24T01:35:09Z

...o-rpc-dubbo/src/main/java/org/apache/dubbo/rpc/protocol/dubbo/LazyConnectExchangeClient.java

+            return closeState;
+        }
+
+        if (client == null) {


return client == null ? false : client.isClosed();

YsoulM · 2021-09-07T06:49:31Z

是不是在3.0.*版本中依旧存在这个问题的？3.0.*中没有找到这次修复提交的代码T_T

coder-michaelwu · 2021-10-04T12:37:39Z

今天在dubbo 2.7.8， spring-cloud-starter-dubbo:2.2.6.RELEASE 也遇见这个问题，不过貌似除了打印出该日志，并没有其他影响，仅在此评论，留作记录。

2021-10-04 20:25:49.943  WARN 46224 --- [nio-8888-exec-8] o.a.d.r.p.d.LazyConnectExchangeClient    : safe guard client , should not be called ,must have a bug.

java.lang.IllegalStateException: safe guard client , should not be called ,must have a bug.
	at org.apache.dubbo.rpc.protocol.dubbo.LazyConnectExchangeClient.warning(LazyConnectExchangeClient.java:138) [dubbo-2.7.8.jar:2.7.8]
	at org.apache.dubbo.rpc.protocol.dubbo.LazyConnectExchangeClient.request(LazyConnectExchangeClient.java:127) [dubbo-2.7.8.jar:2.7.8]
	at org.apache.dubbo.rpc.protocol.dubbo.ReferenceCountExchangeClient.request(ReferenceCountExchangeClient.java:91) [dubbo-2.7.8.jar:2.7.8]
	at org.apache.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:105) [dubbo-2.7.8.jar:2.7.8]
	at org.apache.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:163) [dubbo-2.7.8.jar:2.7.8]
	at org.apache.dubbo.rpc.protocol.AsyncToSyncInvoker.invoke(AsyncToSyncInvoker.java:52) [dubbo-2.7.8.jar:2.7.8]
	at org.apache.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:78) [dubbo-2.7.8.jar:2.7.8]
	at org.apache.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:89) [dubbo-2.7.8.jar:2.7.8]
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:83) [dubbo-2.7.8.jar:2.7.8]
	at org.apache.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:51) [dubbo-2.7.8.jar:2.7.8]
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:83) [dubbo-2.7.8.jar:2.7.8]
	at org.apache.dubbo.rpc.filter.ConsumerContextFilter.invoke(ConsumerContextFilter.java:69) [dubbo-2.7.8.jar:2.7.8]
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:83) [dubbo-2.7.8.jar:2.7.8]
	at org.apache.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:56) [dubbo-2.7.8.jar:2.7.8]
	at org.apache.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:82) [dubbo-2.7.8.jar:2.7.8]
	at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:260) [dubbo-2.7.8.jar:2.7.8]
	at org.apache.dubbo.rpc.cluster.interceptor.ClusterInterceptor.intercept(ClusterInterceptor.java:47) [dubbo-2.7.8.jar:2.7.8]
	at org.apache.dubbo.rpc.cluster.support.wrapper.AbstractCluster$InterceptorInvokerNode.invoke(AbstractCluster.java:92) [dubbo-2.7.8.jar:2.7.8]
	at org.apache.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:88) [dubbo-2.7.8.jar:2.7.8]
	at org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:74) [dubbo-2.7.8.jar:2.7.8]

	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_231]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_231]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_231]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_231]
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) [spring-web-5.2.15.RELEASE.jar:5.2.15.RELEASE]
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) [spring-web-5.2.15.RELEASE.jar:5.2.15.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105) [spring-webmvc-5.2.15.RELEASE.jar:5.2.15.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878) [spring-webmvc-5.2.15.RELEASE.jar:5.2.15.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792) [spring-webmvc-5.2.15.RELEASE.jar:5.2.15.RELEASE]
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) [spring-webmvc-5.2.15.RELEASE.jar:5.2.15.RELEASE]
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) [spring-webmvc-5.2.15.RELEASE.jar:5.2.15.RELEASE]
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) [spring-webmvc-5.2.15.RELEASE.jar:5.2.15.RELEASE]
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) [spring-webmvc-5.2.15.RELEASE.jar:5.2.15.RELEASE]
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) [spring-webmvc-5.2.15.RELEASE.jar:5.2.15.RELEASE]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:626) [tomcat-embed-core-9.0.46.jar:4.0.FR]
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) [spring-webmvc-5.2.15.RELEASE.jar:5.2.15.RELEASE]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:733) [tomcat-embed-core-9.0.46.jar:4.0.FR]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) [tomcat-embed-core-9.0.46.jar:9.0.46]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.46.jar:9.0.46]
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) [tomcat-embed-websocket-9.0.46.jar:9.0.46]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.46.jar:9.0.46]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.46.jar:9.0.46]
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) [spring-web-5.2.15.RELEASE.jar:5.2.15.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.2.15.RELEASE.jar:5.2.15.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.46.jar:9.0.46]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.46.jar:9.0.46]
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) [spring-web-5.2.15.RELEASE.jar:5.2.15.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.2.15.RELEASE.jar:5.2.15.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.46.jar:9.0.46]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.46.jar:9.0.46]
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) [spring-web-5.2.15.RELEASE.jar:5.2.15.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.2.15.RELEASE.jar:5.2.15.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.46.jar:9.0.46]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.46.jar:9.0.46]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) [tomcat-embed-core-9.0.46.jar:9.0.46]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) [tomcat-embed-core-9.0.46.jar:9.0.46]
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542) [tomcat-embed-core-9.0.46.jar:9.0.46]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) [tomcat-embed-core-9.0.46.jar:9.0.46]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.46.jar:9.0.46]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) [tomcat-embed-core-9.0.46.jar:9.0.46]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357) [tomcat-embed-core-9.0.46.jar:9.0.46]
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374) [tomcat-embed-core-9.0.46.jar:9.0.46]
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.46.jar:9.0.46]
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893) [tomcat-embed-core-9.0.46.jar:9.0.46]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1707) [tomcat-embed-core-9.0.46.jar:9.0.46]
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.46.jar:9.0.46]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_231]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_231]
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.46.jar:9.0.46]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_231]

CrazyHZM · 2021-10-09T14:40:07Z

Try it with the latest version.

eoc2015 · 2022-10-13T01:14:34Z

2.7.8 DubboX中复现。。。想知道2.7.8dubboX直接升到2.7.18有什么改动和不兼容吗

AlbumenJ · 2022-10-13T02:46:07Z

2.7.8 DubboX中复现。。。想知道2.7.8dubboX直接升到2.7.18有什么改动和不兼容吗

dubbox 版本是基于 apache/dubbo 2.5.x 的版本，从 dubbox 升级到 dubbo 2.7 参考 dubbo 2.7 升级的文档
https://dubbo.apache.org/zh/docsv2.7/user/versions/version-270/

echojun186 · 2022-12-15T02:13:52Z

mark

OnlyTL · 2023-02-22T09:40:50Z

一定有bug

apache#6959 apache#7737

fix bug: consumer warning "safe guard client , should not be called ,…

ef26026

…must have a bug."

wikiwikiwiki mentioned this pull request Nov 30, 2020

dubbo2.7.6抛出异常must have a bug #6567

Closed

the return value of isClosed() of the lazy client for defensive opera…

92761d9

…tion should always be true

AlbumenJ added the status/waiting-for-feedback Need reporters to triage label Apr 12, 2021

wikiwikiwiki added 2 commits April 28, 2021 13:53

fix bug: consumer warning "safe guard client, should not be called, m…

361a9d4

…ust have a bug."

AlbumenJ closed this May 23, 2021

zhongkaiuu mentioned this pull request May 25, 2021

什么情况下出现safe guard client , should not be called ,must have a bug.异常？ alibaba/spring-cloud-alibaba#1562

Open

yugj mentioned this pull request Jun 29, 2021

升级2.7.12 provider下线造成consumer线程池Terminated #8172

Closed

2 tasks

chickenlj added this to the 2.6.11 milestone Jul 14, 2021

chickenlj reopened this Jul 14, 2021

chickenlj mentioned this pull request Jul 21, 2021

Backport 2.7, lazy connection warn log #8329

Merged

pirateskipper reviewed Aug 24, 2021

View reviewed changes

xiaoheng1 modified the milestones: 2.6.11, 2.6.12 Sep 22, 2021

CrazyHZM closed this Oct 9, 2021

shaowYe pushed a commit to shaowYe/dubbo that referenced this pull request Aug 2, 2024

dubbo issue同步，2.7.12版本已修复

3e6dfaa

apache#6959 apache#7737

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Dubbo-6567] fix consumer warning "safe guard client , should not be called ,must have a bug." #6959

[Dubbo-6567] fix consumer warning "safe guard client , should not be called ,must have a bug." #6959

wikiwikiwiki commented Nov 30, 2020 •

edited

Loading

wow-blacksheep commented Dec 3, 2020

wikiwikiwiki commented Dec 4, 2020

AlbumenJ commented Apr 12, 2021

wikiwikiwiki commented Apr 28, 2021

AlbumenJ commented Apr 30, 2021

xiaoheng1 commented May 23, 2021

AlbumenJ commented May 23, 2021

startjava commented Jun 1, 2021

startjava commented Jun 1, 2021

AlbumenJ commented Jun 3, 2021

sh1yu commented Jun 16, 2021

AlbumenJ commented Jun 21, 2021

masquee commented Jul 8, 2021

pirateskipper Aug 24, 2021

pirateskipper Aug 24, 2021

YsoulM commented Sep 7, 2021

coder-michaelwu commented Oct 4, 2021 •

edited

Loading

CrazyHZM commented Oct 9, 2021

eoc2015 commented Oct 13, 2022

AlbumenJ commented Oct 13, 2022

echojun186 commented Dec 15, 2022

OnlyTL commented Feb 22, 2023

[Dubbo-6567] fix consumer warning "safe guard client , should not be called ,must have a bug." #6959

[Dubbo-6567] fix consumer warning "safe guard client , should not be called ,must have a bug." #6959

Conversation

wikiwikiwiki commented Nov 30, 2020 • edited Loading

wow-blacksheep commented Dec 3, 2020

wikiwikiwiki commented Dec 4, 2020

AlbumenJ commented Apr 12, 2021

wikiwikiwiki commented Apr 28, 2021

AlbumenJ commented Apr 30, 2021

xiaoheng1 commented May 23, 2021

AlbumenJ commented May 23, 2021

startjava commented Jun 1, 2021

startjava commented Jun 1, 2021

AlbumenJ commented Jun 3, 2021

sh1yu commented Jun 16, 2021

AlbumenJ commented Jun 21, 2021

masquee commented Jul 8, 2021

pirateskipper Aug 24, 2021

Choose a reason for hiding this comment

pirateskipper Aug 24, 2021

Choose a reason for hiding this comment

YsoulM commented Sep 7, 2021

coder-michaelwu commented Oct 4, 2021 • edited Loading

CrazyHZM commented Oct 9, 2021

eoc2015 commented Oct 13, 2022

AlbumenJ commented Oct 13, 2022

echojun186 commented Dec 15, 2022

OnlyTL commented Feb 22, 2023

wikiwikiwiki commented Nov 30, 2020 •

edited

Loading

coder-michaelwu commented Oct 4, 2021 •

edited

Loading