Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multithread Deadlock problem at BucketedCounterStream.startCachingStreamValuesIfUnstarted #1521

Open
narusas opened this issue Mar 30, 2017 · 4 comments
Open

Multithread Deadlock problem at BucketedCounterStream.startCachingStreamValuesIfUnstarted #1521

narusas opened this issue Mar 30, 2017 · 4 comments
Labels
bug hystrix-core metrics

Comments

@narusas
Copy link

narusas commented Mar 30, 2017

Using Semephore isolation.

  com.netflix.hystrix.HystrixCommandProperties.Setter prop = HystrixCommandProperties.Setter()
				.withExecutionIsolationStrategy(strategy)
				.withExecutionIsolationSemaphoreMaxConcurrentRequests(50)
				.withFallbackEnabled(false)
				.withCircuitBreakerEnabled(true)
		;
		return HystrixCommand.Setter
				.withGroupKey(HystrixCommandGroupKey.Factory.asKey(name))
				.andCommandKey(HystrixCommandKey.Factory.asKey(name))
				.andCommandPropertiesDefaults(prop)

and launch application on weblogic, multi user's request arraived and all hang.

after600 seconds, Exception has throwed.

  <Mar 30, 2017 9:51:27 AM KST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '43' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "623" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 623540 ms
", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
	rx.internal.operators.OperatorScan$InitialProducer.emitLoop(OperatorScan.java:330)
	rx.internal.operators.OperatorScan$InitialProducer.emit(OperatorScan.java:295)
	rx.internal.operators.OperatorScan$InitialProducer.onNext(OperatorScan.java:202)
	rx.internal.operators.OperatorScan$3.onNext(OperatorScan.java:144)
	rx.subjects.UnicastSubject$State.replay(UnicastSubject.java:298)
	rx.subjects.UnicastSubject$State.onNext(UnicastSubject.java:179)
	rx.subjects.UnicastSubject.onNext(UnicastSubject.java:92)
	rx.internal.operators.OperatorWindowWithSize$WindowOverlap.onNext(OperatorWindowWithSize.java:345)
	rx.observers.Subscribers$5.onNext(Subscribers.java:235)
	rx.observers.SerializedObserver.onNext(SerializedObserver.java:92)
	rx.observers.SerializedSubscriber.onNext(SerializedSubscriber.java:94)
	rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.innerNext(OnSubscribeConcatMap.java:182)
	rx.internal.operators.OnSubscribeConcatMap$ConcatMapInnerSubscriber.onNext(OnSubscribeConcatMap.java:335)
	rx.internal.operators.OnSubscribeFromIterable$IterableProducer.fastPath(OnSubscribeFromIterable.java:173)
	rx.internal.operators.OnSubscribeFromIterable$IterableProducer.request(OnSubscribeFromIterable.java:86)
	rx.internal.producers.ProducerArbiter.setProducer(ProducerArbiter.java:126)
	rx.internal.operators.OnSubscribeConcatMap$ConcatMapInnerSubscriber.setProducer(OnSubscribeConcatMap.java:329)
	rx.internal.operators.OnSubscribeFromIterable.call(OnSubscribeFromIterable.java:63)
	rx.internal.operators.OnSubscribeFromIterable.call(OnSubscribeFromIterable.java:34)
	rx.Observable.unsafeSubscribe(Observable.java:10151)
	rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.drain(OnSubscribeConcatMap.java:286)
	rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.onNext(OnSubscribeConcatMap.java:144)
	rx.internal.operators.OnSubscribeFromArray$FromArrayProducer.slowPath(OnSubscribeFromArray.java:100)
	rx.internal.operators.OnSubscribeFromArray$FromArrayProducer.request(OnSubscribeFromArray.java:63)
	rx.Subscriber.setProducer(Subscriber.java:211)
	rx.internal.operators.OnSubscribeFromArray.call(OnSubscribeFromArray.java:32)
	rx.internal.operators.OnSubscribeFromArray.call(OnSubscribeFromArray.java:24)
	rx.Observable.unsafeSubscribe(Observable.java:10151)
	rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:94)
	rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:42)
	rx.Observable.unsafeSubscribe(Observable.java:10151)
	rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
	rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
	rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
	rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
	rx.Observable.unsafeSubscribe(Observable.java:10151)
	rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
	rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
	rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
	rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
	rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
	rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
	rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
	rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
	rx.Observable.unsafeSubscribe(Observable.java:10151)
	rx.internal.operators.OperatorPublish.connect(OperatorPublish.java:213)
	rx.internal.operators.OnSubscribeRefCount.call(OnSubscribeRefCount.java:67)
	rx.internal.operators.OnSubscribeRefCount.call(OnSubscribeRefCount.java:34)
	rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
	rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
	rx.Observable.subscribe(Observable.java:10247)
	rx.Observable.subscribe(Observable.java:10214)
	rx.Observable.subscribe(Observable.java:10122)
	com.netflix.hystrix.metric.consumer.BucketedCounterStream.startCachingStreamValuesIfUnstarted(BucketedCounterStream.java:88)
	com.netflix.hystrix.metric.consumer.HealthCountsStream.getInstance(HealthCountsStream.java:75)
	com.netflix.hystrix.metric.consumer.HealthCountsStream.getInstance(HealthCountsStream.java:62)
	com.netflix.hystrix.HystrixCommandMetrics.<init>(HystrixCommandMetrics.java:192)
	com.netflix.hystrix.HystrixCommandMetrics.getInstance(HystrixCommandMetrics.java:134)
	com.netflix.hystrix.AbstractCommand.initMetrics(AbstractCommand.java:240)
	com.netflix.hystrix.AbstractCommand.<init>(AbstractCommand.java:166)
	com.netflix.hystrix.HystrixCommand.<init>(HystrixCommand.java:148)
	com.netflix.hystrix.HystrixCommand.<init>(HystrixCommand.java:134)

First thread take lock of HystrixCommandMetrics.java
com.netflix.hystrix.HystrixCommandMetrics.getInstance(HystrixCommandMetrics.java:124)

And call BucketedCounterStream.startCachingStreamValuesIfUnstarted

But it looks like never end.

Another Thread can not take lock of HystrixCommandMetrics.java.

<Mar 30, 2017 9:51:27 AM KST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '53' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "610" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 610283 ms
", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
	com.netflix.hystrix.HystrixCommandMetrics.getInstance(HystrixCommandMetrics.java:124)
	com.netflix.hystrix.AbstractCommand.initMetrics(AbstractCommand.java:240)
	com.netflix.hystrix.AbstractCommand.<init>(AbstractCommand.java:166)
	com.netflix.hystrix.HystrixCommand.<init>(HystrixCommand.java:148)
	com.netflix.hystrix.HystrixCommand.<init>(HystrixCommand.java:134)

All threads is Deathlocked...

Hope some help...
@narusas narusas changed the title Multithread Deadlock problem at com.netflix.hystrix.metric.consumer.BucketedCounterStream.startCachingStreamValuesIfUnstarted(BucketedCounterStream.java:88) Multithread Deadlock problem at BucketedCounterStream.startCachingStreamValuesIfUnstarted Mar 30, 2017
@mattrjacobs
Copy link
Contributor

Thanks for the report @narusas. What version are you seeing this in? Does it happen all the time or occasionally?

@mattrjacobs mattrjacobs mentioned this issue Apr 7, 2017
Merged
@mattrjacobs
Copy link
Contributor

I couldn't replicate, nor have I seen it occur in production, but I believe the change in #1532 should fix your issue

@mattrjacobs
Copy link
Contributor

Did 1.5.11 resolve your issue @narusas ?

@jeho0815
Copy link

jeho0815 commented Nov 14, 2018 •
edited
Loading

@mattrjacobs Hi，I'm using Apache ServiceComb, i have a same issue. An thread was blocked for a long time.

2018-11-14 14:16:53.858 [vertx-blocked-thread-checker] WARN  - [BlockedThreadChecker.java:run:57] - [] - Thread Thread[transport-vert.x-eventloop-thread-7,5,main] has been blocked for 18225 ms, time limit is 2000
io.vertx.core.VertxException: Thread blocked
	at rx.internal.operators.OperatorScan$3.onNext(OperatorScan.java:138)
	at rx.subjects.UnicastSubject$State.onNext(UnicastSubject.java:185)
	at rx.subjects.UnicastSubject.onNext(UnicastSubject.java:92)
	at rx.internal.operators.OperatorWindowWithTime$ExactSubscriber.emitValue(OperatorWindowWithTime.java:231)
	at rx.internal.operators.OperatorWindowWithTime$ExactSubscriber.onNext(OperatorWindowWithTime.java:153)
	at rx.internal.operators.OnSubscribeRefCount$2.onNext(OnSubscribeRefCount.java:124)
	at rx.internal.operators.OperatorPublish$PublishSubscriber.dispatch(OperatorPublish.java:588)
	at rx.internal.operators.OperatorPublish$PublishSubscriber.onNext(OperatorPublish.java:286)
	at rx.subjects.SubjectSubscriptionManager$SubjectObserver.onNext(SubjectSubscriptionManager.java:225)
	at rx.subjects.PublishSubject.onNext(PublishSubject.java:114)
	at rx.internal.operators.NotificationLite.accept(NotificationLite.java:151)
	at rx.observers.SerializedObserver.onNext(SerializedObserver.java:117)
	at rx.subjects.SerializedSubject.onNext(SerializedSubject.java:67)
	at com.netflix.hystrix.metric.HystrixCommandCompletionStream.write(HystrixCommandCompletionStream.java:69)
	at com.netflix.hystrix.metric.HystrixThreadEventStream$3.call(HystrixThreadEventStream.java:87)
	at com.netflix.hystrix.metric.HystrixThreadEventStream$3.call(HystrixThreadEventStream.java:83)
	at rx.internal.util.ActionSubscriber.onNext(ActionSubscriber.java:39)
	at rx.internal.operators.OperatorDoOnEach$1.onNext(OperatorDoOnEach.java:81)
	at rx.internal.operators.NotificationLite.accept(NotificationLite.java:151)
	at rx.internal.operators.OperatorOnBackpressureBuffer$BufferSubscriber.accept(OperatorOnBackpressureBuffer.java:160)
	at rx.internal.util.BackpressureDrainManager.drain(BackpressureDrainManager.java:200)
	at rx.internal.operators.OperatorOnBackpressureBuffer$BufferSubscriber.onNext(OperatorOnBackpressureBuffer.java:155)
	at rx.subjects.SubjectSubscriptionManager$SubjectObserver.onNext(SubjectSubscriptionManager.java:225)
	at rx.subjects.PublishSubject.onNext(PublishSubject.java:114)
	at com.netflix.hystrix.metric.HystrixThreadEventStream.executionDone(HystrixThreadEventStream.java:145)
	at com.netflix.hystrix.HystrixCommandMetrics.markCommandDone(HystrixCommandMetrics.java:340)
	at com.netflix.hystrix.AbstractCommand.handleCommandEnd(AbstractCommand.java:955)
	at com.netflix.hystrix.AbstractCommand.access$000(AbstractCommand.java:60)
	at com.netflix.hystrix.AbstractCommand$2.call(AbstractCommand.java:376)
	at rx.internal.util.ActionSubscriber.onCompleted(ActionSubscriber.java:49)
	at rx.internal.operators.OperatorDoOnEach$1.onCompleted(OperatorDoOnEach.java:47)
	at rx.internal.operators.OperatorMap$MapSubscriber.onCompleted(OperatorMap.java:94)
	at rx.observers.Subscribers$5.onCompleted(Subscribers.java:225)
	at rx.observers.Subscribers$5.onCompleted(Subscribers.java:225)
	at rx.internal.operators.OperatorDoOnEach$1.onCompleted(OperatorDoOnEach.java:54)
	at rx.internal.operators.OperatorDoOnEach$1.onCompleted(OperatorDoOnEach.java:54)
	at rx.internal.operators.OperatorDoOnEach$1.onCompleted(OperatorDoOnEach.java:54)
	at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onCompleted(OperatorOnErrorResumeNextViaFunction.java:101)
	at rx.internal.operators.OperatorDoOnEach$1.onCompleted(OperatorDoOnEach.java:54)
	at rx.internal.operators.OperatorDoOnEach$1.onCompleted(OperatorDoOnEach.java:54)
	at rx.observers.Subscribers$5.onCompleted(Subscribers.java:225)
	at com.netflix.hystrix.AbstractCommand$DeprecatedOnRunHookApplication$1.onCompleted(AbstractCommand.java:1423)
	at com.netflix.hystrix.AbstractCommand$ExecutionHookApplication$1.onCompleted(AbstractCommand.java:1356)
	at org.apache.servicecomb.bizkeeper.BizkeeperCommand.lambda$null$1(BizkeeperCommand.java:88)
	at org.apache.servicecomb.bizkeeper.BizkeeperCommand$$Lambda$274/1397283423.handle(Unknown Source)
	at org.apache.servicecomb.swagger.invocation.AsyncResponse.complete(AsyncResponse.java:46)
	at org.apache.servicecomb.transport.rest.client.http.RestClientInvocation.lambda$processResponseBody$7(RestClientInvocation.java:186)
	at org.apache.servicecomb.transport.rest.client.http.RestClientInvocation$$Lambda$296/15568048.run(Unknown Source)
	at org.apache.servicecomb.core.provider.consumer.ReactiveResponseExecutor.execute(ReactiveResponseExecutor.java:28)
	at org.apache.servicecomb.transport.rest.client.http.RestClientInvocation.processResponseBody(RestClientInvocation.java:179)
	at org.apache.servicecomb.transport.rest.client.http.RestClientInvocation.lambda$handleResponse$6(RestClientInvocation.java:174)
	at org.apache.servicecomb.transport.rest.client.http.RestClientInvocation$$Lambda$295/1162608664.handle(Unknown Source)
	at io.vertx.core.http.impl.HttpClientResponseImpl$BodyHandler.notifyHandler(HttpClientResponseImpl.java:307)
	at io.vertx.core.http.impl.HttpClientResponseImpl.lambda$bodyHandler$0(HttpClientResponseImpl.java:197)
	at io.vertx.core.http.impl.HttpClientResponseImpl$$Lambda$178/1029510944.handle(Unknown Source)
	at io.vertx.core.http.impl.HttpClientResponseImpl.handleEnd(HttpClientResponseImpl.java:261)
	at io.vertx.core.http.impl.ClientConnection.handleResponseEnd(ClientConnection.java:358)
	at io.vertx.core.http.impl.ClientHandler.handleMessage(ClientHandler.java:100)
	at io.vertx.core.http.impl.ClientHandler.handleMessage(ClientHandler.java:36)
	at io.vertx.core.net.impl.VertxHandler.lambda$channelRead$1(VertxHandler.java:150)
	at io.vertx.core.net.impl.VertxHandler$$Lambda$52/1876804217.run(Unknown Source)
	at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:342)
	at io.vertx.core.impl.ContextImpl$$Lambda$24/689749616.run(Unknown Source)
	at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:200)
	at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:148)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

2018-11-14 14:16:45.856 [vertx-blocked-thread-checker] WARN  - [BlockedThreadChecker.java:run:57] - [] - Thread Thread[transport-vert.x-eventloop-thread-7,5,main] has been blocked for 10222 ms, time limit is 2000
io.vertx.core.VertxException: Thread blocked
	at rx.internal.operators.OperatorScan$InitialProducer.emitLoop(OperatorScan.java:332)
	at rx.internal.operators.OperatorScan$InitialProducer.emit(OperatorScan.java:295)
	at rx.internal.operators.OperatorScan$InitialProducer.onNext(OperatorScan.java:202)
	at rx.internal.operators.OperatorScan$3.onNext(OperatorScan.java:144)
	at rx.subjects.UnicastSubject$State.onNext(UnicastSubject.java:185)
	at rx.subjects.UnicastSubject.onNext(UnicastSubject.java:92)
	at rx.internal.operators.OperatorWindowWithTime$ExactSubscriber.emitValue(OperatorWindowWithTime.java:231)
	at rx.internal.operators.OperatorWindowWithTime$ExactSubscriber.onNext(OperatorWindowWithTime.java:153)
	at rx.internal.operators.OnSubscribeRefCount$2.onNext(OnSubscribeRefCount.java:124)
	at rx.internal.operators.OperatorPublish$PublishSubscriber.dispatch(OperatorPublish.java:588)
	at rx.internal.operators.OperatorPublish$PublishSubscriber.onNext(OperatorPublish.java:286)
	at rx.subjects.SubjectSubscriptionManager$SubjectObserver.onNext(SubjectSubscriptionManager.java:225)
	at rx.subjects.PublishSubject.onNext(PublishSubject.java:114)
	at rx.internal.operators.NotificationLite.accept(NotificationLite.java:151)
	at rx.observers.SerializedObserver.onNext(SerializedObserver.java:117)
	at rx.subjects.SerializedSubject.onNext(SerializedSubject.java:67)
	at com.netflix.hystrix.metric.HystrixCommandCompletionStream.write(HystrixCommandCompletionStream.java:69)
	at com.netflix.hystrix.metric.HystrixThreadEventStream$3.call(HystrixThreadEventStream.java:87)
	at com.netflix.hystrix.metric.HystrixThreadEventStream$3.call(HystrixThreadEventStream.java:83)
	at rx.internal.util.ActionSubscriber.onNext(ActionSubscriber.java:39)
	at rx.internal.operators.OperatorDoOnEach$1.onNext(OperatorDoOnEach.java:81)
	at rx.internal.operators.NotificationLite.accept(NotificationLite.java:151)
	at rx.internal.operators.OperatorOnBackpressureBuffer$BufferSubscriber.accept(OperatorOnBackpressureBuffer.java:160)
	at rx.internal.util.BackpressureDrainManager.drain(BackpressureDrainManager.java:200)
	at rx.internal.operators.OperatorOnBackpressureBuffer$BufferSubscriber.onNext(OperatorOnBackpressureBuffer.java:155)
	at rx.subjects.SubjectSubscriptionManager$SubjectObserver.onNext(SubjectSubscriptionManager.java:225)
	at rx.subjects.PublishSubject.onNext(PublishSubject.java:114)
	at com.netflix.hystrix.metric.HystrixThreadEventStream.executionDone(HystrixThreadEventStream.java:145)
	at com.netflix.hystrix.HystrixCommandMetrics.markCommandDone(HystrixCommandMetrics.java:340)
	at com.netflix.hystrix.AbstractCommand.handleCommandEnd(AbstractCommand.java:955)
	at com.netflix.hystrix.AbstractCommand.access$000(AbstractCommand.java:60)
	at com.netflix.hystrix.AbstractCommand$2.call(AbstractCommand.java:376)
	at rx.internal.util.ActionSubscriber.onCompleted(ActionSubscriber.java:49)
	at rx.internal.operators.OperatorDoOnEach$1.onCompleted(OperatorDoOnEach.java:47)
	at rx.internal.operators.OperatorMap$MapSubscriber.onCompleted(OperatorMap.java:94)
	at rx.observers.Subscribers$5.onCompleted(Subscribers.java:225)
	at rx.observers.Subscribers$5.onCompleted(Subscribers.java:225)
	at rx.internal.operators.OperatorDoOnEach$1.onCompleted(OperatorDoOnEach.java:54)
	at rx.internal.operators.OperatorDoOnEach$1.onCompleted(OperatorDoOnEach.java:54)
	at rx.internal.operators.OperatorDoOnEach$1.onCompleted(OperatorDoOnEach.java:54)
	at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onCompleted(OperatorOnErrorResumeNextViaFunction.java:101)
	at rx.internal.operators.OperatorDoOnEach$1.onCompleted(OperatorDoOnEach.java:54)
	at rx.internal.operators.OperatorDoOnEach$1.onCompleted(OperatorDoOnEach.java:54)
	at rx.observers.Subscribers$5.onCompleted(Subscribers.java:225)
	at com.netflix.hystrix.AbstractCommand$DeprecatedOnRunHookApplication$1.onCompleted(AbstractCommand.java:1423)
	at com.netflix.hystrix.AbstractCommand$ExecutionHookApplication$1.onCompleted(AbstractCommand.java:1356)
	at org.apache.servicecomb.bizkeeper.BizkeeperCommand.lambda$null$1(BizkeeperCommand.java:88)
	at org.apache.servicecomb.bizkeeper.BizkeeperCommand$$Lambda$274/1397283423.handle(Unknown Source)
	at org.apache.servicecomb.swagger.invocation.AsyncResponse.complete(AsyncResponse.java:46)
	at org.apache.servicecomb.transport.rest.client.http.RestClientInvocation.lambda$processResponseBody$7(RestClientInvocation.java:186)
	at org.apache.servicecomb.transport.rest.client.http.RestClientInvocation$$Lambda$296/15568048.run(Unknown Source)
	at org.apache.servicecomb.core.provider.consumer.ReactiveResponseExecutor.execute(ReactiveResponseExecutor.java:28)
	at org.apache.servicecomb.transport.rest.client.http.RestClientInvocation.processResponseBody(RestClientInvocation.java:179)
	at org.apache.servicecomb.transport.rest.client.http.RestClientInvocation.lambda$handleResponse$6(RestClientInvocation.java:174)
	at org.apache.servicecomb.transport.rest.client.http.RestClientInvocation$$Lambda$295/1162608664.handle(Unknown Source)
	at io.vertx.core.http.impl.HttpClientResponseImpl$BodyHandler.notifyHandler(HttpClientResponseImpl.java:307)
	at io.vertx.core.http.impl.HttpClientResponseImpl.lambda$bodyHandler$0(HttpClientResponseImpl.java:197)
	at io.vertx.core.http.impl.HttpClientResponseImpl$$Lambda$178/1029510944.handle(Unknown Source)
	at io.vertx.core.http.impl.HttpClientResponseImpl.handleEnd(HttpClientResponseImpl.java:261)
	at io.vertx.core.http.impl.ClientConnection.handleResponseEnd(ClientConnection.java:358)
	at io.vertx.core.http.impl.ClientHandler.handleMessage(ClientHandler.java:100)
	at io.vertx.core.http.impl.ClientHandler.handleMessage(ClientHandler.java:36)
	at io.vertx.core.net.impl.VertxHandler.lambda$channelRead$1(VertxHandler.java:150)
	at io.vertx.core.net.impl.VertxHandler$$Lambda$52/1876804217.run(Unknown Source)
	at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:342)
	at io.vertx.core.impl.ContextImpl$$Lambda$24/689749616.run(Unknown Source)
	at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:200)
	at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:148)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

compare the two thread stacks, looks like have an infinite loop!
Hystrix version is 1.5.10
Hope some help...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug hystrix-core metrics
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mattrjacobs @narusas @jeho0815