[Bug] 使用协程后，运行一段时间后，rpc 服务调用远程服务异常后，服务层耗时增加，无法正常提供服务

[chulong]

Description
使用 Alibaba Dragonwell 8.4.4 版本，发现服务运行一段时间后，服务无法正常提供服务，怀疑和 #226 问题一致，但是不确定是哪个线程引起无法正常切换协程

[chulong]

stack.txt

[joeyleeeeeee97]

@chulong
Hi, 请问是否使用了黑名单机制将 NIOHandler-TimeOut-Thread-pool-8-tid-4 不转为协程？

从jstack输出来看

"NIOHandler-TimeOut-Thread-pool-8-tid-4" #1005 prio=5 os_prio=0 tid=0x00007fba440e7a30 nid=0x5ca waiting on condition [0x00007fb9ce351000]

这种线程未被转为协程

• Coroutine [0x7fba58002690] "AsyncLoggerConfig-1" Add chinese document #14 active=36778324 steal=7499394 steal_fail=87848 preempt=1 park=0/-1
  这种线程是以协程形式运行的

[chulong]

是的，根据 #226 的解决方案，我设置了黑名单，但是发现未切换的线程出现耗时严重，怀疑抢不到资源

[henrysternc]

使用dragonwell 8.9.10运行一段时间后出现Rocketmq消费停止现象,我在消费MQ消息的业务逻辑处理上使用了线程池与多线程,是不是因为这些业务线程都以协程的方式跑在同一个线程上导致了业务线程饥饿而使MQ的消费线程阻塞,如果是线程饥饿引起的,如何把业务线程均匀分布到不同的协程组线程上呢?

[yuleil]

可以提供一下jstack吗？

使用dragonwell 8.9.10运行一段时间后出现Rocketmq消费停止现象,我在消费MQ消息的业务逻辑处理上使用了线程池与多线程,是不是因为这些业务线程都以协程的方式跑在同一个线程上导致了业务线程饥饿而使MQ的消费线程阻塞,如果是线程饥饿引起的,如何把业务线程均匀分布到不同的协程组线程上呢?

[henrysternc]

大佬您好,这是前几天的jstack快照文件 ,因为我这边业务可以分为多个子任务,子任务又可以分为多个子任务,所以存在多个线程池以处理不同的子任务避免造成线程饥饿.但jdk改为咱们的dragonwell之后发现运行一段时间后业务会阻塞,看了jstack快照发现业务线程全跑到一个协程组了,咱们的业务线程与协程工作线程是多对多的关系吧?所以我怀疑是线程饥饿导致的阻塞. 如果是,咱们的dragonwell协程调度有相关的解决方案吗?比如设定某一线程组的任务指定到一个协程组? 从 Windows 版邮件<https://go.microsoft.com/fwlink/?LinkId=550986>发送 发件人: ***@***.***> 发送时间: 2021年12月30日 17:06 收件人: ***@***.***> 抄送: ***@***.***>; ***@***.***> 主题: Re: [alibaba/dragonwell8] [Bug] 使用协程后，运行一段时间后，rpc 服务调用远程服务异常后，服务层耗时增加，无法正常提供服务 (Issue #265) 可以提供一下jstack吗？ 使用dragonwell 8.9.10运行一段时间后出现Rocketmq消费停止现象,我在消费MQ消息的业务逻辑处理上使用了线程池与多线程,是不是因为这些业务线程都以协程的方式跑在同一个线程上导致了业务线程饥饿而使MQ的消费线程阻塞,如果是线程饥饿引起的,如何把业务线程均匀分布到不同的协程组线程上呢? ― Reply to this email directly, view it on GitHub<#265 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFMMTNRMMMPKV63ATMTI54TUTQOJXANCNFSM5GSIR52Q>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***> 2021-12-29 18:00:12 Full thread dump OpenJDK 64-Bit Server VM (25.312-b01 mixed mode): "Attach Listener" #173 daemon prio=9 os_prio=0 tid=0x00007fd514002700 nid=0x49 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "DestroyJavaVM" #142 prio=5 os_prio=0 tid=0x00007fd598009d40 nid=0x7 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Wisp-Prevent-Shutdown-2" #41 prio=5 os_prio=0 tid=0x00007fd5996e6bb0 nid=0x2a runnable [0x00007fd50bafb000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000006f2f84010> (a com.alibaba.wisp.engine.ThreadAsWisp$PreventShutdownThread) at java.lang.Object.wait(Object.java:502) at com.alibaba.wisp.engine.ThreadAsWisp$PreventShutdownThread.run(ThreadAsWisp.java:151) - waiting to lock <0x00000006f2f84010> (a com.alibaba.wisp.engine.ThreadAsWisp$PreventShutdownThread) Locked ownable synchronizers: - None "Service Thread" #20 daemon prio=9 os_prio=0 tid=0x00007fd598217ac0 nid=0x26 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C1 CompilerThread3" #19 daemon prio=9 os_prio=0 tid=0x00007fd598215990 nid=0x25 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread2" #18 daemon prio=9 os_prio=0 tid=0x00007fd59820be50 nid=0x24 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread1" #17 daemon prio=9 os_prio=0 tid=0x00007fd59820a300 nid=0x23 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread0" #16 daemon prio=9 os_prio=0 tid=0x00007fd5982087b0 nid=0x22 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Signal Dispatcher" #15 daemon prio=9 os_prio=0 tid=0x00007fd598206bb0 nid=0x21 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Surrogate Locker Thread (Concurrent GC)" #14 daemon prio=9 os_prio=0 tid=0x00007fd5982052e0 nid=0x20 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Wisp-Root-Worker-0" #4 daemon prio=5 os_prio=0 tid=0x00007fd598203a90 nid=0x1f runnable [0x00007fd4efb72000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:129) at io.netty.channel.epoll.Native.epollWait(Native.java:122) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd51c019210] "lettuce-epollEventLoop-4-2" #107 active=1 steal=0 steal_fail=0 preempt=0 park=0/0 containerId=0 cg=0/0 ttr=0 - Coroutine [0x7fd51c000c80] "Wisp-Root-Worker-0" #9 active=850 steal=0 steal_fail=0 preempt=0 park=0/0 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.runTaskInternal(WispCarrier.java:150) at com.alibaba.wisp.engine.TaskDispatcher.run(TaskDispatcher.java:45) at com.alibaba.wisp.engine.WispScheduler.doExec(WispScheduler.java:536) at com.alibaba.wisp.engine.WispScheduler.access$600(WispScheduler.java:43) at com.alibaba.wisp.engine.WispScheduler$Worker.runCarrier(WispScheduler.java:161) at com.alibaba.wisp.engine.WispScheduler$Worker.run(WispScheduler.java:141) at java.lang.Thread.run(Thread.java:853) - Coroutine [0x7fd51c0432f0] "Nacso-Watch-Task-Scheduler-1" #105 active=1 steal=0 steal_fail=0 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) - parking to wait for <0x00000006f41ef720> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.misc.Unsafe.park(Unsafe.java:1034) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:216) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2087) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd53800b340] "SimplePauseDetectorThread_0" #40 active=244 steal=4 steal_fail=0 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispTask.sleep(WispTask.java:357) at com.alibaba.wisp.engine.WispEngine$5.sleep(WispEngine.java:246) at java.lang.Thread.sleep(Thread.java:384) at java.lang.Thread.sleep(Thread.java:433) at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386) at org.LatencyUtils.TimeServices.sleepNanos(TimeServices.java:62) at org.LatencyUtils.SimplePauseDetector$SimplePauseDetectorThread.run(SimplePauseDetector.java:116) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd51c007c70] "RequestHouseKeepingService" #60 active=2 steal=0 steal_fail=0 preempt=0 park=-1/0 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.park(WispTask.java:484) at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552) - waiting to lock <0x00000006f3a06740> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:505) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) Locked ownable synchronizers: - None "Wisp-Root-Worker-1" #5 daemon prio=5 os_prio=0 tid=0x00007fd598202240 nid=0x1e runnable [0x00007fd560755000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park0(Native Method) at sun.misc.Unsafe.access$200(Unsafe.java:45) at sun.misc.Unsafe$1.park0(Unsafe.java:65) at com.alibaba.wisp.engine.WispScheduler$Worker.doParkOrPolling(WispScheduler.java:188) at com.alibaba.wisp.engine.WispScheduler$Worker.runCarrier(WispScheduler.java:170) at com.alibaba.wisp.engine.WispScheduler$Worker.run(WispScheduler.java:141) at java.lang.Thread.run(Thread.java:853) - Coroutine [0x7fd540097690] "PullMessageServiceScheduledThread" #61 active=21187454 steal=63579 steal_fail=8 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) - parking to wait for <0x00000006f38e7748> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.misc.Unsafe.park(Unsafe.java:1034) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:216) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2087) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd528022c10] "NettyClientWorkerThread_1" #55 active=179217 steal=3048 steal_fail=0 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) - parking to wait for <0x00000006f38cdcc0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.misc.Unsafe.park(Unsafe.java:1034) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:216) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2087) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:471) at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:256) at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:64) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd52805ea80] "com.alibaba.nacos.naming.beat.sender" #120 active=99507 steal=124 steal_fail=0 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) - parking to wait for <0x00000006f2761438> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.misc.Unsafe.park(Unsafe.java:1029) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:176) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2047) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd53010f730] "com.alibaba.nacos.naming.beat.sender" #124 active=85017 steal=141 steal_fail=0 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) - parking to wait for <0x00000006f2761438> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.misc.Unsafe.park(Unsafe.java:1034) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:216) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2087) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd53800cf40] "NettyClientWorkerThread_2" #56 active=447343 steal=10036 steal_fail=36 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) - parking to wait for <0x00000006f38cdc30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.misc.Unsafe.park(Unsafe.java:1034) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:216) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2087) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:471) at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:256) at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:64) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd538006160] "Catalina-utility-1" #27 active=655345 steal=1411 steal_fail=2 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) - parking to wait for <0x00000006f2525140> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.misc.Unsafe.park(Unsafe.java:1034) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:216) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2087) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd52c0039c0] "com.alibaba.nacos.client.naming.updater" #22 active=124128 steal=358 steal_fail=0 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) - parking to wait for <0x00000006f286bd80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.misc.Unsafe.park(Unsafe.java:1034) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:216) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2087) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd524014080] "http-nio-9420-BlockPoller" #108 active=530885 steal=54 steal_fail=0 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEventPump.epollWaitForWisp(WispEventPump.java:260) at com.alibaba.wisp.engine.WispEventPump.access$200(WispEventPump.java:40) at com.alibaba.wisp.engine.WispEventPump$Pool.epollWaitForWisp(WispEventPump.java:116) at com.alibaba.wisp.engine.WispEngine$5.epollWait(WispEngine.java:226) at sun.nio.ch.EPollArrayWrapper.handleEPollWithWisp(EPollArrayWrapper.java:298) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:279) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:96) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000006f53ae290> (a sun.nio.ch.Util$3) - locked <0x00000006f53ae278> (a java.util.Collections$UnmodifiableSet) - locked <0x00000006f53993e8> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:313) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd52c0b0f10] "logback-2" #126 active=3410 steal=12 steal_fail=0 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) - parking to wait for <0x00000006f2871a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.misc.Unsafe.park(Unsafe.java:1029) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:176) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2047) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd5400c1110] "com.alibaba.nacos.naming.beat.sender" #123 active=82198 steal=127 steal_fail=0 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) - parking to wait for <0x00000006f2761438> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.misc.Unsafe.park(Unsafe.java:1029) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:176) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2047) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd540002a10] "logback-1" #16 active=3343 steal=8 steal_fail=0 preempt=0 park=1/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) - parking to wait for <0x00000006f2871a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.misc.Unsafe.park(Unsafe.java:1029) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:176) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2047) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd5280d88a0] "logback-7" #135 active=2528 steal=10 steal_fail=0 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) - parking to wait for <0x00000006f2871a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.misc.Unsafe.park(Unsafe.java:1029) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:176) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2047) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd5304bf600] "logback-3" #127 active=5324 steal=11 steal_fail=0 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) - parking to wait for <0x00000006f2871a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.misc.Unsafe.park(Unsafe.java:1029) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:176) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2047) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd540115670] "<cached>" #158 active=101 steal=0 steal_fail=0 preempt=0 park=0/0 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispCarrier.taskExit(WispCarrier.java:180) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:260) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd540112290] "<cached>" #154 active=101 steal=0 steal_fail=0 preempt=0 park=0/0 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispCarrier.taskExit(WispCarrier.java:180) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:260) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd528141db0] "<cached>" #156 active=2 steal=0 steal_fail=0 preempt=0 park=0/0 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispCarrier.taskExit(WispCarrier.java:180) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:260) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd530448040] "lettuce-eventExecutorLoop-1-2" #38 active=645884 steal=144231 steal_fail=2239 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) - parking to wait for <0x00000006f2f77d20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.misc.Unsafe.park(Unsafe.java:1029) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:176) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2047) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:446) at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:243) at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:64) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd53400edc0] "lettuce-eventExecutorLoop-1-5" #71 active=645657 steal=143502 steal_fail=6107 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) - parking to wait for <0x00000006f2f77ff0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.misc.Unsafe.park(Unsafe.java:1029) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:176) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2047) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:446) at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:243) at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:64) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd53801cb70] "alarm-task-" #94 active=321633 steal=54269 steal_fail=759 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) - parking to wait for <0x00000006f3961a58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.misc.Unsafe.park(Unsafe.java:1029) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:176) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2047) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:446) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd528121e20] "alarm-task-" #95 active=321452 steal=54851 steal_fail=3393 preempt=4 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) - parking to wait for <0x00000006f3961a58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.misc.Unsafe.park(Unsafe.java:1029) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:176) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2047) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:446) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd53801eaa0] "major-task-" #97 active=36239 steal=8468 steal_fail=107 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) - parking to wait for <0x00000006f3961d58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.misc.Unsafe.park(Unsafe.java:1029) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:176) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2047) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:446) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd54001d5b0] "ConsumeMessageThread_8" #129 active=2370 steal=303 steal_fail=4 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) - parking to wait for <0x00000006f39755d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.misc.Unsafe.park(Unsafe.java:1029) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:176) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2047) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:446) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd5301ad360] "major-task-" #93 active=36261 steal=8545 steal_fail=106 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) - parking to wait for <0x00000006f3961d58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.misc.Unsafe.park(Unsafe.java:1029) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:176) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2047) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:446) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd5400450a0] "major-task-" #103 active=36345 steal=8608 steal_fail=244 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) - parking to wait for <0x00000006f3961d58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.misc.Unsafe.park(Unsafe.java:1029) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:176) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2047) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:446) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd5240715a0] "major-task-" #100 active=36265 steal=8566 steal_fail=124 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) - parking to wait for <0x00000006f3961d58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.misc.Unsafe.park(Unsafe.java:1029) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:176) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2047) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:446) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd51c0027f0] "ConsumeMessageThread_7" #35 active=2370 steal=320 steal_fail=1 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) - parking to wait for <0x00000006f39755d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.misc.Unsafe.park(Unsafe.java:1029) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:176) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2047) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:446) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd5240278b0] "ConsumeMessageThread_4" #68 active=3130 steal=389 steal_fail=6 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) - parking to wait for <0x00000006f39755d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.misc.Unsafe.park(Unsafe.java:1029) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:176) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2047) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:446) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd52c08dec0] "ConsumeMessageThread_14" #137 active=1601 steal=244 steal_fail=0 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) - parking to wait for <0x00000006f39755d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.misc.Unsafe.park(Unsafe.java:1029) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:176) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2047) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:446) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd538039560] "ConsumeMessageThread_10" #122 active=1605 steal=214 steal_fail=1 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) - parking to wait for <0x00000006f39755d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.misc.Unsafe.park(Unsafe.java:1029) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:176) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2047) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:446) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd5280e8600] "http-nio-9420-exec-1" #106 active=1 steal=0 steal_fail=0 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) - parking to wait for <0x00000006f53ce6a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.misc.Unsafe.park(Unsafe.java:1029) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:176) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2047) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:446) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:107) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd53800eeb0] "AsyncAppender-Dispatcher-Thread-15" #47 active=8 steal=1 steal_fail=0 preempt=0 park=-1/0 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.park(WispTask.java:484) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.rocketmq.logging.inner.LoggingBuilder$AsyncAppender$Dispatcher.run(LoggingBuilder.java:386) - waiting to lock <0x00000006f38e7688> (a java.util.ArrayList) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) - Coroutine [0x7fd528006400] "com.alibaba.nacos.naming.push.receiver" #23 active=1 steal=0 steal_fail=0 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.alibaba.wisp.engine.WispTask.jdkPark(WispTask.java:473) at com.alibaba.wisp.engine.WispEngine$5.park(WispEngine.java:267) at sun.nio.ch.WispUdpSocketImpl.receive0(WispUdpSocketImpl.java:203) at sun.nio.ch.WispUdpSocketImpl.receive(WispUdpSocketImpl.java:134) at sun.nio.ch.WispUdpSocketImpl.receive(WispUdpSocketImpl.java:225) at java.net.DatagramSocket.receive(DatagramSocket.java:782) - locked <0x00000006f286e7e0> (a java.net.DatagramSocket) at com.alibaba.nacos.client.naming.core.PushReceiver.run(PushReceiver.java:73) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:853) at com.alibaba.wisp.engine.WispTask.runOutsideWisp(WispTask.java:299) at com.alibaba.wisp.engine.WispTask.runCommand(WispTask.java:274) at com.alibaba.wisp.engine.WispTask.access$100(WispTask.java:53) at com.alibaba.wisp.engine.WispTask$CacheableCoroutine.run(WispTask.java:241) at java.dyn.CoroutineBase.startInternal(CoroutineBase.java:62) Locked ownable synchronizers: - None "Wisp-Root-Worker-2" #6 daemon prio=5 os_prio=0 tid=0x00007fd5982009f0 nid=0x1d waiting on condition [0x00007fd560856000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park0(Native Method) at sun.misc.Unsafe.access$200(Unsafe.java:45) at sun.misc.Unsafe$1.park0(Unsafe.java:65) at com.alibaba.wisp.engine.WispScheduler$Worker.doParkOrPolling(WispScheduler.java:188) at com.alibaba.wisp.engine.WispScheduler$Worker.runCarrier(WispScheduler.java:170) at com.alibaba.wisp.engine.WispScheduler$Worker.run(WispScheduler.java:141) at java.lang.Thread.run(Thread.java:853) - Coroutine [0x7fd534009f10] "NettyClientSelector_1" #54 active=879703 steal=19233 steal_fail=122 preempt=0 park=0/-1 containerId=0 cg=0/0 ttr=0 at java.dyn.CoroutineSupport.unsafeSymmetricYieldTo(CoroutineSupport.java:140) at com.alibaba.wisp.engine.WispTask.switchTo(WispTask.java:329) at com.alibaba.wisp.engine.WispCarrier.yieldTo(WispCarrier.java:436) at com.alibaba.wisp.engine.WispCarrier.schedule(WispCarrier.java:274) at com.alibaba.wisp.engine.WispTask.parkInternal(WispTask.java:421) at com.