Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] Build and Test Scheduled On 3.3 might be always failed since 'feat: remove useless dependency(#15125)' #15160

Closed
4 tasks done
zrlw opened this issue Feb 20, 2025 · 5 comments · Fixed by #15163 · May be fixed by #15164
Closed
4 tasks done

[Bug] Build and Test Scheduled On 3.3 might be always failed since 'feat: remove useless dependency(#15125)' #15160

zrlw opened this issue Feb 20, 2025 · 5 comments · Fixed by #15163 · May be fixed by #15164
Labels
component/need-triage Need maintainers to triage type/need-triage Need maintainers to triage

Comments

@zrlw
Copy link
Contributor

zrlw commented Feb 20, 2025 •
edited
Loading

Pre-check

  • I am sure that all the content I provide is in English.

Search before asking

  • I had searched in the issues and found no similar issues.

Apache Dubbo Component

Java SDK (apache/dubbo)

Dubbo Version

dubbo 3.x

Steps to reproduce this issue

see Build and Test Scheduled On 3.3 at https://github.com/apache/dubbo/actions

Almost all Unit Test On windows-latest are failed since feat: remove useless dependency(#15125), i doubt dubbo-test-check might not be removed from some test modules or ReplierDispatcherTest might have some issues about threadpool size.

Error:  Errors: 
Error:    ReplierDispatcherTest.testMultiThread:114 Execution org.opentest4j.AssertionFailedError

@Stellar1999 @CrazyHZM

What you expected to happen

Build and Test Scheduled On 3.3 should be successful.

Anything else

No response

Are you willing to submit a pull request to fix on your own?

  • Yes I am willing to submit a pull request on my own!

Code of Conduct
@zrlw zrlw added component/need-triage Need maintainers to triage type/need-triage Need maintainers to triage labels Feb 20, 2025
@zrlw
Copy link
Contributor Author

zrlw commented Feb 20, 2025

extracted a error message snippet from https://github.com/apache/dubbo/actions/runs/13425007540/job/37506574404

Error:  org.apache.dubbo.remoting.transport.netty4.ReplierDispatcherTest.testMultiThread -- Time elapsed: 11.06 s <<< ERROR!
java.util.concurrent.ExecutionException: org.opentest4j.AssertionFailedError
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at org.apache.dubbo.remoting.transport.netty4.ReplierDispatcherTest.testMultiThread(ReplierDispatcherTest.java:114)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
Caused by: org.opentest4j.AssertionFailedError
	at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:46)
	at org.junit.jupiter.api.Assertions.fail(Assertions.java:161)
	at org.apache.dubbo.remoting.transport.netty4.ReplierDispatcherTest.lambda$testMultiThread$1(ReplierDispatcherTest.java:110)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.util.concurrent.ExecutionException: org.apache.dubbo.remoting.RemotingException: Server side(127.0.0.1,31708) thread pool is exhausted, detail msg:Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-127.0.0.1:8080, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 499 (completed: 299), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in empty://127.0.0.1:8080!
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
	at org.apache.dubbo.remoting.transport.netty4.ReplierDispatcherTest.clientExchangeInfo(ReplierDispatcherTest.java:136)
	at org.apache.dubbo.remoting.transport.netty4.ReplierDispatcherTest.lambda$testMultiThread$1(ReplierDispatcherTest.java:108)
	... 5 more
Caused by: org.apache.dubbo.remoting.RemotingException: Server side(127.0.0.1,31708) thread pool is exhausted, detail msg:Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-127.0.0.1:8080, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 499 (completed: 299), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in empty://127.0.0.1:8080!
	at org.apache.dubbo.remoting.exchange.support.DefaultFuture.doReceived(DefaultFuture.java:265)
	at org.apache.dubbo.remoting.exchange.support.DefaultFuture.received(DefaultFuture.java:208)
	at org.apache.dubbo.remoting.exchange.support.DefaultFuture.received(DefaultFuture.java:196)
	at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleResponse(HeaderExchangeHandler.java:65)
	at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:211)
	at org.apache.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)
	at org.apache.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:64)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at org.apache.dubbo.common.threadlocal.InternalRunnable.run(InternalRunnable.java:39)
	... 1 more

@zrlw zrlw mentioned this issue Feb 20, 2025
Merged
4 tasks
@zrlw
Copy link
Contributor Author

zrlw commented Feb 20, 2025 •
edited
Loading

it seemed that RpcMessageHandler.DEFAULT_PROVIDER#getImplementation should be optimized as it called newInstance for each reply.
@AlbumenJ

@AlbumenJ
Copy link
Member

it seemed that RpcMessageHandler.DEFAULT_PROVIDER#getImplementation should be optimized as it called newInstance for each reply. @AlbumenJ

Merged. Wait for the next test round

@zrlw
Copy link
Contributor Author

zrlw commented Feb 22, 2025 •
edited
Loading

it seemed that RpcMessageHandler.DEFAULT_PROVIDER#getImplementation should be optimized as it called newInstance for each reply. @AlbumenJ

Merged. Wait for the next test round
It doesn't seem like the problem is caused by slow execution because ReplierDispatcherTest.testMultiThread of dubbo-remoting-netty4 still failed after #15163 being merged. sometimes it failed with TimeoutException caused by waiting server-side response timeout. sometimes it failed with RemotingException caused by exhausted thread pool, but the weird thing is the active threads number was much less than exhausted dubbo thread pool size.

  1. TimeoutException at https://github.com/apache/dubbo/actions/runs/13451076793/job/37585864177
2025-02-14T18:30:24.1818982Z [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 67.37 s <<< FAILURE! -- in org.apache.dubbo.remoting.transport.netty4.ReplierDispatcherTest
2025-02-14T18:30:24.1821034Z [ERROR] org.apache.dubbo.remoting.transport.netty4.ReplierDispatcherTest.testMultiThread -- Time elapsed: 65.01 s <<< ERROR!
2025-02-14T18:30:24.1822574Z java.util.concurrent.ExecutionException: org.opentest4j.AssertionFailedError
2025-02-14T18:30:24.1824150Z 	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
2025-02-14T18:30:24.1825034Z 	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
2025-02-14T18:30:24.1826314Z 	at org.apache.dubbo.remoting.transport.netty4.ReplierDispatcherTest.testMultiThread(ReplierDispatcherTest.java:114)
2025-02-14T18:30:24.1827819Z 	at java.lang.reflect.Method.invoke(Method.java:498)
2025-02-14T18:30:24.1828527Z 	at java.util.ArrayList.forEach(ArrayList.java:1259)
2025-02-14T18:30:24.1829231Z 	at java.util.ArrayList.forEach(ArrayList.java:1259)
2025-02-14T18:30:24.1830075Z Caused by: org.opentest4j.AssertionFailedError
2025-02-14T18:30:24.1830902Z 	at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:46)
2025-02-14T18:30:24.1831777Z 	at org.junit.jupiter.api.Assertions.fail(Assertions.java:161)
2025-02-14T18:30:24.1833037Z 	at org.apache.dubbo.remoting.transport.netty4.ReplierDispatcherTest.lambda$testMultiThread$1(ReplierDispatcherTest.java:110)
2025-02-14T18:30:24.1834471Z 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
2025-02-14T18:30:24.1835480Z 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2025-02-14T18:30:24.1836723Z 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2025-02-14T18:30:24.1837879Z 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2025-02-14T18:30:24.1838732Z 	at java.lang.Thread.run(Thread.java:750)
2025-02-14T18:30:24.1849122Z Caused by: java.util.concurrent.ExecutionException: org.apache.dubbo.remoting.TimeoutException: Waiting server-side response timeout by scan timer. start time: 2025-02-14 18:29:17.501, end time: 2025-02-14 18:30:17.526, client elapsed: 0 ms, server elapsed: 60025 ms, timeout: 60000 ms, request: Request [id=1752129614662792547, version=2.0.2, twoWay=true, event=false, broken=false, mPayload=0, data=null], channel: /172.22.80.1:65226 -> /172.22.80.1:39839
2025-02-14T18:30:24.1853182Z 	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
2025-02-14T18:30:24.1854655Z 	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
2025-02-14T18:30:24.1864961Z 	at org.apache.dubbo.remoting.transport.netty4.ReplierDispatcherTest.clientExchangeInfo(ReplierDispatcherTest.java:136)
2025-02-14T18:30:24.1867644Z 	at org.apache.dubbo.remoting.transport.netty4.ReplierDispatcherTest.lambda$testMultiThread$1(ReplierDispatcherTest.java:108)
2025-02-14T18:30:24.1869228Z 	... 5 more
2025-02-14T18:30:24.1871926Z Caused by: org.apache.dubbo.remoting.TimeoutException: Waiting server-side response timeout by scan timer. start time: 2025-02-14 18:29:17.501, end time: 2025-02-14 18:30:17.526, client elapsed: 0 ms, server elapsed: 60025 ms, timeout: 60000 ms, request: Request [id=1752129614662792547, version=2.0.2, twoWay=true, event=false, broken=false, mPayload=0, data=null], channel: /172.22.80.1:65226 -> /172.22.80.1:39839
2025-02-14T18:30:24.1892037Z 	at org.apache.dubbo.remoting.exchange.support.DefaultFuture.doReceived(DefaultFuture.java:261)
2025-02-14T18:30:24.1893995Z 	at org.apache.dubbo.remoting.exchange.support.DefaultFuture.received(DefaultFuture.java:208)
2025-02-14T18:30:24.1895762Z 	at org.apache.dubbo.remoting.exchange.support.DefaultFuture$TimeoutCheckTask.notifyTimeout(DefaultFuture.java:340)
2025-02-14T18:30:24.1897195Z 	at org.apache.dubbo.remoting.exchange.support.DefaultFuture$TimeoutCheckTask.run(DefaultFuture.java:329)
2025-02-14T18:30:24.1898400Z 	at org.apache.dubbo.common.timer.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:654)
2025-02-14T18:30:24.1899508Z 	at org.apache.dubbo.common.timer.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:733)
2025-02-14T18:30:24.1900679Z 	at org.apache.dubbo.common.timer.HashedWheelTimer$Worker.run(HashedWheelTimer.java:455)
  1. RemotingException caused by exhausted thread pool which active was 145, see https://github.com/apache/dubbo/actions/runs/13451076793/job/37585867401
2025-02-14T18:29:17.1177519Z [ERROR] org.apache.dubbo.remoting.transport.netty4.ReplierDispatcherTest.testMultiThread -- Time elapsed: 11.14 s <<< ERROR!
2025-02-14T18:29:17.1179711Z java.util.concurrent.ExecutionException: org.opentest4j.AssertionFailedError
2025-02-14T18:29:17.1180698Z 	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
2025-02-14T18:29:17.1181540Z 	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
2025-02-14T18:29:17.1182714Z 	at org.apache.dubbo.remoting.transport.netty4.ReplierDispatcherTest.testMultiThread(ReplierDispatcherTest.java:114)
2025-02-14T18:29:17.1183831Z 	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
2025-02-14T18:29:17.1184493Z 	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
2025-02-14T18:29:17.1185149Z 	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
2025-02-14T18:29:17.1186761Z Caused by: org.opentest4j.AssertionFailedError
2025-02-14T18:29:17.1187808Z 	at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:46)
2025-02-14T18:29:17.1188999Z 	at org.junit.jupiter.api.Assertions.fail(Assertions.java:161)
2025-02-14T18:29:17.1190526Z 	at org.apache.dubbo.remoting.transport.netty4.ReplierDispatcherTest.lambda$testMultiThread$1(ReplierDispatcherTest.java:110)
2025-02-14T18:29:17.1192086Z 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
2025-02-14T18:29:17.1192945Z 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
2025-02-14T18:29:17.1194873Z 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
2025-02-14T18:29:17.1196680Z 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
2025-02-14T18:29:17.1197637Z 	at java.base/java.lang.Thread.run(Thread.java:1583)
2025-02-14T18:29:17.1219619Z Caused by: java.util.concurrent.ExecutionException: org.apache.dubbo.remoting.RemotingException: Server side(127.0.0.1,36244) thread pool is exhausted, detail msg:Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-127.0.0.1:8080, Pool Size: 200 (active: 145, core: 200, max: 200, largest: 200), Task: 1499 (completed: 1356), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in empty://127.0.0.1:8080!
2025-02-14T18:29:17.1222475Z 	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
2025-02-14T18:29:17.1223247Z 	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
2025-02-14T18:29:17.1229921Z 	at org.apache.dubbo.remoting.transport.netty4.ReplierDispatcherTest.clientExchangeInfo(ReplierDispatcherTest.java:136)
2025-02-14T18:29:17.1231606Z 	at org.apache.dubbo.remoting.transport.netty4.ReplierDispatcherTest.lambda$testMultiThread$1(ReplierDispatcherTest.java:108)
2025-02-14T18:29:17.1232660Z 	... 5 more
2025-02-14T18:29:17.1235838Z Caused by: org.apache.dubbo.remoting.RemotingException: Server side(127.0.0.1,36244) thread pool is exhausted, detail msg:Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-127.0.0.1:8080, Pool Size: 200 (active: 145, core: 200, max: 200, largest: 200), Task: 1499 (completed: 1356), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in empty://127.0.0.1:8080!  
2025-02-14T18:29:17.1238048Z 	at org.apache.dubbo.remoting.exchange.support.DefaultFuture.doReceived(DefaultFuture.java:265)
2025-02-14T18:29:17.1239291Z 	at org.apache.dubbo.remoting.exchange.support.DefaultFuture.received(DefaultFuture.java:208)
2025-02-14T18:29:17.1240702Z 	at org.apache.dubbo.remoting.exchange.support.DefaultFuture.received(DefaultFuture.java:196)
2025-02-14T18:29:17.1241999Z 	at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleResponse(HeaderExchangeHandler.java:65)
2025-02-14T18:29:17.1243405Z 	at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:211)
2025-02-14T18:29:17.1244541Z 	at org.apache.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)
2025-02-14T18:29:17.1245709Z 	at org.apache.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:64)
2025-02-14T18:29:17.1246987Z 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
2025-02-14T18:29:17.1248107Z 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
2025-02-14T18:29:17.1249174Z 	at org.apache.dubbo.common.threadlocal.InternalRunnable.run(InternalRunnable.java:39)
2025-02-14T18:29:17.1249883Z 	... 1 more

@zrlw
Copy link
Contributor Author

zrlw commented Feb 22, 2025

maybe dubbo-test-check should be added back, at least add it back to dubbo-remoting-netty4 for unit testing result comparison.

@zrlw zrlw mentioned this issue Feb 22, 2025
Open
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component/need-triage Need maintainers to triage type/need-triage Need maintainers to triage
Projects
Dubbo Board
Archived in project
Milestone
No milestone
2 participants
@AlbumenJ @zrlw