Use Java Virtual Threads (Loom) (500USD Bounty) #90
Comments
|
I can create a PR for this if this seems like a good idea to @lihaoyi |
|
Go for it, though note we'll need some compatibility story for Java <21 |
|
Is this something that #74 unblocks? |
lihaoyi
changed the title
|
I would recommend against this until JAVA-23 is released, du to the "pinning issue", effectively making Virtual Threads useless if your code/libs uses |
|
Doing it opt-in should be fine. We can run benchmarks etc. without synchronized to see the benefit. That way we can be ready on day one when Java 23 lands with a fix to the thread pinning issue. |
|
I have done this at $work for all my services, with Java 21, it works fine, no need to delay to Java 23. |
|
If there are issues because of undertow, you may consider switching to Helidon, which is the new web server from Oracle, built for working with virtual threads. https://github.com/helidon-io/helidon/tree/main/webserver https://central.sonatype.com/artifact/io.helidon.webserver/helidon-webserver/overview https://javadoc.io/doc/io.helidon.webserver/helidon-webserver/ |
|
FYI for anyone reading, this bounty is still open in case anyone else wants to take a crack at this |
|
@lihaoyi Since I started this issue, I am going to finish it too. :) |
|
while building cask, twirl throws java.lang.reflect.InvocationTargetException |
I get that too when building on JDK >= 21. Maybe the newer 2.0.x works better. |
|
Looks like it wasn't twirl afterall (probably, just a cached coincidence when I switched JDK version). See #154. |
|
@sorumehta @jodersky I bought a MacBook for this :) . |
|
@sideeffffect As all the dispatching layers are done in cask, I think Netty/Vert.x is an option, too? |
Motivation: Add Loom support. refs: #90 , again Modification: 1. add `handlerExecutor()` and some helper methods for virtual threads. 2. some documents. Result: Virtual threads supported. `wrk` is needed to run the benchmark - miniAppWithSleep 100ms : ↑2300% - todoDb: ~↓6% - staticFiles: ~↓6% ```shell ./mill --no-build-lock benchmark.runBenchmarks ``` Results of same 4 Carrier/Platform threads: ```scala [1] staticFilesWithLoom result with (platform threads): [1] Running 30s test @ http://localhost:8080/ [1] 4 threads and 100 connections [1] Thread Stats Avg Stdev Max +/- Stdev [1] Latency 1.23ms 436.88us 28.64ms 98.91% [1] Req/Sec 20.54k 1.08k 22.69k 93.44% [1] 2461250 requests in 30.10s, 342.70MB read [1] Requests/sec: 81766.27 [1] Transfer/sec: 11.38MB [1] [1] staticFilesWithLoom result with (virtual threads): [1] Running 30s test @ http://localhost:8080/ [1] 4 threads and 100 connections [1] Thread Stats Avg Stdev Max +/- Stdev [1] Latency 4.41ms 14.69ms 157.91ms 95.15% [1] Req/Sec 19.00k 4.29k 22.09k 88.63% [1] 2266289 requests in 30.02s, 315.55MB read [1] Requests/sec: 75488.32 [1] Transfer/sec: 10.51MB [1] [1] todoDbWithLoom result with (platform threads): [1] Running 30s test @ http://localhost:8080/ [1] 4 threads and 100 connections [1] Thread Stats Avg Stdev Max +/- Stdev [1] Latency 1.22ms 172.42us 11.67ms 96.46% [1] Req/Sec 20.62k 1.29k 42.72k 93.01% [1] 2466143 requests in 30.10s, 395.12MB read [1] Non-2xx or 3xx responses: 2466143 [1] Requests/sec: 81929.40 [1] Transfer/sec: 13.13MB [1] [1] todoDbWithLoom result with (virtual threads): [1] Running 30s test @ http://localhost:8080/ [1] 4 threads and 100 connections [1] Thread Stats Avg Stdev Max +/- Stdev [1] Latency 3.97ms 13.20ms 160.29ms 95.28% [1] Req/Sec 19.21k 3.68k 22.32k 89.41% [1] 2284539 requests in 30.03s, 366.02MB read [1] Non-2xx or 3xx responses: 2284539 [1] Requests/sec: 76072.80 [1] Transfer/sec: 12.19MB [1] [1] minimalApplicationWithLoom result with (platform threads): [1] Running 30s test @ http://localhost:8080/ [1] 4 threads and 100 connections [1] Thread Stats Avg Stdev Max +/- Stdev [1] Latency 1.05s 575.57ms 1.99s 57.89% [1] Req/Sec 10.05 3.94 30.00 81.66% [1] 1152 requests in 30.03s, 172.12KB read [1] Socket errors: connect 0, read 0, write 0, timeout 1076 [1] Requests/sec: 38.36 [1] Transfer/sec: 5.73KB [1] [1] [1] minimalApplicationWithLoom result with (virtual threads): [1] Running 30s test @ http://localhost:8080/ [1] 4 threads and 100 connections [1] Thread Stats Avg Stdev Max +/- Stdev [1] Latency 106.11ms 2.74ms 126.59ms 77.47% [1] Req/Sec 239.19 36.74 252.00 92.68% [1] 28100 requests in 30.03s, 4.10MB read [1] Requests/sec: 935.74 [1] Transfer/sec: 139.81KB ``` Some design choices: 1. Using MethodHandle/Reflect to make it compile on Java 8 too. 2. Using MethodHandle to name the virtual threads that are needed, JPMS code can be added to open the can by default, but that is a little over-killed, so better with an explicitly `--add-opens java.base/java.lang=ALL-UNNAMED` 3. Add a virtualize or screen method to create a virtual thread executor from a **platform** thread pool, this is useful, especially if you want to limit the underlying queue size, FJP is unbounded. 4. Users can override the `handleExecutor` directly too.
|
@lihaoyi since it was my idea, I deserve a dollar too :P |
|
@sorumehta BTW, I was superized by the performance, which is quite good, at least many systems only have < 1qps. |
Spawn a virtual thread for each incoming request, which would enable asynchronous behaviour and increase performance of the library. The new Java virtual threads are lightweight and can replace the traditional Java threads. Given that Cask does not have any strong async/concurrency model of its own, leveraging the new JVM feature would be really nice.
Haoyi: To incentivize contribution, I'm putting a 500USD bounty on resolving this ticket. This is payable via bank transfer, and at my discretion in case of ambiguity. The acceptance criteria is a PR implementing support for virtual threads, tests exercising them, and performance benchmarks (ad-hoc is OK) demonstrating a performance improvement (e.g. in high concurrency use cases) vs normal threads in Java 21+
The text was updated successfully, but these errors were encountered: