Replies: 3 comments 7 replies
-
|
@nbrady-techempower Let me show you the numbers of running a branch of techempower with server container's restarted (naively implemented, but it's just to show the difference) vs master. Just as reminder , the order of execution of tests:
The plaintext one, given that's the last one and due to how the JVM compilation works, can have a huge variance in its results because of the previous executed (very different) workloads, see:
these results coming from our CI (that's using bare metal/idle/stable hw):
If you have a CI there, you can reproduce yourself just by running a plaintext alone vs all and checking plaintext results. |
Beta Was this translation helpful? Give feedback.
-
|
@nbrady-techempower any news on this? |
Beta Was this translation helpful? Give feedback.
-
|
i think it is better to re-use the existing server for all tests as it currently does - this is the intended purpose of the benchmarks afaict - to test frameworks across a wide range of different tasks, not to see how optimized they can be for each specific task. there is at least one framework already that re-compiles itself based on profiling information so it can better fit the expected load of the test. imho, this tends to obscure any advantage dynamic/JIT languages have over statically compiled ones, which should be something we can learn from these benchmarks. 🤷♂️ |
Beta Was this translation helpful? Give feedback.
-
|
Re
Not sure is what a benchmark would look like, unless the purpose of the benchmark is to stress how good the JIT is to deoptimize and recompile code too (and simpler frameworks obviously are going to benefit by this ^^). If the purpose of the benchmark, instead is to emulate realistic scenarios we should:
|
Beta Was this translation helpful? Give feedback.
-
|
Just noticed that #7475 (comment) mentioned about this as well |
Beta Was this translation helpful? Give feedback.
-
|
i agree with your suggestions re. emulating realistic scenarios! to be fair i think there has been much confusion about the purpose of the TE benchmarks but the TE team has made it quite clear that the primary purpose is to provide a "level playing field" and give a picture of relative performance of frameworks across a range of different scenarios/loads. personally, i would like to be able to see more techniques for max performance but i'm just not sure this fits with the stated objectives of the TE team. re. your JIT issues outlined above - that sounds like very odd behaviour from the jvm - a 2x performance increase for JIT code fitted to a specific test/route? how realistic is it that you would ever build something like this out in the world? surely you want to see how your framework/server/platform works across a variety of different loads rather than having to build a specific server and have the JIT engine optimize if for a single route only? 🤷♂️ |
Beta Was this translation helpful? Give feedback.
-
Yep but that doesn't mean that the same runtime env (always on) must be reused across the range of loads, because it won't make results reproducible (unless proven differently) depending by duration and order of tests, regardless the issue we're hitting for the quarkus framework.
I agree - it seems a kind of overfitting compilation issue on the JIT due to validation runs order making the code to compile in a way that fit very badly with network-limited use cases
From this pov I don't think the all-out-throughput with single workload running for X seconds with Y pipeline level to be realistic as well Isolation is a requirement for reproducibility from any benchmark (as per any scientific experiment); if we give up on this, it's fine, but I would like to understand why and how it relates with the nature of the TE benchmark - although native compiled applications (not using PGO, for example) won't benefit from a compiled code point of view, they would benefit from a reduced memory fragmentation, while GC-backed applications (assuming a copy/moving/compacting GC) won't get any benefit; it's a trade-off that would benefit differently both application types, while keeping results more repeatable. Said that, I agree that the issue we're seeing is pretty weird, but I can tell quarkus isn't the sole affected Java framework; I invite the other authors to share their opinions if they thing it worth to be clarified :) |
Beta Was this translation helpful? Give feedback.
-
Absolutely, and that's part of the problem: it's extremely unrealistic for a server application to be deployed and then be used for some time perfoming exclusively workloads of type A, and then being repurposed (while running the same code) to run exclusively workloads of type B. In a "normal" context, the same application would be handling requests of A anb B concurrently, or at least partially in overlap. |
Beta Was this translation helpful? Give feedback.
-
|
But now there are fws than use a different variant per test, and benefiting from the JIT. So we need to clarify the rules. |
Beta Was this translation helpful? Give feedback.
-
|
@billywhizz @joanhey let me report this as conclusion of our journey to fix the weird JIT behaviour we have seen And funny enough, it's very likely that many frameworks are affected ;) have fun! |
Beta Was this translation helpful? Give feedback.
-
Reading the documentation it seems that the server (docker) istance is kept running across different test types (eg db, update, json, plaintext,...).
Both native and JIT technology stacks can have trouble to keep performance stable across different runs of very different test types:
To keep results between types more stable would be better to restart the server (docker) machine before each validation/warmup/prime run, in order to have repeatable results independent by the order of test types, wdyt?
Beta Was this translation helpful? Give feedback.
All reactions