Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase the concurrency of some tests. #5626

Open
an-tao opened this issue Apr 21, 2020 · 2 comments
Open

Increase the concurrency of some tests. #5626

an-tao opened this issue Apr 21, 2020 · 2 comments

Comments

@an-tao
Copy link
Contributor

an-tao commented Apr 21, 2020

I found in the db and fortune test that the number of client connections is not enough to measure the highest performance of some frameworks (especially the frameworks at the top of the list). I downloaded the latest results of tfb and plotted a chart for each of the above tests. as follows:

截屏2020-04-21 下午9 07 16

截屏2020-04-21 下午8 59 08

Obviously, as long as the concurrency of the test is increased, we can get higher performance of some frameworks(Those frameworks whose qps curve is still rising rapidly). so my suggustion is add new tests for `db` and `fortune` with 1024 and 2048 connections.
@joanhey
Copy link
Contributor

joanhey commented May 19, 2020

It would be good, but it has disadvantages.

  • More time on every run (+ hours)
  • A lot of frameworks drop about 128 or less concurrent connections, so a lot of hours wasted.

but ...

A benchmark need to measure performance and scalabilty

The good solution will be to have a graph to see the data table, in all tests.
But that need more work.

Another fast implementation, will be to add a ⬆️ and 🔽 .

  • #reqs ⬆️ to indicate that still scale
  • #reqs 🔽 to indicate that didn't scale to 512 concurrent connections
    a lot of frameworks are faster with 128 or 256 concurrent and slower with 512.
    But still show faster than others fws scaling better with 512 in the benchmark.

The benchmark only show the faster, when others frameworks scale better and are faster with 512 or more concurrent connections.

A simple change in the fortunes or db, showing only the req/s at 512 concurrency, like do with multiple query and updates, the ranking will change completely .
Perhaps will be good to have this option, show only faster and faster at 512 concurrency. Or show both.

@an-tao
Copy link
Contributor Author

an-tao commented May 20, 2020

What do you think about removing the 16 and 32 concurrency and adding 1024 and 2048?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants