Update performance page with latest data for 20.2

[johnrk-zz]

Hey @rmloveland , to confirm, we're committed from a docs perspective update the performance page if @aayushshah15 and @nvanbenschoten evaluate our existing performance benchmarks? They have already committed to re-running TPC-C 110k.

Here's a breakdown of what is expected to be specifically updated:

For our performance page, it would helpful for our users to have the latest performance numbers for TPC-C and other workloads:

TPC-C 150k (scheduled to be done by @aayushshah15 )

• Max Throughput 1,245,462 tpmC
• Max Warehouses with Max Efficiency 100,000 Warehouses
• Max Number of Rows
• Max Unreplicated Data
• Number of Nodes
• Node Machine Type

Linear Scaling

• Number of nodes Cockroach can achieve linear scaling (in 20.1 it was 256 nodes) - update this sentence if needed: "CockroachDB has no theoretical scaling limit and, in practice, can achieve near-linear performance at X nodes."

• KV 95 Throughput and latency table needs latest data:
  

• Sysbench Throughput Report (QPS) needs to be updated with the latest test results as well
  

• Lastly, Sysbench Average Latency needs to be updated with the latest:
  

Related: #8023

[rmloveland]

@johnrk sounds good to me, thank you for filing this!

[johnrk-zz]

@rmloveland , we're now doing TPC-C 150k

[rmloveland]

John Kendall <notifications@github.com> writes:

@rmloveland , we're now doing TPC-C 150k

OK! Hey @adityamaru I heard that you will be doing the big TPC-C run soon (this week?). As part of the work to update our performance docs I will also need to update the 20.2 docs with instructions for running TPC-C benchmarks, specifically: - TPC-C 1k: https://www.cockroachlabs.com/docs/v20.2/performance-benchmarking-with-tpc-c-1k-warehouses - TPC-C 10k: https://www.cockroachlabs.com/docs/v20.1/performance-benchmarking-with-tpc-c-10k-warehouses - TPC-C 100k (or whatever you end up with: 150k, etc.): https://www.cockroachlabs.com/docs/v20.1/performance-benchmarking-with-tpc-c-100k-warehouses What is the best way to coordinate with you to get the new versions of this information? Is there a doc with your benchmarking instructions written somewhere? Or perhaps a script you are using to do the TPC-C runs? Let me know if I should set up a quick sync with you to discuss. (PS sorry if this is very bad, late timing, I am working across a couple of areas and this fell off my radar for a bit.)

[johnrk-zz]

cc @aayushshah15

[aayushshah15]

Hey @rmloveland, what we've been doing so far is mostly following the set of instructions we have on the docs page for 19.2, while keeping track of where we've deviated from those instructions. We also have a benchmarking script we've been using, but it relies on CRL-specific tools like roachprod, would that still be helpful for you? After the run, we could send you a google doc that contains the updated instructions for running this stuff on 20.2 on non-roachprod clusters. How does that sound? If you have any suggestions / things you'd like us to keep in mind while doing the 150K run, we'll make sure to do so.

[aayushshah15]

In general, we haven't had to deviate too much from the instructions we already have for 19.2 (aside from a few launch flags & cluster settings). Thus, I expect the structure of the new TPCC perf page to be almost identical to the current one.

[rmloveland]

Aayush Shah <notifications@github.com> writes:

Hey @rmloveland, what we've been doing so far is mostly following the set of instructions we have on the docs page for 19.2, while keeping track of where we've deviated from those instructions. We also have a benchmarking script we've been using, but it relies on CRL-specific tools like roachprod, would that still be helpful for you?

It might be, it depends on what ends up in the google doc you mention below. From your description, sounds like I should start with the google doc as the source of truth for updating our perf docs, and ask questions as needed. Does that sound reasonable?

After the run, we could send you a google doc that contains the updated instructions for running this stuff on 20.2 on non-roachprod clusters. How does that sound?

That sounds amazing! Thanks for offering to do that, I know it's extra work.

If you have any suggestions / things you'd like us to keep in mind while doing the 150K run, we'll make sure to do so.

Mainly that the more detailed and specific the google doc is, the better. :-)

In general, we haven't had to deviate too much from the instructions we already have for 19.2 (aside from a few launch flags & cluster settings).

That's really good to know!

Thus, I expect the structure of the new TPCC perf page to be almost identical to the current one.

That would be an excellent outcome! Again, thank you for your work on this. Look forward to seeing the results (also, 150k, wow :-} ).

[jseldess]

Nit to consider while updating the benchmarking docs, @rmloveland: #6564