workload/tpcc: use multiple column families to avoid contention

[nvanbenschoten]

This commit adds multiple column families to the warehouse,
district, and customer tables. These column families split
static columns up from columns that are modified by transactions.

This effectively removes all contention points between the NewOrder
and Payment transactions. The transactions will still contend with
other instances of their same txn type, but that's less of an issue
because they should end up being serialized early on, meaning than
intra-txn contention should rarely lead to transaction retries (for
instance, NewOrder serializes on district.d_next_o_id immediately).
These transactions are run about 90% of the time, so this should be a
performance win, especially without wait times.

Release justification: testing only

[cockroach-teamcity]

This change is 

[nvanbenschoten]

Also FWIW, my investigation in #30213 revealed that transaction restarts were directly correlated with 99th percentile latency in TPC-C. The graphs were almost identical. Tracing confirmed this, as I could see that nearly all long-running txns were restarted at least once. The situation I saw most often in the traces was newOrder txns being restated due to a contending payment txn.

I mentioned above that:

The transactions will still contend with
other instances of their same txn type, but that's less of an issue
because they should end up being serialized early on, meaning than
intra-txn contention should never lead to transaction retries

I'll clarify that intra-txn contention will still lead to transaction retries due to the read-modify-write operation that they often perform when incrementing counters (see #22778 and #29431), but these are performed as the first statement of the newOrder and payment txns. This means that txn restarts here should always be retriable on the server, so this PR eliminates client-side retries for all txns except the delivery txn (which is rare). That should make this a major performance win.

[nvanbenschoten]

This is kind of a crazy idea, but this change would actually allow us to remove the client-side retry loops for all txns except the delivery txn. As discussed in #28105, this retry loop imposes 2 extra client-to-gateway round trips on each transaction. Since we're CPU bound on the upper ends of TPC-C, removing it might actually provide a moderate performance improvement because it would reduce network traffic each gateway handles and the amount of SQL parsing that it needs to perform.

[nvanbenschoten]

I manually verified that without foreign keys, this change, in conjunction with the fixes for #18168 and #30618, break the dependency between newOrder txns and payment txns. The two transactions can be run on the same district and customer without blocking on each other.

In my limited performance testing on my laptop with --wait=false, I found that the change resulted in somewhere between a 4-5% throughput improvement. Note that this was comparing between #30618 being fixed and it not being fixed. The dataset had multiple column families in both versions of the comparison, so it's possible that the overhead of the column families themselves negates some or all of the benefits of this change. We'll have to measure this. On the other hand, we expect txn retries and txn contention to be more expensive in a distributed cluster, so it's also possible that this is an underestimate of the speedup this will provide.

However, the dependency between the txns was not broken with fks enabled. This is because of #30852. newOrder updates a secondary column familiy in the district table and payment inserts into history, which has the fk foreign key (h_w_id, h_d_id) references district (d_w_id, d_id). The corresponding fk check should not need to query all column families in district, but at the moment, it does.

[jordanlewis]

@rohany @solongordon see the last point in @nvanbenschoten's last paragraph. Once the optimizer plans FKs, solving #38280 for lookup joins should break the last dependency here.

[jordanlewis]

Actually, #30618 hasn't been solved yet either (making sure UPDATE RETURNING doesn't fetch too many column families). I think that's on the optimizer team, though. @RaduBerinde is this issue on the horizon?

[RaduBerinde]

Not really, but I can prioritize it.

[nvanbenschoten]

I'm reviving this PR, with slight modifications to the column families so that this all works properly with the handling column families in the optimizer, in foreign key checks, and with implicit SFU locking. With those changes in place, this change achieves its goal of removing all contention points between the NewOrder and Payment transactions.

This showed up very prominently in some of the tests I've been running recently. Specifically, I found that a great way to measure the impact of contention on a workload is to run the workload on our very own high-performance, in-memory database – cockroach demo. Doing so speeds everything else up (after disabling wait times in TPC-C), so that contention becomes significantly more pronounced. With that change in effect, I found that this change almost triples the maximum throughput we can sustain on a single warehouse, increasing from a little less than 2200 tpmC to a little over 5000.

Or course, standard TPC-C with wait times and proper durability isn't nearly as contended, but avoiding dependencies between these transactions still helps. We see this in a complimentary PR I'm about to open.

@jordanlewis do you mind giving this another pass?

[nvanbenschoten]

@jordanlewis friendly ping on this, now that we're close to the end of the stability period.

[jordanlewis]

Yeah, sorry for missing this last time.

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @jordanlewis)

[jordanlewis]

, though I didn't review the column family choices carefully myself . Are you going to regenerate the fixtures and so on? That's probably the most annoying thing about this change, but important, right?

Reviewed 6 of 6 files at r1.
Reviewable status: complete! 1 of 0 LGTMs obtained

[nvanbenschoten]

Thanks for taking a look!

Are you going to regenerate the fixtures and so on?

Nope! We recently ripped out all uses of TPC-C fixtures in perf-related tests in favor of IMPORT, specifically so that they no longer need to be regenerated whenever we make minor tweaks like this. The need to regenerate fixtures was a major impediment to tuning the workload, and that didn't seem well justified.

bors r+

[craig]

Build succeeded:

• GitHub CI (Cockroach)