Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kv: don't pass clock information through Raft log #76095

Conversation

nvanbenschoten
Copy link
Member

This commit eliminates the primary mechanism that we use to pass clock information from a leaseholder, through Raft log entries, to a Range's followers. As we found in #72278, this was only needed for correctness in a few specific cases — namely lease transfers and range merges. These two operations continue to pass clock signals through more explicit channels, but we remove the unnecessary general case.

The allows us to remote one of the two remaining places where we convert a Timestamp to a ClockTimestamp through the TryToClockTimestamp method. As outlined in #72121 (comment), I would like to remove ability to downcast a "data-plane" Timestamp to a "control-plane" CloudTimestamp entirely. This will clarify the role of ClockTimestamps in the system and clean up the channels through which clock information is passed between nodes.

The other place where we cast from Timestamp to ClockTimesatmp is in Store.Send, at the boundary of KV RPCs. I would also like to get rid of this, but doing so needs to wait on #73732.

@nvanbenschoten nvanbenschoten requested a review from a team as a code owner February 5, 2022 00:07
@cockroach-teamcity
Copy link
Member

This change is Reviewable

@nvanbenschoten nvanbenschoten force-pushed the nvanbenschoten/removeRaftWriteTimestamp branch from d016065 to e4aa771 Compare February 5, 2022 00:46
Copy link
Member

@tbg tbg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 1 of 1 files at r1, 10 of 10 files at r2, all commit messages.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @nvanbenschoten)


pkg/kv/kvserver/replica_proposal.go, line 864 at r2 (raw file):

			res.Replicated.WriteTimestamp = ba.WriteTimestamp()
		} else {
			if !r.ClusterSettings().Version.IsActive(ctx, clusterversion.DontProposeWriteTimestampForLeaseTransfers) {

IsActive is moderately expensive (has to unmarshal a protobuf). I assume this code path is "cold enough" for this not to matter (since most requests will apply the timestamp cache) but wanted to point it out anyway.


pkg/kv/kvserver/kvserverpb/proposer_kv.proto, line 132 at r2 (raw file):

  // of the command against the GC threshold and to update the followers'
  // clocks. Only set if the request that produced this command is a write that
  // cares about the timestamp cache.

This is deprecated now, right?

@nvanbenschoten nvanbenschoten force-pushed the nvanbenschoten/removeRaftWriteTimestamp branch from e4aa771 to f16f5cd Compare February 5, 2022 19:10
This commit eliminates the primary mechanism that we use to pass clock
information from a leaseholder, through Raft log entries, to a Range's
followers. As we found in cockroachdb#72278, this was only needed for correctness
in a few specific cases — namely lease transfers and range merges. These
two operations continue to pass clock signals through more explicit
channels, but we remove the unnecessary general case.

The allows us to remote one of the two remaining places where we convert
a `Timestamp` to a `ClockTimestamp` through the `TryToClockTimestamp`
method. As outlined in cockroachdb#72121 (comment),
I would like to remove ability to downcast a "data-plane" `Timestamp` to
a "control-plane" `CloudTimestamp` entirely. This will clarify the role
of `ClockTimestamps` in the system and clean up the channels through
which clock information is passed between nodes.

The other place where we cast from `Timestamp` to `ClockTimesatmp` is
in `Store.Send`, at the boundary of KV RPCs. I would also like to get
rid of this, but doing so needs to wait on cockroachdb#73732.
@nvanbenschoten nvanbenschoten force-pushed the nvanbenschoten/removeRaftWriteTimestamp branch from f16f5cd to 74e2070 Compare February 5, 2022 23:34
Copy link
Member Author

@nvanbenschoten nvanbenschoten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TFTR!

bors r+

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @tbg)


pkg/kv/kvserver/replica_proposal.go, line 864 at r2 (raw file):

Previously, tbg (Tobias Grieger) wrote…

IsActive is moderately expensive (has to unmarshal a protobuf). I assume this code path is "cold enough" for this not to matter (since most requests will apply the timestamp cache) but wanted to point it out anyway.

Yeah, this is cold enough that unmarshalling a proto seems ok. Has IsActive gotten more expensive recently though? I wasn't aware.


pkg/kv/kvserver/kvserverpb/proposer_kv.proto, line 132 at r2 (raw file):

Previously, tbg (Tobias Grieger) wrote…

This is deprecated now, right?

Not fully. It's still used for real MVCC writes that use it to check against the GC threshold (below Raft in checkForcedErr), to assert against closed timestamp violations (assertNoWriteBelowClosedTimestamp), and now for AddSSTable requests with the WriteAtRequestTimestamp flag set.

This first use seemed dubious to me and Andrei, but our attempts to understand it enough to get rid of it were unsuccessful, so I decided not to pick that fight in this PR. The HLC is a formidable enough foe 1-on-1.

@craig craig bot merged commit a7b1c17 into cockroachdb:master Feb 8, 2022
@craig
Copy link
Contributor

craig bot commented Feb 8, 2022

Build succeeded:

@nvanbenschoten nvanbenschoten deleted the nvanbenschoten/removeRaftWriteTimestamp branch February 8, 2022 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants