Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Write a tool that can be used to find the mismatched key values that cause mismatched fingerprints #103072

Closed
rhu713 opened this issue May 10, 2023 · 2 comments
Closed

Write a tool that can be used to find the mismatched key values that cause mismatched fingerprints #103072

rhu713 opened this issue May 10, 2023 · 2 comments
Assignees
@msbutler
Labels
A-disaster-recovery C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-disaster-recovery

Comments

@rhu713
Copy link
Contributor

rhu713 commented May 10, 2023 •
edited by cockroach-jira-scripts
Loading

In backup/restore tests we often use SHOW EXPERIMENTAL_FINGERPRINTS when comparing backed up and restored data. But in the case where fingerprints are mismatched, it's often difficult to determine exactly which keys or values caused the mismatch, especially for large datasets. Write a tool that can be used to bisect and find the mismatch.

Epic: CRDB-27758

Jira issue: CRDB-27822
@rhu713 rhu713 added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) A-disaster-recovery labels May 10, 2023
@blathers-crl
Copy link

blathers-crl bot commented May 10, 2023

cc @cockroachdb/disaster-recovery

@msbutler msbutler self-assigned this May 26, 2023
@msbutler msbutler mentioned this issue May 30, 2023
Merged
msbutler added a commit to msbutler/cockroach that referenced this issue May 30, 2023
@msbutler
testutils: add fingerprint utility package
7bd9ca5 
This patch adds a couple helper methods that make it easier to: fingerprint a
table, database or cluster, compare fingerprints, and identify a mismatch on
the table level. All fingerprinting methods call crdb_internal.fingerprint() on
the table span level with the user provided options.

This patch also integrates the new methods in the backupccl and streamingccl
codebases.

A future patch could add a helper method that identifies mismatched keys within
a mismatched table.

Informs cockroachdb#103072

Release note: None
@msbutler msbutler mentioned this issue Jun 1, 2023
Merged
msbutler added a commit to msbutler/cockroach that referenced this issue Jun 1, 2023
@msbutler
c2c: allow user to pass start time as a decimal to crdb_internal.fing…
90287ed 
…erprint()

Previously, the user could only pass the startTime variable to
crdb_internal.fingerprint() as a TimestampTZ, which only has microsecond
precision. To enable nanosecond precision, this patch allows the user to pass
the startTime as a decimal. This patch is consistent with the user's ability to
pass and AOST timestamp as a decimal as well.

Informs cockroachdb#103072

Release note: None
msbutler added a commit to msbutler/cockroach that referenced this issue Jun 10, 2023
@msbutler
testutils: add fingerprint utility package
08905df 
This patch adds a couple helper methods that make it easier to: fingerprint a
table, database or cluster, compare fingerprints, and identify a mismatch on
the table level. All fingerprinting methods call crdb_internal.fingerprint() on
the table span level with the user provided options.

This patch also integrates the new methods in the backupccl and streamingccl
codebases.

A future patch could add a helper method that identifies mismatched keys within
a mismatched table.

Informs cockroachdb#103072

Release note: None
msbutler added a commit to msbutler/cockroach that referenced this issue Jun 12, 2023
@msbutler
c2c: allow user to pass start time as a decimal to crdb_internal.fing…
bb45084 
…erprint()

Previously, the user could only pass the startTime variable to
crdb_internal.fingerprint() as a TimestampTZ, which only has microsecond
precision. To enable nanosecond precision, this patch allows the user to pass
the startTime as a decimal. This patch is consistent with the user's ability to
pass and AOST timestamp as a decimal as well.

Informs cockroachdb#103072

Release note: None
craig bot pushed a commit that referenced this issue Jun 13, 2023
@msbutler
Merge #104219
b6262d5 
104219: c2c: allow user to pass start time as a decimal to crdb_internal.fingerprint() r=msbutler a=msbutler

Previously, the user could only pass the startTime variable to crdb_internal.fingerprint() as a TimestampTZ, which only has microsecond precision. To enable nanosecond precision, this patch allows the user to pass the startTime as a decimal. This patch is consistent with the user's ability to pass and AOST timestamp as a decimal as well.

Informs #103072

Release note: None

Co-authored-by: Michael Butler <butler@cockroachlabs.com>
msbutler added a commit to msbutler/cockroach that referenced this issue Jun 13, 2023
@msbutler
testutils: add fingerprint utility package
bad4864 
This patch adds a couple helper methods that make it easier to: fingerprint a
table, database or cluster, compare fingerprints, and identify a mismatch on
the table level. All fingerprinting methods call crdb_internal.fingerprint() on
the table span level with the user provided options.

This patch also integrates the new methods in the backupccl and streamingccl
codebases.

A future patch could add a helper method that identifies mismatched keys within
a mismatched table.

Informs cockroachdb#103072

Release note: None
@msbutler msbutler mentioned this issue Jun 13, 2023
Merged
msbutler added a commit to msbutler/cockroach that referenced this issue Jun 14, 2023
@msbutler
c2c: allow user to pass start time as a decimal to crdb_internal.fing…
a76c5b4 
…erprint()

Previously, the user could only pass the startTime variable to
crdb_internal.fingerprint() as a TimestampTZ, which only has microsecond
precision. To enable nanosecond precision, this patch allows the user to pass
the startTime as a decimal. This patch is consistent with the user's ability to
pass and AOST timestamp as a decimal as well.

Informs cockroachdb#103072

Release note: None
craig bot pushed a commit that referenced this issue Jun 14, 2023
@msbutler @yuzefovich @erikgrinaker
Merge #104089 #104828 #104888
ae25f4d 
104089: testutils: add fingerprint utility package r=stevendanna a=msbutler

This patch adds a couple helper methods that make it easier to: fingerprint a table, database or cluster, compare fingerprints, and identify a mismatch on the table level. All fingerprinting methods call crdb_internal.fingerprint() on the table span level with the user provided options.

This patch also integrates the new methods in the backupccl and streamingccl codebases.

A future patch could add a helper method that identifies mismatched keys within a mismatched table.

Informs #103072

Release note: None

104828: base: remove OptionalNodeIDErr r=yuzefovich a=yuzefovich

This method was only used in one place where we need to get the SQL instance ID of the gateway. That place has been refactored to pass that ID explicitly from the DistSQLPlanner.

Addresses: #100826.
Epic: None

Release note: None

104888: kvserver: clarify `kv.raft_log.disable_synchronization_unsafe` r=erikgrinaker a=erikgrinaker

This setting not only disables fsync, it also disables flushing writes to the OS, so it will lose data even on process crashes.

Epic: none
Release note: None

Co-authored-by: Michael Butler <butler@cockroachlabs.com>
Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com>
Co-authored-by: Erik Grinaker <grinaker@cockroachlabs.com>
blathers-crl bot pushed a commit that referenced this issue Jun 14, 2023
@msbutler
testutils: add fingerprint utility package
df6713c 
This patch adds a couple helper methods that make it easier to: fingerprint a
table, database or cluster, compare fingerprints, and identify a mismatch on
the table level. All fingerprinting methods call crdb_internal.fingerprint() on
the table span level with the user provided options.

This patch also integrates the new methods in the backupccl and streamingccl
codebases.

A future patch could add a helper method that identifies mismatched keys within
a mismatched table.

Informs #103072

Release note: None
@blathers-crl blathers-crl bot mentioned this issue Jun 14, 2023
Merged
msbutler added a commit to msbutler/cockroach that referenced this issue Jun 22, 2023
@msbutler
testutils: add fingerprint utility package
6ce335d 
This patch adds a couple helper methods that make it easier to: fingerprint a
table, database or cluster, compare fingerprints, and identify a mismatch on
the table level. All fingerprinting methods call crdb_internal.fingerprint() on
the table span level with the user provided options.

This patch also integrates the new methods in the backupccl and streamingccl
codebases.

A future patch could add a helper method that identifies mismatched keys within
a mismatched table.

Informs cockroachdb#103072

Release note: None
@msbutler
Copy link
Collaborator

closing on #104918

To compare keys, the test needs to grab backups of the two states that ought to be equal, like #110033

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@msbutler msbutler

Labels
A-disaster-recovery C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-disaster-recovery
Projects
No open projects
Disaster Recovery Backlog
Archived in project
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rhu713 @msbutler