Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add closed timestamps to the architecture docs #6793

Closed
rmloveland opened this issue Mar 9, 2020 · 2 comments · Fixed by #11762
Closed

Add closed timestamps to the architecture docs #6793

rmloveland opened this issue Mar 9, 2020 · 2 comments · Fixed by #11762
Assignees
Labels
P-2 Normal priority; secondary task T-missing-info

Comments

@rmloveland
Copy link
Contributor

Currently, the only place where we mention closed timestamps is on the Follower Reads page (AFAICT).

We should find out where they live in the architecture and document them there as well.

@rmloveland rmloveland self-assigned this Mar 9, 2020
@rmloveland rmloveland added A-kv-storage P-2 Normal priority; secondary task T-missing-info labels Mar 9, 2020
@rmloveland
Copy link
Contributor Author

From a comment by @ajwerner on #6801:

This is a bit down in the weeds but a question that has come up before is where does this 4.8 seconds number come from. The answer is due to some cluster settings which we don't make public because we don't think they're easy to use. I'm going to explain how it works here and you can do with it what you will (which could very reasonably be nothing).

The answer is that it comes from the closed timestamp target duration and some other setting. The closed timestamp subsystem effectively makes history immutable. We have it do this periodically. The duration in the past we "close out" is controlled by kv.closed_timestamp.target_duration. We do this every kv.closed_timestamp.close_fraction of the duration. We want to have a high chance of actually hitting a follower read so we want some number of these fractional intervals for the closure to likely propagate.
> SELECT * FROM [SHOW ALL CLUSTER SETTINGS] WHERE variable LIKE '%closed%' OR variable LIKE '%follower%'; 
                        variable                  | value | setting_type | public |                                                                                                                                           description                                           
     ---------------------------------------------+-------+--------------+--------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
       kv.closed_timestamp.close_fraction         | 0.2   | f            | false  | fraction of closed timestamp target duration specifying how frequently the closed timestamp is advanced                                                                                         
       kv.closed_timestamp.follower_reads_enabled | true  | b            |  true  | allow (all) replicas to serve consistent historical reads based on closed timestamp information                                                                                                 
       kv.closed_timestamp.target_duration        | 3s    | d            | false  | if nonzero, attempt to provide closed timestamp notifications for timestamps trailing cluster time by approximately this duration                                                               
       kv.follower_read.target_multiple           | 3     | f            | false  | if above 1, encourages the distsender to perform a read against the closest replica if a request is older than kv.closed_timestamp.target_duration * (1 + kv.closed_timestamp.close_fraction * this) less a clock uncertainty interval. This value also is used to create follower_timestamp().

@nvanbenschoten
Copy link
Member

I stumbled upon this gap in our architecture docs while reading through it. It seems like a nice thing to add when we find the time. As we expand the uses of follower reads (including all reads on GLOBAL tables), closed timestamps will play an increasingly important role in our architecture.

rmloveland added a commit that referenced this issue Sep 22, 2021
Fixes #6793
Addresses #9040
A (soft?) prerequisite for #11016

Summary of changes:

- Update the 'Reading' section of the Transaction Layer page to list
  consistent vs. stale reads

- Add a new 'Closed timestamp' section to the Transaction Layer page

- Link to the above from the Follower Reads page
rmloveland added a commit that referenced this issue Sep 23, 2021
Fixes #6793

Addresses #9040

Also a prerequisite for #11016 (we can't really get into the details of
bounded staleness without more of this background info)

Summary of changes:

- Update the 'Reading' section of the Transaction Layer page to list
  consistent vs. stale reads

- Add a new 'Closed timestamp' section to the Transaction Layer page

- Link to the above from the Follower Reads page

- Update the transaction retry errors page to link to the new closed
  timestamp section of the Transaction Layer page
rmloveland added a commit that referenced this issue Sep 23, 2021
Fixes #6793

Addresses #9040

Also a prerequisite for #11016 (we can't really get into the details of
bounded staleness without more of this background info)

Summary of changes:

- Update the 'Reading' section of the Transaction Layer page to list
  consistent vs. stale reads

- Add a new 'Closed timestamp' section to the Transaction Layer page

- Link to the above from the Follower Reads page

- Update the transaction retry errors page to link to the new closed
  timestamp section of the Transaction Layer page

- Opportunistic update to the 'Timestamp Cache' section of the
  Transaction Layer page based on the contents of the blog post 'An Epic
  Read on Follower Reads'[1] (on which this PR is based)

- Update the 'Read refreshing' section of the Transaction Layer page to
  add links to closed timestamps and to link to more docs on the
  transaction retry error type mentioned there.

[1]: https://www.cockroachlabs.com/blog/follower-reads-stale-data/
rmloveland added a commit that referenced this issue Sep 29, 2021
Fixes #6793

Addresses #9040

Also a prerequisite for #11016 (we can't really get into the details of
bounded staleness without more of this background info)

Summary of changes:

- Update the 'Reading' section of the Transaction Layer page to list
  consistent vs. stale reads

- Add a new 'Closed timestamp' section to the Transaction Layer page

- Link to the above from the Follower Reads page

- Update the transaction retry errors page to link to the new closed
  timestamp section of the Transaction Layer page

- Opportunistic update to the 'Timestamp Cache' section of the
  Transaction Layer page based on the contents of the blog post 'An Epic
  Read on Follower Reads'[1] (on which this PR is based)

- Update the 'Read refreshing' section of the Transaction Layer page to
  add links to closed timestamps and to link to more docs on the
  transaction retry error type mentioned there.

[1]: https://www.cockroachlabs.com/blog/follower-reads-stale-data/
rmloveland added a commit that referenced this issue Sep 30, 2021
Fixes #6793

Addresses #9040

Also a prerequisite for #11016 (we can't really get into the details of
bounded staleness without more of this background info)

Summary of changes:

- Update the 'Reading' section of the Transaction Layer page to list
  consistent vs. stale reads

- Add a new 'Closed timestamp' section to the Transaction Layer page

- Link to the above from the Follower Reads page

- Update the transaction retry errors page to link to the new closed
  timestamp section of the Transaction Layer page

- Opportunistic update to the 'Timestamp Cache' section of the
  Transaction Layer page based on the contents of the blog post 'An Epic
  Read on Follower Reads'[1] (on which this PR is based)

- Update the 'Read refreshing' section of the Transaction Layer page to
  add links to closed timestamps and to link to more docs on the
  transaction retry error type mentioned there.

[1]: https://www.cockroachlabs.com/blog/follower-reads-stale-data/

(Added to v21.1 and v21.2 docs)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P-2 Normal priority; secondary task T-missing-info
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants