Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Protocol Change Request] Improving Time Travel using In-Commit Timestamps #2532

Open
1 of 3 tasks
dhruvarya-db opened this issue Jan 17, 2024 · 1 comment
Open
1 of 3 tasks

[Protocol Change Request] Improving Time Travel using In-Commit Timestamps #2532

dhruvarya-db opened this issue Jan 17, 2024 · 1 comment
Labels
enhancement New feature or request
Milestone
3.2.0

Comments

@dhruvarya-db
Copy link
Collaborator

dhruvarya-db commented Jan 17, 2024 •
edited
Loading

Feature request

Overview

This feature request is about changing Delta commit timestamps to improve time travel.

Motivation

Delta currently relies on the file modification time to identify the timestamp of a commit. This timestamp is used for time travel queries, log cleanup, and staleness checks. However, file modification time is not a very reliable way of getting a timestamp — this can easily change when the files are copied/moved to another directory (e.g. for disaster recovery purposes) or when any manual fixes are performed to the Delta log. In such cases, time travel on the delta table breaks as of today. The possibility of non-monotonic file timestamps also adds lots of code complexity in Delta as we try to handle it heuristically in the best possible way.

Further details

We propose a new Writer feature that will require clients to generate a timestamp just before performing a commit and store it in the commit itself.

Compliant writers will ensure that the timestamp stored in Commit X+1 is always greater than Commit X. To be able to ensure this, the client will need to perform conflict detection for these timestamps.

  1. The writer will write this timestamp in the CommitInfo action. Furthermore, the writer will always write CommitInfo as the first action in a commit.
  2. Clients that understand these new timestamps will now read the commit file to get the actual timestamp. These timestamps will now be used for time travel queries and by other operations that use timestamps.

The detail proposal and the required protocol changes are sketched out in this doc.

Willingness to contribute

The Delta Lake Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature?

  • Yes. I can contribute this feature independently.
  • Yes. I would be willing to contribute this feature with guidance from the Delta Lake community.
  • No. I cannot contribute this feature at this time.
@dhruvarya-db dhruvarya-db added the enhancement New feature or request label Jan 17, 2024
@dhruvarya-db dhruvarya-db changed the title [Feature Request][Spark] Improving Time Travel using In-Commit Timestamps [Feature Request] Improving Time Travel using In-Commit Timestamps Jan 17, 2024
@dhruvarya-db dhruvarya-db mentioned this issue Jan 31, 2024
Closed
5 tasks
@tdas tdas mentioned this issue Feb 1, 2024
Closed
@dhruvarya-db dhruvarya-db changed the title [Feature Request] Improving Time Travel using In-Commit Timestamps [Protocol Change Request] Improving Time Travel using In-Commit Timestamps Feb 2, 2024
This was referenced Feb 2, 2024
Closed
Merged
Merged
5 tasks
allisonport-db pushed a commit that referenced this issue Mar 5, 2024
@dhruvarya-db @allisonport-db
Introduce InCommitTimestamp feature and write monotonically increasin…
b15a2c9 
…g timestamps in CommitInfo

Follow-up for #2532.

Adds a new writer feature called `inCommitTimestamp`. When this feature is enabled, the writer will make sure that it writes `commitTimestamp` in CommitInfo which contains a monotonically increasing timestamp.

This PR is an initial implementation, it does not handle timestamp retrieval efficiently. It does not try to populate the inCommitTimestamp in Snapshot even in places where it is already available, instead Snapshot has to perform an IO to read the timestamp.

Closes #2596

GitOrigin-RevId: 44904e734eee74378ee55f708beb29a484cd93e6
@dhruvarya-db dhruvarya-db mentioned this issue Mar 11, 2024
Merged
5 tasks
allisonport-db pushed a commit that referenced this issue Mar 12, 2024
@dhruvarya-db
[Spark] Rename lastCommitTimestamp to lastCommitFileModificationTimes…
3ae99ae 
…tamp (#2746)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [X] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->
Renames LogSegment.lastCommitTimestamp to
lastCommitFileModificationTimestamp so as to distinguish this file
timestamp from in-commit-timestamp.

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->
Related to #2532 . 

Name-only change.

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
No
@tdas tdas added this to the 3.2.0 milestone Mar 25, 2024
@dhruvarya-db
Copy link
Collaborator Author

This is being released as a preview feature in 3.2 (#2962). The feature will be generally available by 4.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
Linux Foundation Delta Lake Roadmap
Status: Todo
Milestone
3.2.0
Development

No branches or pull requests
2 participants
@tdas @dhruvarya-db