Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

trim WAL on safekeepers #1403

Closed
4 tasks done
Tracked by #543
stepashka opened this issue Mar 24, 2022 · 2 comments
Closed
4 tasks done
Tracked by #543

trim WAL on safekeepers #1403

stepashka opened this issue Mar 24, 2022 · 2 comments
Assignees
@arssher
Labels
c/storage/pageserver Component: storage: pageserver c/storage/safekeeper Component: storage: safekeeper
Milestone
1.0 Technical pre...

Comments

@stepashka
Copy link
Member

stepashka commented Mar 24, 2022 •
edited by arssher
Loading

DOD:
Safekeepers that are not in touch with the pageserver can coordinate and delete WAL after one them confirms that WAL is stored on pageserver and S3.

We need to keep enough WAL for

  • safekeper p2p recovery
  • direct S3 offloading from safekeepers
  • the last checkpoint offloaded to s3 (remote consistent LSN)
@stepashka stepashka added c/storage/safekeeper Component: storage: safekeeper c/storage/pageserver Component: storage: pageserver labels Mar 24, 2022
@stepashka stepashka added this to the 0.6 Towards Tech Prev milestone Mar 24, 2022
@stepashka stepashka mentioned this issue Mar 24, 2022
Closed
10 tasks
@arssher arssher mentioned this issue Apr 20, 2022
Merged
arssher added a commit that referenced this issue Apr 26, 2022
@arssher
Remove old WAL on safekeepers.
791cf60 
Remove when it is consumed by all of 1) pageserver (remote_consistent_lsn) 2)
safekeeper peers 3) s3 WAL offloading.

In test s3 offloading for now is mocked by directly bumping s3_wal_lsn.

ref #1403
arssher added a commit that referenced this issue Apr 26, 2022
@arssher
Remove old WAL on safekeepers.
8b9d523 
Remove when it is consumed by all of 1) pageserver (remote_consistent_lsn) 2)
safekeeper peers 3) s3 WAL offloading.

In test s3 offloading for now is mocked by directly bumping s3_wal_lsn.

ref #1403
@neondatabase-bot neondatabase-bot bot modified the milestone: 1.0 Technical preview May 6, 2022
@arssher
Copy link
Contributor

arssher commented May 11, 2022 •
edited
Loading

Now on staging WAL is removed without waiting for s3 offloading, and there is global (for all timelines) configuration on safekeeper for that. FYI @stepashka
#1662

@arssher
Copy link
Contributor

arssher commented Jun 7, 2022

s3 offloading is enabled on staging, WAL removal happens after it.

@arssher arssher closed this as completed Jun 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@arssher arssher

Labels
c/storage/pageserver Component: storage: pageserver c/storage/safekeeper Component: storage: safekeeper
Projects
None yet
Milestone
1.0 Technical preview
Development

No branches or pull requests
2 participants
@stepashka @arssher