refactor: single phase Timeline::load_layer_map #5074
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
no need to publish the Timeline before sorting out it's layers.
it's a cool pattern but really cements any design.
which we expect to the be the same as from metadata, as in, act of creating a timeline does not advance the disk_consistent_lsn.
koivunej
requested review from
knizhnik and
LizardWizzard
and removed request for
a team
koivunej
changed the title
koivunej
commented
Aug 23, 2023
koivunej
commented
Aug 23, 2023
koivunej
commented
Aug 23, 2023
1624 tests run: 1550 passed, 0 failed, 74 skipped (full report)The comment gets automatically updated with the latest test results
8854f71 at 2023-08-24T12:52:21.627Z :recycle: |
arpad-m
approved these changes
Aug 23, 2023
confusing situation with test_remote_storage_upload_queue_retries documented on the PR discussion. an image layer was remote only and it took me again a while to recollect how is this possible to have future image layers (it is because we can create them for not yet flushed to disk lsns).
Co-authored-by: Christian Schwarz <christian@neon.tech>
Co-authored-by: Christian Schwarz <christian@neon.tech>
jcsp
reviewed
Aug 23, 2023
jcsp
reviewed
Aug 23, 2023
jcsp
reviewed
Aug 23, 2023
|
I really like the overall style of this, makes the code substantially clearer |
jcsp
approved these changes
Aug 23, 2023
|
Only added the one change after your approvals, not waiting for another round. |
koivunej
added a commit
that referenced
this pull request
Aug 24, 2023
I've personally forgotten why/how can we have future layers during reconciliation. Adds `#[cfg(feature = "testing")]` logging when we upload such index_part.json, with a cross reference to where the cleanup happens. Latest private slack thread: https://neondb.slack.com/archives/C033RQ5SPDH/p1692879032573809?thread_ts=1692792276.173979&cid=C033RQ5SPDH Builds upon #5074. Should had been considered on #4837.
This was referenced Aug 24, 2023
koivunej
added a commit
that referenced
this pull request
Oct 26, 2023
…#4938) Implement a new `struct Layer` abstraction which manages downloadness internally, requiring no LayerMap locking or rewriting to download or evict providing a property "you have a layer, you can read it". The new `struct Layer` provides ability to keep the file resident via a RAII structure for new layers which still need to be uploaded. Previous solution solved this `RemoteTimelineClient::wait_completion` which lead to bugs like #5639. Evicting or the final local deletion after garbage collection is done using Arc'd value `Drop`. With a single `struct Layer` the closed open ended `trait Layer`, `trait PersistentLayer` and `struct RemoteLayer` are removed following noting that compaction could be simplified by simply not using any of the traits in between: #4839. The new `struct Layer` is a preliminary to remove `Timeline::layer_removal_cs` documented in #4745. Preliminaries: #4936, #4937, #5013, #5014, #5022, #5033, #5044, #5058, #5059, #5061, #5074, #5103, epic #5172, #5645, #5649. Related split off: #5057, #5134.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Current implementation first calls
load_layer_map, which loads all local layers, cleans up files, leave cleaning up stuff to "second function". Then the "second function" is finally called, it does not do the cleanup and some of the first functions setup can torn down. "Second function" is actually bothreconcile_with_remoteandcreate_remote_layers.This change makes it a bit more verbose but in one phase with the following sub-steps:
Needed by #4938.
It was also noticed that this is blocking code in an
async fnso just do it in aspawn_blocking, which should be healthy for our startup times. Other effects includes hopefully halving ofstatcalls; extra calls which were not done previously are now done for the future layers.