Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set Status.PRESENT after synchronous preload #950

Merged
merged 6 commits into from
Dec 5, 2024
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Merge branch 'dev' into fix-preload
ansoncfit committed Dec 5, 2024
commit 184cfa42a6188926b267b18cf6641005e60661e0
Original file line number Diff line number Diff line change
@@ -458,10 +458,15 @@ protected void handleOneRegionalTask (RegionalTask task) throws Throwable {
}

// Pull all necessary inputs into cache in a blocking fashion, unlike single-point tasks where prep is async.
// Avoids auto-shutdown while preloading. Must be done after loading destination pointsets to establish extents.
// Note we're completely bypassing the async loader here and relying on the older nested LoadingCaches.
// If those are ever removed, the async loader will need a synchronous mode with per-path blocking (kind of
// reinventing the wheel of LoadingCache) or we'll need to make preparation for regional tasks async.
// This is because single point tasks return fast to report progress, while regional tasks currently do not.
// Worker auto-shutdown time should remain very high during this blocking preload step. Destination point sets
// must already be loaded to establish extents before the preloading step begins. Note that we're still using
// the NetworkPreloader which is an AsyncLoader, but calling a method that intentionally skips all the async or
// background proccessing machinery. Usually, N RegionalTasks all try to preload at once, and all block on
// this method. Redundant slow calculation is not prevented by the AsyncLoader class itself, but by the other
// LoadingCaches behind it. Specifically, the TransportNetworkCache and LinkageCache enforce turn-taking and
// prevent redundant work. If those are ever removed, we would need either async regional task preparation, or
// a synchronous mode with per-key blocking on AsyncLoader (kind of reinventing the wheel of LoadingCache).
TransportNetwork transportNetwork = networkPreloader.preloadBlocking(task);

// If we are generating a static site, there must be a single metadata file for an entire batch of results.
You are viewing a condensed version of this merge commit. You can view the full changes here.