chore: load CDK clears partial aggregates at cadence #48551
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ariants don't wrap records. Additional cleanup.
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Skipped Deployment
|
tryangul
changed the title
| * | ||
| * In a future where we deserialize only the info necessary for routing, this could include a dumb | ||
| * container for the serialized, and deserialization could be deferred until the spooled records | ||
| * were recovered from disk. | ||
| */ | ||
| sealed class DestinationRecordWrapped : Sized | ||
| sealed class DestinationStreamEvent : Sized |
| @@ -163,6 +165,10 @@ class DefaultDestinationTaskLauncher( | |||
| } | |||
| } | |||
|
|
|||
| // Start flush task | |||
| log.info { "Starting flush tick task" } | |||
There was a problem hiding this comment.
Nit can we change these log lines to call out the different types of tasks
Starting flush timer for state staged files or something
Starting flush timer for stale checkpoints
There was a problem hiding this comment.
And maybe include $task since that will appear in the logs when the task is launched/completed/canceled
There was a problem hiding this comment.
let me see what I can do
tryangul
force-pushed
the
rbroughan/flush-task
branch
from
cfa272d to
1a5cf93
Compare
johnny-schmidt
requested changes
Nov 19, 2024
airbyte-cdk/bulk/core/load/src/main/kotlin/io/airbyte/cdk/load/task/internal/SpillToDiskTask.kt
Show resolved
Hide resolved
tryangul
force-pushed
the
rbroughan/flush-task
branch
from
1a5cf93 to
c8ee5c7
Compare
johnny-schmidt
approved these changes
Nov 20, 2024
tryangul
added a commit
that referenced
this pull request
Nov 21, 2024
matteogp
pushed a commit
that referenced
this pull request
Nov 22, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Stream aggregates (spilled files) will be closed and flushed at cadence of 15 minutes, so 15-30 minutes to prevent backing up indefinitely on a large number of partial aggregates (chunks of work below the size threshold for high cardinality syncs.
How
FlushTickTaskwhich publishes a flush event on each streams input channel at a cadence — default is 15 minutes but this can be tunedTimeWindowTriggerwhich will respond to flush events and trigger a window publish if the aggregate has been open at least as long as the configured window width — default 15 minutes but this can be tunedDestinationRecordWrapped->DestinationStreamEventsince 2/3 of it's variants do not wrap a record.Not in this PR
Un-wiring the checkpoint time based flush task.
It does not currently work today and may not be relevant with this broader time based mechanism in place.
Reading Order
FlushTickTask
SpillToDiskTask
TimeWindowTrigger