You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on May 29, 2024. It is now read-only.
Current DAG implementation will add edges between existing components based on conflicting ComponentID values. However, this means that pipelines that perform backfills can conflict with ones that are live. Same for backtesting pipelines as well.
Example Scenario
Two pipelines are shown below (RP0, RP1), both of which use the same exact components, but require backfilling from some starting heights (x0, x1) where:
x in Z+ and x <= current_block_height
In this example, since RP0.component_set = RP1.component_set, RP1 will be treated as a duplicate pipeline with the EtlManager since their respective pipeline IDs are equal. This will result in RP1 powering some invariant from whatever point in chain history that RP0 is in; resulting in invariants failing to successfully backfill.
It's important to note that in this scenario, x0 could equal x1 but doesn't necessarily have to. The presence of some x for a registerPipeline denotes that it has backfill requirements to be ran. This means that each pipeline state is:
I. Syncing when monotonic x < current_block_height
II. Live once x >= current_block_height
Problem Solution
Introduce access management logic that performs additional integrity checks before reusing an existing component. I,e. The presence of sync flag for a pipeline deems it globally unique where it's components cannot be reused.
In the example of identical pipelines (P0, P1) with the same IDs but backfilling toggles, the pipelines would merge once their states have transitioned (syncing --> live).
The text was updated successfully, but these errors were encountered:
Bug Description
Current DAG implementation will add edges between existing components based on conflicting
ComponentID
values. However, this means that pipelines that perform backfills can conflict with ones that are live. Same for backtesting pipelines as well.Example Scenario
Two pipelines are shown below
(RP0, RP1)
, both of which use the same exact components, but require backfilling from some starting heights(x0, x1)
where:In this example, since
RP0.component_set = RP1.component_set
,RP1
will be treated as a duplicate pipeline with the EtlManager since their respective pipeline IDs are equal. This will result inRP1
powering some invariant from whatever point in chain history that RP0 is in; resulting in invariants failing to successfully backfill.It's important to note that in this scenario,
x0
could equalx1
but doesn't necessarily have to. The presence of somex
for a registerPipeline denotes that it hasbackfill
requirements to be ran. This means that each pipeline state is:I. Syncing when monotonic
x
< current_block_heightII. Live once
x
>= current_block_heightProblem Solution
Introduce access management logic that performs additional integrity checks before reusing an existing component. I,e. The presence of
sync
flag for a pipeline deems it globally unique where it's components cannot be reused.In the example of identical pipelines (P0, P1) with the same IDs but backfilling toggles, the pipelines would merge once their states have transitioned (
syncing --> live
).The text was updated successfully, but these errors were encountered: