Destinations CDK: Correctly detect when real raw/final table is correct generation during truncate sync #42503
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Skipped Deployment
|
This stack of pull requests is managed by Graphite. Learn more about stacking. |
edgao
force-pushed
the
edgao/refreshes_recover_from_real_raw_table
branch
from
3175892 to
5f9fbcf
Compare
edgao
commented
Jul 24, 2024
| @@ -85,7 +85,7 @@ class AbstractStreamOperationTest { | |||
| val initialState = | |||
| mockk<DestinationInitialStatus<MinimumDestinationState.Impl>> { | |||
| every { streamConfig } returns this@Truncate.streamConfig | |||
| every { initialRawTableStatus } returns mockk<InitialRawTableStatus>() | |||
| every { initialRawTableStatus.rawTableExists } returns false | |||
There was a problem hiding this comment.
these diffs are just to handle the new behavior. They don't affect the test in any relevant way.
edgao
force-pushed
the
edgao/refreshes_recover_from_real_raw_table
branch
2 times, most recently
from
dbcc2c2 to
d59f5d8
Compare
This was referenced Jul 24, 2024
gisripa
approved these changes
Jul 24, 2024
edgao
force-pushed
the
edgao/refreshes_recover_from_real_raw_table
branch
from
d59f5d8 to
3d9cf10
Compare
|
(copying from DM)
trying to do this without that duplication yields a weird if -> if -> early return, which seems confusing. leaving it as-is |
stephane-airbyte
approved these changes
Jul 24, 2024
...rc/main/kotlin/io/airbyte/integrations/base/destination/operation/AbstractStreamOperation.kt
Show resolved
Hide resolved
edgao
force-pushed
the
edgao/refreshes_recover_from_real_raw_table
branch
from
3d9cf10 to
341ae0d
Compare
edgao
changed the base branch from
master
to
edgao/expected_broken_bigquery_tests
This was referenced Jul 26, 2024
edgao
force-pushed
the
edgao/expected_broken_bigquery_tests
branch
from
d9889dc to
bf62d6b
Compare
edgao
force-pushed
the
edgao/refreshes_recover_from_real_raw_table
branch
from
341ae0d to
8a53e44
Compare
edgao
force-pushed
the
edgao/expected_broken_bigquery_tests
branch
from
bf62d6b to
aef6bf6
Compare
edgao
force-pushed
the
edgao/refreshes_recover_from_real_raw_table
branch
from
8a53e44 to
3922cf7
Compare
edgao
force-pushed
the
edgao/expected_broken_bigquery_tests
branch
from
aef6bf6 to
6f61517
Compare
edgao
force-pushed
the
edgao/refreshes_recover_from_real_raw_table
branch
from
3922cf7 to
b9bf10d
Compare
|
from #42514 - this doesn't correctly handle the final table >.> i.e. the raw table is correctly preserved during the second sync, but the final table is still nuked (b/c we unconditionally create a temp final table, T+D ignores the previous sync's records b/c they have nonnull loaded_at, and then we overwrite the final table using the empty temp final table) |
edgao
force-pushed
the
edgao/expected_broken_bigquery_tests
branch
3 times, most recently
from
cc09978 to
020d792
Compare
edgao
force-pushed
the
edgao/refreshes_recover_from_real_raw_table
branch
from
7fdef81 to
ad24e27
Compare
edgao
force-pushed
the
edgao/expected_broken_bigquery_tests
branch
from
020d792 to
be63bdc
Compare
edgao
force-pushed
the
edgao/refreshes_recover_from_real_raw_table
branch
from
ad24e27 to
1d5f768
Compare
edgao
changed the base branch from
edgao/expected_broken_bigquery_tests
to
edgao/refresh_massive_test
edgao
changed the title
edgao
force-pushed
the
edgao/refresh_massive_test
branch
from
7d4f94e to
4c8180c
Compare
edgao
force-pushed
the
edgao/refreshes_recover_from_real_raw_table
branch
2 times, most recently
from
dd2e935 to
707b94d
Compare
edgao
force-pushed
the
edgao/refreshes_recover_from_real_raw_table
branch
3 times, most recently
from
790075d to
3aa5155
Compare
gisripa
approved these changes
Aug 13, 2024
| // In this case, there is no existing temp raw table, and there is a real raw table | ||
| // which already belongs to the correct generation. | ||
| // Check for that case now. | ||
| val realStageGeneration = storageOperation.getStageGeneration(stream.id, NO_SUFFIX) |
edgao
force-pushed
the
edgao/refreshes_recover_from_real_raw_table
branch
3 times, most recently
from
99ecb80 to
30fecb2
Compare
edgao
force-pushed
the
edgao/refreshes_recover_from_real_raw_table
branch
from
95ac402 to
7270022
Compare
edgao
force-pushed
the
edgao/refreshes_recover_from_real_raw_table
branch
from
7270022 to
840e346
Compare
|
/publish-java-cdk
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
from https://github.com/airbytehq/airbyte-internal-issues/issues/8784, for the new interfaces. I'll stack PRs on top of this for bigquery+redshift+snowflake.
add a new check for the real raw table during truncate sync startup, to see if it belongs to the correct generation. Also handle this new behavior during sync finalization (i.e. don't overwrite the raw table if we weren't using a temp raw table to begin with).
Adds two new unit tests for this.