Destination BigQuery: Consolidation of objects to StreamConfig, cleanup #38131
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
2 tasks
This stack of pull requests is managed by Graphite. Learn more about stacking. |
octavia-squidington-iii
added
area/connectors
gisripa
changed the title
gisripa
force-pushed
the
gireesh/05-10-Bigquery_cdk_signature_changes
branch
2 times, most recently
from
b43ce70 to
dc2606b
Compare
2 tasks
gisripa
force-pushed
the
gireesh/05-10-Bigquery_cdk_signature_changes
branch
from
dc2606b to
f6903b7
Compare
gisripa
force-pushed
the
gireesh/05-10-Bigquery_cdk_signature_changes
branch
from
f6903b7 to
cf090a7
Compare
This was referenced May 12, 2024
gisripa
force-pushed
the
gireesh/05-10-Bigquery_cdk_signature_changes
branch
from
cf090a7 to
9c0dfc5
Compare
gisripa
changed the title
gisripa
force-pushed
the
cdk_generation_id
branch
from
0724d77 to
0ded7a7
Compare
gisripa
force-pushed
the
gireesh/05-10-Bigquery_cdk_signature_changes
branch
from
9c0dfc5 to
4328f0a
Compare
gisripa
force-pushed
the
cdk_generation_id
branch
from
0ded7a7 to
23a237f
Compare
edgao
approved these changes
May 13, 2024
| @@ -234,7 +236,8 @@ public SerializedAirbyteMessageConsumer getSerializedMessageConsumer(final JsonN | |||
| final String datasetLocation = BigQueryUtils.getDatasetLocation(config); | |||
| final BigQuerySqlGenerator sqlGenerator = new BigQuerySqlGenerator(config.get(BigQueryConsts.CONFIG_PROJECT_ID).asText(), datasetLocation); | |||
| final Optional<String> rawNamespaceOverride = TypingAndDedupingFlag.getRawNamespaceOverride(RAW_DATA_DATASET); | |||
| final ParsedCatalog parsedCatalog = parseCatalog(config, catalog, datasetLocation, rawNamespaceOverride); | |||
| final ParsedCatalog parsedCatalog = parseCatalog(sqlGenerator, defaultNamespace, | |||
| rawNamespaceOverride.orElse(JavaBaseConstants.DEFAULT_AIRBYTE_INTERNAL_NAMESPACE), catalog); | |||
There was a problem hiding this comment.
random thought: now that we're in kotlin, do you want to make catalogparser accept rawNamespace: String? and do rawNamespace ?: DEFAULT_AIRBYTE_INTERNAL? then we don't need to copy this logic into every connector
There was a problem hiding this comment.
Yeah makes sense to me.
| final ParsedCatalog parsedCatalog, | ||
| final Function<JsonNode, BigQueryRecordFormatter> recordFormatterCreator, | ||
| final Function<String, String> tmpTableNameTransformer) { | ||
| private Map<StreamDescriptor, StreamConfig> createWriteConfigs(final ConfiguredAirbyteCatalog catalog, |
There was a problem hiding this comment.
can this just be parsedCatalog.getStreams().stream()? afaict we don't actually need the raw protocol models for anything
(and then we don't need to plumb the raw configured catalog into this method)
There was a problem hiding this comment.
I think so, but the called method iterated over the Configuredcatalog and populates the map. Didn't want to change any functional logic for the fear of missing some defaultNamespace plumbing. up the stack the whole method will be removed.
| // In Destinations V2, we will always use the 'airbyte' schema/namespace for raw tables | ||
| BigQueryRecordFormatter.SCHEMA_V2, streamConfig.getId().getOriginalName(), | ||
| tableId, streamConfig.getId().getOriginalName()); | ||
| // In Destinations V2, we will always use the 'airbyte' schema/originalNamespace for raw tables |
There was a problem hiding this comment.
Suggested change
| // In Destinations V2, we will always use the 'airbyte' schema/originalNamespace for raw tables | |
| // In Destinations V2, we will always use the 'airbyte_internal' schema/originalNamespace for raw tables |
😅
| DestinationSyncMode.APPEND_DEDUP, | ||
| List.of(new ColumnId("foo", "bar", "fizz")), | ||
| Optional.empty(), | ||
| new LinkedHashMap<>()); | ||
| new LinkedHashMap<>(), 0, 0, 0); |
There was a problem hiding this comment.
.... do you want me to set default values for the generation/sync ID args? These diffs seem kind of dumb
gisripa
force-pushed
the
gireesh/05-10-Bigquery_cdk_signature_changes
branch
from
4328f0a to
3de0d51
Compare
octavia-squidington-iii
added
the
area/documentation
gisripa
force-pushed
the
gireesh/05-10-Bigquery_cdk_signature_changes
branch
from
3de0d51 to
6bf8684
Compare
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Removing redundant references and duplicate information passed around using
WriteConfigobjects. No functional changes and resurrected all the information needed throughStreamConfigand adapted changes accordingly.This PR should be in a mergeable state with no functional changes after the ones down the stack are published.
Review guide
BigQueryWriteConfigand reused already builtStreamConfigStagingOperationsinterface and made concrete class, this will help for later adding a shim on this and refactoring without large changesWriteDispostionetc. Probably remnant of bigquery-denormalized bespoke connector.User Impact
Can this PR be safely reverted and rolled back?