✨ [file based cdk] S3 legacy config adapter #29145

brianjlai · 2023-08-07T04:27:57Z

What

Adds a transformer that allows the new S3 connector built with the file-based CDK to accept configs in the legacy format to be processed. It does not persist the change in the DB, just how the connector handles an operation

How

The CDK allows for a source to override the read_config() method which is responsible for reading in the config JSON file. By overriding this, we can still get the JSON from the file, but we also then parse it into the legacy format instance and then construct a new dictionary representing the new format. By doing it this way, the downstream flow can assume it is always using the new config

Recommended reading order

legacy_config_transformer.py
s3_source.py

github-actions · 2023-08-07T04:28:17Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

brianjlai · 2023-08-07T17:14:05Z

airbyte-integrations/connectors/source-s3/source_s3/v4/legacy_config_transformer.py

+
+    @ classmethod
+    def create_globs(cls, path_pattern: str, path_prefix: str) -> List[str]:
+        if path_prefix:


we could also add trimming to the prefix/pattern to avoid extra / but I also didn't see any of that in the existing version so I kept it simpler

octavia-squidington-iii · 2023-08-07T17:46:42Z

source-s3 test report (commit `ac88970854`) - ❌

⏲️ Total pipeline duration: 22mn01s

Step	Result
Validate airbyte-integrations/connectors/source-s3/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅
Code format checks	❌
Connector package install	✅
Build source-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

girarda · 2023-08-07T17:58:10Z

airbyte-integrations/connectors/source-s3/source_s3/v4/legacy_config_transformer.py

+        }
+
+        if legacy_config.provider.start_date:
+            transformed_config["start_date"] = legacy_config.provider.start_date


do we need to add the extra precision? eg convert 2021-01-01T00:00:00Z to 2021-01-01T00:00:00.000000Z

girarda · 2023-08-07T18:02:44Z

airbyte-integrations/connectors/source-s3/source_s3/v4/legacy_config_transformer.py

+from typing import Mapping, Any, List
+
+
+class LegacyConfigTransformer:


nit: can you add a comment in the spec that this file needs to be updated if the spec changes? I'm not sure how long we'll need this transformer and there are PRs in the pipeline (eg #17334)

maxi297 · 2023-08-08T13:32:39Z

airbyte-integrations/connectors/source-s3/source_s3/v4/legacy_config_transformer.py

+        return transformed_config
+
+    @ classmethod
+    def create_globs(cls, path_pattern: str, path_prefix: str) -> List[str]:


nit: should we make this private?

clnoll

Nice @brianjlai!

airbyte-integrations/connectors/source-s3/source_s3/v4/legacy_config_transformer.py

octavia-squidington-iii · 2023-08-09T21:19:52Z

source-s3 test report (commit `a6159c1cbe`) - ❌

⏲️ Total pipeline duration: 21mn56s

Step	Result
Validate airbyte-integrations/connectors/source-s3/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅
Code format checks	❌
Connector package install	✅
Build source-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

octavia-squidington-iii · 2023-08-09T21:44:15Z

source-s3 test report (commit `5ef32c6e2d`) - ❌

⏲️ Total pipeline duration: 22mn19s

Step	Result
Validate airbyte-integrations/connectors/source-s3/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅
Code format checks	❌
Connector package install	✅
Build source-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

octavia-squidington-iii · 2023-08-09T22:39:46Z

source-s3 test report (commit `391c1ee09d`) - ❌

⏲️ Total pipeline duration: 22mn00s

Step	Result
Validate airbyte-integrations/connectors/source-s3/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

* s3 adapter * pr feedback and updates after rebasing master * add comment * formatting

s3 adapter

ac88970

octavia-squidington-iii added area/connectors Connector related issues connectors/source/s3 labels Aug 7, 2023

brianjlai commented Aug 7, 2023

View reviewed changes

brianjlai changed the title ~~[file based cdk] S3 legacy config adapter~~ ✨ [file based cdk] S3 legacy config adapter Aug 7, 2023

brianjlai requested review from girarda, maxi297 and clnoll August 7, 2023 17:23

brianjlai marked this pull request as ready for review August 7, 2023 17:23

girarda approved these changes Aug 7, 2023

View reviewed changes

maxi297 approved these changes Aug 8, 2023

View reviewed changes

Merge branch 'master' into brian/s3_file_based_adapter

0d5f113

clnoll approved these changes Aug 9, 2023

View reviewed changes

girarda reviewed Aug 9, 2023

View reviewed changes

airbyte-integrations/connectors/source-s3/source_s3/v4/legacy_config_transformer.py Outdated Show resolved Hide resolved

brianjlai added 2 commits August 9, 2023 13:56

pr feedback and updates after rebasing master

a6159c1

add comment

5ef32c6

formatting

391c1ee

brianjlai merged commit 0543099 into master Aug 9, 2023

brianjlai deleted the brian/s3_file_based_adapter branch August 9, 2023 23:09

harrytou pushed a commit to KYVENetwork/airbyte that referenced this pull request Sep 1, 2023

✨ [file based cdk] S3 legacy config adapter (airbytehq#29145)

76afaf7

* s3 adapter * pr feedback and updates after rebasing master * add comment * formatting

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ [file based cdk] S3 legacy config adapter #29145

✨ [file based cdk] S3 legacy config adapter #29145

brianjlai commented Aug 7, 2023 •

edited

Loading

github-actions bot commented Aug 7, 2023 •

edited by brianjlai

Loading

brianjlai Aug 7, 2023

octavia-squidington-iii commented Aug 7, 2023

girarda Aug 7, 2023

girarda Aug 7, 2023

brianjlai Aug 9, 2023

maxi297 Aug 8, 2023

clnoll left a comment

octavia-squidington-iii commented Aug 9, 2023

octavia-squidington-iii commented Aug 9, 2023

octavia-squidington-iii commented Aug 9, 2023

		from typing import Mapping, Any, List


		class LegacyConfigTransformer:

✨ [file based cdk] S3 legacy config adapter #29145

✨ [file based cdk] S3 legacy config adapter #29145

Conversation

brianjlai commented Aug 7, 2023 • edited Loading

What

How

Recommended reading order

github-actions bot commented Aug 7, 2023 • edited by brianjlai Loading

Before Merging a Connector Pull Request

brianjlai Aug 7, 2023

Choose a reason for hiding this comment

octavia-squidington-iii commented Aug 7, 2023

source-s3 test report (commit ac88970854) - ❌

girarda Aug 7, 2023

Choose a reason for hiding this comment

girarda Aug 7, 2023

Choose a reason for hiding this comment

brianjlai Aug 9, 2023

Choose a reason for hiding this comment

maxi297 Aug 8, 2023

Choose a reason for hiding this comment

clnoll left a comment

Choose a reason for hiding this comment

octavia-squidington-iii commented Aug 9, 2023

source-s3 test report (commit a6159c1cbe) - ❌

octavia-squidington-iii commented Aug 9, 2023

source-s3 test report (commit 5ef32c6e2d) - ❌

octavia-squidington-iii commented Aug 9, 2023

source-s3 test report (commit 391c1ee09d) - ❌

brianjlai commented Aug 7, 2023 •

edited

Loading

github-actions bot commented Aug 7, 2023 •

edited by brianjlai

Loading

source-s3 test report (commit `ac88970854`) - ❌

source-s3 test report (commit `a6159c1cbe`) - ❌

source-s3 test report (commit `5ef32c6e2d`) - ❌

source-s3 test report (commit `391c1ee09d`) - ❌