connector-templates: move generated code from src so airbyte-ci builds it
#36428
Conversation
This commit cleans up connector generator, specifically: - Removes java sources and destinations from plopfile - Removes singer source from plopfile - Removes generic and python (non-http) source from plopfile - Cleans up documentation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Ignored Deployment
|
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
This stack of pull requests is managed by Graphite. Learn more about stacking. Join @natikgadzhi and the rest of your teammates on |
natikgadzhi
changed the title
src so airbyte-ci builds itsrc so airbyte-ci builds it
stephane-airbyte
changed the base branch from
ng/connector-generator-cleanup
to
master
natikgadzhi
changed the base branch from
master
to
ng/connector-generator-cleanup
natikgadzhi
changed the base branch from
ng/connector-generator-cleanup
to
master
natikgadzhi
changed the base branch from
master
to
ng/connector-generator-cleanup
natikgadzhi
changed the base branch from
ng/connector-generator-cleanup
to
master
There was a problem hiding this comment.
@natikgadzhi I originally wanted to make connectors follow the src layout as its a good python practice. I might have done it too agressively and the build logic in airbyte-ci might not support it correctly indeed.
I think the bug can be solved with a one line change here: including src instead of connector_snake_case_name. But if we want to support flat and srclayout the logic has to change a bit more. So I'll 👍 this one 😄
Can you please re-generate the scaffolds?
(or delete them? - I think no test are actually checking scaffolds are consistent with the generated code anymore).
./gradlew --no-daemon :airbyte-integrations:connector-templates:generator:generateScaffolds
I noticed you also deleted the java + configuration-based generator.
I'd suggest doing it in a separate PR in case we'd want to revert it.
And you should also remove source-singer as it's legacy too. I'd also challenge the usefulness source-python generator. If anyone wants to build a non-http python connector they can start off from the http one and remove the http related code.
| @@ -12,7 +12,7 @@ readme = "README.md" | |||
| documentation = "https://docs.airbyte.com/integrations/sources/{{dashCase name}}" | |||
| homepage = "https://airbyte.com" | |||
| repository = "https://github.com/airbytehq/airbyte" | |||
| packages = [ { include = "source_{{snakeCase name}}", from="src"}, {include = "main.py", from = "src"} ] | |||
| packages = [ { include = "source_{{snakeCase name}}" }, {include = "main.py" } ] | |||
There was a problem hiding this comment.
This line can be completly remove from all pyproject.toml if we're using the flat layout instead of the src layout
natikgadzhi
deleted the
03-24-connector-generator_move_generated_code_from_src_so_airbyte-ci_builds_it
branch
Merge activity
|
|
Whooops, this got merged with Graphite automatically on approval, but I will follow-up on @alafanechere's asks:
|
|
Seems like java connector has been removed in this PR. Tried to follow steps and it didn't work: https://docs.airbyte.com/connector-development/tutorials/building-a-java-destination |
What
For newly generated python and low-code connectors, move sources from
srctosource_%CONN_NAME%soairbyte-ci buildcan pick them up correctly. Closes #35893.What changed?
srcpyproject.tomltemplatesHow to test?
You can make a new connector with
./generate.shand verify that it works with Poetry and that it builds withairbyte-ci build --name source-test