Source Faker: Add support for PyPi and AirbyteLib entrypoints #34033

aaronsteers · 2024-01-08T23:12:55Z

This adds compatibility with AirbyteLib.

Includes PyPi declaration as defined in:

Add pypi section to connector metadata #33529

This PR replaces the related changes in:

AirbyteLib example notebook with source-faker #33878

This PR also includes a generic Python DevContainer, which can be used for testing in Codespaces. I'm happy to remove it though and move to a different PR if anyone has an objection. I confirmed I was able to run pytest from this decontainer, and that all tests are passing.

vercel · 2024-01-08T23:13:01Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Jan 9, 2024 0:16am

github-actions · 2024-01-08T23:13:16Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

aaronsteers · 2024-01-08T23:52:34Z

.devcontainer/java-connectors-generic/devcontainer.json

@@ -1,6 +1,6 @@
 // For format details, see https://aka.ms/devcontainer.json. For config options, see the
 {
-  "name": "Connector Development DevContainer (Generic)",
+  "name": "Java Development DevContainer (Generic)",


Updated the name to distinguish the generic Java devcontainer from the Python one.

alafanechere

The source-faker changes LGTM.
I raised a couple of concerns / suggestion about the devcontainer configuration.
I'm not using devcontainer myself so my suggestion might be unrealistic / not matching what you like to achieve.

Please engage in a more in-depth discussion if you'd like to broaden the adoption of devcontainer.json among our connector developers. My main concern is to find a way to guarantee a certain level of consistency between the dev environment (managed by devcontainers.json) and the build environment (managed by airbyte-ci) so that developer do not face late CI failures / bad surprises on release.

alafanechere · 2024-01-09T08:36:24Z

.devcontainer/python-connectors-generic/devcontainer.json

Can we use python connector base image as a development container for connectors?

The problem we might have is that:

Python connectors base image is based on Python 3.9.18

airbyte-ci (dagger) requires Python 3.10

So it's kinda hard to have a unified dev container which can match connector runtime with airbyte-ci installed.

We should ideally have a devcontainer with multiple python version installed, 3.9 for connectors, 3.10 for airbyte-ci.

@alafanechere - I tested successfully and was able to run airbyte-ci without issue, thanks to the make option.

Happy to keep iterating on this. For now, it gives a baseline experience that's much better than starting with a blank of generic container.

alafanechere · 2024-01-09T08:45:53Z

.devcontainer/python-connectors-generic/devcontainer.json

+  "initializeCommand": "git config --add safe.directory /workspaces/airbyte",
+
+  // Setup airbyte-ci on the container:
+  "postCreateCommand": "make tools.airbyte-ci-dev.install",


Is airbyte-ci correctly working inside the container?
If it does I think it's thanks to the docker-in-docker feature.
The problem with this approach is that the dagger engine in use will be using your devcontainer docker / buildkit engine, and not your host one.

This can impact performance and caching.

To not use docker-in-docker you can try to:

Mount the docker socket to the dev container so that it can interact with your host docker engine

Set the _EXPERIMENTAL_DAGGER_RUNNER_HOST env var to tell the dagger engine to use this mounted socket. (example in CI here)

It works fine as-is for now. In the cloud, I think docker-in-docker is correct, and probably the best we can get. There's also a 'docker-from-docker' option that might work better locally - but I don't have familiar with that as of yet, and so I'd like to keep as is for now.

Please engage in a more in-depth discussion if you'd like to broaden the adoption of devcontainer.json among our connector developers.

Agreed. No need to push this as of now. I think over time we will get to a good place where we have collective confidence that the containers give a solid and consistent experience.

flash1293

Approving for faker changes, @alafanechere s comments on dev container make sense to me

…ehq#34033)

aaronsteers added 4 commits January 8, 2024 14:57

add generic Python container devcontainer

67bf81c

fix python image name

53bb94c

add cli entrypoint to source-faker

b108e16

version bump

80505db

octavia-squidington-iii added area/connectors Connector related issues connectors/source/faker labels Jan 8, 2024

add changelog

66c4305

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Jan 8, 2024

vercel bot deployed to Preview January 8, 2024 23:17 View deployment

declare PyPi publish in style of #33529

06c7d38

aaronsteers requested review from alafanechere and flash1293 January 8, 2024 23:24

aaronsteers marked this pull request as ready for review January 8, 2024 23:24

aaronsteers mentioned this pull request Jan 8, 2024

AirbyteLib example notebook with source-faker #33878

Closed

airbyte-ci-dev format fix all

f65a44d

aaronsteers changed the base branch from aj/generic-python-devcontainer to master January 8, 2024 23:48

aaronsteers changed the title ~~Faker Source: Add support for standard AirbyteLib entrypoints~~ Faker Source: Add support for PyPi and AirbyteLib entrypoints Jan 8, 2024

aaronsteers changed the title ~~Faker Source: Add support for PyPi and AirbyteLib entrypoints~~ Source Faker: Add support for PyPi and AirbyteLib entrypoints Jan 8, 2024

install airbyte-ci in postCreate

82ade27

aaronsteers commented Jan 9, 2024

View reviewed changes

alafanechere approved these changes Jan 9, 2024

View reviewed changes

flash1293 approved these changes Jan 9, 2024

View reviewed changes

alafanechere mentioned this pull request Jan 9, 2024

Add devcontainer configs for "Python generic" and "Default" #34032

Merged

aaronsteers merged commit 7d7f33c into master Jan 11, 2024
30 checks passed

aaronsteers deleted the aj/source-faker/airbytelib-entrypoints branch January 11, 2024 08:16

jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024

Source Faker: Add support for PyPi and AirbyteLib entrypoints (airbyt…

4b093f1

…ehq#34033)

jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024

Source Faker: Add support for PyPi and AirbyteLib entrypoints (airbyt…

02dc860

…ehq#34033)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Faker: Add support for PyPi and AirbyteLib entrypoints #34033

Source Faker: Add support for PyPi and AirbyteLib entrypoints #34033

aaronsteers commented Jan 8, 2024 •

edited

Loading

vercel bot commented Jan 8, 2024 •

edited

Loading

github-actions bot commented Jan 8, 2024 •

edited by aaronsteers

Loading

aaronsteers Jan 8, 2024

alafanechere left a comment

alafanechere Jan 9, 2024

aaronsteers Jan 11, 2024

alafanechere Jan 9, 2024 •

edited

Loading

aaronsteers Jan 11, 2024

flash1293 left a comment

Source Faker: Add support for PyPi and AirbyteLib entrypoints #34033

Source Faker: Add support for PyPi and AirbyteLib entrypoints #34033

Conversation

aaronsteers commented Jan 8, 2024 • edited Loading

vercel bot commented Jan 8, 2024 • edited Loading

github-actions bot commented Jan 8, 2024 • edited by aaronsteers Loading

Before Merging a Connector Pull Request

aaronsteers Jan 8, 2024

Choose a reason for hiding this comment

alafanechere left a comment

Choose a reason for hiding this comment

alafanechere Jan 9, 2024

Choose a reason for hiding this comment

aaronsteers Jan 11, 2024

Choose a reason for hiding this comment

alafanechere Jan 9, 2024 • edited Loading

Choose a reason for hiding this comment

aaronsteers Jan 11, 2024

Choose a reason for hiding this comment

flash1293 left a comment

Choose a reason for hiding this comment

aaronsteers commented Jan 8, 2024 •

edited

Loading

vercel bot commented Jan 8, 2024 •

edited

Loading

github-actions bot commented Jan 8, 2024 •

edited by aaronsteers

Loading

alafanechere Jan 9, 2024 •

edited

Loading