This guide contains instructions on how to setup Python with Gradle within the Airbyte Monorepo. If you are a contributor working on one or two connectors, this page is most likely not relevant to you. Instead, you should use your standard Python development flow.
Before working with connectors written in Python, we recommend running the following command from the airbyte root directory
python3 tools/bin/update_intellij_venv.py -modules <connector directory name> --install-venv
e.g
python tools/bin/update_intellij_venv.py -modules source-stripe --install-venv
If using Pycharm or IntelliJ, you'll also want to add the interpreter to the IDE's list of known interpreters. You can do this by adding the --update-intellij
flag. More details can be found here
python tools/bin/update_intellij_venv.py -modules <connector directory name> --install-venv --update-intellij
If working with many connectors, you can use the --all-modules
flag to install the virtual environments for all connectors
python tools/bin/update_intellij_venv.py --all-modules --install-venv
This will create a virtualenv
and install dependencies for the connector you want to work on as well as any internal Airbyte python packages it depends on.
When iterating on a single connector, you will often iterate by running
./gradlew :airbyte-integrations:connectors:your-connector-dir:build
This command will:
- Install a virtual environment at
airbyte-integrations/connectors/<your-connector-dir>/.venv
- Install local development dependencies specified in
airbyte-integrations/connectors/your-connector-dir/requirements.txt
- Runs the following pip modules:
To format and lint your code before commit you can use the Gradle command above, but for convenience we support pre-commit tool. To use it you need to install it first:
pip install pre-commit
then, to install pre-commit
as a git hook, run
pre-commit install
That's it, pre-commit
will format/lint the code every time you commit something. You find more information about pre-commit here.
At Airbyte, we use IntelliJ IDEA for development. Although it is possible to develop connectors with any IDE, we typically recommend IntelliJ IDEA or PyCharm, since we actively work towards compatibility.
Install the Pydantic plugin. This will help autocompletion with some of our internal types.
The following setup steps are written for PyCharm but should have similar equivalents for IntelliJ IDEA:
1.python tools/bin/update_intellij_venv.py -modules <your-connector-dir> --update-intellij
- Restart PyCharm
- Go to
File -> New -> Project...
- Select
Pure Python
. - Select a project name like
airbyte
and a directory outside of theairbyte
code root. - Go to
Preferences -> Project -> Python Interpreter
- Find a gear ⚙️ button next to
Python interpreter
dropdown list, click and selectAdd
- Select
Virtual Environment -> Existing
- Set the interpreter path to the one that was created by Python command, i.e.
airbyte-integrations/connectors/<your-connector-dir>/.venv/bin/python
. - Wait for PyCharm to finish indexing and loading skeletons from selected virtual environment.
You should now have access to code completion and proper syntax highlighting for python projects.
If you need to work on another connector you can quickly change the current virtual environment in the bottom toolbar.
By default, the find function in IntelliJ is not scoped and will include all files in the monorepo, including all the libraries installed as part of a connector's virtual environment. This huge volume of files makes indexing and search very slow. You can ignore files from the connectors' virtual environment with the following steps:
- Open the project structure using
cmd-;
- Navigate to the "Project Settings / Modules" section in the right-side of the menu
- Select the top level
airbyte
module so the change is applied to all submodules - Add the following filter to the
Exclude files
option:connectors/**/.venv
- Press OK to confirm your options.