Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[local airbyte cdk tests] update README.md to fix gradle command and … #20821

Merged
merged 1 commit into from
Jan 3, 2023
Merged

[local airbyte cdk tests] update README.md to fix gradle command and … #20821

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions airbyte-cdk/python/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,13 @@ Create a symlink that connects `<connector-directory>/airbyte-cdk` to your local
ln -s ../../../airbyte-cdk/python airbyte-cdk
```

Add the following lines to your connector's `Dockerfile`, before the line that installs dependencies via `pip install -e .`:
Now that you have a symbolic link in your source, you will nedd to install it in the docker image. Depending on how the docker image is build you will have two possibilities:
1. If image building process uses a temporary image called `builder`, add the following just after the copy of the installed packages to the folder `/usr/local`:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you link to an example of an image use a builder tmp image for reference?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't we also need to run pip install when the the image does not use the builder tmp image?

Copy link
Contributor Author

@maxi297 maxi297 Dec 22, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you link to an example of an image use a builder tmp image for reference?

This Dockerfile pip install in a "builder" image and copy the files on the "real" image. This didn't seem to work for me. I assume it is because of the symlink. Therefore instead, I did add the lines:

COPY airbyte-cdk airbyte-cdk
RUN PYTHONUSERBASE=/usr/local pip install -e ./airbyte-cdk

... after https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-square/Dockerfile#L22

don't we also need to run pip install when the the image does not use the builder tmp image?

Yes! This is why I kept point # 2 here

```Dockerfile
COPY airbyte-cdk airbyte-cdk
RUN PYTHONUSERBASE=/usr/local pip install -e ./airbyte-cdk
```
2. Add the following lines to your connector's `Dockerfile`, before the line that installs dependencies via `pip install -e .`:
```Dockerfile
COPY airbyte-cdk airbyte-cdk
RUN pip install -e ./airbyte-cdk
Expand All @@ -128,7 +134,7 @@ airbyteDocker {
You should be able to build your connector with
```bash
# from the airbytehq/airbyte base directory
./gradlew build :airbyte-integrations:connectors:<connector-directory>
./gradlew :airbyte-integrations:connectors:<connector-directory>:airbyteDocker
```
and the installation should use your local CDK. Note that the local CDK is injected at build time, so if you make changes, you will have to run the build command again to see them reflected.
**Note:** if your connector uses a `.dockerignore` file, it cannot have `exclude-all` or `exclude-except` patterns, i.e. the `.dockerignore` must specifically say which files to ignore without using any regex.
Expand Down