airbyte-ci: only install main dependencies when calling poetry install #34945
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Ignored Deployment
|
This stack of pull requests is managed by Graphite. Learn more about stacking. Join @alafanechere and the rest of your teammates on |
alafanechere
force-pushed
the
augustin/02-07-airbyte-ci_only_install_main_dependencies_when_calling_poetry_install
branch
2 times, most recently
from
8e7457f to
8a0003a
Compare
alafanechere
force-pushed
the
augustin/02-07-airbyte-ci_only_install_main_dependencies_when_calling_poetry_install
branch
3 times, most recently
from
8a909b1 to
1b1e55b
Compare
| @@ -159,12 +159,13 @@ def _install_python_dependencies_from_poetry( | |||
| ) -> Container: | |||
| pip_install_poetry_cmd = ["pip", "install", "poetry"] | |||
| poetry_disable_virtual_env_cmd = ["poetry", "config", "virtualenvs.create", "false"] | |||
There was a problem hiding this comment.
Does this imply that inside of the connector images, poetry will install deps into global python package space?
There was a problem hiding this comment.
I think this allows us to then run the connector with the default entry point, as opposed to having to do poetry run whatever.py, right?
There was a problem hiding this comment.
Does this imply that inside of the connector images, poetry will install deps into global python package space?
Yes, building the docker image is a two step approach:
- We provision a builder container on which we run the package installation: this is where poetry is a requirement
- We mount the installed packages to a new container which is transformed into our final image
This means that poetry could eventually not be bundled in our final images, the build dependencies being required only in the builder container.
But for simplicity the builder and the final connector image use the same base image: our python connector base image which has poetry.
If we'd want to optimize the image size we could just install build dependencies in the builder container.
alafanechere
force-pushed
the
augustin/02-07-airbyte-ci_only_install_main_dependencies_when_calling_poetry_install
branch
from
1b1e55b to
05c08a5
Compare
alafanechere
deleted the
augustin/02-07-airbyte-ci_only_install_main_dependencies_when_calling_poetry_install
branch
What
We used the classic
poetry installcommand in a container to install a package.This commands also installs dev dependencies, which is not something we want when no additional depedencie groups are passed to the function
How
Run
poetry install --only mainwhen noadditional_dependency_groupsis passed to the_install_python_dependencies_from_poetryfunction.