Fix dev/Dockerfile + Add uv pip install for faster build time
#997
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dosubot
bot
added
the
size:S
✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.
|
dosubot
bot
added
area:dependencies
tatiana
pushed a commit
that referenced
this pull request
Jun 6, 2024
## Fix dev Dockerfile The Dockerfile was not building because `dbt-postgres==1.8.0` pinned `psycopg2`. To resolve this, you need to apt-get install `build-essential` and `libpq-dev`. I added a `RUN` directive that does that. This is placed near the top as it makes the most sense for layering purposes. ## Add uv While I was at it, I added uv: https://github.com/astral-sh/uv tldr: this makes the Dockerfile build a little faster. This speed is especially important due to how the Dockerfile will typically be used: if a user makes a change to the `cosmos/` directory and then wants to test their code, due to how the layers are stacked, the pip install will need to be re-run and resolved on over 100 Python packages on each code change.
Merged
tatiana
added a commit
that referenced
this pull request
Jun 6, 2024
Bug fixes * Fix the invocation mode for ``ExecutionMode.VIRTUALENV`` by @marco9663 in #1023 * Fix Cosmos ``enable_cache`` setting by @tatiana in #1025 * Make ``GoogleCloudServiceAccountDictProfileMapping`` dataset profile arg optional by @oliverrmaa and @pankajastro in #839 and #1017 * Athena profile mapping set ``aws_session_token`` in profile only if it exists by @pankajastro in #1022 Others * Update dbt and Airflow conflicts matrix by @tatiana in #1026 * Enable Python 3.12 unittest by @pankajastro in #1018 * Improve error logging in ``DbtLocalBaseOperator`` by @davidsteinar in #1004 * Add GitHub issue templates for bug reports and feature request by @pankajkoti in #1009 * Add more fields in bug template to reduce turnaround in issue triaging by @pankajkoti in #1027 * Fix ``dev/Dockerfile`` + Add ``uv pip install`` for faster build time by @dwreeves in #997 * Drop support for Airflow 2.3 by @pankajkoti in #994 * Update Astro Runtime image by @RNHTTR in #988 and #989 * Enable ruff F linting by @pankajastro in #985 * Move Cosmos Airflow configuration to settings.py by @pankajastro in #975 * Fix CI Issues by @tatiana in #1005 * Pre-commit hook updates in #1000, #1019 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Reeves <31971762+dwreeves@users.noreply.github.com> Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com> Co-authored-by: davidsteinar <davidsteinar@gmail.com> Co-authored-by: David Steinar Asgrimsson <david-steinar.asgrimsson@kommuninvest.se> Co-authored-by: Marco Yuen <33394715+marco9663@users.noreply.github.com> Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com> Co-authored-by: Ollie Ma <oliver.zheyi.ma@gmail.com>
arojasb3
pushed a commit
to arojasb3/astronomer-cosmos
that referenced
this pull request
Jul 14, 2024
…tronomer#997) ## Fix dev Dockerfile The Dockerfile was not building because `dbt-postgres==1.8.0` pinned `psycopg2`. To resolve this, you need to apt-get install `build-essential` and `libpq-dev`. I added a `RUN` directive that does that. This is placed near the top as it makes the most sense for layering purposes. ## Add uv While I was at it, I added uv: https://github.com/astral-sh/uv tldr: this makes the Dockerfile build a little faster. This speed is especially important due to how the Dockerfile will typically be used: if a user makes a change to the `cosmos/` directory and then wants to test their code, due to how the layers are stacked, the pip install will need to be re-run and resolved on over 100 Python packages on each code change.
arojasb3
pushed a commit
to arojasb3/astronomer-cosmos
that referenced
this pull request
Jul 14, 2024
Bug fixes * Fix the invocation mode for ``ExecutionMode.VIRTUALENV`` by @marco9663 in astronomer#1023 * Fix Cosmos ``enable_cache`` setting by @tatiana in astronomer#1025 * Make ``GoogleCloudServiceAccountDictProfileMapping`` dataset profile arg optional by @oliverrmaa and @pankajastro in astronomer#839 and astronomer#1017 * Athena profile mapping set ``aws_session_token`` in profile only if it exists by @pankajastro in astronomer#1022 Others * Update dbt and Airflow conflicts matrix by @tatiana in astronomer#1026 * Enable Python 3.12 unittest by @pankajastro in astronomer#1018 * Improve error logging in ``DbtLocalBaseOperator`` by @davidsteinar in astronomer#1004 * Add GitHub issue templates for bug reports and feature request by @pankajkoti in astronomer#1009 * Add more fields in bug template to reduce turnaround in issue triaging by @pankajkoti in astronomer#1027 * Fix ``dev/Dockerfile`` + Add ``uv pip install`` for faster build time by @dwreeves in astronomer#997 * Drop support for Airflow 2.3 by @pankajkoti in astronomer#994 * Update Astro Runtime image by @RNHTTR in astronomer#988 and astronomer#989 * Enable ruff F linting by @pankajastro in astronomer#985 * Move Cosmos Airflow configuration to settings.py by @pankajastro in astronomer#975 * Fix CI Issues by @tatiana in astronomer#1005 * Pre-commit hook updates in astronomer#1000, astronomer#1019 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Reeves <31971762+dwreeves@users.noreply.github.com> Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com> Co-authored-by: davidsteinar <davidsteinar@gmail.com> Co-authored-by: David Steinar Asgrimsson <david-steinar.asgrimsson@kommuninvest.se> Co-authored-by: Marco Yuen <33394715+marco9663@users.noreply.github.com> Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com> Co-authored-by: Ollie Ma <oliver.zheyi.ma@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix dev Dockerfile
The Dockerfile was not building because
dbt-postgres==1.8.0pinnedpsycopg2.To resolve this, you need to apt-get install
build-essentialandlibpq-dev. I added aRUNdirective that does that. This is placed near the top as it makes the most sense for layering purposes.Add uv
While I was at it, I added uv: https://github.com/astral-sh/uv tldr: this makes the Dockerfile build a little faster. This speed is especially important due to how the Dockerfile will typically be used: if a user makes a change to the
cosmos/directory and then wants to test their code, due to how the layers are stacked, the pip install will need to be re-run and resolved on over 100 Python packages on each code change.