Use Airflow constraint file for test environment setup #812

jbandoro · 2024-01-23T22:40:34Z

Description

This PR installs apache-airflow in the hatch pre-install-commands section so Airflow related dependency conflicts do not need to be managed in the override section that has been removed.

Installing from the Airflow constraint file for versions <=2.6 fails because PyYAML is pinned in the constraint file for those versions to 6.0.0 and a workaround is required becaused older versions of PyYAML can no longer be installed from unmodified source or sdist with the release of Cython3.

Related Issue(s)

closes #811

Breaking Change?

None

Checklist

I have made corresponding changes to the documentation (if required)
I have added tests that prove my fix is effective or that my feature works

netlify · 2024-01-23T22:40:40Z

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	`2855bba`
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/65b435df0978d60008756182

jbandoro · 2024-01-23T23:48:51Z

@tatiana do you know how to get the Github actions to run with the updated matrix made to the .github/workflows/test.yml in this PR - it seems to be running the workflow file from the main branch.

tatiana

Thanks for working on this, @jbandoro! This part of the configuration can really use some tidying up.

I left some comments inline.

To validate the changes of this branch before merging to main, you could try to add something like:

on:
  push:
    branches: [<BRANCH_NAME>]

To: https://github.com/astronomer/astronomer-cosmos/blob/main/.github/workflows/test.yml#L3

It may be easier if the PR is made from the main Github repo, but there must be a way to set it up pointing to a fork branch as well.

.github/workflows/test.yml

pyproject.toml

codecov · 2024-01-24T19:11:35Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (9c090a4) 94.76% compared to head (3c9281f) 94.77%.
Report is 4 commits behind head on main.

❗ Current head 3c9281f differs from pull request most recent head 2855bba. Consider uploading reports for the commit 2855bba to get more accurate results

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #812   +/-   ##
=======================================
  Coverage   94.76%   94.77%           
=======================================
  Files          55       55           
  Lines        2447     2450    +3     
=======================================
+ Hits         2319     2322    +3     
  Misses        128      128

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

tatiana

Much more maintainable, thanks a lot, @jbandoro !

This PR installs apache-airflow in the hatch `pre-install-commands` section so Airflow related dependency conflicts do not need to be managed in the override section that has been removed. Installing from the Airflow constraint file for versions <=2.6 fails because PyYAML is pinned in the constraint file for those versions to 6.0.0 and [a workaround ](yaml/pyyaml#736) is required becaused older versions of PyYAML can no longer be installed from unmodified source or sdist with the release of Cython3. Closes: #811 (cherry picked from commit 928ba83)

@jbandoro

Bug fixes * Fix: ensure DbtGraph.update_node_dependency is called for all load methods by @jbandoro in #803 * Fix: ensure operator execute method is consistent across all execution base subclasses by @jbandoro in #805 * Fix custom selector when test node has no depends_on values by @tatiana in #814 * Fix forwarding selectors to test task when using TestBehavior.AFTER_ALL (#816) Others * Docs: Remove incorrect docstring from DbtLocalBaseOperator by @jakob-hvitnov-telia in #797 * Add more logs to troubleshoot custom selector by @tatiana in #809 * Fix OpenLineage integration documentation by @tatiana in #810 * Fix test dependencies after Airflow 2.8 release by @jbandoro and @tatiana in #806 * Use Airflow constraint file for test environment setup by @jbandoro in #812 * pre-commit updates in #799, #807

@jbandoro

Bug fixes * Fix: ensure DbtGraph.update_node_dependency is called for all load methods by @jbandoro in #803 * Fix: ensure operator execute method is consistent across all execution base subclasses by @jbandoro in #805 * Fix custom selector when test node has no depends_on values by @tatiana in #814 * Fix forwarding selectors to test task when using TestBehavior.AFTER_ALL (#816) Others * Docs: Remove incorrect docstring from DbtLocalBaseOperator by @jakob-hvitnov-telia in #797 * Add more logs to troubleshoot custom selector by @tatiana in #809 * Fix OpenLineage integration documentation by @tatiana in #810 * Fix test dependencies after Airflow 2.8 release by @jbandoro and @tatiana in #806 * Use Airflow constraint file for test environment setup by @jbandoro in #812 * pre-commit updates in #799, #807

@jbandoro

Bug fixes * Fix: ensure DbtGraph.update_node_dependency is called for all load methods by @jbandoro in #803 * Fix: ensure operator execute method is consistent across all execution base subclasses by @jbandoro in #805 * Fix custom selector when test node has no depends_on values by @tatiana in #814 * Fix forwarding selectors to test task when using TestBehavior.AFTER_ALL by @tatiana in #816 Others * Docs: Remove incorrect docstring from DbtLocalBaseOperator by @jakob-hvitnov-telia in #797 * Add more logs to troubleshoot custom selector by @tatiana in #809 * Fix OpenLineage integration documentation by @tatiana in #810 * Fix test dependencies after Airflow 2.8 release by @jbandoro and @tatiana in #806 * Use Airflow constraint file for test environment setup by @jbandoro in #812 * pre-commit updates in #799, #807

@jbandoro

**Bug fixes** * Fix: ensure ``DbtGraph.update_node_dependency`` is called for all load methods by @jbandoro in #803 * Fix: ensure operator ``execute`` method is consistent across all execution base subclasses by @jbandoro in #805 * Fix custom selector when ``test`` node has no ``depends_on`` values by @tatiana in #814 * Fix forwarding selectors to test task when using ``TestBehavior.AFTER_ALL`` by @tatiana in #816 **Others** * Docs: Remove incorrect docstring from ``DbtLocalBaseOperator`` by @jakob-hvitnov-telia in #797 * Add more logs to troubleshoot custom selector by @tatiana in #809 * Fix OpenLineage integration documentation by @tatiana in #810 * Fix test dependencies after Airflow 2.8 release by @jbandoro and @tatiana in #806 * Use Airflow constraint file for test environment setup by @jbandoro in #812 * pre-commit updates in #799, #807

@jbandoro

Bug fixes * Fix: ensure DbtGraph.update_node_dependency is called for all load methods by @jbandoro in #803 * Fix: ensure operator execute method is consistent across all execution base subclasses by @jbandoro in #805 * Fix custom selector when test node has no depends_on values by @tatiana in #814 * Fix forwarding selectors to test task when using TestBehavior.AFTER_ALL by @tatiana in #816 Others * Docs: Remove incorrect docstring from DbtLocalBaseOperator by @jakob-hvitnov-telia in #797 * Add more logs to troubleshoot custom selector by @tatiana in #809 * Fix OpenLineage integration documentation by @tatiana in #810 * Fix test dependencies after Airflow 2.8 release by @jbandoro and @tatiana in #806 * Use Airflow constraint file for test environment setup by @jbandoro in #812 * pre-commit updates in #799, #807

@jbandoro

**Bug fixes** * Fix: ensure ``DbtGraph.update_node_dependency`` is called for all load methods by @jbandoro in #803 * Fix: ensure operator ``execute`` method is consistent across all execution base subclasses by @jbandoro in #805 * Fix custom selector when ``test`` node has no ``depends_on`` values by @tatiana in #814 * Fix forwarding selectors to test task when using ``TestBehavior.AFTER_ALL`` by @tatiana in #816 **Others** * Docs: Remove incorrect docstring from ``DbtLocalBaseOperator`` by @jakob-hvitnov-telia in #797 * Add more logs to troubleshoot custom selector by @tatiana in #809 * Fix OpenLineage integration documentation by @tatiana in #810 * Fix test dependencies after Airflow 2.8 release by @jbandoro and @tatiana in #806 * Use Airflow constraint file for test environment setup by @jbandoro in #812 * pre-commit updates in #799, #807 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Justin Bandoro <79104794+jbandoro@users.noreply.github.com> Co-authored-by: Jakob Aron Hvitnov <141235900+jakob-hvitnov-telia@users.noreply.github.com>

This PR installs apache-airflow in the hatch `pre-install-commands` section so Airflow related dependency conflicts do not need to be managed in the override section that has been removed. Installing from the Airflow constraint file for versions <=2.6 fails because PyYAML is pinned in the constraint file for those versions to 6.0.0 and [a workaround ](yaml/pyyaml#736) is required becaused older versions of PyYAML can no longer be installed from unmodified source or sdist with the release of Cython3. Closes: astronomer#811

@jbandoro

**Bug fixes** * Fix: ensure ``DbtGraph.update_node_dependency`` is called for all load methods by @jbandoro in astronomer#803 * Fix: ensure operator ``execute`` method is consistent across all execution base subclasses by @jbandoro in astronomer#805 * Fix custom selector when ``test`` node has no ``depends_on`` values by @tatiana in astronomer#814 * Fix forwarding selectors to test task when using ``TestBehavior.AFTER_ALL`` by @tatiana in astronomer#816 **Others** * Docs: Remove incorrect docstring from ``DbtLocalBaseOperator`` by @jakob-hvitnov-telia in astronomer#797 * Add more logs to troubleshoot custom selector by @tatiana in astronomer#809 * Fix OpenLineage integration documentation by @tatiana in astronomer#810 * Fix test dependencies after Airflow 2.8 release by @jbandoro and @tatiana in astronomer#806 * Use Airflow constraint file for test environment setup by @jbandoro in astronomer#812 * pre-commit updates in astronomer#799, astronomer#807 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Justin Bandoro <79104794+jbandoro@users.noreply.github.com> Co-authored-by: Jakob Aron Hvitnov <141235900+jakob-hvitnov-telia@users.noreply.github.com>

use airflow constraint file for environment setup

1b8488a

jbandoro had a problem deploying to external January 23, 2024 22:40 — with GitHub Actions Error

add comment

e722142

jbandoro had a problem deploying to external January 23, 2024 22:52 — with GitHub Actions Error

fix type-check environment

57c63fa

jbandoro temporarily deployed to external January 23, 2024 22:56 — with GitHub Actions Inactive

jbandoro marked this pull request as ready for review January 23, 2024 23:18

jbandoro requested a review from a team as a code owner January 23, 2024 23:18

dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. area:dependencies Related to dependencies, like Python packages, library versions, etc execution:virtualenv Related to Virtualenv execution environment labels Jan 23, 2024

jbandoro had a problem deploying to external January 23, 2024 23:18 — with GitHub Actions Error

tatiana reviewed Jan 24, 2024

View reviewed changes

.github/workflows/test.yml Outdated Show resolved Hide resolved

pyproject.toml Outdated Show resolved Hide resolved

update use minor instead of patch release

1b60d90

jbandoro had a problem deploying to external January 24, 2024 18:50 — with GitHub Actions Error

Merge branch 'main' into 811-install-airflow-with-constraint-file

a3b7cfe

jbandoro temporarily deployed to external January 24, 2024 18:51 — with GitHub Actions Inactive

jbandoro added this to the 1.3.2 milestone Jan 24, 2024

Merge branch 'main' into 811-install-airflow-with-constraint-file

3c9281f

jbandoro temporarily deployed to external January 25, 2024 17:12 — with GitHub Actions Inactive

jbandoro temporarily deployed to external January 25, 2024 17:29 — with GitHub Actions Inactive

Merge branch 'main' into 811-install-airflow-with-constraint-file

2855bba

tatiana temporarily deployed to external January 26, 2024 22:44 — with GitHub Actions Inactive

tatiana approved these changes Jan 26, 2024

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Jan 26, 2024

tatiana merged commit 928ba83 into astronomer:main Jan 26, 2024
39 checks passed

tatiana mentioned this pull request Jan 26, 2024

Release 1.3.2 #817

Merged

tatiana mentioned this pull request Jan 27, 2024

Release 1.3 #818

Merged

tatiana mentioned this pull request Aug 27, 2024

Fix broken CI #1180

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use Airflow constraint file for test environment setup #812

Use Airflow constraint file for test environment setup #812

jbandoro commented Jan 23, 2024

netlify bot commented Jan 23, 2024 •

edited

Loading

jbandoro commented Jan 23, 2024

tatiana left a comment

codecov bot commented Jan 24, 2024 •

edited

Loading

tatiana left a comment

Use Airflow constraint file for test environment setup #812

Use Airflow constraint file for test environment setup #812

Conversation

jbandoro commented Jan 23, 2024

Description

Related Issue(s)

Breaking Change?

Checklist

netlify bot commented Jan 23, 2024 • edited Loading

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

jbandoro commented Jan 23, 2024

tatiana left a comment

Choose a reason for hiding this comment

codecov bot commented Jan 24, 2024 • edited Loading

Codecov Report

tatiana left a comment

Choose a reason for hiding this comment

netlify bot commented Jan 23, 2024 •

edited

Loading

codecov bot commented Jan 24, 2024 •

edited

Loading