Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Standardize airflow build process and switch to Hatchling build backend #36536

Closed
wants to merge 1 commit into from
Closed

Standardize airflow build process and switch to Hatchling build backend #36536

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions .dockerignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,11 +48,11 @@
!.dockerignore
!RELEASE_NOTES.rst
!LICENSE
!MANIFEST.in
!NOTICE
!.github
!empty
!Dockerfile
!hatch_build.py

# This folder is for you if you want to add any packages to the docker context when you build your own
# docker image. most of other files and any new folder you add will be excluded by default
Expand All @@ -68,8 +68,6 @@
!.bash_completion.d

# Setup/version configuration
!setup.cfg
!setup.py
!pyproject.toml
!manifests
!generated
Expand Down
18 changes: 9 additions & 9 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,7 @@ jobs:

# Push early BuildX cache to GitHub Registry in Apache repository, This cache does not wait for all the
# tests to complete - it is run very early in the build process for "main" merges in order to refresh
# cache using the current constraints. This will speed up cache refresh in cases when setup.py
# cache using the current constraints. This will speed up cache refresh in cases when pyproject.toml
# changes or in case of Dockerfile changes. Failure in this step is not a problem (at most it will
# delay cache refresh. It does not attempt to upgrade to newer dependencies.
# We only push CI cache as PROD cache usually does not gain as much from fresh cache because
Expand Down Expand Up @@ -629,9 +629,9 @@ jobs:
id: cache-doc-inventories
with:
path: ./docs/_inventory_cache/
key: docs-inventory-${{ hashFiles('setup.py','setup.cfg','pyproject.toml;') }}
key: docs-inventory-${{ hashFiles('pyproject.toml;') }}
restore-keys: |
docs-inventory-${{ hashFiles('setup.py','setup.cfg','pyproject.toml;') }}
docs-inventory-${{ hashFiles('pyproject.toml;') }}
docs-inventory-
- name: "Build docs"
run: >
Expand Down Expand Up @@ -693,9 +693,9 @@ jobs:
id: cache-doc-inventories
with:
path: ./docs/_inventory_cache/
key: docs-inventory-${{ hashFiles('setup.py','setup.cfg','pyproject.toml;') }}
key: docs-inventory-${{ hashFiles('pyproject.toml;') }}
restore-keys: |
docs-inventory-${{ hashFiles('setup.py','setup.cfg','pyproject.toml;') }}
docs-inventory-${{ hashFiles('pyproject.toml;') }}
docs-inventory-
- name: "Spellcheck docs"
run: >
Expand Down Expand Up @@ -881,8 +881,9 @@ jobs:
path: old-airflow
- name: "Prepare airflow package: wheel"
run: |
pip install pip==23.3.2 wheel==0.36.2 gitpython==3.1.40
python setup.py egg_info --tag-build ".dev0" bdist_wheel -d ../dist
pip install pip==23.3.2 hatch==1.9.1
# TODO(potiuk) make sure dev0 version is used
hatch --tag-build ".dev0" -t wheel
working-directory: ./old-airflow
- name: >
Install and verify all provider packages and airflow on
Expand Down Expand Up @@ -2072,8 +2073,7 @@ jobs:
path: ".build/.k8s-env"
key: "\
k8s-env-${{steps.breeze.outputs.host-python-version}}-\
${{ hashFiles('scripts/ci/kubernetes/k8s_requirements.txt','setup.cfg',\
'setup.py','pyproject.toml','generated/provider_dependencies.json') }}"
${{ hashFiles('scripts/ci/kubernetes/k8s_requirements.txt','pyproject.toml') }}"
- name: Run complete K8S tests ${{needs.build-info.outputs.kubernetes-combos-list-as-string}}
run: breeze k8s run-complete-tests --run-in-parallel --upgrade
env:
Expand Down
2 changes: 0 additions & 2 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,8 +190,6 @@ dmypy.json
log.txt*

# Provider-related ignores
/provider_packages/CHANGELOG.txt
/provider_packages/MANIFEST.in
/airflow/providers/__init__.py

# Docker context files
Expand Down
38 changes: 19 additions & 19 deletions .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -325,13 +325,6 @@ repos:
files: Dockerfile.*$
pass_filenames: true
require_serial: true
- id: check-setup-order
name: Check order of dependencies in setup.cfg and setup.py
language: python
files: ^setup\.cfg$|^setup\.py$
pass_filenames: false
entry: ./scripts/ci/pre_commit/pre_commit_check_order_setup.py
additional_dependencies: ['rich>=12.4.4']
- id: check-airflow-k8s-not-used
name: Check airflow.kubernetes imports are not used
language: python
Expand All @@ -355,14 +348,6 @@ repos:
exclude: ^airflow/kubernetes/|^airflow/providers/
entry: ./scripts/ci/pre_commit/pre_commit_check_cncf_k8s_used_for_k8s_executor_only.py
additional_dependencies: ['rich>=12.4.4']
- id: check-extra-packages-references
name: Checks setup extra packages
description: Checks if all the libraries in setup.py are listed in extra-packages-ref.rst file
language: python
files: ^setup\.py$|^docs/apache-airflow/extra-packages-ref\.rst$|^airflow/providers/.*/provider\.yaml$
pass_filenames: false
entry: ./scripts/ci/pre_commit/pre_commit_check_setup_extra_packages_ref.py
additional_dependencies: ['rich>=12.4.4']
- id: check-airflow-provider-compatibility
name: Check compatibility of Providers with Airflow
entry: ./scripts/ci/pre_commit/pre_commit_check_provider_airflow_compatibility.py
Expand Down Expand Up @@ -392,19 +377,34 @@ repos:
files: ^airflow/providers/.*/hooks/.*\.py$
additional_dependencies: ['rich>=12.4.4', 'pyyaml', 'packaging']
- id: update-providers-dependencies
name: Update cross-dependencies for providers packages
name: Update dependencies for provider packages
entry: ./scripts/ci/pre_commit/pre_commit_update_providers_dependencies.py
language: python
files: ^airflow/providers/.*\.py$|^airflow/providers/.*/provider\.yaml$|^tests/providers/.*\.py$|^tests/system/providers/.*\.py$
files: ^airflow/providers/.*\.py$|^airflow/providers/.*/provider\.yaml$|^tests/providers/.*\.py$|^tests/system/providers/.*\.py$^scripts/ci/pre_commit/pre_commit_update_providers_dependencies\.py$
pass_filenames: false
additional_dependencies: ['setuptools', 'rich>=12.4.4', 'pyyaml']
additional_dependencies: ['setuptools', 'rich>=12.4.4', 'pyyaml', 'tomli']
- id: check-extra-packages-references
name: Checks setup extra packages
description: Checks if all the extras defined in pyproject.toml are listed in extra-packages-ref.rst file
language: python
files: ^docs/apache-airflow/extra-packages-ref\.rst$|^pyproject.toml
pass_filenames: false
entry: ./scripts/ci/pre_commit/pre_commit_check_extra_packages_ref.py
additional_dependencies: ['rich>=12.4.4', 'tomli', 'tabulate']
- id: check-pyproject-toml-order
name: Check order of dependencies in pyproject.toml
language: python
files: ^pyproject\.toml$
pass_filenames: false
entry: ./scripts/ci/pre_commit/pre_commit_check_order_pyproject_toml.py
additional_dependencies: ['rich>=12.4.4']
- id: update-extras
name: Update extras in documentation
entry: ./scripts/ci/pre_commit/pre_commit_insert_extras.py
language: python
files: ^setup\.py$|^CONTRIBUTING\.rst$|^INSTALL$|^airflow/providers/.*/provider\.yaml$
pass_filenames: false
additional_dependencies: ['rich>=12.4.4']
additional_dependencies: ['rich>=12.4.4', 'tomli']
- id: check-extras-order
name: Check order of extras in Dockerfile
entry: ./scripts/ci/pre_commit/pre_commit_check_order_dockerfile_extras.py
Expand Down
22 changes: 11 additions & 11 deletions BREEZE.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1570,7 +1570,7 @@ The CI image is built automatically as needed, however it can be rebuilt manuall

Building the image first time pulls a pre-built version of images from the Docker Hub, which may take some
time. But for subsequent source code changes, no wait time is expected.
However, changes to sensitive files like ``setup.py`` or ``Dockerfile.ci`` will trigger a rebuild
However, changes to sensitive files like ``pyproject.toml`` or ``Dockerfile.ci`` will trigger a rebuild
that may take more time though it is highly optimized to only rebuild what is needed.

Breeze has built in mechanism to check if your local image has not diverged too much from the
Expand Down Expand Up @@ -2300,7 +2300,7 @@ These are all available flags of ``release-management add-back-references`` comm
Generating constraints
""""""""""""""""""""""

Whenever setup.py gets modified, the CI main job will re-generate constraint files. Those constraint
Whenever ``pyproject.toml`` gets modified, the CI main job will re-generate constraint files. Those constraint
files are stored in separated orphan branches: ``constraints-main``, ``constraints-2-0``.

Those are constraint files as described in detail in the
Expand Down Expand Up @@ -2342,14 +2342,14 @@ These are all available flags of ``generate-constraints`` command:
:width: 100%
:alt: Breeze generate-constraints

In case someone modifies setup.py, the scheduled CI Tests automatically upgrades and
In case someone modifies ``pyproject.toml``, the scheduled CI Tests automatically upgrades and
pushes changes to the constraint files, however you can also perform test run of this locally using
the procedure described in the
`Manually generating image cache and constraints <dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md>`_
which utilises multiple processors on your local machine to generate such constraints faster.

This bumps the constraint files to latest versions and stores hash of setup.py. The generated constraint
and setup.py hash files are stored in the ``files`` folder and while generating the constraints diff
This bumps the constraint files to latest versions and stores hash of ``pyproject.toml``. The generated constraint
and ``pyproject.toml`` hash files are stored in the ``files`` folder and while generating the constraints diff
of changes vs the previous constraint files is printed.

Updating constraints
Expand Down Expand Up @@ -2698,18 +2698,18 @@ disappear when you exit Breeze shell.

When you want to add dependencies permanently, then it depends what kind of dependency you add.

If you want to add core dependency that should always be installed - you need to add it to ``setup.cfg``
to ``install_requires`` section. If you want to add it to one of the optional core extras, you should
add it in the extra definition in ``setup.py`` (you need to find out where it is defined). If you want
to add it to one of the providers, you need to add it to the ``provider.yaml`` file in the provider
If you want to add core dependency that should always be installed - you need to add it to ``pyproject.toml``
to ``dependencies`` section. If you want to add it to one of the optional core extras, you should
add it in the extra definition in ``pyproject.toml`` (you need to find out where it is defined).
If you want to add it to one of the providers, you need to add it to the ``provider.yaml`` file in the provider
directory - but remember that this should be followed by running pre-commit that will automatically update
the ``generated/provider_dependencies.json`` directory with the new dependencies:
the ``pyproject.toml`` with the new dependencies as the ``provider.yaml`` files are not used directly, they
are used to update ``pyproject.toml`` file:

.. code-block:: bash

pre-commit run update-providers-dependencies --all-files


You can also run the pre-commit by ``breeze static-checks --type update-providers-dependencies --all-files``
command - which provides autocomplete.

Expand Down
4 changes: 2 additions & 2 deletions CI.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -617,7 +617,7 @@ those via corresponding command line flags passed to ``breeze shell`` command.
| ``UPGRADE_TO_NEWER_DEPENDENCIES`` | false | false | false\* | Determines whether the build should |
| | | | | attempt to upgrade Python base image and all |
| | | | | PIP dependencies to latest ones matching |
| | | | | ``setup.py`` limits. This tries to replicate |
| | | | | ``pyproject.toml`` limits. Tries to replicate |
| | | | | the situation of "fresh" user who just installs |
| | | | | airflow and uses latest version of matching |
| | | | | dependencies. By default we are using a |
Expand All @@ -638,7 +638,7 @@ those via corresponding command line flags passed to ``breeze shell`` command.
| | | | | |
| | | | | Setting the value to random value is best way |
| | | | | to assure that constraints are upgraded even if |
| | | | | there is no change to setup.py |
| | | | | there is no change to ``pyproject.toml`` |
| | | | | |
| | | | | This way our constraints are automatically |
| | | | | tested and updated whenever new versions |
Expand Down
Loading
Loading