Skip to content

Commit

Permalink
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Fix reproducibility of prepared provider packages (fix flit frontend) (
Browse files Browse the repository at this point in the history
…apache#43683)

After some checks it turned out that reproducibility of produced
packages depends not only on the build backend configured for the
project but also on the build front-end used - because frontend is
the one to modify meta-data in prepared packages - including the build
tool used, it's version and metadata version supported by the front-end.

That's why in order to maintain reproducibility for anyone who builds
the packages, we have to pin not only the build backend in
pyproject.toml (flit-core) but also build fronted used (flit).

Since package preparation is done with breeze, we can do it by
pinning flit (and just in case also flit-core) so that anyone who
builds specific version of the package will use exactly the same flit
as the person who built the original packages.

This way we will avoid reproducibility problems experienced with 1.5.0
release of FAB.

(cherry picked from commit 18ea01c)
potiuk committed Nov 5, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
1 parent 58c18ff commit 520f4d7
Showing 20 changed files with 48 additions and 49 deletions.
1 change: 0 additions & 1 deletion dev/README_RELEASE_PROVIDER_PACKAGES.md
Original file line number Diff line number Diff line change
@@ -335,7 +335,6 @@ export AIRFLOW_REPO_ROOT=$(pwd -P)
rm -rf ${AIRFLOW_REPO_ROOT}/dist/*
```


* Release candidate packages:

```shell script
2 changes: 1 addition & 1 deletion dev/breeze/README.md
Original file line number Diff line number Diff line change
@@ -66,6 +66,6 @@ PLEASE DO NOT MODIFY THE HASH BELOW! IT IS AUTOMATICALLY UPDATED BY PRE-COMMIT.

---------------------------------------------------------------------------------------------------------

Package config hash: f8e8729f4236f050d4412cbbc9d53fdd4e6ddad65ce5fafd3c5b6fcdacbea5431eea760b961534a63fd5733b072b38e8167b5b0c12ee48b31c3257306ef11940
Package config hash: d1d07397099e14c5fc5f0b0e13a87ac8e112bf66755f77cee62b29151cd18c2f2d35932906db6b3885af652defddce696ef9b2df58e21bd3a7749bca82baf910

---------------------------------------------------------------------------------------------------------
28 changes: 14 additions & 14 deletions dev/breeze/doc/images/output_build-docs.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion dev/breeze/doc/images/output_build-docs.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
ac6594538890f8fba65c916aa8672aa1
91166ce4114ea9c162c139d2aff15886
2 changes: 1 addition & 1 deletion dev/breeze/doc/images/output_prod-image.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion dev/breeze/doc/images/output_prod-image.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
55030fe0d7718eb668fa1a37128647b0
d91bcc76b14f186e749efe2c6aaa8682
2 changes: 1 addition & 1 deletion dev/breeze/doc/images/output_prod-image_build.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
88290b22adcd4e5cc9da29aaa8467992
c243f4de16bc858f6202d88922f00109
2 changes: 1 addition & 1 deletion dev/breeze/doc/images/output_setup.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion dev/breeze/doc/images/output_setup.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
d4a4f1b405f912fa234ff4116068290a
08c78d9dddd037a2ade6b751c5a22ff9
10 changes: 5 additions & 5 deletions dev/breeze/doc/images/output_setup_autocomplete.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion dev/breeze/doc/images/output_setup_autocomplete.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
fffcd49e102e09ccd69b3841a9e3ea8e
ec3b4541a478afe5cb86a6f1c48f50f5
2 changes: 1 addition & 1 deletion dev/breeze/doc/images/output_setup_config.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
96e10564034b282769a2c48ebf7176e2
e77da96b508cc4911857d6f1266802b5
2 changes: 1 addition & 1 deletion dev/breeze/doc/images/output_start-airflow.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2fdb4b01e6d949fb40993e3cc416ca5c
834ca1bef0a55889bfccfeb41738a2f6
15 changes: 14 additions & 1 deletion dev/breeze/pyproject.toml
Original file line number Diff line number Diff line change
@@ -48,7 +48,20 @@ dependencies = [
"black>=23.11.0",
"click>=8.1.7",
"filelock>=3.13.0",
"flit>=3.5.0",
#
# We pin flit in order to make sure reproducibility of provider packages is maintained
# It turns out that when packages are prepared metadata version in the produced packages
# is taken from the front-end not from the backend, so in order to make sure that the
# packages are reproducible, we should pin both backend in "build-system" and frontend in
# "dependencies" of the environment that is used to build the packages.
#
# TODO(potiuk): automate bumping the version of flit in breeze and sync it with
# the version in the template for provider packages with pre-commit also add instructions in
# the source packages explaining that reproducibility can only be achieved by using the same
# version of flit front-end to build the package
#
"flit==3.10.1",
"flit-core==3.10.1",
"gitpython>=3.1.40",
"hatch==1.9.4",
# Importib_resources 6.2.0-6.3.1 break pytest_rewrite
Original file line number Diff line number Diff line change
@@ -341,7 +341,6 @@ def remove_old_releases(version, repo_root):
"--version", required=True, help="The release candidate version e.g. 2.4.3rc1", envvar="VERSION"
)
def prepare_airflow_tarball(version: str):
check_python_version()
from packaging.version import Version

airflow_version = Version(version)
Original file line number Diff line number Diff line change
@@ -520,7 +520,6 @@ def prepare_airflow_packages(
version_suffix_for_pypi: str,
use_local_hatch: bool,
):
check_python_version()
perform_environment_checks()
fix_ownership_using_docker()
cleanup_python_generated_files()
@@ -3067,7 +3066,6 @@ def prepare_helm_chart_tarball(
) -> None:
import yaml

check_python_version()
chart_yaml_file_content = CHART_YAML_FILE.read_text()
chart_yaml_dict = yaml.safe_load(chart_yaml_file_content)
version_in_chart = chart_yaml_dict["version"]
@@ -3209,8 +3207,6 @@ def prepare_helm_chart_tarball(
@option_dry_run
@option_verbose
def prepare_helm_chart_package(sign_email: str):
check_python_version()

import yaml

from airflow_breeze.utils.kubernetes_utils import (
Original file line number Diff line number Diff line change
@@ -39,9 +39,8 @@

# IF YOU WANT TO MODIFY THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
# `pyproject_TEMPLATE.toml.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
#
[build-system]
requires = ["flit_core >=3.2,<4"]
requires = ["flit_core==3.10.1"]
build-backend = "flit_core.buildapi"

[project]
8 changes: 1 addition & 7 deletions dev/breeze/src/airflow_breeze/utils/python_versions.py
Original file line number Diff line number Diff line change
@@ -46,16 +46,10 @@ def get_python_version_list(python_versions: str) -> list[str]:


def check_python_version():
error = False
if not sys.version_info >= (3, 9):
get_console().print("[error]At least Python 3.9 is required to prepare reproducible archives.\n")
error = True
elif not sys.version_info < (3, 12):
get_console().print("[error]Python 3.12 is not supported.\n")
error = True
if error:
get_console().print(
"[warning]Please reinstall Breeze using Python 3.9 - 3.11 environment.[/]\n\n"
"[warning]Please reinstall Breeze using Python 3.9 - 3.12 environment.[/]\n\n"
"If you are using uv:\n\n"
" uv tool install --force --reinstall --python 3.9 -e ./dev/breeze\n\n"
"If you are using pipx:\n\n"
3 changes: 0 additions & 3 deletions dev/breeze/src/airflow_breeze/utils/reproducible.py
Original file line number Diff line number Diff line change
@@ -43,7 +43,6 @@
from subprocess import CalledProcessError, CompletedProcess

from airflow_breeze.utils.path_utils import AIRFLOW_SOURCES_ROOT, OUT_DIR, REPRODUCIBLE_DIR
from airflow_breeze.utils.python_versions import check_python_version
from airflow_breeze.utils.run_utils import run_command


@@ -91,7 +90,6 @@ def reset(tarinfo):
tarinfo.mtime = timestamp
return tarinfo

check_python_version()
OUT_DIR.mkdir(exist_ok=True)
shutil.rmtree(REPRODUCIBLE_DIR, ignore_errors=True)
REPRODUCIBLE_DIR.mkdir(exist_ok=True)
@@ -149,7 +147,6 @@ def reset(tarinfo):


def main():
check_python_version()
parser = ArgumentParser()
parser.add_argument("-a", "--archive", help="archive to repack")
parser.add_argument("-o", "--out", help="archive destination")
4 changes: 3 additions & 1 deletion dev/breeze/uv.lock

0 comments on commit 520f4d7

Please sign in to comment.