diff --git a/dev/README_RELEASE_PYTHON_CLIENT.md b/dev/README_RELEASE_PYTHON_CLIENT.md index 5a487cef16ae0..36bdc103b0a0a 100644 --- a/dev/README_RELEASE_PYTHON_CLIENT.md +++ b/dev/README_RELEASE_PYTHON_CLIENT.md @@ -133,6 +133,8 @@ git log 2.8.0..HEAD --pretty=oneline -- airflow-core/src/airflow/api_fastapi/cor cd ${AIRFLOW_REPO_ROOT} rm dist/* breeze release-management prepare-python-client --distribution-format both --python-client-repo "${CLIENT_REPO_ROOT}" --version-suffix "" +breeze release-management prepare-tarball --tarball-type apache_airflow_python_client --version "${VERSION}" --version-suffix "${VERSION_SUFFIX}" + ``` - This should generate both sdist and .whl package in `dist` folder of the Airflow repository. It should @@ -339,15 +341,34 @@ Airflow Python client supports reproducible builds, which means that the package sources should produce binary identical packages in reproducible way. You should check if the packages can be binary-reproduced when built from the sources. -Checkout airflow sources and build packages in dist folder (replace X.Y.Zrc1 with the version + rc candidate) -you are checking): +1) Set versions of the packages to be checked: + +```shell script +cd +VERSION=X.Y.Z +VERSION_SUFFIX=rc1 +VERSION_RC=${VERSION}${VERSION_SUFFIX} +``` + +2) Change directory where your airflow sources are checked out + +```shell +cd "${AIRFLOW_REPO_ROOT}" +``` + +3) Check out the ``python-client`` tag (assume apache is the remote name of the repository): + +```shell +git fetch apache --tags +git checkout providers/${VERSION_RC} +``` + +4) Build the distribution and source tarball: ```shell script -VERSION=X.Y.Zrc1 -git checkout python-client/${VERSION} -export AIRFLOW_REPO_ROOT=$(pwd) rm -rf dist/* breeze release-management prepare-python-client --distribution-format both --version-suffix "" +breeze release-management prepare-tarball --tarball-type apache_airflow_python_client --version "${VERSION}" --version-suffix "${VERSION_SUFFIX}" ``` The last - build step - by default will use Dockerized build and building of Python client packages @@ -356,13 +377,16 @@ will be done in a docker container. However, if you have `hatch` installed loc ```bash breeze release-management prepare-python-client --distribution-format both --use-local-hatch --version-suffix "" +breeze release-management prepare-tarball --tarball-type apache_airflow_python_client --version "${VERSION}" --version-suffix "${VERSION_SUFFIX}" ``` This is generally faster and requires less resources/network bandwidth. -Both commands should produce reproducible `.whl`, `.tar.gz` packages in dist folder. +Both commands should produce reproducible `.whl`, `.tar.gz` packages in dist folder and "-source.tar.gz" +file containing airflow sources in dist folder. -Change to the directory where you have the packages from svn: +4) Change to the directory where you have the packages from svn and check if they are identical to the ones +you just built: ```shell script # First clone the repo if you do not have it @@ -371,7 +395,7 @@ cd .. svn update --set-depth=infinity asf-dist/dev/airflow/clients/python # Then compare the packages -cd asf-dist/dev/airflow/clients/python/${VERSION} +cd asf-dist/dev/airflow/clients/python/${VERSION_RC} for i in ${AIRFLOW_REPO_ROOT}/dist/* do echo "Checking if $(basename $i) is the same as $i" @@ -386,6 +410,63 @@ In case the files are different, you should see: Binary files apache_airflow-client-2.9.0.tar.gz and .../apache_airflow-2.9.0.tar.gz differ ``` +### Licence check + +This can be done with the Apache RAT tool. + +Download the latest jar from https://creadur.apache.org/rat/download_rat.cgi (unpack the binary, the jar is inside) + +You can run this command to do it for you: + +```shell script +wget -qO- https://dlcdn.apache.org//creadur/apache-rat-0.17/apache-rat-0.17-bin.tar.gz | gunzip | tar -C /tmp -xvf - +``` + +Unpack the release source archive (the `-source.tar.gz` file) to a folder + +```shell script +rm -rf /tmp/apache/airflow-python-client-src && mkdir -p /tmp/apache-airflow-python-client-src && tar -xzf ${PATH_TO_SVN}/providers/${RELEASE_DATE}/apache_airflow_python_client-*-source.tar.gz --strip-components 1 -C /tmp/apache-airflow-python-client-src +``` + +Run the check: + +```shell script +java -jar /tmp/apache-rat-0.17/apache-rat-0.17.jar --input-exclude-file /tmp/apache-airflow-python-client-src/.rat-excludes /tmp/apache-airflow-python-client-src/ | grep -E "! |INFO: " +``` + +You should see no files reported as Unknown or with wrong licence and summary of the check similar to: + +``` +INFO: Apache Creadur RAT 0.17 (Apache Software Foundation) +INFO: Excluding patterns: .git-blame-ignore-revs, .github/*, .git ... +INFO: Excluding MISC collection. +INFO: Excluding HIDDEN_DIR collection. +SLF4J(W): No SLF4J providers were found. +SLF4J(W): Defaulting to no-operation (NOP) logger implementation +SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details. +INFO: RAT summary: +INFO: Approved: 15615 +INFO: Archives: 2 +INFO: Binaries: 813 +INFO: Document types: 5 +INFO: Ignored: 2392 +INFO: License categories: 2 +INFO: License names: 2 +INFO: Notices: 216 +INFO: Standards: 15609 +INFO: Unapproved: 0 +INFO: Unknown: 0 +``` + +There should be no files reported as Unknown or Unapproved. The files that are unknown or unapproved should be shown with a line starting with `!`. + +For example: + +``` +! Unapproved: 1 A count of unapproved licenses. +! /CODE_OF_CONDUCT.md +``` + ## Signature check Make sure you have imported into your GPG the PGP key of the person signing the release. You can find the valid keys in diff --git a/dev/breeze/doc/images/output_release-management_prepare-tarball.svg b/dev/breeze/doc/images/output_release-management_prepare-tarball.svg index 03e24473a257c..278d0a18cd86f 100644 --- a/dev/breeze/doc/images/output_release-management_prepare-tarball.svg +++ b/dev/breeze/doc/images/output_release-management_prepare-tarball.svg @@ -127,8 +127,8 @@ ╭─ Tarball flags ──────────────────────────────────────────────────────────────────────────────────────────────────────╮ │--tarball-typeThe type of tarball to build                                                                    │ -│(apache_airflow | apache_airflow_ctl | apache_airflow_providers | apache_airflow_task_sdk |     │ -│helm-chart)                                                                                     │ +│(apache_airflow | apache_airflow_ctl | apache_airflow_providers | apache_airflow_python_client |│ +│apache_airflow_task_sdk | helm-chart)                                                           │ │[default: apache_airflow]                                                                       │ │--versionVersion to build the tarball for. Must have a corresponding tag in git. If not specified, the   │ │HEAD of current branch will be used and version will be retrieved from there.                   │ diff --git a/dev/breeze/doc/images/output_release-management_prepare-tarball.txt b/dev/breeze/doc/images/output_release-management_prepare-tarball.txt index 45c6248ab2c96..0435e78539747 100644 --- a/dev/breeze/doc/images/output_release-management_prepare-tarball.txt +++ b/dev/breeze/doc/images/output_release-management_prepare-tarball.txt @@ -1 +1 @@ -7cb732b47aeebbb6b430cda72776a7a8 +1e525a6dd6717f0ce733a5f679c83489 diff --git a/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py b/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py index 2e9564d9853cc..5f81a334491c9 100644 --- a/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py +++ b/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py @@ -332,6 +332,11 @@ def create_tarball_release( if not version: version = date.strftime(date.today(), "%Y-%m-%d") console_print(f"\n[info]Using current date {version} as tarball version\n") + elif tarball_type == TarBallType.PYTHON_CLIENT: + tag = f"python-client/{version + version_suffix}" if version else "HEAD" + if not version: + version = date.strftime(date.today(), "%Y-%m-%d") + console_print(f"\n[info]Using current date {version} as tarball version\n") else: # pragma: no cover console_print(f"[error]Unsupported tarball type: {tarball_type}") exit(1) diff --git a/dev/breeze/src/airflow_breeze/global_constants.py b/dev/breeze/src/airflow_breeze/global_constants.py index 3aeea527fb57b..26a82821da58a 100644 --- a/dev/breeze/src/airflow_breeze/global_constants.py +++ b/dev/breeze/src/airflow_breeze/global_constants.py @@ -243,6 +243,7 @@ class TarBallType(Enum): PROVIDERS = "apache_airflow_providers" TASK_SDK = "apache_airflow_task_sdk" AIRFLOW_CTL = "apache_airflow_ctl" + PYTHON_CLIENT = "apache_airflow_python_client" HELM_CHART = "helm-chart"