diff --git a/dev/README_RELEASE_AIRFLOW.md b/dev/README_RELEASE_AIRFLOW.md index cd695e78d6ad0..32cf899929076 100644 --- a/dev/README_RELEASE_AIRFLOW.md +++ b/dev/README_RELEASE_AIRFLOW.md @@ -23,10 +23,12 @@ - [Selecting what to put into the release](#selecting-what-to-put-into-the-release) - [Selecting what to cherry-pick](#selecting-what-to-cherry-pick) - [Making the cherry picking](#making-the-cherry-picking) + - [Collapse Cadwyn Migrations](#collapse-cadwyn-migrations) - [Reviewing cherry-picked PRs and assigning labels](#reviewing-cherry-picked-prs-and-assigning-labels) - [Prepare the Apache Airflow Package RC](#prepare-the-apache-airflow-package-rc) - [Update the milestone](#update-the-milestone) - [Build RC artifacts](#build-rc-artifacts) + - [Publish release candidate documentation (staging)](#publish-release-candidate-documentation-staging) - [Prepare production Docker Image RC](#prepare-production-docker-image-rc) - [Prepare Vote email on the Apache Airflow release candidate](#prepare-vote-email-on-the-apache-airflow-release-candidate) - [Verify the release candidate by PMC members](#verify-the-release-candidate-by-pmc-members) @@ -42,8 +44,7 @@ - [Publish release to SVN](#publish-release-to-svn) - [Manually prepare production Docker Image](#manually-prepare-production-docker-image) - [Verify production images](#verify-production-images) - - [Publish documentation](#publish-documentation) - - [Wait and make sure documentation is published on the website before proceeding](#wait-and-make-sure-documentation-is-published-on-the-website-before-proceeding) + - [Publish final documentation](#publish-final-documentation) - [Notify developers of release](#notify-developers-of-release) - [Send announcements about security issues fixed in the release](#send-announcements-about-security-issues-fixed-in-the-release) - [Add release data to Apache Committee Report Helper](#add-release-data-to-apache-committee-report-helper) @@ -140,6 +141,15 @@ to keep a reference to the original commit we cherry picked from. ("cherry picke git cherry-pick -x ``` +## Collapse Cadwyn Migrations + +Before cutting an RC, bump the HEAD date of Cadwyn versioned API (execution api for now: `airflow-core/src/airflow/api_fastapi/execution_api`) +to reflect the tentative release date of Airflow. All the Cadwyn migrations in between the tentative release date and last release date +should be collapsed. + +Refer https://github.com/apache/airflow/pull/49116 as a good example. + + ## Reviewing cherry-picked PRs and assigning labels We have the tool that allows to review cherry-picked PRs and assign the labels @@ -331,6 +341,66 @@ pipx install -e ./dev/breeze --current-release ${VERSION} ``` +## Publish release candidate documentation (staging) + +Documentation is an essential part of the product and should be made available to users. +In our cases, documentation for the pre-release versions is published in `staging` S3 bucket. +The documentation source code and build tools are available in the `apache/airflow` repository, so +you need to run several workflows to publish the documentation. More details about it can be found in +[Docs README](../docs/README.md) showing the architecture and workflows including manual workflows for +emergency cases. + +We have two options publishing the documentation 1. Using breeze commands 2. Manually using GitHub Actions.: + +### Using breeze commands + +You can use the `breeze` command to publish the documentation. +The command does the following: + +1. Triggers [Publish Docs to S3](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml). +2. Triggers workflow in apache/airflow-site to refresh +3. Triggers S3 to GitHub Sync + +```shell script + breeze workflow-run publish-docs --ref --site-env +``` + +The `--ref` parameter should be the tag of the release candidate you are publishing. + +The `--site-env` parameter should be set to `staging` for pre-release versions or `live` for final releases. the default option is `auto` +if the tag is rc it publishes to `staging` bucket, otherwise it publishes to `live` bucket. + +Other available parameters can be found with: + +```shell script +breeze workflow-run publish-docs --help +``` + +### Manually using GitHub Actions + +There are two steps to publish the documentation: + +1. Publish the documentation to the `staging` S3 bucket. + +The release manager publishes the documentation using GitHub Actions workflow +[Publish Docs to S3](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml). By default `auto` selection should publish to the `staging` bucket - based on +the tag you use - pre-release tags go to staging. But you can also override it and specify the destination +manually to be `live` or `staging`. + +You should specify 'apache-airflow docker-stack' passed as packages to be +built. + +After that step, the provider documentation should be available under https://airflow.stage.apache.org// +URL (RC PyPI packages are build with the staging urls) but stable links and drop-down boxes are not updated yet. + +2. Invalidate Fastly cache, update version drop-down and stable links with the new versions of the documentation. + +In order to do it, you need to run the [Build docs](https://github.com/apache/airflow-site/actions/workflows/build.yml) +workflow in `airflow-site` repository - but make sure to use `staging` branch. + +After that workflow completes, the new version should be available in the drop-down list and stable links +should be updated, also Fastly cache will be updated + ## Prepare production Docker Image RC Production Docker images should be manually prepared and pushed by the release manager or another committer @@ -816,57 +886,65 @@ docker pull apache/airflow:${VERSION} breeze prod-image verify --image-name apache/airflow:${VERSION} ``` -## Publish documentation +## Publish final documentation Documentation is an essential part of the product and should be made available to users. -In our cases, documentation for the released versions is published in a separate repository - [`apache/airflow-site`](https://github.com/apache/airflow-site), but the documentation source code and build tools are available in the `apache/airflow` repository, so you have to coordinate between the two repositories to be able to build the documentation. +In our cases, documentation for the released versions is published in the `live` S3 bucket, and the site is +kept in a separate repository - [`apache/airflow-site`](https://github.com/apache/airflow-site), +but the documentation source code and build tools are available in the `apache/airflow` repository, so +you need to run several workflows to publish the documentation. More details about it can be found in +[Docs README](../docs/README.md) showing the architecture and workflows including manual workflows for +emergency cases. -Documentation for providers can be found in the ``/docs/apache-airflow`` directory. +We have two options publishing the documentation 1. Using breeze commands 2. Manually using GitHub Actions.: -- First, copy the airflow-site repository and set the environment variable ``AIRFLOW_SITE_DIRECTORY``. +### Using breeze commands - ```shell script - git clone https://github.com/apache/airflow-site.git airflow-site - cd airflow-site - git checkout -b ${VERSION}-docs - export AIRFLOW_SITE_DIRECTORY="$(pwd)" - ``` +You can use the `breeze` command to publish the documentation. +The command does the following: -- Then you can go to the directory and build the necessary documentation packages +1. Triggers [Publish Docs to S3](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml). +2. Triggers workflow in apache/airflow-site to refresh +3. Triggers S3 to GitHub Sync - ```shell script - cd "${AIRFLOW_REPO_ROOT}" - breeze build-docs apache-airflow docker-stack --clean-build - ``` +```shell script + breeze workflow-run publish-docs --ref --site-env +``` -- Now you can preview the documentation. +The `--ref` parameter should be the tag of the final candidate you are publishing. - ```shell script - ./docs/start_doc_server.sh - ``` +The `--site-env` parameter should be set to `staging` for pre-release versions or `live` for final releases. the default option is `auto` +if the tag is rc it publishes to `staging` bucket, otherwise it publishes to `live` bucket. -- Copy the documentation to the ``airflow-site`` repository, create commit, push changes, open a PR and merge it when the build is green. +Other available parameters can be found with: - ```shell script - breeze release-management publish-docs apache-airflow docker-stack - breeze release-management add-back-references apache-airflow --airflow-site-directory "${AIRFLOW_SITE_DIRECTORY}" - breeze sbom update-sbom-information --airflow-version ${VERSION} --airflow-site-directory ${AIRFLOW_SITE_DIRECTORY} --force --all-combinations --run-in-parallel - cd "${AIRFLOW_SITE_DIRECTORY}" - git add . - git commit -m "Add documentation for Apache Airflow ${VERSION}" - git push - # and finally open a PR - ``` +```shell +breeze workflow-run publish-docs --help +``` + +### Manually using GitHub Actions + +There are two steps to publish the documentation: + +1. Publish the documentation to S3 bucket. + +The release manager publishes the documentation using GitHub Actions workflow +[Publish Docs to S3](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml). By +default `auto` selection should publish to the `live` bucket - based on +the tag you use - pre-release tags go to staging. But you can also override it and specify the destination +manually to be `live` or `staging`. -The `--run-in-parallel` switch allows to speed up SBOM generation significantly, but it might take a lot -of memory - if you are running into memory issues you can limit parallelism by setting `--parallelism N` -where N is a number of parallel `cdxgen` servers that should be started. +After that step, the provider documentation should be available under the https://airflow.apache.org/ URL +(also linked directly from the PyPI packages) but stable links and drop-down boxes should not be yet updated. +That allows the Release Manager to verify if the documentation is published. -## Wait and make sure documentation is published on the website before proceeding +2. Invalidate Fastly Cache, update version drop-down and stable links with the new versions of the documentation. -This is important as it takes time for the documentation to be published. You should exercise some patient -here before proceeding with the next steps. +In order to do it, you need to run the [Build docs](https://github.com/apache/airflow-site/actions/workflows/build.yml) +workflow in `airflow-site` repository. Make sure to use `main` as the branch to run the workflow. +After that workflow completes, the new version should be available in the drop-down list and stable links +should be updated, also Fastly cache will be invalidated. ## Notify developers of release @@ -1007,7 +1085,7 @@ This includes: - For major/minor release, update version in `airflow/__init__.py` and `docs/docker-stack/` to the next likely minor version release. - Sync `RELEASE_NOTES.rst` (including deleting relevant `newsfragments`) and `README.md` changes. - Updating `Dockerfile` with the new version. -- Updating `airflow_bug_report.yml` issue template in `.github/ISSUE_TEMPLATE/` with the new version. +- Updating `1-airflow_bug_report.yml` issue template in `.github/ISSUE_TEMPLATE/` with the new version. ## Update default Airflow version in the helm chart @@ -1046,7 +1124,7 @@ Clients can be found here: ### API Clients versioning policy Clients and Core versioning are completely decoupled. Clients also follow SemVer and are updated when core introduce changes relevant to the clients. -Most of the time, if the [openapi specification](https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/api_fastapi/core_api/openapi/v1-rest-api-generated.yaml) has +Most of the time, if the [openapi specification](https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/api_fastapi/core_api/openapi/v2-rest-api-generated.yaml) has changed, clients need to be released. To determine if you should release API clients, you can run from the airflow repository: diff --git a/dev/README_RELEASE_PROVIDERS.md b/dev/README_RELEASE_PROVIDERS.md index b98bfa5de1edc..08492c7f2dec8 100644 --- a/dev/README_RELEASE_PROVIDERS.md +++ b/dev/README_RELEASE_PROVIDERS.md @@ -35,9 +35,9 @@ - [Build Provider distributions for SVN apache upload](#build-provider-distributions-for-svn-apache-upload) - [Build and sign the source and convenience packages](#build-and-sign-the-source-and-convenience-packages) - [Commit the source packages to Apache SVN repo](#commit-the-source-packages-to-apache-svn-repo) - - [Publish the Regular convenience package to PyPI](#publish-the-regular-convenience-package-to-pypi) + - [Publish the Regular distributions to PyPI (release candidates)](#publish-the-regular-distributions-to-pypi-release-candidates) - [Add tags in git](#add-tags-in-git) - - [Prepare documentation](#prepare-documentation) + - [Prepare documentation in Staging](#prepare-documentation-in-staging) - [Prepare issue in GitHub to keep status of testing](#prepare-issue-in-github-to-keep-status-of-testing) - [Prepare voting email for Providers release candidate](#prepare-voting-email-for-providers-release-candidate) - [Verify the release candidate by PMC members](#verify-the-release-candidate-by-pmc-members) @@ -46,8 +46,8 @@ - [Summarize the voting for the Apache Airflow release](#summarize-the-voting-for-the-apache-airflow-release) - [Publish release to SVN](#publish-release-to-svn) - [Publish the packages to PyPI](#publish-the-packages-to-pypi) - - [Publish documentation prepared before](#publish-documentation-prepared-before) - [Add tags in git](#add-tags-in-git-1) + - [Publish documentation](#publish-documentation) - [Update providers metadata](#update-providers-metadata) - [Notify developers of release](#notify-developers-of-release) - [Send announcements about security issues fixed in the release](#send-announcements-about-security-issues-fixed-in-the-release) @@ -415,7 +415,7 @@ svn rm file_name // repeate that for every file svn commit -m "delete old providers" ``` -## Publish the Regular convenience package to PyPI +## Publish the Regular distributions to PyPI (release candidates) In order to publish release candidate to PyPI you just need to build and release packages. The packages should however contain the rcN suffix in the version file name but not internally in the package, @@ -486,127 +486,72 @@ If you want to disable this behaviour, set the env **CLEAN_LOCAL_TAGS** to false breeze release-management tag-providers ``` -## Prepare documentation +## Prepare documentation in Staging Documentation is an essential part of the product and should be made available to users. -In our cases, documentation for the released versions is published in a separate repository - -[`apache/airflow-site`](https://github.com/apache/airflow-site), but the documentation source code -and build tools are available in the `apache/airflow` repository, so you have to coordinate between -the two repositories to be able to build the documentation. +In our cases, documentation for the released versions is published in the staging S3 bucket, and the site is +kept in a separate repository - [`apache/airflow-site`](https://github.com/apache/airflow-site), +but the documentation source code and build tools are available in the `apache/airflow` repository, so +you need to run several workflows to publish the documentation. More details about it can be found in +[Docs README](../docs/README.md) showing the architecture and workflows including manual workflows for +emergency cases. -Documentation for providers can be found in the `/docs/apache-airflow-providers` directory -and the `/docs/apache-airflow-providers-*/` directory. The first directory contains the package contents -lists and should be updated every time a new version of Provider distributions is released. +We have two options publishing the documentation 1. Using breeze commands 2. Manually using GitHub Actions.: -- First, copy the airflow-site repository and set the environment variable ``AIRFLOW_SITE_DIRECTORY``. +### Using breeze commands -```shell script -git clone https://github.com/apache/airflow-site.git airflow-site -cd airflow-site -export AIRFLOW_SITE_DIRECTORY="$(pwd -P)" -``` - -Note if this is not the first time you clone the repo make sure main branch is rebased: - -```shell script -cd "${AIRFLOW_SITE_DIRECTORY}" -git checkout main -git pull --rebase -``` - -- Then you can go to the directory and build the necessary documentation packages - -```shell script -cd "${AIRFLOW_REPO_ROOT}" -breeze build-docs --clean-build apache-airflow-providers all-providers --include-removed-providers -``` - -Usually when we release packages we also build documentation for the "documentation-only" packages. This -means that unless we release just few selected packages or if we need to deliberately skip some packages -we should release documentation for all Provider distributions and the above command is the one to use. - -If we want to just release some providers you can release them using package names: - -```shell script -cd "${AIRFLOW_REPO_ROOT}" -breeze build-docs apache-airflow-providers cncf.kubernetes sftp --clean-build -``` - -Alternatively, if you have set the environment variable: `DISTRIBUTIONS_LIST` above, just run the command: - -```shell script -cd "${AIRFLOW_REPO_ROOT}" -breeze build-docs --clean-build -``` +You can use the `breeze` command to publish the documentation. +The command does the following: -Or using `--distributions-list` argument: - -```shell script -breeze build-docs --distributions-list PACKAGE1,PACKAGE2 -``` - -- Now you can preview the documentation. - -```shell script -./docs/start_doc_server.sh -``` - -If you encounter error like: +1. Triggers [Publish Docs to S3](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml). +2. Triggers workflow in apache/airflow-site to refresh +3. Triggers S3 to GitHub Sync ```shell script -airflow git:(main) ./docs/start_doc_server.sh -./docs/start_doc_server.sh: line 22: cd: /Users/eladkal/PycharmProjects/airflow/docs/_build: No such file or directory + breeze workflow-run publish-docs --ref --site-env ``` -That probably means that the doc folder is empty thus it can not build the doc server. -This indicates that previous step of building the docs did not work. - -- Copy the documentation to the ``airflow-site`` repository - -All providers (including overriding documentation for doc-only changes) - note that publishing is -way faster on multi-cpu machines when you are publishing multiple providers: +The `--ref` parameter should be the tag of the release candidate you are publishing. +The `--site-env` parameter should be set to `staging` for pre-release versions or `live` for final releases. the default option is `auto` +if the tag is rc it publishes to `staging` bucket, otherwise it publishes to `live` bucket. -```shell script -cd "${AIRFLOW_REPO_ROOT}" - -breeze release-management publish-docs apache-airflow-providers all-providers --include-removed-providers \ - --override-versioned --run-in-parallel +Other available parameters can be found with: -breeze release-management add-back-references all-providers +```shell +breeze workflow-run publish-docs --help ``` -If you have providers as list of provider ids because you just released them you can build them with +### Manually using GitHub Actions -```shell script -cd "${AIRFLOW_REPO_ROOT}" +There are two steps to publish the documentation: -breeze release-management publish-docs amazon apache.beam google .... -breeze release-management add-back-references all-providers -``` +1. Publish the documentation to the staging S3 bucket. -Alternatively, if you have set the environment variable: `DISTRIBUTIONS_LIST` above, just run the command: +The release manager publishes the documentation using GitHub Actions workflow +[Publish Docs to S3](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml). -```shell script -breeze release-management publish-docs -breeze release-management add-back-references all-providers -``` +You should specify the final tag to use to build the docs and list of providers to publish +(separated by spaces) or ``all-providers`` in case you want to publish all providers +(optionally you can exclude some of those providers). You should use `staging` bucket to publish the release +candidate documentation. -Or using `--distributions-list` argument: - -```shell script -breeze release-management publish-docs --distributions-list PACKAGE1,PACKAGE2 -breeze release-management add-back-references all-providers -``` +After that step, the provider documentation should be available under the http://airflow.staged.apache.org URL +(also present in the PyPI packages) but stable links and drop-down boxes should not be yet updated. +2. Invalidate Fastly cache, update version drop-down and stable links with the new versions of the documentation. -Review the state of removed, suspended, new packages in +Before doing it - review the state of removed, suspended, new packages in [the docs index](https://github.com/apache/airflow-site/blob/master/landing-pages/site/content/en/docs/_index.md): +Make sure to use `staging` branch to run the workflow. + +There are few special considerations when the list of provider is updated. - If you publish a new package, you must add it to the list of packages in the index. -- If there are changes to suspension or removal status of a package you must move it appropriate section. +- If there are changes to suspension or removal status of a package, you must move it appropriate section. -- Create the commit and push changes. +- In case you need to make any changes - create the commit and push changes and merge it to `staging` branch. + in [airflow-site](https://github.com/apache/airflow-site) repository. ```shell script cd "${AIRFLOW_SITE_DIRECTORY}" @@ -617,6 +562,15 @@ git commit -m "Add documentation for packages - $(date "+%Y-%m-%d%n")" git push --set-upstream origin "${branch}" ``` +Merging the PR with the index changes to `staging` will trigger site publishing. + +If you do not need to merge a PR, you should manually run the +[Build docs](https://github.com/apache/airflow-site/actions/workflows/build.yml) +workflow in `airflow-site` repository to refresh indexes and drop-downs. + +After that build from PR or workflow completes, the new version should be available in the drop-down +list and stable links should be updated, also Fastly cache will be invalidated. + ## Prepare issue in GitHub to keep status of testing Create a GitHub issue with the content generated via manual execution of the command below. You will use @@ -681,12 +635,20 @@ Make sure the packages are in https://dist.apache.org/repos/dist/dev/airflow/pro Send out a vote to the dev@airflow.apache.org mailing list. Here you can prepare text of the email. -subject: +```shell script +export VOTE_DURATION_IN_HOURS=72 +export IS_SHORTEN_VOTE=$([ $VOTE_DURATION_IN_HOURS -ge 72 ] && echo "false" || echo "true") +export SHORTEN_VOTE_TEXT="This is a shortened ($VOTE_DURATION_IN_HOURS hours vote) as agreed by policy set it https://lists.apache.org/thread/cv194w1fqqykrhswhmm54zy9gnnv6kgm" +export VOTE_END_TIME=$(LANG=en_US.UTF-8 TZ=UTC date -v+"${VOTE_DURATION_IN_HOURS}"H "+%B %d, %Y %H:%M %p") +export RELEASE_MANAGER_NAME="Elad Kalif" +export GITHUB_ISSUE_LINK="LINK_TO_GITHUB_ISSUE" +``` +subject: ```shell script cat < +$GITHUB_ISSUE_LINK The issue is also the easiest way to see important PRs included in the RC candidates. Detailed changelog for the providers will be published in the documentation after the @@ -746,7 +708,7 @@ You can find the RC packages in PyPI following these links: Cheers, - +$RELEASE_MANAGER_NAME EOF ``` @@ -1253,21 +1215,6 @@ Copy links to updated packages, sort it alphabetically and save it on the side. * Again, confirm that the packages are available under the links printed. -## Publish documentation prepared before - -Merge the PR that you prepared before with the documentation. - -If you decided to remove some packages from the release make sure to do amend the commit in this way: - -* find the packages you removed in `docs-archive/apache-airflow-providers-` -* remove the latest version (the one you were releasing) -* update `stable.txt` to the previous version -* in the (unlikely) event you are removing first version of package: - * remove whole `docs-archive/apache-airflow-providers-` folder - * remove package from `docs-archive/apache-airflow-providers/core-extensions/index.html` (2 places) - * remove package from `docs-archive/apache-airflow-providers/core-extensions/connections.html` (2 places) - * remove package from `docs-archive/apache-airflow-providers/core-extensions/extra-links.html` (2 places) - * remove package from `docs-archive/apache-airflow-providers/core-extensions/packages-ref.html` (5 places) ## Add tags in git @@ -1283,12 +1230,103 @@ If you want to disable this behaviour, set the env **CLEAN_LOCAL_TAGS** to false breeze release-management tag-providers ``` +## Publish documentation + +Documentation is an essential part of the product and should be made available to users. +In our cases, documentation for the released versions is published in the `live` S3 bucket, and the site is +kept in a separate repository - [`apache/airflow-site`](https://github.com/apache/airflow-site), +but the documentation source code and build tools are available in the `apache/airflow` repository, so +you need to run several workflows to publish the documentation. More details about it can be found in +[Docs README](../docs/README.md) showing the architecture and workflows including manual workflows for +emergency cases. + +We have two options publishing the documentation 1. Using breeze commands 2. Manually using GitHub Actions.: + +### Using breeze commands + +You can use the `breeze` command to publish the documentation. +The command does the following: + +1. Triggers [Publish Docs to S3](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml). +2. Triggers workflow in apache/airflow-site to refresh +3. Triggers S3 to GitHub Sync + +```shell script + breeze workflow-run publish-docs --ref --site-env +``` + +The `--ref` parameter should be the tag of the final candidate you are publishing. + +The `--site-env` parameter should be set to `staging` for pre-release versions or `live` for final releases. the default option is `auto` +if the tag is rc it publishes to `staging` bucket, otherwise it publishes to `live` bucket. + +Other available parameters can be found with: + +```shell +breeze workflow-run publish-docs --help +``` + +### Manually using GitHub Actions + +There are two steps to publish the documentation: + +1. Publish the documentation to the live S3 bucket. + +The release manager publishes the documentation using GitHub Actions workflow +[Publish Docs to S3](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml). + +You should specify the final tag to use to build the docs and list of providers to publish +(separated by spaces) or ``all-providers`` in case you want to publish all providers +(optionally you can exclude some of those providers). + +The release manager publishes the documentation using GitHub Actions workflow +[Publish Docs to S3](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml). +By default `auto` selection should publish to the `live` bucket - based on +the tag you use - pre-release tags go to staging. But you can also override it and specify the destination +manually to be `live` or `staging`. + +After that step, the provider documentation should be available under the http://airflow.apache.org URL +(also present in the PyPI packages) but stable links and drop-down boxes should not be yet updated. + +2. Invalidate Fastly cache, update version drop-down and stable links with the new versions of the documentation. + +Before doing it - review the state of removed, suspended, new packages in +[the docs index](https://github.com/apache/airflow-site/blob/master/landing-pages/site/content/en/docs/_index.md): +Make sure to use `main` branch to run the workflow. + +There are few special considerations when the list of provider is updated. + +- If you publish a new package, you must add it to the list of packages in the index. +- If there are changes to suspension or removal status of a package, you must move it appropriate section. +- In case you need to make any changes - create the commit and push changes and merge it to `main` branch. + in [airflow-site](https://github.com/apache/airflow-site) repository. + +```shell script +cd airflow-site +export AIRFLOW_SITE_DIRECTORY="$(pwd -P)" +cd "${AIRFLOW_SITE_DIRECTORY}" +branch="add-documentation-$(date "+%Y-%m-%d%n")" +git checkout -b "${branch}" +git add . +git commit -m "Add documentation for packages - $(date "+%Y-%m-%d%n")" +git push --set-upstream origin "${branch}" +``` + +Merging the PR with the index changes to `main` will trigger site publishing. + +If you do not need to merge a PR, you should manually run the +[Build docs](https://github.com/apache/airflow-site/actions/workflows/build.yml) +workflow in `airflow-site` repository to refresh indexes and drop-downs. + +After that build from PR or workflow completes, the new version should be available in the drop-down +list and stable links should be updated, also Fastly cache will be invalidated. + ## Update providers metadata ```shell script cd ${AIRFLOW_REPO_ROOT} git checkout main -git pull +git pull apache main current_date=$(date '+%Y-%m-%d%n') branch="update-providers-metadata-${current_date}" git checkout -b "${branch}" diff --git a/dev/breeze/doc/09_release_management_tasks.rst b/dev/breeze/doc/09_release_management_tasks.rst index b83e6e1730261..f6d7bbbebc321 100644 --- a/dev/breeze/doc/09_release_management_tasks.rst +++ b/dev/breeze/doc/09_release_management_tasks.rst @@ -33,21 +33,21 @@ Those are all of the available release management commands: Airflow release commands ........................ -Running airflow release commands is part of the release procedure performed by the release managers +Running Airflow release commands is part of the release procedure performed by the release managers and it is described in detail in `dev `_ . -Preparing airflow distributions +Preparing Airflow distributions """""""""""""""""""""""""" -You can prepare airflow distributions using Breeze: +You can prepare Airflow distributions using Breeze: .. code-block:: bash breeze release-management prepare-airflow-distributions -This prepares airflow .whl package in the dist folder. +This prepares Airflow .whl package in the dist folder. -Again, you can specify optional ``--distribution-format`` flag to build selected formats of airflow distributions, +Again, you can specify optional ``--distribution-format`` flag to build selected formats of Airflow distributions, default is to build ``both`` type of distributions ``sdist`` and ``wheel``. .. code-block:: bash @@ -60,10 +60,10 @@ default is to build ``both`` type of distributions ``sdist`` and ``wheel``. :alt: Breeze release-management prepare-airflow-distributions -Preparing airflow tarball +Preparing Airflow tarball """"""""""""""""""""""""" -You can prepare airflow source tarball using Breeze: +You can prepare Airflow source tarball using Breeze: .. code-block:: bash @@ -127,10 +127,10 @@ When we prepare final release, we automate some of the steps we need to do. :width: 100% :alt: Breeze release-management start-rc-process -Generating airflow core Issue +Generating Airflow core Issue """"""""""""""""""""""""" -You can use Breeze to generate a airflow core issue when you release new airflow. +You can use Breeze to generate a Airflow core issue when you release new airflow. .. image:: ./images/output_release-management_generate-issue-content-providers.svg :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_generate-issue-content-core.svg @@ -266,7 +266,7 @@ The images are also aliased in dockerhub as appropriate. Adding git tags for providers """"""""""""""""""""""""""""" -This command can be utilized to manage git tags for providers within the airflow remote repository during provider releases. +This command can be utilized to manage git tags for providers within the Airflow remote repository during provider releases. Sometimes in cases when there is a connectivity issue to GitHub, it might be possible that local tags get created and lead to annoying errors. The default behaviour would be to clean such local tags up. @@ -376,7 +376,7 @@ Preparing providers You can use Breeze to prepare providers. The distributions are prepared in ``dist`` folder. Note, that this command cleans up the ``dist`` folder -before running, so you should run it before generating airflow package below as it will be removed. +before running, so you should run it before generating Airflow package below as it will be removed. The below example builds providers in the wheel format. @@ -406,7 +406,7 @@ You can see all providers available by running this command: Installing providers """""""""""""""""""" -In some cases we want to just see if the providers generated can be installed with airflow without +In some cases we want to just see if the providers generated can be installed with Airflow without verifying them. This happens automatically on CI for sdist pcackages but you can also run it manually if you just prepared providers and they are present in ``dist`` folder. @@ -414,7 +414,7 @@ just prepared providers and they are present in ``dist`` folder. breeze release-management install-provider-distributions -You can also run the verification with an earlier airflow version to check for compatibility. +You can also run the verification with an earlier Airflow version to check for compatibility. .. code-block:: bash @@ -438,7 +438,7 @@ just prepared providers and they are present in ``dist`` folder. breeze release-management verify-provider-distributions -You can also run the verification with an earlier airflow version to check for compatibility. +You can also run the verification with an earlier Airflow version to check for compatibility. .. code-block:: bash @@ -455,7 +455,7 @@ Generating Providers Metadata """"""""""""""""""""""""""""" The release manager can generate providers metadata per provider version - information about provider versions -including the associated Airflow version for the provider version (i.e first airflow version released after the +including the associated Airflow version for the provider version (i.e first Airflow version released after the provider has been released) and date of the release of the provider version. These are all of the available flags for the ``generate-providers-metadata`` command: @@ -520,17 +520,17 @@ all or selected python version and single constraint mode like this: Constraints are generated separately for each python version and there are separate constraints modes: -* 'constraints' - those are constraints generated by matching the current airflow version from sources +* 'constraints' - those are constraints generated by matching the current Airflow version from sources and providers that are installed from PyPI. Those are constraints used by the users who want to - install airflow with pip. + install Airflow with pip. * "constraints-source-providers" - those are constraints generated by using providers installed from current sources. While adding new providers their dependencies might change, so this set of providers - is the current set of the constraints for airflow and providers from the current main sources. + is the current set of the constraints for Airflow and providers from the current main sources. Those providers are used by CI system to keep "stable" set of constraints. * "constraints-no-providers" - those are constraints generated from only Apache Airflow, without any - providers. If you want to manage airflow separately and then add providers individually, you can + providers. If you want to manage Airflow separately and then add providers individually, you can use those. These are all available flags of ``generate-constraints`` command: @@ -615,7 +615,7 @@ while publishing the documentation. breeze release-management publish-docs --airflow-site-directory You can also use shorthand names as arguments instead of using the full names -for airflow providers. To find the short hand names, follow the instructions in :ref:`generating_short_form_names`. +for Airflow providers. To find the short hand names, follow the instructions in :ref:`generating_short_form_names`. The flag ``--airflow-site-directory`` takes the path of the cloned ``airflow-site``. The command will not proceed if this is an invalid path. @@ -635,7 +635,7 @@ Adding back referencing HTML for the documentation To add back references to the documentation generated by ``build-docs`` in Breeze to ``airflow-site``, use the ``release-management add-back-references`` command. This is important to support backward compatibility -the airflow documentation. +the Airflow documentation. You have to specify which distributions you run it on. For example you can run it for all providers: @@ -690,7 +690,7 @@ Generating Provider requirements In order to generate SBOM information for providers, we need to generate requirements for them. This is done by the ``generate-providers-requirements`` command. This command generates requirements for the -selected provider and python version, using the airflow version specified. +selected provider and python version, using the Airflow version specified. .. image:: ./images/output_sbom_generate-providers-requirements.svg :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_sbom_generate-providers-requirements.svg @@ -712,17 +712,17 @@ These are all of the available flags for the ``update-sbom-information`` command :width: 100% :alt: Breeze update sbom information -Build all airflow images +Build all Airflow images """""""""""""""""""""""" -In order to generate providers requirements, we need docker images with all airflow versions pre-installed, +In order to generate providers requirements, we need docker images with all Airflow versions pre-installed, such images are built with the ``build-all-airflow-images`` command. -This command will build one docker image per python version, with all the airflow versions >=2.0.0 compatible. +This command will build one docker image per python version, with all the Airflow versions >=2.0.0 compatible. .. image:: ./images/output_sbom_build-all-airflow-images.svg :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_sbom_build-all-airflow-images.svg :width: 100% - :alt: Breeze build all airflow images + :alt: Breeze build all Airflow images Exporting SBOM information @@ -741,16 +741,16 @@ properties of the dependencies. This is done by the ``export-dependency-informat Next step: Follow the `Advanced Breeze topics <10_advanced_breeze_topics.rst>`_ to learn more about Breeze internals. -Preparing airflow Task SDK distributions +Preparing Airflow Task SDK distributions """"""""""""""""""""""""""""""""""" -You can prepare airflow distributions using Breeze: +You can prepare Airflow distributions using Breeze: .. code-block:: bash breeze release-management prepare-task-sdk-distributions -This prepares airflow Task SDK .whl package in the dist folder. +This prepares Airflow Task SDK .whl package in the dist folder. Again, you can specify optional ``--distribution-format`` flag to build selected formats of the Task SDK distributions, default is to build ``both`` type of distributions ``sdist`` and ``wheel``. @@ -765,18 +765,18 @@ default is to build ``both`` type of distributions ``sdist`` and ``wheel``. :alt: Breeze release-management prepare-task-sdk-distributions -Preparing airflow ctl distributions +Preparing Airflow ctl distributions """"""""""""""""""""""""""""""""""" -You can prepare airflow distributions using Breeze: +You can prepare Airflow distributions using Breeze: .. code-block:: bash breeze release-management prepare-airflow-ctl-distributions -This prepares airflow Task SDK .whl package in the dist folder. +This prepares Airflow Task SDK .whl package in the dist folder. -Again, you can specify optional ``--distribution-format`` flag to build selected formats of the airflow ctl distributions, +Again, you can specify optional ``--distribution-format`` flag to build selected formats of the Airflow ctl distributions, default is to build ``both`` type of distributions ``sdist`` and ``wheel``. .. code-block:: bash @@ -787,3 +787,92 @@ default is to build ``both`` type of distributions ``sdist`` and ``wheel``. :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_prepare-airflow-ctl-distributions.svg :width: 100% :alt: Breeze release-management prepare-airflow-ctl-distributions + +Publishing the documentation to S3 +"""""""""""""""""""""""""""""""""" + +To publish the documentation generated by ``build-docs`` in Breeze to ``S3``, +use the ``release-management publish-docs-to-s3`` command: + +.. code-block:: bash + + breeze release-management publish-docs-to-s3 + +The documentation publish to S3 should be done after the ``breeze release-management publish-docs`` command. +Once documentation is available in ``docs-archive`` directory of airflow-site, it can be published to S3. + +The publishing documentation to S3 consists of the following steps: + +.. code-block:: bash + + breeze release-management publish-docs --source-dir-path <> --destination-location <> + +Where ``--source-dir-path`` is a doc-archive location path and ``--destination-location`` is the S3 bucket path. + +.. code-block:: bash + + breeze release-management publish-docs --source-dir-path /User/pavan/airflow-site/docs-archive + --destination-location s3://airflow-docs/docs + +To exclude any documentation from publishing to S3, you can use the ``--exclude`` flag. + +.. code-block:: bash + + breeze release-management publish-docs --source-dir-path /User/pavan/airflow-site/docs-archive + --destination-location s3://airflow-docs/docs --exclude "amazon,apache.kafka" + +To override the versioned directories while publishing the documentation to S3, you can use the ``--overwrite`` flag. + +.. code-block:: bash + + breeze release-management publish-docs --source-dir-path /User/pavan/airflow-site/docs-archive + --destination-location s3://airflow-docs/docs --overwrite + +To check what documents will be published to S3, you can use the ``--dry-run`` flag. + +.. code-block:: bash + + breeze release-management publish-docs --source-dir-path /User/pavan/airflow-site/docs-archive + --destination-location s3://airflow-docs/docs --dry-run + +These are all available flags of ``release-management publish-docs-to-s3`` command: + +.. image:: ./images/output_release-management_publish-docs.svg + :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_publish-docs-to-s3.svg + :width: 100% + :alt: Breeze Publish documentation to S3 + + +Trigger GitHub Actions docs publish workflow +""""""""""""""""""""""""""""""""""""""""""""" +To trigger the GitHub Actions workflow that publishes the documentation to S3, you can use the +``breeze workflow-run publish-docs`` command. + +These are all available flags of ``workflow-run`` command: + +.. image:: ./images/output_workflow-run.svg + :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_workflow-run.svg + :width: 100% + :alt: Workflow run command + + +.. code-block:: bash + + breeze workflow-run publish-docs --ref --exclude-docs --site-env --refresh-site --skip-write-to-stable-folder + + example: + breeze workflow-run publish-docs --ref providers-amazon/1.0.0 --site-env live --refresh-site --skip-write-to-stable-folder amazon apache.kafka + +``--ref`` specifies the Git reference tag checkout and build docs. +``--exclude-docs`` specifies the documentation packages to exclude from the publish process. +``--site-env`` specifies the environment to use for the site (e.g., auto, live, staging). the default is auto, based on the ref it decides live or staging. +``--refresh-site`` specifies whether to refresh the site after publishing the documentation. This triggers workflow on apache/airflow-site repository to refresh the site. +``--skip-write-to-stable-folder`` specifies the documentation packages to skip writing to the stable folder. + + +These are all available flags of ``workflow-run publish-docs`` command: + +.. image:: ./images/output_workflow-run_publish-docs.svg + :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_workflow-run_publish-docs.svg + :width: 100% + :alt: Breeze workflow-run publish-docs diff --git a/dev/breeze/doc/images/output-commands.svg b/dev/breeze/doc/images/output-commands.svg index 2d9668f5a21af..030caaeb7d5b2 100644 --- a/dev/breeze/doc/images/output-commands.svg +++ b/dev/breeze/doc/images/output-commands.svg @@ -1,4 +1,4 @@ - +