Merge latest main to td_develop #14
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This PR is a response to pretty catastrophic issue caused by expiring key on MySQL repository on 14th of December. Oracle does not follow the best practices for signing their packages (while all others do) and their packages and repositories are signed with a key with short expiry date. This basically puts an expiry on their repository, and anyone who releases images following best practices of installation, while keeping the repository after installing mysql libraries has to rebuild their past released images every 2 years or so. This is the last straw for our MySQL client installation problems (we had a few, especially for ARM images) and we decided that we will switch to MariaDB client libraries by default (while allowing our users to build custom images with MySQL libraries). The issue tracked in Airflow repository is: apache#36231 The issues in Oracle's MySQL repo: * https://bugs.mysql.com/bug.php?id=113427 * https://bugs.mysql.com/bug.php?id=113428 * https://bugs.mysql.com/bug.php?id=113432 This PR implements a number of changes: * MariaDB client is now default client used for both ARM and X86 * both pre-2023 and 2023 keys for MySQL are now added to be trusted when custom image with MySQL client is built * MySQL repository is removed after installing MySQL (to avoid repeating similar fiasco for MySQL users in 2025 * changelog added and instructions on how to build custom image with MySQL client * one of our test suites is converted to use "current" image not latest released image (that was bug in our CI). * test was added in `canary` and `release` builds in CI to also test build of custom image with MySQL client
The apache#36243 change switched mysql client type to mariadb by default and fixed "example docker extension" tests - however it also hard-coded branch name for the image to be tested. This PR makes the branch name depend on the output of the build info job.
When we are building CI/PROD images, AIRFLOW_VERSION arg is retrieved
from the current Airflow version set in version.py. This is find for
main build when we are running image build tests, because there, the
version contains `.dev0` and when we extend the image we can use
`pip install airflow-version=${AIRFLOW_VERSION}`. However in release
builds, the version in `version.py` does not contain version suffix,
and this version of Airflow is not released yet.
This PR fixes it by checking if version_suffix_for_pypi is set,
and it case it is and airflow version does not contain it, we
will add the suffix automatically.
* Remove `black` linter example in `STATIC_CODE_CHECKS.rst` * Revert "Remove `black` linter example in `STATIC_CODE_CHECKS.rst`" This reverts commit bd35364dfd48796da31119698c78095e95ed4176. * Update black linter example in `STATIC_CODE_CHECKS`.rst * Update STATIC_CODE_CHECKS.rst Co-authored-by: Josh Fell <48934154+josh-fell@users.noreply.github.com> * Update STATIC_CODE_CHECKS.rst Co-authored-by: Josh Fell <48934154+josh-fell@users.noreply.github.com> --------- Co-authored-by: Josh Fell <48934154+josh-fell@users.noreply.github.com>
…ng airflow and airbyte (apache#36241) Co-authored-by: Elad Kalif <45845474+eladkal@users.noreply.github.com>
…n get_dag_warnings (apache#36256)
…tion (apache#36111) * Use query params to define cassandra keyspace and table in the query * Follow the same pattern used for keys %(...)s * Switch to different approach
…36255) * Stop deserializing pickle when enable_xcom_pickling is False * Fix unit tests
…e#36261) * Fix code typo ObjectStorage Extrenal Integrations example * add ; to make it prettier
--------- Co-authored-by: Aleph Melo <alephmelo@icloud.com>
Add admin permission too on the page and fix some typo
This was made available [as part of v0.1.8 of the Ruff Formatter](https://astral.sh/blog/ruff-v0.1.8#formatting-code-snippets-in-docstrings). Adding this config option to the `ruff-format` pre-commit hook.
There were few small problems in release process:
1) When the SVN directories were in symbolically linked directories, svn
got into locked directory status
2) Cleaning old relases failed to find the files in current directory,
because breeze changes working directory before running and the `.`
directory was really `airflow` repository, thus no released artifacts
could be found.
3) When the old releases contained `_` rather thatn `-` (as it was
in the bdist_wheel prepared packages) then we failed to delet the
files because the file names for them were constructed from the
normalized file names (with `-`).
This PR fixes those problems:
1) `VAR=$(pwd -P)` is used in order to resolve potentially symbolically
linked urrent working directory and `cd ${VAR}` is used to switch dir
to them
2) The same `$(pwd -P)` is used to pass the directory to the
clean-old-provider-artifacts command to avoid interpreting `.` as
"AIRFLOW_ROOT" directory.
3) Filen name is stored in FileVersion and used to run actual `svn rm`
command - this way the right file name will be used, regardless from
normalization of the package names.
There are few steps missing in the provider release process if you followed it meticulously - the working directory was not changed to the AIRFLOW_ROOT and main should be checked out right before we should create branch to update providers metadata. This PR fixes it.
* add pod_template_content field to kubernetes pod operator * add test for pod_template_content * fix * fix test * test labels * accept dictionary instead of yaml * fix staticcheck warnings * fix import error * change import order --------- Co-authored-by: Hussein Awala <hussein@awala.fr> Co-authored-by: Elad Kalif <45845474+eladkal@users.noreply.github.com>
--------- Co-authored-by: Alberto Costa <alberto.costa@windtre.it>
Add more detail and accuracy to audit logs documentation
… non deferrable mode (apache#36108)
Now that Airflow 2.8.0 is released, we can remove common.io from chicken-egg providers.
…he#36286) Once Airflow is relesed to PyPI we should remove chicken-egg providers for that release and cherry-pick them to v2-*-test in order to prepare container images in case the image contains the providers as default extras.
…pache#36283) When generated dependencies are not properly updated, we had a special step where the dependencies were generated "just in case" before CI image was built, because otherwise building the CI image could have failed with strange "failed because of conflicting dependencies" without a clue what was the root cause. However, the pre-commit did not return error exit code - because for the pre-commit, it is enough that a file is modified during pre-commit to fail the pre-commit in general. That had a nasty side effect because the built CI image actually already contained properly generated dependencies (by this step), and it did not properly detected cases where the ones in the repository were added manually and not generated with pre-commit. This PR fixes it - instead of generating and building such image in CI it will now fail the CI image building step but with clear instructions what to do. The CI job step uses now regular breeze command rather than running the script manually but also the script returns error code in case the generated dependencies have been updated.
…he#36280) When we are installing a released version of Airflow in Breeze, we can pass additional extras to install (For example, we need to pass celery extra in order to start airflow with celery executor. The extras could be specified as: ``` breeze start-airflow --use-airflow-version 2.8.0rc4 \ --executor CeleryExecutor --airflow-extras "celery" ``` However recent refactors caused a problem that the extras added were specified after version (which is rejected by newer versions of `pip`). This PR fixes it and also moves the place where CeleryExecutor use triggers adding celery extra when`--use-airflow-version` is used. The warning about this is better visible when moving to Shell Params.
…pache#36288) When Chicken-egg providers are released, we also have to do some manual adjustments of constraints and version of airflow in the v2-8-test branch.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.