Skip to content

Conversation

@potiuk
Copy link
Member

@potiuk potiuk commented Jun 13, 2025

This PR migrates Airflow to connexion 2.15 - which allows Airflow
to bump a number of dependencies such as Flask, Werkzeug, flask-session.

There are a few incompatibilities fixed during the migration introduced
by breaking changes in those dependencies:

  • flask-session >= 0.7.9 switched to use MsgSpec as default serializer
    and while it claims you can switch back to TagJSONSerializer, it's not
    workign out of the box because it relies on encode/decode methods of
    the serializer that are not present in TagJSONSerializer. But we
    cannot use MsgSpec serializer because it does not support some objects
    (Markup) that we use for Flash message serialization. We borrow a hack
    from fix: Redis backend cannot store flash messages with HTML ckan/ckan#8704 and provide our custom
    serializer deriving from TagJSONSerializer with encode/decode methods
    added

  • flask-session >= 0.8.0 switched to use "client" instead of "db" as
    interface field. Our code was adapted accordingly

  • Werkzeug >= 2.3.0 changed the way how percent-encoding is done -
    instead of percent-encoding every reserved character, only the
    reserved characters that have "reserved purpose" in the place of
    the URL that they are used are encoded - following WhatWG
    specification. This means for example that "&" is not encoded in
    path, "/:" are not encoded in url params, and "?" is not encoded
    after it is used to separate url from parameters. Our tests expected
    percent-encoding everywherei (previous Werkzeug behaviour), so they
    had to be updated.

  • Flask test client in newer versions of flask does not have "cookie_jar"

    • it exposes cookies via "get_cookie", tests were updated to use it.
  • Minimum versions of dependencies have been updated so that all the
    changes are compatible with them and we do not have to handle
    back-compatibility with older versions.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@potiuk potiuk changed the base branch from main to v2-11-test June 13, 2025 08:45
@potiuk potiuk marked this pull request as draft June 13, 2025 08:45
@potiuk potiuk closed this Jun 13, 2025
@potiuk potiuk reopened this Jun 13, 2025
@potiuk potiuk force-pushed the attempt-to-migrate-to-connexion-2-15 branch 3 times, most recently from 13467fc to a829068 Compare June 13, 2025 14:03
@potiuk potiuk force-pushed the attempt-to-migrate-to-connexion-2-15 branch from a829068 to d2aac39 Compare June 22, 2025 06:24
@potiuk potiuk force-pushed the attempt-to-migrate-to-connexion-2-15 branch 2 times, most recently from 1ad0113 to 27034c4 Compare June 22, 2025 08:05
@potiuk potiuk linked an issue Jun 22, 2025 that may be closed by this pull request
@potiuk potiuk moved this to In progress in CI / DEV ENV planned work Jun 22, 2025
@potiuk potiuk self-assigned this Jun 22, 2025
@potiuk potiuk changed the title Attempt to migrate to connexion 2 15 Migrate to connexion 2 15 Jun 23, 2025
@potiuk potiuk force-pushed the attempt-to-migrate-to-connexion-2-15 branch 2 times, most recently from 99f112f to 9dac871 Compare June 29, 2025 14:29
@potiuk potiuk moved this from In progress to Dependencies in CI / DEV ENV planned work Jun 30, 2025
@potiuk potiuk force-pushed the attempt-to-migrate-to-connexion-2-15 branch from 9dac871 to fded03e Compare July 1, 2025 10:22
@potiuk potiuk force-pushed the attempt-to-migrate-to-connexion-2-15 branch from a57c33b to 013fb8b Compare October 21, 2025 19:15
@sa1
Copy link
Contributor

sa1 commented Oct 21, 2025

I think cc1a000 needs to be backported to this PR if this is upgrading werkzeug past v3.

EDIT: I see the change is already present.

@potiuk
Copy link
Member Author

potiuk commented Oct 22, 2025

Yeah.. we are one step closer. Constraints are now properly generated for upcoming Airflow 2.11.1 + Fab 1.5.4 .... Now there are few 500 to handle in tests and some warnings when I run it breeze. Anyone who would like to take a look and tackle it are welcome to take a look and help.

This PR migrates Airflow to connexion 2.15 - which allows Airflow
to bump a number of dependencies such as Flask, Werkzeug, flask-session.

There are a few incompatibilities fixed during the migration introduced
by breaking changes in those dependencies:

* flask-session >= 0.8.0 manages it's own session table and we already
  managed it on our own, so we conditionally create/drop the table when
  running migrations.

* flask-session >= 0.7.9 switched to use MsgSpec as default serializer
  and while it claims you can switch back to TagJSONSerializer, it's not
  workign out of the box because it relies on encode/decode methods of
  the serializer that are not present in TagJSONSerializer. But we
  cannot use MsgSpec serializer because it does not support some objects
  (Markup) that we use for Flash message serialization. We borrow a hack
  from ckan/ckan#8704 and provide our custom
  serializer deriving from TagJSONSerializer with encode/decode methods
  added

* flask-session >= 0.8.0 switched to use "client" instead of "db" as
  interface field. Our code was adapted accordingly

* Werkzeug >= 2.3.0 changed the way how percent-encoding is done -
  instead of percent-encoding every reserved character, only the
  reserved characters that have "reserved purpose" in the place of
  the URL that they are used are encoded - following WhatWG
  specification. This means for example that "&" is not encoded in
  path, "/:" are not encoded in url params, and "?" is not encoded
  after it is used to separate url from parameters. Our tests expected
  percent-encoding everywherei (previous Werkzeug behaviour), so they
  had to be updated.

* Flask test client in newer versions of flask does not have "cookie_jar"
  - it exposes cookies via "get_cookie", tests were updated to use it.

* Minimum versions of dependencies have been updated so that all the
  changes are compatible with them and we do not have to handle
  back-compatibility with older versions.
Constraints generation was slow because we run them in a loop and
we tried to run them all on a single machine - trying to utilize
the fact that we only have to build airflow and provider packages
once. But those are pretty fast, comparing to constraint generation
and it's much better to parallelize the constraint jobs and run
them on separatae workers. This will speed up constraint generation
delays that will allow building PROD images and kubernetes checks
faster.
@potiuk potiuk force-pushed the attempt-to-migrate-to-connexion-2-15 branch 3 times, most recently from e94c424 to 13481d0 Compare November 24, 2025 18:35
@potiuk potiuk force-pushed the attempt-to-migrate-to-connexion-2-15 branch from 13481d0 to ff377c9 Compare November 25, 2025 08:31
@potiuk potiuk requested a review from XD-DENG as a code owner December 27, 2025 12:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

all versions If set, the CI build will be forced to use all versions of Python/K8S/DBs area:dev-tools legacy ui Whether legacy UI change should be allowed in PR upgrade to newer dependencies If set, upgrade to newer dependencies is forced

Projects

Status: Dependencies

Development

Successfully merging this pull request may close these issues.

Get rid of the old Werkzeug 2.1 dependency

6 participants