-
Notifications
You must be signed in to change notification settings - Fork 16.3k
Migrate to connexion 2 15 #51681
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: v2-11-test
Are you sure you want to change the base?
Migrate to connexion 2 15 #51681
Conversation
13467fc to
a829068
Compare
a829068 to
d2aac39
Compare
1ad0113 to
27034c4
Compare
99f112f to
9dac871
Compare
9dac871 to
fded03e
Compare
a57c33b to
013fb8b
Compare
|
I think cc1a000 needs to be backported to this PR if this is upgrading werkzeug past v3. EDIT: I see the change is already present. |
|
Yeah.. we are one step closer. Constraints are now properly generated for upcoming Airflow 2.11.1 + Fab 1.5.4 .... Now there are few 500 to handle in tests and some warnings when I run it breeze. Anyone who would like to take a look and tackle it are welcome to take a look and help. |
7562830 to
de0330a
Compare
This PR migrates Airflow to connexion 2.15 - which allows Airflow to bump a number of dependencies such as Flask, Werkzeug, flask-session. There are a few incompatibilities fixed during the migration introduced by breaking changes in those dependencies: * flask-session >= 0.8.0 manages it's own session table and we already managed it on our own, so we conditionally create/drop the table when running migrations. * flask-session >= 0.7.9 switched to use MsgSpec as default serializer and while it claims you can switch back to TagJSONSerializer, it's not workign out of the box because it relies on encode/decode methods of the serializer that are not present in TagJSONSerializer. But we cannot use MsgSpec serializer because it does not support some objects (Markup) that we use for Flash message serialization. We borrow a hack from ckan/ckan#8704 and provide our custom serializer deriving from TagJSONSerializer with encode/decode methods added * flask-session >= 0.8.0 switched to use "client" instead of "db" as interface field. Our code was adapted accordingly * Werkzeug >= 2.3.0 changed the way how percent-encoding is done - instead of percent-encoding every reserved character, only the reserved characters that have "reserved purpose" in the place of the URL that they are used are encoded - following WhatWG specification. This means for example that "&" is not encoded in path, "/:" are not encoded in url params, and "?" is not encoded after it is used to separate url from parameters. Our tests expected percent-encoding everywherei (previous Werkzeug behaviour), so they had to be updated. * Flask test client in newer versions of flask does not have "cookie_jar" - it exposes cookies via "get_cookie", tests were updated to use it. * Minimum versions of dependencies have been updated so that all the changes are compatible with them and we do not have to handle back-compatibility with older versions.
Constraints generation was slow because we run them in a loop and we tried to run them all on a single machine - trying to utilize the fact that we only have to build airflow and provider packages once. But those are pretty fast, comparing to constraint generation and it's much better to parallelize the constraint jobs and run them on separatae workers. This will speed up constraint generation delays that will allow building PROD images and kubernetes checks faster.
e94c424 to
13481d0
Compare
13481d0 to
ff377c9
Compare
This PR migrates Airflow to connexion 2.15 - which allows Airflow
to bump a number of dependencies such as Flask, Werkzeug, flask-session.
There are a few incompatibilities fixed during the migration introduced
by breaking changes in those dependencies:
flask-session >= 0.7.9 switched to use MsgSpec as default serializer
and while it claims you can switch back to TagJSONSerializer, it's not
workign out of the box because it relies on encode/decode methods of
the serializer that are not present in TagJSONSerializer. But we
cannot use MsgSpec serializer because it does not support some objects
(Markup) that we use for Flash message serialization. We borrow a hack
from fix: Redis backend cannot store flash messages with HTML ckan/ckan#8704 and provide our custom
serializer deriving from TagJSONSerializer with encode/decode methods
added
flask-session >= 0.8.0 switched to use "client" instead of "db" as
interface field. Our code was adapted accordingly
Werkzeug >= 2.3.0 changed the way how percent-encoding is done -
instead of percent-encoding every reserved character, only the
reserved characters that have "reserved purpose" in the place of
the URL that they are used are encoded - following WhatWG
specification. This means for example that "&" is not encoded in
path, "/:" are not encoded in url params, and "?" is not encoded
after it is used to separate url from parameters. Our tests expected
percent-encoding everywherei (previous Werkzeug behaviour), so they
had to be updated.
Flask test client in newer versions of flask does not have "cookie_jar"
Minimum versions of dependencies have been updated so that all the
changes are compatible with them and we do not have to handle
back-compatibility with older versions.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.