Use libyaml C library when available. #14577
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Whoops wrong commit. I'm chaning this to use cyaml everywhere. |
ashb
force-pushed
the
use-cyaml-if-available
branch
from
f739e85
to
7f97396
Compare
ashb
requested review from
dimberman,
kaxil,
mik-laj,
potiuk,
ryanahamilton,
turbaszek and
vikramkoka
as code owners
kaxil
approved these changes
Mar 3, 2021
|
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
github-actions
bot
added
the
full tests needed
|
Oh, To make new providers work with 2.0.0 I'm going to have to do Ugh. (Judos to @potiuk for writing tests for this exact case!) |
This makes loading local providers 1/3 quicker -- from 2s down from 3s on my local SSD. The `airflow.utils.yaml` module can be used in place of the normal yaml module, with the bonus that `safe_load` will use libyaml where available instead of always using the pure python version. I have not used this module in tests/docs code etc, as I don't want to force importing `airflow` (and everything in currently brings in) in to those contexts.
ashb
force-pushed
the
use-cyaml-if-available
branch
from
06c3540
to
93c330d
Compare
|
I think this shaves 3 minutes of the "WWW" tests - down to 8 minutes from 11 minutes, and about 6 minutes overall runtime (58mins for Py3.6 sqlite job, down to 52 -- still too high but this is an easy fix. |
ashb
added a commit
that referenced
this pull request
Mar 19, 2021
This makes loading local providers 1/3 quicker -- from 2s down from 3s on my local SSD. The `airflow.utils.yaml` module can be used in place of the normal yaml module, with the bonus that `safe_load` will use libyaml where available instead of always using the pure python version. This shaves 3 minutes off the "WWW" tests - down to 8 minutes from 11 minutes. I have not used this module in tests/docs code etc, as I don't want to force importing `airflow` (and everything in currently brings in) in to those contexts. (cherry picked from commit 7daebef)
ashb
added a commit
that referenced
this pull request
Apr 15, 2021
This makes loading local providers 1/3 quicker -- from 2s down from 3s on my local SSD. The `airflow.utils.yaml` module can be used in place of the normal yaml module, with the bonus that `safe_load` will use libyaml where available instead of always using the pure python version. This shaves 3 minutes off the "WWW" tests - down to 8 minutes from 11 minutes. I have not used this module in tests/docs code etc, as I don't want to force importing `airflow` (and everything in currently brings in) in to those contexts. (cherry picked from commit 7daebef)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This makes loading local providers (
ProvidersManager.initialize_providers_manager()) 1/3 quicker -- from 2s down from 3s on my local SSD.The
airflow.utils.yamlmodule can be used in place of the normal yaml module, with the bonus thatsafe_loadwill use libyaml where available instead of always using the pure python version.I have not used this module in tests/docs code etc, as I don't want to force importing
airflow(and everything in currently brings in) in to those contexts.