-
Notifications
You must be signed in to change notification settings - Fork 14.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Replace sequence concatination by unpacking in Airflow core #33934
Conversation
More readable and fater: $ python -m timeit '[1, 2, 3] + [4] + [5] + [6] + [7]'
2000000 loops, best of 5: 179 nsec per loop
$ python -m timeit '[*[1, 2, 3], 4, 5, 6, 7]'
5000000 loops, best of 5: 79 nsec per loop
|
@@ -1699,7 +1699,7 @@ def _get_task_instances( | |||
if include_subdags: | |||
# Crafting the right filter for dag_id and task_ids combo | |||
conditions = [] | |||
for dag in self.subdags + [self]: | |||
for dag in [*self.subdags, self]: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if itertools.chain
would be better here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I did some tests on it. It seems performance-wise [*[1, 2, 3], 4, 5, 6, 7]
is the best solution
$ python -m timeit '[1, 2, 3] + [4] + [5] + [6] + [7]'
2000000 loops, best of 5: 179 nsec per loop
$ python -m timeit '[*[1, 2, 3], 4, 5, 6, 7]'
5000000 loops, best of 5: 69.2 nsec per loop
$ python -m timeit 'import itertools; itertools.chain([1, 2, 3], [4, 5, 6, 7])'
2000000 loops, best of 5: 177 nsec per loop
$ python -m timeit 'import itertools; [1, 2, 3] + [4] + [5] + [6] + [7]'
1000000 loops, best of 5: 242 nsec per loop
$ python -m timeit 'import itertools; [*[1, 2, 3], 4, 5, 6, 7]'
2000000 loops, best of 5: 131 nsec per loop
$ python -m timeit 'import itertools; list(itertools.chain([1, 2, 3], [4, 5, 6, 7]))'
1000000 loops, best of 5: 305 nsec per loop
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A big chunk of this would come from import itertools
though, which would not be relevant in Airflow since the module is already imported in a lot of places. I would not be surprised if *
is still best for small lists though, since the itertools version still needs to build an additional list (of a single item).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks good to me if @uranusjr 's suggestion is applied
LGTM. I think using itertools for those is indeed over-the-top. I am fine witth the current version :) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
(cherry picked from commit 33e5d03)
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.