-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix attribute forwarding for tasks with dynamic dependencies #2478
Changes from 2 commits
d5c9765
05158a7
e70c98d
a3ac5d1
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,76 @@ | ||
# -*- coding: utf-8 -*- | ||
# | ||
# Copyright 2012-2015 Spotify AB | ||
# | ||
# Licensed under the Apache License, Version 2.0 (the "License"); | ||
# you may not use this file except in compliance with the License. | ||
# You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
# | ||
|
||
from helpers import LuigiTestCase, RunOnceTask | ||
|
||
import luigi | ||
import luigi.scheduler | ||
import luigi.worker | ||
|
||
|
||
forwarded_attributes = set(luigi.worker.TaskProcess.forward_reporter_attributes.values()) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. make this constant in capital letters? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ✔︎ |
||
|
||
|
||
class NonYieldingTask(RunOnceTask): | ||
|
||
# need to accept messages in order for the "scheduler_message" attribute to be not None | ||
accepts_messages = True | ||
|
||
def gather_forwarded_attributes(self): | ||
attrs = set() | ||
for attr in forwarded_attributes: | ||
if getattr(self, attr, None) is not None: | ||
attrs.add(attr) | ||
return attrs | ||
|
||
def run(self): | ||
self.attributes_while_running = self.gather_forwarded_attributes() | ||
|
||
RunOnceTask.run(self) | ||
|
||
|
||
class YieldingTask(NonYieldingTask): | ||
|
||
def run(self): | ||
self.attributes_before_yield = self.gather_forwarded_attributes() | ||
|
||
yield RunOnceTask() | ||
|
||
self.attributes_after_yield = self.gather_forwarded_attributes() | ||
|
||
RunOnceTask.run(self) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is quite unintuitive, maybe add a comment? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ✔︎ |
||
|
||
|
||
class TaskForwardedAttributesTest(LuigiTestCase): | ||
|
||
def run_task(self, task): | ||
sch = luigi.scheduler.Scheduler() | ||
with luigi.worker.Worker(scheduler=sch) as w: | ||
w.add(task) | ||
w.run() | ||
return task | ||
|
||
def test_non_yielding_task(self): | ||
task = self.run_task(NonYieldingTask()) | ||
|
||
self.assertEqual(task.attributes_while_running, forwarded_attributes) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Am i correct that this test passes before AND after the code changes? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, that's correct. |
||
|
||
def test_yielding_task(self): | ||
task = self.run_task(YieldingTask()) | ||
|
||
self.assertEqual(task.attributes_before_yield, forwarded_attributes) | ||
self.assertEqual(task.attributes_after_yield, forwarded_attributes) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. But this one would fail before this code change? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Exactly. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can remove the white-space here and the love above I think.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
✔︎