[AIRFLOW-184] Add mark success to CLI #1590

jgao54 · 2016-06-14T15:26:00Z

Dear Airflow Maintainers,

Please accept this PR that addresses the following issues:

(replace with a link to AIRFLOW-184)

Reminders for contributors:

Your PR's title must reference an issue on Airflow's JIRA. For example, a PR called "[AIRFLOW-1] My Amazing PR" would close JIRA issue Improving the search functionality in the graph view #1. Please open a new issue if required!
You must add an Apache License header to all new files.
Please squash your commits when possible and follow the 7 rules of good Git commits.

criccomini · 2016-06-15T21:48:40Z

Looks like tests are unhappy:

TypeError: mark_success() got an unexpected keyword argument 'confirm_propt'

Typo? :)

Also saw this error, which looks a little more serious:

assert "Wait a minute" in response.data.decode('utf-8')
AssertionError

criccomini · 2016-06-15T21:53:07Z

airflow/bin/cli.py

+    dag.mark_success(
+        start_date=args.start_date,
+        end_date=args.end_date,
+        confirm_propt=not args.no_confirm,


Typo? prompt

bolkedebruin · 2016-06-16T08:57:20Z

I think the overall approach needs to be updated. Ie. you do not mark a "dag" as a success but your mark a "DagRun" and/org its underlying tasksinstances as a success. As such I dont like "dag.mark_success".

Creating TIs outside a dagrun is becoming a big no-no and I would say it is even worse to do so in a "view" component, besides it is indeed not required anymore since #1506 as tasks are eagerly created.

All in all, I really like the functionality but please make it DagRun based.

criccomini · 2016-06-16T20:42:43Z

Agree with @bolkedebruin's comments about DAG run vs dag.

criccomini · 2016-06-29T03:35:33Z

airflow/models.py

@@ -3444,7 +3458,7 @@ def find(dag_id=None, run_id=None, execution_date=None,
        return dr

    @provide_session
-    def get_task_instances(self, state=None, session=None):
+    def get_task_instances(self, state=None, to_list=True, session=None):


Nit: this is backwards incompatible, and could lead to issues if someone was using unnamed params and providing a session. Worth a quick glance, but I'm OK with this change.

Actually, the only place that I see this used is in jobs.py (twice), and in tests. Would prefer removing the to_list param, and just returning tis. Then updating code that makes the call to use .all() on the return object.

Ack that this is backwards compatible, but seems very limited usage.

criccomini · 2016-06-29T04:04:22Z

Took another pass and left some comments.

Make sure to take a look at the failing test. Looks like a legit failure after your changes.

Also, we should block on @bolkedebruin for this one, I think, since it's touching DagRun, and he's refactoring stuff there.

artwr · 2016-07-12T22:21:02Z

Just curious to see if you still intend to refactor this, because I think there is a definite use case for this to be enabled. But maybe this can wait for a REST API.

jgao54 · 2016-07-13T15:50:34Z

@artwr @criccomini @bolkedebruin yeah, definitely still intend to refactor this. The current mark success in the UI has a few bugs, (using mark_success in conjunction with past, upstream, or downstream flag throws the error page in some cases). This pr will address those issues as well. I'm ok with waiting for REST API if it's coming up soon, otherwise I think it may be more beneficial to have this merged (once I clean it up =P ).

r39132 · 2016-07-13T19:04:55Z

I think a mark success feature is a good one. I have been looking for one and don't like running a backfill to achieve it. There is one issue I noticed on the current 1.7.1.3 release.

I was 3 days behind on an hourly run. I was hoping a single backfill -m would suffice to mark everything success till current. However, since DAG Runs/Task Instances are generated in little groups by scheduler cycles, that was not the case.... I had to continuously run mark success.. for an hour.

criccomini · 2016-07-14T21:36:59Z

I'm ok with waiting for REST API if it's coming up soon, otherwise I think it may be more beneficial to have this merged (once I clean it up =P ).

I think it's worth getting this PR done. The REST API is likely a long ways away.

jgao54 · 2016-07-18T16:40:13Z

@bolkedebruin I noticed that subdag's TIs are created independently rather than via DagRun (there are currently no dagrun being created for subdags). Was wondering if this is intentional? imho it would make sense to create dagruns for subdags as well if we want to support mark success / clear a subdag via dagrun.

bolkedebruin · 2016-07-18T16:49:39Z

@jgao54 that is indeed so, but old style. I have a PR outstanding that does add dag runs to backfills (that is how subdags run) but it needs reviewing/testing

jgao54 · 2016-07-18T19:35:34Z

@bolkedebruin ah I see it! In that case I'll hold off this PR until yours is merged. I will need to make a bit of changes accordingly, but it will be cleaner.

bolkedebruin · 2016-07-18T20:15:47Z

If you can please review the other pr and +1 it. Thanks!

criccomini · 2016-07-27T21:37:01Z

@bolkedebruin what's the PR?

jgao54 · 2016-07-27T21:39:25Z

@criccomini #1667

jgao54 · 2016-08-14T02:20:01Z

@criccomini @bolkedebruin @jlowin Updated this PR to reflect the changes from #1667

criccomini · 2016-08-15T21:53:58Z

cc @aoen

criccomini · 2016-08-15T21:54:38Z

Seems like all tests are failing?

jgao54 · 2016-08-16T04:37:56Z

@criccomini thanks! fixed

codecov-io · 2016-08-16T05:59:32Z

Current coverage is 64.75% (diff: 79.51%)

Merging #1590 into master will increase coverage by 0.01%

@@             master      #1590   diff @@
==========================================
  Files           127        127          
  Lines          9498       9528    +30   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits           6149       6170    +21   
- Misses         3349       3358     +9   
  Partials          0          0

Powered by Codecov. Last update 821bdb5...7c5199e

r39132 · 2016-10-06T03:52:23Z

@jgao54 Resurrecting this.. please rebase and prepare for review or request to close.

r39132 · 2016-10-11T16:51:45Z

@jgao54 Do you still want to merge this? If not, I will close for inactivity.

r39132 · 2016-10-12T00:11:43Z

@jgao54 closing for inactivity. Please reopen when ready to submit.

jgao54 changed the title ~~AIRFLOW-184 Add mark success to CLI~~ [AIRFLOW-184] Add mark success to CLI Jun 14, 2016

criccomini reviewed Jun 15, 2016
View reviewed changes

jgao54 force-pushed the add_mark_success_to_cli branch from 3d7c908 to 7e55dfe Compare June 22, 2016 07:06

criccomini reviewed Jun 29, 2016
View reviewed changes

jgao54 force-pushed the add_mark_success_to_cli branch 2 times, most recently from ff8b232 to 11873a0 Compare July 18, 2016 06:37

jgao54 force-pushed the add_mark_success_to_cli branch from 11873a0 to 6d966fc Compare August 14, 2016 02:14

jgao54 force-pushed the add_mark_success_to_cli branch 4 times, most recently from 317a4e7 to f8eb4b2 Compare August 15, 2016 16:22

jgao54 force-pushed the add_mark_success_to_cli branch 2 times, most recently from 94ffbf0 to 7dbb615 Compare August 16, 2016 03:59

[AIRFLOW-184] Add mark success to CLI

7c5199e

jgao54 force-pushed the add_mark_success_to_cli branch from 7dbb615 to 7c5199e Compare August 16, 2016 04:22

asfgit closed this in 9b689d0 Oct 12, 2016

alekstorm pushed a commit to alekstorm/incubator-airflow that referenced this pull request Jun 1, 2017

closes apache#1590 *no movement from submitter*

7e68f68

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AIRFLOW-184] Add mark success to CLI #1590

[AIRFLOW-184] Add mark success to CLI #1590

jgao54 commented Jun 14, 2016

criccomini commented Jun 15, 2016

criccomini Jun 15, 2016

bolkedebruin commented Jun 16, 2016

criccomini commented Jun 16, 2016

criccomini Jun 29, 2016

criccomini Jun 29, 2016

criccomini commented Jun 29, 2016

artwr commented Jul 12, 2016

jgao54 commented Jul 13, 2016

r39132 commented Jul 13, 2016 •

edited

Loading

criccomini commented Jul 14, 2016

jgao54 commented Jul 18, 2016

bolkedebruin commented Jul 18, 2016

jgao54 commented Jul 18, 2016

bolkedebruin commented Jul 18, 2016

criccomini commented Jul 27, 2016

jgao54 commented Jul 27, 2016

jgao54 commented Aug 14, 2016

criccomini commented Aug 15, 2016

criccomini commented Aug 15, 2016

jgao54 commented Aug 16, 2016

codecov-io commented Aug 16, 2016 •

edited

Loading

r39132 commented Oct 6, 2016

r39132 commented Oct 11, 2016

r39132 commented Oct 12, 2016

[AIRFLOW-184] Add mark success to CLI #1590

[AIRFLOW-184] Add mark success to CLI #1590

Conversation

jgao54 commented Jun 14, 2016

criccomini commented Jun 15, 2016

criccomini Jun 15, 2016

Choose a reason for hiding this comment

bolkedebruin commented Jun 16, 2016

criccomini commented Jun 16, 2016

criccomini Jun 29, 2016

Choose a reason for hiding this comment

criccomini Jun 29, 2016

Choose a reason for hiding this comment

criccomini commented Jun 29, 2016

artwr commented Jul 12, 2016

jgao54 commented Jul 13, 2016

r39132 commented Jul 13, 2016 • edited Loading

criccomini commented Jul 14, 2016

jgao54 commented Jul 18, 2016

bolkedebruin commented Jul 18, 2016

jgao54 commented Jul 18, 2016

bolkedebruin commented Jul 18, 2016

criccomini commented Jul 27, 2016

jgao54 commented Jul 27, 2016

jgao54 commented Aug 14, 2016

criccomini commented Aug 15, 2016

criccomini commented Aug 15, 2016

jgao54 commented Aug 16, 2016

codecov-io commented Aug 16, 2016 • edited Loading

Current coverage is 64.75% (diff: 79.51%)

r39132 commented Oct 6, 2016

r39132 commented Oct 11, 2016

r39132 commented Oct 12, 2016

r39132 commented Jul 13, 2016 •

edited

Loading

codecov-io commented Aug 16, 2016 •

edited

Loading