Add UI to engine execution #6647

wisechengyi · 2018-10-17T20:37:08Z

Initially connection between engine execution and the swimlane display. (#6571)

Limitation: having the same task show up in the same lane across updates isn't implemented.

wisechengyi · 2018-10-17T20:47:18Z

Observation: currently the UI is basically showing which tasks have been waiting the longest. E.g. ("Scandir(Dir(\"src/rust/engine/target/debug/build/grpcio-sys-394bf0a62483d9cc/out/build/CMakeFiles/grpc.dir/src/core/lib/transport\"))", 2.817316521s) because there's no way scanning the dir takes 2 sec with only 10 files in it.
Limitation: having the same task show up in the same lane across updates isn't implemented. It's nice to have, but given things update so fast, I am not sure if it has practical impact on users.
Question: The number of swim lanes and whether to show this UI would likely require passing some options from python, how can I go about doing so?

stuhood · 2018-10-17T21:30:48Z

Limitation: having the same task show up in the same lane across updates isn't implemented. It's nice to have, but given things update so fast, I am not sure if it has practical impact on users.

This doesn't block this PR, but it will definitely be an important followup. I suspect that this will significantly improve the readability of the UI, because with no ordering (or even with sorting) something that is rendered 100 times in a row (ie, for about 10 seconds) might hop around to 100 different positions in the task ordering.

But again, not a blocker: easy as a followup.

wisechengyi · 2018-10-17T21:32:30Z

Existing issue: cursor is lost after any pants operation.

stuhood · 2018-10-18T18:00:17Z

Existing issue: cursor is lost after any pants operation.

I think that that qualifies as the "interleaving" mentioned on the ticket. Fixing that is out of scope for this one.

EDIT: Oh, nevermind: I see what you mean. But yes: fine with fixing that in a followup.

stuhood

Thanks! Looks great.

src/python/pants/option/global_options.py

src/python/pants/pantsd/service/scheduler_service.py

src/rust/engine/src/scheduler.rs

stuhood · 2018-10-19T20:10:01Z

src/rust/engine/ui/src/display.rs

@@ -244,7 +244,7 @@ impl EngineDisplay {

  fn render_for_pipe(&self) {
    // TODO: Handle non-tty output w polling interval adjustment + summary rendering.
-    panic!("TODO");
+    // Nothing needs to be printed to pipe by default.


So, while this definitely shouldn't panic, I think that rather than disabling this at this level, we should ensure that when we're inspecting the "should we render the UX" global option, we check whether we're in a tty, and then we explicitly disable the UI rendering if we do not have a TTY.

This information is passed from the pantsd client to the pantsd server via: https://github.com/pantsbuild/pants/blob/master/src/python/pants/java/nailgun_protocol.py#L285-L314 ... but I think that the gist is still that looking at isatty() for stdin/stderr/stdout should tell you whether to render.

Having done that, we should maybe just remove this method entirely if possible...?

Should've clarify a bit that this is not required for this change. Just that I don't want pants to crash on pipe. There is already logic above it which does not call this function at all if it is tty.

pants/src/rust/engine/ui/src/display.rs

Lines 261 to 268 in 298d821

pub fn render(&mut self) {

// TODO: Split this fork out into sub-types of EngineDisplay.

if self.is_tty {

self.render_for_tty()

} else {

self.render_for_pipe()

}

}

Ok. But if my understanding is correct, we should just remove support for attempting to render to a pipe entirely? Maybe that can be a followup ticket: to basically just have the EngineDisplay constructor return a Result::Err if you attempt to create a display against a pipe?

stuhood · 2018-10-19T20:15:06Z

src/python/pants/engine/scheduler.py

@@ -559,14 +560,15 @@ def products_request(self, products, subjects):
      product_results[product].append(state.value)
    return product_results

-  def product_request(self, product, subjects):
+  def product_request(self, product, subjects, render_v2_engine_ui=False):


As discussed on #6654, I think that you should add a @console_rule specific method here, and remove the TODO that Daniel added:

pants/src/python/pants/engine/scheduler.py

Lines 541 to 542 in 3d22e95

# TODO: consider adding a new top-level function adjacent to products_request used for running console tasks,

# so that this code doesn't need to exist in this form.

The idea would be that: a call to product_request(s) would never render the UI, but a call to this new method would, and would also address that TODO. It would also not support batching like product_requests (plural) does.

Made a new method specifically for console rules. However it is not clear to me whether the entire if statement was specifically for console rules, thus can be removed?

pants/src/python/pants/engine/scheduler.py

Lines 591 to 613 in 1b77e56

if throw_root_states:

unique_exceptions = tuple({t.exc for t in throw_root_states})

# TODO: consider adding a new top-level function adjacent to products_request used for running console tasks,

# so that this code doesn't need to exist in this form.

if len(unique_exceptions) == 1 and isinstance(unique_exceptions[0], GracefulTerminationException):

raise unique_exceptions[0]

exception_noun = pluralize(len(unique_exceptions), 'Exception')

if self._scheduler.include_trace_on_error:

cumulative_trace = '\n'.join(self.trace(request))

raise ExecutionError(

'{} encountered:\n{}'.format(exception_noun, cumulative_trace),

unique_exceptions,

)

else:

raise ExecutionError(

'{} encountered:\n {}'.format(

exception_noun,

'\n '.join('{}: {}'.format(type(t).__name__, str(t)) for t in unique_exceptions)),

unique_exceptions

)

stuhood

Thanks!

src/python/pants/engine/scheduler.py

src/python/pants/option/global_options.py

src/rust/engine/src/scheduler.rs

stuhood · 2018-10-22T19:26:00Z

src/rust/engine/ui/src/display.rs

@@ -244,7 +244,7 @@ impl EngineDisplay {

  fn render_for_pipe(&self) {
    // TODO: Handle non-tty output w polling interval adjustment + summary rendering.
-    panic!("TODO");
+    // Nothing needs to be printed to pipe by default.


Ok. But if my understanding is correct, we should just remove support for attempting to render to a pipe entirely? Maybe that can be a followup ticket: to basically just have the EngineDisplay constructor return a Result::Err if you attempt to create a display against a pipe?

src/rust/engine/src/scheduler.rs

wisechengyi · 2018-10-25T05:14:55Z

Ready for another look. Thanks

stuhood

Awesome, thanks!

stuhood · 2018-10-25T19:20:20Z

src/python/pants/engine/scheduler.py

+    :param product: product type for the request.
+    :param subject: subject for the request.
+    :param v2_ui: whether to render the v2 engine UI
+    :return: A dict from product type to lists of products each with length matching len(subjects).


This return type is likely overkill, but easy to fix in a followup.

wisechengyi added 9 commits October 17, 2018 12:24

initial

8da78d7

initial

b287219

single worker works

f22b2a3

basic works

605b788

fmt

cb99460

comments

9a8d71b

fmt

9c288cf

cmt

df8849a

cmt

60700e8

noop instead of panic for non-tty

c2b7e2a

wisechengyi added 3 commits October 17, 2018 16:01

option plumbing

2fcfd7d

optional compiles

a65e348

fmt

c67490a

wisechengyi added 12 commits October 18, 2018 11:00

plumb render option

298d821

fmt

956464f

plubm through run_console_rules codepath

8ac61be

doc

bff4738

Merge branch 'master' into 6571

3f3caf1

fix tests

fb09b8a

correct doc

3ec95d2

clippy if let

75a7474

clippy if let

2a3a45f

clippy

07d4eb5

Merge branch '6571' of https://github.com/wisechengyi/pants into 6571

32547fb

fmt

df64776

wisechengyi changed the title ~~[wip] Add UI to engine execution~~ Add UI to engine execution Oct 19, 2018

stuhood reviewed Oct 19, 2018

View reviewed changes

wisechengyi mentioned this pull request Oct 19, 2018

[wip] Do not hide cursor during v2 UI #6663

Closed

wisechengyi added 4 commits October 19, 2018 15:53

address comments on option

61a3a76

--amend

d9be623

plumb down worker count

96c39bc

fmt

0c95fd9

wisechengyi mentioned this pull request Oct 20, 2018

Improve engine UI #6666

Closed

wisechengyi added 5 commits October 20, 2018 16:28

link improvement ticket

a89dabc

Merge branch 'master' into 6571

ed7d4d0

new method for console rules

1b77e56

doc

d4057e9

extract method

73679c6

wisechengyi force-pushed the 6571 branch from 2efee33 to 20d1101 Compare October 22, 2018 07:33

fix dep and fmt

ac9741d

wisechengyi force-pushed the 6571 branch from 20d1101 to ac9741d Compare October 22, 2018 16:04

stuhood approved these changes Oct 22, 2018

View reviewed changes

wisechengyi added 10 commits October 23, 2018 13:36

adjust interface

bb1839a

refactor exception handling for console rules

a66be04

--v2-ui to global opts + validation

446370b

dangling parenthesis

7739761

make static

a7aa106

display create and initialize works

bc87c72

simplify display logic

3c09ec8

remove non tty case for engine display

a3500fe

fix doc

dda77fd

fmt

725cf8f

stuhood approved these changes Oct 25, 2018

View reviewed changes

stuhood merged commit 6f74c05 into pantsbuild:master Oct 25, 2018

wisechengyi deleted the 6571 branch October 25, 2018 20:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add UI to engine execution #6647

Add UI to engine execution #6647

wisechengyi commented Oct 17, 2018 •

edited

Loading

wisechengyi commented Oct 17, 2018 •

edited

Loading

stuhood commented Oct 17, 2018

wisechengyi commented Oct 17, 2018

stuhood commented Oct 18, 2018 •

edited

Loading

stuhood left a comment

stuhood Oct 19, 2018

wisechengyi Oct 20, 2018

stuhood Oct 22, 2018 •

edited

Loading

stuhood Oct 19, 2018

wisechengyi Oct 21, 2018

stuhood left a comment

stuhood Oct 22, 2018 •

edited

Loading

wisechengyi commented Oct 25, 2018

stuhood left a comment

stuhood Oct 25, 2018

	pub fn render(&mut self) {
	// TODO: Split this fork out into sub-types of EngineDisplay.
	if self.is_tty {
	self.render_for_tty()
	} else {
	self.render_for_pipe()
	}
	}

	# TODO: consider adding a new top-level function adjacent to products_request used for running console tasks,
	# so that this code doesn't need to exist in this form.

	if throw_root_states:
	unique_exceptions = tuple({t.exc for t in throw_root_states})

	# TODO: consider adding a new top-level function adjacent to products_request used for running console tasks,
	# so that this code doesn't need to exist in this form.
	if len(unique_exceptions) == 1 and isinstance(unique_exceptions[0], GracefulTerminationException):
	raise unique_exceptions[0]

	exception_noun = pluralize(len(unique_exceptions), 'Exception')

	if self._scheduler.include_trace_on_error:
	cumulative_trace = '\n'.join(self.trace(request))
	raise ExecutionError(
	'{} encountered:\n{}'.format(exception_noun, cumulative_trace),
	unique_exceptions,
	)
	else:
	raise ExecutionError(
	'{} encountered:\n {}'.format(
	exception_noun,
	'\n '.join('{}: {}'.format(type(t).__name__, str(t)) for t in unique_exceptions)),
	unique_exceptions
	)

Add UI to engine execution #6647

Add UI to engine execution #6647

Conversation

wisechengyi commented Oct 17, 2018 • edited Loading

wisechengyi commented Oct 17, 2018 • edited Loading

stuhood commented Oct 17, 2018

wisechengyi commented Oct 17, 2018

stuhood commented Oct 18, 2018 • edited Loading

stuhood left a comment

Choose a reason for hiding this comment

stuhood Oct 19, 2018

Choose a reason for hiding this comment

wisechengyi Oct 20, 2018

Choose a reason for hiding this comment

stuhood Oct 22, 2018 • edited Loading

Choose a reason for hiding this comment

stuhood Oct 19, 2018

Choose a reason for hiding this comment

wisechengyi Oct 21, 2018

Choose a reason for hiding this comment

stuhood left a comment

Choose a reason for hiding this comment

stuhood Oct 22, 2018 • edited Loading

Choose a reason for hiding this comment

wisechengyi commented Oct 25, 2018

stuhood left a comment

Choose a reason for hiding this comment

stuhood Oct 25, 2018

Choose a reason for hiding this comment

wisechengyi commented Oct 17, 2018 •

edited

Loading

wisechengyi commented Oct 17, 2018 •

edited

Loading

stuhood commented Oct 18, 2018 •

edited

Loading

stuhood Oct 22, 2018 •

edited

Loading

stuhood Oct 22, 2018 •

edited

Loading