[ADAG] Add visualization of compiled graphs

[Bye-legumes]

Why are these changes needed?

#47945

My test case

import ray
import time
import random
from ray.dag.input_node import InputNode
from ray.dag.output_node import MultiOutputNode
ray.init(address = "10.218.163.33:6274")
import os
@ray.remote
class Actor:
    def __init__(self, init_value, fail_after=None, sys_exit=False):
        self.i = init_value
        self.fail_after = fail_after
        self.sys_exit = sys_exit

        self.count = 0


    def echo(self, x):
        self.count += 1
        return x



    def sleep(self, x):
        time.sleep(x)
        return x

    @ray.method(num_returns=2)
    def return_two(self, x):
        return x, x + 1

    def read_input(self, x):
        return x




a = Actor.remote(0)
b = Actor.remote(0)
single_fetch = True
with InputNode() as i:
    o1, o2 = a.return_two.bind(i)
    o3 = b.echo.bind(o1)
    o4 = b.echo.bind(o2)
    dag = MultiOutputNode([o3, o4])

compiled_dag = dag.experimental_compile()
for _ in range(3):
    refs = compiled_dag.execute(1)
    if single_fetch:
        for i, ref in enumerate(refs):
            res = ray.get(ref)
            assert res == i + 1
    else:
        res = ray.get(refs)
        assert res == [1, 2]
compiled_dag.visualize()
compiled_dag.teardown()

plot

Related issue number

Checks

• [√] I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• [√] I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[rkooo567]

cc @kevin85421 can you take a look?

[ruisearch42]

Thanks for the contribution.

[kevin85421]

@ruisearch42 is this different from the visualization function in your overlap PR?

[ruisearch42]

After addressing these comments, I think we are good to merge!

Over time we can always improve and add more features. This will be very useful.

[ruisearch42]

@ruisearch42 is this different from the visualization function in your overlap PR?

This is to visualize the DAG itself, the overlap PR visualizes the schedule. Both will be useful.

[kevin85421]

DAG (non-compiled) seems to have visualization support: https://github.com/ray-project/ray/blob/master/python/ray/dag/vis_utils.py.

What's the difference between the existing one and this PR?

[Bye-legumes]

Hi, @ruisearch42 , I just update it and solved you comments. The new plots is above with different color and shape for different node. The tests are also included.

[Bye-legumes]

I suspect that graphviz is not available on the test environment so we need to pip install graphviz and sudo apt-get install graphviz for visualization test. Do you know how we can achieve that? I need to add dependency to which file for pip and apt-get? @ruisearch42

[kevin85421]

I suspect that graphviz is not available on the test environment so we need to pip install graphviz and sudo apt-get install graphviz for visualization test.

@Bye-legumes maybe you can try to use runtime environment.

Btw, would you mind answering my question here: #47958 (review)? Thanks!

[Bye-legumes]

I suspect that graphviz is not available on the test environment so we need to pip install graphviz and sudo apt-get install graphviz for visualization test.

@Bye-legumes maybe you can try to use runtime environment.

Btw, would you mind answering my question here: #47958 (review)? Thanks!

This is for the complied graph with different nodes that we have different node handling for different node for complied graph. Also the data transport method is also included as we have hint. This is the result of that plot.

How I can set runtime environment for this test case? Should install packages in the test.py with pip install graphviz and sudo apt-get install graphviz

[ruisearch42]

There are test failures for this file. Please fix.

[Bye-legumes]

I suspect that graphviz is not available on the test environment so we need to pip install graphviz and sudo apt-get install graphviz for visualization test. Do you know how we can achieve that in the test environments? I need to add dependency to which file for pip and apt-get? @ruisearch42

[ruisearch42]

I think we can introduce something like a test requirement.txt , something like python/requirements/ml/data-test-requirements.txt
If that is too complex, we can skip the test for now.

[Bye-legumes]

I just skip these tests and I think it's OK now.

[ruisearch42]

Thanks for the change!

[ruisearch42]

@Bye-legumes Auto merge is enabled, but tests are failing, please fix

[Bye-legumes]

@Bye-legumes Auto merge is enabled, but tests are failing, please fix

I see, thx you keep the eye on this! I think this tests failed is not related as other PR also failed on same test. Let me merge the master and check if it will recurs!

[Bye-legumes]

@ruisearch42 Hi, I just merged the master and it's OK now!

[ruisearch42]

@rkooo567 to merge