[Runtime] Extend Graph Runtime To Support Cuda Graph Launch #7616

zhuochenKIDD · 2021-03-09T09:22:11Z

We are currently using graph runtime to run some CTR models on NV-GPU, for our in-house model (around 100 nodes in tvm json graph ) cuGraphLaunch can reduce 5% to 10% percent latency vs the original for-loop cuda kernel launch.

So I wonder if the extension might benefits other workloads, I haven't test other types of models.

comaniac · 2021-03-09T18:33:33Z

@zhuochenKIDD is this ready for review? Please modify the description if so; otherwise please mark this PR as a draft first. Thanks.

zhuochenKIDD · 2021-03-11T11:52:37Z

@comaniac I've added test case, would you please help review, thanks.

tests/python/unittest/test_runtime_graph_cugraph.py

CMakeLists.txt

comaniac · 2021-03-11T20:23:40Z

CMakeLists.txt

+  if(USE_CUDA)
+    if(USE_GRAPH_RUNTIME_CUGRAPH)


This makes USE_GRAPH_RUNTIME_CUGRAPH silent when CUDA is OFF and may confuse users. We should have

if(USE_GRAPH_RUNTIME_CUGRAPH) if(NOT USE_CUDA) // error out saying please config with USE_CUDA=ON.

I moved this to CUDA.cmake to better check CUDA version > 10, so it might like cudnn/cublas feature, is that ok?

python/tvm/contrib/cu_graph/cugraph_runtime.py

comaniac · 2021-03-11T20:25:06Z

python/tvm/contrib/cu_graph/cugraph_runtime.py

+    except ValueError:
+        raise ValueError(
+            "Please set '(USE_GRAPH_RUNTIME_CUGRAPH ON)' in "
+            "config.cmake and rebuild TVM to enable cu_graph test mode"


Why test mode?

It is because we are currently evaluating CUDA graph API vs kernel launch, and it's keep on going, using TVM is more convenient to do so on new workloads than TF Runtime. And also currently only Kernel-kind cuda node is in captured CUDA graph, in might be more benefits when Memcpy-kind node or using manually created cuda graph, so currently I am not sure current stream-capture way is the optimal way, perhaps need more test

I see. We usually call it "experimental". I'll suggest the following:

To enable CuGraph (experimental), please set '(USE_GRAPH_RUNTIME_CUGRAPH ON)' in config.cmake and rebuild TVM

src/runtime/graph/cugraph/graph_runtime_cugraph.cc

tests/python/unittest/test_runtime_graph_cugraph.py

Co-authored-by: Cody Yu <comaniac0422@gmail.com>

This reverts commit f286711.

comaniac

Just some miner changes but overall is good. Two additional points:

I found two terms cuGraph and CUDA graph are used in this PR. It would be better to just use cuGraph.
It would be great if you could send a follow-up PR for a tutorial to explain how to use the two interfaces.

cmake/config.cmake

cmake/modules/CUDA.cmake

comaniac · 2021-03-16T16:55:46Z

cmake/modules/CUDA.cmake

+    if(CUDAToolkit_VERSION_MAJOR LESS "10")
+      message(FATAL_ERROR "CUDA Graph requires at least CUDA 10, got=" ${CUDAToolkit_VERSION})
+    endif()
+    message(STATUS "Build with Graph runtime cuGraph support...")


It would be better to have one terminology in this PR. Either cuGraph or CUDA graph.

Yes I removed all cuGraph or cu_graph name, use CUDA Graph instread

comaniac · 2021-03-16T16:58:16Z

python/tvm/contrib/nvcc.py

+            return False
+        return True
+    except RuntimeError:
+        warnings.warn("Cannot find cuda path")


This warning has no information and can consider to remove.

python/tvm/testing.py

src/runtime/graph/graph_runtime_factory.cc

tests/python/unittest/test_runtime_graph_cugraph.py

Co-authored-by: Cody Yu <comaniac0422@gmail.com>

zhuochenKIDD · 2021-03-17T08:11:52Z

Just some miner changes but overall is good. Two additional points:

I found two terms cuGraph and CUDA graph are used in this PR. It would be better to just use cuGraph.

It would be great if you could send a follow-up PR for a tutorial to explain how to use the two interfaces.

I removed cuGraph and changed code to CUDA graph because it's NV official terminology and found cuGraph is another lib for graph algorithms
I will add more docs when ready, by tutorial do you mean I add a py in tutorials/frontend or a rst in docs/dev?

comaniac

LGTM. I'm going to merge this first and the doc could be the next PR.
For the doc location, it's reasonable to put it under TVM runtime along with debugger (.rst), but to me this is a feature not limited to developers, so it would be more impactful if we put it under tutorial (.py). @tqchen @hogepodge could you please advice?

comaniac · 2021-03-17T16:39:16Z

Thanks @zhuochenKIDD

tqchen · 2021-03-17T17:20:41Z

a tutorial/howto guide would be nice

hogepodge · 2021-03-18T00:56:59Z

Agree with Tianqi. A how-to guide would be best. You can write it as a Sphinx-Gallery document, under the tvm/tutorials directory. I'm not entirely certain which subdirectory it should go under (you should avoid the get_started directory). Maybe a new directory if it doesn't fit into classifications for the others.

) * add graph runtime cuGraph poc * lint format * add unittest * fix review comments * Update CMakeLists.txt Co-authored-by: Cody Yu <comaniac0422@gmail.com> * build cuda graph runtime in gpu test * Revert "build cuda graph runtime in gpu test" This reverts commit f286711. * rename cuGraph to CUDA Graph * rename cuda_graph * rename cuda_graph * lint format * Update src/runtime/graph/graph_runtime_factory.cc Co-authored-by: Cody Yu <comaniac0422@gmail.com> * Update python/tvm/testing.py Co-authored-by: Cody Yu <comaniac0422@gmail.com> * fix lint error * remove unnecessary warn * add test, fix lint * fix lint W0223 Co-authored-by: Cody Yu <comaniac0422@gmail.com>

zhuochenKIDD added 2 commits March 9, 2021 16:54

add graph runtime cuGraph poc

300d8ac

lint format

cc97f6b

zhuochenKIDD mentioned this pull request Mar 9, 2021

[Runtime] Extend Graph Runtime To Support Cuda Graph Launch #7573

Closed

tqchen assigned comaniac Mar 9, 2021

tqchen added the status: need review label Mar 9, 2021

add unittest

ed3b250

comaniac requested changes Mar 11, 2021

View reviewed changes

zhuochenKIDD and others added 5 commits March 16, 2021 19:30

fix review comments

520e532

Merge branch 'main' into main

5754f5a

Update CMakeLists.txt

96f94eb

Co-authored-by: Cody Yu <comaniac0422@gmail.com>

build cuda graph runtime in gpu test

f286711

Revert "build cuda graph runtime in gpu test"

cc36e34

This reverts commit f286711.

comaniac requested changes Mar 16, 2021

View reviewed changes

zhuochenKIDD and others added 6 commits March 17, 2021 15:28

rename cuGraph to CUDA Graph

604980d

rename cuda_graph

41e6b9a

rename cuda_graph

f437a2d

lint format

88812ac

Update src/runtime/graph/graph_runtime_factory.cc

f128580

Co-authored-by: Cody Yu <comaniac0422@gmail.com>

Update python/tvm/testing.py

67080c3

Co-authored-by: Cody Yu <comaniac0422@gmail.com>

zhuochenKIDD added 5 commits March 17, 2021 16:59

fix lint error

ab4f8c3

remove unnecessary warn

6ae69c5

add test, fix lint

eac3ce9

Merge branch 'main' of github.com:zhuochenKIDD/incubator-tvm into main

7f844c0

fix lint W0223

2627831

comaniac approved these changes Mar 17, 2021

View reviewed changes

comaniac merged commit 60ff0c7 into apache:main Mar 17, 2021

comaniac added status: accepted and removed status: need review labels Mar 17, 2021

junrushao mentioned this pull request Nov 1, 2021

Apache TVM v0.8 Release Note Candidate #9416

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Runtime] Extend Graph Runtime To Support Cuda Graph Launch #7616

[Runtime] Extend Graph Runtime To Support Cuda Graph Launch #7616

zhuochenKIDD commented Mar 9, 2021 •

edited

Loading

comaniac commented Mar 9, 2021

zhuochenKIDD commented Mar 11, 2021

comaniac Mar 11, 2021

zhuochenKIDD Mar 16, 2021

comaniac Mar 11, 2021

zhuochenKIDD Mar 16, 2021

comaniac Mar 16, 2021

comaniac left a comment

comaniac Mar 16, 2021

zhuochenKIDD Mar 17, 2021

comaniac Mar 16, 2021

zhuochenKIDD Mar 17, 2021

zhuochenKIDD commented Mar 17, 2021

comaniac left a comment

comaniac commented Mar 17, 2021

tqchen commented Mar 17, 2021

hogepodge commented Mar 18, 2021

[Runtime] Extend Graph Runtime To Support Cuda Graph Launch #7616

[Runtime] Extend Graph Runtime To Support Cuda Graph Launch #7616

Conversation

zhuochenKIDD commented Mar 9, 2021 • edited Loading

comaniac commented Mar 9, 2021

zhuochenKIDD commented Mar 11, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

comaniac left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhuochenKIDD commented Mar 17, 2021

comaniac left a comment

Choose a reason for hiding this comment

comaniac commented Mar 17, 2021

tqchen commented Mar 17, 2021

hogepodge commented Mar 18, 2021

zhuochenKIDD commented Mar 9, 2021 •

edited

Loading