[RELAY][COMPILER] Initial Relay interpreter and compiler for TVM runtime system. #1954

jroesch · 2018-10-22T20:05:23Z

This PR implements the evaluation machinery for Relay, including basic passes for lowering and transforming Relay programs to make them suitable for compilation.

Passes

FuseOps, is a phase which implements the machinery for operator fusion, as well as marking defintions which can be fused and lowered. A future PR will complete the fusion pass by actually partitioning the ops like the NNVM algorithm.
LowerOps, is a phase which collects the set of operations used by an expression and converts them into LoweredFunc using the appropriate TVM operator and schedule.

Interpreter

This PR ships a simple interpreter for Relay programs, which can be used as its reference semantics.
The interpretation of operator calls utilizes TVM compiled operators on CPU, but the evaluation of other expressions is naive.

TVM Graph Runtime.

This PR ships a compiler located in to_tvm.py which converts a Relay program into a TVM graph, as well as using the above passes to generate a tvm::Module enabling users to deploy a subset of Relay programs to the TVM RTS.

Testing

We ship some more tests for both the evaluator and the RTS, as well as a couple new cases for type inference.

This is blocked on #2026 and #2025.

python/tvm/relay/testing/init.py

MarisaKirisame · 2018-10-22T20:10:32Z

python/tvm/relay/testing/init.py

+    factor_type: str, optional
+        Can be ``'avg'``, ``'in'``, or ``'out'``.
+
+    magnitude: float, optional


Optional[float]

MarisaKirisame · 2018-10-22T20:13:38Z

src/relay/evaluator.cc

+  }
+
+  Value VisitExpr_(const OpNode* id) override {
+    // TODO(@jroesch): Eta-expand and return in this case.


fix this? it is not hard

MarisaKirisame · 2018-10-22T20:14:17Z

src/relay/evaluator.cc

+    Value v = Eval(op->cond);
+    if (const TensorValueNode* bv = v.as<TensorValueNode>()) {
+      // TODO(@jroesch): Ask TQ
+      if (reinterpret_cast<uint8_t*>(bv->data->data)[0]) {


there is something in dce that sort of do what we want. maybe reuse.

MarisaKirisame · 2018-10-22T20:14:42Z

src/relay/evaluator.cc

+        return Eval(op->false_branch);
+      }
+    } else {
+      throw EvalError("type error, type system should have caught this");


use check as check has stack trace

MarisaKirisame · 2018-10-22T20:15:36Z

src/relay/pass/monomorph.cc

+namespace tvm {
+namespace relay {
+
+using MMCacheKey = std::pair<Expr, Array<Type>>;


for compatibility

> > for compatibility

tqchen · 2018-10-24T20:59:05Z

@eqy please help review this

eqy · 2018-10-24T21:55:04Z

python/tvm/relay/testing/init.py

+        if "depthwise" in name:
+            factor = 3 * 3
+        scale = np.sqrt(self.magnitude / factor)
+        if self.rnd_type == "uniform":


Do we want to allow the user to pass in their own generator/initializer, or should they write a subclass of Initializer?

eqy · 2018-10-24T21:57:39Z

python/tvm/relay/to_tvm.py

+    op_attrs = attr.ib()
+    is_output = attr.ib(default=False)
+
+    def to_json(self):


Will these contain concrete shapes/params for operators, or will we need another way to extract them?
e.g., stride, padding, kernel dims, input dims for conv2d

This should put all the attributes in the operator just like the NNVM format does unless I misunderstood something.

slyubomirsky · 2018-10-24T22:38:33Z

src/relay/pass/monomorph.cc

+  }
+};
+
+struct MonoMorphizer : ExprMutator {


nit: Is the camel-case really suitable here? "Monomorphize" is one word

zhiics

I did a quick review. I will do another round when the PR is ready. Really nice work. One question, do we have a plan to use a pass manager? It looks that we are starting to add more passes.

zhiics · 2018-10-26T03:22:00Z

include/tvm/relay/base.h

+  CHECK(fdebug) << "Could not find Relay Python debugger function."; \
+  (*fdebug)("RELAY_DEBUG", __FILE__, __LINE__, __VA_ARGS__); \
+}
+
 /*!
 * \brief we always used NodeRef for referencing nodes.


zhiics · 2018-10-26T03:24:16Z

include/tvm/relay/interpreter.h

+ * \brief An interpreter for Relay.
+ *
+ * This file implements a simple reference interpreter for Relay programs.
+ * Given a Relay environment, an a Relay expression it produces a value.


s/an/and or delete "an"

zhiics · 2018-10-26T03:38:00Z

include/tvm/relay/pass.h

@@ -125,10 +142,12 @@ tvm::Array<TypeVar> FreeTypeVars(const Expr& expr);

 /*! \brief Remove expressions which does not effect the program result.
 *
- * It will remove let binding that are not referenced, and if branch that are not entered.
+ * It will remove let binding that are not referenced, and if branch that are


s/branch/branches/

zhiics · 2018-10-26T03:42:41Z

include/tvm/relay/pass.h

+
+// template<typename T>
+// struct hash<typename std::enable_if<std::is_base_of<tvm::relay::Expr, T>::value, T>::type> {
+//     size_t operator()(const T& expr) const {


remove the SFINAE if it is not used?

zhiics · 2018-10-26T03:44:00Z

include/tvm/relay/pass.h

+struct hash<tvm::Array<T>> {
+  size_t operator()(const tvm::Array<T>& array) const {
+    size_t hash = std::hash<std::string>()(array->_type_key);
+    for (auto elem : array) {


zhiics · 2018-10-26T04:31:24Z

include/tvm/tuple.h

+
+template <class Tuple, size_t Index = std::tuple_size<Tuple>::value - 1>
+struct TupleForEach {
+  static void apply(std::vector<NodeRef>& fields, Tuple const& tuple) {


I understand why we need &fields here, but I am not sure if we prefer passing by pointers or not.

zhiics · 2018-10-26T04:41:46Z

python/tvm/relay/op/_tensor.py

+import topi
+from . import register
+
+def add_compiler(attrs, inputs, output_type):


s/add_compiler/add_compute ?

Should we also have target for fcompute?

@zhiics I tried to keep the signatures mostly the same, @tqchen thoughts?

I think it is not a bad idea to also add target for compute

zhiics · 2018-10-26T05:18:11Z

python/tvm/relay/to_tvm.py

+
+@attr.s
+class OpNode(Node):
+    """An operator node in the graph representation we lower to before NNVM's graph."""


May I ask if this graph is more like the one generated after graphcompile? If so, it seems that it not before nnvm. Am I missing something?

Yeah sorry this is a typo, this is TVM's graph input.

zhiics · 2018-10-26T05:27:39Z

src/relay/interpreter.cc

+    return ClosureNode::make(captured_env, func);
+  }
+
+  Value invoke_operator(const Op& op, tvm::Array<Value>& args, const tvm::Array<Type>& type_args, const Attrs& attrs, Type ret_type) {


const tvm::Array& args ?

jroesch · 2018-10-27T09:58:24Z

@zhiics Tianqi has been pushing back on implementing a pass manager for the past couple weeks. He feels like we should wait until we have a few more passes and use cases before we go all in on designing the pass manager. I'm happy to build one sooner then later for various reasons which I'm happy to elaborate on.

tqchen · 2018-10-28T23:56:26Z

include/tvm/relay/expr.h

@@ -219,7 +222,8 @@ class FunctionNode : public ExprNode {
  TVM_DLL static Function make(tvm::Array<Var> params,
                               Expr body,
                               Type ret_type,
-                               tvm::Array<TypeVar> ty_params);
+                               tvm::Array<TypeVar> ty_params,
+                               tvm::Attrs attrs = DictAttrsNode::make({}));


Can we allow attrs to be None?

yeah sure, do you want it to default to None? right now the invariant is that it is always DictAttrs since it is most general.

I think it might not be a bad idea to make it default to None

tqchen · 2018-10-29T01:43:01Z

python/tvm/relay/to_tvm.py

@@ -0,0 +1,394 @@
+"""
+A compiler from a Relay expression to TVM's graph runtime.
+


graph_runtime_codegen

Add tuple.h Improve Evaluator Complete initial refactor of evaluator Repair evaluator tests Address some feedback and clean up uneeded code. Remove one piece of the hack Repair broken tests Add better module docstring Refactor to make Function attributes immutable This PR changes around how the passes work in order to accomadate immutable attributes Fix failing type_infer test Add fix up analyses in lower_ops.cc Forgot to add renamed to_tvm.py Remove experiment of adding name to Function Fix

python/tvm/relay/graph_runtime_codegen.py

MarisaKirisame · 2018-10-29T05:00:55Z

python/tvm/relay/graph_runtime_codegen.py

+                heads.append(NodeRef(i).to_json())
+
+        # Compute "node_row_ptr".
+        # TODO


The above comment, currently I'm using NNVM to fill the field in.

MarisaKirisame · 2018-10-29T05:02:23Z

tests/python/relay/test_type_infer.py

 def test_tuple():
-    tp = relay.TensorType((10,))


MarisaKirisame · 2018-10-29T05:02:40Z

tests/python/relay/test_evaluator.py

+
+# failing due to numeric issues
+
+# def test_dense():


restore or remove

There is a numeric precision issue I want feedback on. Can anyone tell me how to get dense to give results near numpy's mult.

Is the difference very large? We could just check for being within some epsilon of each other. I think it is important that we have tests that use important and complex ops

MarisaKirisame · 2018-10-29T05:04:12Z

src/relay/interpreter.cc

+
+struct Frame {
+  // In the efficient version this should seperate args, locals, and return
+// address.


src/relay/interpreter.cc

src/relay/ir/base.cc

src/relay/interpreter.cc

src/relay/pass/lower_ops.cc

jroesch · 2018-10-29T05:15:17Z

@eqy @zhiics @tqchen @MarisaKirisame @slyubomirsky @joshpoll this is ready for review if you could all gives some eyeballs, @junrushao1994 @ZihengJiang ?

junrushao · 2018-10-29T05:17:12Z

I am very interested in this PR. Will do tomorrow!

Co-Authored-By: jroesch <roeschinc@gmail.com>

tqchen · 2018-10-30T22:34:16Z

Thanks, @jroesch @junrushao1994 @joshpoll @zhiics @slyubomirsky @MarisaKirisame , this is now merged

…ystem. (apache#1954)

jroesch force-pushed the relay-eval branch from 91be553 to 5a8a593 Compare October 22, 2018 20:08

MarisaKirisame reviewed Oct 22, 2018

View reviewed changes

tqchen added the status: need review label Oct 22, 2018

tqchen self-assigned this Oct 22, 2018

jroesch mentioned this pull request Oct 23, 2018

[RELAY][TypeSystem] Add support for populating type args #1962

Merged

jroesch force-pushed the relay-eval branch 2 times, most recently from 52ddd94 to d39abb3 Compare October 23, 2018 07:59

eqy approved these changes Oct 24, 2018

View reviewed changes

slyubomirsky reviewed Oct 24, 2018

View reviewed changes

jroesch force-pushed the relay-eval branch 2 times, most recently from d5dec61 to 4d74ced Compare October 25, 2018 23:16

zhiics reviewed Oct 26, 2018

View reviewed changes

jroesch force-pushed the relay-eval branch from 19bac52 to 82bec21 Compare October 27, 2018 09:56

tqchen requested changes Oct 28, 2018

View reviewed changes

jroesch mentioned this pull request Oct 29, 2018

Add attrs package #2025

Merged

jroesch changed the title ~~[RELAY][WIP] Add Relay evaluator and compiler for TVM runtime system.~~ [RELAY][RUNTIME] Add Relay interpreter and compiler for TVM runtime system. Oct 29, 2018

tqchen requested changes Oct 29, 2018

View reviewed changes

jroesch force-pushed the relay-eval branch from b043252 to 26451d0 Compare October 29, 2018 05:01

jroesch added 3 commits October 28, 2018 22:06

Remove uneeded C++ test

4d3d9e0

Reformat file

e5f2d87

jroesch force-pushed the relay-eval branch from 64d0aea to e5f2d87 Compare October 29, 2018 05:07

MarisaKirisame reviewed Oct 29, 2018

View reviewed changes

Add target field for schedule

efe62d3

jroesch added 2 commits October 28, 2018 22:17

Type Compute and Schedule

eb130ea

Clean up deadcode and whitespace

21a52cf

jroesch and others added 18 commits October 30, 2018 11:07

Address a few more comments

98ae9b9

Add doc strings

b4d5d31

Update include/tvm/relay/pass.h

b69485f

Co-Authored-By: jroesch <roeschinc@gmail.com>

Update include/tvm/relay/pass.h

3f9dc5e

Co-Authored-By: jroesch <roeschinc@gmail.com>

Update include/tvm/relay/pass.h

4e3f92d

Co-Authored-By: jroesch <roeschinc@gmail.com>

Update include/tvm/relay/pass.h

604ea7b

Co-Authored-By: jroesch <roeschinc@gmail.com>

Add more docs

a61d663

Rename tests for graph runtime.

95e49a0

Remove old debugging

341a0c3

Update include/tvm/relay/interpreter.h

002f376

Co-Authored-By: jroesch <roeschinc@gmail.com>

Update python/tvm/relay/graph_runtime_codegen.py

c562275

Co-Authored-By: jroesch <roeschinc@gmail.com>

Update src/relay/pass/fuse_ops.cc

9a229c3

Co-Authored-By: jroesch <roeschinc@gmail.com>

A little more clean up

a050815

Fix CPP lint

28bfdba

Fix lint issue

ecfe6e5

Fix bad space

315f9c0

Fix testing error (hopefully)

764bc5e

Fix integer types issue

86ff607

tqchen approved these changes Oct 30, 2018

View reviewed changes

tqchen merged commit 10ea05e into apache:master Oct 30, 2018

tqchen changed the title ~~[RELAY][RUNTIME] Add Relay interpreter and compiler for TVM runtime system.~~ [RELAY][COMPILER] Initial Relay interpreter and compiler for TVM runtime system. Oct 30, 2018

tqchen added status: accepted and removed status: need review labels Oct 30, 2018

jroesch mentioned this pull request Nov 1, 2018

[RELAY][OP] Relay Op Sprint (Part 2) #2051

Closed

66 tasks

FrozenGene pushed a commit to FrozenGene/tvm that referenced this pull request Dec 27, 2018

[RELAY][RUNTIME] Add Relay interpreter and compiler for TVM runtime s…

be4b7c1

…ystem. (apache#1954)

ZihengJiang mentioned this pull request Feb 1, 2019

TVM 0.5 Release Note #2448

Closed

wweic pushed a commit to neo-ai/tvm that referenced this pull request Feb 20, 2019

[RELAY][RUNTIME] Add Relay interpreter and compiler for TVM runtime s…

43c2ea6

…ystem. (apache#1954)

wweic pushed a commit to neo-ai/tvm that referenced this pull request Feb 20, 2019

[RELAY][RUNTIME] Add Relay interpreter and compiler for TVM runtime s…

4531d41

…ystem. (apache#1954)

jroesch deleted the relay-eval branch February 4, 2021 04:39

		@@ -0,0 +1,394 @@
		"""
		A compiler from a Relay expression to TVM's graph runtime.

[RELAY][COMPILER] Initial Relay interpreter and compiler for TVM runtime system. #1954

[RELAY][COMPILER] Initial Relay interpreter and compiler for TVM runtime system. #1954

Conversation

jroesch commented Oct 22, 2018 • edited Loading

Passes

Interpreter

TVM Graph Runtime.

Testing

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tqchen commented Oct 24, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhiics left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jroesch commented Oct 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

slyubomirsky Oct 29, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jroesch commented Oct 29, 2018

junrushao commented Oct 29, 2018

tqchen commented Oct 30, 2018

jroesch commented Oct 22, 2018 •

edited

Loading

zhiics left a comment •

edited

Loading

slyubomirsky Oct 29, 2018 •

edited

Loading