[DOCS] Add docs for Pass Instrument #8220

chiwwang · 2021-06-09T08:03:03Z

Hi,
This is a follow-up PR on #7952
A How-To-Use-Pass-Instrument tutorial is added.
Pass Instrument sections are also added to pass_infra.rst.

[DOCS] Add docs for Pass Instrument

Add a tutorial about how to use pass instrument.
Add related sections in Pass Infrastructure documents.

@zackcquic @tqchen @areusch @tqchen @tkonolige @altanh

Thanks for contributing to TVM! Please refer to guideline https://tvm.apache.org/docs/contribute/ for useful information and tips. After the pull request is submitted, please request code reviews from Reviewers by @ them in the pull request thread.

Initiate a Path object from TEST_DATA_ROOT_PATH to fix the error: AttributeError: 'str' object has no attribute 'mkdir'

- Add a tutorial about how to use pass instrument. - Add related sections in Pass Infrastructure documents.

zackcquic

Hi @chiwwang:

Thanks a lot :)

Something should be added:

What happens when exceptions occur in different instrument point.
Standard Instrument section: PassTimingInstrument, PrintBefore(TODO), PrintAfter(TODO), ..
Global PassContext and override_instrument examples
use_pass_infra.py's comments should be updated, sorry, I forgot to update it.
conf.py should be updated.

docs/api/python/ir.rst

zackcquic · 2021-06-09T09:13:30Z

docs/dev/pass_infra.rst

+``PassInstrument`` provides callbacks run by these methods. Multiple
+``PassInstrument`` instances can be registed into a single ``PassContext``.
+They are called sequentially in the order of ``instruments`` member.
+


Add exception situations.

Done. But it's a little long section. Please help review. Thanks:)

I think it maybe better to move python frontend ahead, use it to introduce what is a pass instrument and instrument points, then goes to the discussion in PassContext::Instrument*

It's hard to introduce python frontend first because pass_infra.txt put a big section called "Python Frontend" downward.
All python things are put there. If I don't follow this, pass_instra.txt become fragmented.
(Although I agreed it is easier to introduce call-sequences and concepts with Python code...)

docs/dev/pass_infra.rst

zackcquic · 2021-06-09T09:19:05Z

tutorials/dev/use_pass_instrument.py

+As more and more passes are implemented, it becomes interesting to instrument
+passes execution, analyze per-pass effects and observe various events.
+We have extended :py:class:`tvm.transform.PassContext` to accept a list of
+instrument classes. Also a decorator :py:func:`tvm.ir.instrument.pass_instrument` is provided to easily implement instrument classes.


Pass instrument framework extended ... instrument instances, Also a decorator :py:func:tvm.instrument.pass_instrument ... instrument instances.

I think pass_instrument is used to help implement a "class", not a instance.
And argument "instruments" accept a list of instances. Do I understand correctly?

yes, you are right.

So it should be,
Pass instrument framework extended ... instrument instances, Also a decorator :py:func:tvm.instrument.pass_instrument ... instrument classes.
Sorry for miss-copied the second one.

I don't quite get what "..." should be in "Pass instrument framework extended ... instrument instances".
But wordings around here is changed. Could you take a look again?

zackcquic · 2021-06-09T09:20:42Z

tutorials/dev/use_pass_instrument.py

+instrument classes. Also a decorator :py:func:`tvm.ir.instrument.pass_instrument` is provided to easily implement instrument classes.
+
+This tutorial demostrates how developers can use ``PassContext`` to instrument
+passes. For more details, please refer to the :ref:`pass-infra`


How about remove "For more details, ... " ?

Seems no change.

I changed it to "please also refer to ...."
Maybe you were referring to old commits?

tkonolige

Thanks for this hard work @chiwwang! Documentation is a really important part of this project. I've left some feedback.

tkonolige · 2021-06-09T16:49:10Z

docs/dev/pass_infra.rst

@@ -109,7 +109,8 @@ configure the compilation options, including optimization level and
 required/disabled passes, etc. For instance, we may have a configuration which
 performs all passes at ``opt_level=3`` with some disabled passes using
 ``disabled_pass=xx`` provided by ``PassContext``. Now we could glob all passes
-at ``opt_level=3`` and exclude those in the disabled pass list.
+at ``opt_level=3`` and exclude those in the disabled pass list. ``PassContext``
+also provides pass-instruments mechanism, which will be introduced latter.


Suggested change

also provides pass-instruments mechanism, which will be introduced latter.

also provides a way to instrument all passes. See section [fill in the section here].

tkonolige · 2021-06-09T16:50:41Z

docs/dev/pass_infra.rst

@@ -123,16 +124,23 @@ Python APIs to create a compilation pipeline using pass context.

    class PassContextNode : public Object {
     public:
-      ErrorReporter err_reporter;


What happened to the error reporter?

It seems to be replaced by diag_ctx in #6274.
It is not related to this PR. I fixed it just because it's handy.
Though there aren't lots of descriptions about error reporters in pass_infra.txt, I think it should present due to this sentence:

PassContext carries useful information for an optimization pass. For example, it contains the error reporting system so optimization authors can provide diagnostic...

tkonolige · 2021-06-09T16:52:12Z

docs/dev/pass_infra.rst

+    TVM_DLL bool InstrumentBeforePass(const IRModule& mod, const PassInfo& info) const;
+    TVM_DLL void InstrumentAfterPass(const IRModule& mod, const PassInfo& info) const;
+
+The first two methods are called respectively in entering/exiting context scope. The latter two are called while passes is being applied(`src/ir/transform.cc`_).


I think pseudo code of how these functions are called would be useful here.

Done. But you might want to take a look. I am not quite confident of what syntax should be used for pseudo code.

if (pass_ctx.InstrumentBeforePass(ir_module, pass_info)) { new_ir_module = run_pass(ir_module, pass_ctx); pass_ctx.InstrumentAfterPass(new_ir_module, pass_info); return new_ir_module; }

tkonolige · 2021-06-09T16:53:16Z

docs/dev/pass_infra.rst

+Note that ``InstrumentBeforePass()`` return a boolean indicating this pass should
+be run or not.


Suggested change

Note that ``InstrumentBeforePass()`` return a boolean indicating this pass should

be run or not.

Note that ``InstrumentBeforePass()`` returns a boolean indicating whether or not the pass should

be run.

tkonolige · 2021-06-09T16:54:00Z

docs/dev/pass_infra.rst

+
+``PassInstrument`` provides callbacks run by these methods. Multiple
+``PassInstrument`` instances can be registed into a single ``PassContext``.
+They are called sequentially in the order of ``instruments`` member.


Is this the same as the order in which they are registered?

Yes. I tried to change wordings to "in the order of instruments argument passed to PassContext"

tkonolige · 2021-06-09T16:57:52Z

docs/dev/pass_infra.rst

+
+- ``exit_pass_ctx``
+
+  * This callback is run at the moement of exiting ``PassContext``.


Suggested change

* This callback is run at the moement of exiting ``PassContext``.

* This callback is run when exiting the ``PassContext``.

tkonolige · 2021-06-09T16:58:16Z

docs/dev/pass_infra.rst

+
+- ``should_run``
+
+  * This callback is run before a pass is executed, returning a boolean indicating if the pass should be run.


Suggested change

* This callback is run before a pass is executed, returning a boolean indicating if the pass should be run.

* This callback is run before a pass is executed. It returns a boolean indicating if the pass should be run.

tkonolige · 2021-06-09T16:59:24Z

python/tvm/ir/instrument.py


    Examples
    --------
-    The following code block decorates a pass instrument class.
+    The following code block show how to decorate a pass instrument class.


Suggested change

The following code block show how to decorate a pass instrument class.

The following code block shows how to decorate a pass instrument class.

tkonolige · 2021-06-09T17:00:00Z

tutorials/dev/use_pass_instrument.py

+==============================
+**Author**: `Chi-Wei Wang <https://github.com/chiwwang>`_
+
+As more and more passes are implemented, it becomes interesting to instrument


Suggested change

As more and more passes are implemented, it becomes interesting to instrument

As more and more passes are implemented, it becomes useful to instrument

tkonolige · 2021-06-09T17:17:23Z

tutorials/dev/use_pass_instrument.py

+# We create a Relay program from a Pytorch model.
+# Here we pick up ``mobilenet_v2`` from torchvision.
+import torch
+import torchvision
+
+model_name = "mobilenet_v2"
+model = getattr(torchvision.models, model_name)(pretrained=True)
+model = model.eval()
+
+input_shape = [1, 3, 224, 224]
+input_data = torch.randn(input_shape)
+scripted_model = torch.jit.trace(model, input_data).eval()
+
+shape_list = [("input0", input_shape)]
+relay_mod, relay_params = relay.frontend.from_pytorch(scripted_model, shape_list)


Importing a pytorch model seems to unnecessarily complicate this tutorial. Why not use relay.testing.resnet?

Thanks for pointing out this API. It is not used just because I am not aware of it.
Change the example model to resnet-18.

chiwwang · 2021-06-11T14:22:37Z

Thanks for prompt feedbacks @zackcquic @tkonolige !

Here are some comments for Zack's questions:

What happens when exceptions occur in different instrument point.
Added in pass_infra.txt. But it is a little long. You might want to take a look again.
Standard Instrument section: PassTimingInstrument, PrintBefore(TODO), PrintAfter(TODO), ..
I think it might be better to maintain these in the doc string of related Python class/function.
So I add example to instrument.py. It will be shown in Python API reference.
Global PassContext and override_instrument examples
Done. Sorry for not aware of this approach.
use_pass_infra.py's comments should be updated, sorry, I forgot to update it.
Done.
conf.py should be updated.
Done. But actually it seems to automatically append un-listed tutorials to the end.
How do you think about the current order of tutorial?

zackcquic · 2021-06-11T14:33:44Z

docs/dev/pass_infra.rst

+
+Python interfaces are provided to implement ``PassInstrument`` quickly.
+
+Following four methods are invoked in the life-cycle of ``PassContext``.


Currently we introduces four instrument point in the life-cycle of PassContext. (introduce in python frontend maybe better)
Include the text graph in

tvm/include/tvm/ir/instrument.h

Line 52 in d69011d

* with PassContext(instruments=[pi]): # pi = a PassInstrument implementation

maybe helpful to understand quickly (move this to python frontend, too?)

Added text-graph.

zackcquic · 2021-06-11T14:34:34Z

docs/dev/pass_infra.rst

+``PassInstrument`` provides callbacks run by these methods. Multiple
+``PassInstrument`` instances can be registed into a single ``PassContext``.
+They are called sequentially in the order of ``instruments`` member.
+


I think it maybe better to move python frontend ahead, use it to introduce what is a pass instrument and instrument points, then goes to the discussion in PassContext::Instrument*

zackcquic · 2021-06-11T14:37:27Z

docs/dev/pass_infra.rst

+    TVM_DLL void InstrumentAfterPass(const IRModule& mod, const PassInfo& info) const;
+
+``InstrumentEnterPassContext`` is called immediately when the scope
+of the ``PassContext`` instance is entered.


when entering the scope of PassContext instance.

It's hard to introduce python frontend first because pass_infra.txt put a big section called "Python Frontend" downward.
All python things are put there. If I don't follow this, pass_instra.txt become fragmented.
(Although I agreed it is easier to introduce call-sequences and concepts with Python code...)

zackcquic · 2021-06-11T14:38:10Z

docs/dev/pass_infra.rst

+``InstrumentEnterPassContext`` is called immediately when the scope
+of the ``PassContext`` instance is entered.
+
+``InstrumentExitPassContext`` is called when the scope of ``PassContextNode``


when leaving the scope of PassContext instance

zackcquic · 2021-06-11T14:40:17Z

docs/dev/pass_infra.rst

+
+- ``InstrumentBeforePass``
+
+  * ``ShouldRun`` callbakc is executed if the pass is not listed as a required pass.


i think you could just delete the word callback here. ShouldRun is an interface method.

zackcquic · 2021-06-11T14:40:59Z

docs/dev/pass_infra.rst

+- ``InstrumentBeforePass``
+
+  * ``ShouldRun`` callbakc is executed if the pass is not listed as a required pass.
+    If the pass is a required pass, ``ShouldRun`` will not be executed for that pass.


Seems this sentence is same with above. Maybe able to remove.

zackcquic · 2021-06-11T14:45:38Z

docs/dev/pass_infra.rst

+
+``InstrumentExitPassContext`` is called when the scope of ``PassContextNode``
+is being leaved, or exceptions occur during the execution of passes.
+This method is also called when instruments is being overriden by ``override_instruments`` in ::py:class:`tvm.transform.PassContext`.


I think override_instruments should be placed in another little section to describe along with the concept of global PassContext

Done.
Because override_instruments appears as a Python function, I put it after Python Frontend introduction in pass_infra.txt.

zackcquic · 2021-06-11T15:00:20Z

docs/dev/pass_infra.rst

+Pass Instrument
+~~~~~~~~~~~~~~~
+
+``PassInstrument`` provides callbacks run when entering/exiting ``PassContext`` and before/after executing passes.


provide callbacks to run

Since PassInstrument is an interface, it might be more helpful to speak about the interface as a whole rather than mention callbacks, which are just one way of implementing the interface. e.g.

The PassInstrument interface allows you to run code when entering/exiting PassContext and before ...

zackcquic · 2021-06-11T15:09:54Z

tutorials/dev/use_pass_instrument.py

+As more and more passes are implemented, it becomes interesting to instrument
+passes execution, analyze per-pass effects and observe various events.
+We have extended :py:class:`tvm.transform.PassContext` to accept a list of
+instrument classes. Also a decorator :py:func:`tvm.ir.instrument.pass_instrument` is provided to easily implement instrument classes.


yes, you are right.

So it should be,
Pass instrument framework extended ... instrument instances, Also a decorator :py:func:tvm.instrument.pass_instrument ... instrument classes.
Sorry for miss-copied the second one.

zackcquic · 2021-06-11T15:10:32Z

tutorials/dev/use_pass_instrument.py

+instrument classes. Also a decorator :py:func:`tvm.ir.instrument.pass_instrument` is provided to easily implement instrument classes.
+
+This tutorial demostrates how developers can use ``PassContext`` to instrument
+passes. For more details, please refer to the :ref:`pass-infra`


Seems no change.

zackcquic · 2021-06-11T15:15:27Z

Thanks for prompt feedbacks @zackcquic @tkonolige !

Here are some comments for Zack's questions:

What happens when exceptions occur in different instrument point.
Added in pass_infra.txt. But it is a little long. You might want to take a look again.

Standard Instrument section: PassTimingInstrument, PrintBefore(TODO), PrintAfter(TODO), ..
I think it might be better to maintain these in the doc string of related Python class/function.
So I add example to instrument.py. It will be shown in Python API reference.

Global PassContext and override_instrument examples
Done. Sorry for not aware of this approach.

use_pass_infra.py's comments should be updated, sorry, I forgot to update it.
Done.

conf.py should be updated.
Done. But actually it seems to automatically append un-listed tutorials to the end.
How do you think about the current order of tutorial?

Thanks a lot for the hard work, its very helpful. @chiwwang

For 2, I think it should be mentioned in this doc, like vm optimizations
If maybe a little hard to find (eg grep the whole source without knowing the name).

For 3, I think it needs a little section:
Since passes can be used without explicit PassContext, so we introduced override...
// ... more description

For 5. I would like to group pass related together, it seems done in the latest fix.

areusch

thanks for contributing @chiwwang ! added some comments inline.

areusch · 2021-06-11T15:49:01Z

docs/dev/pass_infra.rst

+Pass Instrument
+~~~~~~~~~~~~~~~
+
+``PassInstrument`` provides callbacks run when entering/exiting ``PassContext`` and before/after executing passes.


Since PassInstrument is an interface, it might be more helpful to speak about the interface as a whole rather than mention callbacks, which are just one way of implementing the interface. e.g.

The PassInstrument interface allows you to run code when entering/exiting PassContext and before ...

areusch · 2021-06-11T15:49:37Z

docs/dev/pass_infra.rst

+
+``PassInstrument`` provides callbacks run when entering/exiting ``PassContext`` and before/after executing passes.
+Multiple ``PassInstrument`` instances can be registed into a single ``PassContext``.
+Instrument instances are called sequentially in the order of ``instruments`` argument passed to ``PassContext``.


Don't abbreviate class names, it will get very confusing: s/Instrument/PassInstrument/

areusch · 2021-06-11T15:50:17Z

docs/dev/pass_infra.rst

+
+    }  // namespace instrument
+
+Python interfaces are provided to implement ``PassInstrument`` quickly.


Perhaps it would be good to include a pointer to these, since a pointer to the C++ ones are above.

Yes, I add a new tag for python frontend here. Thanks.

areusch · 2021-06-11T15:50:54Z

docs/dev/pass_infra.rst

+This method is also called when instruments is being overriden by ``override_instruments`` in ::py:class:`tvm.transform.PassContext`.
+
+``InstrumentBeforePass`` is called before pass-execution.
+``InstrumentAfterPass`` is called after pass-executioon if the pass should be run. The behavir is like:


nit: execution, behavior

areusch · 2021-06-11T15:52:57Z

docs/dev/pass_infra.rst

+
+- ``InstrumentBeforePass``
+
+  * ``ShouldRun`` callbakc is executed if the pass is not listed as a required pass.


i think you could just delete the word callback here. ShouldRun is an interface method.

areusch · 2021-06-11T15:53:49Z

docs/dev/pass_infra.rst

@@ -389,6 +396,103 @@ To allow other C++ modules to apply this pass, we declare a free function in

    TVM_DLL Pass FoldConstant();

+.. _pass_instrument_section_tag:
+
+Pass Instrument


it seems like PassContext may be introduced after this section, no? Perhaps it makes sense to move it downward.

I fixed wordings.
As for the order of introduction, It's a little embarrassing. I would like to put all Pass Instrument descriptions together and after PassContext introduction.
But the pass_infra.txt is divided to C++ backend and Python Frontend parts.
I tried to follow that. So the order of introduction becomes:

PassContext C++ backend

PassInstrument C++ backend

PassContext Python frontend

PassInstrument Python frontend

The PassContext Python frontend is after PassInstrument C++ backend.

ah i see. what do you think about documenting the PassInstrument first, since that's the developer-facing thing? TVM calls InstrumentEnterPassContext.

i think explaining the C++ thing first and then the Python thing is workable. i agree it's a bit tricky to work around the split. I don't think the Python docs need to be as extensive since they mostly wrap the C++ stuff

It would be nice! But after trying and reading around, I find PassInstrument might require understanding PassContext first to know about what each instrument points means.
Perhaps we can split PassInstrument doc out once it becomes bigger (such as more and more built-in instruments are added, or new functionality added to provide more instrumentation possibility?)

I also tried to shrink Python docs a little. Thanks a lot for suggestions @areusch :)

areusch · 2021-06-11T15:57:39Z

tutorials/dev/use_pass_instrument.py

+relay_mod["main"] = bind_params_by_name(relay_mod["main"], relay_params)
+# timing_inst is put after call_node_inst.
+# So the execution time of ``call_node.inst.run_after_pass()`` is also counted.
+with tvm.transform.PassContext(opt_level=3, instruments=[call_node_inst, timing_inst]):


it would be super awesome to add another section demonstrating an exception case here!

Added. Thanks for suggestions!

chiwwang · 2021-06-14T14:23:01Z

Ah! I saw what confused me. It's the level of "Pass Registration" section.
The current pass_infra.rst section-hierachy is

Pass Infrastructure (Topmost)

The Design
The design of backend and frontend are described here.
- C++ Backend
  - PassContext
  - Pass Constructs
  - Module-Level Passes
  - Function-Level Passes
  - Sequential Passes
- Pass Registration <-----This section has the same level with Backend/Frontend.
- Python Frontend
  - PassContext
  - Pass Objects

Now I add Pass Instrument as:

Pass Infrastructure (Topmost)

The Design
The design of backend and frontend are described here.
- C++ Backend
  - PassContext
  - Pass Constructs
  - Module-Level Passes
  - Function-Level Passes
  - Sequential Passes
  - Pass Registration <----- May I fix this to have the same level with other sub-sections in C++ backend?
  - Pass Instruments <--- Added in this PR.
  - Built-in Instrument <--- Added in this PR.
- Python Frontend
  - PassContext
  - Pass Objects
  - Pass Instrument <--- Added in this PR.
  - Override Instruments in Current PassContext <--- Added in this PR.

This might looks matching with descriptions in "The Design" section.
Or, could we isolate Pass Instrument, and have another topmost section as Pass Infrastructure?
May I know your thoughts @zackcquic @areusch ?
Thanks a lot!

zackcquic · 2021-06-15T01:51:56Z

docs/dev/pass_infra.rst

+        ):
+        # ...
+
+One can implement a ``PassInstrument`` by using the ``pass_instrument`` decorator(`python/tvm/ir/instrument.py`_) on a class implementing following methods:


Nit: Maybe it should be emphasized to use decorator, instead of overriding/subclassing.

zackcquic · 2021-06-15T01:57:40Z

docs/dev/pass_infra.rst

+Pass Instrument
+^^^^^^^^^^^^^^^
+
+Currently we introduce four instrument point in the life-cycle of ``PassContext``.


instrument points

zackcquic · 2021-06-15T02:02:11Z

docs/dev/pass_infra.rst

+        ):
+        # ...
+
+One can implement a ``PassInstrument`` by using the ``pass_instrument`` decorator(`python/tvm/ir/instrument.py`_) on a class implementing following methods:


Nit: I think it maybe better to emphasize use decorator instead of subclassing.

zackcquic · 2021-06-15T02:12:44Z

Ah! I saw what confused me. It's the level of "Pass Registration" section.
The current pass_infra.rst section-hierachy is

Pass Infrastructure (Topmost)

The Design
The design of backend and frontend are described here.

C++ Backend

PassContext

Pass Constructs

Module-Level Passes

Function-Level Passes

Sequential Passes

Pass Registration <-----This section has the same level with Backend/Frontend.

Python Frontend

PassContext

Pass Objects

Now I add Pass Instrument as:

Pass Infrastructure (Topmost)

The Design
The design of backend and frontend are described here.

C++ Backend

PassContext

Pass Constructs

Module-Level Passes

Function-Level Passes

Sequential Passes

Pass Registration <----- May I fix this to have the same level with other sub-sections in C++ backend?

Pass Instruments <--- Added in this PR.

Built-in Instrument <--- Added in this PR.

Python Frontend

PassContext

Pass Objects

Pass Instrument <--- Added in this PR.

Override Instruments in Current PassContext <--- Added in this PR.

This might looks matching with descriptions in "The Design" section.
Or, could we isolate Pass Instrument, and have another topmost section as Pass Infrastructure?
May I know your thoughts @zackcquic @areusch ?
Thanks a lot!

Just know this PR put all in Pass Infrastructure
I previous assume this a separate section like Pass Infrastructure with the sequence:

Introduce what is a pass instrument
1. instrument points
PassContext and multiple instance details
Examples

But this PR's organization is ok for me.
Wait for @areusch @tkonolige to see if they have more comments.

chiwwang · 2021-06-24T06:55:47Z

Gentle ping to see whether we have any comment on this change. Thanks!
@areusch @tkonolige @ZihengJiang

areusch

thanks for the ping and sorry for the delay @chiwwang ! just a few more comments.

areusch · 2021-06-25T20:11:52Z

docs/dev/pass_infra.rst

+Pass Instrument
+^^^^^^^^^^^^^^^
+
+Currently we introduce four instrument points in the life-cycle of ``PassContext``.


maybe could you add a short sentence explaining what a "Pass Instrument" is? e.g. what does it mean to "instrument" the pass manager?

Done. Thanks!

areusch · 2021-06-25T20:29:45Z

docs/dev/pass_infra.rst

+
+There are several built-in instruments. Those marked with *TODO* are not implemented yet.
+
+PassTimmingInstrument (see `src/ir/instrument.cc`_)


nit: Timing

Done. Thanks!

areusch · 2021-06-25T21:19:50Z

docs/dev/pass_infra.rst

@@ -389,6 +396,103 @@ To allow other C++ modules to apply this pass, we declare a free function in

    TVM_DLL Pass FoldConstant();

+.. _pass_instrument_section_tag:
+
+Pass Instrument


ah i see. what do you think about documenting the PassInstrument first, since that's the developer-facing thing? TVM calls InstrumentEnterPassContext.

i think explaining the C++ thing first and then the Python thing is workable. i agree it's a bit tricky to work around the split. I don't think the Python docs need to be as extensive since they mostly wrap the C++ stuff

areusch · 2021-06-25T21:21:05Z

docs/dev/pass_infra.rst

+
+PassTimmingInstrument (see `src/ir/instrument.cc`_)
+
+PrintBefore(TODO)


could you give a short explanation what these are?

areusch · 2021-06-25T21:22:49Z

docs/dev/pass_infra.rst

+See :ref:`pass_instrument_overriden`.
+
+``InstrumentBeforePass`` is called before execution.
+``InstrumentAfterPass`` is called after executioon if the pass should be run. The behavior is like:


nit: execution

Done. Thanks!

areusch · 2021-06-28T15:49:54Z

thanks @chiwwang , please address the CI failure:

/workspace/docs/tutorials/dev/use_pass_infra.rst:307: WARNING: undefined label: pass_instrument_section_tag

tkonolige · 2021-06-28T20:33:10Z

python/tvm/ir/instrument.py

+        Examples
+        --------
+
+        The following code-block shows how to use this instrument.


You can just delete this line.

tkonolige · 2021-06-28T20:33:19Z

python/tvm/ir/instrument.py


    Examples
    --------
-    The following code block decorates a pass instrument class.
+    The following code block shows how to decorate a pass instrument class.


You can delete this line.

tkonolige · 2021-06-28T20:35:07Z

tutorials/dev/use_pass_instrument.py

+**Author**: `Chi-Wei Wang <https://github.com/chiwwang>`_
+
+As more and more passes are implemented, it becomes useful to instrument
+passes execution, analyze per-pass effects and observe various events.


Suggested change

passes execution, analyze per-pass effects and observe various events.

pass execution, analyze per-pass effects, and observe various events.

I think we generally use the Oxford comma?

tkonolige · 2021-06-28T20:46:36Z

tutorials/dev/use_pass_instrument.py

+Pass infrastructure provides instrument mechanism. One can pass a list of
+instrument instances to :py:class:`tvm.transform.PassContext`.
+Also a decorator :py:func:`tvm.instrument.pass_instrument` is provided
+to easily implement instrument classes.


Suggested change

Pass infrastructure provides instrument mechanism. One can pass a list of

instrument instances to :py:class:`tvm.transform.PassContext`.

Also a decorator :py:func:`tvm.instrument.pass_instrument` is provided

to easily implement instrument classes.

We can instrument passes by providing a list of :py:class:`tvm.ir.instrument.PassInstrument` instances to :py:class:`tvm.transform.PassContext`. We provide a pass instrument for collecting timing information (:py:class:`tvm.ir.instrument.PassTimingInstrument`), but an extension mechanism is available via the :py:func:`tvm.instrument.pass_instrument` decorator.

tkonolige · 2021-06-28T20:47:21Z

tutorials/dev/use_pass_instrument.py

+image_shape = (3, 224, 224)
+output_shape = (batch_size, num_of_image_class)
+relay_mod, relay_params = resnet.get_workload(num_layers=18, batch_size=1, image_shape=image_shape)
+print(relay_mod.astext(show_meta_data=False))


Can you add a print statement before this that mentions what we are printing.

tkonolige · 2021-06-28T21:04:09Z

tutorials/dev/use_pass_instrument.py

+
+###############################################################################
+# If an exception occur in ``enter_pass_ctx``, ``PassContext`` disable the pass
+# instrumentation. And it will run ``exit_pass_ctx`` of each ``PassInstrument``


Suggested change

# instrumentation. And it will run ``exit_pass_ctx`` of each ``PassInstrument``

# instrumentation. And it will run the ``exit_pass_ctx`` of each ``PassInstrument``

tkonolige · 2021-06-28T21:05:46Z

tutorials/dev/use_pass_instrument.py

+# Also, all ``PassInstrument`` are cleared.
+# So nothing printed while ``override_instruments`` is called.


Suggested change

# Also, all ``PassInstrument`` are cleared.

# So nothing printed while ``override_instruments`` is called.

```suggestion

# Exceptions in ``PassInstrument``s cause all instruments of the current ``PassContext`` to be cleared, so nothing is printed when ``override_instruments`` is called.

Done. But the 's' in "PassInstruments" seems to cause rendering problems. (Sphinx 3.5.4)
I changed it to PassInstrument instances.

tkonolige · 2021-06-28T21:06:17Z

tutorials/dev/use_pass_instrument.py

+demo_ctx.override_instruments([])  # no PassFine_0 exit_pass_ctx printed....etc
+
+###############################################################################
+# If an exception occur in ``exit_pass_ctx``, pass instrumentation is disabled.


Suggested change

# If an exception occur in ``exit_pass_ctx``, pass instrumentation is disabled.

# If an exception occurs in ``exit_pass_ctx``, then the pass instrument is disabled.

tkonolige · 2021-06-28T21:07:17Z

tutorials/dev/use_pass_instrument.py

+
+###############################################################################
+# If an exception occur in ``exit_pass_ctx``, pass instrumentation is disabled.
+# Then exception is thrown. That means ``PassInstrument`` registered


Suggested change

# Then exception is thrown. That means ``PassInstrument`` registered

# This exception is propagated. ``PassInstrument``s registered

tkonolige · 2021-06-28T21:07:42Z

tutorials/dev/use_pass_instrument.py

+
+###############################################################################
+# Exceptions occured in ``should_run``, ``run_before_pass``, ``run_after_pass``
+# are not handled explicitly -- that means, we rely on the context manager


Suggested change

# are not handled explicitly -- that means, we rely on the context manager

# are not handled explicitly -- we rely on the context manager

chiwwang · 2021-06-29T02:21:49Z

@areusch Oops sorry I forgot to change the tag name. Fixed and checked with task_sphinx_precheck.sh. Hope it works this time.

@tkonolige Thank you for so detailed review!
And sorry for my some careless grammar mistakes :(

areusch · 2021-06-29T15:32:16Z

@tkonolige @zackcquic please take a look and explicitly approve if you're good with this PR

tkonolige

@chiwwang Everyone makes grammar mistakes, thats why we have code review :)

zackcquic

LGTM

chiwwang · 2021-07-05T01:21:45Z

Thanks @zackcquic @tkonolige !!
@areusch you might want to check if this PR is ready for merging. Thanks :)

areusch · 2021-07-07T18:03:31Z

thanks @chiwwang, the PR is now merged!

chiwwang · 2021-07-08T02:59:44Z

Thanks you all!! I learn so much from all of feedbacks :)

* Fix AttributeError when TEST_DATA_ROOT_PATH is set Initiate a Path object from TEST_DATA_ROOT_PATH to fix the error: AttributeError: 'str' object has no attribute 'mkdir' * [DOCS] Add docs for Pass Instrument - Add a tutorial about how to use pass instrument. - Add related sections in Pass Infrastructure documents. * Fix ir.rst, the length of separator. * Fix unused local name * Fix linting errors * Fix linting errors * Fix linting errors * Address code-review feedbacks * Fix linting * Fix the order of tutorial. * Add exception handling. Address feedbacks. * Fix CI error -- clearing instruments in global pass_ctx * Clarify section hierachy. * Emphasize to use decorator instead of subclassing * Add a sentence to explain Pass Instrument. Fix typo. * Shrink python docs a little. * Fix tag name. * Address feedbacks.

chiwwang and others added 9 commits May 14, 2021 22:34

Fix AttributeError when TEST_DATA_ROOT_PATH is set

7c9d5dd

Initiate a Path object from TEST_DATA_ROOT_PATH to fix the error: AttributeError: 'str' object has no attribute 'mkdir'

Merge branch 'apache:main' into main

c0bc83b

Merge branch 'apache:main' into main

af35621

[DOCS] Add docs for Pass Instrument

43d40c4

- Add a tutorial about how to use pass instrument. - Add related sections in Pass Infrastructure documents.

Fix ir.rst, the length of separator.

42df16e

Fix unused local name

0485c45

Fix linting errors

7d81b58

Fix linting errors

4f8b8e4

Fix linting errors

3c666b5

zackcquic suggested changes Jun 9, 2021

View reviewed changes

tqchen added the status: need review label Jun 9, 2021

tqchen assigned areusch and ZihengJiang Jun 9, 2021

tkonolige requested changes Jun 9, 2021

View reviewed changes

chiwwang added 2 commits June 11, 2021 21:48

Address code-review feedbacks

18317ce

Fix linting

66596a9

Fix the order of tutorial.

98d4fbf

zackcquic suggested changes Jun 11, 2021

View reviewed changes

areusch reviewed Jun 11, 2021

View reviewed changes

chiwwang added 3 commits June 13, 2021 17:42

Add exception handling. Address feedbacks.

229991b

Fix CI error -- clearing instruments in global pass_ctx

80aaa37

Clarify section hierachy.

b6083aa

zackcquic reviewed Jun 15, 2021

View reviewed changes

Emphasize to use decorator instead of subclassing

2c201c1

areusch requested changes Jun 25, 2021

View reviewed changes

chiwwang added 2 commits June 28, 2021 10:58

Add a sentence to explain Pass Instrument. Fix typo.

b0a270f

Shrink python docs a little.

b5cffef

tkonolige requested changes Jun 28, 2021

View reviewed changes

chiwwang added 2 commits June 29, 2021 09:23

Fix tag name.

fd32622

Address feedbacks.

078db70

tkonolige approved these changes Jun 29, 2021

View reviewed changes

zackcquic approved these changes Jun 29, 2021

View reviewed changes

areusch approved these changes Jul 7, 2021

View reviewed changes

areusch merged commit a4775c2 into apache:main Jul 7, 2021

	also provides pass-instruments mechanism, which will be introduced latter.
	also provides a way to instrument all passes. See section [fill in the section here].

		Note that ``InstrumentBeforePass()`` return a boolean indicating this pass should
		be run or not.


		- ``exit_pass_ctx``

		* This callback is run at the moement of exiting ``PassContext``.

	* This callback is run at the moement of exiting ``PassContext``.
	* This callback is run when exiting the ``PassContext``.


		- ``should_run``

		* This callback is run before a pass is executed, returning a boolean indicating if the pass should be run.

	The following code block show how to decorate a pass instrument class.
	The following code block shows how to decorate a pass instrument class.

	As more and more passes are implemented, it becomes interesting to instrument
	As more and more passes are implemented, it becomes useful to instrument


		Python interfaces are provided to implement ``PassInstrument`` quickly.

		Following four methods are invoked in the life-cycle of ``PassContext``.


		- ``InstrumentBeforePass``

		* ``ShouldRun`` callbakc is executed if the pass is not listed as a required pass.


		} // namespace instrument

		Python interfaces are provided to implement ``PassInstrument`` quickly.


		There are several built-in instruments. Those marked with TODO are not implemented yet.

		PassTimmingInstrument (see `src/ir/instrument.cc`_)


		PassTimmingInstrument (see `src/ir/instrument.cc`_)

		PrintBefore(TODO)

	passes execution, analyze per-pass effects and observe various events.
	pass execution, analyze per-pass effects, and observe various events.

	# instrumentation. And it will run ``exit_pass_ctx`` of each ``PassInstrument``
	# instrumentation. And it will run the ``exit_pass_ctx`` of each ``PassInstrument``

		# Also, all ``PassInstrument`` are cleared.
		# So nothing printed while ``override_instruments`` is called.

	# If an exception occur in ``exit_pass_ctx``, pass instrumentation is disabled.
	# If an exception occurs in ``exit_pass_ctx``, then the pass instrument is disabled.

	# Then exception is thrown. That means ``PassInstrument`` registered
	# This exception is propagated. ``PassInstrument``s registered

	# are not handled explicitly -- that means, we rely on the context manager
	# are not handled explicitly -- we rely on the context manager

[DOCS] Add docs for Pass Instrument #8220

[DOCS] Add docs for Pass Instrument #8220

Conversation

chiwwang commented Jun 9, 2021 • edited Loading

zackcquic left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tkonolige left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chiwwang Jun 11, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chiwwang commented Jun 11, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chiwwang Jun 13, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zackcquic commented Jun 11, 2021 • edited Loading

areusch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chiwwang Jun 28, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chiwwang commented Jun 9, 2021 •

edited

Loading

zackcquic left a comment •

edited

Loading

chiwwang Jun 11, 2021 •

edited

Loading

chiwwang commented Jun 11, 2021 •

edited

Loading

chiwwang Jun 13, 2021 •

edited

Loading

zackcquic commented Jun 11, 2021 •

edited

Loading

chiwwang Jun 28, 2021 •

edited

Loading

chiwwang commented Jun 14, 2021 •

edited

Loading