[TVM EP] support of TVM Virtual Machine #10341

vvchernov · 2022-01-20T13:14:28Z

Support TVM Virtual Machine on TVM EP side

Waiting for merging of TVM PR (done) and ORT PR#10260 (done) and ORT PR#10505 (done) before check this PR

onnxruntime/python/providers/tvm/ort.py

onnxruntime/core/providers/tvm/tvm_execution_provider.cc

tmoreau89 · 2022-02-15T21:43:50Z

onnxruntime/core/providers/tvm/tvm_api.cc

+void TVM_VM_SetInputs(TvmModule& mod,
+ std::vector<DLTensor>& inputs)
+{
+ // TODO(vvchernov): set_input_zero_copy is more preferable but it does not satisfy alignment conditions.


Can you elaborate more on what these alignment conditions are?

I've observed that VM does not support zero copy on TVM side. I've supported zero copy on TVM EP side for GE after data alignment check.

tmoreau89 · 2022-02-15T21:44:59Z

onnxruntime/core/providers/tvm/tvm_api.h

 void TVMSetInputs(TvmModule& mod, std::vector<size_t>& inds, std::vector<DLTensor>& inputs);
+ void TVM_VM_SetInputs(TvmModule& mod, std::vector<DLTensor>& inputs);
+ void TVMGetOutputs(TvmModule& mod, std::vector<DLTensor>& outputs);
+ void TVMGet_VM_Outputs(TvmModule& mod, std::vector<DLTensor>& outputs);


We should name these functions either TVM_VM_SetInputs and TVM_VM_GetInputs or TVMSet_VM_Inputs and TVMGet_VM_Inputs for consistency.

Thanks! done

tmoreau89 · 2022-02-16T05:07:34Z

onnxruntime/python/providers/tvm/ort.py

+ # TODO(vvchernov): replace prints by logger, but investigate ORT logging system for python before
+ # see lines 69, 77, 119, 136
+ if executor == "vm":
+ # print("Build TVM virtual machine")


clean up this line

tmoreau89 · 2022-02-16T05:07:40Z

onnxruntime/python/providers/tvm/ort.py

+ target_host=target_host,
+ )
+ elif executor == "graph":
+ # print("Build TVM graph executor")


clean up this line

tmoreau89 · 2022-02-16T05:10:47Z

onnxruntime/core/providers/tvm/tvm_execution_provider.cc

- tvm::TVMSetInputs(*mod_, inds, dl_tensors_inputs);
+ if (use_vm_) {
+ tvm::TVM_VM_SetInputs(*mod_, dl_tensors_inputs);
+ // Infer once for calculating of output shapes


How will this affect benchmarking?

At first run we should do probe inference to define output tensor shapes. It needs for VM only. It does not affect our benchmarking because of warmup with several inferences is done before performance test.

vvchernov · 2022-02-22T06:41:10Z

Hello @xadupre! Could you review this PR again? I did not skip WIP status due to we are waiting for merging of PR on TVM side, just now ONNX runtime references on TVM on local branch, but further the TVM commit hash will be changed only. I do not think it confuses you.

onnxruntime/python/providers/tvm/ort.py

cgmanifests/cgmanifest.json

xadupre · 2022-02-24T09:17:04Z

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux Nuphar CI Pipeline, Linux OpenVINO CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline

azure-pipelines · 2022-02-24T09:17:53Z

Azure Pipelines successfully started running 9 pipeline(s).

…rosoft:onnxruntime)

…M EP

…ng related to target_host

vvchernov · 2022-03-01T12:33:43Z

Hello @xadupre, I think the PR is ready to merge. Could you recheck last changes and start CI tests.

xadupre · 2022-03-01T14:30:16Z

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux Nuphar CI Pipeline, Linux OpenVINO CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline

azure-pipelines · 2022-03-01T14:31:11Z

Azure Pipelines successfully started running 9 pipeline(s).

xadupre · 2022-03-01T14:36:27Z

/azp run Windows CPU CI Pipeline, Windows GPU CI Pipeline, Windows GPU TensorRT CI Pipeline, Windows WebAssembly CI Pipeline, orttraining-amd-gpu-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed, onnxruntime-python-checks-ci-pipeline

azure-pipelines · 2022-03-01T14:37:08Z

Azure Pipelines successfully started running 8 pipeline(s).

vvchernov · 2022-03-01T14:56:37Z

@xadupre I've fixed flake8 issues, please restart CI pipelines

xadupre · 2022-03-01T16:02:36Z

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux Nuphar CI Pipeline, Linux OpenVINO CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline

xadupre · 2022-03-01T16:03:20Z

/azp run Windows CPU CI Pipeline, Windows GPU CI Pipeline, Windows GPU TensorRT CI Pipeline, Windows WebAssembly CI Pipeline, orttraining-amd-gpu-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed, onnxruntime-python-checks-ci-pipeline

azure-pipelines · 2022-03-01T16:03:30Z

Azure Pipelines successfully started running 9 pipeline(s).

azure-pipelines · 2022-03-01T16:04:03Z

Azure Pipelines successfully started running 8 pipeline(s).

vvchernov · 2022-03-01T17:47:40Z

Good evening @xadupre Could you restart orttraining-amd-gpu-ci-pipeline, there is failure due to samples-per-second mismatching for huggingface-gpt2 models. It does not related to the PR in any case.

xadupre · 2022-03-01T17:50:38Z

/azp run orttraining-amd-gpu-ci-pipeline

azure-pipelines · 2022-03-01T17:50:50Z

Azure Pipelines successfully started running 1 pipeline(s).

vvchernov · 2022-03-02T10:00:10Z

Hello @xadupre! All checks have passed successfully, it can be approved.

* add executor option (vm or graph) and support virtual machine methods * nullptr check for compile and run methods (see also PR#10211 from microsoft:onnxruntime) * get output shapes for VM * remove run_with_benchmark. remove run methods from python api, get it from native side * get outputs method for VM was implemented * support multiple input for VM * update python logging and exception * small fix * update tvm with patch for VM API * update nhwc transformations for TVM EP * add data alignment check and support set_input_zero_copy for GE in TVM EP * fix logger name * return back to apache/tvm with VM fixes instead of local dev branch * hide customized tvm logger while issue is not resolved. fix tvm warning related to target_host * flake8 fix Co-authored-by: Valery Chernov <valery.chernov@deelvin.com> (cherry picked from commit 62cc981)

vvchernov force-pushed the vc/tvm-vm branch 2 times, most recently from 07724d5 to 1ba8fd5 Compare February 4, 2022 06:28

vvchernov force-pushed the vc/tvm-vm branch from 1ba8fd5 to 9cd4b5d Compare February 9, 2022 14:35

This was referenced Feb 9, 2022

[TVM EP] Rename Standalone TVM (STVM) Execution Provider to TVM EP #10260

Merged

[TVM EP] upstream Apache/TVM after internal fixes #10515

Merged

vvchernov force-pushed the vc/tvm-vm branch from 9cd4b5d to f74e9b4 Compare February 15, 2022 17:30

vvchernov changed the title ~~WIP: [TVM EP] support of TVM Virtual Machine~~ [TVM EP] support of TVM Virtual Machine Feb 15, 2022

xadupre reviewed Feb 15, 2022

View reviewed changes

onnxruntime/python/providers/tvm/ort.py Outdated Show resolved Hide resolved

xadupre reviewed Feb 15, 2022

View reviewed changes

onnxruntime/core/providers/tvm/tvm_execution_provider.cc Show resolved Hide resolved

tmoreau89 reviewed Feb 15, 2022

View reviewed changes

tmoreau89 reviewed Feb 16, 2022

View reviewed changes

vvchernov changed the title ~~[TVM EP] support of TVM Virtual Machine~~ WIP: [TVM EP] support of TVM Virtual Machine Feb 16, 2022

vvchernov force-pushed the vc/tvm-vm branch 5 times, most recently from 35e520d to 97da135 Compare February 18, 2022 11:25

xadupre reviewed Feb 22, 2022

View reviewed changes

onnxruntime/python/providers/tvm/ort.py Show resolved Hide resolved

xadupre reviewed Feb 22, 2022

View reviewed changes

onnxruntime/python/providers/tvm/ort.py Outdated Show resolved Hide resolved

xadupre reviewed Feb 22, 2022

View reviewed changes

cgmanifests/cgmanifest.json Outdated Show resolved Hide resolved

vvchernov mentioned this pull request Feb 24, 2022

[TVM EP] code refactor #10655

Merged

xadupre previously approved these changes Feb 24, 2022

View reviewed changes

vvchernov added 2 commits February 25, 2022 10:27

add executor option (vm or graph) and support virtual machine methods

d2e755c

nullptr check for compile and run methods (see also PR#10211 from mic…

7c8b91a

…rosoft:onnxruntime)

vvchernov added 2 commits February 25, 2022 10:27

add data alignment check and support set_input_zero_copy for GE in TV…

8766ff3

…M EP

fix logger name

fa3c259

vvchernov dismissed xadupre’s stale review via fa3c259 February 25, 2022 07:29

vvchernov force-pushed the vc/tvm-vm branch from 0e69cf0 to fa3c259 Compare February 25, 2022 07:29

return back to apache/tvm with VM fixes instead of local dev branch

765d740

vvchernov changed the title ~~WIP: [TVM EP] support of TVM Virtual Machine~~ [TVM EP] support of TVM Virtual Machine Feb 26, 2022

vvchernov changed the title ~~[TVM EP] support of TVM Virtual Machine~~ WIP: [TVM EP] support of TVM Virtual Machine Feb 26, 2022

hide customized tvm logger while issue is not resolved. fix tvm warni…

1a9a7fc

…ng related to target_host

vvchernov changed the title ~~WIP: [TVM EP] support of TVM Virtual Machine~~ [TVM EP] support of TVM Virtual Machine Mar 1, 2022

xadupre previously approved these changes Mar 1, 2022

View reviewed changes

flake8 fix

b459617

vvchernov dismissed xadupre’s stale review via b459617 March 1, 2022 14:55

xadupre approved these changes Mar 2, 2022

View reviewed changes

xadupre merged commit 62cc981 into microsoft:master Mar 2, 2022

vvchernov deleted the vc/tvm-vm branch March 5, 2022 11:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TVM EP] support of TVM Virtual Machine #10341

[TVM EP] support of TVM Virtual Machine #10341

vvchernov commented Jan 20, 2022 •

edited

Loading

tmoreau89 Feb 15, 2022

vvchernov Feb 22, 2022

tmoreau89 Feb 15, 2022

vvchernov Feb 18, 2022

tmoreau89 Feb 16, 2022

vvchernov Feb 18, 2022

tmoreau89 Feb 16, 2022

vvchernov Feb 18, 2022

tmoreau89 Feb 16, 2022

vvchernov Feb 18, 2022

vvchernov commented Feb 22, 2022 •

edited

Loading

xadupre commented Feb 24, 2022

azure-pipelines bot commented Feb 24, 2022

vvchernov commented Mar 1, 2022

xadupre commented Mar 1, 2022

azure-pipelines bot commented Mar 1, 2022

xadupre commented Mar 1, 2022

azure-pipelines bot commented Mar 1, 2022

vvchernov commented Mar 1, 2022

xadupre commented Mar 1, 2022

xadupre commented Mar 1, 2022

azure-pipelines bot commented Mar 1, 2022

azure-pipelines bot commented Mar 1, 2022

vvchernov commented Mar 1, 2022

xadupre commented Mar 1, 2022

azure-pipelines bot commented Mar 1, 2022

vvchernov commented Mar 2, 2022

[TVM EP] support of TVM Virtual Machine #10341

[TVM EP] support of TVM Virtual Machine #10341

Conversation

vvchernov commented Jan 20, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vvchernov commented Feb 22, 2022 • edited Loading

xadupre commented Feb 24, 2022

azure-pipelines bot commented Feb 24, 2022

vvchernov commented Mar 1, 2022

xadupre commented Mar 1, 2022

azure-pipelines bot commented Mar 1, 2022

xadupre commented Mar 1, 2022

azure-pipelines bot commented Mar 1, 2022

vvchernov commented Mar 1, 2022

xadupre commented Mar 1, 2022

xadupre commented Mar 1, 2022

azure-pipelines bot commented Mar 1, 2022

azure-pipelines bot commented Mar 1, 2022

vvchernov commented Mar 1, 2022

xadupre commented Mar 1, 2022

azure-pipelines bot commented Mar 1, 2022

vvchernov commented Mar 2, 2022

vvchernov commented Jan 20, 2022 •

edited

Loading

vvchernov commented Feb 22, 2022 •

edited

Loading