Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[5/6] Arm(R) Ethos(TM)-U NPU codegen integration #8849

Merged
merged 1 commit into from
Sep 29, 2021

Conversation

manupak
Copy link
Contributor

@manupak manupak commented Aug 25, 2021

This commit integrates the codegen for Arm® Ethos™-U.

  • Adding Conv2D tests and a mobilenet_v1 conv2d offload test.

This PR is blocked on merging #8795 , #8806 and #8811. Since this is built on top of all those PR branches, this contains the accumulation of contents of the all the PRs.

Co-authored-by: Grant Watson grant.watson@arm.com
Co-authored-by: Leandro Nunes leandro.nunes@arm.com
Co-authored-by: Christopher Sidebottom chris.sidebottom@arm.com
Co-authored-by: Matthew Barrett matthew.barrett@arm.com

@manupak
Copy link
Contributor Author

manupak commented Sep 13, 2021

@Mousius @grant-arm , #8274 fails test_networks.py . I cant immediately figure out why. See if you guys see something.

cc: @mbaret

@manupak
Copy link
Contributor Author

manupak commented Sep 13, 2021

From the looks of it,

TVMBAW calls are not generated any more for static allocates for kDLCPU. I wonder why the AoT tests did not break.

*/
void SaveToFile(const std::string& file_name, const std::string& format) final {
std::string fmt = GetFileFormat(file_name, format);
LOG(INFO) << "format=" << fmt << ";;\n";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like some debug logging that was left in by mistake.
It should probably be removed.

@@ -113,16 +113,6 @@ class BuiltinLower : public StmtExprMutator {
op = stmt.as<AllocateNode>();
// Get constant allocation bound.
int64_t nbytes = GetVectorBytes(op->dtype);
if (device_type_.defined()) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @areusch @mbrookhart this is the code we discussed yesterday, currently CI passes with this code? that seems to contradict the conversations we had, but also I think these should be stack allocations for performance on cloud cpus.

Copy link
Contributor Author

@manupak manupak Sep 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, if we want the allocates to be placed on stack in CPU PrimFuncs, maybe we should make them have storage_scope = 'local' and we should generate TVMBAWs for 'global' allocates -- that could be made to work for both cases.

Copy link
Contributor

@mbs-octoml mbs-octoml Sep 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Filed #9022.
Just checking your use case: You want to see the allocas so you can later rewrite them, right? It's a bit confusing because I thought if the final c code generator sees these Allocates it rewrites them to use globals which I assumed was what you wanted.


ss << "#include <stdio.h>\n";
ss << "#include <stdlib.h>\n";
ss << "#include <dlpack/dlpack.h>\n";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dlpack.h is no longer required.

@manupak manupak marked this pull request as ready for review September 27, 2021 19:38
This commit integrates the codegen for Arm® Ethos™-U.
* Adding Conv2D tests and a mobilenet_v1 conv2d offload test.

Co-authored-by: Grant Watson <grant.watson@arm.com>
Co-authored-by: Leandro Nunes <leandro.nunes@arm.com>
Co-authored-by: Christopher Sidebottom <chris.sidebottom@arm.com>
Co-authored-by: Matthew Barret <matthew.barrett@arm.com>
module=module,
inputs=inputs,
outputs=outputs,
output_tolerance=10,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the default value of output_tolerance should be 0 and build_source should have an optional parameter to set it to something else, if necessary

Copy link
Contributor

@mbs-octoml mbs-octoml left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM modulo some nits and comment requests.

I didn't dig into the tests much, esp the reference_system stuff, will trust your local reviewers.

* specific language governing permissions and limitations
* under the License.
*/
#include <dmlc/filesystem.h>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: file level preamble comment

/*!
* \brief The ethos runtime module.
*
* \param cmms A array of external symbol 1, serialized command stream 1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: comment seems garbled, might as well explain them all here or on the fields.

namespace tvm {
namespace runtime {

class EthosUModuleNode : public ModuleNode {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overview comment, especially since this is doing the final host codegen.


private:
String c_source;
Array<String> func_names_;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth changing to a single main_func_name_ since a deep assumption throughout?



@tvm._ffi.register_func("relay.ext.ethosu.constant_updater")
def constant_updater(expr, symbol): # pylint: disable=unused-argument
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: move below main entry point and comment it is called back from utils.h UpdateConstants after lowering.

}

TVM_REGISTER_GLOBAL("runtime.module.ethosu.create").set_body([](TVMArgs args, TVMRetValue* rv) {
*rv = EthosUModuleNode::Create(args[0], args[1], args[2], args[3], args[4], args[5]);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: consider TypedPackedFunc



@tvm._ffi.register_func("relay.ext.ethosu")
def ethosu_compiler(ref):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update argument name/improve doc string here?

Copy link
Member

@jroesch jroesch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor nits, otherwise mostly Ethos-U specific and looks good to me

# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""Test infrastructure for Arm(R) Ethos(TM)-U NPU related tests"""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: grammar

@manupak
Copy link
Contributor Author

manupak commented Sep 29, 2021

Thanks all!
I'll address the minors in a follow up (as updated here : #8482).

manupak added a commit to manupak/incubator-tvm that referenced this pull request Sep 29, 2021
This is a follow up commit to address the
comments of apache#8849

Change-Id: I02d8de64f3bce0e7b544d652eee8737ec1ecbb80
@jroesch jroesch merged commit 86ce111 into apache:main Sep 29, 2021
@Mousius Mousius deleted the ethosu_upstreaming_p5 branch September 29, 2021 16:54
AndrewZhaoLuo added a commit to AndrewZhaoLuo/tvm that referenced this pull request Sep 29, 2021
* main:
  Fix flaky NMS test by making sure scores are unique (apache#9140)
  [Relay] Merge analysis/context_analysis.cc and transforms/device_annotation.cc (apache#9038)
  [LLVM] Make changes needed for opaque pointers (apache#9138)
  Arm(R) Ethos(TM)-U NPU codegen integration (apache#8849)
  [CI] Split Integration tests out of first phase of pipeline (apache#9128)
  [Meta Schedule][M3b] Runner (apache#9111)
  Fix Google Mock differences between Ubuntu 18.04 and 16.04 (apache#9141)
  [TIR] add loop partition hint pragma (apache#9121)
  fix things (apache#9146)
  [Meta Schedule][M3a] SearchStrategy (apache#9132)
  [Frontend][PyTorch] support for quantized conv_transpose2d op (apache#9133)
  [UnitTest] Parametrized test_conv2d_int8_intrinsics (apache#9143)
  [OpenCL] Remove redundant visit statement in CodeGen. (apache#9144)
  [BYOC] support arbitrary input dims for add/mul/relu of dnnl c_src codegen (apache#9127)
  [Relay][ConvertLayout] Support for qnn.conv2d_transpose (apache#9139)
  add nn.global_avgpool to fq2i (apache#9137)
  [UnitTests] Enable minimum testing on Vulkan target in CI (apache#9093)
  [Torch] Support returning quantized weights and bias for BYOC use cases (apache#9135)
  [Relay] Prepare for new plan_devices.cc (part II) (apache#9130)
  [microTVM][Zephyr] Add MIMXRT1050 board support (apache#9068)
manupak added a commit to manupak/incubator-tvm that referenced this pull request Sep 30, 2021
This is a follow up commit to address the
comments of apache#8849

Change-Id: I02d8de64f3bce0e7b544d652eee8737ec1ecbb80
manupak added a commit to manupak/incubator-tvm that referenced this pull request Sep 30, 2021
This is a follow up commit to address the
comments of apache#8849

Change-Id: I02d8de64f3bce0e7b544d652eee8737ec1ecbb80
manupak added a commit to manupak/incubator-tvm that referenced this pull request Sep 30, 2021
This is a follow up commit to address the
comments of apache#8849

Change-Id: I02d8de64f3bce0e7b544d652eee8737ec1ecbb80
AndrewZhaoLuo added a commit to AndrewZhaoLuo/tvm that referenced this pull request Sep 30, 2021
* main: (80 commits)
  Introduce centralised name transformation functions (apache#9088)
  [OpenCL] Add vectorization to cuda conv2d_nhwc schedule (apache#8636)
  [6/6] Arm(R) Ethos(TM)-U NPU codegen integration with `tvmc` (apache#8854)
  [microTVM] Add wrapper for creating project using a MLF (apache#9090)
  Fix typo (apache#9156)
  [Hotfix][Testing] Wait for RPCServer to be established (apache#9150)
  Update find cublas so it search default path if needed. (apache#9149)
  [TIR][LowerMatchBuffer] Fix lowering strides when source region has higher dimension than the buffer (apache#9145)
  Fix flaky NMS test by making sure scores are unique (apache#9140)
  [Relay] Merge analysis/context_analysis.cc and transforms/device_annotation.cc (apache#9038)
  [LLVM] Make changes needed for opaque pointers (apache#9138)
  Arm(R) Ethos(TM)-U NPU codegen integration (apache#8849)
  [CI] Split Integration tests out of first phase of pipeline (apache#9128)
  [Meta Schedule][M3b] Runner (apache#9111)
  Fix Google Mock differences between Ubuntu 18.04 and 16.04 (apache#9141)
  [TIR] add loop partition hint pragma (apache#9121)
  fix things (apache#9146)
  [Meta Schedule][M3a] SearchStrategy (apache#9132)
  [Frontend][PyTorch] support for quantized conv_transpose2d op (apache#9133)
  [UnitTest] Parametrized test_conv2d_int8_intrinsics (apache#9143)
  ...
manupak added a commit to manupak/incubator-tvm that referenced this pull request Oct 1, 2021
This is a follow up commit to address the
comments of apache#8849

Change-Id: I02d8de64f3bce0e7b544d652eee8737ec1ecbb80
jroesch pushed a commit that referenced this pull request Oct 1, 2021
This is a follow up commit to address the
comments of #8849

Change-Id: I02d8de64f3bce0e7b544d652eee8737ec1ecbb80
ylc pushed a commit to ylc/tvm that referenced this pull request Jan 7, 2022
This commit integrates the codegen for Arm® Ethos™-U.
* Adding Conv2D tests and a mobilenet_v1 conv2d offload test.

Co-authored-by: Grant Watson <grant.watson@arm.com>
Co-authored-by: Leandro Nunes <leandro.nunes@arm.com>
Co-authored-by: Christopher Sidebottom <chris.sidebottom@arm.com>
Co-authored-by: Matthew Barret <matthew.barrett@arm.com>

Co-authored-by: Grant Watson <grant.watson@arm.com>
Co-authored-by: Leandro Nunes <leandro.nunes@arm.com>
Co-authored-by: Christopher Sidebottom <chris.sidebottom@arm.com>
Co-authored-by: Matthew Barret <matthew.barrett@arm.com>
ylc pushed a commit to ylc/tvm that referenced this pull request Jan 7, 2022
This is a follow up commit to address the
comments of apache#8849

Change-Id: I02d8de64f3bce0e7b544d652eee8737ec1ecbb80
ylc pushed a commit to ylc/tvm that referenced this pull request Jan 13, 2022
This commit integrates the codegen for Arm® Ethos™-U.
* Adding Conv2D tests and a mobilenet_v1 conv2d offload test.

Co-authored-by: Grant Watson <grant.watson@arm.com>
Co-authored-by: Leandro Nunes <leandro.nunes@arm.com>
Co-authored-by: Christopher Sidebottom <chris.sidebottom@arm.com>
Co-authored-by: Matthew Barret <matthew.barrett@arm.com>

Co-authored-by: Grant Watson <grant.watson@arm.com>
Co-authored-by: Leandro Nunes <leandro.nunes@arm.com>
Co-authored-by: Christopher Sidebottom <chris.sidebottom@arm.com>
Co-authored-by: Matthew Barret <matthew.barrett@arm.com>
ylc pushed a commit to ylc/tvm that referenced this pull request Jan 13, 2022
This is a follow up commit to address the
comments of apache#8849

Change-Id: I02d8de64f3bce0e7b544d652eee8737ec1ecbb80
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants