…g` (apache#9817)

…#10232)

This file is included every time TVM is build, regardless of whether
any support for Hexagon is enabled or not. This refactoring is meant
to remove underlying assumptions about what features are enabled and
what the compilation targets are. Now, when there is nothing needed
from Hexagon, the script exits early (although it doesn't need to),
and the rest of it is (and should remain) safe to execute regardless
of build configuration.

Disable "runtime.module.loadfile_hexagon" from the offload runtime,
since it conflicts with device_api.hexagon.v2.
It was only used with offload on Android, which is being deprecated.

* [Torch] Fix conv2d transpose with group

* lint

* wrong issue number

* do not run test on cuda

* upd

* upd

* upd

* lint

* fix

* upd docstring

* upd

…ailed. (apache#10244)

There is an error message to show the body when 'MatchCopyPattern' is failed,
but the error message not give the information why this function get failed.
Adding the detail error information to help trouble shooting.

lint.sh complain for an addtional space line in 'utils.cc', just fix it.

* wip

* revert for now

* simplify blocking

* add bench script

* update type rel

* refactor tests

* end to end compilation working

* paralleize outer loop

* add shape check

* fused schedule first cut

* restore original test

* black

* add vnni check

* add relay test

* skip on ci

* check dtype

* lint

* make it tunable

* minor cleanup

* hack to ExpandDataflow

* add test from mei

* Update DeviceAnalyzer to inherit from MixedModeVisitor

* indent

* 0;276;0cinitial commit

* register a bunch of ops

* unary ops

* add a bunch of tests

* 0;276;0crefactor tests

* add tests to qnn

* comments on macros

* add back in log to pattern utils

* update floating point func description

* proper creating of calls to quantize and dequantize

* fix lowering process for using dequantize and quantize ops

* Fix spacing

* Add try..finally everywhere

* trigger ci

* Fix pytest invocations

* Remove junit collection where no files exist

Co-authored-by: driazati <driazati@users.noreply.github.com>

* Fix bug

* Fix whitespace

* lint

* Move the other consts out of PoolInfo

…e#10259)

* [Hexagon] Don't use cmake glob for auto-generated source files

Glob treats inputs as patterns: if the file with a given name
does not exist (is to be generated later), it won't be added to
the output.

* Restart CI

* [RPC] Link in whole archive with BUILD_STATIC_RUNTIME

* Restart CI

Co-authored-by: pfk-beta <this_email_isnot_working@gmail.com>

Co-authored-by: pfk-beta <this_email_isnot_working@gmail.com>

* upd

* simplify

* upd

* fix

* upd

* fix docstring

See apache#10275

cc @masahi

Co-authored-by: driazati <driazati@users.noreply.github.com>

Required to pass `test_cuda_bf16_vectorize_add` in `tests/python/unittest/test_target_codegen_cuda.py`.

…ache#10256)

* Add an end_to_end benchmarking argument to TVMC run.

* Add command line test.

* Fix comment syntax.

* Set device to cpu if end_to_end is on.

* Tickle CI

This adds `$TVM_LIBRARY_PATH` to the search directory of
`vta.libinfo.find_libvta`, matching the behavior of
`tvm._ffi.libinfo.find_lib_path`.

* [microNPU] Add support for LeakyReLU

Adds support for offloading an int8 Leaky ReLU activation function
to the NPU by legalizing to a LUT.

Change-Id: I63dd5b16a1a2a747b11f15a5b8124810e2ebf491

* refactor LeakyReLUParams to inherit from LutActivationParams

Change-Id: I35b59200b16a7eff1915f771ab6b5d9181d4f3ab

…e#10277)

* Use /usr/bin/env bash in shebang

This makes scripts executable on system without /bin/bash (NixOS)

* Use `set -e` in script instead of `bash -e` in shebang

* Add a conversion of individual operations in FQ2I pass.

* apply review comments

* apply review comments 2

* removed include header path, which are invalid

* 1. added target tvm in a cmake export group
2. added cmake package config file

* added Threads as public dependency

* changed temp config file name for better understanding

* Hexagon docker files added

* trigger

See apache#10297

cc @mehrdadh

Co-authored-by: driazati <driazati@users.noreply.github.com>

* [TIR] Add software pipelining

Co-authored-by: Junru Shao <junrushao1994@gmail.com>
Co-authored-by: Xiyou Zhou <xiyou@octoml.ai>
Co-authored-by: Bohan Hou <32121147+spectrometerHBH@users.noreply.github.com>
Co-authored-by: Siyuan Feng <Hzfengsy@sjtu.edu.cn>
Co-authored-by: Hongyi Jin <3231950289@qq.com>
Co-authored-by: Ruihang Lai <lairuihangdongdong@qq.com>

* fix

* fix

* lint

* fix

* format

* doc

* remove print

* lint

* lint

* doc

* Apply suggestions from code review

Co-authored-by: Junru Shao <junrushao1994@gmail.com>

* address comments

* address comments

* refactor FragmentInfo::GetSize

* remove unused

* refactor

* address comments

Co-authored-by: Junru Shao <junrushao1994@gmail.com>
Co-authored-by: Xiyou Zhou <xiyou@octoml.ai>
Co-authored-by: Bohan Hou <32121147+spectrometerHBH@users.noreply.github.com>
Co-authored-by: Siyuan Feng <Hzfengsy@sjtu.edu.cn>
Co-authored-by: Hongyi Jin <3231950289@qq.com>
Co-authored-by: Ruihang Lai <lairuihangdongdong@qq.com>

* initial commit

* lint

* Add -i option to fix ASF headers to lint scripts.

* address driazati comments

…xagon (apache#10289)

…gon (apache#10290)

See apache#10300

Co-authored-by: driazati <driazati@users.noreply.github.com>

These shouldn't take longer than 5 minutes but since they have to poll they can end up running for a long while (e.g. this failure: https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/2534/pipeline).

cc @mehrdadh

Co-authored-by: driazati <driazati@users.noreply.github.com>

This adds a template so we can report (and label) flaky test issues separately from CI infra problems. This also helps others report flaky tests by pointing them to the relevant documentation.

cc @areusch @denise-k @hpanda-naut @masahi

Co-authored-by: driazati <driazati@users.noreply.github.com>

…e#10229)

* [UnitTest] Disable ptx mma tests on unsupported nvcc versions.

- Modified `tvm.contrib.nvcc.get_cuda_version` to return a
  `(major,minor,release)` tuple rather than a float.

- Implemented `tvm.testing.requries_nvcc_version` decorator to specify
  the minimum `(major,minor,release)` version needed to run a unit
  test.

- Applied decorated to unit tests in `test_tir_ptx_mma.py` that fail
  on earlier nvcc versions.

* Fix lint errors.

* Updated a few of the cuda version checks.

* More lint fixes.

* Only compare major/minor in find_libdevice, not release version.

* [Hexagon] Fix getting/setting DMA state

The bits [3:0] of the first word of the descriptor (both 16- and 32-byte)
is the DMA state. It must be set to 0 before starting a DMA transaction.

* Restart CI

* [TOPI] Add support for groupped conv3d

Change conv3d to use generic conv implementation which supports groupped
convolutions. Also, remove support for non-float16 tensorcore operations
as they cause large degradation in accuracy. Generic conv now supports
autoscheduler.

* correct none check

* add tests for floordiv simplification

* fixed incorrect test for autoscheduler

* formatting

* add groups to winograd

* fix tensorcore

* manually simplify index instead of relying on simplifier

* formatting

* add groups argument to conv3d_ncdhw_winograd_without_weight_transform

* formatting

…ual_device_ field (apache#9848)

* VStore function result virtual devices in virtual_device_ field

* Address Mark's 'mega nit'

* Promote function result virtual device to first class

* Add kVirtualDevice

* move kVirtualDevice

* Fix annotation test

* Progress on parsing & printing

* Fix printing of virtual device attribute

* flake

See apache#10312

cc @masahi

Co-authored-by: driazati <driazati@users.noreply.github.com>

See apache#10314

cc @masahi

Co-authored-by: driazati <driazati@users.noreply.github.com>

* [TE][Fix] Comparison of the output tensor

* fix hybrid op issue

* fix tensor replacement in schedule ops

* fix compute inline

…pache#9996)

* add ceildiv and shapediv

* add boundary checking in layout_transform

* support multi-blocking and shape padding

* refine the log for shape transform

* add test for multi-blocking layout transform

* delete unwanted comments

* remove workaround

* fix lint errors

)

* Deal with BufferLoad

* Test for BufferLoad

* Fix comments and CI

This updates the docker image build to use Python 3.7 and Ubuntu 18 as discussed in apache#9703. The update is mostly straightforward except that the `apt` boost isn't built with 3.7 so we must now build it from source.

This adds a script that runs on a cron to discover the last commit where CI all passed (every job was successful and `tvm-ci/branch` is included) and updates a git tag `green` to point to this commit on `main`. This can be used for checking out the latest unbroken TVM, which can be useful for developers wanting a good changeset to base their changes on or for infra needing a clean, up-to-date TVM.

This commit enables USMP in the microNPU codegen
and tests. The microNPU codegen is modified to
support Let nodes that are produced as from USMP.

It seems the sentence is from relay.repeat() and not related to
relay.reverse().

We found this while converting an RNN model.

The relay tflite frontend use squeeze at converting unpack, but when the
unpack.axis=0, `None` is passed to relay.squeeze(), which would squeeze
all dimensions with length 1, causing different results from TFLite.

A possible fix might be, assign the unpack.axis as-is to relay.squeeze()

As for stridedslice, when the tflite frontend handles shrink_axis_mask,
the wrapped `begin` should be used, instead of the original one which
can be negative. It can cause errors at
https://github.com/apache/tvm/blob/d65ff6594d4d6db0062537a1d43c0504173b8e5c/include/tvm/topi/detail/strided_slice.h#L140

Related cases are also added to the python test.

…passes (apache#10172)

[TIR] Fix Ramp int32~64 mismatch in VectorizeLoop and NarrowDataType passes

[BYOC-DNNL] add support for more ops and fusion patterns

* RelayViz interface and terminal ast-dump.

This PR follows apache#8668, with splitting
out interfaces class and terminal ast-dump implementation.

This visualizer is aimed for quick look-then-fix, so the interface is
simple. Despite that, customization is still possbile through
implementing interfaces defined in `interface.py` or overriding existent
implementations inside a renderer module, like `terminal.py`.

A tutorial is also provided in this PR.

A graphviz renderer will also be contributed after this PR.

* lint and typo

)

Due to some restructuring of the Ethos(TM)-N driver library it is no
longer necessary to link the compiler library (AKA Support library)
into the runtime.

…he#10330)

* Add new Metadata classes and base implementation.

 * These were autogenerated in the original PR, but checking them in
   as plain code until we can revisit the auto-generator approach.

* address masa comments

* Add documentation per Manupa's comments, and move kMetadataVersion namespace.

* remove get_name function, used for debugging

* clang-format

* [ONNX] only broadcast matmul if the shape has changed

* fix copy-pasta mistake

* [TIR] Introduce tir.allocate_const to TIR

This PR is adding non-scalar constant representation in TIR. This is used to
express constants (i.e., parameters) in the TIR instead of bypassing the
TIR as it's done until now.

Change-Id: Id3afc4d7197260cb43ecde60f05ccbce3fc42430

Co-authored-by: Giuseppe Rossini <giuseppe.rossini@arm.com>
Change-Id: Id4a09a637c9c1fd7d49989c6c10f474a78569e18

* [TIR] Integrate tir constant nodes in compilation pipeline

This PR integrates tir.allocate_const to the compilation pipeline to support --link-params.

Change-Id: Ic8d0cb75d596299fcae7078b304598afbf0c5494

Co-authored-by: Giuseppe Rossini <giuseppe.rossini@arm.com>
Change-Id: Id98cc682bbfacfe75c4d8b260fd41658f1f196b2

* [TIR] tir.const extraction

This commit tries to implement an amendment to tir.constant RFC
with centralized storage of constant data within the IRModule
Please note that data and irmod_storage_idx are not mutual exclisive
further more the irmod_storage_idx is valid only immediatly after
prim func addition to the mod or after update within the mod.
If prim func is out of the the module scope then the index become
meangless. irmod_storage_idx also is not used in calculation of hash
function of the tir.constant node.

Change-Id: I40742ed580468b0252ea3fec02184cba65e20871

* unit test fixed

Change-Id: Ied2186554d4cbad44b2346216c8be92449e55732

* cmsis-nn codegen fix

Now handled case when params of the functions came as constants

Change-Id: I5874e182e34ef94e23048eaf3c61b01a56d91131

* Fixes for unittests

Change-Id: I5b82ee3f80337155706b5470973f494a301b5d90

* Rebasing tests fixes

Change-Id: I94ac87907081bab53c1dd1ab2db106ae057b4b19

* Linter: added method param description

Change-Id: I2f8c4c8d244b74c794abaa6079c46cc593ffcbdb

* Printing removal fix

This patch removes forgotten print in fuse_ops

Change-Id: I4bb5934f3b4cd5fde19d36a8e3319aae136bce8a

* Bugfix

Fixed concurrent map update bug here

Change-Id: Ifec3bf5030086d9079b9e493096f17dfd82297ec

* Reworked logic for not to introduce empty constant list to modue attrs

Change-Id: I082c85b3b4b70c218f0d714f5613ef6e178bd020

* Added support for tir builtin::tvm_access_ptr

This fixed unit tests for tests/python/integration/test_arm_mprofile_dsp.py

Change-Id: I10919f301ef9ddc3fd87f0e1a8414e9a52fc7938

* Unit test fix

Fixes unit tests in torch frontend

Change-Id: I6c179834f93dd202605d1ce5a7f07d987b9dc469

* Addressed requested changes

Addressed changes requested upstream

Change-Id: I741e52b89eb285732c23b1ac7ff277e757a088c3

* Namespace usage changed to conform earlier C++ standard

Change-Id: I1b29238cfe2a6bedb525f4f823a3a540f631d836

* Bugfix

Change-Id: I57a44b714b307278a243817ec2864e53ad31366b

* updated IRModuleNode::ExtractPrimFuncConstants

Updated IRModuleNode::ExtractPrimFuncConstants as per
request upstream.

Change-Id: I35db0145fb5827efd0445ce665d0c99465274016

* Minor changes

typo fixd
renamed ExtractPrimFuncConstants to ExtractConstants
removed getters/setters from FuseMutator and added parametrized
constructor

Change-Id: Ib2326805781779b88c963a8642ff683c8755956e

* Moved LinkedParam/LinkedParamNode

Moved LinkedParam/LinkedParamNode from tvm::tir namespace to tvm
namespace

Change-Id: Ie3f0303bd4f7890c6d680268c91f2051977bc7f4

* Addressed upstream comments

Changed BindParams argument to Array<NDArray>
Removed 'name' argument from te.const
Switched to in-depth comparision of NDArrays in constant de-duplication
Removed extra final comma from NDArrayToTIR
Changed return type of ConstantAllocationSize to int64_t
Made link_param a tvm.testing.parameter for test_fuse_take and test_fuse_gather_nd

Change-Id: I4285099cc63756aa5ebe91a5bd207d4135499b41

* Removed unnecessary forward declaration

+linter

Change-Id: I2a6c0d1f97773aeb1ae3f458da252a22079ccdb1

* Constant extractor now is a separate pass

Change-Id: Ia4adca9d3315b26fbdc006ef7c115900c081e303

* Added forgotten file + unit test fix

Change-Id: Ice305f4fefd13fe95e97574e6d63ffeb664621df

* Changed to IRModule pass

Refactored ExtractPrimFuncConstants to IRModule pass.
deDup -> DeDup
Refactored logic of Applicator supplementary class

Change-Id: I6c120d175eb6790ba90f176c4f856bde8f0c7c94

* bugfix after rebasing

Change-Id: Ie3ee6ea2479476a30f486baef74f20070f117942

* -v -> -vv to have more debug information

Change-Id: I12c63731663b9c9ea574b9ed5cb17311ba3cf701

Co-authored-by: Giuseppe Rossini <giuseppe.rossini@arm.com>

…est (apache#10342)

* Simple workaround for PyTorch symbol crash problem in meta schedule test

* workaround for CI

…se (apache#10304)

* [ARM_CPU] Conv2d int8 intrinsic for cortex-A72

Add an intrinsic that performs a dot product of 8 4-element vectors at
once. Also conditionally inline fused operators into the main
convolution loop depending on convolutions size. Small convolution = no
inlining. Performance improves by ~20% on mobilenet on raspberry pi 4
and ~30% improvement on performance for the individual convolutions.

* ignore incorrect lints

* fixup fstring

* revert changes to conv2d_NCHWc (not int8)

* remove error check, apparently tests rely on it

* refactor alter op layout

* Add hexagon tests to CI Hexagon

* Fix CRT libs

* cleanup and fix Jenkins

* Address @areusch comments

…pache#10343)

Looks like we did need the redirect in (apache#10247), otherwise you get a
blank redirect response and `tar` doesn't like that very much:

```
tar: This does not look like a tar archive

gzip: stdin: unexpected end of file
```

…che#10354)

* [CI] Fix Flaky Test `test_task_scheduler_gradient`

A change to fix the issue of flaky test mentioned in apache#10356 by increase the `chain_rule` factor and avoid small gradient.

* Retrigger CI.

* add test

* compute added

* schedule works

* reuse dense_vnni schedule

* try an alternative approach to scheduling layout transform

* introduce a tunable knob to decide if compute_root

* check transpose condition

* support s8 + s8 input

* pylint

* tir schedule misc update

* Trigger Build

Following an investigation from apache#10022,
it turns out, currently the workspace
calculation assumes there would be a single
lowered PrimFunc could be produced per
primitive Relay Function.

However, the exception turned out to
be the CMSIS-NN codegen that produces
multiple calls/PrimFuncs in the place
of a single call to single relay PrimFunc.

This commit adds changes to workspace
calculation to be done on lowered IRModule.

Additionally, changes the test utils to
not to generate any stack allocator code
when USMP is used to make the tests more
strict.

This change also removes the confusing
"run_model" which has semantics identitical
to "__tvm_main__" in TIR.

)

This PR introduces a function signature printer in the `TypedPackedFunc` part, so that the log information in `detail::unpack_call` will be more complete. This PR allows users to obatin the original function signature when the `detail::unpack_call` fails.

* refactored GraphProto.from_onnx into smaller functions

* black formatted file

* removed line that does not seem to make sense. Is there a purpose that I missed?

* just to trigger CI pipeline

This was broken in apache#10267, not sure how that commit passed CI (maybe some logic to figure out the PR diff in pylint is broken).

Co-authored-by: driazati <driazati@users.noreply.github.com>

* fix canonical simplifier

* improve comments

…n of all conditions can not be none. (apache#10340)

Co-authored-by: sqing <qing.siqi@intellif.com>

)

The 21.11 update for the Ethos(TM)-N driver is slightly more strict in
accepting various operator attributes.

* [microNPU][4] Add the cascader Proposal generator

The Proposal generator takes optimal Plans and combines
them to find optimal 'Proposals' - sets of disjoint
Plans that cover every Part in a CascaderGraph. It
ultimately produces a Pareto-frontier of 'optimal'
Proposals in terms of estimated cycles and memory usage.

Change-Id: Id42099819a596496a5769bae22f08eeb75ec69b6

* Fixes

Change-Id: I4f5f2a298bd3bb379c7c8d179150358923b0dd66

… forwarding notification mechanism. (apache#10234)

* [Runtime][Pipeline Executor] multiple threads management and the
data forwarding notification mechanism.

In this patch we create working threads for each runtime of pipeline.
the threads would be terminated once the runtime class gets destroyed.

We also add a notification mechanism derived from the 'binding configuration'
of the runtime to forward the data notification.

* address review comments.

* address review comments.

* fix typo.

* fix typo.

* trigger build.

* address review comments.

* address review comments.

* address review comments.

* address review comments.

This is the C++ code for running Hexagon code on simulator via the
RPC mechanism. It is intended to be integrated into the current
HexagonLauncher, although the integration will require further changes
to the launcher python code.

The final goal is to be able to run the same file.py on either
hardware or simulator without needing to edit the python file, but
simply by changing the configuration of the execution platform
(i.e. something like --exectute-on=simulator as a command line or
in an environment variable). The exact details are still to be
determined.

* update AMP table to enable ResNet50 conversion

* add runtime datatype dispatch for BFloat16

* skip asserts for uint16 for bf16 compatibility

* add bf16 cast for the unary intrinsic operators

* enable "bf16<-->fp32<-->any dtype" casting

* support inconsistent input for bf16 BIOP legalize

* add treatments for bfloat16 in if statements

* add bfloat16 dtype casts in binary OP

* delete unnecessary treatments for bfloat16

* add test for bfloat16 building

* code style

* restore the modifications in .gitignore

* restore the changes to AMP lists

* fix typos

* fix lint errors

* fix typo

* [ci] Check more events before pinging reviewers

This was missing some events before (reviews without comments, PR updated from a draft -> ready for review) so these were being ignored when finding the latest event. This PR adds them and restructures the code a bit to make it more clear what is happening for each PR. This addresses some of the issues from apache#9983

* fix tests

Co-authored-by: driazati <driazati@users.noreply.github.com>

…#10365)

* Lower cache_read and cache_write to Hexagon DMA via tensorize

* rework test to be compatible with launcher

* remove cpu device api mem_copy implementation and test

Adding a few tests to confirm memory usage
with and without USMP.

- Supporting the toggle to disable storage_rewrite.
- There is a slight change to tir_to_cs_translator to
   add index of Load nodes associated with NpuAddressRange objects

…arameter virtual devices in virtual_device_ field (apache#10352)

* parent 33082e0
author electriclilies <lilyorthsmith@gmail.com> 1643141097 -0800
committer Lily Orth-Smith <lilyorthsmith@gmail.com> 1645560059 -0800

Store function param virtual devices in virtual_device_ field

Fix test_annotation.py and change result_virtual_device to virtual_device

* Change plan devices tests to use the new syntax for function parameters

* Fix free var problem

* Fix attribute parsing if there is virtual device; most device planning tests passgit status

* fixed lambda lifting

* Debugging high order functions -- right now FunctionOnDevice and Bind are mutually recursive. This needs to not be the case.

* tests pass wootgit status

* Remove FunctionOnDevice from device planner

* Don't use MaybeFunctionOnDevice in VM compiler

* Remove MaybeFunctionOnDevice from lambda lifter

* Delete FunctionOnDevice and MaybeFunctionOnDevice!

* Reomve GetFunctionResultVirtualDevice

* Remove GetFunctionParamVirtualDevice

* lint

* lint

* Python formatting

* Remove FunctionOnDevice python test

* Fix bug in binds & debug output

* Fix text printer

* lint

* Remove function on device from fold constant tests

* Mark nits

* Revert behavior of bind

* clean up debug

* Make ExprBinder public interface and use instead of Bind

* Fix lambda lift

* This is broken but not sure how to fix

* passes all device planning tests yay!

* Add substitution helper and use in device planner

* Remove unnecessary check

* Respond to comments

* Update comment