Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Relay][Memory][VM] #3560

Merged
merged 1 commit into from
Nov 1, 2019
Merged

[Relay][Memory][VM] #3560

merged 1 commit into from
Nov 1, 2019

Conversation

jroesch
Copy link
Member

@jroesch jroesch commented Jul 17, 2019

Explicitly manifest memory and tensor allocations in Relay. This is to enable future optimizations and transformations.

v0.0.3
fn (%x: Tensor[(10,), float32], %y: Tensor[(10,), float32]) -> Tensor[(10,), float32] {
  %1 = fn (%p0: Tensor[(10,), float32], %p1: Tensor[(10,), float32], Primitive=1) -> Tensor[(10,), float32] {
    %0 = add(%p0, %p0) /* ty=Tensor[(10,), float32] */;
    subtract(%0, %p1) /* ty=Tensor[(10,), float32] */
  };
  let %x1: Tensor[(10,), float32] = %1(%x, %y) /* ty=Tensor[(10,), float32] */;
  %x1
}
v0.0.3
fn (%x: Tensor[(10,), float32], %y: Tensor[(10,), float32]) -> Tensor[(10,), float32] {
  %0 = annotation.alloc_storage(320);
  let %x1: Tensor[(10,), float32] = let %output: Tensor[(10,), float32] = annotation.alloc_tensor(%0, meta[relay.Constant][0], meta[relay.attrs.AllocTensorAttrs][0]);
  %2 = fn (%p0: Tensor[(10,), float32], %p1: Tensor[(10,), float32], Primitive=1) -> Tensor[(10,), float32] {
    %1 = add(%p0, %p0) /* ty=Tensor[(10,), float32] */;
    subtract(%1, %p1) /* ty=Tensor[(10,), float32] */
  };
  %3 = (%x, %y);
  %4 = (%output,);
  let %v = annotation.invoke_tvm_op(%2, %3, %4);
  %output;
  %x1
}

I have an initial version of the PR nearly ready for review. The general goal of this PR is to perform all calling convention regularization which has historically been done in the runtime in the . IR instead, so we can further optimize. The final remaining piece is the transform of dynamic shape handling code into the IR as well.

@ajtulloch
Copy link
Contributor

@jroesch what kind of optimizations will this enable?

@wweic
Copy link
Contributor

wweic commented Jul 19, 2019

@ajtulloch This PR models memory allocation/deallocation explicitly in relay expression, it will open the opportunity to write passes to analyze memory operation and try allocation coalescing etc. It will also simplify vm bytecode generation.

@icemelon
Copy link
Member

icemelon commented Aug 3, 2019

Any updates? @jroesch

Copy link
Member

@zhiics zhiics left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jroesch I just started the review and had a quick glance on it. I will do a more careful review by this week. Jared, could you please provide a little more description for this PR, i.e. the mechanism of the memory optimization and why we introduce these annotations. That would help the review as well. Thank you.

include/tvm/runtime/vm.h Show resolved Hide resolved
python/tvm/relay/memory_alloc.py Outdated Show resolved Hide resolved
src/relay/op/memory/memory.cc Outdated Show resolved Hide resolved
src/relay/op/memory/memory.cc Outdated Show resolved Hide resolved
include/tvm/runtime/object.h Outdated Show resolved Hide resolved
include/tvm/runtime/object.h Outdated Show resolved Hide resolved
python/tvm/relay/memory_alloc.py Show resolved Hide resolved
python/tvm/relay/memory_alloc.py Outdated Show resolved Hide resolved
src/relay/ir/module.cc Outdated Show resolved Hide resolved
src/runtime/vm/vm.cc Outdated Show resolved Hide resolved
tests/python/relay/test_vm.py Outdated Show resolved Hide resolved
Copy link
Member

@zhiics zhiics left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks good. I have a few questions:

  • How is live memory released/killed currently?
  • Can we reuse the allocated memory as the current memory planning pass does? Or we just get memory from the pool directly?

include/tvm/relay/attrs/annotation.h Outdated Show resolved Hide resolved
include/tvm/relay/attrs/annotation.h Outdated Show resolved Hide resolved
include/tvm/relay/expr_functor.h Outdated Show resolved Hide resolved
include/tvm/relay/module.h Outdated Show resolved Hide resolved
include/tvm/runtime/vm.h Outdated Show resolved Hide resolved
src/relay/op/memory/memory.cc Outdated Show resolved Hide resolved
src/relay/op/memory/memory.cc Outdated Show resolved Hide resolved
src/relay/op/memory/memory.cc Outdated Show resolved Hide resolved
src/relay/op/memory/memory.cc Outdated Show resolved Hide resolved
src/relay/op/tensor/reduce.cc Outdated Show resolved Hide resolved
@jroesch jroesch marked this pull request as ready for review October 16, 2019 05:51
@jroesch jroesch force-pushed the memory branch 2 times, most recently from bf9555a to 2806a08 Compare October 18, 2019 09:02
@jroesch jroesch force-pushed the memory branch 2 times, most recently from 6a43034 to d7fefb9 Compare October 21, 2019 23:59
Copy link
Member

@zhiics zhiics left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am half way done. I will spend more time tomorrow to look at the design.

include/tvm/relay/attrs/annotation.h Outdated Show resolved Hide resolved
include/tvm/relay/attrs/memory.h Outdated Show resolved Hide resolved
include/tvm/relay/attrs/memory.h Outdated Show resolved Hide resolved
python/tvm/relay/op/memory/memory.py Outdated Show resolved Hide resolved
python/tvm/relay/op/memory/memory.py Outdated Show resolved Hide resolved
src/relay/backend/vm/compiler.cc Outdated Show resolved Hide resolved
src/relay/backend/vm/compiler.cc Outdated Show resolved Hide resolved
src/relay/op/memory/memory.cc Outdated Show resolved Hide resolved
src/relay/op/memory/memory.cc Outdated Show resolved Hide resolved
include/tvm/relay/attrs/memory.h Show resolved Hide resolved
python/tvm/relay/memory_alloc.py Outdated Show resolved Hide resolved
python/tvm/relay/memory_alloc.py Outdated Show resolved Hide resolved
python/tvm/relay/memory_alloc.py Outdated Show resolved Hide resolved
python/tvm/relay/memory_alloc.py Outdated Show resolved Hide resolved
python/tvm/relay/std/core.rly Show resolved Hide resolved
src/relay/op/memory/memory.cc Outdated Show resolved Hide resolved
include/tvm/relay/attrs/memory.h Outdated Show resolved Hide resolved
Copy link
Member

@icemelon icemelon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

half-way through. will continue after the flight

python/tvm/relay/op/memory/memory.py Outdated Show resolved Hide resolved
python/tvm/relay/memory_alloc.py Show resolved Hide resolved
src/runtime/vm/vm.cc Outdated Show resolved Hide resolved
src/runtime/vm/vm.cc Outdated Show resolved Hide resolved
include/tvm/relay/attrs/memory.h Outdated Show resolved Hide resolved
include/tvm/relay/expr_functor.h Outdated Show resolved Hide resolved
include/tvm/runtime/vm.h Show resolved Hide resolved
include/tvm/runtime/vm.h Show resolved Hide resolved
include/tvm/runtime/vm.h Show resolved Hide resolved
python/tvm/relay/memory_alloc.py Outdated Show resolved Hide resolved
python/tvm/relay/op/memory/memory.py Outdated Show resolved Hide resolved
python/tvm/relay/op/memory/memory.py Outdated Show resolved Hide resolved
src/runtime/vm/vm.cc Outdated Show resolved Hide resolved
src/relay/backend/vm/compiler.cc Outdated Show resolved Hide resolved
@zhiics
Copy link
Member

zhiics commented Oct 27, 2019

@jroesch I checked out the PR and debugged it. The compilation error is caused by converting tensors from ndarray to raw shape. On i386, I believe it is int32 but x86 it int64.

src/relay/backend/vm/compiler.cc Outdated Show resolved Hide resolved
src/relay/op/memory/memory.cc Outdated Show resolved Hide resolved
@jroesch jroesch force-pushed the memory branch 2 times, most recently from 1df4d3d to c4eb59f Compare October 29, 2019 22:35
src/runtime/vm/vm.cc Outdated Show resolved Hide resolved
Copy link
Member

@tqchen tqchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some high level comments

src/relay/ir/module.cc Outdated Show resolved Hide resolved
include/tvm/relay/expr_functor.h Outdated Show resolved Hide resolved
explicit TypeName( \
::tvm::runtime::ObjectPtr<::tvm::runtime::Object> n) \
: ParentType(n) {} \
ObjectName* operator->() { \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The mutational def was a bit dangerous, can we explicitly use the define_ref_methods and add the mutationable ref methods directly? This will also reduce one macro

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can not do that as the definitions are incompatible overloads

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

they are compatible, because operator->() and operator->()const are two different functions

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got an error about return type overload not working when I tried to do it.

include/tvm/relay/module.h Show resolved Hide resolved
src/relay/op/memory/memory.cc Outdated Show resolved Hide resolved
src/runtime/vm/vm.cc Show resolved Hide resolved
src/runtime/vm/vm.cc Outdated Show resolved Hide resolved
python/tvm/relay/memory_alloc.py Outdated Show resolved Hide resolved
src/relay/backend/vm/compiler.cc Show resolved Hide resolved
Copy link
Contributor

@wweic wweic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few minor comments.

include/tvm/expr_operator.h Outdated Show resolved Hide resolved
python/tvm/relay/op/memory/memory.py Outdated Show resolved Hide resolved
Copy link
Member

@zhiics zhiics left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Manifest memory allocations

Fix

Windows fixes

Clean up Storage memory manager

Address final review

Fix lint

Wei's comments

Disable dynamic input for now
@tqchen tqchen merged commit 2083513 into apache:master Nov 1, 2019
@jroesch jroesch deleted the memory branch February 4, 2021 04:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants