-
Notifications
You must be signed in to change notification settings - Fork 6.8k
Image normalize operator - GPU support, 3D/4D inputs #13802
Image normalize operator - GPU support, 3D/4D inputs #13802
Conversation
src/operator/image/normalize_op.cc
Outdated
[](const NodeAttrs& attrs) { | ||
return std::vector<std::pair<int, int> >{{0, 0}}; | ||
}) | ||
.set_attr<nnvm::FGradient>("FGradient", ElemwiseGradUseNone{ "_copy" }) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
copy gradient is not correct IMO, even though this OP rarely pass gradient back, we still need to take care of this
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for flagging this. Yes you are right, it is should out_grad * 1/std_dev for each channel. Fixed and also added tests for symbolic forward/backward and gradient checks.
0b5dd72
to
7fdced9
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some high-level issues with this PR:
Make normalize as operator. No contribution to gradient.
- based on the description, this PR seems to assume that
normalize
is not yet an operator. but it is!.normalize
is an operator in theimage
namespace. - the unnecessary file movement should be reverted so that edit history of the code are best preserved. suggest to make changes in the existing files.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Meant to request changes for the above.
@stu1130 @zhreshold - Thanks for your review. Addressed your comments. Can you please take a look? Thanks. |
|
From first glance there doesn't seem to be that many changes to the extent that it would cause confusion. Let's at least give it a try. |
I can surely do that, but, I don't understand/convinced the motivation for doing that. Can you please help me understand? If code change history is the only reason, this commit, PR will surely help track and provide the history. For doing this way, like I said earlier, this is just the way every other operator is done in MXNet, and more over normalize is not a random augmentation operator to be part of image-random-inl.h. Also, there will be more functionalities for example supporting "list input" and so on. |
Similar to #13837 (comment) |
7011380
to
149eb37
Compare
@zhreshold , @szha - I moved back all changes to same files as per suggestion. Can you please take a look at this PR? Thanks! |
@zhreshold - Thanks! I addressed your review comments. |
a803806
to
83c21a9
Compare
* CPU version of normalize operator is working and unit test added * Add GPU implementation and tests * Working GPU normalize transforms * Add default values, fix imports, fix documentation * Add backward implmentation for image normalize * Add tests for backward pass * Move back operators to its original files * Add review comments * Add 4D example * Make infer type generic * Fix inline function build error * make functions as inline to avoid multiple definition conflict across cc and cu * Fix build errors * Fix failing GPU tests
* CPU version of normalize operator is working and unit test added * Add GPU implementation and tests * Working GPU normalize transforms * Add default values, fix imports, fix documentation * Add backward implmentation for image normalize * Add tests for backward pass * Move back operators to its original files * Add review comments * Add 4D example * Make infer type generic * Fix inline function build error * make functions as inline to avoid multiple definition conflict across cc and cu * Fix build errors * Fix failing GPU tests
* CPU version of normalize operator is working and unit test added * Add GPU implementation and tests * Working GPU normalize transforms * Add default values, fix imports, fix documentation * Add backward implmentation for image normalize * Add tests for backward pass * Move back operators to its original files * Add review comments * Add 4D example * Make infer type generic * Fix inline function build error * make functions as inline to avoid multiple definition conflict across cc and cu * Fix build errors * Fix failing GPU tests
Description
Make normalize as operator. No contribution to gradient.Add backward function to normalize operator.Background
We can enable users to fuse transformation operators with network graph and export it. This end to end network with transforms + network will greatly simplify inference deployment.
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.
Changes
Comments
@apeforest @stu1130 @zhreshold @nswamy - For review