[FEATURE] Add feature of retain_grad

[KexinFeng]

Description

The PR adds the support for fetching the gradients of intermediate variables in a computation graph.

The motivation of this feature comes from this issue.

Checklist

Essentials

• PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage
• Code is well-documented

Changes

• Imperative::MarkVariables in src/imperative/imperative.cc to include the case of marking nonleaf nodes
• Imperative::Backward in src/imperative/imperative.cc to retrieve nonleaf nodes and link node->info.out_grads[] to buffered arrays.
• Graph BuildGradientGraph to map forward node to the corresponding gradient node entry.

Comments

• Interesting edge cases to note here: 1. the gradient-queried node is the output node, z.retain_grad(). 2. when it is used combined with detaching computation.
• Note that u.retain_grad() is called outside the scope of ag.record(). Otherwise, u.retain_grad() would be recorded as operation and trigger building of computational node. This is conceptually not desired.

[mxnet-bot]

Hey @KexinFeng , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

• To trigger all jobs: @mxnet-bot run ci [all]
• To trigger specific jobs: @mxnet-bot run ci [job1, job2]

---

CI supported jobs: [centos-gpu, centos-cpu, windows-cpu, unix-gpu, website, sanity, edge, miscellaneous, unix-cpu, windows-gpu, clang]

---

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

[mxnet-bot]

Jenkins CI successfully triggered : [unix-cpu]

[barry-jin]

@mxnet-bot run ci [unix-gpu, centos-gpu]

[mxnet-bot]

Jenkins CI successfully triggered : [centos-gpu, unix-gpu]

[barry-jin]

@mxnet-bot run ci [centos-gpu]

[mxnet-bot]

Jenkins CI successfully triggered : [centos-gpu]