[FEATURE] Add feature of attach_grad to nonleaf variables in HybridizedBlock.

[KexinFeng]

Description

The PR adds the support for fetching the gradients of intermediate variables in a gluon HybridizedBlock. This applies uniformly to both when block.hybridize() is on and off. This generates the attach_grad implemented in implemented in PR#20500.

The motivation of this feature comes from this issue#11865.

Checklist

Essentials

• PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage
• Code is well-documented

Changes

• block.py where mark_vars and get_mark_vars are added along with MXNDArrayMarkDCVariables.
• cached_op.invoke in cpp backend and CachedOp.__call__ have been editted to include the pass of marked nonleaf ndarrays.
• set_nleafs method is added into CachedOp class to store the marked nonleaf ndarrays.
• Inside void RunGraph, marked nonleaf ndarrays are linked to the marked computational node for autograd computation.

Comments

• This feature is built on top of PR#20500. The modification here is mainly in the invoke of CachedOp computation.

[mxnet-bot]

Hey @KexinFeng , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

• To trigger all jobs: @mxnet-bot run ci [all]
• To trigger specific jobs: @mxnet-bot run ci [job1, job2]

---

CI supported jobs: [sanity, windows-gpu, clang, edge, unix-cpu, windows-cpu, unix-gpu, centos-cpu, website, miscellaneous, centos-gpu]

---

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

[leezu]

For a good user experience, it would be great if we have an API that supports both hybrid and imperative mode. It doesn't need to use the same backend implementation, but it would be good to have a consistent frontend API.

[KexinFeng]

Now both hybridized and unhybrized are unified. This is shown in the last two examples in unittest/test_autograd.py