[Feature] Detect anomalous parameters

[jshilong]

Motivation

Sometimes there are some anomalous parameters in the training phase, mainly two cases

• Some parameters are not be used in a forward pass due to our wrong implementation, which causes the error when doing loss.backward()
• Some parameters are not be used to produce loss, which causes the deadlock when doing loss.backward()

In both cases，anomalous parameters are not included in the computational graph that is with loss as the root.

Modification

I add a debug option named detect_anomalous_params to OptimizerHook, which can help you find anomalous parameters

BC-breaking (Optional)

None

[jshilong]

Ping @ZwwWayne

[zhouzaida]

After resolving the above comments and the CI passes, the PR can be merged.

[ZwwWayne]

Can we check whether it slows down the training speed if we set detect_anomalous_params=True?
This PR can be merged after checking the influence on training speed and updating this information in docstring.

[zhouzaida]

Can we check whether it slows down the training speed if we set detect_anomalous_params=True? This PR can be merged after checking the influence on training speed and updating this information in docstring.

It probably slows down the training speed but the flag should only be opened for debugging.

[zhouzaida]

Please merge the upstream master branch to this PR which will resolve the error when building documentation.

[jshilong]

Can we check whether it slows down the training speed if we set detect_anomalous_params=True? This PR can be merged after checking the influence on training speed and updating this information in docstring.

It probably slows down the training speed but the flag should only be opened for debugging.

Can we check whether it slows down the training speed if we set detect_anomalous_params=True? This PR can be merged after checking the influence on training speed and updating this information in docstring.

It is only used for debugging

[jshilong]

Can we check whether it slows down the training speed if we set detect_anomalous_params=True? This PR can be merged after checking the influence on training speed and updating this information in docstring.

I will add a notice in the docstr to indicate this is only used for debugging

[zhouzaida]

I will add a notice in the docstr to indicate this is only used for debugging

Hi @jshilong , we will merge the PR after you add a notice in the docstring.

[jshilong]

I will add a notice in the docstr to indicate this is only used for debugging

Hi @jshilong , we will merge the PR after you add a notice in the docstring.

Done