Add Anymal-D torchrl cfg

[fyu-bdai]

Description

Adds TorchRL training configuration for the Anymal-D velocity environment as a template for training IsaacLab environments with the new torchrl training workflow.

You can try training Anymal with TorchRL (after merging all TorchRL PRs) using
/workspace/isaaclab/source/standalone/workflows/torchrl python train.py --task Isaac-Velocity-Flat-Anymal-D-v0 --num_envs 4096

Related PRs:
#1178, #1179

This is the last PR in the group of 3 that adds the TorchRL training pipeline.

⚠️ The not so good parts ⚠️

Unfortunately, the Anymal-D environment converges slower than RSL-RL. This is probably due to policy architecture and PPO implementation differences which requires different hyperparameter settings. I have provided the best hyperparameters that have worked for me so far. While it is possible to speed up the convergence by increasing desired_kl targets and entropy_coef to closely match RSL-RL, torchrl policies seem to crash late during training due to spurious KL/action noise spikes after the reward has long converged.

Training curves and video

Video_3197_7d9452698382ed1512bd.mp4

Type of change

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

Checklist

• I have run the pre-commit checks with ./isaaclab.sh --format
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have updated the changelog and the corresponding version in the extension's config/extension.toml file
• I have added my name to the CONTRIBUTORS.md or my name already exists there

[Toni-SM]

Question: How having the model manually defined in the agent config will support changing it from CLI (e.g.: using hydra)?

[fyu-bdai]

I suppose we can add an argument to the torchrl CLI args to specify which model definition method the user would like to use if we want to support both methods?