Remove TensorDict usage

[Toni-SM]

Description

Remove TensorDict usage from Isaac Lab

Type of change

• Breaking change... for those ones using TensorDict specific API such as camera.data.output.to_dict() or camera.data.output.sorted_keys

Screenshots

Before

After

Checklist

• I have run the pre-commit checks with ./isaaclab.sh --format
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have updated the changelog and the corresponding version in the extension's config/extension.toml file
• I have added my name to the CONTRIBUTORS.md or my name already exists there

[pascal-roth]

Before a more detailed review, I was wondering what is the reason why we should prefer normal dicts instead of Tensordict (thought we would rather use more and not less tensordicts in the code)?

[Toni-SM]

Before a more detailed review, I was wondering what is the reason why we should prefer normal dicts instead of Tensordict (thought we would rather use more and not less tensordicts in the code)?

1. Env API handles state, observation, action, reward, terminated, truncated and info.

   • For state, observation and action, we support fundamental and composite spaces (at least in Direct workflow): Box (tensor), Discrete (tensor), MultiDiscrete (tensor), Dict (dict), Tuple (tuple). TensorDict does not support tuples, so there is no point in using it for spaces in single or multi-agent envs.
   • For single-agent envs, reward, terminated and truncated are just tensors not so in multi-agent (dict of tensors)... but since it cannot be used for spaces, it does not make sense that some components use it and others do not.

2. TensorDict is a PyTorch-scpecific thing which limits the implementation to that framework (making it difficult to scale Isaac Lab to other ML frameworks).

3. Nested TensorDicts is pythonic ....
   

[pascal-roth]

Okay makes sense to me, it was anyway basically the only place where we use them. @Mayankm96 if I remember correctly then you were in favor of using them, so tagging you here for viz