Remove superfluous datatype and device conversions

[shadeMe]

Description

Removes a couple of unnecessary type conversions and calls to torch.to when preparing inputs for the inner PyTorch transformer model.

Types of change

Fix

Checklist

• I confirm that I have the right to submit this contribution under the project's MIT license.
• I ran the tests, and all new and existing tests passed.
• My changes don't require a change to the documentation, or if they do, I've added all required information.