[WIP] Add tests for lora finetune single device act offload

[janeyx99]

Need to rebase this afer #1443

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/1604

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 4 Cancelled Jobs

As of commit b3ef790 with merge base df29d8a ():

NEW FAILURES - The following jobs have failed:

• GPU tests / gpu_test (3.11, stable) (gh)
  tests/recipes/test_qat_distributed.py::TestQATDistributedRecipe::test_loss[llama3/8B_qat_full-llama3-tune]
• Recipe Tests / recipe_test (3.10) (gh)
  tests/recipes/test_lora_finetune_single_device.py::TestLoRAFinetuneSingleDeviceRecipe::test_loss[llama3/8B_lora_single_device-llama3-tune-True-False-False]

CANCELLED JOBS - The following jobs were cancelled. Please retry:

• GPU tests / gpu_test (3.10, stable) (gh)
  ##[error]The operation was canceled.
• GPU tests / gpu_test (3.9, stable) (gh)
• Recipe Tests / recipe_test (3.11) (gh)
• Recipe Tests / recipe_test (3.9) (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.