Experimental Flag for tune ls

[pbontrager]

Context

What is the purpose of this PR? Is it to

• add a new feature
• fix a bug
• update tests and/or documentation
• other (please add here)

Currently we have a number of recipes under dev that we would like to make accessible to users but still clearly signal to users that these are not as well tested and may have special dependencies. The current behavior is to list these all in "tune ls" which isn't ideal as it doesn't disclose the experimental nature of these recipes. One option would be to exclude them entirely from "tune ls" but then it makes it very difficult to use them outside of github.

The proposal here is to add an "experimental" flag to tune ls that displays the extra recipes. All of the recipes are always available in tune run, but if you want to view them in ls you must pass the extra flag. This is accomplished by adding an extra parameter to the recipe dataclass in _recipe_registry that allows some recipes to be marked as experimental. Then get_all_recipes can accept an argument "include_experimental". This is True for "tune run" and False by default for "tune ls".

Changelog

• include_experimental added to get_all_recipes
• experimental added to recipe dataclass
• experimental flag added to tune ls
• Updated unit test and cli rst
• Fixed bug where dev recipes weren't included in project package

Test plan

Ran tests and test tune ls in terminal for expected outputs

• run pre-commit hooks and linters (make sure you've first installed via pre-commit install)
• add unit tests for any new functionality
• update docstrings for any new or updated methods or classes
• run unit tests via pytest tests
• run recipe tests via pytest tests -m integration_test
• manually run any new or modified recipes with sufficient proof of correctness
  • include relevant commands and any other artifacts in this summary (pastes of loss curves, eval results, etc.)

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/1163

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• [PREEMPTIVE] CPU amd64 non-GPU instances migration to Linux AMZN2023

✅ No Failures

As of commit f2dae01 with merge base bbc48e0 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[ebsmothers]

Can we just use something like configs/**.yaml here?

[RdoubleA]

it should be configs/**/*.yaml to match any number of subdirectories

[ebsmothers]

I think we already landed the existing version in another PR. Also after thinking about it more maybe it's better to be explicit here anyways