[refactor] dynamically import TrainSpec

[tianyu-l]

which handles import more efficiently and avoids accidental failure.

After this PR, each new model or experiment need to define a get_train_spec function in their __init__.py file.

[fegin]

left some comments, stamp to unblock critical fix

[fegin]

We should do a sanity check here.

assert _supported_models. isdisjoint(_supported_experiments)

This allows us to avoid having duplicated name. You can change _supported_models and _supported_experiments to be set.

[fegin]

Both tests failure are real.

[wwwjn]

Need to fix model name here:

torchtitan/torchtitan/experiments/simple_fsdp/tests/integration_tests.py

Line 46 in 85d92de

"--model.name llama3_simple_fsdp",

[ruisizhang123]

hmmm get a question for this PR: I'm trying to import either deepseek_simple_fsdp or llama_simple_fsdp from simple_fsdp/__init__.py. I understand you want a simple_fsdp indentifier to import this folder in torchtitan/protocols/train_spec.py.

Wonder if there is a way that I could specific one of the two model(deepseek or llama) Train_Spec in simple_fsdp folder without significantly refactor the codebase?