Circular imports

[ebsmothers]

Bug description

Running e.g.

from torchtitan.components.validate import BaseValidator

throws an error about attempting to import from a partially initialized module. The import chain is

1. In torchtitan/components/validate.py: from torchtitan.distributed import utils as dist_utils
2. In torchtitan/distributed/utils.py: from torchtitan.models.attention import ScaledDotProductAttention
3. In torchtitan/models/__init__.py: import torchtitan.models.deepseek_v3
4. In torchtitan/models/deepseek_v3/__init__.py: from torchtitan.experiments.llama4.optimizer import build_llama4_optimizers
5. In torchtitan/experiments/__init__.py: import torchtitan.experiments.llama4
6. In torchtitan/experiments/llama4/__init__.py: from torchtitan.models.llama3 import pipeline_llama
7. In torchtitan/models/llama3/__init__.py: from torchtitan.components.validate import build_validator

(4) seems like the problem, since the core repo shouldn't be importing from the experiments directory. I assume this is from DeepSeek's recent move to core, probably build_llama4_optimizers can also be moved to core (and maybe renamed)?

Versions

n/a

[allela-roy]

any update on this issue?

[tianyu-l]

hmm I don't seem to have issues locally...

Anyways, will do some refactor soon, hopefully we can get rid of this.

[msaroufim]

I'm not getting a crash but the import is super slow

(myenv) ➜  torchtitan git:(main) time python -c "from torchtitan.components.validate import BaseValidator"
python -c "from torchtitan.components.validate import BaseValidator"  16.03s user 1.15s system 179% cpu 9.583 total
(myenv) ➜  torchtitan git:(main) 

[tianyu-l]

Let me do some refactor and see if it's helpful.

[tianyu-l]

#1534 fixes 2 and 4, but honestly there still seem to be other circular dependencies.

Also I'm getting

time python -c "from torchtitan.components.validate import BaseValidator"
real    0m5.247s
user    0m5.140s
sys     0m6.916s

[tianyu-l]

After #1740, I get

time python -c "from torchtitan.components.validate import BaseValidator"
real    0m3.758s
user    0m3.776s
sys     0m6.913s