[Feature]: Stream from HF into TPU HBM

[samos123]

🚀 The feature, motivation and pitch

GPU allows streaming from HF directly into GPU memory. This is required when attached disk or local disk isn't available / big enough.

Alternatives

No response

Additional context

I tried removing --download-dir but it still downloads to a local directory

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[carlesoctav]

iirc, you can use runai streamer from vllm to stream the weight from cloud storage to cpu ram directly. and for flax_nnx impl, it'll be enabled on this #955.

[samos123]

vllm on GPU supports streaming from HF to GPU memory natively without using runai streamer. Note I'm not talking about cloud storage, that's a separate FR.