downloads are unpredictable

[chris-aeviator]

System Info

1.3.4 (docker)

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

When trying to bring up a container model downloads start fine. Then downloading suddenly stops and won't resume, also no error is beeing thrown.

WARN[0000] The "HUGGING_FACE_HUB_TOKEN" variable is not set. Defaulting to a blank string.
tgi-tgi-1 | {"timestamp":"2024-01-22T09:51:12.247781Z","level":"INFO","fields":{"message":"Args { model_id: "mistralai/mixtral-8x7b-Instruct-v0.1", revision: None, validation_workers: 2, sharded: None, num_shard: Some(1), quantize: Some(BitsandbytesNF4), speculate: None, dtype: None, trust_remote_code: false, max_concurrent_requests: 128, max_best_of: 2, max_stop_sequences: 4, max_top_n_tokens: 5, max_input_length: 2048, max_total_tokens: 16000, waiting_served_ratio: 1.2, max_batch_prefill_tokens: 4096, max_batch_total_tokens: None, max_waiting_tokens: 20, hostname: "e14228c55b0b", port: 80, shard_uds_path: "/tmp/text-generation-server", master_addr: "localhost", master_port: 29500, huggingface_hub_cache: Some("/data"), weights_cache_override: None, disable_custom_kernels: false, cuda_memory_fraction: 1.0, rope_scaling: None, rope_factor: None, json_output: true, otlp_endpoint: None, cors_allow_origin: [], watermark_gamma: None, watermark_delta: None, ngrok: false, ngrok_authtoken: None, ngrok_edge: None, env: false }"},"target":"text_generation_launcher"}
tgi-tgi-1 | {"timestamp":"2024-01-22T09:51:12.248008Z","level":"INFO","fields":{"message":"Starting download process."},"target":"text_generation_launcher","span":{"name":"download"},"spans":[{"name":"download"}]}
tgi-tgi-1 | {"timestamp":"2024-01-22T09:51:17.527458Z","level":"INFO","fields":{"message":"File model-00001-of-00019.safetensors already present in cache.\n"},"target":"text_generation_launcher"}
tgi-tgi-1 | {"timestamp":"2024-01-22T09:51:17.527575Z","level":"INFO","fields":{"message":"Download: [1/19] -- ETA: 0:00:00\n"},"target":"text_generation_launcher"}
tgi-tgi-1 | {"timestamp":"2024-01-22T09:51:17.528150Z","level":"INFO","fields":{"message":"Download file: model-00002-of-00019.safetensors\n"},"target":"text_generation_launcher"}
^[[Btgi-tgi-1 | {"timestamp":"2024-01-22T09:54:48.522264Z","level":"INFO","fields":{"message":"Downloaded /data/models--mistralai--mixtral-8x7b-Instruct-v0.1/snapshots/125c431e2ff41a156b9f9076f744d2f35dd6e67a/model-00002-of-00019.safetensors in 0:03:30.\n"},"target":"text_generation_launcher"}
tgi-tgi-1 | {"timestamp":"2024-01-22T09:54:48.522386Z","level":"INFO","fields":{"message":"Download: [2/19] -- ETA: 0:29:45\n"},"target":"text_generation_launcher"}
tgi-tgi-1 | {"timestamp":"2024-01-22T09:54:48.523191Z","level":"INFO","fields":{"message":"Download file: model-00003-of-00019.safetensors\n"},"target":"text_generation_launcher"}
WARN[0000] The "HUGGING_FACE_HUB_TOKEN" variable is not set. Defaulting to a blank string.
tgi-tgi-1 | {"timestamp":"2024-01-22T09:51:12.247781Z","level":"INFO","fields":{"message":"Args { model_id: "mistralai/mixtral-8x7b-Instruct-v0.1", revision: None, validation_workers: 2, sharded: None, num_shard: Some(1), quantize: Some(BitsandbytesNF4), speculate: None, dtype: None, trust_remote_code: false, max_concurrent_requests: 128, max_best_of: 2, max_stop_sequences: 4, max_top_n_tokens: 5, max_input_length: 2048, max_total_tokens: 16000, waiting_served_ratio: 1.2, max_batch_prefill_tokens: 4096, max_batch_total_tokens: None, max_waiting_tokens: 20, hostname: "e14228c55b0b", port: 80, shard_uds_path: "/tmp/text-generation-server", master_addr: "localhost", master_port: 29500, huggingface_hub_cache: Some("/data"), weights_cache_override: None, disable_custom_kernels: false, cuda_memory_fraction: 1.0, rope_scaling: None, rope_factor: None, json_output: true, otlp_endpoint: None, cors_allow_origin: [], watermark_gamma: None, watermark_delta: None, ngrok: false, ngrok_authtoken: None, ngrok_edge: None, env: false }"},"target":"text_generation_launcher"}
tgi-tgi-1 | {"timestamp":"2024-01-22T09:51:12.248008Z","level":"INFO","fields":{"message":"Starting download process."},"target":"text_generation_launcher","span":{"name":"download"},"spans":[{"name":"download"}]}
tgi-tgi-1 | {"timestamp":"2024-01-22T09:51:17.527458Z","level":"INFO","fields":{"message":"File model-00001-of-00019.safetensors already present in cache.\n"},"target":"text_generation_launcher"}
tgi-tgi-1 | {"timestamp":"2024-01-22T09:51:17.527575Z","level":"INFO","fields":{"message":"Download: [1/19] -- ETA: 0:00:00\n"},"target":"text_generation_launcher"}
tgi-tgi-1 | {"timestamp":"2024-01-22T09:51:17.528150Z","level":"INFO","fields":{"message":"Download file: model-00002-of-00019.safetensors\n"},"target":"text_generation_launcher"}
^[[Btgi-tgi-1 | {"timestamp":"2024-01-22T09:54:48.522264Z","level":"INFO","fields":{"message":"Downloaded /data/models--mistralai--mixtral-8x7b-Instruct-v0.1/snapshots/125c431e2ff41a156b9f9076f744d2f35dd6e67a/model-00002-of-00019.safetensors in 0:03:30.\n"},"target":"text_generation_launcher"}
tgi-tgi-1 | {"timestamp":"2024-01-22T09:54:48.522386Z","level":"INFO","fields":{"message":"Download: [2/19] -- ETA: 0:29:45\n"},"target":"text_generation_launcher"}
tgi-tgi-1 | {"timestamp":"2024-01-22T09:54:48.523191Z","level":"INFO","fields":{"message"

this happens with
HF_HUB_ENABLE_HF_TRANSFER=0 as well as =1
This is on a business fibre line.

Expected behavior

Downloads should work or TGI should error

[chris-aeviator]

basically I'm restarting the docker container 19 times to download mixtral successfully ;/

[sunnysmith]

I have seen the same behavior. Have you tried it with a hf token?

[spew]

Try setting your HF token:

The "HUGGING_FACE_HUB_TOKEN" variable is not set

[chris-aeviator]

I think the issue lies in tgi not erroring on either a) my HF_token is missing (though it was not needed to download model parts one by one) b) the download stalling. This should not be intetended behavior ?!

[OlivierDehaene]

Duplicate of #1186.
Some system had some download issues until #1486.
The ghcr.io/huggingface/text-generation-inference:sha-9c320e2 should fix your problem.