common : fix parallel shard download interleaving output

[TevinWang]

Motivation

Currently when downloading model shards, the progress output shown is interleaved, which is difficult to interpret. To fix this, we case on whether the download is a shard, and create a global table that stores the progress information for each model shard. We continuously update a progress bar for each of the shards in llama_load_model_from_url until all downloads are complete.
Closes #6537

Changes

• • Add boolean value to llama_download_file to indicate whether a download is a shard
• • Add curl property to use a custom progress callback
• • Create a progressTable global to store progress information
• • Print information to console every second as progress bar output while shards are downloading
• • Buffer the download finished/file saved output and print to stderr once ALL downloads complete to prevent interleaving output

Before and after:

Testing:

Tested using the following command on CPU:

./main --hf-repo ggml-org/models \
       --hf-file tinyllamas/split/stories15M-00001-of-00003.gguf \
       --model models/test-model-00001-of-00003.gguf \
       -ngl 0 \
       --prompt "I believe the meaning of life is"

[phymbert]

Well, I would prefer to properly implement a global progress callback rather than completely remove progression.

[TevinWang]

@phymbert implemented a global progress callback and updated the PR description!

[phymbert]

It must be done without global variables.

[phymbert]

Is it something easily possible you think @TevinWang ?

[phymbert]

@TevinWang do you need help on the global variable issue ?

@ggerganov do you confirm the issue with the proposed approach? IMHO this contribution is valuable as the console output is currently hard to read

[ggerganov]

Yes, the globals should be avoided