[Ecosystem] enable saving and loading FP8 model(#53) #1683
Conversation
|
@xin3he Could you please remove 'software ticket' and 'OHF' from the title? This PR is for OH |
|
@xin3he can you please address the comments. everything else sounds good to me! |
xin3he
changed the title
Co-authored-by: Yaser Afshar <yaser.afshar@intel.com>
Co-authored-by: Yaser Afshar <yaser.afshar@intel.com>
Co-authored-by: Yaser Afshar <yaser.afshar@intel.com>
Co-authored-by: Yaser Afshar <yaser.afshar@intel.com>
Co-authored-by: Yaser Afshar <yaser.afshar@intel.com>
Co-authored-by: Yaser Afshar <yaser.afshar@intel.com>
Co-authored-by: Yaser Afshar <yaser.afshar@intel.com>
|
Surely, thank you @yafshar, sorry for the delay response. |
Signed-off-by: Xin He <xinhe3@habana.ai>
| --bucket_internal \ | ||
| --load_quantized_model_with_inc | ||
| ``` | ||
|
|
There was a problem hiding this comment.
remove this section and just give examples of llama3
There was a problem hiding this comment.
Hi, @libinta I don't understand why we need to remove the loading section. Saving and loading is a pair that cannot work without each other.
There was a problem hiding this comment.
I think we misled you, what we save in huggingface format is different with existing huggingface models. The parameter name in neural magic models is different with INC quantized model. Also, INC only support one card save&load situation due to link. So we cannot reuse the load part for neural_magic models.
There was a problem hiding this comment.
@xin3he But the command is the same whether loading a model locally or from the hub right? We need to pass --load_quantized_model_with_inc and specify the model name (local path or hub name). So I agree with Libin that it would be better to have only one section.
|
A reminder of TODO:
|
| parser.add_argument( | ||
| "--saved_model_path", | ||
| type=str, | ||
| default="saved_results", |
There was a problem hiding this comment.
Maybe let's set the default to a better name? Like "inc_quantized_model" or so?
Co-authored-by: regisss <15324346+regisss@users.noreply.github.com>
| After quantizing the model, we can save it to a local path. | ||
|
|
||
| > [!NOTE] | ||
| > Before executing the command below, please refer to the "Running FP8 Models on a Single Device" section to measure the model quantization statistics. |
There was a problem hiding this comment.
| > Before executing the command below, please refer to the "Running FP8 Models on a Single Device" section to measure the model quantization statistics. | |
| > Before executing the command below, please refer to the ["Running FP8 Models on a Single Device" section](#running-fp8-models-on-a-single-device) to measure the model quantization statistics. |
Let's add a link to it too, it will be easier for readers.
Can you also add the same here please? https://github.com/HabanaAI/optimum-habana-fork/blob/98539470b6c6dbf39145844317d878ddf5482167/examples/text-generation/README.md?plain=1#L600
I had missed that.
| --bucket_internal \ | ||
| --load_quantized_model_with_inc | ||
| ``` | ||
|
|
There was a problem hiding this comment.
@xin3he But the command is the same whether loading a model locally or from the hub right? We need to pass --load_quantized_model_with_inc and specify the model name (local path or hub name). So I agree with Libin that it would be better to have only one section.
What does this PR do?
Fixes # (issue)
Before submitting