Inference tutorial - Part 3 of e2e series [WIP] #2343
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/2343
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit ce675b8 with merge base 101c039 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
jainapurva
added
the
topic: documentation
| print("Response:", output_text[0][len(prompt):]) | ||
|
|
||
|
|
||
| [Optional] Float8 Dynamic Quantization + Semi-structured (2:4) sparsity |
There was a problem hiding this comment.
@jerryzh168 @jcaip Does this look good? Should I keep sparsity as a optional section or just mention it in note
There was a problem hiding this comment.
can we just add to huggingface torchao page?
There was a problem hiding this comment.
Remove it from here, and just add keep the note for hf-torchao page ?
docs/source/inference.rst
Outdated
|
|
||
| vllm serve pytorch/Phi-4-mini-instruct-float8dq --tokenizer microsoft/Phi-4-mini-instruct -O3 | ||
|
|
||
| Inference with vLLM |
There was a problem hiding this comment.
should we move this after Inference with Transformers
There was a problem hiding this comment.
cc @jainapurva I think if vLLM is our recommended serving solution, this should go before transformers.
docs/source/inference.rst
Outdated
|
|
||
| vLLM automatically leverages torchao's optimized kernels when serving quantized models, providing significant throughput improvements. | ||
|
|
||
| Setting up vLLM with Quantized Models |
There was a problem hiding this comment.
nit: this doesn't have to be a new section I think
| Performance Breakdown | ||
| ===================== | ||
|
|
||
| When using vLLM with torchao: |
There was a problem hiding this comment.
this is not a comprehensive list, probably just remove, do we have a exhaustive list of all the techniques that we support?
There was a problem hiding this comment.
I don't think we've a comprehensive list. If we decide to make it, that could be another doc page or readme
There was a problem hiding this comment.
I think we can remove this section in that case
|
Hi @jainapurva, by the way I'm adding a 
|
jainapurva
force-pushed
the
inference_tutorial
branch
from
b93b892 to
ce675b8
Compare
| .. note:: | ||
| For more information on supported quantization and sparsity configurations, see `HF-Torchao Docs <https://huggingface.co/docs/transformers/main/en/quantization/torchao>`_. | ||
|
|
||
| Inference with vLLM |
There was a problem hiding this comment.
for this section, can you replace with https://huggingface.co/pytorch/Qwen3-8B-int4wo-hqq#inference-with-vllm
it might be easier to do command line compared to code
No description provided.