Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Set TP argument correctly when instantiating PagedKVCacheManager (#94)
#### Motivation Users are seeing runtime errors when trying to use TP>1 with speculative decoding. #### Modifications We need to set the tensor parallel argument correctly when we instantiate the PagedKVCacheManager. #### Result I have verified that this change resolves the reported issue. #### Related Issues https://huggingface.co/ibm-fms/llama3-8b-accelerator/discussions/1 Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
- Loading branch information