Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow passing of can_stream in openai_models.py #600

Merged
merged 3 commits into from
Nov 6, 2024
Merged

Allow passing of can_stream in openai_models.py #600

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6a68078
Allow passing of can_stream in openai_models.py
cmungall Oct 31, 2024
fe58ea0
Only set can_stream: false if it is false
simonw Nov 6, 2024
b8b1374
Docs for can_stream: false
simonw Nov 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/other-models.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,8 @@ You can set `api_key_name` to the name of a key stored using the {ref}`api-keys`

Add `completion: true` if the model is a completion model that uses a `/completion` as opposed to a `/completion/chat` endpoint.

If a model does not support streaming, add `can_stream: false` to disable the streaming option.

Having configured the model like this, run `llm models` to check that it installed correctly. You can then run prompts against it like so:

```bash
Expand Down
4 changes: 4 additions & 0 deletions llm/default_plugins/openai_models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,9 @@ def register_models(register):
api_version = extra_model.get("api_version")
api_engine = extra_model.get("api_engine")
headers = extra_model.get("headers")
kwargs = {}
if extra_model.get("can_stream") is False:
kwargs["can_stream"] = False
if extra_model.get("completion"):
klass = Completion
else:
Expand All @@ -71,6 +74,7 @@ def register_models(register):
api_version=api_version,
api_engine=api_engine,
headers=headers,
**kwargs,
)
if api_base:
chat_model.needs_key = None
Expand Down
Loading