Skip to content

Commit

Permalink
Config: Fix draft model migration and loading
Browse files Browse the repository at this point in the history
The loader takes in the "draft" parameter, so map the config model
to that when creating kwargs for initial load.

Also map the old "draft" key to the new "draft_model" key.

Signed-off-by: kingbri <bdashore3@proton.me>
  • Loading branch information
bdashore3 committed Sep 18, 2024
1 parent a34bd9a commit 754fb15
Show file tree
Hide file tree
Showing 2 changed files with 6 additions and 2 deletions.
2 changes: 1 addition & 1 deletion common/tabby_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ def _from_file(self, config_path: pathlib.Path):

if model_cfg.get("draft"):
legacy = True
cfg["draft"] = model_cfg["draft"]
cfg["draft_model"] = model_cfg["draft"]

if model_cfg.get("lora"):
legacy = True
Expand Down
6 changes: 5 additions & 1 deletion main.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,11 @@ async def entrypoint_async():
model_path = model_path / model_name

# TODO: remove model_dump()
await model.load_model(model_path.resolve(), **config.model.model_dump())
await model.load_model(
model_path.resolve(),
**config.model.model_dump(),
draft=config.draft_model.model_dump(),
)

# Load loras after loading the model
if config.lora.loras:
Expand Down

0 comments on commit 754fb15

Please sign in to comment.