Skip to content

[FEATURE]: Register model under multiple names for LoRA #2267

@grahamking

Description

@grahamking

Feature request

A worker should be able to 1. register itself under multiple names, and 2. the used name should be passed on to the worker. It will use the name to decide which LoRA to use.

Thoughts on 1:

  • register_llm could accept either a string or an array of string
  • In Rust we put the handler in the map multiple times. The handler will probably be in an Arc

Thoughts on 2:

  • If we don't do pre-processing the worker should already get the model name, we pass on the request.
  • If we do pre-processing, check that the model name ends up in the JSON that we send the model, with the token ids.

Metadata

Metadata

Assignees

Labels

dynamo-llmRelates to dynamo-llm componentenhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions