feat: Introduce RFC for Provider Configuration API #1359
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cdoern
requested review from
ashwinb,
dineshyv,
dltn,
ehhuang,
hardikjshah,
raghotham,
sixianyi0721,
terrytangyuan,
vladimirivic and
yanxi0830
as code owners
facebook-github-bot
added
the
CLA Signed
cdoern
changed the title
cdoern
force-pushed
the
config-proposal
branch
2 times, most recently
from
9b1a1e3 to
5d337fb
Compare
rfcs/RFC-0002-configuration.md
Outdated
There was a problem hiding this comment.
There can also be sensitive configurations such as API keys so we probably want to categorize fields that require redactions.
There was a problem hiding this comment.
hopefully this will be handled by the UserConfig class. I would imagine a user should not be able to see any API keys in the returned config
raghotham
reviewed
Mar 4, 2025
There was a problem hiding this comment.
Can we try to reduce the scope by calling it /v1/providers. That way we know that we are going to support provider CRUD. We can rejigger /inspect to move out provider specific methods.
The only concern here is the requirement for different software packages based on the provider. Right now, getting dependencies is a build time operation. For now, maybe we support adding new provider endpoints only for providers whose dependencies are already in the distro. Otherwise, we may get into situations where we package everything and cause bloat. Thoughts?
dmartinol
reviewed
Mar 4, 2025
rfcs/RFC-0002-configuration.md
Outdated
There was a problem hiding this comment.
we also need to check that user_field key in field.json_schema_extra is True, right?
Yeah Yep, it seems fair for an initial version to just be re-configuring already initialized providers! I hadn't thought of using it outside of that realm, good call out! |
cdoern
changed the title
Introduce an RFC for a Provider Configuration API, allowing for provider configuration changes of an existing stack to aid in common runtime changes like model swapping, endpoint switching, etc Signed-off-by: Charlie Doern <cdoern@redhat.com>
cdoern
added a commit
to cdoern/llama-stack
that referenced
this pull request
Mar 13, 2025
currently the `inspect` API for providers is really a `list` API. Create a new `providers` API which has a GET `providers/{provider_id}` inspect API
which returns "user friendly" configuration to the end user. Also add a GET `/providers` endpoint which returns the list of providers as `inspect/providers` does today.
This API follows CRUD and is more intuitive/RESTful.
This work is part of the RFC at llamastack#1359
Signed-off-by: Charlie Doern <cdoern@redhat.com>
ashwinb
pushed a commit
that referenced
this pull request
Mar 13, 2025
# What does this PR do?
currently the `inspect` API for providers is really a `list` API. Create
a new `providers` API which has a GET `providers/{provider_id}` inspect
API
which returns "user friendly" configuration to the end user. Also add a
GET `/providers` endpoint which returns the list of providers as
`inspect/providers` does today.
This API follows CRUD and is more intuitive/RESTful.
This work is part of the RFC at
#1359
sensitive fields are redacted using `redact_sensetive_fields` on the
server side before returning a response:
<img width="456" alt="Screenshot 2025-03-13 at 4 40 21 PM"
src="https://github.com/user-attachments/assets/9465c221-2a26-42f8-a08a-6ac4a9fecce8"
/>
## Test Plan
using llamastack/llama-stack-client-python#181 a
user is able to to run the following:
`llama stack build --template ollama --image-type venv`
`llama stack run --image-type venv
~/.llama/distributions/ollama/ollama-run.yaml`
`llama-stack-client providers inspect ollama`
<img width="378" alt="Screenshot 2025-03-13 at 4 39 35 PM"
src="https://github.com/user-attachments/assets/8273d05d-8bc3-44c6-9e4b-ef95e48d5466"
/>
also, was able to run the new test_list integration test locally with
ollama:
<img width="1509" alt="Screenshot 2025-03-13 at 11 03 40 AM"
src="https://github.com/user-attachments/assets/9b9db166-f02f-45b0-86a4-306d85149bc8"
/>
Signed-off-by: Charlie Doern <cdoern@redhat.com>
cdoern
added a commit
to cdoern/llama-stack
that referenced
this pull request
Apr 8, 2025
add `v1/providers/` which uses PUT to allow users to change their provider configuration this is a follow up to llamastack#1429 and related to llamastack#1359 a user can call something like: `llama_stack_client.providers.update(api="inference", provider_id="ollama", provider_type="remote::ollama", config={'url': 'http:/localhost:12345'})` or `llama-stack-client providers update inference ollama remote::ollama "{'url': 'http://localhost:12345'}"` this API works by adding a `RequestMiddleware` to the server which checks requests, and if the user is using PUT /v1/providers, the routes are re-registered with the re-initialized provider configurations/methods for the client, `self.impls` is updated to hold the proper methods+configurations this depends on a client PR, the CI will fail until then but succeeded locally Signed-off-by: Charlie Doern <cdoern@redhat.com>
|
This pull request has been automatically marked as stale because it has not had activity within 60 days. It will be automatically closed if no further activity occurs within 30 days. |
|
This pull request has been automatically closed due to inactivity. Please feel free to reopen if you intend to continue working on it! |
cdoern
added a commit
to cdoern/llama-stack
that referenced
this pull request
Jun 27, 2025
add `v1/providers/` which uses PUT to allow users to change their provider configuration this is a follow up to llamastack#1429 and related to llamastack#1359 a user can call something like: `llama_stack_client.providers.update(api="inference", provider_id="ollama", provider_type="remote::ollama", config={'url': 'http:/localhost:12345'})` or `llama-stack-client providers update inference ollama remote::ollama "{'url': 'http://localhost:12345'}"` this API works by adding a `RequestMiddleware` to the server which checks requests, and if the user is using PUT /v1/providers, the routes are re-registered with the re-initialized provider configurations/methods for the client, `self.impls` is updated to hold the proper methods+configurations this depends on a client PR, the CI will fail until then but succeeded locally Signed-off-by: Charlie Doern <cdoern@redhat.com>
cdoern
added a commit
to cdoern/llama-stack
that referenced
this pull request
Jun 30, 2025
add `v1/providers/` which uses PUT to allow users to change their provider configuration this is a follow up to llamastack#1429 and related to llamastack#1359 a user can call something like: `llama_stack_client.providers.update(api="inference", provider_id="ollama", provider_type="remote::ollama", config={'url': 'http:/localhost:12345'})` or `llama-stack-client providers update inference ollama remote::ollama "{'url': 'http://localhost:12345'}"` this API works by adding a `RequestMiddleware` to the server which checks requests, and if the user is using PUT /v1/providers, the routes are re-registered with the re-initialized provider configurations/methods for the client, `self.impls` is updated to hold the proper methods+configurations this depends on a client PR, the CI will fail until then but succeeded locally Signed-off-by: Charlie Doern <cdoern@redhat.com>
cdoern
added a commit
to cdoern/llama-stack
that referenced
this pull request
Jun 30, 2025
add `v1/providers/` which uses PUT to allow users to change their provider configuration this is a follow up to llamastack#1429 and related to llamastack#1359 a user can call something like: `llama_stack_client.providers.update(api="inference", provider_id="ollama", provider_type="remote::ollama", config={'url': 'http:/localhost:12345'})` or `llama-stack-client providers update inference ollama remote::ollama "{'url': 'http://localhost:12345'}"` this API works by adding a `RequestMiddleware` to the server which checks requests, and if the user is using PUT /v1/providers, the routes are re-registered with the re-initialized provider configurations/methods for the client, `self.impls` is updated to hold the proper methods+configurations this depends on a client PR, the CI will fail until then but succeeded locally Signed-off-by: Charlie Doern <cdoern@redhat.com>
cdoern
added a commit
to cdoern/llama-stack
that referenced
this pull request
Jul 1, 2025
add `v1/providers/` which uses PUT to allow users to change their provider configuration this is a follow up to llamastack#1429 and related to llamastack#1359 a user can call something like: `llama_stack_client.providers.update(api="inference", provider_id="ollama", provider_type="remote::ollama", config={'url': 'http:/localhost:12345'})` or `llama-stack-client providers update inference ollama remote::ollama "{'url': 'http://localhost:12345'}"` this API works by adding a `RequestMiddleware` to the server which checks requests, and if the user is using PUT /v1/providers, the routes are re-registered with the re-initialized provider configurations/methods for the client, `self.impls` is updated to hold the proper methods+configurations this depends on a client PR, the CI will fail until then but succeeded locally Signed-off-by: Charlie Doern <cdoern@redhat.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Introduce an RFC for a Provider Configuration API, allowing for provider configuration changes of an existing stack to aid in common runtime changes like model swapping, endpoint switching, etc
Closes #993