-
Notifications
You must be signed in to change notification settings - Fork 1.4k
Add Hugging Face as a provider #1911
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
35 commits
Select commit
Hold shift + click to select a range
2c3b9cb
add hf inference providers support
hanouticelina 537a657
update dependencies
hanouticelina af602a5
nit
hanouticelina 1f3f7a2
update docstring
hanouticelina bea050c
add tests
hanouticelina 40aef2e
add docs and known models for hf
hanouticelina af6fa42
Merge branch 'main' of github.com:hanouticelina/pydantic-ai into hf-i…
hanouticelina 7a4b9a4
fix imports in test
hanouticelina a153081
fix tests
hanouticelina 2f0ec51
fix provider test
hanouticelina 69aee55
adapt cli test
hanouticelina f68dace
re-record vcr cassettes
hanouticelina cc982e5
fix token name
hanouticelina 00da46e
fix examples test
hanouticelina bb20a34
Merge remote-tracking branch 'origin/main' into hf-inference-providers
Kludex 922fd13
Add API docs and refactor a bit the wording
Kludex c779503
Merge branch 'main' of github.com:hanouticelina/pydantic-ai into hf-i…
hanouticelina 873f090
Merge branch 'hf-inference-providers' of github.com:hanouticelina/pyd…
hanouticelina adfc254
review suggestions
hanouticelina e4af59e
more tests
hanouticelina fbead3d
Merge branch 'main' of github.com:hanouticelina/pydantic-ai into hf-i…
hanouticelina cd76d78
fix test
hanouticelina 13ebbf9
fix another test
hanouticelina b96b710
add more vcr tests
hanouticelina 9b0edee
split thinking part
hanouticelina 6da1cf2
fix tests
hanouticelina d546e04
add more context to hugging face models page
burtenshaw 789c261
Update docs/models/huggingface.md
hanouticelina 17b74d2
Merge pull request #1 from burtenshaw/docs/inference-providers-docs
hanouticelina 96a9c96
add more tests
hanouticelina 0c37e41
Merge branch 'hf-inference-providers' of github.com:hanouticelina/pyd…
hanouticelina 60e5c74
coverage
hanouticelina f2a74dd
remove no-cover
hanouticelina eb4af3f
Merge remote-tracking branch 'origin/main' into hf-inference-providers
Kludex 9539909
replace exception handling
Kludex File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| # `pydantic_ai.models.huggingface` | ||
|
|
||
| ## Setup | ||
|
|
||
| For details on how to set up authentication with this model, see [model configuration for Hugging Face](../../models/huggingface.md). | ||
|
|
||
| ::: pydantic_ai.models.huggingface |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,95 @@ | ||
| # Hugging Face | ||
|
|
||
| [Hugging Face](https://huggingface.co/) is an AI platform with all major open source models, datasets, MCPs, and demos. You can use [Inference Providers](https://huggingface.co/docs/inference-providers) to run open source models like DeepSeek R1 on scalable serverless infrastructure. | ||
|
|
||
| ## Install | ||
|
|
||
| To use `HuggingFaceModel`, you need to either install `pydantic-ai`, or install `pydantic-ai-slim` with the `huggingface` optional group: | ||
|
|
||
| ```bash | ||
| pip/uv-add "pydantic-ai-slim[huggingface]" | ||
| ``` | ||
|
|
||
| ## Configuration | ||
|
|
||
| To use [Hugging Face](https://huggingface.co/) inference, you'll need to set up an account which will give you [free tier](https://huggingface.co/docs/inference-providers/pricing) allowance on [Inference Providers](https://huggingface.co/docs/inference-providers). To setup inference, follow these steps: | ||
|
|
||
| 1. Go to [Hugging Face](https://huggingface.co/join) and sign up for an account. | ||
| 2. Create a new access token in [Hugging Face](https://huggingface.co/settings/tokens). | ||
| 3. Set the `HF_TOKEN` environment variable to the token you just created. | ||
|
|
||
| Once you have a Hugging Face access token, you can set it as an environment variable: | ||
|
|
||
| ```bash | ||
| export HF_TOKEN='hf_token' | ||
| ``` | ||
|
|
||
| ## Usage | ||
|
|
||
| You can then use [`HuggingFaceModel`][pydantic_ai.models.huggingface.HuggingFaceModel] by name: | ||
|
|
||
| ```python | ||
| from pydantic_ai import Agent | ||
|
|
||
| agent = Agent('huggingface:Qwen/Qwen3-235B-A22B') | ||
| ... | ||
| ``` | ||
|
|
||
| Or initialise the model directly with just the model name: | ||
|
|
||
| ```python | ||
| from pydantic_ai import Agent | ||
| from pydantic_ai.models.huggingface import HuggingFaceModel | ||
|
|
||
| model = HuggingFaceModel('Qwen/Qwen3-235B-A22B') | ||
| agent = Agent(model) | ||
| ... | ||
| ``` | ||
|
|
||
| By default, the [`HuggingFaceModel`][pydantic_ai.models.huggingface.HuggingFaceModel] uses the | ||
| [`HuggingFaceProvider`][pydantic_ai.providers.huggingface.HuggingFaceProvider] that will select automatically | ||
| the first of the inference providers (Cerebras, Together AI, Cohere..etc) available for the model, sorted by your | ||
| preferred order in https://hf.co/settings/inference-providers. | ||
|
|
||
| ## Configure the provider | ||
|
|
||
| If you want to pass parameters in code to the provider, you can programmatically instantiate the | ||
| [`HuggingFaceProvider`][pydantic_ai.providers.huggingface.HuggingFaceProvider] and pass it to the model: | ||
|
|
||
| ```python | ||
| from pydantic_ai import Agent | ||
| from pydantic_ai.models.huggingface import HuggingFaceModel | ||
| from pydantic_ai.providers.huggingface import HuggingFaceProvider | ||
|
|
||
| model = HuggingFaceModel('Qwen/Qwen3-235B-A22B', provider=HuggingFaceProvider(api_key='hf_token', provider_name='nebius')) | ||
| agent = Agent(model) | ||
| ... | ||
| ``` | ||
|
|
||
| ## Custom Hugging Face client | ||
|
|
||
| [`HuggingFaceProvider`][pydantic_ai.providers.huggingface.HuggingFaceProvider] also accepts a custom | ||
| [`AsyncInferenceClient`][huggingface_hub.AsyncInferenceClient] client via the `hf_client` parameter, so you can customise | ||
| the `headers`, `bill_to` (billing to an HF organization you're a member of), `base_url` etc. as defined in the | ||
| [Hugging Face Hub python library docs](https://huggingface.co/docs/huggingface_hub/package_reference/inference_client). | ||
|
|
||
| ```python | ||
| from huggingface_hub import AsyncInferenceClient | ||
|
|
||
| from pydantic_ai import Agent | ||
| from pydantic_ai.models.huggingface import HuggingFaceModel | ||
| from pydantic_ai.providers.huggingface import HuggingFaceProvider | ||
|
|
||
| client = AsyncInferenceClient( | ||
| bill_to='openai', | ||
| api_key='hf_token', | ||
| provider='fireworks-ai', | ||
| ) | ||
|
|
||
| model = HuggingFaceModel( | ||
| 'Qwen/Qwen3-235B-A22B', | ||
| provider=HuggingFaceProvider(hf_client=client), | ||
| ) | ||
| agent = Agent(model) | ||
| ... | ||
| ``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How can we keep the list of those models up-to-date? Do you folks have an endpoint that we can call to list a lot of them, or something?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes! we've just shipped https://router.huggingface.co/v1/models that returns the list of models (sorted by trending score on the Hugging Face Hub). Not sure how can I update
KnownModelNamedynamically? i can do that for [LatestHuggingFaceModelNames](https://github.com/hanouticelina/pydantic-ai/blob/873f090197b6c7b8c2ae1c0760db17ef54814f86/pydantic_ai_slim/pydantic_ai/models/huggingface.py#L69, butKnownModelNameneeds to be update as well.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can create a test that: if run locally, it would call that endpoint and it will check against this list of models.
In CI it should use the cassette.