-
Notifications
You must be signed in to change notification settings - Fork 1.4k
Document performance implications of async vs sync tools #2298
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
PR Change SummarySpecified the requirements for concurrent tool execution in the documentation.
Modified Files
How can I customize these reviews?Check out the Hyperlint AI Reviewer docs for more information on how to customize the review. If you just want to ignore it on this PR, you can add the Note specifically for link checks, we only check the first 30 links in a file and we cache the results for several hours (for instance, if you just added a page, you might experience this). Our recommendation is to add |
|
@Kludex Can you please check this out? You had some related comments in our internal Slack that I think should also be documented. |
|
BTW Alex Hall commented in the public Slack about documenting a bit how to use the baggage to add attributes in the spans of an agent run. I was about to document this also, do you care if I add that bit to this PR? |
|
@GuillermoBlasco I'd prefer to have that in a separate PR. As for the thing I pinged Marcelo for, he's going to be out next week, so I'll quote it here so you can see if you can incorporate it into this PR yourself: [Other person]
Marcelo Trylesinski
[Other person]
Marcelo Trylesinski
[Other person]
Marcelo Trylesinski
|
|
Docs updated with the detail Marcelo provided! |
Co-authored-by: Douwe Maan <me@douwe.me>
|
|
||
| ### Parallel tool calls & concurrency | ||
|
|
||
| When a model returns multiple tool calls in one response, Pydantic AI schedules them concurrently using `asyncio.create_task`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would prefer to not have the "using asyncio.create_task" in this sentence.
That's an implementation detail, which I don't think is correct (and if it is, we might need to change it).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Kludex Can you remove/reword it please?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can @claude do it?
|
|
||
| When a model returns multiple tool calls in one response, Pydantic AI schedules them concurrently using `asyncio.create_task`. | ||
|
|
||
| Async functions are run on the event loop, while sync functions are offloaded to threads. To get the best performance, _always_ use an async function _unless_ you're doing blocking I/O (and there's no way to use a non-blocking library instead) or CPU-bound work (like `numpy` or `scikit-learn` operations), so that simple functions are not offloaded to threads unnecessarily. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is okay.
* Add `priority` `service_tier` to `OpenAIModelSettings` and respect it in `OpenAIResponsesModel` (pydantic#2368) * Add an example of using RunContext to pass data among tools (pydantic#2316) Co-authored-by: Douwe Maan <douwe@pydantic.dev> * Rename gemini-2.5-flash-lite-preview-06-17 to gemini-2.5-flash-lite as it's out of preview (pydantic#2387) * Fix toggleable toolset example so toolset state is not shared across agent runs (pydantic#2396) * Support custom thinking tags specified on the model profile (pydantic#2364) Co-authored-by: jescudero <jescudero@itos.es> Co-authored-by: Douwe Maan <douwe@pydantic.dev> * Add convenience functions to handle AG-UI requests with request-specific deps (pydantic#2397) * docs: add missing optional packages in `install.md` (pydantic#2412) * Include default values in tool arguments JSON schema (pydantic#2418) * Fix "test_download_item_no_content_type test fails on macOS" (pydantic#2404) * Allow string format, pattern and others in OpenAI strict JSON mode (pydantic#2420) * Let more `BaseModel`s use OpenAI strict JSON mode by defaulting to `additionalProperties=False` (pydantic#2419) * BREAKING CHANGE: Change type of 'source' field on EvaluationResult (pydantic#2388) Co-authored-by: Douwe Maan <douwe@pydantic.dev> * Fix ImageUrl, VideoUrl, AudioUrl and DocumentUrl not being serializable (pydantic#2422) * BREAKING CHANGE: Support printing reasons in the console output for pydantic-evals (pydantic#2163) * Document performance implications of async vs sync tools (pydantic#2298) Co-authored-by: Douwe Maan <douwe@pydantic.dev> * Mention that tools become toolset internally (pydantic#2395) Co-authored-by: Douwe Maan <douwe@pydantic.dev> * Fix tests for Logfire>=3.22.0 (pydantic#2346) * tests: speed up the test suite (pydantic#2414) * google: add more information about schema on union (pydantic#2426) * typo in output docs (pydantic#2427) * Deprecate `GeminiModel` in favor of `GoogleModel` (pydantic#2416) * Use `httpx` on `GoogleProvider` (pydantic#2438) * Remove older deprecated models and add new model of Anthropic (pydantic#2435) * Remove `next()` method from `Graph` (pydantic#2440) * BREAKING CHANGE: Remove `data` from `FinalResult` (pydantic#2443) * BREAKING CHANGE: Remove `get_data` and `validate_structured_result` from `StreamedRunResult` (pydantic#2445) * docs: add `griffe_warnings_deprecated` (pydantic#2444) * BREAKING CHANGE: Remove `format_as_xml` module (pydantic#2446) * BREAKING CHANGE: Remove `result_type` parameter and similar from `Agent` (pydantic#2441) * Deprecate `GoogleGLAProvider` and `GoogleVertexProvider` (pydantic#2450) * BREAKING CHANGE: drop 4 months old deprecation warnings (pydantic#2451) * Automatically use OpenAI strict mode for strict-compatible native output types (pydantic#2447) * Make `InlineDefsJsonSchemaTransformer` public (pydantic#2455) * Send `ThinkingPart`s back to Anthropic used through Bedrock (pydantic#2454) * Bump boto3 to support `AWS_BEARER_TOKEN_BEDROCK` API key env var (pydantic#2456) * Add new Heroku models (pydantic#2459) * Add `builtin_tools` to `Agent` (pydantic#2102) Co-authored-by: Marcelo Trylesinski <marcelotryle@gmail.com> Co-authored-by: Douwe Maan <douwe@pydantic.dev> * Bump mcp-run-python (pydantic#2470) * Remove fail_under from top-level coverage config so <100% html-coverage step doesn't end CI run (pydantic#2475) * Add AbstractAgent, WrapperAgent, Agent.event_stream_handler, Toolset.id, Agent.override(tools=...) in preparation for Temporal (pydantic#2458) * Let toolsets be built dynamically based on run context (pydantic#2366) Co-authored-by: Douwe Maan <douwe@pydantic.dev> * Add ToolsetFunc to API docs (fix CI) (pydantic#2486) * tests: change time of evals example (pydantic#2501) * ci: remove html and xml reports (pydantic#2491) * fix: Add gpt-5 models to reasoning model detection for temperature parameter handling (pydantic#2483) Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Co-authored-by: Douwe Maan <DouweM@users.noreply.github.com> Co-authored-by: Marcelo Trylesinski <marcelotryle@gmail.com> * History processor replaces message history (pydantic#2324) Co-authored-by: Marcelo Trylesinski <marcelotryle@gmail.com> * ci: split test suite (pydantic#2436) Co-authored-by: Douwe Maan <douwe@pydantic.dev> * ci: use the right install command (pydantic#2506) * Update config.yaml (pydantic#2514) * Skip testing flaky evals example (pydantic#2518) * Fix error when parsing usage details for video without audio track in Google models (pydantic#2507) * Make OpenAIResponsesModelSettings.openai_builtin_tools work again (pydantic#2520) * Let Agent be run in a Temporal workflow by moving model requests, tool calls, and MCP to Temporal activities (pydantic#2225) * Install only dev in CI (pydantic#2523) * Improve CLAUDE.md (pydantic#2524) * Add best practices regarding to coverage to CLAUDE.md (pydantic#2527) * Add support for `"openai-responses"` model inference string (pydantic#2528) Co-authored-by: Claude <noreply@anthropic.com> * docs: Confident AI (pydantic#2529) * chore: mention what to do with the documentation when deprecating a class (pydantic#2530) * chore: drop hyperlint (pydantic#2531) * ci: improve matrix readability (pydantic#2532) * Add pip to dev deps for PyCharm (pydantic#2533) Co-authored-by: Marcelo Trylesinski <marcelotryle@gmail.com> * Add genai-prices to dev deps and a basic test (pydantic#2537) * Add `--durations=100` to all pytest calls in CI (pydantic#2534) * Cleanup snapshot in test_evaluate_async_logfire (pydantic#2538) * Make some minor tweaks to the temporal docs (pydantic#2522) Co-authored-by: Douwe Maan <douwe@pydantic.dev> * Add new OpenAI GPT-5 models (pydantic#2503) * Fix `FallbackModel` to respect each model's model settings (pydantic#2540) * Add support for OpenAI verbosity parameter in Responses API (pydantic#2493) Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Douwe Maan <douwe@pydantic.dev> * Add `UsageLimits.count_tokens_before_request` using Gemini `count_tokens` API (pydantic#2137) Co-authored-by: Douwe Maan <douwe@pydantic.dev> * chore: Fix uv.lock (pydantic#2546) * Stop calling MCP server `get_tools` ahead of `agent run` span (pydantic#2545) * Disable instrumentation by default in tests (pydantic#2535) Co-authored-by: Marcelo Trylesinski <marcelotryle@gmail.com> * Only wrap necessary parts of type aliases in forward annotations (pydantic#2548) * Remove anthropic-beta default header set in `AnthropicModel` (pydantic#2544) Co-authored-by: Marcelo Trylesinski <marcelotryle@gmail.com> * docs: Clarify why AG-UI example links are on localhost (pydantic#2549) * chore: Fix path to agent class in CLAUDE.md (pydantic#2550) * Ignore leading whitespace when streaming from Qwen or DeepSeek (pydantic#2554) * Ask model to try again if it produced a response without text or tool calls, only thinking (pydantic#2556) Co-authored-by: Douwe Maan <douwe@pydantic.dev> * chore: Improve Temporal test to check trace as tree instead of list (pydantic#2559) * Fix: Forward max_uses parameter to Anthropic WebSearchTool (pydantic#2561) * Let message history end on ModelResponse and execute pending tool calls (pydantic#2562) * Fix type issues * skip tests requiring API keys * add `google-genai` dependency * add other provider deps * add pragma: no cover for untested logic --------- Co-authored-by: akenar <52220260+akenarsari@users.noreply.github.com> Co-authored-by: Tony Woland <16152581+tonyxwz@users.noreply.github.com> Co-authored-by: Douwe Maan <douwe@pydantic.dev> Co-authored-by: Yi-Chen Lin <103916325+ethan01x@users.noreply.github.com> Co-authored-by: José I. Escudero <joseignacioescudero@gmail.com> Co-authored-by: jescudero <jescudero@itos.es> Co-authored-by: Marcelo Trylesinski <marcelotryle@gmail.com> Co-authored-by: William Easton <bill.easton@elastic.co> Co-authored-by: David Montague <35119617+dmontagu@users.noreply.github.com> Co-authored-by: Guillermo <guillermo@mankind.technology> Co-authored-by: Hamza Farhan <thehamza96@gmail.com> Co-authored-by: Mohamed Amine Zghal <medaminezghal@outlook.com> Co-authored-by: Yinon Ehrlich <Tiksagol@users.noreply.github.com> Co-authored-by: Matthew Brandman <matthb6@gmail.com> Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Co-authored-by: Douwe Maan <DouweM@users.noreply.github.com> Co-authored-by: Alex Enrique <41076109+AlexEnrique@users.noreply.github.com> Co-authored-by: Jerry Yan <jerry@heygen.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Mayank <83648453+spike-spiegel-21@users.noreply.github.com> Co-authored-by: Alex Hall <alex.mojaki@gmail.com> Co-authored-by: Jerry Lin <jerry@reevo.ai> Co-authored-by: Raymond Xu <raymond.y.xu@gmail.com> Co-authored-by: kauabh <56749351+kauabh@users.noreply.github.com> Co-authored-by: Victorien <65306057+Viicos@users.noreply.github.com> Co-authored-by: Ethan Brooks <ethanabrooks@gmail.com> Co-authored-by: eballesteros <44843469+eballesteros@users.noreply.github.com>
Added minor change to documentation so specify the requirements for tools to be concurrently executed.
Slack conversation reference: https://pydanticlogfire.slack.com/archives/C083V7PMHHA/p1753301883710229