feat(llmobs): langgraph submits agent manifest #13904
Conversation
|
|
Bootstrap import analysisComparison of import times between this PR and base. SummaryThe average import time from this PR is: 279 ± 6 ms. The average import time from base is: 279 ± 6 ms. The import time difference between this PR and base is: -0.3 ± 0.3 ms. The difference is not statistically significant (z = -1.11). Import time breakdownThe following import paths have shrunk:
|
BenchmarksBenchmark execution time: 2025-07-16 19:52:50 Comparing candidate commit 3a30555 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 548 metrics, 2 unstable metrics. |
…langgraph-agentic-model
brettlangdon
left a comment
brettlangdon
left a comment
There was a problem hiding this comment.
apm-core-python file changes lgtm
This PR adds instrumentation for the agent manifest for our LLM Obs agentic integrations (minus LangGraph which will be covered in @sabrenner's [PR](#13904)). I have been using [this doc](https://docs.google.com/document/d/1k-vHYQZfCdckvoH29Mi_TzXuNGBLf3KpypZDpm4vFio/edit?pli=1&tab=t.0) to track what information is available in the Agent Manifest for each integration, but a summary is below: ## Agent Manifest Schema ### Crew AI - Framework - Name - Goal - Backstory - Model - Model Settings - Tools - Handoffs - Code Execution Permissions - Max Iterations <img width="1744" height="918" alt="image" src="https://github.com/user-attachments/assets/d85ac5a7-bbf9-4098-a4a2-93116be6d007" /> ### Open AI Agents - Framework - Name - Instructions - Handoff Description - Model - Model Settings - Tools - Handoffs - Guardrails <img width="1650" height="938" alt="image" src="https://github.com/user-attachments/assets/88f5034c-1846-441d-9495-49ddf7b075ad" /> ### PydanticAI - Framework - Name - Instructions - System Prompts - Model - Model Settings - Tools - Dependencies <img width="548" height="830" alt="image" src="https://github.com/user-attachments/assets/4cbf3466-9de0-475f-808a-43c86e3c45cf" /> ## Next Steps - Make the current Agent Manifest UI work for the new fields that have been added ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --------- Co-authored-by: Kyle Verhoog <kyle@verhoog.ca>
…langgraph-agentic-model
Performance SLOsCandidate: sabrenner/langgraph-agentic-model (91cdcc3) 🔵 No Baseline Data (24 suites)🔵 coreapiscenario - 12/12 (2 unstable)🔵 No baseline data available for this suite
|
ncybul
left a comment
ncybul
left a comment
There was a problem hiding this comment.
looks good so far, had some stylistic comments as well as some things that could be clarified in the agent manifest!
tests/llmobs/llmobs_cassettes/openai/openai_chat_completions_post_b45798ae.yaml
Show resolved
Hide resolved
…langgraph-agentic-model
ncybul
left a comment
ncybul
left a comment
There was a problem hiding this comment.
Left some more questions and comments but overall looks great! Lmk when I can take another look!
tests/llmobs/llmobs_cassettes/openai/openai_chat_completions_post_b45798ae.yaml
Show resolved
Hide resolved
ncybul
left a comment
ncybul
left a comment
There was a problem hiding this comment.
LGTM! Thanks for the thorough clarifications and implementation. All looks good to me :)
Annotates LangGraph agents (compiled state graphs, even those created through `create_react_agent`) with the following agent metadata information: - name (always) - tools (always - best effort, depending on the presence of `ToolNode`s within the agent) - instructions (if done through `create_react_agent`) - model name, provider, & settings (if done through `create_react_agent`) - handoffs (always - assuming the agent is directly a part of a supergraph) - framework (always) - dependencies (since state should always be passed in, always) - max_iterations (always, defaulting to 25 if not provided explicitly) Additionally, to help accomplish this, we patch the `create_react_agent` function to record agent manifest data on agent instance creation, should it be invoked. The means of doing this is a bit strange, as it is in completely different submodule (`langgraph.prebuilt`), and it required me to do what I think are some pretty not normal things for patching 😅 so, happy to discuss/change as needed! MLOB-3102 ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)
Annotates LangGraph agents (compiled state graphs, even those created through
create_react_agent) with the following agent metadata information:ToolNodes within the agent)create_react_agent)create_react_agent)Additionally, to help accomplish this, we patch the
create_react_agentfunction to record agent manifest data on agent instance creation, should it be invoked. The means of doing this is a bit strange, as it is in completely different submodule (langgraph.prebuilt), and it required me to do what I think are some pretty not normal things for patching 😅 so, happy to discuss/change as needed!MLOB-3639
Checklist
Reviewer Checklist