Bedrock Tools returning documents fail with ValidationException

[wdavew]

Initial Checks

• I confirm that I'm using the latest version of Pydantic AI
• I confirm that I searched for my issue in https://github.com/pydantic/pydantic-ai/issues before opening this issue

Description

When a tool returns BinaryContent with document types (PDF, etc.), Bedrock throws a ValidationException:

The model returned the following errors: messages.2: `tool_use` ids were found without 
`tool_result` blocks immediately after: tooluse_XXX. Each `tool_use` block must have a 
corresponding `tool_result` block in the next message.

This is in contrast to returning ImageUrl, which seems to work fine.

In my testing, I found that this works after modifying models/bedrock.py to merge multimodal content responses inside the toolResult block instead of spreading it into a subsequent user message.

Before:

[
  {
    "role": "user",
    "content": [
      {
        "toolResult": {
          "toolUseId": "tooluse_XXX",
          "content": [{"text": "See file ABC"}],
          "status": "success"
        }
      }
    ]
  },
  {
    "role": "user",
    "content": [
      {"text": "This is file ABC:"},
      {"document": {"name": "Document 1", "format": "pdf", "source": {"bytes": "..."}}}
    ]
  }
]

After:

{
  "role": "user",
  "content": [
    {
      "toolResult": {
        "toolUseId": "tooluse_XXX",
        "content": [
          {
            "text": "See file ABC"
          },
          {
            "text": "This is file ABC:"
          },
          {
            "document": {
              "name": "Document 1",
              "format": "pdf",
              "source": {
                "bytes": "..."
              }
            }
          }
        ],
        "status": "success"
      }
    }
  ]
}

I did not find any documentation in AWS Bedrock stating that tool call returns must be formatted this way.
However, including the contents inside the toolReturn part seems better aligned to the Bedrock API.

Example Code

import base64
from pydantic_ai import Agent
from pydantic_ai.models.bedrock import BedrockConverseModel
from pydantic_ai.messages import BinaryContent

SIMPLE_PDF = b"%PDF-1.4\n1 0 obj<</Pages 2 0 R/Type/Catalog>>endobj 2 0 obj<</Count 1/Kids[3 0 R]/Type/Pages>>endobj 3 0 obj<</MediaBox[0 0 612 792]/Parent 2 0 R/Type/Page>>endobj\nxref\n0 4\n0000000000 65535 f\n0000000009 00000 n\n0000000058 00000 n\n0000000111 00000 n\ntrailer<</Root 1 0 R/Size 4>>\nstartxref\n174\n%%EOF"
SIMPLE_IMAGE = base64.b64decode("iVBORw0KGgoAAAANSUhEUgAAAAgAAAAIAQMAAAD+wSzIAAAABlBMVEX///+/v7+jQ3Y5AAAADklEQVQI12P4AIX8EAgALgAD/aNpbtEAAAAASUVORK5CYII=")

model = BedrockConverseModel('us.anthropic.claude-sonnet-4-5-20250929-v1:0')
agent = Agent(model)


@agent.tool_plain
def get_document() -> BinaryContent:
    return BinaryContent(data=SIMPLE_PDF, media_type='application/pdf')

@agent.tool_plain
def get_image() -> BinaryContent:
    return BinaryContent(data=SIMPLE_IMAGE, media_type='image/png')

if __name__ == '__main__':

    # This works
    result = agent.run_sync('Call get_image and tell me what you see')
    print(result.output)

    # This fails
    result = agent.run_sync('Call get_document and tell me what the document contains')
    print(result.output)

Python, Pydantic AI & LLM client version

1.5.0

[DouweM]

@wdavew Thanks for the report. I didn't realize that Bedrock supports files directly inside tool results. OpenAI does not, hence the workaround to include the files as user parts following the tool result. But I'm seeing now that Anthropic and Google (now?) support multi-modal tool responses as well, so it may be time to reevaluate this and make OpenAI the only one that gets the user-parts-following-tool-return behavior.

Note that I do not get this error when using BedrockConverseModel('us.amazon.nova-pro-v1:0'), so it looks like it's Anthropic-specific.

I also think the Bedrock error is incorrect here. It says:

The model returned the following errors: messages.2: `tool_use` ids were found without `tool_result` blocks immediately after: tooluse_bvAnQEa1QpSOhcoD0B_mDA. Each `tool_use` block must have a corresponding `tool_result` block in the next message.

While the tool_result block is definitely immediately after the tool_use block:

[
    {'content': [{'text': 'Call get_document and tell me what the document contains'}], 'role': 'user'},
    {
        'content': [{'toolUse': {'input': {}, 'name': 'get_document', 'toolUseId': 'tooluse_bvAnQEa1QpSOhcoD0B_mDA'}}],
        'role': 'assistant',
    },
    {
        'content': [
            {
                'toolResult': {
                    'content': [{'text': 'See file 588c4d'}],
                    'status': 'success',
                    'toolUseId': 'tooluse_bvAnQEa1QpSOhcoD0B_mDA',
                }
            },
            {'text': 'This is file 588c4d:'},
            {
                'document': {
                    'format': 'pdf',
                    'name': 'Document 1',
                    'source': {
                        'bytes': b'%PDF-1.4\n1 0 obj<</Pages 2 0'
                        b' R/Type/Catalog>>endobj 2 0 '
                        b'obj<</Count 1/Kids[3 0 R]/Ty'
                        b'pe/Pages>>endobj 3 0 obj<</M'
                        b'ediaBox[0 0 612 792]/Parent '
                        b'2 0 R/Type/Page>>endobj\nxref'
                        b'\n0 4\n0000000000 65535 f\n'
                        b'0000000009 00000 n\n000000005'
                        b'8 00000 n\n0000000111 00000 n'
                        b'\ntrailer<</Root 1 0 R/Size 4'
                        b'>>\nstartxref\n174\n%%E'
                        b'OF'
                    },
                }
            },
        ],
        'role': 'user',
    },
]

(And the message structure when returning an image is identical, which also suggests something's going wrong with the mapping on their side)

[DouweM]

@wdavew I have a few other things to finish this month, but I aim to look at this in November.

In the meantime, could you share your bedrock.py modification in case it benefits anyone else?

[wdavew]

Of course, this is the modification I tested in _map_messages in bedrock.py to merge the subsequent user message into the tool call result. Note to those reading -- this is just an experimental patch to verify the issue. I have not otherwise thoroughly tested this.

    async def _map_messages(  # noqa: C901
        self, messages: list[ModelMessage]
    ) -> tuple[list[SystemContentBlockTypeDef], list[MessageUnionTypeDef]]:
        """Maps a `pydantic_ai.Message` to the Bedrock `MessageUnionTypeDef`.

        Groups consecutive ToolReturnPart objects into a single user message as required by Bedrock Claude/Nova models.
        """
        profile = BedrockModelProfile.from_profile(self.profile)
        system_prompt: list[SystemContentBlockTypeDef] = []
        bedrock_messages: list[MessageUnionTypeDef] = []
        document_count: Iterator[int] = count(1)
        for message in messages:
            if isinstance(message, ModelRequest):
                parts_list = list(message.parts)
                skip_next = False

                for i, part in enumerate(parts_list):
                    if skip_next:
                        skip_next = False
                        continue

                    if isinstance(part, SystemPromptPart) and part.content:
                        system_prompt.append({'text': part.content})
                    elif isinstance(part, ToolReturnPart):
                        assert part.tool_call_id is not None

                        tool_result_content: list[Any] = [
                            {'text': part.model_response_str()}
                            if profile.bedrock_tool_result_format == 'text'
                            else {'json': part.model_response_object()}
                        ]

                        # If next part is UserPromptPart, merge its content into toolResult.
                        # and skip the next user message
                        if i + 1 < len(parts_list) and isinstance(parts_list[i + 1], UserPromptPart):
                            next_part = parts_list[i + 1]
                            user_messages = await self._map_user_prompt(next_part, document_count)
                            tool_result_content.extend(user_messages[0]['content'])
                            skip_next = True

                        bedrock_messages.append({
                            'role': 'user',
                            'content': [{
                                'toolResult': {
                                    'toolUseId': part.tool_call_id,
                                    'content': tool_result_content,
                                    'status': 'success',
                                }
                            }],
                        })
                    elif isinstance(part, UserPromptPart):
                        bedrock_messages.extend(await self._map_user_prompt(part, document_count))

Regarding your other points, I agree this seems like a quirk in Bedrock. It had me very confused for some time given that the tool call and return parts are in fact correct. I also had only encountered this with Anthropic models in Bedrock (no issue with the Anthropic provider).

Thanks for your attention on it. It sounds like there's a broader change you're planning anyway to address this. If however you do end up getting busy with other things, I'm happy to clean up what I have for bedrock at least and submit a PR.

[DouweM]

@wdavew The problem with merging user parts into tool return parts is that if there are 2 or more parallel tool calls that both generate multi-modal content, the ModelRequest will end up looking like (pseudocode) [ToolReturn(1, "see file A"), ToolReturn(2, "see file B"), "file A:", file_a, "file B:", file_b].

The correct result would be [ToolReturn(1, ["see file A", "file A", file_a]), ToolReturn(2, ["see file B", "file B", file_b]) or [ToolReturn(1, file_a), ToolReturn(2, file_b), while naive code could end up with all user parts on the second tool return, and none on the first, as there's nothing on the user parts to indicate which tool result they belong to.

If the tool return is "See file X" and there's a "File X:" message followed by a file, that can be used to reconstruct the original tool return, but that won't work with ToolReturn.content: https://ai.pydantic.dev/tools-advanced/#advanced-tool-returns.

So ideally, we would not split the multi-modal return values out into user parts at all, keeping them on ToolReturnPart.content, and leaving it up to the model class implementation to either move them into user parts (which I think OpenAI and similar APIs still require) or send them to the API directly in the tool return part (as Bedrock, Anthropic, and Google all support). We'd have to think about backward compatibility with message histories stored in DBs, and we'd need to keep supporting ToolReturn.content (https://ai.pydantic.dev/tools-advanced/#advanced-tool-returns) with the current behavior.

Another option would be to store a reference back to the tool call (i.e. the tool call ID) on the multi-modal user parts, so that the merging can be made smarter, but that seems like doubling down on a now-unnecessary workaround, and it'd be cleaner from a data model perspective to just keep the multi-modal content on ToolReturnPart.content and deal with pulling out only when a specific model API requires it.

So I'd be open to a PR, but it would be a bit more involved than the merging logic you're using currently.

[wdavew]

@DouweM Yes that makes sense, my dumb example above breaks down with parallel tool calls.

For a real fix, I think it would be best to just put the multi-modal content into ToolReturnPart.content as that's more the rule than the exception now, but I understand there's further consideration required around stored message history deserialization. I don't think I'll have time to get to it before November, so I'm happy to wait until you have time to take care of it. Thanks!