openai.BadRequestError when Agent tries to structure an output

[lukekh]

I am getting this error from an Agent trying to structure an output using a tool call

openai.BadRequestError: Error code: 400 - {'error': {'message': "An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. The following tool_call_ids did not have response messages: call_GYxJmEaeT1ZbLYzo21Cz2jT8", 'type': 'invalid_request_error', 'param': 'messages.[4].role', 'code': None}}

Here's the full stacktrace (with some censoring of file paths)

Traceback (most recent call last):
  File "<...>/venv/bin/<...>", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "<...>/venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<...>/venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "<...>/venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<...>/venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<...>/venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<...>/venv/lib/python3.11/site-packages/<...>/ui/cli.py", line 28, in init
    explore_and_cache()
  File "<...>/venv/lib/python3.11/site-packages/<...>/explore/system.py", line 317, in explore_and_cache
    tree = exploration(root, ignore_path, default_ignore_git, default_ignore_<...>) # FOR TESTING: tree = load_cached_exploration()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<...>/venv/lib/python3.11/site-packages/<...>/explore/system.py", line 77, in exploration
    return asyncio.run(recursive_exploration(root, folder, ignore))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/work/.pyenv/versions/3.11.4/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/Users/work/.pyenv/versions/3.11.4/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/work/.pyenv/versions/3.11.4/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "<...>/venv/lib/python3.11/site-packages/<...>/explore/system.py", line 100, in recursive_exploration
    await recursive_exploration(root, d, ignore)
  File "<...>/venv/lib/python3.11/site-packages/<...>/explore/system.py", line 100, in recursive_exploration
    await recursive_exploration(root, d, ignore)
  File "<...>/venv/lib/python3.11/site-packages/<...>/explore/system.py", line 100, in recursive_exploration
    await recursive_exploration(root, d, ignore)
  File "<...>/venv/lib/python3.11/site-packages/<...>/explore/system.py", line 109, in recursive_exploration
    file = await vague_file.specify()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<...>/venv/lib/python3.11/site-packages/<...>/schema.py", line 78, in specify
    return await Code.generate(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<...>/venv/lib/python3.11/site-packages/<...>/schema.py", line 264, in generate
    gen_fields = await cls._generate_fields(context)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<...>/venv/lib/python3.11/site-packages/<...>/schema.py", line 254, in _generate_fields
    result = await CODE_AGENT.run(contents)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<...>/venv/lib/python3.11/site-packages/pydantic_ai/agent.py", line 251, in run
    model_response, request_cost = await agent_model.request(messages, model_settings)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<...>/venv/lib/python3.11/site-packages/pydantic_ai/models/openai.py", line 151, in request
    response = await self._completions_create(messages, False, model_settings)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<...>/venv/lib/python3.11/site-packages/pydantic_ai/models/openai.py", line 189, in _completions_create
    return await self.client.chat.completions.create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<...>/venv/lib/python3.11/site-packages/openai/resources/chat/completions.py", line 1720, in create
    return await self._post(
           ^^^^^^^^^^^^^^^^^
  File "<...>/venv/lib/python3.11/site-packages/openai/_base_client.py", line 1843, in post
    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<...>/venv/lib/python3.11/site-packages/openai/_base_client.py", line 1537, in request
    return await self._request(
           ^^^^^^^^^^^^^^^^^^^^
  File "<...>/venv/lib/python3.11/site-packages/openai/_base_client.py", line 1638, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': "An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. The following tool_call_ids did not have response messages: call_GYxJmEaeT1ZbLYzo21Cz2jT8", 'type': 'invalid_request_error', 'param': 'messages.[4].role', 'code': None}}

What's funny is that looking in logfire, I don't see any data with call_GYxJmEaeT1ZbLYzo21Cz2jT8 in the posted request ...

This leads me to believe that some funny business is going on when this function is being executed.

[sydney-runkle]

Hmph, definitely looks like a bug. Thanks for the report.

We'll look into this 👍

[samuelcolvin]

@lukekh please can you share an MRE so we can investigate.