Noisy logs in the botocore implementation

[csydvs-whi]

Describe your environment

Python version: Python 3.10 (I would guess 3.10.17, but we're just using the 3.10 tagged docker image)
Package version: opentelemetry-instrumentation-botocore==0.53b1, botocore==1.38.7, boto3==1.38.7

What happened?

I'm getting a bunch of errors with a stacktrace that looks like this:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/opentelemetry/instrumentation/botocore/__init__.py", line 401, in _safe_invoke
    function(*args)
  File "/usr/local/lib/python3.10/site-packages/opentelemetry/instrumentation/botocore/extensions/bedrock.py", line 327, in before_service_call
    for event in message_to_event(message, capture_content):
  File "/usr/local/lib/python3.10/site-packages/opentelemetry/instrumentation/botocore/extensions/bedrock_utils.py", line 456, in message_to_event
    if tool_calls := extract_tool_calls(message, capture_content):
  File "/usr/local/lib/python3.10/site-packages/opentelemetry/instrumentation/botocore/extensions/bedrock_utils.py", line 377, in extract_tool_calls
    tool_uses = [
  File "/usr/local/lib/python3.10/site-packages/opentelemetry/instrumentation/botocore/extensions/bedrock_utils.py", line 378, in <listcomp>
    item for item in content if item.get("type") == "tool_use"
AttributeError: 'str' object has no attribute 'get'

and the message

Error when invoking function 'before_service_call'

As near as I can tell, the problem is just one of noisy logs, I actually still see botocore calls to bedrock in my traces but I'm seeing an error log like this basically every time I call Bedrock.

Steps to Reproduce

I have two calls to Bedrock that it could be, not sure if it's one or both, but I call both bedrock.invoke_model and bedrock_runtime.apply_guardrail. I've been doing this for a while without issues, but I think this error appeared when I updated a bunch of my libraries.

Expected Result

Just not the errors. Again, not sure what they're causing besides noise, my app still runs fine and my traces look reasonable as well.

Actual Result

The noisy log

Additional context

No response

Would you like to implement a fix?

No

[xrmx]

@csydvs-whi thanks for reporting, what model are you using with invoke_model? I'll try to reproduce

[csydvs-whi]

@xrmx I'm guessing it's us.anthropic.claude-3-5-haiku-20241022-v1:0, but anthropic.claude-3-haiku-20240307-v1:0 isn't outside the realm of possibility either.

[xrmx]

Since we are testing with claude already maybe it's that we are not limiting our tracing to only the expected operations.

[SrdjanLL]

I've tried to reproduce this issue without any luck for both us.anthropic.claude-3-5-haiku-20241022-v1:0 and anthropic.claude-3-haiku-20240307-v1:0. I've tried the following:

• invoke_model
• apply_guardrail
• invoke_model with specifying the guardrail
• invoke_model with tool calling (simple tool mock in my own codebase).

The happy path scenarios of all of the above seem to work and get instrumented based on the level of instrumentation implemented (note here for guardrails that they're currently not supported for genai instrumentation with that work pending - #3210).

In order to try and make a reproducer, would you be able to share how you're using invoke_model APIs (i.e if you use tools, how do you pass them, how do you manage the conversation turns, etc.)? It would be great to have a code snippet with the mentioned details.

I'm writing this with the assumption that the quick and easy fix would be to add a dict type check that would fix the noise, but won't help us understand the root cause and the instrumentation code could miss capturing potentially useful information just because it's in the format we're not currently expecting it in.

[csydvs-whi]

Hey @SrdjanLL, thanks for your effort on this. I spent some time trying to figure out what's happening, so the way I'm calling is with the body being something like this:

           json.dumps({
                "anthropic_version": "bedrock-2023-05-31",
                'messages': [{"role": "user", "content": "Placeholder text, but I'm giving them some sources, a question, and a json schema and asking them to respond using that json schema"}, {"role": "assistant", "content": "{"}],
                "system": 'system prompt goes here',
                "max_tokens": 4096,
                'temperature': 0.0,
            })

And then, when I get down into extract_tool_calls, the content is just the string {, so I'm guessing it has something to do with the format of my messages call. I can try and spin up a repo later if that's not enough, but maybe this will give enough to go on?

[SrdjanLL]

Thanks for sharing, that got me a reproducer which should help come up with a bug fix.