FastAPI instrumentor stops recording exception event starting v0.55b0

[RogerHYang]

Describe your environment

OS: MacOS
Python version: (e.g., Python 3.13)
Package version: (e.g., 0.56b0)

What happened?

Starting 0.55b0, exception event is no longer captured. Screenshot below shows that the span has ended before it gets to record the exception.

Steps to Reproduce

Below is a script to reproduce the issue. By changing the version from 0.55b0 down to 0.54b1, we can see the exception event in the span. But for 0.55b0 onward, the span has no event.

# /// script
# dependencies = [
#   "fastapi",
#   "uvicorn",
#   "opentelemetry-sdk",
#   "opentelemetry-instrumentation-fastapi==0.55b0",
# ]
# ///
from fastapi import FastAPI
import uvicorn
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

tracer_provider = TracerProvider()
tracer_provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(tracer_provider)
tracer = trace.get_tracer(__name__)

app = FastAPI()
FastAPIInstrumentor.instrument_app(app)

@app.get("/")
def root():
    raise RuntimeError()

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000)

Expected Result

exception event should be recorded

Actual Result

empty event list

Additional context

No response

Would you like to implement a fix?

None

[alexmojaki]

cc @adriangb @Kludex , this is caused by #3012

Closely related: #3642

[outergod]

@RogerHYang would you mind checking whether #3664 fixes the issue for you?

[RogerHYang]

@outergod Thanks for the update. I'm still not seeing the event in the span, and the debugger still shows that the span has ended before it reaches the except block as shown below. (This version is installed from the PR branch.)

[outergod]

Thank you @RogerHYang for checking.

I realize this is a conundrum: Either the OTEL FastAPI middleware acts as the outermost middleware, which it does since #3012 and still does with #3664, which enables user defined exception handlers to have access to OTEL tracing (which is certainly desired!), but that also means the exceptions are caught before the exception recording code gets to see them. The other (old) way around, user exception handlers don't have access to the active OTEL context because it's already over, but the OTEL middleware gets to deal with the exceptions.

I think this might only be solvable using an approach with two individual middlewares where one is responsible for enacting the context and root span, and the other one dealing with exception recording, but I don't know how easy that will be.

[outergod]

I just tried the described approach by just slamming in another instance of OpenTelemetryMiddleware as an additional, inner middleware, and it indeed records the exceptions - but will, as excepted, also record requests twice in general. But at least it shows it's a viable approach with a custom middleware.

[alexmojaki]

The span is being ended here:

opentelemetry-python-contrib/instrumentation/opentelemetry-instrumentation-asgi/src/opentelemetry/instrumentation/asgi/__init__.py

Line 982 in 74536f1

server_span.end()

At that point the exception is still being handled by ServerErrorMiddleware. Adding import sys; server_span.record_exception(sys.exc_info()[1]) just before works.

[outergod]

@alexmojaki do you suggest actually adding exception handling code to that place?

[alexmojaki]

yes

[outergod]

I just experimented with the approach and I see a fundamental problem arising from it, even though I'd love to see it work.

We are changing ASGI middleware code, which has an effect on each and every framework somehow based on it, not just FastAPI. In other words, non FastAPI middlewares based on ASGI will encounter double exception recording, so I think this approach is not viable and we need a solution that's limited to FastAPI.

[alexmojaki]

It could be configurable on OpenTelemetryMiddleware, and FastAPI could enable it.

Side note, the original problem of not recording the http status code as 500 also (still) exists on starlette, not just fastapi.

[outergod]

I've added a commit to #3664 that handles it with a custom middleware. While testing the previous approach, I've also found problems with exceptions getting recorded multiple times with no obvious reason.

If Starlette faces the same problem, the middleware solution could be moved up one level.

[outergod]

@RogerHYang could you maybe check whether the latest state of #3664 fixes your use case? I've added tests that show exceptions are recorded properly.