Restore otel chat message

[chenghao-mou]

Add back otel messages from chat_ctx.

This should close #4116 and the comment

[theomonnom]

The tricky thing is how we're going to handle PII on a lot of different places in the future

[ChenghaoMou]

The tricky thing is how we're going to handle PII on a lot of different places in the future

If we do PII redaction inside the ChatContext, we can keep changes to other places minimal, right?

[theomonnom]

The PII isn't realtime, it's going to be done at the end of the session. (On LiveKit Cloud)
So it's already too late, another version of the chat_ctx is already inside some traces (in the attributes)

[chenghao-mou]

I see. But currently, we are still reporting LLM output, which might also contain PII. Do we intend to:

1. Keep the codebase as is: no input report, but keep output;
2. Or drop output too;

Either way, this would create a discrepancy between Agent Observability and Otel traces for the moment. If we are going to support Otel PII handling eventually, I think we should at least provide consistency for now, especially considering users have to explicitly set up Otel today.

I like how Logfire handles this by:

1. providing a set of predefined regexes for typical sensitive data
2. providing a callback for more control (but the user takes the responsibility for it being slow/complex)
   , similar to how Langfuse handles sensitive data too.

[theomonnom]

agree, consistency for now makes sense