Pickle error when logging aiohttp exceptions in IPC subprocess: CIMultiDictProxy not pickleable

[dana-augment]

Description

When running LiveKit agents with process-based job execution, logging aiohttp exceptions (e.g., from Cartesia, Deepgram, or other aiohttp-based plugins) causes a pickle error because CIMultiDictProxy objects from aiohttp response headers cannot be pickled.

Root Cause

The LogQueueHandler in livekit/agents/ipc/log_queue.py pickles log records to send them from worker subprocesses to the main process. When plugins log exceptions in the extra dict, such as:

# In livekit-plugins-cartesia/tts.py line ~520
except Exception as e:
    logger.error(
        "Cartesia connection error...",
        extra={"cartesia_context_id": cartesia_context_id, "error": e},
    )

The exception object e (when it's an aiohttp.ClientError or subclass) contains request_info.headers which is a multidict.CIMultiDictProxy - an unpickleable type.

Error Message

TypeError: can't pickle multidict._multidict.CIMultiDictProxy objects

Steps to Reproduce

import logging
import pickle
from multidict import CIMultiDict, CIMultiDictProxy
from aiohttp import ClientResponseError

# Simulate aiohttp response headers
headers = CIMultiDict([("content-type", "application/json")])
proxy = CIMultiDictProxy(headers)

# Mock request info (aiohttp stores headers here)
class MockRequestInfo:
    def __init__(self):
        self.headers = proxy
        self.url = "https://api.cartesia.ai"
        self.method = "GET"
        self.real_url = "https://api.cartesia.ai"

# Create an aiohttp exception (as would occur on connection error)
err = ClientResponseError(
    request_info=MockRequestInfo(),
    history=(),
    status=500,
    message="Server Error"
)

# Create a log record with the exception in extra (as plugins do)
record = logging.LogRecord(
    name="livekit.plugins.cartesia",
    level=logging.ERROR,
    pathname="tts.py",
    lineno=520,
    msg="Cartesia connection error",
    args=(),
    exc_info=None
)
record.error = err  # This is what extra={"error": e} does

# This fails - same as LogQueueHandler.emit() trying to pickle
pickle.dumps(record)
# TypeError: can't pickle multidict._multidict.CIMultiDictProxy objects

Affected Code Locations

1. Cartesia TTS (livekit-plugins-cartesia/livekit/plugins/cartesia/tts.py):

   • Line ~493: extra={"cartesia_context_id": cartesia_context_id, "error": data}
   • Line ~520: extra={"cartesia_context_id": cartesia_context_id, "error": e}

2. Potentially other aiohttp-based plugins that log exceptions in extra

3. LogQueueHandler (livekit-agents/livekit/agents/ipc/log_queue.py):

   • The emit() method pickles records but doesn't sanitize extra attributes

Suggested Fix

Option 1: Sanitize in LogQueueHandler (recommended)

Modify LogQueueHandler.emit() to convert non-pickleable objects to strings:

def emit(self, record: logging.LogRecord) -> None:
    try:
        # ... existing code ...
        
        # Sanitize any extra attributes that might not be pickleable
        for key in list(vars(record).keys()):
            if key not in logging.LogRecord.__dict__:
                try:
                    pickle.dumps(getattr(record, key))
                except (TypeError, pickle.PicklingError):
                    setattr(record, key, str(getattr(record, key)))
        
        self._send_q.put_nowait(pickle.dumps(record))
    except Exception:
        self.handleError(record)

Option 2: Fix in plugins

Convert exceptions to strings before logging:

except Exception as e:
    logger.error(
        "Cartesia connection error...",
        extra={"cartesia_context_id": cartesia_context_id, "error": str(e)},
    )

Environment

• livekit-agents: 1.3.6
• livekit-plugins-cartesia: 1.3.6
• Python: 3.11
• multidict: 6.7.0
• aiohttp: 3.x

Impact

This error occurs during Cartesia (or other aiohttp-based service) outages/errors, causing the logging itself to fail. This can mask the original error and make debugging difficult.

[dana-augment]

Note: I see there have been recent related pickle fixes in LogQueueHandler:

• fix traceback print when using LogQueueHandler #4128 - fix traceback print when using LogQueueHandler
• fix record.exc_info is not pickable when using LogQueueHandler #4185 - fix record.exc_info is not pickable when using LogQueueHandler

These fixes address exc_info and stack_info not being pickleable, but the issue with extra attributes containing unpickleable objects (like aiohttp exceptions with CIMultiDictProxy headers) is still present.

The pattern of logging exceptions in extra dicts appears in multiple plugins, so a centralized fix in LogQueueHandler.emit() to sanitize all non-standard record attributes would be more robust than fixing each plugin individually.