ConnectionErrors and ReadTimeouts

[anze3db]

Description

I have installed logfire in my Django app that I have running on my Raspberry Pi. From what I can tell the logs are all showing up correctly in Logfire, but I'm constantly seeing exceptions in Sentry.

Raw stack trace for the exception that's most common:

RemoteDisconnected: Remote end closed connection without response
  File "urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
  File "urllib3/connectionpool.py", line 537, in _make_request
    response = conn.getresponse()
  File "urllib3/connection.py", line 466, in getresponse
    httplib_response = super().getresponse()
  File "http/client.py", line 1419, in getresponse
    response.begin()
  File "http/client.py", line 331, in begin
    version, status, reason = self._read_status()
  File "http/client.py", line 300, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"

ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
  File "requests/adapters.py", line 589, in send
    resp = conn.urlopen(
  File "urllib3/connectionpool.py", line 847, in urlopen
    retries = retries.increment(
  File "urllib3/util/retry.py", line 470, in increment
    raise reraise(type(error), error, _stacktrace)
  File "urllib3/util/util.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
  File "urllib3/connectionpool.py", line 537, in _make_request
    response = conn.getresponse()
  File "urllib3/connection.py", line 466, in getresponse
    httplib_response = super().getresponse()
  File "http/client.py", line 1419, in getresponse
    response.begin()
  File "http/client.py", line 331, in begin
    version, status, reason = self._read_status()
  File "http/client.py", line 300, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"

ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
  File "/home/home/projects/flakytest/.venv/lib/python3.12/site-packages/opentelemetry/sdk/metrics/_internal/export/__init__.py", line 534, in _receive_metrics
    self._exporter.export(
  File "/home/home/projects/flakytest/.venv/lib/python3.12/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 181, in export
    resp = self._export(serialized_data.SerializeToString())
  File "/home/home/projects/flakytest/.venv/lib/python3.12/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 152, in _export
    return self._session.post(
  File "requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "requests/adapters.py", line 604, in send
    raise ConnectionError(err, request=request)

Let me know if I can share anything else to help debug this!

Python, Logfire & OS Versions, related packages (not required)

logfire="0.35.0"
platform="Linux-6.1.21-v8+-aarch64-with-glibc2.31"
python="3.12.1 (main, Dec 21 2023, 15:21:50) [GCC 10.2.1 20210110]"
[related_packages]
requests="2.32.2"
protobuf="4.25.3"
rich="13.7.1"
executing="2.0.1"
opentelemetry-api="1.24.0"
opentelemetry-exporter-otlp-proto-common="1.24.0"
opentelemetry-exporter-otlp-proto-http="1.24.0"
opentelemetry-instrumentation="0.45b0"
opentelemetry-instrumentation-django="0.45b0"
opentelemetry-instrumentation-wsgi="0.45b0"
opentelemetry-proto="1.24.0"
opentelemetry-sdk="1.24.0"
opentelemetry-semantic-conventions="0.45b0"
opentelemetry-util-http="0.45b0"

[alexmojaki]

Thanks for the report. Now and then our backend has problems that we're still working on, but I hadn't seen any signs of such a consistent and high error rate. Has it been like that since the beginning, and is it still like that now?

I've added some more retrying to the SDK in #214 and released, please can you upgrade logfire and see if that helps?

[anze3db]

The error rate has been very consistent since I configured Logfire, but I had to turn it off yesterday because I was running out of issue quota on sentry. I've reenabled now and I'll let you know if it happens again.

PS: I think you need to add structlog as a dependency for logfire. I had to install it manually to get around this exception:

     import logfire
  File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/logfire/__init__.py", line 20, in <module>
    from .integrations.structlog import LogfireProcessor as StructlogProcessor
  File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/logfire/integrations/structlog.py", line 3, in <module>
    from structlog.types import EventDict, WrappedLogger
ModuleNotFoundError: No module named 'structlog'

[alexmojaki]

PS: I think you need to add structlog as a dependency for logfire. I had to install it manually to get around this exception:

woops, that's not meant to be imported eagerly, thanks for pointing it out. I've released a fix for that.

[anze3db]

Great, I've redeployed the new version and everything looks good! I also haven't seen any timeout issues, so I think my issue is resolved so I'm closing it.

Thank you for fixing this so quickly!