Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Occasional Connection reset by peer by content app #2334

Open
goosemania opened this issue Mar 15, 2022 · 1 comment
Open

Occasional Connection reset by peer by content app #2334

goosemania opened this issue Mar 15, 2022 · 1 comment

Comments

@goosemania
Copy link
Member

Occasionally the pulp-content-app will return a 500 error due to a Connection reset by peer error. Subsequent requests complete successfully.

It has been observed that once this occurs it will occur repeatedly albeit infrequently until the pod is deleted/restarted.
Additionally the error typically occurs in only one of the three pulp-content-app replicas at a time.

[2022-02-24 19:10:18 +0000] [10] [ERROR] Error handling request
Traceback (most recent call last):
  File "/venv/lib64/python3.8/site-packages/aiohttp/web_protocol.py", line 435, in _handle_request
    resp = await request_handler(request)
  File "/venv/lib64/python3.8/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
  File "/venv/lib64/python3.8/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/venv/lib64/python3.8/site-packages/pulpcore/content/authentication.py", line 41, in authenticate
    return await handler(request)
  File "/venv/lib64/python3.8/site-packages/pulpcore/cache/cache.py", line 212, in cached_function
    bk = await self.default_base_key(request, self)
  File "/venv/lib64/python3.8/site-packages/pulpcore/content/handler.py", line 156, in find_base_path_cached
    index_p1 = await cached.exists(base_key=multiplied_base_paths)
  File "/venv/lib64/python3.8/site-packages/pulpcore/cache/cache.py", line 50, in wrapper
    return await func(*args, **kwargs)
  File "/venv/lib64/python3.8/site-packages/pulpcore/cache/cache.py", line 153, in exists
    return await self.redis.exists(*base_key)
  File "/venv/lib64/python3.8/site-packages/aioredis/client.py", line 1061, in execute_command
    conn = self.connection or await pool.get_connection(command_name, **options)
  File "/venv/lib64/python3.8/site-packages/aioredis/connection.py", line 1365, in get_connection
    if await connection.can_read():
  File "/venv/lib64/python3.8/site-packages/aioredis/connection.py", line 848, in can_read
    return await self._parser.can_read(timeout)
  File "/venv/lib64/python3.8/site-packages/aioredis/connection.py", line 362, in can_read
    return self._buffer and bool(await self._buffer.can_read(timeout))
  File "/venv/lib64/python3.8/site-packages/aioredis/connection.py", line 274, in can_read
    return bool(self.length) or await self._read_from_socket(
  File "/venv/lib64/python3.8/site-packages/aioredis/connection.py", line 247, in _read_from_socket
    data = await self._stream.read(self.socket_read_size)
  File "/usr/lib64/python3.8/asyncio/streams.py", line 665, in read
    raise self._exception
  File "/usr/lib64/python3.8/asyncio/selector_events.py", line 848, in _read_ready__data_received
    data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer
2022-02-24 19:10:18,043 INFO gunicorn.access: X.X.X.X [24/Feb/2022:19:10:18 +0000] "GET /api/automation-hub/v3/artifacts/collections/published/ansible-tower-3.8.5.tar.gz?validate_token=539ad80e030608fa60520eb76a9553050727b4a2b95889b7de48ab81553be496:56be102481103ad85a7d0c4ec8b762a86767cdd21e84186e717b8419966014cb HTTP/1.1" 500 244 "-" "python-requests/2.26.0" 

pulpcore==3.17.4
aioredis==2.0.0
AWS Elasticache for Redis: 5.0.5

@goosemania
Copy link
Member Author

ipanova: it can be this issue aio-libs-abandoned/aioredis-py#778

@gerrod3 gerrod3 self-assigned this Mar 31, 2022
@gerrod3 gerrod3 removed their assignment Aug 31, 2022
@dralley dralley removed the Sprint label Jul 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants