Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix an issue with client connection errors raised from initial request #512

Merged
merged 2 commits into from
Aug 22, 2020

Conversation

nolar
Copy link
Owner

@nolar nolar commented Aug 20, 2020

What do these changes do?

Suppress all connection errors from the initial streaming request the same way as all connection errors from the continuous stream itself.

Description

An issue was caused by an aiohttp.client_exceptions.ServerDisconnectedError raised from the initial streaming request:

[2020-05-25 10:44:44,924] kopf.reactor.running [ERROR ] Root task 'watcher of pods' is failed:	
Traceback (most recent call last):	
File "/usr/local/lib/python3.7/dist-packages/kopf/reactor/running.py", line 453, in _root_task_checker	
await coro	
File "/usr/local/lib/python3.7/dist-packages/kopf/reactor/queueing.py", line 109, in watcher	
async for raw_event in stream:	
File "/usr/local/lib/python3.7/dist-packages/kopf/clients/watching.py", line 75, in infinite_watch	
async for raw_event in stream:	
File "/usr/local/lib/python3.7/dist-packages/kopf/clients/watching.py", line 112, in streaming_watch	
async for raw_event in stream:	
File "/usr/local/lib/python3.7/dist-packages/kopf/clients/watching.py", line 146, in continuous_watch	
async for raw_input in stream:	
File "/usr/local/lib/python3.7/dist-packages/kopf/clients/auth.py", line 78, in wrapper	
async for item in fn(*args, **kwargs, context=context):	
File "/usr/local/lib/python3.7/dist-packages/kopf/clients/watching.py", line 215, in watch_objs	
sock_connect=settings.watching.connect_timeout,	
File "/usr/local/lib/python3.7/dist-packages/aiohttp/client.py", line 504, in _request	
await resp.start(conn)	
File "/usr/local/lib/python3.7/dist-packages/aiohttp/client_reqrep.py", line 847, in start	
message, payload = await self._protocol.read() # type: ignore # noqa	
File "/usr/local/lib/python3.7/dist-packages/aiohttp/streams.py", line 591, in read	
await self._waiter	
aiohttp.client_exceptions.ServerDisconnectedError

The issue is not with the stream itself broken, but with the initial streaming handshake broken.

This kind of errors should be treated the same as the streaming error: i.e., by ignoring them and restarting the stream request.

Issues/PRs

Issues: fixes #369

Replaces #368 zalando-incubator/kopf#368

Type of changes

  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Refactoring (non-breaking change which does not alter the behaviour)
  • Mostly documentation and examples (no code changes)
  • Mostly CI/CD automation, contribution experience

Checklist

  • The code addresses only the mentioned problem, and this problem only
  • I think the code is well written
  • Unit tests for the changes exist
  • Documentation reflects the changes
  • If you provide code modification, please add yourself to CONTRIBUTORS.txt

The issue is not with the stream itself broken, but with the initial
streaming handshake broken, e.g. due to `ServerDisconnectedError`.

This kind of errors should be treated the same as the streaming error:
i.e., by ignoring them and restarting the stream request.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Streaming request error caused operator crash
1 participant