Error on already removed object patch

[dbazhal]

Expected Behavior

When I create and quickly remove object, that's an odd behavior, but I expect that kopf will handle it as non-error.

Actual Behavior

Insetad, I get errors in log and in error tracker:

ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-store', 'Content-Type': 'application/json', 'Date': 'Wed, 10 Jul 2019 10:03:20 GMT', 'Content-Length': '222'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"increments.tutu.ru \"inc-e5c1\" not found","reason":"NotFound","details":{"name":"inc-e5c1","group":"tutu.ru","kind":"increments"},"code":404}


  File "kopf/reactor/queueing.py", line 145, in worker
    await handler(event=event)
  File "kopf/reactor/handling.py", line 139, in custom_object_handler
    await patching.patch_obj(resource=resource, patch=patch, body=body)
  File "kopf/clients/patching.py", line 41, in patch_obj
    await loop.run_in_executor(config.WorkersConfig.get_syn_executor(), functools.partial(patch_func, **request_kwargs))
  File "concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "kubernetes/client/apis/custom_objects_api.py", line 1951, in patch_namespaced_custom_object
    (data) = self.patch_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, body, **kwargs)
  File "kubernetes/client/apis/custom_objects_api.py", line 2057, in patch_namespaced_custom_object_with_http_info
    collection_formats=collection_formats)
  File "kubernetes/client/api_client.py", line 334, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "kubernetes/client/api_client.py", line 168, in __call_api
    _request_timeout=_request_timeout)
  File "kubernetes/client/api_client.py", line 393, in request
    body=body)
  File "kubernetes/client/rest.py", line 286, in PATCH
    body=body)
  File "kubernetes/client/rest.py", line 222, in request
    raise ApiException(http_resp=r)

I'm not sure what behavior will be correct in such situation, but maybe passing such situations will be ok?

except ApiException as e:
    if e.reason != 'NotFound':
        raise

Steps to Reproduce the Problem

1. Empty handler with enough sleep on create
2. Create and remove object faster than sleep

Specifications

kopf==0.17

[nolar]

@dbazhal Thanks for reporting.

Side-notes: it was introduced in #24 — when the finalizers were made optional in case there are no deletion handlers. Now, the k8s-side deletion process does not wait for the operator's "approval" (i.e. removing its finalizers). Kopf should be ready for a sudden disappearance of the individual resources during the handling cycle (especially when the handling is not instant). Though, only when a DELETED event is received on the watch-stream (presumably, it is always the latest event for each resource); otherwise, it should continue treating such cases as regular errors.

[embano1]

Hi, perhaps the issue I'm facing during a delete is related to this one here:

[2019-07-30 21:46:53,387] kopf.reactor.queuein [ERROR   ] functools.partial(<function custom_object_handler at 0x10a0c1a60>, lifecycle=<function asap at 0x10a884510>, registry=<kopf.reactor.registries.GlobalRegistry object at 0x109c2c5c0>, resource=Resource(group='vsphere.vmware.com', version='v1alpha1', plural='vmgroups'), event_queue=<Queue at 0x10bac1550 maxsize=0 _getters[1]>, freeze=<asyncio.locks.Event object at 0x10a0e1400 [unset]>) failed with an exception. Ignoring the event.
Traceback (most recent call last):
  File "/Users/mgasch/GO/src/github.com/embano1/kopf-operator-vmworld/lib/python3.7/site-packages/pykube/http.py", line 241, in raise_for_status
    resp.raise_for_status()
  File "/Users/mgasch/GO/src/github.com/embano1/kopf-operator-vmworld/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://localhost:59982/apis/vsphere.vmware.com/v1alpha1/namespaces/default/vmgroups/kopf-vmworld-example-1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/mgasch/GO/src/github.com/embano1/kopf-operator-vmworld/lib/python3.7/site-packages/kopf/reactor/queueing.py", line 148, in worker
    await handler(event=event)
  File "/Users/mgasch/GO/src/github.com/embano1/kopf-operator-vmworld/lib/python3.7/site-packages/kopf/reactor/handling.py", line 131, in custom_object_handler
    await patching.patch_obj(resource=resource, patch=patch, body=body)
  File "/Users/mgasch/GO/src/github.com/embano1/kopf-operator-vmworld/lib/python3.7/site-packages/kopf/clients/patching.py", line 35, in patch_obj
    await loop.run_in_executor(config.WorkersConfig.get_syn_executor(), obj.patch, patch)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/mgasch/GO/src/github.com/embano1/kopf-operator-vmworld/lib/python3.7/site-packages/pykube/objects.py", line 142, in patch
    self.api.raise_for_status(r)
  File "/Users/mgasch/GO/src/github.com/embano1/kopf-operator-vmworld/lib/python3.7/site-packages/pykube/http.py", line 248, in raise_for_status
    raise HTTPError(resp.status_code, payload["message"])
pykube.exceptions.HTTPError: vmgroups.vsphere.vmware.com "kopf-vmworld-example-1" not found
[2019-07-30 21:46:53,496] kopf.objects         [DEBUG   ] [default/kopf-vmworld-example-1] Invoking handler 'vm_operator'.
deleting vm group "kopf-vmworld-example-1"
[2019-07-30 21:47:02,621] kopf.objects         [INFO    ] [default/kopf-vmworld-example-1] Handler 'vm_operator' succeeded.

Note: I am using @kopf.on.event raw event handling and switching between event types with:

event_type = event['type']
    if event_type == "DELETED":

This should be OK imho...