Resource grows too big with multiple operators for same resource #372
Comments
kopf-archiver
bot
changed the title
This was referenced Sep 9, 2020
|
The issue is addressed in #539: all Kopf-originating annotations, regardless of the specific operators or their identities, will be ignored now. This required a little injection of Kopf's own identity, so it would be impossible to hide that the operators are Kopf-based, if it was a concern at all. The reproduction is show there too. Tested manually — both reproduced and fixed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Long story short
I'm writing an operator that uses the "multiple operator on same resource" feature in 0.27 but found an issue that I'm not sure whether it's a bug or if I'm doing something wrong.
The annotations used to persist kopf diffbase and progress data end up recursively included, causing the resource object to grow very large.
Description
If there are 2 kopf operators that manage the same resource, with custom persistence settings, they will include the annotations stored by themselves and the other operator in their own snapshot, escaping it while doing so. The result is that annotations end up including the entire history of changes until the resource grows too big and updates start failing with error 422 Unprocessable Entity.
The code snippet to reproduce the issue
Write the following code into op1.py and op2.py
The exact command to reproduce the issue
The full output of the command that failed
Environment
Python packages installed
Also, is there a way to make kopf write state data only in resources that are of actual interest to the operator? For example, I have Pod event handlers that are filtered by label but all pods in the cluster get annotated by kopf.
akojima For the latter question: That's strange. If you use
labels=orannotations=orwhen=or any other filters on the handlers, the unrelated objects should not be touched in any way. If that is the case, it is a bug.And especially, the
@kopf.on.eventhandlers should not cause any annotations stored by Kopf, as they are a low-level "spying" type of handlers.Can you please report this as a separate issue?
For the former (main) question: That behaviour is explainable, and maybe is desired. Here is what happens:
Since both of the operators are unaware of each other, they do not purge the annotations of each other from the snapshots. Other operator's annotations look like just any other annotations, incl. the human-added ones.
The same would be true for the status fields or any other ways of storing the state on the resource itself — as long as they are indeed unaware of each other.
The logic of cleaning up the resource's body is here: https://github.com/nolar/kopf/blob/0.27/kopf/storage/progress.py#L233-L235 and in other
clear()methods in the same module.Another bit of it is here: https://github.com/nolar/kopf/blob/0.27/kopf/storage/diffbase.py#L78-L79
As you can see, either the own prefixes are removed, or
kopf.zalando.org/-prefixed annotations are removed (regardless of own identity). Everything else is considered as essential.To work around this problem you have to make them aware of each other. There are three ways:
kopf.zalando.orgprefix.kopf.AnnotationsProgressStorage, and extending itsclear()method with few more cleanups of "related" operators — now, both of them will be aware of each other.The inclusion of own data (op1's data by op1 operator) is not actually happening. In fact, the op1 includes the op2's data, which includes the op1's data.
On a side-note: this is indeed a weird behaviour. I see how confusing it can be. And I am open to any suggestions to make it smoother.
Perhaps, I can add a tiny marker to the content of all progress-/diffbase-related annotations of all Kopf-based operators, and exclude such annotations regardless of their prefixes.
Technically, the content of the annotations is not promised to be a json. It can be any string in any format/notation/syntax. JSON is only used for debuggability. So, adding any kind of markers seems acceptable.
nolar Thank you for the thorough explanation.
I filed issue #374 for the secondary issue.
As for the main issue, always keeping the default kopf.zalando.org across all operators (+ unique handler names) seems like the simplest workaround.
I also have the impression that sometimes a ping-pong effect is triggered with multiple operators, where one handler notices the other operator adding annotations, then adding data of its own, which is noticed and handled the same way by the other operator. This loops with a flood of messages in --verbose mode, until the patch requests start failing because the resources can't grow any longer.
I switched to a single operator model for my project so this doesn't immediately affect me anymore, but still, anyone using 2 Kopf operators in the same cluster would be affected.
The marker to identify data added by Kopf sounds like a good idea to me.
Another idea would be to use a fixed key pre-prefix/suffix (e.g. myprefix/handler_name-kopf.private instead of myprefix/handler_name).
But generally speaking, what we want is the ability to advertise certain fields as being private to a specific operator instance and thus of no interest to anyone else, regardless of what framework is used to write them. Ideally there would be a standard way of doing that in all of Kubernetes-land, but since there isn't one (as far as I know, I'm new to Kubernetes tho), the only way I can think would be a configurable list of prefixes or suffixes to filter out known private field patterns.
Yes, it is a known issue. Or a known feature. If 2 operators use the same
kopf.zalando.org/last-handled-configurationannotation, they will be triggering each other's reactions — like ping-pong indeed — and noticing the changes happening (the annotation is used as a base value for diff calculation).We had this issue in our apps too. The only proper solution is to have different operator identities (this is why this feature was added). But — back to the original issue — the operators should be somehow aware of each other and/or ignore each other even if unaware.
I'll take it as a priority for version 0.28, together with K8s API throttling (it is quite easy to kill the cluster with all those 422s and ping-pongs going way too fast; see #351).
The text was updated successfully, but these errors were encountered: