[Framework] Improvements to delete netities on real time #571
Conversation
omby8888
changed the title
| { | ||
| "property": "$identifier", | ||
| "operator": "contains", | ||
| "value": entity.identifier, | ||
| }, | ||
| { | ||
| "property": "$blueprint", | ||
| "operator": "contains", | ||
| "value": entity.blueprint, | ||
| }, |
There was a problem hiding this comment.
| { | |
| "property": "$identifier", | |
| "operator": "contains", | |
| "value": entity.identifier, | |
| }, | |
| { | |
| "property": "$blueprint", | |
| "operator": "contains", | |
| "value": entity.blueprint, | |
| }, | |
| { | |
| "property": "$identifier", | |
| "operator": "=", | |
| "value": entity.identifier, | |
| }, | |
| { | |
| "property": "$blueprint", | |
| "operator": "=", | |
| "value": entity.blueprint, | |
| }, |
| @@ -209,6 +195,37 @@ async def _register_in_batches( | |||
| ) | |||
| return entities, errors | |||
|
|
|||
| async def _search_deleted_entities( | |||
There was a problem hiding this comment.
you are not searching deleted entities, you are searching entities to delete
There was a problem hiding this comment.
I dont agree with that either
Nothing in this function says delete
| diffs: list[EntitySelectorDiff] = await asyncio.gather( | ||
| *( | ||
| self._register_resource_raw(resource, results, user_agent_type, True) | ||
| for resource in resource_mappings | ||
| ) | ||
| ) | ||
|
|
There was a problem hiding this comment.
will this be optimized in your other PR? as there could be thousand of entities at once and spawning all can have a huge decrease in performance and take a lot of memory
There was a problem hiding this comment.
Did he change anything?
There was a problem hiding this comment.
i didn't changed anything there, if it causes weak performance i'll handle it in the other PR but there isn't suppose to be any new performance issue from the current PR
| logger.info( | ||
| f"Skipping deletion for {len(entities_to_delete) - len(filtered_entities_to_delete)} entities " | ||
| f"as we couldn't find matching entities that's related to the current integration." | ||
| ) |
There was a problem hiding this comment.
could be very spamming message, as it will happen most of the time probably. maybe just add a log before the deletion, that you are going to perform a delete
CHANGELOG.md
Outdated
|
|
||
| - Delete entities on real time events improvements |
There was a problem hiding this comment.
explain the improvements
| *( | ||
| self._register_resource_raw(resource, results, user_agent_type, True) | ||
| for resource in resource_mappings | ||
| ) | ||
| ) | ||
|
|
||
| registered_entities: list[Entity] = [ | ||
| item for d in diffs for item in d["passed"] |
| *( | ||
| self._register_resource_raw(resource, results, user_agent_type, True) | ||
| for resource in resource_mappings | ||
| ) | ||
| ) | ||
|
|
||
| registered_entities: list[Entity] = [ |
There was a problem hiding this comment.
Use zip to create 2 lists instead of 3 double iteration
CHANGELOG.md
Outdated
| - Delete entities on real time events only if they didn't pass any of the selectors | ||
| - Calculated JQ of only identifier and blueprint to raw entities that didn't pass any selector on real time events |
There was a problem hiding this comment.
| - Delete entities on real time events only if they didn't pass any of the selectors | |
| - Calculated JQ of only identifier and blueprint to raw entities that didn't pass any selector on real time events | |
| - Implemented real-time entity deletion exclusively for instances that haven't matched any selectors. | |
| - Change the JQ calculation to process only identifier and blueprint for raw entities not selected during real-time events to only get the required data for the delete. |
…-time-events-improvements
…-time-events-improvements
Description
What - Improvements to delete entities on real time, denying inefficient cases
Why - previous changes might resulted a case of trying to calculate full jq mapping of entities that wouldn't being registered eventually, and also performing lots of api requests to port api.
How - using multiple rules search api when checking entities to delete and calculating only relevant for deletion jq fields
Type of change
Please leave one option from the following and delete the rest:
Screenshots
Include screenshots from your environment showing how the resources of the integration will look.
API Documentation
Provide links to the API documentation used for this integration.