[Framework] Allow bulk operations result logging #2117
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
timgrein
changed the title
… sending (operations can fail)
timgrein
changed the title
vidok
reviewed
Mar 22, 2024
wangch079
reviewed
Mar 22, 2024
connectors/es/sink.py
Outdated
| result = item[action_item].get("result") | ||
| operation_failed = result not in SUCCESSFUL_RESULTS | ||
|
|
||
| if operation_failed: |
There was a problem hiding this comment.
For failed operation, you should check
if "error" in item[action_item]There was a problem hiding this comment.
See comment above. Maybe it makes sense to name the condition non_successful_result/ failed_result?
Co-authored-by: Dmitriy Burlutskiy <dmitrii.burlutckii@elastic.co>
timgrein
changed the title
seanstory
reviewed
Mar 22, 2024
config.yml
Outdated
| @@ -147,6 +147,10 @@ | |||
| #elasticsearch.bulk.retry_interval: 10 | |||
| # | |||
| # | |||
| ## Enable to log ids of created/indexed/deleted/updated documents during a sync. | |||
There was a problem hiding this comment.
help me make sure that this isn't lost with #2280 (depending on which merges first)
There was a problem hiding this comment.
mine merged first. To resolve the conflict, this'll just need to be moved to config.yml.example
Co-authored-by: Sean Story <sean.j.story@gmail.com>
…imgrein/log-deleted-ids
timgrein
added a commit
that referenced
this pull request
Mar 27, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We've an open tech improvement, which states that it's important for some customers for audit purposes to be able to log ids of documents, which were deleted.
This PR introduces one new configuration setting
elasticsearch.bulk.enable_operations_loggingto enable this option. The logs will be logged onDEBUGlevel.Sample logs for successful operations:
Sample log for failed operations:
Checklists
Pre-Review Checklist
v7.13.2,v7.14.0,v8.0.0)- [ ] if you added or changed Rich Configurable Fields for a Native Connector, you made a corresponding PR in KibanaRelease Note
Add configuration settings to be able to log ids of deleted documents.