Elasticsearch bulk request enhancements #5811
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
urso
force-pushed
the
enh/elasticsearch-document-id
branch
2 times, most recently
from
eecce76 to
5bcc5c3
Compare
ruflin
reviewed
Dec 12, 2017
| meta := createEventBulkMeta(index, pipeline, event) | ||
| meta, err := createEventBulkMeta(index, pipeline, event) | ||
| if err != nil { | ||
| logp.Err("Failed to encode event meta dat: %s", err) |
| Pipeline string `json:"pipeline" struct:"pipeline"` | ||
| var id interface{} | ||
| if m := event.Meta; m != nil { | ||
| id = m["id"] |
There was a problem hiding this comment.
Similar comment to other PR: Should we really allow here interface or make sure this is converted to a string? Do we know the key "id" exists in m ?
There was a problem hiding this comment.
can _id be a number instead of a string?
There was a problem hiding this comment.
I'm assuming that _id on the ES side is a keyword. So if 12 is put in it ends up also as "12". Even if that is not the case, I think we should enforce string values on beats side.
|
@urso Needs a rebase. |
added 6 commits
January 5, 2018 12:01
- Log error and drop event if no index name can be computed (ES API returns error if index is empty) - Set `_id` if event.Meta["id"] is set - Use `index` action if no `id` is set and `create` if `id` is set.
urso
force-pushed
the
enh/elasticsearch-document-id
branch
from
933da88 to
93aaea6
Compare
houndci-bot
reviewed
Jan 5, 2018
libbeat/beat/event.go
Outdated
| @@ -22,6 +22,13 @@ var ( | |||
| errNoTimestamp = errors.New("value is no timestamp") | |||
| ) | |||
|
|
|||
| func (e *Event) SetID(id string) { | |||
There was a problem hiding this comment.
exported method Event.SetID should have comment or be unexported
|
@ruflin rebased and ensure |
ruflin
approved these changes
Jan 9, 2018
urso
pushed a commit
to urso/beats
that referenced
this pull request
Sep 18, 2019
Add support to configure a key for setting the document ID in the harvester JSON settings. The ID will be store in the events Meta["id"] for the output to pick up. With elastic#5811 will the elasticsearch output is the Meta["id"] field to set the document its ID (uses op_type="create" to count duplicate inserts of same ID). For other output type, the document ID will be forwarded via `@metadata.id`.
urso
pushed a commit
that referenced
this pull request
Sep 19, 2019
* Add support to set the document ID in the filebeat json reader Add support to configure a key for setting the document ID in the harvester JSON settings. The ID will be store in the events Meta["id"] for the output to pick up. With #5811 will the elasticsearch output is the Meta["id"] field to set the document its ID (uses op_type="create" to count duplicate inserts of same ID). For other output type, the document ID will be forwarded via `@metadata.id`.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Requires: #5810
returns error if index is empty)
_idif event.Meta["id"] is setindexaction if noidis set andcreateifidis set.