Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Beats will not log monitoring bulk index failure #14303

Closed
cwurm opened this issue Oct 29, 2019 · 0 comments · Fixed by #14356
Closed

Beats will not log monitoring bulk index failure #14303

cwurm opened this issue Oct 29, 2019 · 0 comments · Fixed by #14356
Assignees

Comments

@cwurm
Copy link
Contributor

cwurm commented Oct 29, 2019

When the monitoring user does not have the required privileges to write into .monitoring-beats-* but does have other indexing privileges (e.g. to write to auditbeat-*), a Beat will not log any errors about the failed monitoring bulk index requests.

The reason is that while Elasticsearch will return 403 Forbidden when the user has no write privileges at all, it will return 200 for a _bulk request if a user has at least one write privilege - even if it's not for the monitoring index.

Instead, errors are going to be returned in the body, e.g. see the following (I had to add a debug statement into the code, this is not what a Beat actually prints):

2019-10-29T10:33:46.579Z        DEBUG   [elasticsearch] elasticsearch/client.go:835     POST http://localhost:9200/.monitoring-beats-7-2019.10.29/_bulk 200 {"took":0,"errors":true,"items":[{"index":{"_index":".monitoring-beats-7-2019.10.29","_id":null,"status":403,"error":{"type":"security_exception","reason":"action [indices:admin/create] is unauthorized for user [auditbeat]"}}}]} {}

For normal event publishing, Libbeat parses the return body in a function called bulkCollectPublishFails:

// bulkCollectPublishFails checks per item errors returning all events
// to be tried again due to error code returned for that items. If indexing an
// event failed due to some error in the event itself (e.g. does not respect mapping),
// the event will be dropped.
func bulkCollectPublishFails(

But this does not happen for the body of monitoring data bulk requests.

I tried to fix this, but the two logics for sending collected vs. monitoring data are quite different so couldn't do it easily.

/cc @ycombinator - seems you've touched this code most recently

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants