Simplify Beats security docs

[dedemorton]

Closes #10241

Documents the privileges required by Beats 6.7 7.0 users. I've rewritten the content to provide info that is easier to reference, and removed the instructions to create specific roles and users. The idea is to provide users with the ability to create roles that are more suitable for their needs (rather than proscribing specific users and roles).

You can preview the changes here: https://filebeatsecurityupdates.firebaseapp.com/securing-beats.html

Please read and respond to questions tagged for REVIEWERS in the review. I have a lot of questions because the combination of possible features makes things more complicated. This was a very time consuming project considering the output, but I hope that means I've suffered so that our users don't have to anymore. :-)

@LeeDr I've tested the combinations, but would like a second pair of eyes, if you have time. I'm trying to boil things down to essentials, but not sure I am 100% there.

@ph Could you check out the beats CM privileges?

@urso Can we get someone on the security team to take a look at this, too? I feel like our security info has been pretty confusing to users, so I'd like to get this fixed before 6.7, if we can.

@karenzone @bmorelli25 FYI because APM and LS either inherit or reuse the same general style of instructions. Would like your feedback, too, if you have time.

Remaining todo items:

• Test updates against 6.7 then backport.
• Test updates against 7.0
• Make sure to backport correct monitoring settings for 7.1 vs 7.2. (xpack.monitoring.enabled in 7.1 and monitoring.enabled in 7.2

[ph]

@dedemorton add the two roles required.

[urso]

Yeah, this is why we should encourage users to run filebeat setup with another user.

[dedemorton]

I did set up filebeat in advance (with a different user), but the manage_ingest_pipelines privilege was still required. I will retest to see if I can reproduce the behavior in 7.0 as well as 6.7.

[dedemorton]

This is ready for review again. I moved the section about built-in roles because I worked them into the sections about privileges.

@urso I am still setting that manage_pipeline is required even after I have preloaded the ingest pipeline. Is there some way to tell Beats to not try to load the pipeline. The error I'm seeing is:

	ERROR	fileset/factory.go:142	Error loading pipeline: 1 error: Error loading pipeline for fileset system/auth: couldn't load pipeline: couldn't load json. Error: 403 Forbidden: {"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:admin/ingest/pipeline/put] is unauthorized for user [filebeat_publish]"}],"type":"security_exception","reason":"action [cluster:admin/ingest/pipeline/put] is unauthorized for user [filebeat_publish]"},"status":403}. Response body: {"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:admin/ingest/pipeline/put] is unauthorized for user [filebeat_publish]"}],"type":"security_exception","reason":"action [cluster:admin/ingest/pipeline/put] is unauthorized for user [filebeat_publish]"},"status":403}

[dedemorton]

@ycombinator When I remove xpack from this, I no longer see Beats monitoring data in Kibana 7.0.

[ycombinator]

Hi @dedemorton, that would be expected as the new monitoring.* settings were only introduced in 7.1.0: #9260.

Just to avoid confusion, the xpack.monitoring.* settings will continue to be available in 7.1.0 and above. However they will be deprecated in favor of the monitoring.* settings.

[dedemorton]

Ok, I need to backport this to 7.0, so I will use xpack.monitoring in this PR and then change it back after I backport to 7.1.

[dedemorton]

Note that I've updated the docs here to reflect the latest changes: https://filebeatsecurityupdates.firebaseapp.com/securing-beats.html

[kvch]

@dedemorton Filebeat always tries to load pipelines on startup if at least one module is configured. There is no way to skip it right now. So manage_pipelines is indeed required.

[roncohen]

The website is still up and it appeared in a google search for me just now. Can we kill it? Or at least ensure it’s not being indexed, for example using robots.txt?

[dedemorton]

Yup. I’ll remove the project tomorrow