Indexing permissions as part of the Elastic Agent policy

[aleksmaus]

What does this PR do?

Implements the Indexing permissions as part of the Elastic Agent policy
#101

1. Adds a new field policy_output_permissions_hash into the Agent document that stores the hash of output_permissions from the policy.
2. Retrieves the policy from .fleet-policies upon the agent enrollment and uses the output_permissions default namespace in order to create the role descriptors for the output API key generation.
3. Checks the output_permissions upon every checking. If the hash doesn't match with the stored in the agent document the new key is generated.

This PR:

1. doesn't invalidate the old API key see the comment for the next iteration [Discuss] Indexing permissions as part of the Elastic Agent policy #101 (comment)
2. is not fully validated end-to-end with kibana for the output_permissions changes, pending on further kibana changes.

Should only be merged after the corresponding kibana changes:
elastic/kibana#94591

Why is it important?

Addresses: #101

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation

Related issues

• Addresses [Discuss] Indexing permissions as part of the Elastic Agent policy #101

Screenshots

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #169 updated

• Start Time: 2021-03-29T13:25:33.463+0000

• Duration: 4 min 11 sec

• Commit: f625155

Test stats 🧪

Test	Results
Failed	0
Passed	66
Skipped	0
Total	66

Trends 🧪

[ruflin]

I hope we can get rid of this bit as it is planned that Kibana will add the index part to the output so we don't need to do the additional processing here.

[aleksmaus]

Yeah, Alejandro reached out about the upcoming change. Might add more things here then.

[blakerouse]

Overall this change looks good. See my inline comment about fetching the policy from elasticsearch.

[blakerouse]

The policy.Monitor keeps the latest revision of each policy in memory. It would be better to fetch the policy from there instead of going to elasticsearch, as fleet-server already has it cached in memory.

[aleksmaus]

Removed the output key generation upon enrollment. Now this is driven by the policy update pushes only, one code path.
Updated to the latest kibana changes, removed output permissions to the role descriptors transformation.

[mergify]

This pull request is now in conflicts. Could you fix it @aleksmaus? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature/indexing_permissions upstream/feature/indexing_permissions
git merge upstream/master
git push upstream feature/indexing_permissions

[ruflin]

@aleksmaus I see this got merged but elastic/kibana#94591 is still open. How is the current behaviour?