[Fleet] Allow packages to specify index privileges

[hop-dev]

Summary

Closes #109047

This PR will not be ready to merge until we have tested it end to end.

Allow packages to specify a restricted set of index privileges as part of their manifest. There was previously some stub code which handled permissions however this was never fully implemented, @axw proposed moving these to elasticsearch.privileges.indices.

if a package has elasticsearch.privileges.indices in their data stream definition then:

• remove any invalid or forbidden privileges e.g delete is not allowed
• add these privileges to the package policy saved object
• read these fields from the package policy saved object when generating the agent policy

Testing Steps

First I had to modify a package to specify index privileges in its manifest, for testing i used the custom logs package, these steps won't be necessary once elastic/apm-server#6139 is merged.

• in the integrations repo edit packages/log/data_stream/log/manifest.yml to include the following at the top level:

elasticsearch:
  privileges:
    indices: [auto_configure, create_doc, maintenance, monitor, read]

• in the package directory then run elastic-package build
• run elastic-package stack up -v -d --services package-registry to start the package registry in docker locally
• in kibana.dev.yml set xpack.fleet.registryUrl: http://localhost:8080

We can now test if these privileges are added to the API key when I install the integration (on mac):

• add the custom logs package to a policy
• in dev tools run GET /.fleet-agents/_search
• find an agent that is part of your test policy and copy the default_api_key
• in a terminal run echo -n "<your token>" | base64
• copy that output into the following curl command, note you may need to change the index pattern depending on the test data stream you have used:

curl "http://localhost:9200/_security/user/_has_privileges" \
-H "Connection: keep-alive" \
-H "Content-Type: application/json" \
-H "Authorization: ApiKey <your base64 encoded token>" \
-d '{ "index" : [ { "names": [ "logs-generic-default" ] , "privileges": ["auto_configure", "create_doc", "maintenance", "monitor", "read"] } ] }'

• the response should contain "has_all_requested":true

Checklist

Delete any items that are not applicable to this PR.

• Unit or functional tests were updated or added to match the most common scenarios

[joshdover]

Testing

I haven't found a good way of testing this in fleet yet, I can see I am adding the correct privileges to the agent policy but I am struggling to verify that the key is then created with the correct privileges:

• I dont believe I can use the has_privileges API for an API key
• I cant see how to retrieve the API key to use it manually in a curl request to check its privileges (the keys are created by fleet-server)

I believe you should be able to use the _has_privileges API, at least I was able to with a key I created myself. That said, I don't think you'll be able to retrieve the API key since it is created by Fleet Server as you mentioned.

I think the best course of action is to explore adding a test to the e2e-testing repo once the changes are in for the APM package as well. However, I'm not sure we have examples that use apm-server and test ingesting APM traces, so we would need to add that.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[hop-dev]

Moving out of draft as I have been able to test e2e, I've updated the test steps in the description.

[nchaulet]

I am wondering if we should update the saved object mapping here https://github.com/elastic/kibana/blob/master/x-pack/plugins/fleet/server/saved_objects/index.ts/#L251-L253 (as input is not enabled it probably change nothing)

[hop-dev]

Hi @nchaulet I've added it now. Privileges may include cluster privileges in the future as well, I've set it as flattened as I dont think we would ever need to search on individual privileges?

[axw]

I think the best course of action is to explore adding a test to the e2e-testing repo once the changes are in for the APM package as well. However, I'm not sure we have examples that use apm-server and test ingesting APM traces, so we would need to add that.

FWIW, I intend to add a functional test for this in the apm-server repo. We have tests which run apm-server along with Elasticsearch, Kibana, and fleet-server already (e.g. see https://github.com/elastic/apm-server/blob/master/systemtest/fleet_test.go).

Of course it would be great to have functional testing for Fleet specifically too, but it could perhaps be deferred.

[hop-dev]

@elasticmachine merge upstream

[nchaulet]

🚀

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: fd71710
• Storybooks Preview
• Documentation Changes

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
fleet	1075	1076	+1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
fleet	130.1KB	130.2KB	+163.0B

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/development-plugin-saved-objects.html#_mappings

id	before	after	diff
ingest-package-policies	33	35	+2

Unknown metric groups

API count

id	before	after	diff
fleet	1176	1177	+1

History

• 💚 Build #154179 succeeded c5da9dd
• 💚 Build #153880 succeeded dd8ff25
• 💔 Build #153829 failed 55d9c7d
• 💔 Build #153793 failed 3595370

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @hop-dev

[kibanamachine]

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.