[Change Proposal] Add support for specifying cluster privileges

[stuartnelson3]

This proposal is a continuation of work that was initially begun with #203.

In that issue it was decided to add elasticsearch.privileges.indices, but not specifically cluster privileges.

As was recently surfaced in elastic/elastic-agent#145, access to specific cluster privileges are required for apm-server's stack monitoring ui to correctly while while being managed by elastic-agent.

The specific problem is that the cluster:monitor/main privilege is required.

I could use some additional insight from others on the specifics of how to achieve this. Assuming this is approved, I have a few questions:

• Where would this should be added to package spec? The current implementation seems datastream-focused, while this would be specifically so that the apm-server is able to request data from elasticsearch's root path.
• Would this require additional changes to kibana? Again, the linked code from the previous issue makes me believe it is data-stream focused

However it ends up being structured, I think the original proposal's core looks like a good start:

elasticsearch:
  privileges:
    cluster: ['monitor/main']

[jen-huang]

Where would this should be added to package spec? The current implementation seems datastream-focused, while this would be specifically so that the apm-server is able to request data from elasticsearch's root path.

There is an in-flight spec proposal that, among other things, adds spec for a top-level elasticsearch directory. I think this new directory should have a corresponding manifest.spec.yml file that defines the elasticsearch.privileges.cluster setting structure. @mtojek does that make sense to you?

Would this require additional changes to kibana? Again, the linked code from the previous issue makes me believe it is data-stream focused

Yes, we will need to update Fleet's code to read from the new [package directory]/elasticsearch/manfiest.yml file and add the cluster privileges to the generated agent policy, i.e.:

output_permissions:
  default:
    apm-1:
+     cluster:
+       - monitor/main
      indices:
        - names:
            - metrics-apm.app.*-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-apm.error-default
          privileges:
            - auto_configure
            - create_doc

I believe no changes are required on Fleet Server or Elastic Agent because we already send cluster privileges for agent monitoring permissions. @hop-dev, cc'ing you for awareness as you originally implemented the Fleet changes for the data stream privileges (elastic/kibana#112397). Will probably need your help in implementing this change if the proposal is approved :)

[mtojek]

We have a similar section in data stream manifests:

package-spec/versions/1/data_stream/manifest.spec.yml

Line 164 in 1e17043

elasticsearch:

Maybe it would be good enough to keep it in the package manifest?

cc @ruflin

[stuartnelson3]

@jen-huang assuming this goes through, would you have bandwidth to get the necessary changes in on the kibana side by FF on oct 19?

[ruflin]

Package manifest sounds like a good place for me as it seems it does not tie into a specific data stream.

[jen-huang]

@jen-huang assuming this goes through, would you have bandwidth to get the necessary changes in on the kibana side by FF on oct 19?

It will be tight but yes, I believe so. I'll go ahead and open the Kibana issue now, assuming that structure under package manifest. Actually, I will wait to confirm in elastic/elastic-agent#145.

[stuartnelson3]

I'm going to go ahead and call this "accepted". I'll create implementation issues and then move forward from there.

[jsoriano]

Closing this as it seems it was already implemented in #226. Please reopen if there was anything pending to do.