[Fleet] Expose permissions as part of the agent policy

[afgomez]

Summary

Closes #94058

Fleet server needs to generate an API key for each agent so it can communicate with elasticsearch. As of today, these API keys have coarse permissions; every agent can write on any logs-*,metrics-*,traces-* index. We want to narrow down the permissions for each agent based on their integrations.

This PR does some groundwork to move forward this goal.

1. Specify how to add permissions to the agent policy,

To do so we need to specify in the agent policy which permissions are needed for that agent. We have added a outputPermissions key to the agent containing this information.

The type signature is as follows:

// Same structure as https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html#security-api-create-api-key-example
type Permission {
  names: string[];
  privileges: string[];
}

interface FullAgentPolicy {
  // ...
  // As the time of writing this PR the only possible output is `elasticsearch`,
  // but that might not always be the case. There might be outputs that don't support
  // permissions at all, so we make the property optional.
  output_Permissions?: {
    // Only `default` for now, since it's the only `output` that we have
	[outputKey: string]: {
      // Each integration will specify their own permissions.
      // This PR uses `fallback` as an "integration" for now.
      [roleForIntegration: string]: {
        indices: Permission[];
      }
    } 
  }
}

2. Move current hardcoded permissions from fleet server to the policy itself.

Since the integrations don't specify their permissions yet, we will use the current broad permissions in the policy, under the fallback key.

Checklist

Delete any items that are not applicable to this PR.

• Unit or functional tests were updated or added to match the most common scenarios

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[ruflin]

I was probably too quick to test this as it is still in draft. I was curious how the full output currently looks and was opening the yaml view but got the following error:

[afgomez]

I was curious how the full output currently looks and was opening the yaml view but got the following error

Yeah sorry about that. The code had a silly error. It's fixed now

[aleksmaus]

#94058 (comment)

[elasticmachine]

Pinging @elastic/fleet (Feature:Fleet)

[aleksmaus]

testing with osquerybeat that ships the data using libbeat, the fallback permissions don't seem to be sufficient

{"log.level":"error","@timestamp":"2021-03-25T10:46:57.811-0400","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/output.go","file.line":154},"message":"Failed to connect to backoff(elasticsearch(http://localhost:9200)): 403 Forbidden: {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [cluster:monitor/main] is unauthorized for API key id [SsuOaXgBlpA2k-27cUU0] of user [elastic], this action is granted by the cluster privileges [monitor,manage,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [cluster:monitor/main] is unauthorized for API key id [SsuOaXgBlpA2k-27cUU0] of user [elastic], this action is granted by the cluster privileges [monitor,manage,all]\"},\"status\":403}","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-03-25T10:46:57.819-0400","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/output.go","file.line":145},"message":"Attempting to reconnect to backoff(elasticsearch(http://localhost:9200)) with 1 reconnect attempt(s)","ecs.version":"1.6.0"} 

the missing "cluster": ["monitor"], probably is the reason.

[ruflin]

I think to get this moving, lets add cluster: ['monitoring'] back in so we can continue with the implementation. But I would like to understand why we ned monitoring cluster permissions.

[scunningham]

@urso Can we get rid of monitor privs? I see at least one case of the /_xpack API being called:

https://github.com/elastic/beats/blob/master/libbeat/monitoring/report/elasticsearch/client.go#L73

We probably do not want the endpoint to have that much information about the output cluster.

[urso]

Beats in xpack (the ones used for agent) will always query Elasticsearch for the right license, but I'm not sure if this API call really requires the monitoring permissions. At least my assumption was 'no'.

Originally monitoring permissions have been required to check if the template, policy, index, or alias do exist. But the agent should create a configuration that suppresses these checks. If monitoring is still required for the license check, we might consider to disable the check via agent as well (and maybe have agent do the check).

[ruflin]

To make sure we don't block this PR, I suggest we readd the monitoring permissions in this PR but follow up with trying to get it removed.

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: c0bab6e
• Storybooks Preview

Metrics [docs]

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
fleet	344.4KB	344.4KB	+21.0B

History

• 💚 Build #115193 succeeded 73043d2
• 💔 Build #115124 failed 3831a8a
• 💔 Build #114893 failed b9180fc
• 💚 Build #113932 succeeded dccd319
• 💔 Build #113708 failed 31381ca

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[nchaulet]

🚀

[ruflin]

LGTM. nice job!

[ruflin]

@nchaulet one to cleanup as soon we will not support 7.9 agents anymore.

[dikshachauhan-qasource]

Hi @jen-huang

If any testing is required to be done on this ticket from our end, could you please let know more details about how to validate this.

Thanks
QAS

[jen-huang]

Hi @dikshachauhan-qasource, no specific for this for now, the risk areas here would be covered by existing test cases.

[dikshachauhan-qasource]

Ok Thanks for the confirmation @jen-huang