[Fleet] Create new service account elastic/fleet-server-remote for remote agent connections

[hop-dev]

Description

Background
As part of elastic/kibana#104986 we will be adding the ability for Elastic Agents to send agent monitoring data to a remote Elasticsearch cluster. In order to achieve this, we will need a service token for the remote cluster which fleet server will use.

This service account will need less permissions than the elastic/fleet-server as integration data will not be sent to remote clusters yet.

Service Account Details

Name: elastic/fleet-server-remote

Permissions:

• for indices logs-*, metrics-* privileges "write", "create_index", "auto_configure"

[elasticmachine]

Pinging @elastic/es-security (Team:Security)

[bytebilly]

Hi, a couple of questions:

the ability for Elastic Agents to send agent monitoring data to a remote Elasticsearch cluster

Is the Agent sending data, or the Fleet Server? Is there any compatibility constraint?

This service account will need less permissions than the elastic/fleet-server as integration data will not be sent to remote clusters yet.

Does it mean that it will do in the future?

[hop-dev]

Hi thanks for the quick reply!

Is the Agent sending data, or the Fleet Server? Is there any compatibility constraint?

Both, the fleet server is just an agent with the fleet server integration installed. When you say compatibility constraint do you mean between elastic agent versions and elasticsearch version? Not that I am aware of. (CC @ph )

Does it mean that it will do in the future?

I think that is the intention one day but I don't know the timeframe. Is it possible to expand the permissions of a service account in a future release?

[bytebilly]

When you say compatibility constraint do you mean between elastic agent versions and elasticsearch version?

Yes, since they "belong" to different deployments. Let's assume that Fleet is a newer version, sending data to an older Elasticsearch cluster: could this cause any problem, for example if the permission set or schema changed between the two?

I think that is the intention one day but I don't know the timeframe. Is it possible to expand the permissions of a service account in a future release?

Yes it is, permissions can be redefined in newer versions.
I was wondering because, if we already know that we'll need this service account to be exactly the same as the fleet-server one in a reasonable timeframe, I would discuss if we really need to create a new one to limit permissions only in a few versions. I love the least privilege principle, but I also really love having the minimum number of service accounts 🙂

So the question is: which is the right tradeoff between security and simplicity? Do we really need a new service account?

[hop-dev]

Thanks for this input, it's really useful. My answer to both questions is: let me talk to some people and get back to you on that :)

[joshdover]

Yes it is, permissions can be redefined in newer versions. I was wondering because, if we already know that we'll need this service account to be exactly the same as the fleet-server one in a reasonable timeframe, I would discuss if we really need to create a new one to limit permissions only in a few versions. I love the least privilege principle, but I also really love having the minimum number of service accounts 🙂

So the question is: which is the right tradeoff between security and simplicity? Do we really need a new service account?

I don't think we'll ever want elastic/fleet-server-remote to have the same permissions, specifically, we'd never need any access to any of the .fleet* system indices from the remote cluster.

A user who has manage_service_accounts can create service account tokens for either account, so presumably they have access to the .fleet* system indices anyways. The main issue that this would help us protect against is a vulnerability in Kibana or Fleet Server that exposes the service account token for the remote cluster that we'll storing in the .kibana (encrypted) or .fleet-policies (not currently encrypted) indices for the remote cluster.

In today's architecture, an RCE in Kibana would enable an attacker to read this token and use it to gain control of Agents being managed by the remote cluster. If we used a less privileged token, a vulnerability would only allow an attacker to ingest data into the integration data streams, but not the Fleet system indices.

In the future, we intend for Kibana to not have any read access to Fleet-managed secrets, only write access. This would eliminate the potential for an RCE in Kibana to expose this token at all. A Fleet Server vulnerability could still expose it, however Fleet Server does have a much smaller API surface to defend.

In general though, I think avoiding situations where this sensitive token could be used to gain access to control Agents on any cluster is valuable. Agents on the remote monitoring cluster are likely to be running on the same hosts as Elasticsearch of the main cluster which could be leveraged to further escalate privilege (say, by finding a way to reset the elastic password).

[hop-dev]

Thanks for the input Josh. Also regarding @bytebilly's point here:

Let's assume that Fleet is a newer version, sending data to an older Elasticsearch cluster: could this cause any problem, for example if the permission set or schema changed between the two?

Discussed this with @joshdover. If we were to introduce integration data in the future, we would stipulate that the remote cluster must have a minimum version for this to work, so we would be happy to proceed with a minimum set of privileges required for agent data for now.

[nimarezainia]

fyi @blakerouse

I believe the original intent was to limit the permissions to the new remote cluster.

[ywangd]

@hop-dev

Is the Agent sending data, or the Fleet Server? Is there any compatibility constraint?

Both, the fleet server is just an agent with the fleet server integration installed.

Does that mean an agent (without fleet server integration) installed on an end-user's environment can also send data directly to Elasticsearch using the service token as the credential? Service accounts were designed to be used by services that can have "a few" instances. It was not intended to be used by a large number of agents (at least that's my understanding) for which API keys are more suitable. Could you please clarify the intended usage of this new service account on this aspect?

[blakerouse]

@ywangd The elastic/fleet-server-remote service token would still only be used by Fleet Servers it would not be handed down to each Elastic Agent. The elastic/fleet-server-remote token would be used by Fleet Server to create API keys for that cluster to send those down to each Elastic Agent.

The idea of the elastic/fleet-server-remote service token was that we can remove access to the .fleet-* system indexes because the Fleet Server does not need to use those in the remote elasticsearch case, as the Fleet Server is using its main elasticsearch for that.

[juliaElastic]

Hi @elastic/es-security team, we are picking this work up now, and would like to proceed with the creation of elastic/fleet-server-remote service account.
We have a new question related to serverless: since the ES service token API will not be available to users in serverless, we need an alternative to allow users setting up a service token for remote ES.
One solution to this can be to let users create a service token with kibana_system user through a kibana API call. Any concerns with this approach? What would be the reasonable privileges to require users to call this API?
cc @joshdover

[tvernum]

Why do users need to create a service token?

In the prior discussion we said that fleet-server would use the service token, why do we need an API for users to create the token?

The recommended model on Serverless would be for the controller to create a token and configure it in the fleet server, there shouldn't be any need to create a token via an API.

[juliaElastic]

We plan to ask users to create the service token because this is a remote cluster configuration: cluster 1 doesn't have access to create a service token in cluster 2 without user intervention.
So if the users create the service token in cluster 2, they can configure it in cluster 1 to enable the remote ES connection.

[tvernum]

I don't think that's a viable model for Serverless, can we discuss offline?

[juliaElastic]

As discussed, we are going to defer on the serverless implementation now and focus on the stateful support.
So we would proceed with the addition of elastic/fleet-server-remote service account in stateful deployments (cloud and self-managed) unless there are any concerns.