Metricbeat errors due to missing admin/aliases/get permission for remote_monitoring_user user

[mark-vieira]

The default metricbeat configuration using the built-in remote_monitoring_user user fails due to a missing permission.

The workaround is to create a separate user with the appropriate permissions but the expectation should be that using the built-in users should "just work". My understand would be that either a) metricbeat is doing something it shouldn't or b) the built-in user should have this permission added.

Connection marked as failed because the onConnect callback failed: failed to check for alias 'metricbeat-7.9.1': (status=403) {"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/aliases/get] is unauthorized for user [remote_monitoring_user]"}],"type":"security_exception","reason":"action [indices:admin/aliases/get] is unauthorized for user [remote_monitoring_user]"},"status":403}: 403 Forbidden: {"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/aliases/get] is unauthorized for user [remote_monitoring_user]"}],"type":"security_exception","reason":"action [indices:admin/aliases/get] is unauthorized for user [remote_monitoring_user]"},"status":403}

[elasticmachine]

Pinging @elastic/es-security (:Security/Authorization)

[tvernum]

@ycombinator Can you confirm what we need to support here?

From previous conversation, it seems to be:

• GET /_alias/metricbeat-*
• HEAD /metricbeat-*

Is that all?
We can manage the PR once we're confident about what needs to be fixed.

[ycombinator]

tl;dr: Yes, those are the only two API calls we are currently missing support for. Further, support for them should be added to the remote_monitoring_agent role (not the remote_monitoring_collector role). For details on how I arrived at this conclusion, see below.

---

I started with a fresh install of Elasticsearch and Metricbeat. Using Wireshark I looked at the HTTP API calls Metricbeat makes to Elasticsearch right after start up (which is when it performs any index management related setup actions). Here's what that looks like:

Looking at the definition of the remote_monitoring_user built-in user, I see that it is assigned the remote_monitoring_agent and remote_monitoring_collector roles:

elasticsearch/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/user/RemoteMonitoringUser.java

Lines 13 to 22 in 5197043

	public class RemoteMonitoringUser extends User {
	public static final String NAME = UsernamesField.REMOTE_MONITORING_NAME;
	public static final String COLLECTION_ROLE_NAME = UsernamesField.REMOTE_MONITORING_COLLECTION_ROLE;
	public static final String INDEXING_ROLE_NAME = UsernamesField.REMOTE_MONITORING_INDEXING_ROLE;
	public RemoteMonitoringUser(boolean enabled) {
	super(NAME, new String[]{ COLLECTION_ROLE_NAME, INDEXING_ROLE_NAME }, null, null, MetadataUtils.DEFAULT_RESERVED_METADATA, enabled);
	}
	}

elasticsearch/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/user/UsernamesField.java

Lines 29 to 31 in 5197043

	public static final String REMOTE_MONITORING_NAME = "remote_monitoring_user";
	public static final String REMOTE_MONITORING_COLLECTION_ROLE = "remote_monitoring_collector";
	public static final String REMOTE_MONITORING_INDEXING_ROLE = "remote_monitoring_agent";

Looking at the definition of these roles...

elasticsearch/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/store/ReservedRolesStore.java

Lines 70 to 99 in 5197043

	.put("remote_monitoring_agent", new RoleDescriptor("remote_monitoring_agent",
	new String[] {
	"manage_index_templates", "manage_ingest_pipelines", "monitor",
	GetLifecycleAction.NAME, PutLifecycleAction.NAME,
	"cluster:monitor/xpack/watcher/watch/get",
	"cluster:admin/xpack/watcher/watch/put",
	"cluster:admin/xpack/watcher/watch/delete"
	},
	new RoleDescriptor.IndicesPrivileges[] {
	RoleDescriptor.IndicesPrivileges.builder().indices(".monitoring-*").privileges("all").build(),
	RoleDescriptor.IndicesPrivileges.builder()
	.indices("metricbeat-*").privileges("index", "create_index").build() },
	null, MetadataUtils.DEFAULT_RESERVED_METADATA))
	.put("remote_monitoring_collector", new RoleDescriptor(
	"remote_monitoring_collector",
	new String[] {
	"monitor"
	},
	new RoleDescriptor.IndicesPrivileges[] {
	RoleDescriptor.IndicesPrivileges.builder()
	.indices("*").privileges("monitor").allowRestrictedIndices(true).build(),
	RoleDescriptor.IndicesPrivileges.builder()
	.indices(".kibana*").privileges("read").build()
	},
	null,
	null,
	null,
	MetadataUtils.DEFAULT_RESERVED_METADATA,
	null
	))

... I see that they have the following privileges already:

• Cluster-level:
  • manage_index_templates, which resolves to:
    • indices:admin/template/*
    • indices:admin/index_template/*,
    • cluster:admin/component_template/*
  • manage_ingest_pipelines, which resolves to:
    • cluster:admin/ingest/pipeline/*
  • monitor, which resolves to:
    • cluster:monitor/*
  • cluster:admin/ilm/get
  • cluster:admin/ilm/put
• Index-level:
  • On all (*) indices:
    • monitor, which resolves to:
      • indices:monitor/*
  • On .monitoring-*:
    • all
  • On metricbeat-*:
    • index, which resolves to:
      • indices:data/write/index*
      • indices:data/write/bulk*
      • indices:data/write/update*
    • create_index, which resolves to:
      • indices:admin/create
      • indices:admin/auto_create
      • indices:admin/data_stream/create
  • On .kibana:
    • read, which resolves to:
      • indices:data/read/*

Comparing the API calls made by Metricbeat to Elasticsearch at start up with the privileges already available to the remote_monitoring_user built-in user, it appears we're missing privileges to support the GET /_alias/metricbeat-* and HEAD /metricbeat-* API calls. These could be satisfied by the indices:admin/aliases/get and indices:admin/get index-level privileges or the broader view_index_metadata index-level privilege.

As noted earlier, the remote_monitoring_user built-in user is assigned the remote_monitoring_agent and remote_monitoring_collector roles. Of these, the former is intended for use by Metricbeat to index monitored data into .monitoring-* and metricbeat-* indices. The latter is intended for Metricbeat to monitor Elasticsearch and gather data from it. As such, the missing privileges should be granted to the former role, not the latter one.