APM Server monitoring using apm_system role fails

[axw]

Describe the bug

The Monitoring APM Server section in the docs describes how to configure the server to send monitoring data to Elasticsearch, and recommends using the built-in apm_system user/role.

In 7.2 this fails with an error in the logs like this:

Failed to publish events: 403 Forbidden: {"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:data/write/bulk] is unauthorized for user [apm_system]"}],"type":"security_exception","reason":"action [indices:data/write/bulk] is unauthorized for user [apm_system]"},"status":403}

To Reproduce
Steps to reproduce the behavior:

1. Create a 7.2 stack with security enabled. Set a password for the apm_system user
2. Configure apm-server.yml according to https://www.elastic.co/guide/en/apm/server/7.2/monitoring.html, using the password from step 1.
3. Start the APM Server
4. Check the logs, see there's an error

Expected behavior

Monitoring should work using the apm_system user/role.

[axw]

This appears to be a regression in 7.2, relating to changes in the underlying Beats infrastructure.

The apm_system role has the same cluster-level privileges as beats_system, but the latter has some additional index-level privileges. These were added to beats_system in elastic/elasticsearch#40876; apm_system should presumably have been updated at the same time.

[cachedout]

I have opened elastic/elasticsearch#47302 to start work on this issue.

[simitt]

@cachedout can you confirm this issue was fixed with elastic/elasticsearch#47302 (comment)?

[cachedout]

@simitt I am still waiting for reviews on this: elastic/elasticsearch#47668

[cachedout]

Hi again @simitt and @axw.

I have merged elastic/elasticsearch#47668 into master in Elasticsearch.

Backports have been opened to 7.5 and 7.x. Please let me know if you have any concerns. Thanks!

[simitt]

LGTM

[axw]

Backports are all done, so closing this out. Thanks @cachedout!

[simitt]

Tested with apm_system user by enabling xpack security in Elasticsearch and following the steps in the APM Server docs:

monitoring:
  enabled: true
  elasticsearch:
    username: apm_system
    password: somepassword

Kibana Monitoring UI:

Using built in apm_system user works as expected.