apmpackage: fix cluster privilege

[axw]

Motivation/summary

The Fleet system tests are failing with this error message in the apm-server logs:

{"log.level":"error","@timestamp":"2021-10-20T10:09:10.395Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/output.go","file.line":154},"message":"Failed to connect to backoff(elasticsearch(http://elasticsearch:9200)): 400 Bad Request: {\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"unknown cluster privilege [monitor/main]. a privilege must be either one of the predefined cluster privilege names [manage_own_api_key,none,cancel_task,delegate_pki,grant_api_key,manage_autoscaling,manage_enrich,manage_index_templates,manage_logstash_pipelines,manage_oidc,manage_saml,manage_service_account,manage_token,monitor_ml,monitor_rollup,monitor_snapshot,monitor_text_structure,monitor_watcher,read_ccr,read_ilm,read_pipeline,read_slm,transport_client,create_snapshot,manage_ccr,manage_ilm,manage_ml,manage_rollup,manage_slm,manage_watcher,monitor_data_frame_transforms,monitor_transform,manage_api_key,manage_ingest_pipelines,manage_pipeline,manage_data_frame_transforms,manage_transform,manage_security,monitor,manage,all] or a pattern over one of the available cluster actions\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"unknown cluster privilege [monitor/main]. a privilege must be either one of the predefined cluster privilege names [manage_own_api_key,none,cancel_task,delegate_pki,grant_api_key,manage_autoscaling,manage_enrich,manage_index_templates,manage_logstash_pipelines,manage_oidc,manage_saml,manage_service_account,manage_token,monitor_ml,monitor_rollup,monitor_snapshot,monitor_text_structure,monitor_watcher,read_ccr,read_ilm,read_pipeline,read_slm,transport_client,create_snapshot,manage_ccr,manage_ilm,manage_ml,manage_rollup,manage_slm,manage_watcher,monitor_data_frame_transforms,monitor_transform,manage_api_key,manage_ingest_pipelines,manage_pipeline,manage_data_frame_transforms,manage_transform,manage_security,monitor,manage,all] or a pattern over one of the available cluster actions\"},\"status\":400}","service.name":"apm-server","ecs.version":"1.6.0"}

Per elastic/elastic-agent#145, the privilege we need is cluster:monitor/main, not monitor/main.

Checklist

- [ ] Update CHANGELOG.asciidoc
- [ ] Documentation has been updated

How to test these changes

Cherry-pick #6395, check that the Fleet system tests pass.

Related issues

None

[apmmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-10-20T10:45:00.726+0000

• Duration: 40 min 3 sec

• Commit: 7d0ee42

Test stats 🧪

Test	Results
Failed	0
Passed	6293
Skipped	18
Total	6311

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /hey-apm : Run the hey-apm benchmark.

• /package : Generate and publish the docker images.

[marclop]

I ran the Fleet system tests and they all passed:

$ go test -v -run \.\*Fleet .
2021/11/04 17:12:37 INFO: starting stack containers...
apm-server_elasticsearch_1 is up-to-date
apm-server_package-registry_1 is up-to-date
apm-server_kibana_1 is up-to-date
apm-server_fleet-server_1 is up-to-date
2021/11/04 17:12:37 INFO: setting up fleet...
2021/11/04 17:12:38 INFO: running system tests...
=== RUN   TestFleetIntegration
2021/11/04 17:12:58 Building image elastic-agent-systemtest:8.1.0-2fdb91c4-SNAPSHOT...
2021/11/04 17:12:58 Building apm-server...
2021/11/04 17:12:59 Built /Users/marclop/repos/elastic/apm-server/build/apm-server-linux
2021/11/04 17:13:02 Built image elastic-agent-systemtest:8.1.0-2fdb91c4-SNAPSHOT
2021/11/04 17:13:02 Starting container id: 345ddb8e55be image: quay.io/testcontainers/ryuk:0.2.3
2021/11/04 17:13:02 Waiting for container id 345ddb8e55be image: quay.io/testcontainers/ryuk:0.2.3
2021/11/04 17:13:02 Container is ready id: 345ddb8e55be image: quay.io/testcontainers/ryuk:0.2.3
2021/11/04 17:13:02 Starting container id: a4b8555932af image: elastic-agent-systemtest:8.1.0-2fdb91c4-SNAPSHOT
2021/11/04 17:13:03 Waiting for container id a4b8555932af image: elastic-agent-systemtest:8.1.0-2fdb91c4-SNAPSHOT
2021/11/04 17:13:09 Container is ready id: a4b8555932af image: elastic-agent-systemtest:8.1.0-2fdb91c4-SNAPSHOT
--- PASS: TestFleetIntegration (38.63s)
=== RUN   TestFleetIntegrationAnonymousAuth
2021/11/04 17:13:26 Building image elastic-agent-systemtest:8.1.0-2fdb91c4-SNAPSHOT...
2021/11/04 17:13:29 Built image elastic-agent-systemtest:8.1.0-2fdb91c4-SNAPSHOT
2021/11/04 17:13:29 Starting container id: 5aa69172ac33 image: elastic-agent-systemtest:8.1.0-2fdb91c4-SNAPSHOT
2021/11/04 17:13:30 Waiting for container id 5aa69172ac33 image: elastic-agent-systemtest:8.1.0-2fdb91c4-SNAPSHOT
2021/11/04 17:13:35 Container is ready id: 5aa69172ac33 image: elastic-agent-systemtest:8.1.0-2fdb91c4-SNAPSHOT
--- PASS: TestFleetIntegrationAnonymousAuth (23.06s)
=== RUN   TestFleetPackageNonMultiple
--- PASS: TestFleetPackageNonMultiple (9.14s)
PASS
ok  	github.com/elastic/apm-server/systemtest	72.697s

Using 7.16.0 BC3 in staging, migrated the APM server to managed mode and:

1. Ran this program to generate some data:

package main

import (
	"context"
	"log"
	"time"

	"go.elastic.co/apm"
)

func main() {
	tx := apm.DefaultTracer.StartTransaction("apm-server-7.16.0", "type")
	ctx := apm.ContextWithTransaction(context.Background(), tx)
	span, ctx := apm.StartSpan(ctx, "capture-error", "type")
	span.Duration = time.Second
	span.End()
	tx.Duration = 2 * time.Second
	tx.End()
	apm.DefaultTracer.Flush(nil)
	log.Printf("sent %+v\n", apm.DefaultTracer.Stats())
	log.Println("done")
}

2. Issues the following query and validated that the events are indexed in their corresponding Data Streams:

GET traces-*/_search
{
  "_source": [false], 
  "fields": [
    "transaction.name",
    "span.name"
  ], 
  "query": {
    "bool": {
      "should": [
        {
          "match": {
            "transaction.type": "type"
          }
        },
         {
          "match": {
            "span.type": "type"
          }
        }
      ]
    }
  }
}

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 0.2876821,
    "hits" : [
      {
        "_index" : ".ds-traces-apm-default-2021.11.04-000001",
        "_type" : "_doc",
        "_id" : "G9dE6nwBY0MW94dhbIdS",
        "_score" : 0.2876821,
        "_source" : { },
        "fields" : {
          "span.name" : [
            "capture-error"
          ]
        }
      },
      {
        "_index" : ".ds-traces-apm-default-2021.11.04-000001",
        "_type" : "_doc",
        "_id" : "HNdE6nwBY0MW94dhbIdS",
        "_score" : 0.2876821,
        "_source" : { },
        "fields" : {
          "transaction.name" : [
            "apm-server-7.16.0"
          ]
        }
      }
    ]
  }
}

[mergify]

This pull request is now in conflicts. Could you fix it @axw? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b apmpackage-monitor-main-privilege upstream/apmpackage-monitor-main-privilege
git merge upstream/master
git push upstream apmpackage-monitor-main-privilege