Fix libbeat metrics replacement

[axw]

Motivation/summary

Fix a race between apm-server and libbeat code replacing the libbeat.output.events metrics registries. When the output is reloaded, libbeat clears the libbeat registry, undoing any replacement made by apm-server code. Because the libbeat code caches the libbeat metrics registry at init time, we can remove it and replace it with a new libbeat metrics registry under our own control.

Checklist

- [ ] Update CHANGELOG.asciidoc (not yet released)
- [ ] Update package changelog.yml (only if changes to apmpackage have been made)
- [ ] Documentation has been updated

How to test these changes

1. Run apm integration
2. Ingest some events
3. Check libbeat metrics in /stats output, or stack monitoring

Related issues

Closes #7139

[apmmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Reason: null

• Start Time: 2022-01-27T12:09:01.917+0000

• Duration: 63 min 17 sec

• Commit: 4a17630

Test stats 🧪

Test	Results
Failed	0
Passed	5625
Skipped	20
Total	5645

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /hey-apm : Run the hey-apm benchmark.

• /package : Generate and publish the docker images.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[axw]

/test

[axw]

/test

[mergify]

This pull request is now in conflicts. Could you fix it @axw? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b modelindexer-libbeat-metrics upstream/modelindexer-libbeat-metrics
git merge upstream/master
git push upstream modelindexer-libbeat-metrics

[marclop]

@Mergifyio refresh

[mergify]

refresh

✅ Pull request refreshed

[marclop]

It seems that the current diff cannot be back ported as is to 7.17.

[stuartnelson3]

confirmed with RC2:

start apm-integration-test:

./scripts/compose.py start 8.1.0 --bc 024d23aa  --remove-orphans

in the fleet ui:

• create agent policy
• copy the service token + server policy id for enrolling

create an elastic-agent.yml that enables http monitoring:

agent:
  monitoring:
    enabled: true
    use_output: default
    namespace: default
    logs: true
    metrics: true
    http:
      enabled: true
      host: localhost
      port: 6791

Start the elastic-agent using the service token, policy id, and elastic-agent.yml:

docker run --name elastic-agent-local -it \
    --env KIBANA_FLEET_SETUP=1  \
    --env FLEET_SERVER_ENABLE=1 \
    --env ELASTICSEARCH_HOST="http://localhost:9200" \
    --env FLEET_SERVER_SERVICE_TOKEN=$TOKEN \
    --env FLEET_SERVER_POLICY_ID=$POLICY_ID \
    --env FLEET_INSECURE=1 \
    --env FLEET_ENROLL=1  \
    --env KIBANA_FLEET_HOST="http://localhost:5601" \
    --env KIBANA_FLEET_PASSWORD="changeme" \
    --env KIBANA_FLEET_USERNAME="admin"  \
    --env ELASTIC_APM_LOG_LEVEL=debug \
    --env ELASTIC_APM_LOG_FILE=stderr \
    --network host \
    -v $(pwd)/elastic-agent.yml:/usr/share/elastic-agent/elastic-agent.yml \
    --rm docker.elastic.co/beats/elastic-agent:8.1.0

add the apm-integration to the agent policy

confirm the apm-server has been started: curl http://localhost:6791/processes | jq .

start metricbeat with the following modules.d/beat-xpack.yml:

# modules.d/beat-xpack.yml
- module: beat
  xpack.enabled: true
  period: 10s
  hosts: ["http://localhost:6791"]
  basepath: "/processes/apm-server-default"

navigate to the stack monitoring ui

ingest events with the following program:

package main

import (
	"context"
	"fmt"
	"os"
	"time"

	"go.elastic.co/apm"
)

func main() {
	version := "undefined"
	if len(os.Args) > 1 {
		version = os.Args[1]
	}
	name := fmt.Sprintf("apm-server-%s", version)

	for i := 0; i < 1000; i++ {
		tx := apm.DefaultTracer.StartTransaction(name, "type")
		ctx := apm.ContextWithTransaction(context.Background(), tx)
		span, ctx := apm.StartSpan(ctx, name, "type")
		span.Duration = time.Second
		span.End()
		tx.Duration = 2 * time.Second
		tx.End()
		<-time.After(time.Millisecond)
	}
	<-time.After(time.Second)
	apm.DefaultTracer.Flush(nil)
	fmt.Printf("%s: %+v\n", name, apm.DefaultTracer.Stats())
}

verify that events appear in the output events rate graph