Support benchmarking an unreleased locally built APM Server

[marclop]

Description

We should document or automate (if it proves easy enough) how to package and run a locally built version of APM Server with some changes in ESS / ECE.

[simitt]

@marclop could you summarize the road bumps you hit here please.

[marclop]

My approach for this issue has been to reuse and extend some of the code that we use to build the APM Server for the system tests and re-use the same packaging to produce a Docker Image that would be able to run in ESS or ECE.

For that purpose I introduced a new buildapm binary (link) that builds a docker image from the fleet-server service as specified in our docker-compose.yml (docker.elastic.co/beats/elastic-agent).

I also introduced a new -cloud flag in that binary which instead of using the upstream elastic-agent standard image, uses the docker image that would be used in ESS (docker.elastic.co/cloud-release/elastic-agent-cloud).

As documented in the Observability-dev documentation PR, the packaging should work and it's a fairly easy experience to do build, package and build a docker image which is then uploaded to a repository which is allowed to run in ESS / ECE.

When using buildapm and running the image manually (or just using runapm which uses a very similar code path) to run the image locally, the injected apm-server runs without any problems and testing changes locally is pretty easy. The same is not applicable for the apm-server in the cloud-release image. It seems that something is missing or the packaging doesn't work exactly right. Although the APM Server binary is executed by the Elastic Agent in the built container, some libbeat code (before any of the APM Server code is run) returns with an error:

Exiting: error initializing publisher: missing field accessing 'output.elasticsearch.hosts.0'

The code that returns that error is: https://github.com/elastic/beats/blob/ff1041fc099ec69caea11c5c9c83d6ca59170ee7/libbeat/cmd/instance/beat.go#L387

Which is run before the APM Server's Creator func.

It's unclear to me how to debug this further and make headway since debugging is not straightforward.

[axw]

I haven't gotten to the bottom of why, but commenting out

apm-server/cmd/root.go

Lines 57 to 67 in e960ca3

	{
	Check: func(_ *agentconfig.C) bool {
	return os.Getenv(cloudEnv) != ""
	},
	Config: func() *agentconfig.C {
	return agentconfig.MustNewConfigFrom(map[string]interface{}{
	// default to medium compression on cloud
	"output.elasticsearch.compression_level": 5,
	})
	}(),
	},

allows things to proceed. IIANM we should get rid of that override anyway, since we set the default for modelindexer in

apm-server/elasticsearch/config.go

Line 76 in e960ca3

CompressionLevel: 5,

[marclop]

Verified that the internal docs work while testing the automaxprocs change (#8278). Since @axw removed the problematic code snipped in 8.3.0, I am going to close off this issue.

I am unsure how to best point APM Server developers to the internal docs, since we aren't referencing those anywhere in this repository. I am open to suggestions as to how to best do that. https://github.com/elastic/apm-server/blob/main/TESTING.md#manual-testing asks readers to consult the internal docs, but no references as to how to find these.