Experimental self-profiling

[axw]

This PR adds experimental support for continuously profiling the server itself, and recording the data in Elasticsearch.

By default the feature is disabled, and must be explicitly enabled through the config file. There is a new /intake/v2/profile endpoint, which is enabled when CPU or heap profiling is enabled.

We emit an event per profile sample, roughly as described in the proposal document. Some of the field names have changed. Each document includes a hash of the function names in the call stack. We use https://github.com/OneOfOne/xxhash for hashing, which is both faster and of higher quality than the FNV1a algorithm used in the prototype.

TODO:

• complete and merge the Go agent branch
• disable the profile endpoint by default
• define response format for profile endpoint
• generate a unique ID for each profile, so we can obtain samples for a specific profile
• add tests

[codecov-io]

Codecov Report

Merging #2839 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #2839   +/-   ##
=======================================
  Coverage   78.47%   78.47%           
=======================================
  Files          89       89           
  Lines        4711     4711           
=======================================
  Hits         3697     3697           
  Misses       1014     1014

[simitt]

Would you mind to create issues for the TODO comments and link to them?

I am missing some integration tests. We usually have some integration test covering the incoming payload being processed and transformed. You can have a look at some integration tests where we capture the event before calling the libbeat publishing.
We also usually add a test to the system_tests, checking that the actual ingestion works as expected (ES template + ingested event fit together, ..)

Do the agent devs have a possibility to get the specs for how the payload has to look like? Usually they pull the json spec and test against it, so they don't have to actually run the APM Server in their integration tests.

[axw]

Would you mind to create issues for the TODO comments and link to them?

#2954
#2955

I am missing some integration tests. We usually have some integration test covering the incoming payload being processed and transformed. You can have a look at some integration tests where we capture the event before calling the libbeat publishing.
We also usually add a test to the system_tests, checking that the actual ingestion works as expected (ES template + ingested event fit together, ..)

👍 Looking into this now.

Do the agent devs have a possibility to get the specs for how the payload has to look like? Usually they pull the json spec and test against it, so they don't have to actually run the APM Server in their integration tests.

Yes, it's protobuf encoded: https://github.com/google/pprof/blob/master/proto/profile.proto. If/when we open this feature up to applications as well as the server, then this will be better documented.

[axw]

@simitt I've added integration tests. I'll try and have a look at system tests next week.

[simitt]

Shouldn't the id of a profile be unique, and the timestamp usually differ per profile? Is it possible to extract 2 or 3 unique profiles per example?

[axw]

A profile is just a collection of samples, and each document corresponds to one sample. I added the profile.id field so that we could group all samples by the profile that they came from.

I've since removed this field for a few reasons:

• It's not needed right now
• We can potentially use the @timestamp field instead (all samples from a profile will have the same timestamp value)
• The field won't be very meaningful if we start performing additional aggregation of samples in the server

[axw]

Jenkins run the tests please

[axw]

We also usually add a test to the system_tests, checking that the actual ingestion works as expected (ES template + ingested event fit together, ..)

I added a couple of system tests that show that with appropriate config, the server will profile its own CPU and heap usage.

While I was doing that I found that there was an issue :) I'd missed adding "profile" as an event type to the idxmgmt package.

[simitt]

Thanks a bunch for bringing this to the server!