self-instrumentation triggers mapping exceptions

[graphaelli]

start apm-server with self-instrumentation enabled

./apm-server -E instrumentation.enabled=true -e -d '*'

wait a minute or less and notice:

{"log.level":"error","@timestamp":"2022-08-09T13:37:42.589-0400","log.logger":"modelindexer","log.origin":{"file.name":"modelindexer/indexer.go","file.line":412},"message":"failed to index event (strict_dynamic_mapping_exception): mapping set to strict, dynamic introduction of [parent] within [process] is not allowed","service.name":"apm-server","trace.id":"58bdee8653d6445c590ad24451d57302","transaction.id":"58bdee8653d6445c","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-09T13:37:42.589-0400","log.logger":"modelindexer","log.origin":{"file.name":"modelindexer/indexer.go","file.line":430},"message":"bulk request completed: 2 indexed, 5 failed (0 exceeded capacity)","service.name":"apm-server","trace.id":"58bdee8653d6445c590ad24451d57302","transaction.id":"58bdee8653d6445c","ecs.version":"1.6.0"}

successes and failures continue to be logged after that.

[axw]

Was this with 8.5.0-SNAPSHOT? Which commit of Kibana are you running? There was a change a couple of days ago to switch from process.ppid to process.parent.pid (#8783). The latest 8.5.0 integration package in snapshot storage is up to date: elastic/package-storage@4c2a9cc

If Kibana is stale, then the bundled integration package would also be stale.

[graphaelli]

My kibana snapshot is older than that package so I suspect you're right and this is already fixed. I'll close this after confirming with a newer kibana build.

[graphaelli]

Relaunched the whole stack and still seeing the same with:

Kibana (image built: 2022-08-09 11:40:11 UTC):
	Version: 8.5.0-SNAPSHOT
	Branch: main
	Build SHA: 6655eeff50554078e3f427828846399066cc29e3
	Build number: 55383

Elastic Agent managed APM Server (image built: 2022-08-10 04:31:22 UTC):
	Binary: 8.5.0-SNAPSHOT (build: dc5b1a2ec19c04e54f663be166e54f892d3c3385 at 2022-08-10 04:25:59 +0000 UTC)

The APM Server displaying the errors is running standalone from:

apm-server version 8.5.0 (amd64), libbeat 8.5.0 [32ace4565111ef7162bebe99a3a4940e4b31c96a built 2022-08-09 04:01:52 +0000 +0000]

How can the integration package version be determined?

[axw]

How can the integration package version be determined?

It's not possible to directly check it, at the moment, as we replace the package version during development. I might look at adding the commit hash in the integration README.

The 8.5.0-SNAPSHOT images are currently bundling the 8.4.0 integration package, so that would explain the issue you're seeing. This will be fixed by elastic/kibana#137759 (comment)

[axw]

The integration package was updated, therefore this should no longer be an issue.