Persist ingestion pipeline modification across updates

[Holi0317]

Currently (v8.3) from the documentation on ingestion pipeline stated that:

It is not currently possible to edit or add pipelines that persist through upgrades.

https://www.elastic.co/guide/en/apm/guide/8.3/ingest-pipelines.html

I am trying to correcting the service.environment for APM events as some my services are not emitting them properly. Currently I am achieving this by defining a pipeline for correcting the environment and edit *apm*${version}* ingestion pipelines to use it after every upgrade. This has become a tedious process.

If the ingestion pipeline defined by APM include something like run pre-logs-apm and post-logs-apm pipelines with ignore failed, I could persist my pipeline customization across update as I (as administrator) can define and hook into the ingestion pipeline safely across APM server updates.

[axw]

@Holi0317 sorry for the slow response.

In 8.4 there is a new feature coming which will introduce the ability to customise ingest pipelines in a persistent manner: elastic/kibana#133740

The gist is that you will be able to create an ingest pipeline called traces-apm@custom, and it will be called at the end of the built-in traces-apm-<version> pipeline. It is not possible to specify pre/post, so if that's something you need I would be interested to hear more about your use case.

I've also been thinking about introducing a similar feature which allow you to define a custom ingest pipeline (e.g. apm@custom) that would be called at the beginning of every APM ingest pipeline (traces, metrics, logs, ...). That way you wouldn't need to edit every ingest pipeline, and the pipeline would be picked up by any new data streams we introduce. I'd be interested to hear if that would be helpful in your case.

[Holi0317]

Thanks for the response. That's a really useful feature and I am looking forward to the 8.4 release 😃

It is not possible to specify pre/post, so if that's something you need I would be interested to hear more about your use case.

The feature mentioned should be sufficient to cover my use case. I don't think adding pre/post would be necessary.

I've also been thinking about introducing a similar feature which allow you to define a custom ingest pipeline (e.g. apm@custom)

That would be a great addition. But I think the apm@custom would be a nice-to-have instead of necessary. The problem I am facing right now is I have to monitor our elastic deployment on the new version deployed and modify the newly-created pipelines after upgrades. *-apm@custom pipelines should be sufficient to cover my use case.

[axw]

@Holi0317 thanks for your feedback!

I'll close this issue since the change is coming: 8.4.0 will be out soon, ~next month or so. Once it's out, please reopen the issue if for some reason your need is not fulfilled.