[Metricbeat] Initialize processors in manifest.yml with TestData function

[kaiyan-sheng]

Describe the enhancement:
For light weight modules, sometimes we have processors defined in manifest.yml file. But the processors are not initialized when running TestData function in integration test file to generate data.json.

For example: with the rename processor

beats/x-pack/metricbeat/module/aws/s3_daily_storage/manifest.yml

Line 14 in 7d78d64

- rename:

we should see different names for the fields in the data.json file generate by TestData function in s3_daily_storage_integration_test.go:

beats/x-pack/metricbeat/module/aws/s3_daily_storage/_meta/data.json

Line 13 in 7d78d64

"NumberOfObjects": {

Link: #21703 (comment)

cc @jsoriano

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[kaiyan-sheng]

After this is done, I will rerun TestData to generate new data.json files for s3_daily_storage and s3_request metricsets.

[jsoriano]

I have been investigating this issue and I think that the solution is not very straigth-forward.

The thing is that TestData, and mbtest helpers in general, call directly the native Fetch function, what has very little boilerplate.
Processors in light modules are initialized as part of the publisher, that is not instantiated at all in these integration tests.

So to be able to use the mbtest helpers with processors we would need to do one of these things:

• Make the helpers aware of light modules so they can instantiate and execute the processors.
• Refactor light modules to execute the processors as part of the fetch.

First option would be a bit tricky because it wouldn't be testing the real pipelines, but it would serve to generate the data.json files. Second one would be quite an intrusive change, and I think we already discarded this option while implementing processors in light modules.

For testing, there is an alternative, pipelines are executed in system tests, maybe we can use them to generate the data.json too.

I have pushed my tests here: master...jsoriano:lightmodules-processors-test
They include a system test where processors work.
Go tests fail, confirming that processors are not working with mbtest.

As there is a possible workaround with system tests, I am not sure if it worths the effort. In any case I will give a quick try to make the helpers aware of light modules processors.

@kaiyan-sheng do you think that system tests may work in your case?

[kaiyan-sheng]

@jsoriano Thank you for the investigation! Yes if system tests can generate data.json for light modules with processors, that's fine for me!

[jsoriano]

In any case I will give a quick try to make the helpers aware of light modules processors.

Finally I tried this, and it seems to work well, PR open: #22854