[Metricbeat] CI: update stack to 7.14

[endorama]

What does this PR do?

Update CI docker images to 7.14.0

Why is it important?

7.12.0 (currently in use for testing Metricbeat) is old and unsupported.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Blocks [Metricbeat] gcp: add GKE metricset #26824

Use cases

Screenshots

Logs

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[endorama]

@aspacca was suggesting to update to 7.15 instead, per #27220

As 7.15 is the next minor and to my understanding we still don't have builds to tests against.

We would be testing against nightly, but that's not something we usually do (or am I wrong?)

[aspacca]

#27220 was merged yesterday requiring Kibana 7.15 and ndjson format for dashboards.
I wonder if we should rather bump CI versions to 7.x or similar

[endorama]

@aspacca what if we do this in 2 steps? First let's update to 7.14 (we are already late for this), to unblock other work streams.

Then we update to 7.15 as soon as the first BC is out to test the new dashboards format.

[aspacca]

@endorama , it would be fine for me.

[jsoriano]

#27220 was merged yesterday requiring Kibana 7.15 and ndjson format for dashboards.

This format was supported by Kibana before 7.0, there shouldn't be any problem with using it in any 7.x version of Kibana, 7.14 should be fine.

I wonder if we should rather bump CI versions to 7.x or similar.

We should use released versions whenever possible. Using development versions has the risk of introducing unstability in our pipelines.

We should also use released versions of Kibana to create dashboards, so there are less risks of using features that are not available or are different in final versions. Also, the older the version of Kibana used to create the dashboard, the broader support for older versions the dashboard has.

[jsoriano]

LGTM if it looks good to Mario and Jenkins 🙂

[jsoriano]

Then we update to 7.15 as soon as the first BC is out to test the new dashboards format.

To be clear, 7.15 is not needed to test or use the new dashboard format (proof of it is that builds are green in #27220 - well, the relevant tests for dashboards are 🙂 ), this format was already available in all 7.x versions and before. The reason to make the change in Beats 7.15 is because Kibana is going to remove the old format at the latest in 8.0, so Beats needs to migrate to the new format before 7.last.

[aspacca]

@jsoriano thanks for the clarification: I might have missed some context indeed.

On #27220 I saw a version check in libbeat requiring minimum Kibana 7.15.0 ff1ca85 and was not sure if this would be hit on CI

[jsoriano]

@jsoriano thanks for the clarification: I might have missed some contest indeed.

On #27220 I saw a version check in libbeat requiring minimum Kibana 7.15.0 ff1ca85 and was not sure if this would be hit on CI

Umm, this is interesting, I may be wrong then 🙂 Not sure why adding this limitation, in principle the Saved Objects API is supported way before 7.15 (https://www.elastic.co/guide/en/kibana/7.14/saved-objects-api.html).

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-08-17T14:25:55.508+0000

• Duration: 106 min 45 sec

• Commit: 4579588

Test stats 🧪

Test	Results
Failed	0
Passed	9644
Skipped	2540
Total	12184

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	9644
Skipped	2540
Total	12184

[endorama]

So for some reason now the integration tests fail with

E               Exception: Key 'beat.stats.info.version' found in event is not documented!

Any idea why?

[jsoriano]

So for some reason now the integration tests fail with

E               Exception: Key 'beat.stats.info.version' found in event is not documented!

Any idea why?

It seems that the stats metricset in the beat module is collecting a (new?) field that is not documented. @sayden could you take a look?

@endorama as you only need to update Kibana and Elasticsearch to unblock your PR, an option could be to update them by now and keep the metricbeat version in the docker compose file as is.

[jsoriano]

🤞

[endorama]

All green, merging.

[jsoriano]

Adding backport labels for this, as seems required by #27470. @endorama there was any reason to don't backport this change to 7.x? cc @kvch

[endorama]

@jsoriano not at all, most probably I forgot in the heat of the moment.

A sidenote is that this PR did not update beats-metricbeat so now there are different versions for elasticsearch, kibana and metricbeat.