Add a persistent cache for cloudfoundry metadata based on badger

[jsoriano]

What does this PR do?

Add a persistent cache on disk for Cloudfoundry metadata, based on badger.

Why is it important?

Improve restart of beats when monitoring very big Cloudfoundry deployments.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• Look for some way to avoid so much json unmarshalling.
• Cache should be closed at some moment (to be continued in Fix leaks with metadata processors #16349).
• Add settings to make it optional?
• Use different store names for different Cloudfoundry deployments, to support monitoring multiple clusters from the same Beat (maybe not needed because it is difficult to have uuid collisions).

How to test this PR locally

• Use add_cloudfoundry_metadata with Filebeat or Metricbeat.
• Restart the Beat.
• The Beat should reuse information on disk to enrich events without querying cloudfoundry API.
  A way to confirm it is to check the http connections done with a reverse proxy like mitmproxy.

Related issues

• Would replace Move Auditbeat datastore to Libbeat #20707, where I evaluate the use of Auditbeat datastore for the same purpouse.
• Would also replace Add a persistent cache for cloudfoundry metadata #20739, where I started using statestore for this purpouse.
• Fixes Proposal: Cache metadata to local disk/db #19511.

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [jsoriano commented: jenkins run the tests please]

• Start Time: 2020-10-05T22:50:54.940+0000

• Duration: 78 min 48 sec

Test stats 🧪

Test	Results
Failed	0
Passed	16271
Skipped	1343
Total	17614

[jsoriano]

jenkins run the tests please

[jsoriano]

jenkins run the tests please

[jsoriano]

@urso @exekias I am going to open this for review as I would like to reach 7.10, it'd be nice if you could take a look.

There are still some "controversial" points, but I think they can be considered non-blockers:

• Database is not being properly closed, because we don't have a way to finalize processors. This is a more general problem (see Fix leaks with metadata processors #16349). After many tests I haven't seen it causing problems that require some manual operation. Main problem here is that one may lose some entries written during beat shutdown, but I don't think this is such a big problem for this kind of cache. So I think we can go on as is by now, and continue with Fix leaks with metadata processors #16349 as a later fix.
• Caches are requested using package-level methods, and references are kept in a global registry, this is done to allow sharing the same cache, what is needed if multiple features use it. It is done at the package level to avoid adding some additional object in the main beater that needs to be passed to processors. There are methods to instantiate additional registries, this could be used for other uses or future refactors.
• CBOR encoding is done at the end with github.com/ugorji/go/codec, that is also a dependency of badger. I found problems with the codec in github.com/elastic/go-structform. In any case the code for this is quite isolated, so we can refactor it later to use any other library if needed.

Let me know if you prefer this split in two PRs, one for the cache and another one for the changes for Cloud Foundry.

[exekias]

LGTM, besides lint issues which need some attention

[jsoriano]

jenkins run the tests please