Validate "base-field.yml" files

[mtojek]

All base-field.yml files must be provided by integrations.

The package-storage will validate PRs to check if mentioned fields are present.

[mtojek]

It might be a breaking change, so I will rollout changes in steps:

• Update EPR: create missing files (base-field.yml) if missing.
• Update all integrations: add missing files.
• Update packages in package-storage: add missing files.
• Update EPR: only validate base-field.yml.
• Update dependency on package-registry in integrations
• Update dependency on package-registry in package-storage
• Update import-beats script to create the base-field.yml file.

Are you ok with the scenario @ruflin ?

[ruflin]

@mtojek Steps look great. Like this from a consumer perspective things never break.

One problem we have is that in old packages the fields don't exist, for example https://github.com/elastic/package-storage/tree/master/packages/cisco/0.1.0/dataset/asa/fields Now assuming we release 0.1.1 of this package, the old package will not validate in the package-registry. Should we update the 0.1.0 package too?

It brings up the more general question: What do we do if the package registry adds more validation and an old package cannot be validated anymore?

Not a big issue at the moment as we can still change things but probably worth discussing separately. Lets not block on this.

[mtojek]

Now assuming we release 0.1.1 of this package, the old package will not validate in the package-registry. Should we update the 0.1.0 package too?

Yes, I included also these in point 4.

It brings up the more general question: What do we do if the package registry adds more validation and an old package cannot be validated anymore?

We can document package format and version it.

[ruflin]

We have a package format version today: https://github.com/elastic/package-registry/blob/master/dev/packages/example/base/0.2.0/manifest.yml#L1 +1 on using it, but it would mean we keep legacy code in the registry for quite a while. I would say for now, lets fix the existing packages until we ship 7.9.

[mtojek]

All action items done. Resolving.