[kbn/optimizer] implement "requiredBundles" property of KP plugins

[spalger]

Plugin teams who were pinged for review

This change triggers a ton of review because it changes a lot of kibana.json files. These changes are validated at build time by the @kbn/optimizer and I've talked to @kobelb and we've agreed that we're going to merge this even if we don't get review from all the plugins. I encourage you to see and understand the change described here as it will impact your plugin during development, but please don't feel like you need to schedule review unless you're on the @elastic/kibana-platform or @elastic/kibana-operations team.

---

Closes #69458

This PR requires that plugin define the bundles which they import from, so that when the bundle for that plugin is loaded we can be certain that the imported-from bundles are loaded even when those plugins are disabled.

When a plugin tries to import from another bundle but the bundle is not mentioned in the kibana.json file the following error will be logged:

import [../../../../../src/plugins/kibana_react/public] references a public export of the [kibanaReact] bundle, but that bundle is not in the "requiredPlugins" or "requiredBundles" list in the plugin manifest [{path to}/kibana.json]

When users fix this problem while running @kbn/optimizer in --watch mode (or just using yarn start), webpack will notice the change to the kibana.json and rebuild, adapting to changes in kibana.json at runtime without needing to restart the optimizer.

Similarly, when requiredBundles has an extra plugin listed another error will be logged and again the error can be cleared by fixing the kibana.json file.

Finally, in order to make sure that the kibana.json file is tracked by webpack when it watches files for changes learned about compilation.fileDependencies. This is an array of every file path that webpack interacted with while creating the build, including files that don't make it into the final bundle or impact the output of the bundle. This list is useful for measuring how many files each bundle forces webpack to consult, which was originally the intention behind the moduleCount metric. The size of this list, plus a static value of 100 for each .scss module created, are used to create a new workUnits value that is tracked by the optimizer. Using this metric to allocate bundles to workers rather than moduleCount makes @kbn/optimizer marginally more efficient on systems with lots of CPU cores.

We still track the moduleCount for reporting purposes as it's still a very useful metric for identifying surprisingly heavy imports and other unexpected side-effects of changing imports in public bundles.

[paul-tavares]

FYI - this PR for Security (Endpoint) is dependent on this change - #71198

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[spalger]

@elasticmachine merge upstream

[smith]

APM change looks good.

[spalger]

@elasticmachine merge upstream

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 7f31df2

Build metrics

✅ unchanged

History

• 💚 Build #60525 succeeded d7801d0
• 💔 Build #60484 failed 27bd3de
• 💚 Build #60430 succeeded 75f8240
• 💚 Build #60285 succeeded 7c33060
• 💔 Build #59944 failed 3ec96ff

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[sebelga]

@spalger do you mind explaining the difference between requiredPlugins and requiredBundles? I thought that "esUiShared" was also a plugin like any other. Thanks!

[spalger]

@sebelga it is, the difference is that requiredPlugins defines the dependency tree for plugin contracts. If you're only using static imports for a plugin you don't need it's plugin contract, and therefore can execute even if the dependent plugin is disabled as long as their static code is still loaded and available.