[new-platform] UI plugins

[spalger]

Part of #16253

See #18840 to get more details on kibana.json manifest and serve-side part of plugin support.

details are probably going to change here

The UI plugin will look something like this:

// foo/public/ui_plugin.js
export default class Plugin {
  start(core) {
    console.log('starting example plugin');
    const { applicationService } = core;

    applicationService.register('example', async ({ targetDomElement }) => {
      console.log('mounting example app');

      const { bootstrapApp } = await import('./app');
      return bootstrapApp({ targetDomElement });
    });
  }
}

Notice the await import() statement that pulls in app.js. This will cause webpack to split the newPlatform bundle into two chunks, one containing the bootstrapping code and one containing the app for foo plugin. This strategy may still be used plugin authors when they become responsible for their own build systems down the road to allow Kibana to load less code on startup.

// foo/public/app.js
import React from 'react';
import { render, unmountComponentAtNode } from 'react-dom';

export function bootstrapApp({ targetDomElement }) {
  console.log('render example app into targetDomElement');
  
  render(<div>Example App</div>, targetDomElement);
  
  return () => {
    console.log('cleanup example app');
    // called when app is unmounted
    unmountComponentAtNode(targetDomElement);
  };
}

[epixa]

This is how I envision it working at a high level:

server-side

When loading Kibana, we inject a sorted array of plugin names (only UI plugins).

client-side

The core system would defer this logic to a client-side plugin service.

class CoreSystem {
  start() {
    // ...
    this.pluginService.start();
    // ...
  }
}

The plugin service would retrieve the list of plugins from injected vars and
asyncronously load each plugin entry point.

class PluginService {
  async start() {
    const plugins = this.injectedMetadata.getInjectedVar('plugins');
    for (const pluginName of plugins) {
      const { plugin } = await loadPlugin(pluginName);
      plugin().start();
    }
  }
}

Plugin entry points would be written in the same convention as the server-side
plugins, so they'll contain a class definition for the plugin but ultimately
it will be the exported plugin function that the core system invokes.

class Plugin {
  start() {}
  stop() {}
}

export function plugin(...args) {
  return new Plugin(...args);
}

The open question to me is the implementation of the loadPlugin in this
proposal. If we put aside how we construct the plugin module path, which I
didn't propose a solution for here but I think is pretty straightforward, we're
left with how we import code in a world without webpack.

For plugins that ship in Kibana, we can just use dynamic import.

async function loadPlugin(name) {
  return import(name);
}

Since dynamic import is processed by webpack, third party plugins couldn't be
imported in this way unless we forever shipped webpack with the distributable.

A traditional async script injector using a reserved map on window should
work for this sort of plugin once the plugin is built.

async function loadPlugin(name) {
  return new Promise(resolve => {
    const script = document.createElement('script');

    script.addEventListener('load', () => {
      resolve(window.__kbn__[name]);
    });

    script.src = name;
    document.body.appendChild(script);
  });
}

But this would require that built plugin entry files write their exported
interface to the non-standard reserved map on windows. The resulting bundle
would need to look sort of like this:

class Plugin {
  start() {}
  stop() {}
}

function plugin(...args) {
  return new Plugin(...args);
}

window.__kbn__.mypluginname = { plugin };

It's not clear to me what is the best way to manage this from a build tooling
standpoint. I suspect we'll want a combination of both loaders, where we use
dynamic loading via webpack in development and then have different build output
for creating the distributable versions of Kibana or third party plugins that
creates these "immutable" script-injected, window-proxied entry files.

Thoughts?

[spalger]

A traditional async script injector using a reserved map on window should
work for this sort of plugin once the plugin is built.

I agree, lets just do that

It's not clear to me what is the best way to manage this from a build tooling standpoint. I suspect we'll want a combination of both loaders, where we use dynamic loading via webpack in development and then have different build output for creating the distributable versions of Kibana or third party plugins that creates these "immutable" script-injected, window-proxied entry files.

With separate webpack builds for each new platform plugin, once we get there, we'll be able to configure webpack to write the default exports of the entry file to a global on window like suggested, so we'd need to update the example to export the plugin function, but otherwise the webpack config will be sufficient to handle this. Until we have separate webpack compilers for each plugin we can just do const { plugin } = async import('new_platform_plugins/{id}'), so that should be fine.

[epixa]

@spalger To clarify, are you proposing that we move forward with dynamic import for all plugins (core, x-pack, third party) and then only tackle the traditional script injector loader whenever we move to immutable webpack bundles for plugins?

[joshdover]

I'm planning on breaking this work up into 3 PRs:

• Adding ui plugin information to injectedMetadata to be read by the client-side
• Loading plugin bundles and booting plugins with the client-side CoreContext and plugin dependencies' start contracts
• Injecting new platform plugins start contracts into the legacy world for consumption by legacy plugins

[joshdover]

After prototyping the Webpack-based dynamic import()s of plugin bundles, here's what I've found.

Option 1: Magic Webpack Dynamic Bundles

tl;dr: this will work, with a bit of a weird implementation hack needed to hint to Webpack what to optimize.

So first of all, what actually happens when Webpack sees code like this?

// src/core/public/plugins/plugin_loader.ts
import(`../../../plugins/${pluginId}/public`);

Obviously, Webpack can't rely on runtime variables to know which bundles to build at compile time. So it needs to make a decision at compile time on what bundles to build.

You may be thinking that we would need to configure Webpack to build these files by specifying them as entry files, but this is not how it works. In fact, when Webpack parses this file it recognizes that this import() is called with a dynamically generated string at runtime, so it expands that template string and then builds every possible bundle that could match that pattern on disk.

This can be potentially problematic. For instance, using a very generic string like import(`${pluginPath}/public`) would cause Webpack to build a bundle for every single file matching **/public/index.(js|ts|tsx) and **/public.(js|ts|tsx).

Due to this behavior, we will need to "hint" as much as we possibly can to webpack by having multiple import statements that are specific to each directory we support plugins to live in.

// How we match `pluginPath` to the different paths should probably be loaded from an
// Env service based on the full directory path instead of hardcoded like this,
// but you get the idea.
let plugin;
if (pluginPath.endsWith(`kibana/src/plugins/${pluginName}`)) {
  plugin = (await import(`../../../plugins/${pluginName}/public`)).plugin;
} else if (pluginPath.endsWith(`x-pack/plugins/${pluginName}`)) {
  plugin = (await import(`../../../../x-pack/plugins/${pluginName}/public`)).plugin;
} else if (pluginPath.endsWith(`kibana/plugins/${pluginName}`)) {
  plugin = (await import(`../../../../plugins/${pluginName}/public`)).plugin;
} else if (pluginPath.endsWith(`kibana-extra/${pluginName}`)) {
  plugin = (await import(`../../../../../kibana-extra/${pluginName}/public`)).plugin;
} else {
  throw new Error(`Unsupported bundle path! ${pluginPath}`);
}

The other weirdness with this option is that with the kibana-extra directory. If the directory does not exist, the Webpack build will fail. This means we'd need to do some pre-processing on this file before Webpack processes it in order to only add that if block when that directory exists. Yuck.

Option 2: Explicit Webpack Bundles

Webpack also supports not doing any magic to detect bundles that need to be built:

// src/core/public/plugins/plugin_loader.ts
import(/* webpackIgnore: true */ `./plugins/${pluginId}.bundle.js`);

This instead will use the normal ES2015 Loading Spec to load a bundle at a URL path relative to the bundle running the code.

With this method, we would instead add the plugin paths we want to support directly to our Webpack configuration. Because these are not actual application entry points, we could make this path configuration static so that optimizer would not have to build any bundles that are
already built when a plugin is disabled and instead only build any new plugins that were installed.

Conclusion

So while the magic option would work, I prefer the explicit configuration because it gives us more explicit control, avoids any need to do preprocessing, and provides a simpler path to true immutable bundles for each plugin (in fact, if the import() polyfill supports IE11 there may be very little work required here).

[joshdover]

More on Option 2 after some more testing:

Webpack doesn't seem to support building Dynamic Modules explicitly in a way that works with older browsers unless you use their import() magic. So it appears the only way to make this work in any fashion is to either:

• Configure Webpack to assign an entry point's plugin export to a global object, such as window.__kbn_new_platform__.myPlugin, which I have been able to do. We'd then load this bundle using a script tag and pull the function from window.__kbn_new_platform__; or
• Fetch the bundle with fetch and then call eval on the resulting text to get the plugin function. (I suspect this is how Webpack's import() magic works under the hood, though I'm having a very hard time hunting down the implementation).

[joshdover]

I'm planning on moving forward with configuring our optimizer to create entry points that attach themselves to a global object and loading these scripts dynamically with <script> tags. This approach will not use import() and will be compatible with IE 11.

This seems like the best approach because:

• It relies on less magic
• Avoids any need to template source files before loading into Webpack
• Eliminates the risk of building unnecessary bundles (due to Webpack's "build all possible bundles" strategy for import(); and
• Is the long-term solution we were envisioning anyways

I've got a small POC of it working already and I don't anticipate the changes to optimizer to be substantial. In fact, I think this path requires less changes to the optimizer than making import() work safely with our multiple plugin paths.

@epixa @spalger: Is there anything you know about that I may not be considering that I need to think about before moving forward?

[epixa]

I can't think of anything off-hand. I've long imagined a similar approach to what you're describing, but I didn't know how to work it into webpack in a seamless way.

[spalger]

I'm all about this if you can get this working in Kibana. I feel like I tried to load multiple entry points on the same page and ran into serious issues, but my information could be out of date or totally wrong. How does webpack handle dependencies that are shared between the multiple entry paths?

[mshustov]

(I suspect this is how Webpack's import() magic works under the hood, though I'm having a very hard time hunting down the implementation).

If I'm not mistaken it uses jsonp pattern - creates a script tag that loads a chunk, chunk already wrapped in global special function window["webpackJsonp"] that registers it. This code is generated during the build phase and shipped as a core part of webpack runtime. You can find examples in their repo. check the raw *md file sourcecode https://raw.githubusercontent.com/webpack/webpack/07d4d8560060c102a1aed97844d452547d02b1d4/examples/code-splitting/README.md

How does webpack handle dependencies that are shared between the multiple entry paths?

Webpack does its best to places shared deps in a parent chunk. AFAIR can be configured by splitChunks.

[joshdover]

How does webpack handle dependencies that are shared between the multiple entry paths?

Our splitChunks configuration in the base optimizer appears to be working fine. In my PR, I've created two different plugins that both import React and render a component and when I use webpack-bundle-analyzer each bundle only has its own source files and does not include React. Think we're good to go on this.

[joshdover]

One edge I've hit in testing, this scenario is valid:

• pluginA has server-side code, but no client-side code
• pluginB has server-side code and client-side code
• pluginB depends on pluginA

This state causes pluginB to startup in the browser with no contract for pluginA in its dependencies. This is essentially a non-issue with plugins written in TypeScript, because you wouldn't be able to import the PluginAStart for the client-side code because it wouldn't exist. However, in a pure JS scenario, this may be unexpected and core can't throw an exception because this scenario is a totally valid use case.

Options I see to solve this to make the development experience smoother:

1. Just don't, this is an edge case. Document this behavior somewhere in code.
2. Update kibana.json to have separate server dependencies and client dependencies. This adds a bit of complexity/boilerplate but could be useful and I imagine it more closely maps to the reality of most plugin dependency trees.
3. Log an info or debug message in server. This is probably going to be pretty noisy, as like I said, this scenario is totally valid and likely common.

Personally, I'm a fan of (2), but I'd like to hear others' feedback or ideas.

[mshustov]

This state causes pluginB to startup in the browser with no contract for pluginA in its dependencies.

Actually, I already thought about the problem and decided for myself that when you declare a dependency on a plugin you know what API you want to consume and you know it is available via contracts. Otherwise, you can also call the non-existing method and get the same problems.
Also, I'm afraid of new corner cases, say pluginA provides both server and client plugin, but pluginB uses only one of them, etc.

[joshdover]

Actually, I already thought about the problem and decided for myself that when you declare a dependency on a plugin you know what API you want to consume and you know it is available via contracts. Otherwise, you can also call the non-existing method and get the same problems.

This is a good point. I think to make this more explicit we should not even populate an undefined key in the dependencies we pass to the plugin's setup function. It should just not be present at all. I'd be ok with this for now and solve for it later if it becomes more of an issue.

Aside: I'm learning towards requiring that plugins at least provide a index.d.ts file if they decide they want to implement in JS.

Also, I'm afraid of new corner cases, say pluginA provides both server and client plugin, but pluginB uses only one of them, etc.

I'm not sure why this would be an issue, could you elaborate more?

[mshustov]

I'm not sure why this would be an issue, could you elaborate more?

Ok, only one case. For example, PluginA provides both client and server parts. PluginB the only consumer of PluginA in the codebase, but declares dependency only on PLuginaA client part. This client part of PluginA expects some data injected on a page by server part of PluginA. Now it brings about more questions: Should we include both parts of a plugin if they tightly coupled? Who makes a decision? What if a user really wants only the client part and wants to swap server part with a custom implementation?

Aside: I'm learning towards requiring that plugins at least provide a index.d.ts file if they decide they want to implement in JS.

👍

[epixa]

Ok, only one case. For example, PluginA provides both client and server parts. PluginB the only consumer of PluginA in the codebase, but declares dependency only on PLuginaA client part. This client part of PluginA expects some data injected on a page by server part of PluginA.

This was the exact scenario we had in mind when we made the original decision to make plugins an all or nothing thing rather than being able to disable or depend on only one portion of a plugin. I think the right approach here is to not include the plugin name at all in the dependencies argument as @joshdover mentioned in #18874 (comment). That's the behavior we want for optional dependencies as well, for what it's worth.

Aside: I'm learning towards requiring that plugins at least provide a index.d.ts file if they decide they want to implement in JS.

This just seems like a recipe for inaccurate types. If a plugin doesn't want to expose types, then they won't want to keep types up to date either.

[joshdover]

This just seems like a recipe for inaccurate types. If a plugin doesn't want to expose types, then they won't want to keep types up to date either.

Sure if they don't want to expose types then they shouldn't return anything from their setup. But if they do expose a contract, any plugin using their contract will write their own types. Now we just have n type definitions to keep up to date rather than one.

[epixa]

I don't think "not providing extension points at all" is the same thing as "not providing types for the extension point". We intend to have all of the Kibana repo be typescript, so the multiple types thing won't impact mainline development down the road. But requiring types puts an unnecessary restriction on third party development that doesn't need to exist.