H4HIP: Plugin-first architecture

[scottrigby]

Preview: https://github.com/scottrigby/community/blob/h4hip-plugin-first-architecture/hips/hip-9999.md

To-do:

• Finish simplifying language in this HIP
• Create follow-up H4HIP to deprecate legacy CLI plugins (require converting to CLI-type Wasm plugins)

[mattfarina]

While I point out a couple sub-items below, here is my quick high level review ....

• The scope here seems like it's larger than the time we have to deliver. Feature freeze is in August. Can we get something documented where the feature functionality can be delivered by then?
• There seems to be a concept that SDK authors will want to have plugins as WASM for core functionality. The WASM runtime delivered with Helm will not be a high performance one. The high performance ones require cgo and are not portable to the extent we want. For an SDK user (who knows Go) to take functionality, compile it WASM, and then have that executed by a low performance runtime seems like a lower quality UX than an SDK user having clean interfaces and components all in Go. Am I wrong?
• There is a lot open for discussion in this proposed change. What is the design being proposed for Helm v4? I think the rest can be cut out so that we do not go down rabbit holes discussing the parts not for Helm v4.
• I like the idea of swappable components. For example, rendering. I can see that being a cross platform CLI plugin (and WASM could work well here).
• What about the idea of chart plugins being packaged with the chart itself? This is a trade-off. You have increased chart size but consumers of the chart do not need to find the plugins needed to use the chart.

[mattfarina]

I have a few thoughts on this...

1. How will Helm verify where a plugin came from and stop it from running? Is this going to be with some cryptographic key for signing/validation? Writing this isn't going to be straight forward.
2. Who will do the verification for trusted plugins? And, how do we do this in a way that's vendor neutral? There is going to be a fair amount of people/process here. If there is a verification process, this is going to be required. Where do the people come from to execute on this?
3. Companies are going to want to have private plugins. How is this going to work?

[scottrigby]

I agree. I've removed the verification process line in fb797e5. That was a (very conservative) stopgap for discussion, from a concern that a bad actor chart author may try to take advantage of users testing this out during development of the feature and before a secure process is established for discovery and distribution of plugins. I think that was overkill for this HIP, and yes not realistic for our timeline.

[mattfarina]

Feature freeze is in approximately 6 months. Can you put this implementation plan into a timeline that meets the freeze time? Note, docs don't need to be completed in that window but the other parts do. I'm skeptical that we have the time.

[sabre1041]

Some suggestions based on the initial draft proposal

[sabre1041]

"the default plugins" as described by the two kinds is somewhat confusing.

[scottrigby]

ok thanks—will update the HIP to make sure this part is self-explanatory.

[scottrigby]

@sabre1041 does this make more sense? bbdca4e

[sabre1041]

Suggested change

	- define different types of plugins for each part of existing Helm functionality (we could summarize all in-scope Helm functionality as fitting into different phases of the Helm packaging and release lifecycle)
	- define different types of plugins for each part of the existing Helm functionality (we could summarize all in-scope Helm functionality as fitting into different phases of the Helm packaging and release lifecycle)

[sabre1041]

Would we want to define different categories of plugins?

[scottrigby]

Absolutely. This is what's meant by this step under "Implementation plan":

Define Plugin Interface and Types: Develop standardized interface for plugin execution, and define the initial plugin types.

[scottrigby]

Actually I think this is clear in the first goal at the top:

• define different types of plugins for each part of existing Helm functionality (we could summarize all in-scope Helm functionality as fitting into different phases of the Helm packaging and release lifecycle)

@sabre1041 what do you think?

[sabre1041]

Is this a separate bullet item? Also, what is the definition of Helm's "core value proposition"

[scottrigby]

These indented bullets below the goal to "ultimately move as much existing functionality/code as possible out of Helm core into default plugins" are meant to be a list of benefits of doing that. I will clarify this.

[sabre1041]

Suggested change

	- Thanks also to Fermyon, Cosmonic, and Dylibso for guidance and collaboration on Wasm integration in Helm.
	- Thanks also to Fermyon, Cosmonic, and Dylibso for guidance and collaboration on WASM integration in Helm.

[scottrigby]

Should be Wasm not WASM, even though the doc already contains lots of the latter. Will fix all of these after I commit your other applicable suggestions.

[nilslice]

[...] The WASM runtime delivered with Helm will not be a high performance one. The high performance ones require cgo and are not portable to the extent we want.

@mattfarina - it's not uncommon to think CGO is required for high-performance Wasm execution in Go programs, but thanks to the many great contributors to the Wazero pure-go runtime, it's not the case.

Wazero does not require CGO, and in the same way other high-performance Wasm runtimes do, compiles Wasm bytecode to native x86 and arm64 instructions. On other architectures, it would run the interpreter, which is less performant, but quite uncommon to encounter.

Extism's Go SDK simply provides a more accessible & generic interface to Wazero so that plugin developers are not burdened with lower-level memory management like they would be directly using Wazero. In effect, you get high-performance plugin code execution, with all the benefits of the sandboxed environment brought by Wasm, without the cross-compilation and linking complexity of CGO.