Plugin support

[andremedeiros]

As a user, I would like to be able to implement custom functionality and protocols and expose it in the JavaScript VM so that I can write the heavy lifting work in Golang but write tests in JS.

Feature Description

As a protocol developer, I'd like to be able to develop a client that talks to my protocol via the JavaScript VM that k6 runs. To do this, I would prefer to be able to write a plugin that doesn't have to live in the main codebase and isn't the responsibility of the k6 team.

Effectively, a plugin should expose:

• A preflight function (to start any background services it might need)
• A postflight function (to shutdown any background services it might have started)
• A map[string]interface{} that gets added to the module mapping so that the plugin's functions can be imported from JavaScript.

Suggested Solution (optional)

My MVP proposal would be to use Golang's plugin package. Despite not supporting Windows in its current state, Windows users could use Docker to run plugins. Moreso, it would not introduce any new dependencies to k6.

Plugins would be loaded by passing a -plugin argument when launching k6, which should receive a .so file path, and would be initialized (preflight) once, on test start, and cleaned up (postflight) once, on test finish.

A plugin struct could look something like:

type Plugin struct {
	Name       string
	Preflight  func() error
	Postflight func() error
    Modules    map[string]interface{}
}

For each plugin file, k9 would run something to the lines of:

// Omitting error checking for brevity
for _, path := range plugins {
	p, _ := plugin.Open(path)
	m, _ := p.Lookup("Plugin")
	meta := m.(Plugin)

	// Add plugin modules to JavaScript module registry
	// Add preflight and postflight functions to a callback list
}

Tagging @na--, @imiric, and @mstoykov as indicated in Slack.

[na--]

Thank you for creating this issue! As I said in Slack, we've discussed plugins a few times internally, but the lack of cross-platform support has always stopped us. Docker is an acceptable workaround for Windows though, and if a public plugin ever becomes very popular, we can always work on upstreaming it or having similar functionality in core k6, so I think this should be fine.

A few comments about the proposal itself. Even though this is a v1, and we likely won't be making any API stability guarantees, we probably should think a bit about supporting other things besides new JS modules. Not actually implement anything else, just figure out a mechanism by which a plugin can declare its capabilities. The only capability currently available and supported by k6 would be extra modules exported to the JS runtime, but in the near future we'll probably want to include outputs in the mix.

The reason for this is that I'd like the CLI plugin flag to one, --plugin. You should be able to just execute k6 run --plugin extra-js-functions.so --plugin timescale-db-output.so --plugin mix-new-js-objects-and-output.so --plugin whatever.so my-cool-but-demaning-script.js.

So, I'd like k6 to be able to dynamically determine what a plugin adds. Maybe type assertions with a bunch of pre-defined interface plugin types would be enough, one per type of plugin? Not sure if plugins restrict that in any way, but maybe the generic k6 plugin and the JS plugin interface can look somewhat like this:

    type K6Plugin interface {
        Name() string
        Preflight() error  // though maybe Setup() or Init() would be a bit better?
        Postflight() error  // Teardown()?
    }
    type JavaScriptPlugin interface{
        K6Plugin
        GetModules() map[string]interface{}
    }

We'll discuss this internally again tomorrow, and if we think of any other potential issues, we'll mention them here.

[na--]

Hey, @andremedeiros, we discussed this internally again and we didn't see any obstacles, so k6 plugins are a go! 🎊 So, if you're interested in working on this, don't hesitate to ask any questions.

As a practical matter, the actual configuration handling of --plugin foo.so --plugin bar.so (and probably K6_PLUGINS="foo.so,bar.so" would also be useful) should be done as part of the RuntimeOptions in https://github.com/loadimpact/k6/blob/master/cmd/runtime_options.go. Then, somewhere around here you could probably inject the new JS modules in the current map: https://github.com/loadimpact/k6/blob/567e5a0e225e3bb8f2d5dec00c36fd4fdd9a5d7a/cmd/run.go#L115-L119

Also, we'd like the initial PR to k6 that adds the JS plugins to also feature a very simple demo plugin, and a small integration test in CI that the plugin actually works. The plugin can be dead simple though, a function that reverses a string or a leftpad() implementation would suffice 😅 . Or something more useful as long as it's not too complex.

[andremedeiros]

Sounds great to me! I'll get to work on it!

[andremedeiros]

Friends, I've pushed a preliminary PR (#1396) that loads plugins, exposes them, and does the setup/teardown lifecycle.

I don't have tests yet, and I'm not 100% happy with how clunky writing plugins feels right now, but it's doing what it's supposed to.

I'd love to have a more involved chat with the team re: types and how things "feel." Basically, with the way that Go's plugin package works, plugins are only useful if they export variables or functions.

We have two options:

1. Be implicit in what is exported, as in, loading a plugin will try a series of variables that are defined by conventions (ie. if the plugin exports a JavaScriptPlugin variable we know what to expect.) This will result in n lookups, but less boilerplate.
2. Be explicit in what is exported, as in, a function that is called JavaScriptPlugin and that returns nil when it doesn't expose anything of that type. This allows for a bit of logic but is also easier to break in the long run if we expect all functions for plugin types to be present (although we could also ignore the things we don't find.)

Curious to hear your thoughts on the approach.

[simskij]

Curious to hear your thoughts on the approach.

🥳

I've submitted some initial feedback. While this looks great so far, I think it will be easier to reason about once a runnable proof of concept (the previously mentioned demo) has been added to it.

Regarding the options:
Given that we load and execute external code at runtime, the n additional lookups added by option 1 instinctively feel like a diminishing overhead. At the same time option 2, in my opinion, adds more issues than it solves.

[na--]

Sorry, I'm probably not going to be able to review the PR until next week - I'm swamped by other things today, and after that I'm off until next Tuesday.

[andremedeiros]

Had another pass today, and I'm pretty happy with the way it turned out. Addressed some of the concerns, and decided to push a couple of things for future me to handle.

With regards to what I mentioned above, and after digesting @simskij's feedback, I decided to go with option 1. It's negligible in the broader context of running a load test, so n lookups for the sake of compatibility is a good thing.

At this point, baring an issue in the tests where the plugin is being built in the exact same container as the tests are being run, but somehow golang thinks it's built with a different runtime, I think we have a solid v1 implementation of plugins. It's useful (at least for the use case I originally had) and it sort of builds... which leads me to...

Can one of you look at the tests?

I'm not quite sure why they're failing right now, and it seems to be pretty happy On My Machine (tm).

Also, as a way to have a plugin that can be loaded in tests, I pushed a repo to andremedeiros/leftpad that contains a very minimal plugin example. This gets loaded in tests and we run assertions on the output too. Part of me feels that this should be done inside the test code, in some dynamically created test directory, but I can't for the life of me find an acceptable pattern for this use case in golang. I'm willing to hear what other people think and adapt.

[mostafa]

My evaluation of the plug-in system for k6

The plug-in system will soon be the quintessential part of the k6 tooling ecosystem. The benefits of such a system are not hidden to anyone, with the most significant being that load testing various protocols become easy, in addition to various useful plugins that can help with current issues with JavaScript external modules (NodeJavaScript APIs, browser APIs and ...). These are my observations:

The Good

I was able to quickly create three plugins:

1. A plugin for load testing Apache Kafka with custom messages: k6-plugin-kafka
2. An ICMP plugin (ping4 and ping6): k6-plugin-icmp
3. A SQL plugin to load test SQL servers, supporting PostgreSQL, MySQL and SQLite: k6-plugin-sql

It took me a long time to create the first plugin because I was learning Golang and dealing with various parts of the language and the k6 APIs. But the second plugin was created in an afternoon. Before creating the third plugin, I started to think that a pattern is emerging, so I created a template repository on GitHub and started forking it for the third plugin. The template is not perfect and it also includes the hacky parts for getting state and pushing metrics. Thus, once the plugin system is merged, the template repo should be updated.

The Bad

It was really difficult to understand how k6 and the plugin system works behind the scenes, mostly because of the lack of documentation and examples. This was significantly more difficult in the first plugin, but became easier over time. The hardest part was figuring out that there's a magic Context passed as the first argument to each function. The second hardest was the conversion (reflection) of types between JavaScript and Golang. These should be documented extensively. Another annoying issue was the API differences between JavaScript and Golang and it was mostly due to the error handling of Golang. We should (maybe) have a documented convention for writing functions and exposing plugin APIs, so that we don't end up dealing with a multitude of API differences in terms of UX and DX.

The Ugly

Using a hacky solution to get (current) state from the Context was the ugliest part of the experience. The lib.GetState didn't work on v0.26.2 and v0.27.0, which forced me to write a workaround for it. I have modified Context printing function to retrieve the state (instead of printing).
Also, porting the plugin system to v0.27.0 caused the k6 to not stop sometimes, especially with higher VUs, even with duration and gracefulStop and other related configs set. Then, I must kill it manually, which made it not to return any results.

[mardukbp]

In my opinion, using Go plugins is not the way to go.

Facts:

• Using plugins in a project with vendored dependencies (like k6) is a mess
• Windows support is not coming any time soon
• Requiring to install Docker, which consumes gigabytes of hard disk and RAM, to run a 1 KB plugin is absurd

As the 5th top contributor to the go repo put it:

It is really indisputable that plugins are a half baked feature. They have significant problems even on platforms where they (mostly) work. It was perhaps a mistake to add them at all. I think I approved the change to add plugins myself; I'm sorry for the trouble they are causing.

I think the k6 team has to consider a cross-platform solution, built on a more solid foundation. One possibility is to use yaegi, a Go interpreter implemented in Go and designed to add plugins to Go applications.

[na--]

@mardukbp, I hear you. We're not blind about the issues with Go plugins, though I have to admit I hadn't considered the potential problems of vendored dependencies. Another issue you don't mention, which we recently found out about, is the requirement to use CGO_ENABLED=1 in the k6 builds, which we currently don't do (1, 2, 3, 4).

Regarding the vendor issue - I don't think it'd be a huge problem in the case of k6. The likely use case of a k6 JS plugin would be to expose some new functionality to k6 scripts (DB queries, DNS, raw TCP, queue messages, some SDK for something, etc.) and maybe measure the things it does and emit metrics. So, in the unlikely event it shares a dependency with k6, it's not going to be an issue if it has its own copy of it. The architecture of how VUs are executed and how they emit metrics is reasonably decoupled and shouldn't require any specific dependency beyond k6.

Also, k6 is meant to be a batteries-included sort of tool. If a particular well-written plugin becomes very popular, it's very likely we'll try to bring it into the k6 core. And, your original suggestion (#1595) is never going to not be an option for anyone who knows Go. Compiling a k6 version with a plugin bundled in is probably going to be very easy, possibly also automate-able. @mstoykov actually suggested that instead of plugins, we should have a "k6 bundler" which builds a unique k6 binary, for any platform, allowing you to just select which things out of a list of features/plugins you want. Sort of like these JS library builders - for example, babel. I don't think that would offer better UX, but it's not completely out of the question that we pivot in that direction, if the current version of plugins proves to be too problematic for us. We'd be very explicit that we don't offer any guarantees about backwards compatibility yet, when it comes to plugins.

Lack of Windows support for Go plugins isn't ideal either, but having Docker as a workaround is mostly OK, in my mind. The k6 docker image sits at ~10MB, which seems reasonable to me. And, I'm not certain, but the Linux k6 binary should work on the the Windows Subsystem for Linux (WSL), which means you should be able to use k6 plugins that way also...

yaegi deserves some investigation - we've previously only considered it as a way to allow non-JS scripts to be executed by k6. As I explained in #751, there are some issues with that use case, mostly refactoring that needs to happen in k6 before we can try it. We haven't considered how suitable it would be for plugins though, and my biggest worry is performance.

It's not out of the question for a single Go dependency to be tens of thousands of lines of Go code. For example, the https://github.com/Shopify/sarama/ dependency, which is currently used only for the Kafka output in k6. Our vendored copy of it sits at 10k lines of Go code, and that's not including any of the dependencies it itself has. Judging by the #617 PR that added the Kafka output, it might be closer to 25k. So, a kafka plugin is going to pull in anywhere from 10k to 25k lines of Go code, and there are probably much larger things (e.g. AWS SDK or some crypto library?). It seems to me like we'd want them to be compiled instead of interpreted, especially in a load testing tool. We already have issues with huge JS scripts. goja, the JS runtime we use, is reasonably performant with most scripts, but we're hitting performance issues when dealing with huge ones. I don't want to also have to deal with performance issues because we're interpreting huge Go files...

[mostafa]

Hi @mardukbp,

Regarding your comment about yaegi and support for running Go code in k6, I've made a plugin that does exactly that, although I haven't publicly published it yet. But there are some issues with it:

1. A good sum of the standard library is not supported, thus, many external libraries that depend on them cannot be used: Can't use external libraries in the code: "unable to find source related to xxxx" traefik/yaegi#656.
2. As @na-- stated, it is not as fast as the compiled Go code. This became more evident while I tested it using the plugin system, whereby a chunk of Go code is passed around from k6 to yaegi like k6 <=> Goja <=> Plugin system <=> plugin <=> yaegi. There are lots of contexts/states being passed around.

Regarding the half-baked plugin system and no support for Windows, we talked about it a lot. We also considered using the go-plugin from Hashi Corp. instead, but the current implementation is going to be THE version 1 of the plugin system, which may change in the future, possibly breaking backward compatibility. The overall goal of the plugin system is to let users write JS plugins specifically, so no Go for now and no Windows support until golang/go#19282 is resolved.

[mardukbp]

The overall goal of the plugin system is to let users write JS plugins specifically, so no Go for now

I firmly believe that k6 should be extensible both at the Go and JS levels. None is more important than the other. They just serve different purposes.

Having said that, I think that a k6 bundler is the way to go. That is how the plugin system of the Caddy webserver works. This offers a cross-platform solution leveraging the Go build system, which is one of the platform's greatest strengths.

In any case, I would suggest adding a section to the Documentation explaining how to extend k6 with Go. It is simple, useful and works now and in all platforms.

[na--]

hmm I didn't know about https://github.com/caddyserver/xcaddy, this has a lot of potential...

[na--]

Hmm I'm not sure what implications the AGPL-3.0 k6 license is going to have on a bundler - would all k6 plugins in such a system have to be open source?

[mstoykov]

I think that part of the clauses only apply if you are distributing the binary or are providing a service that directly uses it ... I think :D

[mostafa]

@na-- xcaddy looks somewhat like my idea of a version/package/environment manager. 😉

[na--]

@mstoykov, maybe, but that wasn't my impression 🤷‍♂️ if we go that route, we might need to offer some guidance what's kosher and not in the docs...

@mostafa, in as much as it has versioned things, xcaddy looks like every version/package/environment manager... 😉 But it's very different compared to pretty much every other plugin system I've seen, considering something like that is going to build a bespoke k6 binary with the desired plugins built-in... Moreover, and mind you, this is someting that I very much like, it doesn't seem to require any infrastructure from our side. It seems to basically operate with git repos, which is a nice benefit. Another plus is that it seems to be built on top of Go modules, which from @mstoykov's experiments caused a ton of issues with the Go plugins from #1396... @mardukbp, thank you very much for bringing all of these issues and potential solutions to our attention!

[mstoykov]

Sorry for writing so late, but I have been running tests and trying to get a plugin to work with the now go modules.
The k6 code is in https://github.com/loadimpact/k6/tree/pr1396-rebased and the plugin code is in https://github.com/mstoykov/leftpad
Things that I have learned:

1. it's possible 🎉
2. As far as I can see in order for this to work you need to have k6 to be in the EXACT same path and to be the EXACT same commit when you compile against it the plugin in order for it to work (on other machines.)
3. Requires all the replace stuff, the script I've added helps there but it's a PITA. Also requires that the replaced module has a go.mod which some don't and some do but go mod vendor didn't put them there for some reason 🤷‍♂️ . The commit where I add the go.mod isn't needed in k6, but it means that the plugin developer will need to add them. Also go mod vendor in k6 deletes them which means that we couldn't (at least not without more changes) just add them for all dependencies.
4. Using -trimpath ... doesn't work, it seems like it does if you are in the plugin folder and do go run -trimpath ../k6 run --plugin leftpad.so ... but this is some kind of go module stuff where it actually compiles it in some strange way because you are in the plugin directory. If you are not it will not work :(
5. My initial tries of -ldflags="-s -w" doesn't seem to matter but I haven't done cross computer compilation with it :D So it might break something
6. While go.mod makes this seem ... harder it is actually making things .. work correct. As otherwise, you can be using a different version (not that this isn't impossible even with them ... but it will probably be much harder). I am pretty sure the initial problems @mostafa was having with getting the lib.State was because he was using different versions of k6 or at least different import paths.
7. This means that if k6 and a plugin share a library, the library will need to use the version of k6 no matter what, this is something that if we weren't using go modules would've not been required and would've been fine as long as no object from the plugin version needs to put in place where the k6 version is used or vice versa ... Which is what I think happened with lib.State as I mentioned above.
8. From my reading of some issues, the replace trick might get broken (?) and there are apparently things that will change around it which might break this whole thing.

As a whole, I think this is not a great solution given all of the above problems and the fact that it doesn't support windows and with all of those and the requirement for CGO_ENABLED=1 leading to even more k6 builds I think this puts the last nail in the ⚰️ for this approach until there is some movement on the golang side...

While the xcaddy approach has bigger technical expectations from users, I don't think it will be impossible to (in the future) make it more encapsulated, while it does make "plugin" development much better, especially in the sense that you will get a lot of the above problems fixed. Notably:

1. it works on windows
2. because of the fact it more or less makes another module dependency of k6 and then builds k6 it is possible for a module to use an updated version of a library as long as it is backward compatible (which given go modules and semvers should be expected).
3. CGO_ENABLED=0, -trimpath and all other flags we(or someone else) might want to use have no reason to break any more than with normal k6( obviously if the plugin will, for example, require C than it won't be possible ;) )
4. Removes the requirements for placing k6 in a specific place, which will likely make things harder for some users to develop plugins or make it impossible (windows, albeit there are other problems there as it already would've required docker to run)
5. Requires small changes to k6 that are likely to be beneficial even outside the context of plugins. Which means that if we decide to abandon this we won't really break compatibility ;)

Thanks a lot to @andremedeiros for getting the ball rolling(and the implementation), no matter what the final decision/solution will be.
Also Thanks to @mardukbp to giving us some starting points to see what different problems with this would be
And of course to @mostafa for trying out the system early and hitting some problems, hopefully, whatever we decide to implement (if we decide on something) will be able to benefit from his work :)

[andremedeiros]

So before I go and rebase the PR, what's the plan?

[na--]

@andremedeiros, we're re-evaluating the way we want to go forward with k6 plugins. We still want plugins, or at least a way to allow users to more easily extend the k6 functionality, but unfortunately Go plugins don't seem to be the best way to go about solving this issue... 😞 There are just too many technical hurdles and limitations - for plugin authors, for plugin users, and for k6 developers.

So, I'd say don't rebase the PR... 😞 We're not going to close it until we've fully committed ourselves to some other approach, but there's no need for you to spend any more time on it while things are in limbo.

We plan to investigate the xcaddy approach more thoroughly in the coming weeks. The current plan for k6 v0.28.0 (slated for release mid-September) is not to commit to any plugin/extension architecture yet. At most, we might refactor the k6 module registration slightly, to make it easier to extend k6, both in an xcaddy-like manner and as even a fork. This will allow us to work on an PoC for a xcaddy-like tool (xk6? k6bilder? something else) independently of the k6 release cycle. Hopefully by k6 v0.29.0 (~mid-November), we'll have an official and somewhat stable way to make and use k6 plugins/extensions.

Thank you for your huge contributions with the plugins and sorry again things are so complicated! Whatever we settle on in the end, you'd get credit in the release notes at the very least, since if it weren't for your suggestions and PR, we probably wouldn't even be talking about this now, much less exploring potential solutions.

[mardukbp]

I suggest calling the k6 builder 'pack6`.

[mardukbp]

I know there is an issue about k6-Telegraf integration, but I think it is relevant to bring up Telegraf in this issue since it is mostly plugin-driven. Maybe there are things to learn and/or reuse from their implementation.

[na--]

@mardukbp, a valid point, though as far as I know, all of their "plugins" are baked into the main repo. They just have some very clean interfaces for the different types of plugins (see https://godoc.org/github.com/influxdata/telegraf) and then a whole bunch of in-tree implementations of these plugins (see https://godoc.org/github.com/influxdata/telegraf#pkg-subdirectories, all that start with /plugins). Not that we can't learn from that, since I presume they can easily do an xtelegraf implementation with that clean-looking architecture themselves, but it doesn't quite fit our use case, unless I'm missing something?