Plugins in gotop

Why Did I Choose to Use Go Plugins?

Update 2020-04-09:

For context, please see issue 78.

I was hoping for some comment from the Go team to https://github.com/golang/go/issues/31354, but there's been no communication on the ticket. Some people are discussing it, but they're only users; any conversations the Go team is having on this is behind closed doors,or in another forum.

Since there's no indication about which way this will go -- or even whether it's considered a priority, or something we'll limp along with for another couple of years -- I've decided to change my short-term tactic on extensions in gotop.

I'm a fan of Caddy, and they have the same issue: how to let users enable extensions without bloating the executable. I discuss this more below, but what they've done recently with the major version change is publish a build tool. While this forces everyone who wants plugins to compile their own -- which I really hate -- it does provide a solution, and I don't see an alternative until the weaknesses in Go's plugin implementation are addressed.

So: I'll be changing the gotop extensions API and documenting how to compile extensions in, with the ultimate objective of writing a build tool to make finding and compiling a custom gotop executable easier. If and when the Go plugins system becomes usable, I plan to switch back to that.

In the meantime, read on for an overview of the problem and my rationale for wanting to use plugins.

Old explanation

gotop uses Go's plugin library for a subset of features. Go's plugin support is in its infancy, and currently has a number of drawbacks. It is extremely brittle and sensitive to version differences, which results in libraries that are tightly coupled to a specific build of the executable. It can be challenging to reliably build plugins that don't fail because of version differences. It's so problematic that alternative solutions thrive despite there being a canonical, built-in support for plugins. Articles are easily found that detail real-world negative experiences with Go plugins.

So why did I choose them?

As support for edge cases¹ grows, I face the problem of how to add these features for some without bloating the program for everybody. There are five options that I can see:

Every person for themselves

We can choose to not provide binaries, and force everybody to compile gotop themselves. This would allow us to put feature flags, in the form of build flags, on all optional features. If you want a feature, you enable it at build time. This is a tried and true approach used by generations of ./configure && make projects.

There are three downsides to this:

1. The main one is that everybody has to compile gotop themselves.
2. All dependency sources will still be pulled, even if it's not compiled into the binary.
3. It doesn't work for distribution package maintainers.

I find this approach odious.

Choose Your Adventure

A quite nice variation is an approach taken by Caddy: provide a build server and a web interface that allows users to select which features they want to enable. Then, dynamically compile a custom executable while they wait. This is a fantastic, clever solution with one downside: it present distribution maintainers the same conundrum as the first approach: they're left with the decision to either include the kitchen sink, or provide a binary that's crippled for some subset of people.

While nice, I'm not in a position to support or pay for a custom build server for the project.

Bundle everything

This would mean that as gotop grows and features are added, everybody gets them whether or not they use it -- and even when they can't use it. It's guaranteed bloat and effectively dead code for a substantial percentage of users. Examples are:

• Support for non-ubiquitous hardware, such as GPUs. Many people have Nvidia cards, but probably just as many or more do not.
• gotop over TCP. Most people won't have a use for monitoring remotely running VMs, or they will have other, better ways of doing it
• Integrated email reader. as Zawinski's Law tells us, gotop will someday be able to read email.

Ok, so that last one is fluff; I don't have a lot of plans for wild feature expansion at the moment, but I have two, which is enough to draw a rudimentary trend line. While I don't know what it is yet, I'm certain there will be at least three. I want to solve this problem up front.

More dependencies

We could use Hashicorp's plugin library, an alternative plugin library that has none of Go Plugin's weaknesses.

I like go-plugin -- I really like most of Hashicorp's stuff -- but I think it's not right for gotop. For one thing, it's a bit kichen-sinky already. It's a fairly big library, weighing in at 2,374 lines of code. That's 14% larger than gotop is right now (2,064 LLOC). Pulling in go-plugin would double the size of gotop. It also has a lot of features that are great in a company, but are not strictly needed for something like gotop.

Mainly, the size thing steers me away from this.

Use Go plugins.

So, I'm using Go plugins, with the fervent hope that the Go team's traditional rapid progress in improving core libraries will soon be applied to plugins/ and most of the negatives will fall away.

And this is a FOSS project. If you can't be a little bleeding edge and have some fun in your spare-time projects, then what's the point?