Load external .so plugins, fixes #1717

[Timidger]

Original patch written by sparrc, brought up to date by timidger.
Original from 92d8a2e message follows:

support for the Go 1.8 shared object feature of loading external
plugins.

this support relies on the developer defining a Plugin symbol in their
.so file that is a telegraf plugin interface.

So instead of the plugin developer "Adding" their own plugin to the
telegraf registry, telegraf loads the .so, looks up the Plugin symbol,
and then adds it if it finds it.

The name of the plugin is determined by telegraf, and is namespaced
based on the filename and path.

see #1717

Required for all PRs:

• Signed CLA.
• Associated README.md updated.
• Has appropriate unit tests.

Since this adding a new capability to inject plugins I wasn't sure where to document this or how to test this. So looking for feed back on how to do those things 😄

[Timidger]

ping? Does anyone know a good way to test this or refactor this so it can be tested and merged?

[danielnelson]

@Timidger I'm not ready to give this change a thumbs up or down yet, we are still trying to decide in which ways we want to support these types of extensions, and don't want to provide any methods officially that we aren't prepared to support.

[mtp401]

@danielnelson I think supporting some sort DSO based interface like what
@Timidger has implemented here is incredibly important for most IoT/embedded
scenarios for a couple of reasons. Firstly, in these types of resource
constrained environments the cost of everything really counts. The existing
generic methods for integrating with telegraf (exec, socket input, etc) are all
relatively expensive compared to an in memory solution. In addition to
performance concerns, the state of the art in most of these environments is
either C or C++. It would be much easier to adopt Telegraf into these
environments there was a straightforward way of integrating plugins written
in one of these languages. For most other projects, this integration point is
usually a C FFI.

Is there some existing preferred direction for an extensions API that looks
like this?

[danielnelson]

@mtp401 Thanks, I appreciate you taking the time to provide this feedback. To be sure, I agree about the value of calling into native code and that it is a big limitation right now.

When this code was originally written we ran into some issues that required Telegraf and all plugins to be compiled together; it didn't seem possible to compile ones own plugin and then use it with official builds or use a compiled plugin on multiple Telegraf versions. I don't think we looked at plugins using CGO, I imagine they would require both Telegraf and the plugin to be compiled with CGO_ENABLED. There were other shortcomings when using libraries from pure Go plugins and the lack of Windows support meant we couldn't move anything standard into plugins. Given these limitations, at the time I didn't feel like the solution was a large enough improvement over creating a new plugin in a fork to be included.

I do encourage you to compile with this code report back with any feedback you have, it is always good to gather experience with using a plugin.

[Timidger]

I do encourage you to compile with this code report back with any feedback you have, it is always good to gather experience with using a plugin.

There's an example of how I plan to use it here, which is from this pr using this glue code that relies on these changes.

For the IoT/embedded usecase that @mtp401 mentioned generally some sort of cross build tool is used that compiles from source the entire target image. Thus the versioning limitations you mentioned wouldn't affect this usecase since telegraf would essentially be vendored (and thus always at one version).

For other projects that want to use this feature they can weight the limitations vs the benefits. If being tied to one version and compiling with CGo is not an option then there's always the exec functionality. However, if they are working in a limited environment the ability to avoid expensive logging by doing everything in a shared library is a huge boon.

[danielnelson]

If you are building the plugin and Telegraf together it works quite well, however I think it also reduces the need for the plugin code quite a bit. For example, I think this would work and only requires a single line change to Telegraf itself, give it a try if you have time:

Add a init function to your glue code:

func init() {
    inputs.Add("rust_plugin", func() telegraf.Input { return &RustPlugin{} })
}

Then add the plugin import path to Telegraf's input all.go:

import (
	_ "github.com/StarryInternet/telegraf-plugin-go-glue"

Compile Telegraf and no need to compile a go-plugin.

One thing I'm considering is just to always load all go-plugins from a directory, this could allow you to skip the one line change and also allow third party sql packages to be loaded as described here, without committing to a particular API on the Telegraf side.

[Timidger]

I tried your suggestion and that does seem much simpler. I don't know about the SQL use case that well, but I know it definitely solves our use case with minimal support needed from telegraf itself.

If that is a solution you are willing to purse I can close this PR and implement that instead. Let me know what you decide.

[danielnelson]

Yeah let's do it, we will have the plugin directory option and just try to Open any plugins in there. We won't do any special handling of the plugin, leaving them totally free form for now.

[Timidger]

Addressed your comments. Just to double check we should still be trying to load the Plugin structure from the go plugin, like I had it before?

[danielnelson]

For now let's only Open the plugin, and plugins can use the init() function to register new plugins.

[Timidger]

Got it, will fix this up on Monday

[Timidger]

Ok, the plugin is now simply loaded and it is expected for the init method to handle all injection into the input plugin list.

[adrianlzt]

Why is the var name defined?
Looks like is not being used.

Maybe it was thought to register the plugin with that name?

[adrianlzt]

Also, TOML doesn't like names with slasesh:

[[inputs.extplugins/ext-kernel]]

2019-08-05T11:01:53Z E! [telegraf] Error running agent: Error parsing prueba.conf, line 5: invalid TOML 

[Timidger]

The name is leftover of the previous implementation, you're right that it should be removed.

Also, TOML doesn't like names with slasesh

You can include a slash using ""

[[inputs."extplugins/ext-kernel"]]

[danielnelson]

You probably already have this figured out, but I removed the name code when merging.
Plugins can be loaded using the name specified in the registration function called from init().

I also have an example that I was using for testing over at https://github.com/danielnelson/telegraf-plugins