Plugin system

[okushchenko]

Plugin system

The plugin system is implemented using https://github.com/hashicorp/go-plugin, which is successfully used in Terraform, Packer, Nomad, and Vault. It's based on either gRPC or net/rpc over the local network connection. Plugins are Go interface implementations, which maps quite good to the current backends implementation.

The core set of plugins is defined in builtin/ directory. They are compiled in the main binary.

Builtin plugins are called using the special cli command:

confd internal-plugin <pluginType> <pluginName>

The only pluginType in use is database.

Plugin discovery mechanism will look for binaries named using confd-database-* format in the following locations:

1. A path where Confd is installed
2. A path where Confd is invoked

To preserve backward compatibility, the configuration is passed to built-in plugins in the form of a map[string]string as a parameter to the Configure method.

A third-party plugin should manage its own config directly. The map[string]string passed to a third-party plugin in Configure method will be empty.

Interface changes

The new interface which plugins implement has the new method Configure, which is used to decouple plugin configuration from the core of Confd. This method is called to initialize a plugin with a config. A config is passed to a plugin in a form of map[string]string. Config deserialization is done using github.com/mitchellh/mapstructure, which is a simple and a safe way to convert a map to a Go struct. A plugin may extract values from a map directly, but this behavior is unsafe and discouraged.

stopChan and waitIndex parameters were removed from WatchPrefix method because they are redundant.

type Database interface {
	GetValues(keys []string) (map[string]string, error)
	WatchPrefix(prefix string, keys []string, results chan string) error
	Configure(map[string]string) error
}

How to write a plugin

Any builtin plugin can be used as an example of how to write a plugin for Confd.
All builtin plugins can be compiled to external plugins by running go build from the respectful plugin directory in builtin/bins. For example, to compile env plugin to an external plugin:

cd builtin/bins/env
go build -o ../../../bin/confd-database-env-test

The code which is used to run a plugin is very simple:

package main

import (
	"github.com/kelseyhightower/confd/builtin/databases/env"
	"github.com/kelseyhightower/confd/plugin"
)

func main() {
	plugin.Serve(&plugin.ServeOpts{
		DatabaseFunc: env.Database,
	})
}

You can use generated confd-database-env-test plugin binary like that:

./bin/confd --confdir ./integration/confdir --backend env-test --one-time

[kelseyhightower]

After reviewing this I feel that this approach introduces a tone of complexity for such a simple tool. confd was supposed to be simple -- take a key/value pair store and generate a template.

What is really required here is either scope: which backends will confd support, or one generic backend that will allow people to build shims to add support for everything else. I'm thinking the shim can either be an exec or a simple REST API that allows confd to fetch the key/values it needs to run.

Thoughts?

[okushchenko]

This plugin system is kind of like an exec one, the only difference is that communication between a plugin and a host is done using a local unix socket (can be a tcp socket instead) and not stdin/stdout. The main confd binary will exec a plugin, launching a net/rpc server to communicate with. When you are writing a plugin you are implementing a Go interface, as it was done before. All complexity of host-plugin communication is hidden away. The key is that this plugins system doesn't introduce any changes to the way confd should be started or configured so is backward compatible.

The new version of hashicorp/go-plugin uses gRPC instead of Go net/rpc. It allows plugins to be written in any language. This PR can be updated to use the new version of hashicorp/go-plugin, but it will require changing a channel based WatchPrefix method.

Also, I don't think that this implementation is by any mean complex, it's under 500 LOC. It reuses an established plugin library (hashicorp/go-plugin), it's easy to test, understand and build on.

[okushchenko]

I've switched the plugins system to gRPC transport. Also, I've updated the documentation with an example on how to write a plugin, added more clarifications about the plugin system.

[abtreece]

What is the status of this PR and the plugin system as a whole? I've been looking for ways to contribute to this project and it seems a lot of things are on hold for this. Has @kelseyhightower given the approval to move forward with it?

[johnwards]

I've attempted to review this, however it looks like that the biggest block is @kelseyhightower's acceptance that this is the correct direction he wants the project to move in.

Kelsey, I know you are a super busy man, is their anything we can do to help?

From my teams perspective we could assist in re-writing some of the backend plugin's we use to this new architecture if that is appropriate, for example.

[okushchenko]

@abtreece @johnwards let's have a discussion anyway. I don't see any other plugin system implementations on the horizon and I've put quite a lot of work into this one to give up on it so easily 😉

[abtreece]

From my perspective, albeit limited, I agree with both @kelseyhightower and @okushchenko

The current list of backends confd supports is 11 and there are currently 6 PRs for 5 additional backends. I do not think it prudent to continue accepting every new backend for which a PR is submitted. However, AWS Secrets Manager, GCP Metadata, Azure Metadata and certainly Kubernetes are stores that would provide confd useable parity across the major platforms. Thus scoping the backends confd will support is definitely needed.

That said, if confd is going to limit the supported backends it should also implement a plugin system. I appreciate the desire to keep it simple, but confd has perhaps become a victim of its own success. Barring a simpler way to implement a pluggable API I think go-plugin makes sense. And of course there are good examples of its use in Terraform and Packer.

[okushchenko]

@abtreece I have an idea of implementing metadata providers outside of the current backends. The metadata is always needed and is relevant to the cloud provider or the OS used, not the key-value store, e.g. Consul or Etcd.
For the new backends, I would suggest to implement them exclusively as plugins first and then, if they will become mature enough and wide-used, they could be merged into the main repo.

[patrickdevivo]

Just adding a +1 to this plugin system. I would love to be able to add a backend implementation as a plugin and would love to get a sense of the direction of this PR and how this project will handle support (or not) for a variety of backends

[abtreece]

@okushchenko what needs to happen to move this PR and subsequently this project forward? This PR and Confd as a whole is starting to feel a bit neglected at this point. I am using confd extensively for my primary job and would like to see it continue to advance. I'm sure many others feel similarly.

[vkristjan]

@okushchenko what needs to happen to move this PR and subsequently this project forward? This PR and Confd as a whole is starting to feel a bit neglected at this point. I am using confd extensively for my primary job and would like to see it continue to advance. I'm sure many others feel similarly.

+1

[justizin]

Bumping this to echo the feelings of stagnation. We are using a severely out of date fork based on #672 in order to have GCE support, it's frustrating that both: (a) no new backends will be accepted, and (b) the plugin system is deemed undesirable.

I understand @kelseyhightower's concerns about growing complexity, but it feels a bit like that ship has sailed. To have 11 supported backends in core with no path forward for currently unsupported backends is a frustrating sort of limbo.

I'm more than happy to try and help with necessary work, at a minimum it's clear some tedious rebasing would be necessary, since this change is over four years old.

UPDATE: sounds like we are no longer relying on the fork, so our concern is less urgent, though this still seems like a good idea.