Plugin architecture for storages

[marpio]

Is your feature request related to a problem? Please describe.
There are still interesting storage options which we do not support i.e. B2. We cannot possibly support all storages in our code base but we can make it possible for the user to provide one i.e. through https://golang.org/pkg/plugin/

Describe alternatives you've considered
Implementing them like we do now but it will make the maintenance difficult.

cc: @marwan-at-work since you mentioned that on slack

[marwan-at-work]

Another alternative would be is to provide an HTTP interface to storages, almost an extension of the download protocol. And also we can look at how K8s does plugin extensions

[tylerchr]

I've implemented an http storage type in an effort to support an integration with a new storage backend (Artifactory) in a more generic way than simply integrating with that backend (this came out of a Slack conversation with @arschles). It works essentially as @marwan-at-work described, in that it maps the storage operations to generic HTTP GET/PUT/DELETE calls with BasicAuth, and looks pretty much just like the download protocol with some extensions. Perhaps that would be applicable or useful here?

My intention is to open an proposal issue and then a PR, though I've been held up somewhat (internal review by my employer). However that's nearly resolved—barring any surprises I expect to do that this week.

[arschles]

@tylerchr so awesome! Looking forward to this

I'm +1 on this over the Go plugin interface, so that folks can build storage interfaces and have flexibility to run them either on localhost or over a network

[tylerchr]

I opened #1130 (issue describing my needs) and #1131 (prototype implementation of an HTTP storage type).

Would be interested to hear your thoughts (@marpio, @marwan-at-work) especially since you're coming from the "storage plugins" angle which I hadn't initially considered.

[Kunde21]

Missed this when it was first discussed, but there's an option to use https://github.com/hashicorp/go-plugin and define a gRPC service as the interface. Then, storage plugins can be created and maintained in a separate repo, so long as they fulfill the service definition.

[marwan-at-work]

@Kunde21 the readme says:

While the plugin system is over RPC, it is currently only designed to work over a local [reliable] network. Plugins over a real network are not supported and will lead to unexpected behavior.

Is that outdated?

[Kunde21]

@marwan-at-work the readme kind of buries the gRPC option:

Cross-language support. Plugins can be written (and consumed) by almost every major language. This library supports serving plugins via gRPC. gRPC-based plugins enable plugins to be written in any language.

[arschles]

Is everyone ok with sticking with HTTP for now? It's less specification work for us because we can use the module download protocol we all know and love already.

More conversation at #1131 (comment) and #1110 (comment)