Feature: allow plugin sources other than GitHub

[digitalfiz]

We have our own private gitlab instance and it would be great if we could keep our plugins private and pulling locally. Can you add support to pull from the same places at terraform can for modules?

[bendrucker]

Can you add support to pull from the same places at terraform can for modules?

Short answer is no, but it would be helpful to at least collect some more info here. We can probably provide support for private GitHub installations but I don't anticipate TFLint will attempt to support anything other than GitHub for plugin discovery.

We have our own private gitlab instance

Don't know what to make of this, hopefully you can read the info below and explain what you're actually expecting.

Bit of Terraform background:

• Terraform registries are hosted services that conform to the registry API. It is entirely up to the implementer to decide where to store data and how to ingest new packages.
• The public Terraform registry, aka registry.terraform.io, is run by HashiCorp and only supports GitHub sources
• Registries can host modules and providers. The public registry uses the GitHub releases API to find available module versions. For providers, the registry expects the release to contain built artifacts, i.e. the provider plugin binary for each supported architecture.
• Incidentally, GitLab seems to offer their own module registry but not a provider registry, maybe this is what you're referring to?

As far as TFLint, there are some light similarities, but the implementation of the plugin "API" is entirely different. There is no hosted service. Release data is loaded directly from the GitHub API. There is no intermediate layer that serves module and provider data to clients. With Terraform's registry API, you could at least run your own registry proxy that sources release data/binaries from GitLab. TFLint's plugin system doesn't allow for that because it doesn't define its own API.

Supporting GitLab would mean building a plugin discovery system with multiple clients for different VCS providers. With limited maintainer resources, that's probably not something we're going to tackle.

[digitalfiz]

A team member of mine is working on adding gitlab support because the apis for releases are very similar so it should be possible. He got it working with gitlab directly but I asked him to make it work for both so we aren't forced to fork it for our needs.

I wasn't aware of how the plugin system works but after playing with it for a few hours I understand quite a bit more about how it works.

Would you be willing to accept a PR that adds support for gitlab?

I'm sure we can also add some gitlab/gorelease examples to the plugin template to show the different possibilities.

[bendrucker]

At this stage, I don't think it's appropriate for TFLint to be taking on support for another release API. The plugin discovery system is very new and adding support for a VCS provider that Terraform's own registry doesn't support seems like a very niche feature.

Supporting a custom GH URL would provide users a path for whatever custom plugin host they want, whether that's GitLab or any other server. You'd be able to write a proxy that behaves like the GitHub releases API. The indirection is not great, but there's some prior art around this idea:

https://github.com/philips-labs/terraform-registry

Would you be willing to accept a PR that adds support for gitlab?

In the abstract I'm not really persuaded that there is enough value for the community (beyond yourself) but submitting the code would be the best way to highlight how minimal the required changes are, if that is indeed the case.

[bendrucker]

TFLint also exposes objects that are involved in configuration parsing, so you could conceivably write your own binary (replacing tflint --init) that could resolve/download plugins from sources other than GitHub.

[digitalfiz]

We use gitlab every day for our modules so I'm not sure why you keep saying it doesn't support it?

I understand not wanting to support extra systems. But personal I think adding support for both public/private github and gitlab instances would be an extremely useful addition to this great tool.

I imagine since there really isn't an alternative outside of shoving some validation stuff into tfsec there really isn't an alternative to this featureset. The need for this project will grow and with it the userbase and their diverse setup of systems each with their own requirements. Protecting IP will be a big issue with any project like this so supporting private instances will be a must.

Maybe we can put up a PR anyways for you to have a gander at?

[bendrucker]

We use gitlab every day for our modules so I'm not sure why you keep saying it doesn't support it?

The public Terraform registry that HashiCorp runs only supports synchronizing modules and providers from GitHub. Hashicorp has effectively decided that supporting only GitHub is good enough for the growth of open source development for Terraform modules/providers. TFLint chose GitHub for similar reasons, it's where the community is.

HashiCorp made the smart investment in a dedicated, well-specified registry service so that other products/implementations can do things differently. TF Cloud and TFE's private registries support other VCS providers. GItLab will run a module-only registry above, I wasn't aware of that until this issue, but you'll see I linked it above. Or you can do something clever like Phillips and write your own server with whatever artifact backend you'd like. HashiCorp is a multi-billion dollar company—building a registry is the right choice for them, but probably not for TFLint with a sole lead maintainer and me helping out.

Maybe we can put up a PR anyways for you to have a gander at?

Yes, this is/was always welcome! But FWIW the comments about how companies are gonna need this are not, TFLint is volunteer-maintained, well-documented, and extensible, there is no "must" in OSS.

[scabana]

FIrst, thanks a lot for maintaining TFLint! I would have loved to see the same api as the provider api in the terraform registry. Afterall, it's the same pattern. Too bad custom "provider types" can't be added to the terraform registry api. At least, Would have been nice to see the exact same format for the files for custom TFLint plugins. At a minimum, allowing a custom github api dns name would help quite a bit.

[bendrucker]

This is considered in my comment right above yours:

HashiCorp is a multi-billion dollar company—building a registry is the right choice for them, but probably not for TFLint with a sole lead maintainer and me helping out.

Hosting a public registry comes with real challenges, both technical and organizational, that require staffing. Scaling for user load is hard enough. Dealing with abuse, name disputes, etc. is equally hard. It's not impossible to do this with community funding instead of a for-profit company but recruiting the sponsors to make that possible is very hard.

At a minimum, allowing a custom github api dns name would help quite a bit.

Easy enough, we can/should support GitHub Enterprise (#1643).

[wata727]

One challenging idea might be a way to support more abstract storage rather than GitHub API.

For example, there is a project called OCI Registry as Storage. This is a project aiming to use OCI registries as generic artifact stores, and by complying with it TFLint will be able to download plugins uploaded to any OCI registry (GitHub Container Registry, GitLab Container Registry, ...etc.).

Trivy seems to put this idea into practice. Trivy has the ability to install WASM modules from an OCI registry.
https://aquasecurity.github.io/trivy/v0.36/docs/advanced/modules/
https://github.com/aquasecurity/trivy/blob/v0.36.1/pkg/module/command.go#L25

There's a lot to investigate, such as whether this provides a comparable experience to current functionality, but it's not a bad idea. I don't strongly promote this idea, but it's worth considering as an alternative.