Plugin Mechanism

[delvh]

What should it be?

• it should allow adding additional functionality to Gitea without bloating the main binary or being responsible for maintaining it
• it should not allow to modify the core functionality of Gitea

Architecture

• Plugins are called using Gos plugin package. (see discussion below, was generally disliked)
• Plugins are called over RCP using the go-plugin package.
• Reasons:
  • active community to maintain the package
  • works on every platform and even across languages
  • minimalistic and understandable -> no technical debt to have to consider
  • OS is responsible for setting plugin resource limits, not Gitea
  • Protocol versioning built-in
• Reasons against that architecture:
  • requests over RCP make everything slow
  • extra dependency
  • vendor lock-in for plugins to go-plugin

Communication: Gitea ➡️ plugin ➡️ Gitea

• common data structures and methods needed
• recommendation: an extra package that both Gitea and the plugins use
  • during application startup, Gitea should use the package from the receiver side and the plugins from the sender side
  • while dispatching calls to the plugins, the roles should be switched around so that Gitea becomes the sender and the plugins the receivers
• things that plugins, as well as Gitea, should have access to (at best only as a wrapper that is set once at runtime because the underlying implementation can change quite often):
  • especially things located in modules/:
  • logger
  • context
  • i18n
• again, to avoid vendor lock-in, this package cannot depend on anything that it doesn't define on its own, or is provided by Go itself
• to reduce the chance of plugins breaking with a new Gitea release, this extra package should be developed to be "all-inclusive" from the beginning meaning that it entails everything a plugin might wish to configure
  • it is recommended that the extra package uses easily extendable data structures, i.e. map[string]?, where possible to avoid breaking changes

Communication: plugin ➡️ Gitea ➡️ plugin

• Let Plugins use the Gitea API when requesting additional information from Gitea. Reasons:
  • easy to implement (literally NO work at all 😃 )
  • easy to maintain (again, because there is literally NO code to maintain 😃 )
  • safe to use for the plugins because the API guarantees stability
  • API gets used more, hence more people want to see it well-maintained and there is an actual incentive to add an API for a feature that at that moment only exists for the web UI
• Reasons against this approach:
  • slow, because it will always require HTTP requests
  • plugins need to parse the answer instead of simply being able to use a predefined method/ datatype

Plugin Lifecycle (What the plugins can set/ "use")

• Plugin Initialization
• Application is completely initialized and running - Hook
• Plugin Dependencies (to Gitea (versions), to other plugins?)
• Setting Plugin-wide settings as well while parsing the server-wide settings (requires (a) section(s) reserved for plugins)
• Custom translation keys to allow for localized new features
• Adding custom tabs to different units:
  • most basic:
  • user
  • org
  • user+org
  • all repos
  • public repo
  • private repo
  • additionally possible:
  • issue
  • PR
  • projects
  • packages
  • ...
  • to make parsing this easier, it would probably be good to use an approach that allows setting a list of tags where this tab will be shown. Problem: How to handle OR/ AND relations (depending on the chosen algorithm)?
• Adding consistency checks
• Cron Job Hooks?
• Shutdown Hook

What data is needed by every/ some plugin(s)?

• unique plugin name (i.e. by following reverse naming such as io.gitea.example_plugin)
• human printable plugin name (i.e. Example Plugin V1)
• plugin version in semver form
• list of core units affected by this plugin (rather as a code of honor the plugin intends to keep than a security feature)
• dependencies to other plugins
• custom translations/ translation files
• custom settings/ setting files
• custom web routes
• custom API routes (problem: how to update the swagger file with the new operations?)
• provided hooks
• minimum Gitea version the plugin needs

Why this long text rambling?

1. Everyone should know without a doubt how it should look like in the end
2. To make the work for the (eventual) reviewers easier by simply letting them cross-check the actual functionality with its intention
3. Most of the text I wrote here can be copy-pasted into the docs, making it easier for me to write the docs
4. To allow for discussion if I missed something while planning this feature
5. To see if the scope others expect from this feature differs from what I imagine from it

[42wim]

I think https://github.com/hashicorp/go-plugin should also be taking in consideration, afaik the "normal" plugin system has a lot of shortcomings (eg crashing the whole process because of issues with the plugin), that the Hashicorp one does not suffer from.

See also:

• https://www.youtube.com/watch?v=SRvm3zQQc1Q
• https://alperkose.medium.com/things-to-avoid-while-using-golang-plugins-f34c0a636e8
• https://www.sobyte.net/post/2022-03/golang-plugin/

[delvh]

Yeah, I've also seen their implementation when looking for a plugin system.
However, to me, it appears as if they don't support executing dynamic functions as I think I read the sentence sending functions is not possible anywhere, and then I stopped reading and didn't investigate further as that is something we most likely need.
After having read through their docs for a little, it seems I was a bit too fast with that.

Regarding the unexpected crashes: I intended to call every plugin by using something like

func PluginCall(function func(plugin *Plugin)err) {
  for _, plugin := range plugins {
    executePluginCall(plugin)
  }
}

func executePluginCall(plugin *Plugin, function func(plugin *Plugin)err) {
  defer func(){
    if err := recover(); err != nil { // Error handling
    }
  }()
  function(plugin)
}

and thus mitigating the potential for those errors.

So, the question is now, should we use the RPC version which might offer a few benefits and higher latency, or do we stay with the Go native approach that we can completely customize to our liking?

[Gusted]

However, to me, it appears as if they don't support executing dynamic functions as I think I read the sentence sending functions is not possible anywhere, and then I stopped reading and didn't investigate further as that is something we most likely need.

Do you want to compile/execute Go code on-the-fly?

[delvh]

Not really, but we certainly need to handle the custom routes which can best be achieved dynamically, and making this interface-based might be possible, but (a little bit) more work.

[Gusted]

Wish you many luck with this advanced logic.

[lunny]

I prefer the plugin system based on javascript or nodejs because that will allow it include frontend code.

Possbile library be used. https://github.com/dop251/goja, there is also a plugin support nodejs .

[delvh]

Interesting how many different views there are regarding this feature.
Now we have three completely different options and no agreement on what to use...

[lunny]

Interesting how many different views there are regarding this feature. Now we have three completely different options and no agreement on what to use...

Use javascript based plugin system, we can implement a plugin market and one click installation. That's impossible for Go based plugin system.

[delvh]

The problem I can see is: We do not even have the same definition of what we want in the end:
What you want seems to be a per-user plugin mechanism, what I thought of was a per-instance plugin mechanism...

[lunny]

The problem I can see is: We do not even have the same definition of what we want in the end: What you want seems to be a per-user plugin mechanism, what I thought of was a per-instance plugin mechanism...

I don't think so. I think I'm also talking about per-instance plugin. If I made you confusing, I'm sorry.

[jolheiser]

-1 for the go plugin package.
It can't be removed per compatibility promise, but it is absolutely not stable, and there are far too many caveats to get it working properly.

[lunny]

I think we can begin from a theme-kind plugin, which is simple and useful. A plugin could be a zip/tar file with a main.js or index.js and other files like theme.sass, theme.css and images and etc.

[wxiaoguang]

I do not think the Go's plugin package is the future. Is there any success story of using it?

Most large Go projects uses gPRC for inter-process commutation, including plugins, but it's also heavy.