Support a plugin infrastructure

[NiklasRosenstein]

It would be very cool if we could write plugins for Gogs to add features to the web interface. These should be easy to deploy, too.

[unknwon]

Plugin is a quite big topic... I don't have any idea. 😵

Please give more specific details.

[NiklasRosenstein]

Some examples that I believe would make sense to implement as "plugins" or "add-ons" rather than being merged into the code base:

• Import from other Git hosting services (eg. GitHub, BitBucket, GitLab including Issues and Wiki)
• Overall new features/apps (eg. Code Snippets)
• Third Party Issue Tracker integration

and stuff like that. It would make contributions of new non-core features easier as it would not need a merge into the codebase.

[unknwon]

Thanks your info!

Import from other Git hosting services (eg. GitHub, BitBucket, GitLab including Issues and Wiki)

I suggest use APIs instead.

Overall new features/apps (eg. Code Snippets)

The Code Snippets will eventually be implemented, so... any other examples?

Third Party Issue Tracker integration

How deep the integration would be?

Sorry I don't have a clear idea about plugin thing, never done that before and sounds too general to me.

[bkcsoft]

With regards to #2507 we now have a valid reason for a plugin-system. Since I've written a few of those I'd like to express my take on this from a purely architectural point of view.

First of, I'd recommend setting it up as external services that plugin to Gogs REST-APIs, so it would use the regular API for accessing gogs-data, and expose its own API (standardized of course) for injecting data (menus, panes and configuration). This would make them easy to create, install, upgrade, and manage separate from the Gogs-installation, and you could have them on a separate machine/VM/container/whatever so as to keep separation of concerns clean (IIRC this is a huge concern for PCI-validation which larger companies might have to have, though I might be wrong). Another good thing is that you don't have to restart Gogs to (un)install plugins.

Second thing is that you'd have to be able to setup authorization on each plugin (authorization as in what the plugin is, and isn't allowed to do; and with what). I'd recommend that each plugin is sent an individual token on initialization, which the plugin uses for talking to the Gogs-API, and that Gogs would use for accessing the service-API. Tokens should be changed on each initialization (tokens could be changed after a fixed time as well, though it's not a necessity, more of a recommendation).

Each plugin would tell Gogs what it does (in wide terms), as in intent="issues", inject="repo" for an Issue-tracker; or intend="repos,issues,milestones,orgs,teams", inject="global" for system-wide mirroring services (mirror everything to e.g. Github/Bitbucket). intent would tell Gogs which parts of the Gogs-API it wants access to, and inject would tell Gogs which level of authentication it would need; such as per-user, per-repo, per-orga, and global (for admin-rights). Gogs would obviously have to have local settings for this as well, one might not want it to be global but per-repo, one might only want it to mirror repos and issues but not the rest, etc etc. So Gogs would have to tell the service what the service is allowed to do.

Now, the biggest issue with this approach, is that the Gogs-API isn't done (not even close to done unfortunately), so the API would have to mature before this could be implemented.
The second issue I see with this is injecting menues and panes, not sure how this would be handled with the current templating-system Gogs uses, but that is secondary to the API-issue...

Anyhow, that's my recommendation and should be taken as such. Feel free to comment/bash on anything I've said and I'll try to answer as best as I can 😃

[NiklasRosenstein]

I like the idea with plugins over a REST API. This would also allow to implement them in any language. I only have concerns regarding the speed of the communication. Wouldn't there be much overhead if the server is talking to the plugin via REST (eventually on a different server?) and only then sends the resulting webpage to the user?

[andreynering]

Writting a plugin infraestructure is hard and I don't know if it is worth. I am not the best person to propose something, but will try to do some brainstorming 😄.

Backend

An alternative to what @bkcsoft proposed would be doing it with Go itself. Since Go doesn't support dinamic libraries (.so or .dll) a recompile would be necessary. Gogs would have to have a Plugin API to make possible to register callbacks for specific actions:

package plugins

import "github.com/gogits/gogs/plugin"

func init() {
    plugin.RegisterPlugin("myPluginName", func(ctx *plugin.context) {
        ctx.InjectHTML("anyTemplate.tmpl", "header", func() string { return "<foo>bar</foo>" })
        ctx.Route("/foo/bar", func() { ... })
        ctx.OnPushRepository(func(repository) { ... })
        // etc...
    })
}

This would be hard work, though.

Client side

Doing simple things in the client side would be much simpler. We could have a folder where every .js and .css file would be automatically imported in the HTML. This would permit simple tweaks with jQuery or whatever.

Also, we could have an alternative public folder which would override the existing one. Example: I want to change the favicon.ico, so I would just paste it there and Gogs would serve that one instead of the original (on /public folder).

Injecting templates

Another folder could exists to inject templates on Gogs. Example: new_issue_header.tmpl would be automatically injected in the header of a new issue page, etc.

[bkcsoft]

@NiklasRosenstein Speed is obviously an issue when talking about micro-services (which is what this is called BTW). On the other hand, REST-APIs should be as light as possible, and should never send entire webpages 😉.

Information on where menus and panels should be could easily be send on initialization and stored by Gogs so as to not ask on each page-render "does anyone wanna render here??". And you could most likely send and cache templates as well for rendering panels. So the only data that is sent at run-time is the actual data that changes 😃. Though for some things (like code-comments which might wanna render after any line) this might still be an issue, not sure how to implement such a feature as a pure template-fix. Though overriding templates as @andreynering suggested could work here as well (though not as secure IMO)

On a less serious note, neither Amazon or Netflix has any real performance issue with their micro-services, so I'm guessing that even a fairly large plugin for Gogs would not be an issue 😉

[bkcsoft]

@andreynering Actually it wouldn't take much to implement this within Gogs (except for the lacking API...). I could probably implement a rough draft over the week.

There are X key components that needs implementation:

• Config storage (a model is required)
• Initialization (token generation, exchanging configs, storing templates)
• And templates needs to be changed in some places to allow for menu-injections

And since Gogs already has token-based auth, there's no need to put extra thought on security (other than at token-generation)

[andreynering]

@bkcsoft Sorry. Which alternative are you talking about? Doing it in Go or using a REST API?

[bkcsoft]

@andreynering REST-API.

A few thoughts on your ideas:

• Backend
  Generally speaking a good idea, and could be made simple and contained enought. Though like you say, you'd have to recompile for at least installing plugins (plugins could be disabled at run-time). Generally this isn't something you'd wanna do 😒
  Also the slight issue of having to merge back into Gogs-repo...
• Client-side
  This is gonna be a bitch if you consider stuff like cross-domain requests. And the fact that it won't allow for much other than injecting data into the rendered page. (no mirror-service, no issue-tracking, etc)
• Templates
  This part I liked, and recommend even for my approach 😃

[unknwon]

My initial idea seems too big right now... I was thinking to support both code modifications and REST APIs...

[andreynering]

Backend

Generally speaking a good idea, and could be made simple and contained enought. Though like you say, you'd have to recompile for at least installing plugins (plugins could be disabled at run-time). Generally this isn't something you'd wanna do 😒
Also the slight issue of having to merge back into Gogs-repo...

This would require the user to manually download the plugin files, paste in a specific folder (package) and compiling. Not much intuitive, but it is an idea. And it would be very powerful. Maybe there's a better alternative to link Go code?

Client-side

This is gonna be a bitch if you consider stuff like cross-domain requests. And the fact that it won't allow for much other than injecting data into the rendered page. (no mirror-service, no issue-tracking, etc)

I didn't get what would be the security issue here. If you could change the HTML, why not the CSS? (and maybe JS, too). It would be necessary to create custom themes (see below).

Templates

This part I liked, and recommend even for my approach 😃

Relevant issues: #1956 #1126

[bkcsoft]

@unknwon well, with a complete REST-API the compiled version wouldn't be necessary, since everything would be exposed over an API.

Though most of this could be implemented with webhooks, though you couldn't inject the returned data into Gogs (yet... that could also be possible)

[unknwon]

I am thinking using gRPC or protobuf, might be faster?

[bkcsoft]

@unknwon well, protobuf is just a packer/unpacker so it wouldn't do that much difference. Websockets is more interesting since it can be used from within the browser as well (live updating of comments etc)

WRT auth-plugins (talk from gitter, to mention it here as well), this I wouldn't be comfortable with putting in a REST-plugin and should be implemented like @andreynering mentioned with compiled plugins into Gogs itself... Mostly because it would be a PITA to have multiple authentications, one for plugins and another over a plugin for users and whatnot...
There also isn't much necessity with being able to swap them out easily, since one usually don't change the auth-setup that often 😆

[bkcsoft]

@andreynering WRT #1956 and #1126. Themeing shouldn't be implemented by changing the templates, but rather by changing (or appending) css-files. (see this )
And I don't really see any point in being able to change or override templates, since they're tightly coupled with the template-model (the .go-files in /routers/ 😃 )

And for those that wanna change them anyway, you can do that today, though you have to replace the existing one 😩

[andreynering]

@andreynering WRT #1956 and #1126. Themeing shouldn't be implemented by changing the templates, but rather by changing (or appending) css-files.
And I don't really see any point in being able to change or override templates, since they're tightly coupled with the template-model (the .go-files in /routers/ :smiley: )

And for those that wanna change them anyway, you can do that today, though you have to replace the existing one 😩

Yep. We have to think whether we want to be possible to replace the entire template or just inject more HTML on the existing templates.

See my comment above:

Another folder could exists to inject templates on Gogs. Example: new_issue_header.tmpl would be automatically _injected_ in the header of a new issue page, etc.

Agree that for theming CSS would be better.

[bkcsoft]

Well, I'd want to both inject templates into existing templates, and inject entirely new ones 😄. Though never replace, don't think that a plugin should be allowed to TBH...

On the topic of "not worth it", since Gists could be implemented thusly, and both Issues and Wikis could be migrated to such, I'd say it's worth it.

Reasons:

• Less cruft in the main application.
  Everyone might not need issues/wikis, and they both take up space, and add complexity to Gogs as a code-base
• Easy to implement new features; such as injecting build-statuses to PRs, and Github issue-mirroring (they require a token which Gogs can't provide at the moment without implementing it as a full feature in Gogs)
• If the notification-system is exposed throught this, it would be dead easy to implement new webhooks as needed

So I'd say that it's very much "worth it" 😛

[bkcsoft]

@NiklasRosenstein @andreynering Feel free to comment on this 😃
https://docs.google.com/document/d/1DB7quADTPpQphGTtOEEcaDoEt0Spo6BIVd6c4vyOrz4/edit?usp=sharing

[andreynering]

Just some brainstorming.

An alternative to allowing plugins to be written in Go, would be using a scripting language:

• Anko: a Go-like scripting language: https://github.com/mattn/anko
• A JavaScript interpreter for Go: https://github.com/robertkrimen/otto

Pros:

• Both allow runtime execution, so no Go environment require to write plugins. It would even be possible to install/update scripts without requiring a Gogs restart
• Both allow register functions and variables to be available inside the script. This would made the interaction with the Gogs codebase possible

Cons:

• Performance penalty compared to using native Go

[tboerger]

Caddy is an example how to integrate plugins, it's compiled into the binary as already suggested, you extend the source similar to caddyext.

[tboerger]

For further reference https://github.com/caddyserver/caddyext

[shytikov]

Hi,

I'm not a Go guy, so maybe my implementation idea how to work around dynamic loading issue in Go will face some issues during actual coding, but what kind of possible problems you can foresee, if:

1. Gogs should be statically linked with proxy library written in C;
2. Proxy library would have the only functionality — dynamically load so plugin files created in Go;
3. Library will try to transparently proxy all calls to plugin.
4. PROFIT (???)

I believe it could be some memory leaking on C side, but other than that it seems to be feasible. Or I missing something?

Can you share your thoughts? Thanks in advance!

[tboerger]

The most idiomatic way should be a RPC based plugin system. There are already multiple examples in the wild. Famous examples are terraform and packer from Hashicorp.

[bkcsoft]

One step closer 😈 https://tip.golang.org/pkg/plugin/

[tboerger]

One step closer 😈 https://tip.golang.org/pkg/plugin/

I still prefer the RPC based plugins like the plugin system of Hashicorp :)

[Kaliamos]

Hi,

I would like to add some features on my GOGS. I can see on previous comments that it is not yet possible to add plugin.
Do you know when it could be possible to write plugin ?

In fact, I have realised a little script to verify the syntax of my code. Then, I have added its on the suitable hook, and this works. However, currently, the return information of my script are written on a log file on my server. It is not convenient. I have to logon on the server to know this information. So, I would like to add a way to display on GOGS this information, for example a button or a popup or whatever. How can I do this ?

Thank you so much,

[lavvy]

http://gitbucket-plugins.github.io/

[cornfeedhobo]

I would like to sidecar package repos. like, add an option for releases to get published to a built-in repo, e.g. apt, pip, etc