Extensionservice: Discussing the future of this interface

[davidgraeff]

To make the Eclipse Marketplace possible Kai introduced the interface ExtensionService.
Because the Marketplace will be shutdown at some point the future of this interface need to be discussed.

Personally I really require this interface for several things. I can go in detail (by copying my texts off the community forum about this topic) if @openhab/core-maintainers want that.

I made some changes to this interface though that I like to present soon, nontheless an interface like that makes sense for openHAB IMO.

WDYT?

[kaikreuzer]

The interface had been introduced in ESH to allow solutions to implement their custom mechanisms on how to provide add-ons.
It was targeted as a framework feature at build a proper solution, not as a solution feature to allow extending it in arbitrary ways.

For openhab-core, I absolutely agree that the interface should stay as it is useful for anyone building custom solutions.
Wrt the openHAB distro: Yes, we will need a replacement implementation for the Eclipse IoT Marketplace. As discussed somewhere else briefly, my idea would be to have a Git repo, which hosts the meta-information about external add-ons as a catalogue (referencing to external urls just like the Eclipse IoT Market, not hosting the binaries itself) and to which people can add their add-ons through PRs.
I would not want to accept any additional extension service in the openHAB distro - but people could certainly be free to use the "addon" folder to install additional 3rd party extension services.
I'd compare it to the PlayStore on Android: There is only one official store, which is curated by the platform provider, but if people download an apk of an alternative store, they are still free to install it on their device.

Does that make sense?

[davidgraeff]

I don't exclusively want the ExtensionService for java bundles. I need it to include an npm namespace into openHAB.

How do you want to distribute web-app otherwise? Or Json automation rules and rule templates? Or javascript and python automation files?

I have the feeling that you want everything to be packaged as java bundle (via maven coordinates?) which is unnecessary overhead and the most non-intuitive distribution way. Front end developers want to use npm and we want to attract new developers. Rule helper library developers (jython) want to use pip.

ExtensionService allows us to support all those different packaging systems and provide the user a unified way to install any of the above mentioned types.

I also think that the current karaf bundle is mostly undocumented and consists of an unfortunate mix of hardcoded configuration with a bunch (more than one) configuration file. Not well maintainable by any other then the original author.

ExtensionService allows to bypass that bundle to work with other mechanisms like OSGi subsystems or an alternative implementation that uses a simple initial repository configuration file with a StorageService backend to be altered dynamically during runtime.

[kaikreuzer]

No, I didn't talk about any specific packaging type. The rule template implementation is already using raw JSON files today.

[davidgraeff]

I should probably picture the work-flow that I have in mind for ExtensionServices. They should be provided by Core, not by addon developers.

Producers:

1. A community member creates a github repo where he starts to publish useful rule templates or script libraries
2. A fronend developer creates a user-interface and publishes to npm
3. A binding developer created a binding in his own repo and wants community members to test it

Consumers:

1. A user just adds the github repo URL in his openHAB Setup&Maintenance user-interface, because we provide a GitHub Crawler ExtensionService that can read some form of to-be-defined index file (package.json etc). Moments later he can install provided rule templates or rule scripts.
2. A user just installs a 3rd party user-interface, because we provide an npm ExtensionService
3. A Github releases ExtensionService allows users to add a repository and the service will scan the releases section for .kar files.

The current openHAB2-addons as well as OH1-addons should also be provided via ExtensionService and ExtensionServiceRepository, so that the user has maximum transparency where addons are downloaded from. ExtensionServiceRepository are configurable (polling time, enable/disable).

The idea is that each addon repository mainly publishes a karaf repository file.
And the distro is adding that one url per repository to a static ExtensionServiceRepository initializer.

And if we add more repositories (like the suggested addons-staging repository), it is the work of adding one url.

openHAB cannot stay the walled garden for normal users in my opinion (the techie user is already now able to change one of the three configuration files that define maven and karaf repos).

[MHerbst]

One of the most important features of openHAB has always been security. Installation from arbitrary sources would be a security vulnerability (especially for normals users). That's why smartphone allows the installation of apps by default only from their stores where they can check the uploaded apps for vulnerabilities.

In my opinion, especially npm is a high risk. In the past, there have already been some malicious JavaScript components that were distributed by npm. There is a recent study of "Security threats in the npm ecosystem" that comes to the conclusion

The overall conclusion is that npm is a small world with high risks. It is “small” in the sense that packages are densely connected via dependencies. The security risk are “high” in the sense that vulnerable or malicious code in a single package may affect thousands of others, and that a single
misbehaving maintainer, e.g., due to a compromised account, may have a huge negative impact

[davidgraeff]

One of the most important features of openHAB has always been security.

I don't see that. openHAB does exactly nothing for security. Otherwise it would use OSGi's bundle restriction features or karaf's security features or javas security features. I googled just a few days ago what could all be done with permissions.perm files per bundle etc.

You can restrict file access, socket access, url access, thread access, bundle management, and event-admin topic access. We could show the user exactly what a binding needs if we'd force bindings to ship the mentioned permissions.perm file (and actually kill bundle's that don't comply).

The last barrier were always the reviews. But think of the Eclipse marketplace. You can install random addons with no reviews at all.

The current situation doesn't scale. If I want to develop a binding in my own repo and make it available to users for testing, I like them to only copy&paste a repo link, or select it from a registry (like the Eclipse marketplace). If you run other people's software you are always depending on their good will. Does any non java developer can be sure that we don't hide malicious software in openHAB? Probably not.

npm would be used for UI's and wouldn't be any different than the marketplace. If there is really a bitcoin miner or similar in one of the UI's javascripts (which is the worst that can happen, cross-site scripting is no longer a topic), we can blacklist such a packet.

Btw, that can happen with any java maven dependency as well. At least personally I don't check all code of all linked dependencies of a new addon pull request while reviewing. Do you?

We must also look at how other solutions handle this. And our big competitors (HomeAssistant) do allow and even encourage people to install extensions from other repos. And the community thinks of this as the simpler solution. We can't loose on this end (usability!) or openHAB will die.

[MHerbst]

I definitely agree with you that openHAB does a lot regarding security. That's what I have said in my post. And I don't want to have a system with that is less secure.

I was also skeptical about the marketplace because it allowed the installations of add-ins that had not been reviewed.

If there is really a bitcoin miner or similar in one of the UI's javascripts (which is the worst that can happen, cross-site scripting is no longer a topic), we can blacklist such a packet.

Have you ever counted the number of components that are installed together with the openHAB add-in in VSC (the node_modules folder contains 201 folders!)? The problem is not the direct dependencies but the large number of indirect dependencies and they are very hard or nearly impossible to track.

Btw, that can happen with any java maven dependency as well. At least personally I don't check all code of all linked dependencies of a new addon pull request while reviewing. Do you?

Of course not, but the number of dependencies is much smaller and therefore better to control if you compare it with the JavaScript dependencies. But I would try to limit the number of 3rd party libraries as much as possible.

We must also look at how other solutions handle this. And our big competitors (HomeAssistant) do allow and even encourage people to install extensions from other repos. And the community thinks of this as the simpler solution. We can't loose on this end (usability!) or openHAB will die.

We are working on an open source tool and not a commercial product. So I won't speak of "competitor".

I don't have problems with other repos but the user must know about the consequences and by default it should not be allowed. Similar to Android. By default only installations from the official store are allowed. But if you want you can install apps from other sources.

[davidgraeff]

Your last point is not entirely correct anymore. Google has changed from a global on/off switch to a "please allow installing apps from this app" style. It's a button click now. That way you can have the Amazon store, open source stores, diverse chinese app stores.

Apple is the only company that still does full app reviews for every app.

The default should be "on", eg allowing to add additional repos and registries. A smarthome user is usually technical affine and should know the risks, but of course I agree that we need some warning messages and explain what permissions or access an add-on may gain.

Regarding dependencies: that will change in java space as well and actually has already. There is no single Apache commons or Google library anymore. Upload tools improved, GitHub started to establish maven repositories per git repository, so you can now deploy to GitHub directly. That's just how it works nowadays, everything has tons of dependencies. I needed to include 12 just for the moquette broker and that's comparable old software. An android app usually has around 30 dependencies (with hidden transitive dependencies). I disagree that we don't have this npm problem also in java.

I have the feeling that you didn't read my first paragraph to the detail. We don't do all of what I have listed. Openhab does NOTHING in terms of security. And we shouldn't pretend to be a safe solution by excluding external add-ons, but by restricting what they are allowed to do.

Just thing a bit on our development language itself: Java on a JVM.

• C++ & Rust would be the much better choice for our usual target: An embedded system.
• Python and JavaScript is way easier to contribute to.

So how to we get people engaged in binding development and convince them to use a language that has less and less popularity. Easy: we build on the strengths of the JVM.

Every call, every thread creation, every file access goes through the security manager in java. Java just does have the best reflection system existing IMO. Let's actually make use of that and protect the user from malicious add-ons and not forbidding add-ons per se.

[MHerbst]

I have the feeling that you didn't read my first paragraph to the detail. We don't do all of what I have listed. Openhab does NOTHING in terms of security.

Sorry, but I did not see the point after the first sentence which changed the meaning. And it was too late in the evening. But even if the openHAB implementation does not everything that's technically possible, we have a review process that in my opinion is the best protection against malicious code (and license violations).

Regarding dependencies: that will change in java space as well and actually has already. ....

... and I definitely don't like it. Too many developers don't care about the components (and their licenses) they are using and the dependencies and possible problems. But this discussion would go too far here.
Same is true for a discussion about the best programming language.

Let's actually make use of that and protect the user from malicious add-ons and not forbidding add-ons per se.

Here, I agree with you. But I would prefer if a user had to explicitly decide that he wants to install extensions from unofficial sources. He can do it but at its own risk.

Let me add two additional aspects that have to do with extensions that don't go through the review process:

• We cannot prevent an extension developer from violating 3rd party licenses, e.g. by embedding libraries that he is not allowed to.
• If a user installs a poorly programmed extension it is possible that this extension influences other (official) extensions negatively (e.g. by blocking resources like ports or by spawning too many threads, ...) or causes problems in the openHAB core. This can make the support of the "official extensions" more expensive,.

[davidgraeff]

We cannot prevent an extension developer from violating 3rd party licenses, e.g. by embedding libraries that he is not allowed to.

We don't need to. We are only "liable" for what we are shipping. And to be honest, almost all linux related stuff is GPL, which we do not really allow in openhab2-addons. So we are excluding a whole lot of useful utility bindings just because of the license. Wouldn't it be great to just add another repo with a button click for GPL bindings?

If a user installs a poorly programmed extension it is possible that this extension influences other (official) extensions negatively

We already execute binding code in a "safe" context (protect us from exceptions). The mentioned instrumentation mechanisms of Java allows us to kill a bundle that uses "new Thread" (see https://stackoverflow.com/a/15868800/3235044). We would force bindings to use a core factory service to get a thread (we already have that). And again we would kill a bundle that requests more threads then what the user has configured as the maximum.

That's the advantage of Java that I'm talking about. We can allow arbitrary addons from any sources, because the core can restrict everything. Like on Android. An Android App without permissions can launch a visible Activity. That's it.

But I would prefer if a user had to explicitly decide that he wants to install extensions from unofficial sources

Absolutely. In the workflow that I have pictured, the user would explicitly and deliberately add an external source with a warning note. But that's still safer than me adding an apt-get source to my linux installation, and I'm doing that quite often.

Just one exception to that: I would like npm with the organisation "openhab" to be enabled by default to allow users to install reviewed user interfaces. The current "we bundle everything as java bundle" philosophy (see webui repo) is stupid IMO. Web based interfaces are npm bundles! We don't bundle the iOS or Android App, so why making it so complicated for web developers to contribute?

[MHerbst]

We don't need to. We are only "liable" for what we are shipping.

Legally speaking, yes. But it will can be bad for the reputation if such things happen too often but ATM I don't think that this will be a really big problem. But e.g. Kodi had some big problem because of some illegal streaming add-ons.

Extensive thread usage was only one example. But maybe we need some additional monitoring in the core to make it easier to identify extensions that are causing problems. E.g. if one extension uses to much CPU time or reserves a port that is also needed by another extension (just an example).

But that's still safer than me adding an apt-get source to my linux installation

Sounds familiar to me .... :-).

The current "we bundle everything as java bundle" philosophy (see webui repo) is stupid IMO

Maybe there was a good reason in the past for bundling them as java bundles. But you are right, this makes the installation of the Web UI too complicated. You can't even simply change a style sheet to get a different L & F.
I agree with you that it would really be better to separate all UI components from the core. This would allow a user to install a "headless" openHAB server on one machine and the web UI on the same or on a different machine where you only need a web server.

[davidgraeff]

Maybe there was a good reason in the past for bundling them as java bundles

Yes there was. About 10 years ago when the project started, the javascript part in web-browsers was still very infantil and not consistent. A lot was done on backends instead, that was also where Java with JSP pages etc tried to get a marketshare.

Nowadays you have a clear distinction. The backend is just an API. The frontend is highly dynamic. Creating a new frontend for a documented API takes you about 2 hours with all the scaffolding cli tools nowadays.

This shift in paradigms is the reason why we need to kill the sitemaps interface which tries to do too much on the backend side and is not flexible at all IMO. Instead we'd just provide

• Things, Items, Rules etc (just like a database),
• a very generic way for frontends to store state (like "pages" or "widgets" or "cards"),
• and a real-time interface (websocket or server-send-events).

A frontend will do whatever it needs to do to present data. I think it was Wouter that worked on a Grafana interface for example. Personally I have developed a Setup&Maintenance interface.

And that's the real reason why I want the extension interface in openHAB with an npm backend.

When Yannick loses interest and the new general user-interface that he is developing at the moment dies a lack-of-maintenance dead (like HaBot and the Flow Editor and HabPanel, you see the pattern), I want it to be super easy for users to switch to an alternative. You could also develop your own personal small Vue-javascript based interface (I intend to write a small developer guide) and share it with the world.

The current situation is sub-optimal and I see HomeAssistant leading right now, we need to change that.