[openhab.core] Split: Storage+Auth+Persistence

[davidgraeff]

The core bundles purpose is not well defined and contains almost everything in a mix.

I could understand if that is API only ("defining the openHAB smarthome APIs"), but that's not the case.
Also newer concepts like automation and things are not part of this core bundle which speaks against the one API core bundle theory.

I therefore think that the core bundle should shrink down to bare utility methods (OSGI/Registries/Threadpool) and all other API and impl should be distributed.
(This is also true for the core.rest bundle which I'm splitting here simultaneously)

Advantages:

• Semantic versioning of parts of the core will be possible in the future (eg OH-persistence 3.0 but OH-storage 3.0.1)
• Related functionality is in one place each
• Solutions can easily ship without a specific subsystem by just not using bundles of a specific pattern.
• Inter-dependencies between subsystems can be easily seen by looking at the pom.xml files. Allows to group subsystems into Tiers in the future (like C++/Qt and other multi-subsystem projects do it).

This is the first of a series of PRs and handles

• Storage
• Authentication
• Persistence
• core.net

Further planned PRs:

• core.extension
• core.binding
• core.i18n
• core.item (analoge to the also separate Thing bundle)

I'm backporting this from my own fork and not all pom dependencies are yet correct.
Please consider this WIP until travis is satisfied.

Signed-off-by: David Graeff david.graeff@web.de

[5iver]

@davidgraeff, I'm curious (and anxious!) to see what you've done. But it seems to me that this should go into a feature branch. It will require maintainance in the future to keep it in sync, but keeping it out of master will allow us to keep moving forward with OH2.x.

[davidgraeff]

@openhab-5iver This is not breaking any API, it is just moving files between bundles. The core bundle will even list the splitted bundles (at least while we are in OH 2) as dependencies so nothing changes for bindings or solution consumers.

The huge OH 3 package renaming is still to come at some point.

And the bigger picture for me is: I want to contribute my re-worked extension, extension service, extension repository handling (for npm and github repositories as extension sources).
But that requires .core to be split into smaller units first otherwise the diff will be too huge to review.

[5iver]

I wasn't sure if this was to go into OH3 or not. Just mitigating risk... we could end up ready to finally get a release out (hopefully we're getting close!), but be stuck behind such large commits. Keeping it separate allows us to release around your commits.

[davidgraeff]

Hm but this is 250 lines of code change ^^. I carefully backported to make this tiny on purpose.
The thing is, we are always waiting in this repo. The last 4 months were only small bug fixes and just a ton of buildsystem related commits.

[5iver]

Just my opinion... and I tend to be overly cautious when it comes to release management. We'll see what the maintainers think!

[maggu2810]

I am not fully convinced yet.
Personally I prefer to split things into different bundles instead of one big one.

The split you are providing does not split API and implementation.
So, while using an API, we still need to use the official implementation and cannot use a custom one.
Shoudn't we if we start a split already think about how to differ API and implementation?
Or does it make any sense to split the difference "subsystems" now into separate bundles containing API and implementation and later to split API and implementation. Then we need to think about if we want to use one openHAB Core API bundle or separate ones, ...

I am also not sure if solutions can drop all that subsystems. Are you sure that all of them are optional?
I am pretty sure, that you cannot build a solution without e.g. the storage bundle as it will be used by most of the other ones.

[davidgraeff]

I do think that API+Impl bundles is the way to go.
But I also do think that openHAB subsystems should have the chance to evolve independently (aka semantic versioning).

The mentioned storage service bundle but also authentication are API bundles. They cannot be dropped from a solution, true, but they might be versioned independently in the future.

The .net bundle exists because of easy discovery and to entangle dependencies within .core.

If you have a better suggestion on how to tackle the overloaded .core bundle I'm interested to hear your opinion.

I really just want to contribute my extension stuff.
And at the moment I'd need to change the .core bundle and .core.io.rest bundle for that and I really think that is a design flaw.

[maggu2810]

In OSGi you use versioning per package.
So, one API bundle can contain v1.1 for package x v2.3 for package y, v4.5 for package z etc.
"One" bundle only means that you need to release and update all the packages at the same time.

I really just want to contribute my extension stuff.
And at the moment I'd need to change the .core bundle and .core.io.rest bundle for that and I really think that is a design flaw.

I see your point, but if we split the core bundles, we should not do it, just to easily contribute something and then do it another way after that again.

I really agree that we should split API + impl and also split the different subsystems.
But such changes needs a well defined concept.

[davidgraeff]

In OSGi you use versioning per package.

Something learned. So OSGi bundles use package-info for version information I guess.

But such changes needs a well defined concept.

I have my fork for a month now, I'm just afraid that I will have that for another 6 months or even worse that package renaming starts and I cannot contribute my stuff back in the end. I really love to use my Paper UI replacement, but core is missing so much fundamental basics. The REST API cannot even tell what OH version is in use.

I tried to open a few discussion PRs here on core, but I usually get no response.
Now I have tried to make a first move with a PR but indeed there is no roadmap, no vision, no written down idea how core should be structured and the PR can only be declined. I'm really frustrated with OH core contributing.

Can @openhab/core-maintainers please write down a vision and roadmap. There are willing contributors. And if that means that we need to spend an afternoon in a discord channel to discuss and argue, then let it be.

[maggu2810]

The REST API cannot even tell what OH version is in use.

That's another topic we already discussed in ESH.
What's the version at all?
The version of package x, the version of bundle y, the version of ...
You can mix whatever you want in your env.
The decision has been made to use a manual incremented version that should define something similar to an "REST API" version information.
But I assume this has not been maintained correctly.

I tried to open a few discussion PRs here on core, but I usually get no response.

Hm, IIRC Kai added responses to that PRs.

but indeed there is no roadmap, no vision, no written down idea how core should be structured and the PR can only be declined
Can @openhab/core-maintainers please write down a vision and roadmap. There are willing contributors.

So, you tell me that you change this stuff without a vision and expect it does not get declined?
I would be happy to write down a vision and roadmap but hey, it is my spare time.
Perhaps my understanding of open source development is wrong, but if I want to change something in a project, I need to provide a concept.
I cannot tell the maintainers, that they need to give me a concept. At least if no one gets paid for.

I'm really frustrated with OH core contributing.

I am sorry. I try to read every PR and comment all of them. It is just a matter of time.

[davidgraeff]

By reading through my comment I realized that it might sound offending towards you. That wasn't the purpose, I'm sorry.

Warning, long post.

Vision

I do have a bigger picture of what I like openHAB to be.

I have created this PaperUI design study (https://davidgraeff.github.io/paperui-ng/) where I have embedded all services and API endpoints that I like openHAB to have. My vision is encoded in code, if you want so.

(Not really for read just for reference).
See https://github.com/davidgraeff/paperui-ng/blob/develop/js/storage-webworker/rewriteToNotYetSupportedStoreLayout.js add https://github.com/davidgraeff/paperui-ng/blob/develop/js/storage-webworker/addNotYetSupportedStoreData.js.

I'm adding 14 REST API endpoints and modify 7 endpoints for the design study to work.
I have written some kind of missing services roadmap here: https://davidgraeff.github.io/paperui-ng/roadmap.html and have implemented most of them.

I really like to contribute my developed services and core changes, but I don't know how and that frustrates me. I have split the core bundle in my fork to not touch so many bundles when re-working subsystems.

REST

I know that it is hard to define the version that is send via REST, but the user is actually interested in the current distribution version. And in the boot bundle that exact version is read in and shown in the dashboard.

The current REST design in this project follows no real vision at least none that I understand.
Sometimes endpoint data is "enhanced" (like embedding the configuration description into thing types, or adding metadata to items) and sometimes not (why on earth do I have services and services/config).

I actually know the why behind those decisions. They were a lot driven by PaperUI. But In my design study, that is like a lot more user friendly, I'm showing more info from more endpoints and actually end up calling about 3 to 5 endpoints for a Thing edit page etc.

There is also a bigger long term plan that I have in mind:

Getting rid of REST and substitute with GraphQL. When designing REST endpoints you never know what the client will make out of it, with GraphQL the client will request exactly what it needs and we don't need to guess in the code and send "enhanced" DTOs when it is not required.

And even if REST should stay until EOL, in my opinion there shouldn't be separate rest packages and we just expose all registries automatically. (Not talking about endpoints dedicated for control here, just for setup&maintenance).

[davidgraeff]

How do I progress in this matter?

I guess I first abandon this PR.
My first post always includes a reasoning (a concept / a vision) and that was not accepted / not convincing so far. I still think that my split idea is superior, but the status quo also works somehow.

Should I maybe first tell about my ideas? I mean I tried (#665, #594) but I really get no responses from maintainers in this repo if I only create Issues.
So how does it work to contribute or is this project in maintenance mode only?

[maggu2810]

I personally do not use sitemaps (#594) or the extension service (#665) at the moment.
So it is not on my high priority list and I cannot judge. Or let's say, other ones can judge better.

If no maintainer uses this code anymore it does not mean no other one is using it. And we should not break all the consumers that just want to use the framework and don't follow the issues.

You mentioned Qt in this thread -- perhaps because you use it.
Do you read every ticket of the Qt? Would you be happy if they rework its API and break all your software that relies on Qt?
For openHAB Core as a framework we should not break just because we have an idea how we could make things better.
There are products that rely on interfaces etc.

It does not mean we need to live with a "bad design" forever but changes should be done considered and if it is merged, we should just fixes bugs.

Please don't get me wrong, I love progress and improvements, but don't expect too much as long as we don't get paid for all the time invest.

Perhaps it would make sense to introduce meetings or something similar where people discuss the technical stuff.

While working on the migration and the other openHAB repositories I also made my experiences and drew conclusions.

But hey, this is very off-topic -- at least it is not an issue for the code base (this issue tracker is used for) but for the management.

[davidgraeff]

But hey, this is very off-topic -- at least it is not an issue for the code base (this issue tracker is used for) but for the management.

We have no other means on the other hand. There is no mailing list or board (like the old Eclipse Dev Board) or IRC.

Do you read every ticket of the Qt? Would you be happy if they rework its API and break all your software that relies on Qt?

Just a few words about Qt and their way of doing it, but I'm sure everyone knows about the versioning issue we have here at the moment.

Qt does break API periodically, roughly every 5 years. But each time (major update) the changes got less intense because they emerged into a very fine well defined product and API. All big projects that I know of adapted because of the benefits of the new APIs.

• They release the framework in so called modules, and specific modules have specific stability promises. Like with the automation subsystem here. And we need more of this experimental bundles, that are allowed to change a couple times, without API guarantees, before turning into their final shape.
• If you have multiple modules, I rather call this subsystems in openHAB context, you run into inter-dependency problems. Qt solved this by grouping their modules into Tiers. A Tier 2 bundle may only rely on Tier 1 bundles and so on.

It does not mean we need to live with a "bad design" forever but changes should be done considered and if it is merged, we should just fixes bugs.

I know what you mean. This model also makes this project very lethargic when it comes to pull requests and no fun to contribute. And that is what I'm afraid of. No experimentation, no trying out, basically a full stop.

I've spend like 300 hours or probably more for developing bindings and my own core bundles to have a great experience for my HA system at home, so I have invested a great amount of time.

But I feel like if openhab-core continues it's current model it will die a slow death (which I do not want for mentioned reasons). Because of the usage of OSGi and Java we are already in a back position compared to other competing frameworks like HomeAssistant (I'm talking about the ease of contributing). The language itself just recently (Java 11/12) is catching up with Kotlin and Scala. With our Java 8 min requirement, there is still a lot of boilerplate code to write.

The tooling must be so much easier to attract more developers (maven/bnd a big plus, gradle/bnd even better of course), the API must be easier to make the review process simpler (annotation based binding development. Yes I have created core bundles for that) and so on.

To wrap this up, I understand that continuity is important because some solutions are relying on it. But if people of those solutions (and I'm not talking about you Markus) are not contributing, why should openHAB-core care about them?

Still, I respect the idea, and that's the reason for this PR. Not breaking the APIs, but sorting bundles into actual subsystems. Those then can be independently changed expect if a veto is coming in.

[5iver]

We have no other means on the other hand. There is no mailing list or board (like the old Eclipse Dev Board) or IRC.

This is what the team discussions are for... https://github.com/orgs/openhab/teams/core-maintainers.

Perhaps it would make sense to introduce meetings or something similar where people discuss the technical stuff.

I think some technical discussions would be very useful. If it worked out well, and there were enough people, it could turn into a recurring call/video chat where developers could ask questions, show new designs, work through larger issues, etc. Depending on the platform used, this could be opened up to the rest of the community, so that they can hear/see the direction that the developers are taking the project in.

[kaikreuzer]

We used certain guidelines for this code in the past:

• In OSGi, Java packages are the first citizen. They define what belongs together as an API or implementation and they are the entities that are versioned and can break compatibility.
• Differentiation between API and Impl is done by using internal packages and exposing packages.
• Bundles are an arbitrary way to package Java packages.
• We aim to keep the number of bundles small, since every additional bundle means a performance impact on small embedded systems.
• We ONLY split bundles into pieces if there is the need to deploy them independently.

Wrt this PR, I don't really see the need to have a more fine-grained split into functional pieces. Imho a solution is likely to require those parts and even if not, the additional not-used features would not be a huge burden.

Note that it is not really in the interest of the project to foster a plethora of new solutions being built. We rather acknowledge the fact that there are other solutions than the openHAB distro itself and make sure to keep this support. But imho the architectural design should not primarily be driven by this requirement.

[davidgraeff]

We aim to keep the number of bundles small, since every additional bundle means a performance impact on small embedded systems.

I thought about this as well until I have read that Eclipse Concierge (https://www.eclipse.org/concierge/) for example extracts jar files, so the amount of bundles is not important to that IoT tailored OSGi container. I guess that Felix or Equinox can do or do the same?

We ONLY split bundles into pieces if there is the need to deploy them independently.

I get you there, but the current bundle system here in core is inconsistent, isn't it? You can't build a OH solution without things, but things still have their own bundle. Discovery has it's own bundle. All those API's could have been in .core then. I'm just for going either way, but be consistent. So if splitting is not wanted, merging is?

[kaikreuzer]

So if splitting is not wanted, merging is?

I am all for it. The main reason why this hasn't been done for many places is that we have the rule that bundle ids should be chosen to reflect the java packages names. With this rule, it is easy to split existing bundles, but hard to merge them as you would then end up with inconsistent naming...

[maggu2810]

We aim to keep the number of bundles small, since every additional bundle means a performance impact on small embedded systems.

Has there been some analysis between the runtime overhead and the number of bundles?
Can you point me to some details?

[kaikreuzer]

I cannot provide you with any detailed statistics, but those were clear results from our experiences at DT.
Additional bundles cause different kinds of things:

• the OSGi fw needs to extract/install them in some way
• they need to be resolved, which is an expensive operation on embedded systems
• they cause lifecycle events, which are consumed by others, which then become active (like OSGi ds to check for XMLs or our bundle watcher services and others, which e.g. scan the manifest for specific headers)
• we are doing checks for ESH-INF folders and create ready markers, etc.

All those things might seem neglectable, but when you end up with several hundred bundles in an installation, they add up to quite something ("Kleinvieh macht auch Mist")...

[openhab-bot]

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/need-some-help-with-setup-to-create-new-binding-using-maven-bnd/70901/45

[maggu2810]

Isn't it much more complex?

• Using "bigger" bundles will lead into much more classes in the classpath on compile time.
• If the big artifact file changes all dependent artifacts needs to be rebuild to check for breakage. A lot of more bundles needs to be recompiled.
• If one class of that big JAR is used and so its package imported from another bundle, it takes much more time before that logic can start because the whole bundle needs to be started.
• If a class of a bundle needs to be fixed, you just need to update that bundle (big win of the modular design of OSGi), we don't need to restart all your stuff. If the bundles goes to be bigger, all stuff that depend on any package of that class will be affected by the "stop, update, (e.g. dependent refresh,) start" cycle.

I don't state that we need a bundle per package, I just want to state that IMHO both directions can lead to a bad behaviour.

[davidgraeff]

All I can say as a contributor is that the current situation is confusing. There are two ways a project can be organised:

1. It's very own, organic way but with a detailed documentation where to find what and the reasoning for specific taken decisions
2. Organised by convention

It's like with maven. The convention is enforced on a java project and as long as you follow the convention you don't need to alter the buildsystem or document anything.

But OH does not follow the OSGi/bnd API-Impl bundle convention AND it has no documentation or specific pattern.
And I do not see a change with OH 3 as long as you guys are not on the same boat.

Personal opinion

I personally thing there is no "technical" aspect to consider, it is purely about taste. As my research in this are indicates, all OSGi frameworks are using sophisticated bundle caches. And that xml parsing is slow (aka finding ESH-INF files) is known. XML is not necessary for OH and I would like an alternative (not a replacement ; ) for OH 3 bindings.

I can understand both parties.

• I mean I have proposed this PR, so I like the idea of having everything organised by "subsystems". Even if there are strong dependencies between those subsystems, a new contributor can quickly find its way around in the codebase. No guesswork. And yes, IMO, having more contributors is more important than a 2% increased start-up performance.

• I also like the idea of one .core bundle that contains the essentials of OH. In my opinion this is Items, Things, Discovery. But .core should not contain utilities. Utilities tend to change API to fast (newer technologies, newer java versions etc). All registry and OSGi and network stuff should not be in .core. A lot of network stuff was or still is IPv4 specific for example. Newer OSGi versions make part of the OSGi utility classes obsolete. And nowadays the http whiteboard pattern is to be used instead of the self-made OH factories. And so on.

I would like every of the 4 OH-core maintainers to participate in this discussion actually. This bundle situation can be solved BEFORE OH 3 and should be. There will be enough confusion later on with package renaming.

[davidgraeff]

Not much for me to do here, closing.

[splatch]

Just waking up this issue to see if its suitable for OH3 or not.. is it going to be considered or not?