Proposal/Discussion: Self-contained add-on metadata

[ThomDietrich]

Hey everyone.
I'd like to discuss a unified way of adding metadata to extensions/add-ons. While most extensions are still managed in 2-3 consolidated repositories, more and more extensions are developed in dedicated repositories and e.g. published through the Eclipse IoT Marketplace. That shift is unavoidable and I believe also accepted. It however brings a few challenges, e.g. for the Extensions/Add-ons overview in Paper UI or the addons section over at docs.openhab.org.

I propose to introduce a mandatory add-on metadata file to provide information about an add-on in a defined and self-contained matter. The idea is not new, a limited version is already available in the form of ESH-INF/binding/binding.xml for bindings (example) and many other extensions based system do the same thing. See:

• https://getcomposer.org/doc/04-schema.md
• https://docs.npmjs.com/files/package.json
• https://www.python.org/dev/peps/pep-0426

The following proposal was derived from the info given above and is open for discussion:

ESH-INF/extension.xml

(replaces binding.xml)

• name - mandatory - The add-on name, no reference to specific solution (e.g. openHAB). Unclear if type should be included ("Air Quality" vs. "Air Quality Binding")
• description - mandatory - A description of the purpose, specialization and limitations of the add-on, should be between 160-240 characters.
• authors - mandatory - List of the main developers, can also be a community or company name
• type - mandatory - One out of a counted list, e.g. "Binding", "Action", ...
• keywords - optional but recommended - Keywords (categories) out of an open list to filter against, e.g. "heating, lighting, graphing"
• version - unclear - Needed or redundant to OSGi "Bundle-Version"?
• connection - optional - highlight if a binding/action/... interacts locally or if it connects to an online service (seldom or continuously)
• repository_url - mandatory if available - The root or branch/tag-specific URL of the git repository online
• homepage_url - optional - (preferably solution independent, e.g. for non-open-source add-ons)
• documentation_url - optional - URL to end user documentation (preferably solution independent, e.g. for non-open-source extensions)
• image_url - optional, if possible - Link to the extension icon/logo (critical topic due to copyright implications)

More fields could be added, some of the above should be mandatory while most are optional.

Wdyt?

[maggu2810]

Adding some metadata that could be used by which code ever will need them some day is okay.
But I don't think we need to at redundant information.
Why do we need to add the version here? All add-ons are OSGi bundles and so the version is already placed (supported by build tools etc.) in that metainfo.
Isn't the name already be part of the metainfo?
Do you expect that the author of the addon differ from the author of the binding?
Or do we make author in the binding XML optional and use the one in add-on instead?

[ThomDietrich]

Hey Markus,
I've intentionally started with "Proposal/Discussion" 😉 The above is a first design and I'd gladly remove all redundant information. Please keep in mind that I am not an experienced binding/add-on developer.

• Version: Sounds logical. If you are certain the field is superfluous I can surely remove it. Would you be so kind and point me towards the existing version in the OSGi bundle?
• Name/Authors: These are indeed already available in binding.xml. I did not specifically propose that but yes of course, the new file should ideally replace the binding specific binding.xml. Would that be an option?

[kaikreuzer]

I don't think we should add urls at all in the extensions. The idea of ESH extensions is that they are potentially compatible with many different solutions that are based on ESH. So the link to the a suitable documentation will highly depend on the solution and thus cannot be given within the extension.
That's why currently the link to documentation and icons is provided by the extension service implementation that makes the extensions available to the solution. openHAB e.g. does that here.

[maggu2810]

I've intentionally started with "Proposal/Discussion"

That has been my reading. 😉 I only want to give you a comment.

point me towards the existing version in the OSGi bundle?

Hm, look at the META-INF/MANIFEST.MF file that is part of every OSGi bundle.
There should be a manifest entry Bundle-Version.
See e.g.: https://www.osgi.org/bundle-headers-reference/

[ThomDietrich]

@kaikreuzer I had that thought in mind. I believe you are talking about documentation_url and/or homepage_url only? The idea was to provide those for e.g. externally developed add-ons. A developer that doesn't open-source his add-on should be able to provide a link to his private/corporate add-on-related resources? I don't need those and we can remove them (remember that they are already optional).

All other links are pointed at the development repository, which is/can be very much solution independent. If you consider to remove those we can honestly stop the discussion here because they are what I'd especially like to benefit from in documentation... 😉

Btw. maybe you want to study the links I've given for npm and co to get a feeling how other/bigger ecosystems do it.

[ThomDietrich]

@maggu2810 I have no strong feelings towards the version information and where it is stored. For me it feels like the position next to all other meta data would be useful for some use cases later. Yes, redundant information is bad. I've updated the description above, let's wait for other's opinion.
Any thoughts on the other fields, additional ones or the question regarding binding.xml?

[ThomDietrich]

@kaikreuzer @maggu2810 @SJKA @martinvw
friendly reminder that the issue is still open for discussion. Any further arguments, uncertainties or problems? What's needed to push this one further?

[martinvw]

The idea of ESH extensions is that they are potentially compatible with many different solutions that are based on ESH.

I don't think that that should lead to a problem, different solutions can just ignore or at least not expose the fields to the users. For a lot of background processes such as the marketplace and documentation i can see a large benefit in this proposal.

[kaikreuzer]

All other links are pointed at the development repository, which is/can be very much solution independent. If you consider to remove those we can honestly stop the discussion here because they are what I'd especially like to benefit from in documentation...

How would the end user documentation benefit from this? Usually, pointing at source repos should be avoided (especially as you should use links to tags and not to master, because things might be moved around and such links can easily break). Source repo links should only be interesting for developers, not for users.

Regarding image_url: There isn't a good place to put them. For copyright reasons, we must not include them in the ESH repo and you probably would not want to point to any image that you find on the web. For openHAB, being non-commercial, the situation can be different, but again, this is nothing that I see in the generic meta-data.

Btw. maybe you want to study the links I've given for npm and co to get a feeling how other/bigger ecosystems do it.

I don't really think they apply here as they are about libraries for developers - it is not meta-information that is meant for solutions that are built on npm to be shown to the end users, right?

So to summarize: You want to rename the binding.xml to extension.xml (that's how add-ons are called in ESH, sorry) and add type (which list we would still have to define) and keywords (list to be defined as well) to it (all the rest of the list seems to be rather unclear).

Yes, we could do so. We would have to adapt the parsing to correctly read that information and provide it through the REST API. And it would mean an API break as all existing bindings need to be adapted - something we should be careful about, but I do not rule it out if it makes it simpler for us to add new extension types easily.

To come back to your original needs: I feel that it is probably the best solution to maintain dedicated meta-data (including links, images, openHAB specific keywords, etc.) somewhere in openHAB (likewise in any other solution). After all, every solution has special needs and different ways to store such information. Nothing wrong to put in the ESH-INF what clearly is in common for everyone, but I doubt that we can avoid having to maintain a separate list anyhow (see e.g. the image comment above).

[ThomDietrich]

Hello Kai,
first and foremost I want to say that @martinvw highlighted a very important aspect of the base idea: The metadata is primarily intended for behind-the-scene systems and may/must not exposed to the end user. Decisions wrt to that can be made on a per solution level.
Regardless the solution I believe it is important to provide a considerate set of metadata for an add-on in order to integrate fully and intuitively, especially in future scenarios. Every solution or solution part might have different needs or might be restricted in those needs (e.g. copyright) but the sum of data available should not restrict those needs.
If we now also consider the (accepted) trend that add-ons/extensions are being developed unhinged from the existing structures, provided metadata is getting more and more important for compatible and transparent integration of those.

How would the end user documentation benefit from this?

Currently the documentation page goes a great length to retrieve all needed data about add-ons. The process is complicated, error-prone and not able to satisfy all needs. Possible interesting proposed/requested features (e.g.) are currently simply not feasible. This is especially true as new add-ons are integrated from dedicated repositories. Instead of trying to predict properties of an add-on, the add-on should provide this data (up to a certain degree) in a self-contained matter. That seems logical and future-ready. Of course not everything can be provided without loosing solution-independency, I believe to have respected that fact.

Usually, pointing at source repos should be avoided (... tags)

Why exactly? I can't think of any serious reason why any kind of computer software (esp. open source) should NOT contain information about its origin 😲 The author(s) and home of the add-on/extension are irrespective of the solution using it. Again: The information doesn't have to be solution-specific, doesn't have to be utilized and may/must not be exposed to the end user!
Regarding tags: The proposal doesn't say otherwise but that is an important fact I'll add to it.

[Arguments about] image_url

I fully agree. Images/Logos are a complicated issue. In general, this might indeed be a field that can not be filled in most cases.
Optional fields in the proposal should be seen as a chance rather than an burden or obligation! The same goes for most other fields. Consider this 3rd party extension example: If company XY recognizes an ESH based solution as a major market player and wants to develop and provide an extension, they should be able to provide their logo, documentation URL and so on.

I feel that it is probably the best solution to maintain dedicated meta-data (...) somewhere in openHAB

After all I've said above I did not exclude that, the image subject is a good example. Still the proposed metadata will help with that process/effort.
The best solution in my eyes is to provide as much (solution-independent) relevant metadata as possible through extension.xml, then work with it the best we can in solution-specific systems like docs.openhab.org.

but I doubt that we can avoid having to maintain a separate list anyhow

I believe that this is the subsequent task, that will in any case only benefit from already more-directly provided metadata. Those separate lists (or the chance of needing them) can be minimized by a good proposal now. I did not expect this much doubt on that end of the proposal.

To summarize:

1. The proposal is (already) solution independent
2. The option of externally developed extensions is especially considered
3. Metadata will not automatically be accessible by the end user
4. Almost all fields can be defined as optional (and their necessity depends on the extension type/source)
5. At no point will any given but unused metadata be a hindrance to anything (!?)

I'll update the OP with the aspects you've mentioned now.

[kaikreuzer]

I think most of my confusion is that you say that the information is NOT meant to be shown to the end user, while you suggest at the top that this new file should replace the existing binding.xml, which IS meant for the end user (it is parsed at runtime, localisable, shown in the Paper UI, etc.).

If the suggestion is to add some additional source/developer/tool-processing-oriented meta-data, that would be fine with me.

I think in order to move forward with the discussion, it would be good to look at a concrete example. What file/information would we e.g. add to the hue binding and which to the JsonPath transformation?

[triller-telekom]

I was just pointed to this discussion because I did quite a lot of reviews for bindings in openhab2-addons.

As a user that values privacy I am a bit concerned about more and more bindings using a cloud API to access the local hardware. Thus I would suggest a metadata information for bindings which express that they connect to the hardware locally or need some kind of internet connection to work.

Solutions like openHAB could even sort the bindings in the documentation by this "flag", so users which only want to use local bindings know which ones do not connect to cloud services without having to look into the README files or even scan through the source code.

[ThomDietrich]

Hey @triller-telekom sounds good to me. Privacy is (still) one of the corner stones of openHAB and a clear classification like that would be interesting. Could you please propose a line as in the OP and I'll add it above.

[ThomDietrich]

@kaikreuzer I just read your #5176 (comment) and wonder what it is you wanted to discuss here. Do you want to remove the authors field? I don't think this is a good idea. We could clarify the options to allow mentioning of a community or company though.

[triller-telekom]

@ThomDietrich The problem with "adding a line" to you proposal above is the following:

You have type in there which might be any of action, binding, etc...

My "flag" would be for binding only, so it would have to be a child node of the type=binding definition.

Maybe the type node may have parameters, so we could have:

<type connection="(local|internet)">binding</type>

But these parameters would also exist for <type>action</type>...

[ThomDietrich]

Fields can be optional. If the connection field is not meaningful for a UI, don't mention it. 😃

The overall idea here is to provide metadata when meaningful and useful. I'd only require very few.

[kaikreuzer]

I just read your #5176 (comment) and wonder what it is you wanted to discuss here

Nothing really to discuss - I merely suggested to have an optional "author" here in the technical metadata, but don't include it in the binding.xml anymore, which is user-facing content (within the Paper UI).

[ThomDietrich]

Data as proposed here isn't necessarily user-facing. It is not necessarily part of the Paper UI presentation.
I have the feeling we still didn't reach a common understanding of the goal here. Maybe you want to read my last few comments again.

@triller-telekom I've added connection to the list above

[kaikreuzer]

Yes, it seems you again misunderstood me.
What I said is that the current content of binding.xml is user-facing. That why I suggested to move the author information to the "Data as proposed here" (which I referred to as "technical metadata").

[ThomDietrich]

I see. I indeed misunderstood that part.

Yet again, I would question the meaning of "content of binding.xml is user-facing" as I believe what you actually mean is "every detail available in binding.xml is by chance also presented by Paper UI". Nothing forbids the addition of more information that is not or only conditionally presented in Paper UI or other user-facing graphical interfaces... - Correct? Did I match your meaning of "user-facing"?

The "data as proposed here" yet again is what I envision for a file called similar to ESH-INF/extension.xml and which supersedes binding.xml. Besides needed code changes there shouldn't be a good reason to keep both, is there?

Once again the content in this file MAY be used by Paper UI, however nothing dictates that. In conclusion I don't see why we should not add as much (useful, relevant and harmless) data as could be useful to Paper UI, the documentation, or any other future system that handles extensions - user-facing or not.

Any other concerns with the proposal? It would be nice to settle this issue at some point, even though I have to admit that I am the wrong person to ask anything of anyone as I am currently not active myself.

Best!

[maggu2810]

I did not read the whole topic from the beginning again just the new parts.
I don't see any problem to replace the binding.xml with a more generic extension.xml and move all user or technical metadata to that one file.
The UI itself can display the fields it is interested in and hide the other ones. I don't see any reason to use separate files for different information.
Perhaps another UI (not the Paper UI) would also display other parts (and perhaps not all but some of the one file some of the other).

[kaikreuzer]

@ThomDietrich & @maggu2810 What you are obviously missing here is that the binding.xml is just "the tip of the iceberg". Similar to thing xmls, it is just one way to provide runtime information about a binding. The major part of that infrastructure is the BindingInfo, BindingInfoProvider, the BindingInfoRegistry and BindingResource with all the content being localisable through the i18n infrastructure as well. I don't really see any value in spending the effort to extend all this to information that is anyhow not meant to be available at runtime and/or being user-facing.
Especially stuff that is technical/internal must imho not be exposed through the REST API.

As I said, I am ok to include a package descriptor file (as the ones given as examples above) in the bundles, but I would not want to mix this with the UI infrastructure.

[ThomDietrich]

@kaikreuzer I'm deeply sorry but even after countless comments I am still unclear what exactly is the issue, why are we still so focused on "user-facing" and what all of this has to do with "UI infrastructure".

The goal is to provide a rich metadata file, so that each addon (not just bindings, therefore the change of file name) is accompanied by information that describes itself. Where this data is then utilized, processed or presented is almost irrelevant and the next step.

[ThomDietrich]

Any new thoughts on this?

[kaikreuzer]

why are we still so focused on "user-facing" and what all of this has to do with "UI infrastructure".

Because this is what the current binding.xml is all about as explained above and you suggest to remove it and replace it by something else.

If you'd alter your suggestion to simply add an additional metadata file, it all would make much more sense to me.