[JSR223] Most Java classes are no longer available after ESH migration

[5iver]

I'm testing this on S1534. Classes that were previously available can no longer be imported. Here is a list of the ones I have tested, when trying to update the openhab2-jython libraries. The ones with ##### worked, but the others did not.

# core.date
from org.openhab.core.library.types import DateTimeType as LegacyDateTimeType
#####from org.eclipse.smarthome.core.library.types import DateTimeType

#####from org.eclipse.smarthome.core.items import Metadata
#####from org.eclipse.smarthome.core.items import MetadataKey

from org.joda.time import DateTime

from org.eclipse.smarthome.automation.core.util import RuleBuilder
#####from org.openhab.core.automation import Rule as SmarthomeRule
#####from org.openhab.core.automation.handler import TriggerHandler

# core.triggers
from org.eclipse.smarthome.automation.core.util import TriggerBuilder

#####from org.openhab.core.automation import Trigger
#####from org.eclipse.smarthome.config.core import Configuration
#####from org.eclipse.smarthome.core.thing import ChannelUID, ThingUID, ThingStatus
from org.eclipse.smarthome.core.thing.type import ChannelKind
#####from org.eclipse.smarthome.core.types import TypeParser

# DirectoryEventTrigger
from org.eclipse.smarthome.core.service import AbstractWatchService

# JythonTransform
from org.eclipse.smarthome.core.transform import TransformationService

# JythonThingTypeProvider
#####from org.eclipse.smarthome.core.thing.binding import ThingTypeProvider

# JythonThingProvider
#####from org.eclipse.smarthome.core.thing import ThingProvider

# JythonItemChannelLinkProvider
from org.eclipse.smarthome.core.thing.link import ItemChannelLinkProvider

# JythonBindingInfoProvider
from org.eclipse.smarthome.core.binding import BindingInfoProvider

# personal.utils
from org.eclipse.smarthome.model.script.actions.Exec import executeCommandLine
from org.eclipse.smarthome.model.persistence.extensions import PersistenceExtensions

# core.actions
from org.openhab.core.scriptengine.action import ActionService
from org.eclipse.smarthome.model.script.engine.action import ActionService

@lewie reported this in openhab1-addons/#5281. @vbier also reported something similar in the forum.

[maggu2810]

I assume the code that executes your scripts defines package imports only for the packages that are used by its Java code.
So, while using the bundle's classloader for the script execution the non-imported packages will not be available.
I don't know what makes more sense, using a special classloader with a defined set of classes that are allowed to be access or adding all the packages to the script execution bundle's import declarations.

[5iver]

I assume the code that executes your scripts defines package imports only for the packages that are used by its Java code.

If that were the case, then I don't see how they were accessible before. There are classes that are available, but have nothing to do with scripted automation, like Metadata. So, I think there is more to it. I'm still investogating.

BTW, I realize org.joda.time was removed, but left it in the list. Or was that reverted? Wait... I just imported org.joda.time.DateTime into a DSL Rule... hmmm.

[maggu2810]

Before the migration the imports has been added in the manifest manually.
Now the manifest is generated by bnd that inspect what is used by the code.

I was not aware that org.joda.time has been removed. IIRC in ESH (so in front of the migration) there has been a PR that has been reverted.

[vbier]

I also have problems with missing classes in my groovy rules. The most simple example does not work:

scriptExtension.importPreset("RuleSimple")
def r = new SimpleRule();

It does not help if I try to import either
org.openhab.automation.module.script.rulesupport.shared.simple.SimpleRule or
org.eclipse.smarthome.automation.module.script.rulesupport.shared.simple.SimpleRule, it always gives me

unable to resolve class SimpleRule

[maggu2810]

@openhab-5iver I never used JSR223 -- at least I cannot remember 😉
Is it about DSL (model.script and model.rule) or the new automation engine?

[5iver]

@maggu2810, JSR223 is under org.openhab.automation, and was introduced with ESH #1783. If you'd like to reproduce the issue, just install the new rule engine, and save this to a .js file in OPENHAB_CONF/automation/jsr223/...

var RuleBuilder = Java.type("org.eclipse.smarthome.automation.core.util.RuleBuilder");

I'm no help at all when it comes to Maven and bnd, but from researching, I found a number of references where using DynamicImport-Package: * resolved similar issues.

[maggu2810]

I can try to reproduce and solve it but I need to know:
Which bundles are necessary that are not part of openHAB Core?
Which bundles needs to be present in the runtime?

[5iver]

The list I provided in the OP was everything that I could find that was missing. The only thing not in openhab-core was org.joda.time, but I wouldn't say that is necessary but nice to have.

[maggu2810]

@openhab-5iver You need to start from the beginning 😉

I added the automation bundles to the my runtime.
That's the openHAB Core bundles that are part of my runtime:

openHAB Core :: Bundles :: Audio (2.5.0.201903071601)
openHAB Core :: Bundles :: Automation (2.5.0.201903071602)
openHAB Core :: Bundles :: Automation Media Modules (2.5.0.201903071602)
openHAB Core :: Bundles :: Automation REST API (2.5.0.201903071602)
openHAB Core :: Bundles :: Automation Script Modules (2.5.0.201903071602)
openHAB Core :: Bundles :: Automation Script RuleSupport (2.5.0.201903071602)
openHAB Core :: Bundles :: Binding XML (2.5.0.201903071558)
openHAB Core :: Bundles :: Configuration Core (2.5.0.201903071558)
openHAB Core :: Bundles :: Configuration Discovery (2.5.0.201903071602)
openHAB Core :: Bundles :: Configuration Dispatcher (2.5.0.201903071558)
openHAB Core :: Bundles :: Configuration XML (2.5.0.201903071558)
openHAB Core :: Bundles :: Console (2.5.0.201903071600)
openHAB Core :: Bundles :: Console for OSGi Console RFC 147 (2.5.0.201903071600)
openHAB Core :: Bundles :: Core (2.5.0.201903071600)
openHAB Core :: Bundles :: HTTP Interface (2.5.0.201903071600)
openHAB Core :: Bundles :: ID (2.5.0.201903071601)
openHAB Core :: Bundles :: MapDB Storage (2.5.0.201903071602)
openHAB Core :: Bundles :: Model Core (2.5.0.201903071601)
openHAB Core :: Bundles :: Model Items (2.5.0.201903071601)
openHAB Core :: Bundles :: Model Items Runtime (2.5.0.201903071601)
openHAB Core :: Bundles :: Model Persistence (2.5.0.201903071601)
openHAB Core :: Bundles :: Model Persistence Runtime (2.5.0.201903071602)
openHAB Core :: Bundles :: Model Rules (2.5.0.201903071602)
openHAB Core :: Bundles :: Model Rules Runtime (2.5.0.201903071603)
openHAB Core :: Bundles :: Model Script (2.5.0.201903071602)
openHAB Core :: Bundles :: Model Script Runtime (2.5.0.201903071603)
openHAB Core :: Bundles :: Model Sitemap (2.5.0.201903071601)
openHAB Core :: Bundles :: Model Sitemap Runtime (2.5.0.201903071601)
openHAB Core :: Bundles :: Model Thing (2.5.0.201903071602)
openHAB Core :: Bundles :: Model Thing Runtime (2.5.0.201903071602)
openHAB Core :: Bundles :: Monitor (2.5.0.201903071601)
openHAB Core :: Bundles :: Network I/O (2.5.0.201903071601)
openHAB Core :: Bundles :: Persistence (2.5.0.201903071601)
openHAB Core :: Bundles :: REST Interface (2.5.0.201903071601)
openHAB Core :: Bundles :: REST Interface (2.5.0.201903071602)
openHAB Core :: Bundles :: REST JAX-RS Optimizations (2.5.0.201903071601)
openHAB Core :: Bundles :: Scheduler Service (2.5.0.201903071602)
openHAB Core :: Bundles :: Semantics (2.5.0.201903071602)
openHAB Core :: Bundles :: SSE Interface (2.5.0.201903071601)
openHAB Core :: Bundles :: Thing (2.5.0.201903071602)
openHAB Core :: Bundles :: Thing XML (2.5.0.201903071602)
openHAB Core :: Bundles :: Transformation Service (2.5.0.201903071602)
openHAB Core :: Bundles :: Voice (2.5.0.201903071602)

After that I created a directory automation/jsr233 in my conf directory and put a test.js file with the above line into that directory.

And now?

[5iver]

Add the rule engine and watch the log

[maggu2810]

What a bundle is missing (can you tell me the BSN or GAV)?

[5iver]

Sorry... I meant start the rule engine, if it isn't. Are you trying to reproduce this in the IDE, or in a build that you've made? It looks to me like you have everything.

[maggu2810]

Hm, more or less both.

I habe an IDE that contains the openHAB Core bundles only.
But if I need other ones, I will add them.

The source tree of my openHAB Core code base is only minimal modified.

I am using the Bnd based launch demo configuration (only added bundles).

[maggu2810]

Sorry, I named the directory jsr233 instead of jsr223...

[maggu2810]

I assume if I see

23:29:52.890 [ool-12-thread-1] TRACE o.o.c.a.m.s.i.ScriptEngineManagerImpl:132 - scriptLoaded() not defined in script: file:/home/maggu2810/data/shared/bin/pkgs/eclipse/oh-core/git/openhab-core/demo/app/runtime/conf/automation/jsr223/test1.js

instead of

23:31:52.025 [ool-12-thread-1] ERROR o.o.c.a.m.s.i.ScriptEngineManagerImpl:138 - Error during evaluation of script 'file:/home/maggu2810/data/shared/bin/pkgs/eclipse/oh-core/git/openhab-core/demo/app/runtime/conf/automation/jsr223/test2.js': java.lang.ClassNotFoundException: org.openhab.core.automation.util.RuleBuilder

it is better

[5iver]

Yes, the use of the scriptLoaded() and scriptUnloaded() functions in your script are optional, so that message is normal.

[maggu2810]

The automation has been moved from ESH to OH namespace (package names).
If we restore all the accessible classes / packages, the scripts needs still be changed to use the OH automation package instead of the ESH packages.

Let me ask if it make sense to access that Java classes in scripts at all.
Every time we change the core code and e.g. all addons to compile successful it could break scripts because they access that classes.
This will detected at runtime only and could forbid us to improve the Java code handling (perhaps).

Wouldn't it make sense to create separate classes -- perhaps they only wrap the whole current function.
We could use a namespace that is prefixed by the scriptsupport one and ensure that classes stay stable also if we change internals.
If we ever need to change also that public interfaces, we know -- as it is the only use case for that classes -- that we change the API for scripts and can add it to a release documentation or perform a migration of the scripts on update automatically.

[5iver]

If we restore all the accessible classes / packages, the scripts needs still be changed to use the OH automation package instead of the ESH packages.

The majority of imports will be in the helper libraries. When this issue is resolved, I will be providing a release of the libraries to support the changes. This release will require that they are using a minimum version of OH that contains the fix. There have been other breaking changes in the API that required upgrades of the libraries. Users may need to change some imports in their scripts and modules, but that shoiuld be minor, and these will be the more experienced users. Anyone not using the libraries will have some work to do, but it would also be a good opportunity for them to switch over to using the libraries.

Let me ask if it make sense to access that Java classes in scripts at all.

For the future, maybe OH3, the helper libraries, which show what users actually want to be doing with scripted automation, can be absorbed into some of the Automation classes and a new scripting API. Some (most?) can also be turned into Actions, which could then also be accessible directly through the NGRE. The ESH reintegration, repo moves, and lack of UI in the Core Development IDE have really been holding me back. But things seem to be settling down now, and with #635 finally committed, I will be working on Jython and Groovy bundles, and some of the Automation bugs. While doing this, I plan to open some discussions in the Developer section of the forum to discuss plans for moving forward with the NGRE and scripted automation... specifically, what needs to be done to get it mainstream in OH3. I would really like to see some Github projects setup in this repo for better visibility, prioritization, and coordination of efforts for Automation.

I like what you're thinking, but for the time being, I think they just need to be made accessible again in order to get things working. 😄

BTW, could you please share the Demo app changes that you used to be able to test this? I think I am at the same place you were, where the engine is running, but scripts aren't loading. At some point, I really need to get Jython scripts working in the IDE again too.

[lewie]

When the classes are assessable again, I will adjust the JS helper classes, so that they are backward compatible to older openHAB versions with ESH .

[5iver]

When I replied to Markus, I completely forgot that I had planned to do the same! This avoids requiring a minimum OH version for the libraries, and can be implemented now, if this is the path we're taking.

try:
    from org.eclipse.smarthome.automation.core.util import RuleBuilder
except:
    from org.openhab.core.automation.util import RuleBuilder

[CrazyIvan359]

With the ESH integration finished, is the entire core accessible again? I have been working on some scripts that require access to add, modify, and identify the class of items, metadata, things, and channel links. I had upgraded to S1522 some weeks ago but had to rollback to S1512 because most of the classes in automation and the core were not available.

@openhab-5iver let me know if I can help in any way to get the libraries work again or with the integration of scripting into a bundle

[maggu2810]

@openhab-5iver: I will fix the missing imports and explain what to changes are required for the demo application.

Till then we hopefully get this one merged: #640

For the work with openHAB Core and Maven + Bnd bundles I assume that you would like to use that commit in your working copy: maggu2810/openhab-core@f1c74e5
See also: https://github.com/openhab/openhab2-addons/pull/5050#issuecomment-470459971

[maggu2810]

@openhab-5iver: Should joda still be provided? AFAIK we already tried to drop it...

[5iver]

@maggu2810, it was available before the reintegration, so I woud think it should be available after. It would be a breaking change if it were not included.

[maggu2810]

Yes, it will be a breaking change.
But AFAIK changing the package names of all automation bundles has been a breaking change, too -- and it has been done.

[maggu2810]

@openhab-5iver I cannot test myself ATM, but you could perhaps check the branch "dev" in my fork and give it a try.

[CrazyIvan359]

@maggu2810 I think it would be a good idea to remove joda given its depreciation in favour of java.time. On the other hand, my DSL rules make heavy use of it, so it disappearing suddenly would be a shock. Perhaps a warning in the log on parsing rules files that import it in snapshots starting now and up until the release that phases it out.

For the purposes of Jython that I am converting all my rules to I am using java.time, and the portion of Scott's scripts that use it can be rewritten to handle the possibility that it isn't available gracefully.

[5iver]

But AFAIK changing the package names of all automation bundles has been a breaking change, too -- and it has been done.

@maggu2810, that's very true. When this issue is resolved, we will need to communicate that to the JSR223 users. I will have an update to the Jython helper libraries released to mitigate that, and @lewie will hopefully have one for JS. But removing Joda from JSR223 would be MUCH harder to mitigate, as it will effect people's scripts/rules.

I cannot test myself ATM, but you could perhaps check the branch "dev" in my fork and give it a try.

I'm still working on getting automation working in the IDE, but I'll do a build of your branch.

[maggu2810]

If you are using the demo app in my branch, the automation bundles will be present at runtime (I modified the demo app "now", so update your working copy).

[maggu2810]

@openhab-5iver
Added some minor updates of the configuration.
Have a look at the "runtime" directory in the demo app.
It contains a logback.xml file that contains the logging configuration (the root tag declared that it should be reloaded every 30 seconds if you used the last commits).
There is also a conf directory where you can create your automation/jsr223 directory and place files into.

[5iver]

I setup a Core Development IDE, imported your dev branch, removed the OH repo, did a Run as Maven Build... with clean install. Added /conf/automation/jsr223/ with a JS script. Debugged and got...

13:41:13.800 [ool-13-thread-1] ERROR o.o.c.a.m.s.i.ScriptEngineManagerImpl:138 - Error during evaluation of script 'file:/home/scott/git/openhab-core/demo/app/runtime/conf/automation/jsr223/testHelloWorld.js': java.lang.ClassNotFoundException: org.eclipse.smarthome.automation.core.util.RuleBuilder

And then I changed the import 😊 and got this (made it past the RuleBuilder import!)...

13:59:31.543 [le-processing-6] ERROR o.o.c.a.m.s.i.ScriptEngineManagerImpl:138 - Error during evaluation of script 'file:/home/scott/git/openhab-core/demo/app/runtime/conf/automation/jsr223/testHelloWorld.js': TypeError: Can not find a common class loader for ScriptObject and org.openhab.core.automation.module.script.rulesupport.shared.simple.SimpleRule. in <eval> at line number 7

... using this script...

'use strict';
scriptExtension.importPreset("RuleSupport");
scriptExtension.importPreset("RuleSimple");
var log = Java.type("org.slf4j.LoggerFactory").getLogger("jsr223.javascript");

var rawAPIRule = new SimpleRule() {
    execute: function(module, input) {
        log.info("This is a 'hello world!' from a raw API Javascript rule.");
    }
};

rawAPIRule.setTriggers([
    TriggerBuilder.create()
        .withId("aTimerTrigger")
        .withTypeUID("timer.GenericCronTrigger")
        .withConfiguration(new Configuration({"cronExpression": "0/10 * * * * ?"}))
        .build()
    ]);

automationManager.addRule(rawAPIRule);

Line 7 is where SimpleRule is called, so it looks like the ScriptExtensions aren't working. This will be a lot easier to debug using Jython (I can see everything available in the context using dir()), so I'm going to try getting it setup. With PDE, there was an option for adding environment variable, where I could add the Jython jar into the classpath and specify the home and path directories. In a non-ide install, these go into EXTRA_JAVA_OPTS. Any suggestions of how to do this using the Demo app?

Edit: The JVM arguments look like a good spot. Now I need to figure out the paths.

[maggu2810]

@openhab-5iver Please also follow openhab/openhab-distro#889
Perhaps I can upstream openhab-demo.
I would like to simplify the demo app usage and the option to add also non core bundles.

Have a look at these options: https://bnd.bndtools.org/chapters/300-launching.html
The app.bndrun file is a text file that can be edited simple

[5iver]

@maggu2810, I saw that, and will try it out next... thank you!

[5iver]

@maggu2810, I have Jython running in the IDE now, using...

-runvm: -Xbootclasspath/a:${.}/runtime/conf/automation/jython/jython-standalone-2.7.0.jar,-Dpython.home=${.}/runtime/conf/automation/jython,-Dpython.path=${.}/runtime/conf/automation/lib/python

Everything I've checked so far in org.openhab.core appears to be available. But I found some external libraries that are used in the modules that I did not include in the original list. After adding these...

org.quartz;resolution:=optional,\
org.osgi.service.event;resolution:=optional,\
org.osgi.service.cm;resolution:=optional,\

...to the org.openhab.core.automation.module.script bdn.bdn, everything seems back to normal except for...

from org.openhab.core.scriptengine.action import ActionService

... which is odd, since it looks like you had taken care of that one. There's also still a problem with SimpleRule n JS, but Jython doesn't have this problem...

05:09:19.186 [Dir Watcher] ERROR o.o.c.a.m.s.i.ScriptEngineManagerImpl:138 - Error during evaluation of script 'file:/home/scott/git/openhab-core/demo/app/runtime/conf/automation/jsr223/Javascript/testHelloWorld.js': TypeError: Can not find a common class loader for ScriptObject and org.openhab.core.automation.module.script.rulesupport.shared.simple.SimpleRule. in <eval> at line number 7

In regards to SimpleRule, these looked interesting... eclipse-archived/smarthome#3819, eclipse-archived/smarthome#3361.

[maggu2810]

The compat1x ActionService is optional.
You need to ensure that the bundle is added to the runtime (add the bundle) and it should be available.

Is the jython-standalone-2.7.0.jar platform independent?
I could modify my openhab-demo setup to include your changes.
If it is platform independent, can you point me to the official place to download it?

[5iver]

You need to ensure that the bundle is added to the runtime (add the bundle) and it should be available.

By 'add the bundle', do you mean in the console? I got it installed, but there was an unresolved module...

  115|Installed  |    1|openHAB Core :: Bundles :: 1.x Compatibility Layer (2.5.0.201903102136)|2.5.0.201903102136
g! start 115
org.osgi.framework.BundleException: Could not resolve module: org.openhab.core.compat1x [115]
  Unresolved requirement: Import-Package: org.apache.commons.httpclient; version="[3.1.0,4.0.0)"

Adding this one as optional didn't help. I'm still banging on it though.

Is the jython-standalone-2.7.0.jar platform independent?

Yes, it's all Java.

If it is platform independent, can you point me to the official place to download it?

There is some instruction for installing Jython with the helper libraries. Here is a link to Jython 2.7.0. I recommend the standalone jar for simplicity. That website is supposedly moving to the new official site, where you can find the latest version, 2.7.1, but there are some quirks to using it. I have updates for the helper libraries so that 2.7.1 can be used, but I was waiting to release them with any changes needed for the ESH reintegration. Details for some of the issues with using 2.7.1 can be found here, along with a link to the Jython Maven repo with 2.7.1.patch2618, which resolved one if the issues.

I haven't been successful in getting Groovy to load in the demo app.

[maggu2810]

@openhab-5iver I have added a commit that contains the modification for compat1x support to maggu2810/openhab-demo. I assume you missed only to press the "Resolve" button.

[maggu2810]

@openhab-5iver I have created a branch "jython" that you could perhaps use to work on.

[5iver]

OK, thank you... I just pulled the other one and got everything back in place 😄.

[maggu2810]

@openhab-5iver Are all missing Java packages available now?
If yes, I will create a PR.

[5iver]

@maggu2810, sorry, I got distracted fixing up my PR. From my testing, it looks like everything that the Jython helper libraries accessed is available now. I'm sure there will be more to add as the libraries evolve, but I am pretty clear on how to add them in the future, if needed. The only thing I never got to completely check on was org.openhab.core.scriptengine.action, but it will be much easier to test this after what you have is merged.

I had setup a recent snapshot (S1553) with automation jars that I'd build from your branch, and found that the error that was happening with Javascript was not occurring. So, that may be an issue in the IDE only. However, Groovy still seemed to have an issue.

We're much better off after these changes, but there will need to be some more cleanup. Thank you for all you do!