[jsscripting] ES6+ Support

[jpg0]

Parent PR to track all the changes required to get ES6+ support into OH. This supercedes #6399 and was reopened from #8441

Remaining dependencies:

• script module access ([automation] Extracted accessor into interface and added to script engine context … openhab-core#1843)
• engine identifier (blocks [automation] Added script filename to engine, if engineIdentifier is a file openhab-core#1196) [automation] Pass script context to script engines openhab-core#1837
• dependency tracking ([automation] Script dependency tracking openhab-core#1884)

[TravisBuddy]

Travis tests were successful

Hey @jpg0,
we found no major flaws with your code. Still you might want to look at this logfile, as we usually suggest some optional improvements.

[cpmeister]

Is this still being worked on? I can't review this if this PR stays a draft.

[jpg0]

@cpmeister yes it is, but it has a number of prereqs that need to go in first. These are the ones that are linked in the description. The intent is that once these are complete I can move this one out of 'WIP'. I can certainly push code against this PR so that you can see what it will be like, however it wouldn't compile and would be likely to change as the prereqs go in.

(Also, I can certainty speed up opening more of the PRs in the description, however I have not been pushing too hard on this as the currently open ones have not been progressing very quickly. Any help on these would be appreciated!)

[CrazyIvan359]

Does this replace ES5 with ES6 or does it allow running them in parallel?

[jpg0]

@CrazyIvan359 great question. The way I have it working is that it replaces the script engine for any files ending in js when used in scripted automation. So yes it replaces ES5 (although ES6 should be backwards compatible), but it does not replace JS transforms (yet).

[CrazyIvan359]

Great news! I don't think JS Transform needs to be a priority anyway. Such transforms are not the most common and are fairly simple so I don't think the limitations of ES5 are felt as much.

[jpg0]

This is pretty much ready for review, pending merge of the last remaining dependency. The surface area between this and the dependency are small, so I'll move this out of WIP now as it's pretty much ready for review.

(It won't quite compile until that PR is merged, I'm expecting this.)

[jpg0]

Added minor change to remove hard dependency on script dependency tracking. Will now warn if this feature is not present, but otherwise will work (including compiling here).

[jpg0]

org.openhab.automation.jsscripting-3.0.0-SNAPSHOT.jar.zip

Sample jar from my build. Untested, but should work on 3.0.

[LukasA83]

org.openhab.automation.jsscripting-3.0.0-SNAPSHOT.jar.zip

Sample jar from my build. Untested, but should work on 3.0.

I'm willing it to test it over the next week. Right now my setup is based on js files containing the rules with scripts around it. What do you recommend as a starting point?

[Confectrician]

Maybe this would also be a good start for adding some content for the readme @LukasA83 and @jpg0.

I have reduced the jsr223 generic documentation a bit for openHAB 3 since it makes more sence to me, to cover the actual state within those addons, to get the latest changes included just in time with the merge.

If you can provide some starting point we can collect useful informations and file a pull request against your repository. :)

[jpg0]

Great idea!

Some notes whilst I'm thinking about and in case anyone wants any hints...

• Language - this is basically just ES2020. We can link to the GraalJS docs for more details. Note that whilst I've explicitly enabled Nashorn compatibility, not everything has been ported over.
• Modules - standard CommonJS support.
  • I believe that ES6 modules may also work now in the later versions of GraalJS that I've included, but I have not tested this.
  • 3rd parties modules work fine (and can be installed in the standard way with npm/yarn/whatever), unless they rely on native code.
  • Openhab extensions are now imported via the module system, under a reserved name. Currently this is @runtime (although I'm open to changing it). So to import rulesupport: const rulesupport = require('@runtime/RuleSupport');.
• Directory layout
  • Scripts are loaded from automation/jsr223/javascript/(personal|core|community)/ as normal
  • Libraries are a little different, because the module loader in GraalJS is very specific to NodeJS, and requires loading from a single node_modules library, directly implementing the node spec. This is currently set to automation/lib/javascript/personal. Currently I keep personal libraries in the node_modules subdirectory here, and 3rd party libraries in ../node_modules (relative to personal). This all follows the node spec, but is unwieldy. (TBH I think the automation directory layout needs an overhaul, but this works for the moment.) It would be possible to redirect the library loading stuff by writing a custom Graal Filesystem, but too much divergence would lose compatibility with npm etc.

My personal configuration is online: https://github.com/jpg0/oh-config/tree/master/automation so may be a useful reference.

Should all the documentation be in the add-on repo?

[Confectrician]

Should all the documentation be in the add-on repo?

Yes.

Jython docs are currently moving in the corresponding addon too and groovy docs will be served from this repo as well.
This way we can keep up with all changes in the codebase.

[lewie]

@jpg0, I copied your org.openhab.automation.jsscripting-3.0.0-SNAPSHOT.jar into addons folder. And I have a started bundle
"208 │ Active │ 80 │ 3.0.0.202012182330 │ openHAB Add-ons :: Bundles :: Automation :: JSScripting" in my list output.
Copied your "oh-config/automation/" tree into conf/automation/ in openHAB 3 folder.

Runs on Windows 10, installed is Java 11.

But nothing happens. What am I missing for happiness?

[jpg0]

Nothing at all happens? Even without my add-on something should happen as the Nashorn engine should attempt to process the JS files (and fail).

[csowada]

Nothing at all happens? Even without my add-on something should happen as the Nashorn engine should attempt to process the JS files (and fail).

I'm able to load simple scripts, but the code is not compatible with openhab 3.0? I'm not able to load services etc.

Here an example

Error: Failed to get any services of type(s): org.openhab.core.items.MetadataRegistry,org.eclipse.smarthome.core.items.MetadataRegistry

win 10, graalvm, oh 3.0 release !

But you already said untested for oh3.0 !

But I like you way you want to go, but is it not better to use the node_modules folder within the jsr223 folder? My own extension I already build for oh 2.5 only loaded files on the root level and skipped all files in folders. This was the work of require`(...) to load these files. I use a new jsr223 folder and add an own FileWatcher to adjust the original Watcher behaviour.

https://github.com/csowada/org.eclipse.smarthome.automation.module.script.jsext

[jpg0]

@csowada the error you are seeing there is triggered by the ohj library, not basic JS execution. If you are copying my configuration, you may see these problems. If you want to get ohj working, you'll need to follow the instructions on jpg0/oh-config#2 (comment). I believe that the problems here are related to startup order regressions in OH3.

As for where the node_modules folder should reside, I don't really mind. I picked what I thought was closest to the current OH directory structure (putting libraries in lib), although TBH I think that the OH structure needs an overhaul. Node does not allow multiple library roots (personal/community/core), and also the subpath matters (so you can't just pick the parent directory, as modules would then need to be referenced as 'personal/library'). Also the current layout seems overspecified to me; there should probably not be per-language folders, nor the personal/community/core split, nor jsr223 in the path etc.

I'd be perfectly happy to see the structure change, or to support it, but I'm not a maintainer of that area or anything.

[simonihmig]

As an OH user and an early adopter of @jpg0's great work here (the GraalVM integration bundle + his ohj helper library), let me bring in my thoughts here (and sorry if this is getting a bit longer, and goes beyond the scope of this initial PR).

So first of all, I feel really excited about what's happening here. I think a solid integration of modern JavaScript could provide a huge boost for the future of OH, as...

• JavaScript is the lingua franca of the web
• as such opens the door for a huge community of developers, who are comfortable with building stuff in JS (basically the lingua franca of the web) but not so much in Java (people like me)
• has a huge ecosystem of libraries (npm), covering also a ton of IoT/smarthome stuff
• seems like a perfect fit for building applications (see below) on top of the core Java stuff (core, OSGI, bindings etc.)

Besides the modern language features of ES201x that we get here (the DX in Nashorn in comparison feels like browsing today's web with Netscape Navigator 😜), the ability to easily share that code is at least just as important IMO, building on top of the existing node module support (ESM hopefully in the future) and npm infrastructure.

I always felt like there was some abstraction layer missing in OH, to build applications on top of the basic infrastructure (bindings etc.), that are more elaborate than just some simple if-then rules, are easily sharable and ideally can also be held accountable to some higher engineering principles (less spaghetti code wrapped in rules, better abstractions, modularization, test coverage etc.). With regards to sharing, it seems currently you have the choice of either reinventing the wheel, or looking through the forums to find some xtend based rule snippets with very ad-hoc style bespoke implementations, with zero separation between common logic and use-specific aspects (e.g. hard-coded item references all over the place).

To give you some examples for applications (vs. simple rules) that I have on my to do list:

• home security alarm system
• sprinkler automation (e.g. based on configurable zones with individual properties, evotranspiration calculation based on weather data, basically what opensprinkler is already doing)
• rollershutter/blind automation, e.g. sun protection (based on individual window orientation, sun position etc.)

All of these examples have in common that they share common logic, are more elaborate that just a few simple if-then rules (e.g. could benefit from some higher level abstractions like using a finite state machine library like xstate) and require some configuration to bring in the user-specific aspects (e.g. defining specific conditions, wiring to existing items).

Tbh I am not really aware if those "rule templates" are a thing already and if they will support use cases like this (i.e. apps vs. rules)? But I do see a story of sharable code unfolding here with the support of modern JS and sharable npm packages (which the new GraalVM bundle supports, in contrast to the basic Nashorn engine AFAIK). Maybe in the distant future this could be made available in the UI, with some app repository (e.g. based on npm), generative forms for configuration and what not. But even in the most basic form as it would be with this PR, you could share code like this:

• Someone could write some generic awesome-oh-alarm-system library and publish it on npm
• users then would be told to
  • go into their automation folder and yarn add awesome-oh-alarm-system (or npm)
  • create a simple script entrypoint like:

const alarm = require('awesome-oh-alarm-system');

alarm({
  // specific config comes here, like listing all contact items, alarm signal devices, customized timeouts etc.
});

So, to support use cases like this, the work here is obviously super important. But based on my early experimentation with this, I see a couple of serious problems though:

• as has been mentioned before, the file layout seems pretty weird. Especially the fact that libraries are supposed to live in a totally different place than the scripts:

  • This may be a common thing in other ecosystems (I frankly don't know), but is highly unusual in the JavaScript/Node.js world. You run npm install whatever in your project's root folder, and that places that library (and all its transitive dependencies) into ./node_modules, and not into a distant arbitrarily defined folder. This would create a new barrier to entry for JS-savvy folks that are new to OH.
  • While node.js did historically support arbitrary library folders (using $NODE_PATH), this has become out of practice, and is even discouraged in favor of its normal module resolution algorithm, see https://nodejs.org/dist/latest-v14.x/docs/api/modules.html#modules_loading_from_the_global_folders:

  It is strongly encouraged to place dependencies in the local node_modules folder. These will be loaded faster, and more reliably.

  • Using a non-standard library path will probably make using popular tools like ESLint or TypeScript difficult or impossible, when they are not able to understand what is getting imported (require()). Especially TypeScript support can be very beneficial, even when writing your own code in "just" plain JS due to the vastly improved Intellisense support you get in VSCode when libraries ship their typings. But it will totally freak out when it cannot "see" what the imports refer to.

  Therefore I would suggest to use simple local node_modules that "just work", even if only for the JS scripting part. But obviously that is something more "political", that others have to decide. Implementation note: I once tried out a local node_modules folder, and OH totally freaked out and came to a crawl, probably because the JS engine tried to load each and every ./node_mopdules/**/*.js file as a script. In this case we would probably need some mechanism to define "entrypoints", like a naming convention like *.script.js (similar to *.test.js files)

• Next point is that the current integration apparently lacks support for async rules (see Async rules not supported? jpg0/ohj#14). However asynchronicity is ubiquitous in modern JS. Currently it seems anything that would normally be scheduled for a future event loop (be it a Promise or callback) will just not be executed. That would mean anything as simple as a network request would inherently be impossible to do (currently not possible anyway due to the next point), at least in JS-land without resorting to calling into (sync) Java APIs.

• Somewhat related, currently you can import external vanilla-JS modules, but not any of the core node.js modules (like fs), as there is no full-blown node.js environment. That also means you cannot use any other npm package that (directly or indirectly) expect those APIs to be available (which most non-browser libraries do!?). AFAIK GraalVM also supports a full node.js environment, but it is beyond my pay-grade to asses what's needed to bring this into the OH integration.

Maybe not all of those concerns are relevant for the first iteration of integrating GraalVM, but I wanted to bring this up as early as possible nevertheless. At least it would be beneficial when adding things iteratively to not cause too much churn for users by introducing breaking changes (which mostly would be the case for the first point of library paths).

That's it for now, sorry for the long rant! 😆

[jpg0]

@simonihmig I think I agree with pretty much everything you have said!

As another example to libraries - I actually have my HVAC implemented into a JS library just as you suggest, such that config drives when you want certain temperature ranges in different zones, plus which items control the zones, monitor the temp per zone, monitor the doors and windows in the zone etc.

Regarding the problems you mention:

• I believe that the filesystem layout is the most important issue here. The challenge is reconciling it with other languages. I have not attempted to make significant changes here, because when I've raised it in the past I've got nowhere. This is something which is not simple to change later. If there is support from core OH maintainers, I'm sure that we could propose a new layout here; I'd be happy to write the code to implement it.
• I agree that lack of async support is an issue, and it prevents the use of various 3rd party libraries. I do however believe that it will be possible to add in, such that it unlocks the use of these libraries. I'll certainly attempt this once we have GraalJS in OH. I don't see it as a serious issue, rather an incremental piece of functionality that has not yet been added. This would only change if it led to many OH users forking / reimplementing libraries that could otherwise be used as-is. Certainly something I want to add (there are some libs that I want to use that need it...)
• I feel the same way about the core node.js modules; some things work, not all, but let's get support for core modules next. Fortunately the GraalJS devs changed the resolution behaviour after I requested it so that we can provide our own implementations (possibly Java-backed), although rebuilding them is not trivial. I am optimistic that we can get their core node.js modules in somehow; it does seem that they are gradually porting functionality over to the embedded scripting runtime (like they have for modules support).

[wborn]

It seems to break the build, can you have a look at it @jpg0?

See: https://ci.openhab.org/job/openHAB-Addons/370/console

[jpg0]

I'm looking at it. I'm not sure what has changed (as it compiled before), but it looks like something related to nullability for type parameters. Compiles fine in my ide which is annoying...

[simonihmig]

I am happy this is going forward now! 🎉

However what I am still missing is any decision or at least comment / note / thought from core folks regarding the issues that have been raised here, most importantly the issue of where libraries should live (in the current way in separate folders or in local ./node_modules which is the de facto standard for the whole JS ecosystem). As explained in my longer comment I think this decision has implications on DX and thus adoption of JS-in-Openhab as a whole that you just cannot overestimate!

And as @jpg0 stated, and I totally agree, the other "issues" can all be seen basically as missing features that can be added over time. But changing the module resolution later is a breaking change, and would be super annoying for early adopters of this new exciting JS integration!

So at least something along the lines of "problem understood, our vision is this, next step is that" would be good to hear from the people who can make these calls! /cc @wborn @kaikreuzer

[wborn]

Would it make sense for this add-on to also use any libraries in $OPENHAB_CONF/automation/lib similar to Jython Scripting? To allow such configuration to be overridden, you could use a system property or use add-on configuration parameter(s).

[jpg0]

Would it make sense for this add-on to also use any libraries in $OPENHAB_CONF/automation/lib similar to Jython Scripting? To allow such configuration to be overridden, you could use a system property or use add-on configuration parameter(s).

This is precisely how it works currently. The problem is that by default, node will always search ./node_modules for libraries (which has now become a standard), not this directory. It's possible to workaround this to some degree by changing to this directory before install libraries (with npm / yarn / etc), which works for most cases, but if libraries themselves (or more likely - build tooling) relies on the hardcoded (relative) directory then it won't work.

Additionally, the current ScriptFileWatcher will always try to load all scripts from under the jsr223 folder, so would try to load any libraries in a contained folder.

Possibly what could be done is:

• disable script loading from jsr233/javascript/node_modules
• try to load JS libs from this directory
• fall back to loading from the automation/lib/.. path

This would likely provide compatibility with the existing lib split, and also allow users to switch to the "node native" approach. The only nasty part is having to build JS-specifics into ScriptFileWatcher, although I guess this could be contributed by the plugin.

[csowada]

What is with the idea of an .scriptignore file similar to a .gitignore file. Then everyone can add files or folders that should be ignored.

[jpg0]

@wborn thanks for fixing up the build errors - was nighttime in Aus so couldn't get to it until later.

[jpg0]

@csowada The more I think about the use-cases related to this, the more I question whether we actually want that level of configurability. I see 3 main use-cases:

• Novice script writer who simply wants to add basic scripts with no libraries. Most likely these people will be using the UI rather than files anyway.
• Intermediate script writer who does want to use libraries. In this case they will want to use files. They may possibly be familiar with another scripting language so would expect the existing openHAB directory layout. However this may not be the case, as if they were more familiar with JS, they may expect the standard JS project layout.
• Advanced script writer who wants a full JS project style experience.

I'm not sure that we care about the first, as it will likely be handled by the UI, and generally does not require libs. The second use-case I personally would opt for the standard JS layout rather than a generic openHAB one (note that it would prevent mixing across language types - e.g. consuming a lib from a different language - but I'm not sure that we care about this). The third use-case prioritises the standard JS layout. This means that we should either optimise for one layout for JS (the JS/node one), or two (this plus the existing openHAB one). This feels much simpler than providing configuration allowing many. I'm sensitive to fragmentation of the user experience here, so prefer to have fewer options (as well as less code!).

[csowada]

@jpg0 I have adjusted your PR a bit and it is missing just a little to implement more complex javascript. My change was the following.

Customize ScriptFileWatcher

• Only scan on Javascript files in the root directory, also skip node_modules directory

Here I see the ignore file to provide a common and simple approach for all script engines.

Modify your PR

• Use the node_modules directory for the require loading or the default fallback

These two changes are enough to implement real node.js like projects. For this I chose a different directory than automation\jsr223 to run the change as a separate bundle.

See the bit outdated changes here
https://github.com/csowada/openhab2-addons/tree/jsscripting/bundles/org.openhab.automation.jsscripting/src/main/java/org/openhab/automation/jsscripting/internal

So two small changes would be enough for all openHAB users to be more flexible. And not too much code changes.

1. enhance ScriptFileWatcher to allow an ignore file like .gitignore. Then the user can exclude the unwanted folders and subfolders by himself. It would be a new PR regarding the ScriptFileWatcher.
2. try to load modules with require from node_modules or the default library folder if it fails

I would also see myself more as "Intermediate" or "Advanced" scripter as I use TypeScript for all of my rules :-)

[jpg0]

@csowada instead of adding a custom ScriptFileWatcher, would it not be better to allow the existing one to be configured by script add ons? So when the JS script add-on loads, it registers itself specifying both the script and lib paths? I'm concerned that a custom file (and hence user, rather than add-on configuration) will make things more complex.

[csowada]

@jpg0 Oh, you are right. Wrong written. I mean that it makes more sense to modify the current ScriptFileWatcher, the custom one was only for my own test implementation. But you think it would be better to manage/modify the ScriptFileWatcher by the ScriptEngine. But this approach is more complex than a simple exclude file or not?

[jpg0]

I would expect an exclude file to be more complex. This is because in either approach, the ScriptFileWatcher needs to be able to accept and respect some kind of exclusion list. However with the file approach, it also needs to load and parse the file, monitor it for changes, apply changes during script loading etc. It's also something thing users would need to understand, rather than for it to 'just work' like node does.

[ghys]

Hi, I'm just becoming aware now that we have a new modern JS script engine, and I'm thrilled, that's great!

From the UI perspective, could you confirm that once it's installed, it replaces the Nashorn engine for the application/javascript media type, and if so, is there a way to distinguish whether we have Nashorn or the new engine installed?

I'm bringing that up because in the CodeMirror ecosystem that's used in the UI, you can choose which ECMAScript revision you want to target, and it will present you with different autocompletion hints. If there's no indication on the version of ECMAScript that's being used, the autocompletion might be inaccurate, and we don't want that.

Also - I'm confident this new JS implementation can also work for Blockly script actions but we'll have to do thorough testing to confirm that.

[jpg0]

Currently, the implementation hides the Nashorn engine, but only in the scripted automation part of openHAB - not (for example) in the transforms part.

As for how to tell whether it's installed, there is no explicit mechanism that I'm aware of to do this - I suspect that this may be the first example of two different engines implementing the same language type.

One way would be to lean on OSGi and see if you can get a reference to one of the components this bundle exports. It does sound like it may be worth adding a feature into openHAB itself to make available the engine details for each script type.

[jpg0]

FYI to watchers of this PR, I have submitted another PR to have openHAB core ignore "./node_modules" for scripts, so that we can use this for JS libraries instead, as requested in this thread and elsewhere, to allow compatibility with 3rd party JS tools and libs.

openhab/openhab-core#2408