[hdpowerview] PowerView Gen 3 support

[jlaur]

This issue is created as a placeholder for gathering information about the new PowerView Gen 3 and discussing how to support it. I have requested API documentation from Hunter Douglas. They have responded: "This information has not been released to the general public yet, we are working in beta with selected groups as needed."

What we know so far:

• Proprietary RF communication will be replaced by standards-based Bluetooth Low Energy (BLE) protocol.
• The new protocol offers instant two-way synchronization and improved reliability and range.
• A new PowerView Gen 3 Gateway will be introduced, probably similar to PowerView Gen 2 Hub, but communicating over BLE with shades and accessories.
• A new PowerView Pebble Remote will be introduced, communicating over BLE.

Open questions:

• Will the PowerView Gen 3 Gateway support the proprietary RF communication for backwards compatibility with legacy shades and accessories? Answer: No, two hubs/gateways are needed when having a mix of Gen 1/2 and Gen 3 shades.
• Will the PowerView Gen 3 Gateway remain open for integrators like the PowerView Gen 2 Hub? Answer: Yes.
• Will the PowerView Gen 3 Gateway implement a new API and/or support the existing API? Answer: There is a new API - see issue comments.
• Will the PowerView Gen 3 Gateway support the Matter standard at some point?

Sources:

• https://restechtoday.com/hunter-douglas-debuts-powerview-gen-3-automation/
• https://www.cepro.com/control/motorized-shades/hunter-douglas-powerview-gen-3-series/

[jlaur]

@andrewfg, @arroyoj - FYI.

[kingy444]

The fact they havent released the api is not a great sign for any backwards compatability

but fingers crossed for the new hub to support the old protocol - be nice if shades automatically updated their postion with the hub when finished moving

[andrewfg]

The fact they havent released the api is not a great sign for any backwards compatability

Or it might mean that the API did not change :)

[kingy444]

Likely some bad news unfortunately

https://community.home-assistant.io/t/hunter-douglas-powerview-gen-3-integration/424836/3

[andrewfg]

The current (older) API uses plain text over HTTP with no encryption and no authentication. So it is pretty archaic. Therefore my guess is that they will have moved to HTTPS and some form of authentication (perhaps password based or token based).

{ Note: I had the exact same experience in the last month with the Heatmiser Neohub where the manufacturer migrated from plain text HTTP to a token based web socket connection instead #12915 (comment) and in this case, they kept support for the legacy plain text HTTP but now it requires the user to explicitly switch it on in the App }

[kingy444]

Yea that makes sense - would be good if they added it via firmware to the old hubs too tbh - just make it optional

[andrewfg]

Yeah. In the Heatmiser App, it is as simple as this..

[andrewfg]

would be good if they added it via firmware to the old hubs

PS in my App there is an option 'Receive Early Access Updates' (Opt in to receive early access to firmware updates before official release) ... however I am extremely nervous about whether or not to enable it ...

[jlaur]

PS in my App there is an option 'Receive Early Access Updates' (Opt in to receive early access to firmware updates before official release) ... however I am extremely nervous about whether or not to enable it ...

IMHO you should try it. 😆

[jlaur]

Btw, in the linked thread a port scan was made, and port 443 wasn't open.

[andrewfg]

you should try it

LOL

[andrewfg]

@jlaur here is another article that answers some of your questions about Gen 2 vs Gen 3 inter-operation..

https://www.slashgear.com/850488/hunter-douglas-smart-blinds-embrace-bluetooth-why-thats-a-big-deal/

[andrewfg]

@jlaur someone made the following post on the HA forum; it's not definitive, but the guy sounds like he knows something..

https://community.home-assistant.io/t/hunter-douglas-powerview-gen-3-integration/424836/17?u=andrewfg

[andrewfg]

@jlaur I found the following link, which implies that some commercial companies have developed integrations for Gen3 hubs. The web page has an email link, and I have already emailed them asking for information about the Gen 3 API. I will keep you informed if I get any response.

https://controlconcepts.net/manufacturers-modules-and-drivers/hunter-douglas/

[andrewfg]

@jlaur the response was as follows..

Do you have access to a Gen 3 Gateway and shades? The Gen 3 API documentation lives on the Gateway itself using Swagger. Swagger also allows for easy testing of the REST API, showing command structure and results.

By default, Swagger is disabled on Gateways as it does utilize system resources and is only needed during integration development. http://powerview-g3.local/gateway/swagger?enable=true will enable Swagger. Note that Swagger must be re-enabled after each Gateway power cycle.

To get to Swagger once enabled, use http://powerview-g3.local:3002/

[vves]

@andrewfg @jlaur Hi, I wanted to introduce myself. I am the product architect for Gen3 can provide some unofficial guidance and support for this integration. Cheers and Code On! @jlaur is correct that the starting point for the integration is the Gateway Swagger docs listed above.

[jlaur]

@andrewfg @jlaur Hi, I wanted to introduce myself. I am the product architect for Gen3 can provide some unofficial guidance and support for this integration. Cheers and Code On! @jlaur is correct that the starting point for the integration is the Gateway Swagger docs listed above.

Thanks for stepping in and offering support, @vves. For supporting the Gen 3 gateway we currently lack documentation and/or users (in order to get hands on), since we are currently not in procession of the new gateway. I recently added new TDBU shades to my setup, so I now have 11 shades on a Gen 2 Hub, so for me personally I'm unfortunately nowhere near a Gen 3 upgrade.

[andrewfg]

@vves I would also like to thank you for your support; perhaps the best is if you can send us the swagger data (presumably a yaml file?); however alternatively perhaps you can expose one of your hubs via a port forwarding, or vpn connection, so we can talk to it ourselves?

[kingy444]

Thanks for reaching out @vves

I do Dev work on the HomeAssistant side and have been working with @andrewfg closely over the last couple of months (great to see collaboration between ‘competing’ platforms)

As Andrew suggested if we could get access to the api doco in some alternative way that would be great.

While offline development without a hub isn’t great (blindly hoping your code logic works) our only other option is to hope another ‘developer’ gets a gen3 hub installed - I couldn’t afford to spring for new blinds just to help the community out 😂

Again really appreciate the response - I’d personally tried to reach out to Hunter Douglas and had no response.

Not sure if you have seen some of our other conversations but one thing that would be insanely helpful is the definition of your shade capability and types. Everything we currently have is community driven and relies solely on someone providing their json output to us.

Even just a listing of shades and their capability type would be great for aesthetics but really would be great if some of our ‘guesses’ on capability were substantiated.

[vves]

My preference is to use the Gateway Swagger endpoint as the HD dev team is still making minor tweaks to the API. That said, I'll provide static swagger doc here but first I need to cull some private data from swagger. Expect an update by end-of-week.

[andrewfg]

@vves many thanks for your support; much appreciated.

[andrewfg]

@jlaur as "homework" for this integration, you may want to take a look at the tado binding which uses Swagger CodeGen to parse the Swagger 'yaml' file and auto- generate the respective Java classes for the DTOs and the web service.

[andrewfg]

FWIW a Home Assistant user kindly did a screen capture from their Gen 3 hub (see below). It is interesting, but I am still hoping that @vves can post us the yaml file :)

@jlaur of immediate interest to me are the following..

• The urls in Gen 2 hubs like '/api/shades' seem to now be '/homes/shades' instead.
• There seem to be ten capabilities rather than nine ;)
• .. more no doubt to come ..

[vves]

@andrewfg Even better - I am putting a 'Getting Started with Integrations' document together. A deep integration only needs around 10 total calls. The rest of the API is for internal Hunter Douglas tools and will end up being redacted in the future.

[andrewfg]

putting a 'Getting Started with Integrations' document together

@vves please don't feel that you need to make that extra effort on account of us. We already have full integration for Gen 1 and Gen 2 hubs so (dare I say it) we are beyond the 'getting started' phase. All we need from you is the extra information concerning how Gen3 hubs differ from Gen 1/2 ones..

A deep integration only needs around 10 total calls

Yes. In fact our integration for the Gen 1/2 hubs goes a bit deeper than ten. We use seventeen calls as you can see from the HDPowerViewWebTargets.java code. Albeit some of these are GET/PUT variations on the same uri.

[vves]

@andrewfg there are some significant changes in Gen3 as all shade information is now live and event driven instead of requiring polling for shade and scene state. It's these changes that I want to highlight and provide best practices for all integrations - not just openHAB. Coming soon!

[andrewfg]

all shade information is now .. event driven

Ah yes. Had you not mentioned it, I would most probably have overlooked it. ;)

[andrewfg]

@jlaur the first Pull Request #13352 is ready for your review. This PR changes the architecture by splitting "hub facing" classes into base abstract classes that define the interfaces, and specific implementation classes for V1. This PR should function exaclty as the existing binding since only the code structure has changed.

---

EDIT: I had originally opened #13339 but I abandoned it and closed it because I made a real mess of the GIT commits history. And replaced it with #13352 instead

[andrewfg]

@jlaur I have now added the second PR #13355 which is based on the Generation 3 documentation we had already recieved from the HA user community. It does however still require some further work depending on the clarifications that @vves will be providing.

[andrewfg]

@jlaur I made the attached PDF to as an aid to navigate the files in my two PR's ..

Class Structure.pdf

[andrewfg]

@jlaur a few days ago you asked if we need a completely new Bridge class; the answer is no because when its initialize() is called we can send an HTTP GET 'ping' to either a Gen 1/2 hub API url end point, and/or a Gen 3 hub API url end point, and depending on which GET call succeeds, we can produce and consume either a HDPowerViewWebTargetsV1 or a HDPowerViewWebTargetsV3 class from that point onwards.

And if HDPowerViewWebTargetsV1 has been chosen, it in turn produces and consumes ShadeDataV1, ShadePositionV1, SceneV1, and ScheduledEventV1 class instances; whereas if HDPowerViewWebTargetsV3 has been chosen it produces and consumes the respective xxxV3 class instances.

[jlaur]

@andrewfg - Gen 1/2 vs. Gen 3 detection IMHO should happen during discovery or manual bridge configuration, so that after this point there is no need for pinging any bridge to discover own identity.

I haven't had much time prioritized for this project lately, so I'll be back with input for some of the other open questions.

[andrewfg]

should happen during discovery or manual bridge configuration

I think choosing during discovery would make sense; actually discovery code is something that I am 'reserving' to do in PR #3..

Nevertheless I am NOT convinced about your suggestion about manual configuration: why ask the user to configure something manually when it can easily be done automatically? Perhaps the compromise solution is a kind of auto-manual approach; if the config param is initially empty, then we ping the hubs and set the config param accordingly? After that the user can (if they have any reason to do so) force override it.

By the way, apropos config params: as you know there are currently two params 'polling interval' and 'hard refresh interval' which I am using as follows..

• V1 hubs: Execute poll requests continuously at the 'polling interval', and execute hard refreshes continuously at the 'hard refresh interval' (no change from the current binding)
• V2 hubs: At start up it executes just one poll to initialize the start state, and after that there is no more polling because all state changes are notified via SSE. However at the hard refresh interval it will still execute a poll in order to resynchronise the state in case any SSE events had been accidentally lost, (and it also checks that the SSE socket is still open, and if not, it re-initializes the SSE connection).

haven't had much time prioritized for this project lately

No problemo :)

[jlaur]

I think choosing during discovery would make sense; actually discovery code is something that I am 'reserving' to do in PR #3..

Nevertheless I am NOT convinced about your suggestion about manual configuration

Not much to discuss. Discovery is only for managed things, so for file-based configuration you need to be able to provide this information manually. That's also why there is properties/configuration interoperability (can dig up documentation and/or examples later). For sure I don't want my hub "pinged" on each initialization just because I'm not using managed things.

[andrewfg]

I don't want my hub "pinged" on each initialization

I don't understand your resistance. The 'ping' is just the get firmare request. And that IS done every time that you initialise the thing. So in reality we are not doing anything different than what happens already.

---

EDIT: this is the code..

    /**
     * Instantiate the web targets.
     *
     * @param host the ip address
     * @return instance of HDPowerViewWebTargets class (either V1 or V3).
     * @throws InstantiationException if neither a V1 nor a V3 web target was instantiated.
     */
    private HDPowerViewWebTargets newWebTargets(String host) throws InstantiationException {
        HDPowerViewWebTargets webTargets = this.webTargets;
        if (webTargets != null) {
            return webTargets;
        }
        try {
            // try communicating via V1 web targets
            webTargets = new HDPowerViewWebTargetsV1(httpClient, host);
            webTargets.getFirmwareVersions();
            this.webTargets = webTargets;
            return webTargets;
        } catch (HubProcessingException | HubMaintenanceException e) {
            // fall through
        }
        try {
            // try communicating via V3 web targets
            webTargets = new HDPowerViewWebTargetsV3(httpClient, host);
            webTargets.getFirmwareVersions();
            this.webTargets = webTargets;
            return webTargets;
        } catch (HubProcessingException | HubMaintenanceException e) {
            // fall through
        }
        throw new InstantiationException("Unable to instantiate the web targets");
    }

[jlaur]

I don't understand your resistance. The 'ping' is just the get firmare request. And that IS done every time that you initialise the thing. So in reality we are not doing anything different than what happens already.

That's a valid point. But if the firmware request is different between Gen 1/2 and Gen 3, then it will still make one unneeded call for Gen 3 (from code above), unless I'm missing something?

[andrewfg]

^
Yes. On Gen 1 there is no excessive call. Whereas on Gen 3 there will be one extra call to a uri which the hub will fail (with an HTTP 503 error according to the Swagger draft).

[jlaur]

Yes. On Gen 1 there is no excessive call. Whereas on Gen 3 there will be one extra call to a uri which the hub will fail (with an HTTP 503 error according to the Swagger draft).

So where do you see the problem in storing the hub/gateway type during discovery/manual configuration as opposed to having to rediscover it on each initialization?

[vves]

I recommend using mdns (bonjour) to check for Gen3 availability. The Primary gateway and REST interface - if a Gen3 Gateway is online - will always be found at 'powerview-g3.local'.

[andrewfg]

So where do you see the problem in storing the hub/gateway type during discovery/manual configuration as opposed to having to rediscover it on each initialization?

I simply don't want to force the user to do manual configuration.
AND to be specific I do not want to break any existing Gen 1/2 installations i.e. it must be a non breaking change

[andrewfg]

if a Gen3 Gateway is online - will always be found at 'powerview-g3.local'.

Thanks @vves for the tip; actually I was indeed planning to do something like that since we already use MDNS to discover Gen 1/2 hubs on '_powerview._tcp.local'.

PS my prior discussion with @jlaur is not really about using MDNS or not; but rather about using hard config settings (his suggestion) as opposed to auto-config (mine, and apparently yours too..)

[andrewfg]

the problem in storing the hub/gateway type during discovery/manual configuration

@jlaur proposed solution as follows..

1. for auto discovered things we can use mDNS as @vves suggests to set a 'representationProperty' called (say) 'generation' that can have an integer value or 1 or 3 depending on which mDNS target had discovered the thing; and when the thing is instantiated that causes a respective configuration parameter of the same name to have that value.
2. for manual defined things, the new configuration parameter 'generation' can be optional; (this ensures that it is a non breaking change on Gen 1/2 installations); if it exists (with a value 1/2 or 3) then the webTargets are created specifically for that generation; and if it does not exist (int value == 0) then we can use my auto-initialise code as follows..

    private HDPowerViewWebTargets newWebTargets(String host) throws InstantiationException {
        HDPowerViewWebTargets webTargets = this.webTargets;
        if (webTargets != null) {
            return webTargets;
        }
        HDPowerViewHubConfiguration config = getConfigAs(HDPowerViewHubConfiguration.class);
        int hubGeneration = config.generation;
        switch (hubGeneration) {
            case 0: {
                // optional config.generation parameter is missing; instantiate webTargets automatically
                try {
                    // try communicating via V1 web targets
                    webTargets = new HDPowerViewWebTargetsV1(httpClient, host);
                    webTargets.getFirmwareVersions();
                    hubGeneration = 1;
                    break;
                } catch (HubProcessingException | HubMaintenanceException e) {
                    // fall through
                }
                try {
                    // try communicating via V3 web targets
                    webTargets = new HDPowerViewWebTargetsV3(httpClient, host);
                    webTargets.getFirmwareVersions();
                    hubGeneration = 3;
                    break;
                } catch (HubProcessingException | HubMaintenanceException e) {
                    // fall through
                }
                break;
            }
            case 1: 
            case 2: {
                webTargets = new HDPowerViewWebTargetsV1(httpClient, host);
                break;
            }
            case 3: {
                webTargets = new HDPowerViewWebTargetsV3(httpClient, host);
                break;
            }
        }
        if (webTargets != null) {
            this.webTargets = webTargets;
            return webTargets;
        }
        throw new InstantiationException("Unable to instantiate the web targets");
    }

[vves]

@andrewfg We are on the same page. First time auto-discovery using both mdns addresses is user friendly as (from our experience) most users are not aware of Gen1/2/3 differences or even what generation they have - and then cache what you found and configured for future use to reduce your integrations boot time.

[jlaur]

We are on the same page. First time auto-discovery using both mdns addresses is user friendly as (from our experience) most users are not aware of Gen1/2/3 differences or even what generation they have - and then cache what you found and configured for future use to reduce your integrations boot time.

This is exactly what I proposed, so now it seems all three of us are on the same page. :-) Perhaps the confusion was from the fact that we have both managed and unmanaged things in openHAB. Managed things can be created from discovery and would then according to my proposal be "born" with the Gen version cached. For unmanaged things (configured in files) manual configuration is needed in order for them to have the same information at hand. Thereby they will also have reduced boot time/initialization phase, which was my point. And on top of that, there will be no differences in the logic after creation, so the implementation will be simpler.

@andrewfg - we can consider the lack of configured "Gen" version configuration as Gen 1/2. That will provide the backwards compatibility and eliminate any need for detecting Gen version for already configured things. We could simply use 0 for Gen 1/2 and 1 for Gen 3, unless there would ever be any need to distinguish between Gen 1 and 2, @vves?

[andrewfg]

consider the lack of configured "Gen" version configuration as Gen 1/2. That will provide the backwards compatibility and eliminate any need for detecting Gen version for already configured things.

Yes. Ok.

We could simply use 0 for Gen 1/2 and 1 for Gen 3

Not quite. I suggest the following

        HDPowerViewHubConfiguration config = getConfigAs(HDPowerViewHubConfiguration.class);
        int hubGeneration = config.generation;
        switch (hubGeneration) {
            case 0: 
            case 1: 
            case 2:
                webTargets = new HDPowerViewWebTargetsV1(httpClient, host);
                break;
            case 3:
                webTargets = new HDPowerViewWebTargetsV3(httpClient, host);
                break;
        }
        if (webTargets != null) {
            this.webTargets = webTargets;
            return webTargets;
        }
        throw new InstantiationException("Unable to instantiate the web targets");
    }

[andrewfg]

unless there would ever be any need to distinguish between Gen 1 and 2

@jlaur as far as I recall we recently had an issue (see below) where the functionality needed to be different for Generation 1 than Generation 2. And I think you also had something about scene groups not being supported in Generation 1? So we could consider re-factoring those PR's to handle those Gen 1 / 2 differences based on config.generation, rather than how we actually did it in those PR's. .. (??) .. but on the other hand, we might just 'let sleeping dogs lie'..

https://community.openhab.org/t/hd-powerview-binding-issue-openhab-3-3-0/136921

[jlaur]

@andrewfg - the only two differences known so far:

• Gen 1 doesn't provide capabilities, only type.
• Gen 2 redirects /api/scenecollections to /api/sceneCollections while Gen 1 only supports /api/scenecollections.

For both of these differences I'm not sure if it's worth distinguishing API version, although we could get a small optimization for Gen 2 out of directly using /api/sceneCollections to avoid redirection.

But that leads back to what should be persisted in bridge config. We have currently two different API's and three different hardware units (Gen 1, Gen 2 and Gen 3). Especially for the Gen 3 unit, the API is likely to evolve over time, so software version might be significant to look at also. However, since software version can change after creation of bridge, we can probably only reliably detect major API version at time of discovery. So we might have to dynamically handle minor differences in API versions after that, for example based on firmware version detected.

Now, the question is if we should link configuration directly to hardware version, or if we simply need to have incremental API major version numbering in order to abstract the hardware completely? Gen 4 might be based on the same API as Gen 3, so I'm not sure we'll have benefit of storing "3", and we have no guarantee we'll be able to detect actual hardware version rather than API version. Like "1" and "2" would pretty much refer to the same API version, namely current API.

[andrewfg]

^
It sounds like you are discussing naming issues again?

Probably the users are more aware of what hardware version they have. So the primary user decision point is if they have a gen1, gen2, or gen3 hardware. For that reason I think the config param should reflect the hardware too.

The secondary (and for the user hidden) point is the api version that runs on that hardware.

As far as I am concerned the gen1 and gen2 hardware both support the V1 api. Albeit with some minor nuances, as you hilighted above.

And the gen3 hub supports the V3 api.

So to summarise we have three hub hardware versions, (representated by an integer config param), that map to two api versions.

• hardware v 1, 2 (or unknown =0) uses V1 api
• hardware v 3 uses V3 api

[andrewfg]

@jlaur some background on my two PRs..

• the first PR is simply refactoring the existing functionality to split classes into a non version specific declaration class, and a (V1) version specific implementation class; there are 24 files involved.
• the second PR adds implementaion classes for V2; it includes the 24 changes above, plus a further 27 for V3, Junit tests, and mDNS discovery; in total 51 files have changed.

So if you are reviewing my code, you may consider whether to bother with looking at PR 1 or whether to jump straight into PR 2. Its up to you..

FYI I myself am already running the full PR 2 code on my V1 operative system.

[andrewfg]

@vves just wondering what is the status concerning your revised document? In particular the api call for the binding to register itself with the gateway.