Add webhook support to onvif #91485

bdraco · 2023-04-16T03:01:01Z

Proposed change

Add webhook support to ONVIF

Upstream changelog: openvideolibs/python-onvif-zeep-async@v1.2.11...v1.3.0

Compliant cameras are required to support event notifications (webhooks) or PullPoint subscriptions. Some implement both, and we now support both. Webhook support means we can support events on another subset of devices. We will automatically switch over to webhooks as soon as the first webhook is received to ensure that the webhook is reachable by the camera.

Webhooks are generally preferred since they have lower latency (at least for most of the tested cameras) and are better for motion events than the long poll of the pull point, which must be restarted when events happen. Webhooks are more efficient since we have much less activity when nothing is happening with the camera, which is anticipated to be the state most of the time. Additionally its less overhead to process the webhook than it is to do the long polling.

Adjust subscriptions to 3 minutes and renew when they are at 50% expiry. Previously if Home Assistant was restarted and the unsubscribe was not processed, it with tie up the slot for 24h, and events would stop working. If you restarted Home Assistant a few times, events would be broken until the next day because all the slots were full. This is especially likely to happen for cameras that do not implement unsubscribe.

Relative times are used everywhere since it's clear that vendors implement time zone support poorly or never update for daylight savings time changes. This should fix a whole class of issues related to time zones since we never need to care about which timezone the camera supports. There are a significant amount that don't event support utc

I've done a full range of performance testing on this PR to make sure its efficient as well.

Type of change

Dependency upgrade
Bugfix (non-breaking change which fixes an issue)
New integration (thank you!)
New feature (which adds functionality to an existing integration)
Deprecation (breaking change to happen in the future)
Breaking change (fix/feature causing existing functionality to break)
Code quality improvements to existing code or addition of tests

Additional information

This PR fixes or closes issue: fixes ONVIF integration problem with Hikvision DS-2CD2432F-I camera - "Failed to set up" #91471 fixes XM530AI AZISHN 1080P 2.0MP Wifi IP Camera live stream not working #91678
This PR is related to issue:
Link to documentation pull request:

Checklist

The code change is tested and works locally.
Local tests pass. Your PR cannot be merged unless tests pass
There is no commented out code in this PR.
I have followed the development checklist
I have followed the perfect PR recommendations
The code has been formatted using Black (black --fast homeassistant tests)
Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

The manifest file has all fields filled out correctly.
Updated and included derived files by running: python3 -m script.hassfest.
New or updated dependencies have been added to requirements_all.txt.
Updated by running python3 -m script.gen_requirements_all.
For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
Untested files have been added to .coveragerc.

To help with the load of incoming pull requests:

I have reviewed two other open pull requests in this repository.

home-assistant · 2023-04-16T03:01:08Z

Hey there @hunterjm, mind taking a look at this pull request as it has been labeled with an integration (onvif) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of onvif can trigger bot actions by commenting:

@home-assistant close Closes the pull request.
@home-assistant rename Awesome new title Renames the pull request.
@home-assistant reopen Reopen the pull request.
@home-assistant unassign onvif Removes the current integration label and assignees on the pull request, add the integration domain after the command.

bdraco · 2023-04-16T08:20:19Z

Now we need a way to test if the webhook is working and the camera can reach hass

bdraco · 2023-04-16T09:10:29Z

Need to keep running pull points until the webhook gets its first callback so we know hass is reachable

bdraco · 2023-04-16T09:13:18Z

Track time interval is probably a bad choice. Should do call later in a finally instead in case renew takes a while or restart

bdraco · 2023-04-16T09:25:29Z

Should use config entry id for webhook id so it's easy and to cleanup later

homeassistant/components/onvif/__init__.py

homeassistant/components/onvif/event.py

bdraco · 2023-04-17T01:01:40Z

I've beat this up the best I can. I'm going to move some of the code to the lib to reduce this a bit

homeassistant/components/onvif/event.py

bdraco · 2023-04-17T03:09:41Z

Lots of power pulls with both webhook and pull point.

Everything recovers well

bdraco · 2023-04-17T03:36:48Z

I'm happy with this now.

Need to split out those two parts into the lib

bdraco · 2023-04-17T07:34:01Z

I ended up with a smart plug that reboots the camera over and over again to make sure it resumes

bdraco · 2023-04-17T08:53:30Z

Existing problem but if I power cycle the camera while motion is detected its never cleared because we don't clear the events when we loose the subscription. That would be a bit difficult to be sure we can.

For a future PR

bdraco · 2023-04-17T08:56:20Z

I'm happy that is recovers well when the power is cycled many times.

bdraco · 2023-04-17T16:33:32Z

We should have a failed state for when the initial pull point or notify subscription can't happen because the camera doesn't support it or has a quirk that prevents it from working

We can also cleanup the sensor and binary sensor a bit more

bdraco · 2023-04-17T18:05:26Z

I have one that doesn't support pull point but supports webhooks

    "events": {
      "webhook_manager_state": {
        "__type": "<enum 'WebHookManagerState'>",
        "repr": "<WebHookManagerState.STARTED: 1>"
      },
      "pullpoint_manager_state": {
        "__type": "<enum 'PullPointManagerState'>",
        "repr": "<PullPointManagerState.FAILED: 3>"
      }
    }

bdraco · 2023-04-17T18:06:01Z

and the reverse

    "events": {
      "webhook_manager_state": {
        "__type": "<enum 'WebHookManagerState'>",
        "repr": "<WebHookManagerState.FAILED: 2>"
      },
      "pullpoint_manager_state": {
        "__type": "<enum 'PullPointManagerState'>",
        "repr": "<PullPointManagerState.STARTED: 1>"
      }
    }

bdraco · 2023-04-17T19:53:12Z

I've done hundreds of restarts now and everything always comes up now.

Previously it had a 15-35% failure rate.

bdraco · 2023-04-17T20:07:18Z

Existing problem but if I power cycle the camera while motion is detected its never cleared because we don't clear the events when we loose the subscription. That would be a bit difficult to be sure we can.

For a future PR

I will do another PR after this to clear stale events when a renew fails (it will need a remote protocol retry as well because of http://datatracker.ietf.org/doc/html/rfc2616#section-8.1.4) but its outside the scope of this PR and I didn't want to make this any bigger

edit: #91567

bdraco · 2023-04-17T20:19:23Z

Going to squash this since I'm going to build on top of it and I don't think anyone has looked at it yet 🤞

This will fix all the motion sensors being in a stuck on state when the camera reboots needs #91485

bdraco · 2023-04-17T20:41:35Z

If/when mvantellingen/python-zeep#1369 gets merged we can simplify this all a bit

frenck

This looks good (and works well for me locally).

homeassistant/components/onvif/event.py

home-assistant · 2023-04-22T11:47:07Z

Please take a look at the requested changes, and use the Ready for review button when you are done, thanks 👍

Learn more about our pull request process.

frenck

Thanks, @bdraco 👍

../Frenck

home-assistant bot added cla-signed integration: onvif labels Apr 16, 2023

home-assistant bot assigned hunterjm Apr 16, 2023

home-assistant bot added the Quality Scale: No score label Apr 16, 2023

bdraco mentioned this pull request Apr 16, 2023

Bump onvif-zeep-async to 1.2.11 #91472

Merged

20 tasks

bdraco mentioned this pull request Apr 16, 2023

ONVIF integration problem with Hikvision DS-2CD2432F-I camera - "Failed to set up" #91471

Closed

bdraco commented Apr 16, 2023

View reviewed changes

homeassistant/components/onvif/__init__.py Outdated Show resolved Hide resolved

bdraco commented Apr 16, 2023

View reviewed changes

homeassistant/components/onvif/event.py Outdated Show resolved Hide resolved

bdraco commented Apr 16, 2023

View reviewed changes

homeassistant/components/onvif/event.py Outdated Show resolved Hide resolved

bdraco commented Apr 17, 2023

View reviewed changes

homeassistant/components/onvif/event.py Outdated Show resolved Hide resolved

bdraco commented Apr 17, 2023

View reviewed changes

homeassistant/components/onvif/event.py Outdated Show resolved Hide resolved

bdraco mentioned this pull request Apr 17, 2023

Add support for webhooks/notifications openvideolibs/python-onvif-zeep-async#25

Merged

bdraco marked this pull request as ready for review April 17, 2023 19:53

bdraco requested a review from hunterjm as a code owner April 17, 2023 19:53

Add webhook support to onvif

e360240

bdraco force-pushed the onvif_webhooks branch from 4d1aa85 to e360240 Compare April 17, 2023 20:19

bdraco added a commit that referenced this pull request Apr 17, 2023

Mark onvif events as stale when the subscription renewal fails

c27a1bd

This will fix all the motion sensors being in a stuck on state when the camera reboots needs #91485

bdraco mentioned this pull request Apr 17, 2023

Mark onvif events as stale when the subscription renewal fails #91567

Merged

20 tasks

bdraco added new-feature dependency labels Apr 17, 2023

frenck requested changes Apr 22, 2023

View reviewed changes

homeassistant/components/onvif/event.py Show resolved Hide resolved

home-assistant bot marked this pull request as draft April 22, 2023 11:47

bdraco marked this pull request as ready for review April 22, 2023 15:28

home-assistant bot requested a review from frenck April 22, 2023 15:28

frenck approved these changes Apr 22, 2023

View reviewed changes

frenck merged commit 3beb6e9 into dev Apr 22, 2023

frenck deleted the onvif_webhooks branch April 22, 2023 15:49

joanwa pushed a commit to joanwa/home-assistant that referenced this pull request Apr 23, 2023

Add webhook support to onvif (home-assistant#91485)

dd5b415

github-actions bot locked and limited conversation to collaborators Apr 23, 2023

frenck added the noteworthy Marks a PR as noteworthy and should be in the release notes (in case it normally would not appear) label Apr 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add webhook support to onvif #91485

Add webhook support to onvif #91485

bdraco commented Apr 16, 2023 •

edited

Loading

home-assistant bot commented Apr 16, 2023

bdraco commented Apr 16, 2023

bdraco commented Apr 16, 2023

bdraco commented Apr 16, 2023

bdraco commented Apr 16, 2023

bdraco commented Apr 17, 2023

bdraco commented Apr 17, 2023

bdraco commented Apr 17, 2023

bdraco commented Apr 17, 2023

bdraco commented Apr 17, 2023 •

edited

Loading

bdraco commented Apr 17, 2023

bdraco commented Apr 17, 2023

bdraco commented Apr 17, 2023

bdraco commented Apr 17, 2023

bdraco commented Apr 17, 2023

bdraco commented Apr 17, 2023 •

edited

Loading

bdraco commented Apr 17, 2023

bdraco commented Apr 17, 2023

frenck left a comment

home-assistant bot commented Apr 22, 2023

frenck left a comment

Add webhook support to onvif #91485

Add webhook support to onvif #91485

Conversation

bdraco commented Apr 16, 2023 • edited Loading

Proposed change

Type of change

Additional information

Checklist

home-assistant bot commented Apr 16, 2023

bdraco commented Apr 16, 2023

bdraco commented Apr 16, 2023

bdraco commented Apr 16, 2023

bdraco commented Apr 16, 2023

bdraco commented Apr 17, 2023

bdraco commented Apr 17, 2023

bdraco commented Apr 17, 2023

bdraco commented Apr 17, 2023

bdraco commented Apr 17, 2023 • edited Loading

bdraco commented Apr 17, 2023

bdraco commented Apr 17, 2023

bdraco commented Apr 17, 2023

bdraco commented Apr 17, 2023

bdraco commented Apr 17, 2023

bdraco commented Apr 17, 2023 • edited Loading

bdraco commented Apr 17, 2023

bdraco commented Apr 17, 2023

frenck left a comment

Choose a reason for hiding this comment

home-assistant bot commented Apr 22, 2023

frenck left a comment

Choose a reason for hiding this comment

bdraco commented Apr 16, 2023 •

edited

Loading

bdraco commented Apr 17, 2023 •

edited

Loading

bdraco commented Apr 17, 2023 •

edited

Loading