Exception while attempting to get latest data

[s28400]

Noticed my integration stopped working on 3/16. HA logs show the following:

Logger: custom_components.uhoo
Source: custom_components/uhoo/__init__.py:85 
Integration: uHoo ([documentation](https://github.com/csacca/uhoo-homeassistant), [issues](https://github.com/csacca/uhoo-homeassistant/issues)) 
First occurred: 9:42:19 AM (382 occurrences) 
Last logged: 4:11:48 PM

Error: an exception occurred while attempting to get latest data: Error requesting data from https://auth.uhooinc.com/renewusertoken: 410, message='Gone', url=URL('https://auth.uhooinc.com/renewusertoken')

I tried deleting integration and re-authenticating. Looks like it got one data point but none after.

Running HASS: 2022.3.5 and latest integration version from HACS.

[startledhighlandhippo]

Seeing the same thing with mine - started on the same day (3/16). Any time the integration is restarted, it seems to grab one data point and then all sensors read "unavailable".

Confirmed that data is still being received in the phone app, so this seems to be something with how HomeAssistant is interfacing with the sensor.

[s28400]

I am thinking uhooair changed their link. I tried that URL manually and it is a blank page.

[s28400]

Did some debugging this weekend, the issue is within pyuhooair package. I wrote a comparable python package that can be used for this home assistant integration here: https://pypi.org/project/pyuhooairq/0.0.1 I am happy to help integrate to get this integration up and running again. If not, I can create a comparable home assistant integration to maintain. Thanks!

[csacca]

@s28400 Sorry for not responding earlier. This is a real issue, mostly due to using a reverse engineered API. I just wrote uHoo again and I'm hoping they'll give me access to their API now that they're supporting more third party integrations that just IFTTT. This integration uses the pyuhoo package, not pyuhooair. I'm a little busy right now with life, but if you submit a pull request to pyuhoo air that fixes the problem I'll take a look and if it works I'll push it and update both pyuhoo and this package. Thank you!

[howlermonkeys]

Did some debugging this weekend, the issue is within pyuhooair package. I wrote a comparable python package that can be used for this home assistant integration here: https://pypi.org/project/pyuhooairq/0.0.1 I am happy to help integrate to get this integration up and running again. If not, I can create a comparable home assistant integration to maintain. Thanks!

Any chance you'd be willing to teach others how to use your package to fix our uHoo intergrations?

[yingyann]

Hello, I’m interrested to know how to solve this issue also please:-) thank you

[startledhighlandhippo]

I have been trying to look into this. Unfortunately, I haven't found a fix yet. I did want to share what I've done though, in case it helped anyone out. I am far from an expert, so definitely take all this with a grain of salt. Sorry for the novel!

First, I have tried both @s28400's package and the PR created by @adig. Neither has been able to run successfully for me. I still see that after a short amount of time, the script fails to refresh the user token. The error is:
pyuhoo.errors.RequestError: Error requesting data from https://auth.uhooinc.com/renewusertoken: 410, message='Gone', url=URL('https://auth.uhooinc.com/renewusertoken')

I did learn how to set up an Android emulator and intercept HTTP traffic from the app. I suspect @adig did too as some updates from their PR seem related to what I was seeing (e.g. using the allconsumerdata endpoint). However, I saw something else not covered by that PR:

It seems the POST to refresh the user token now goes to a URL like:
https://auth.uhooinc.com/renewusertoken?millis=#############
Where ############# is a Unix epoch timestamp expressed in milliseconds.
However, trying this with the current UTC time has not worked. I also noticed there was an offset between the origination time of the request and the value used for millis (varying by the session, sometimes hours, sometimes just seconds). Trying to replicate that hasn't worked for me either.

Posting to plain-old https://auth.uhooinc.com/renewusertoken does work, but only for about a minute after the last renewal, after that it somehow expires/disappears.

A really brute-force fix here might just be to refresh the token every 30s or something. The uHoo app refreshes its token every 10 minutes, though, so it must be possible to go longer.

My uneducated suspicion is that the way the millis parameter is determined needs to be reverse engineered and then pyuhoo updated to include it.

I'll keep plugging at it in my free time, but maybe this write-up will give someone an idea.

[s28400]

@startledhighlandhippo startledhighlandhippo do you have the latest pypi package installed? Version 1.1.0 of pyhuooairq had some bug fixes around this issue and support for python 3.10. I simply re-authenticated by logging in once the token was determined to be expired. I have been trying to make an integration in my free time with little progress but hopefully this is helpful to get this one fixed.

[startledhighlandhippo]

@s28400 - yep, good call, turns out I was still on v1.0.0. Updating to v1.1.0 did get data to stream continuously.

In the interest of the shortest path to getting something working, I did go ahead and open up a PR that implements the same fix, but on the pyuhoo package used by @csacca's existing integration to hopefully get it up and running again. Ran it on my computer for a few hours and it seemed to work.

@csacca - if you could take at look at that PR it would be much appreciated. Should be quick - only touches 5 lines of code.

[howlermonkeys]

Any update on this, Does it still work via the updated pyuhoo library?
I'd much prefer to access Uhoo through a free method vs their 120 dollar a year "premium" serivce.
or anything I can do to help?

[frankvandenhurk]

+1

[yingyann]

+1

[dvhub]

+1... also very interested in the fix.

[dvhub]

Can someone fork with the fix in it?

[andrewleech]

@startledhighlandhippo would you happen to have the logs of the http traffic still? which version of the app did you capture these from? I'm inspecting the code in v5.1 and can't find reference to millis in the refresh url's, I might have to try to find a copy of v5.2 to look into.

However it looks like there could be other token information in the headers, a REFRESH: Token header and RefreshToken: header - perhaps the millis number it's referring to one of these from the previous refresh?

[davidcsally]

@startledhighlandhippo and @s28400 , if I am reading this right, you both got this to work by swapping pyuhoo to pyuhooairq ? Do you have any instructions on how to swap them? Are you using pyuhooairq inside of uhoo-homeassistant ?

[davidcsally]

I am thinking uhooair changed their link. I tried that URL manually and it is a blank page.

@s28400 I did some snooping in uHoo's premium web dashboard. I don't think they changed their link, because it is still being hit. I did notice that everytime it is called, it is called twice. First as an OPTIONS request, then as a POST request (with the Token and userDeviceId keys)

[andrewleech]

@davidcsally I don't know that pyuhooair is needed, this patch in pyuhoo adds a workaround to just login every time rather than refreshing the auth: https://github.com/csacca/pyuhoo/pull/57/files

This is not an ideal solution by any means, but unless we can figure out the added details needed to make refresh work it's possibly the best option.

[andrewleech]

I did notice that everytime it is called, it is called twice. First as an OPTIONS request, then as a POST request (with the Token and userDeviceId keys)

Great idea checking the web interface, I might be able to use that to figure out the refresh! Much easier than inspecting a phone app.

Update: yep I've got some logged traffic from this that follows the login / refresh workflow, hopefully that'll have enough detail to help me fix it properly!

[andrewleech]

Ok so the logged traffic was very interesting... in that it really did look identical to the flow used here / in pyuhoo! I didn't see any sign of the url arg millis mentioned earlier or anything else particularly different.

I also extended the test script in pyuhoo to run a number of user refresh / get data with a 5 second delay between each one.
Strangely, usually the first free refreshes work perfectly! Sometimes they all work. Other times though it works a few times, then fails in the same way seen here.

It took me quite a few looks back and first, and lots of extra logging being added to the pyuhoo test script, before I really realised the glaring difference; with online logged data the client_id looks like a token, a long (48char) hex string, whereas in pyuhoo it's a fixed string of mostly 0.

I believe this is the issue!

It makes sense that if the login / refresh tokens are generated against the client id, as soon as a second person logs in with the same client id the first users token will be invalidated.

I've raised the PR csacca/pyuhoo#75 which generates a client id in a relatively unique fashion, though I don't know how it compares to the official app. It's a long enough string the chance of user class should be essentially zero now.

[davidcsally]

@andrewleech That's really interesting, good find. I'm a front end guy, but I forked this and pyuhoo to make my own package with your changes, unfortunately I'm still seeing the same error. Were you able to get it to work consistently? I can try digging into the web api a little more as well. It's crazy, this code and the web api look like they are doing exactly the same thing.

[andrewleech]

@davidcsally I've now got my development fork of this repo set up to depend on my pyuhoo PR branch so it pulls it in directly from GitHub automatically.

I did a clean install of that a few hours ago and it seems to be still working properly; the sensors have full history during this time and I can't see any errors in the HA logs.

If you want to test it, you can remove this repo from hacs and add mine instead then re-install uhoo integration. https://github.com/andrewleech/uhoo-homeassistant
No guarantees I'll keep this stable long term, it's still where I do development, but does let the PR get easily tested for now!

[dvhub]

@davidcsally I've now got my development fork of this repo set up to depend on my pyuhoo PR branch so it pulls it in directly from GitHub automatically.

I did a clean install of that a few hours ago and it seems to be still working properly; the sensors have full history during this time and I can't see any errors in the HA logs.

If you want to test it, you can remove this repo from hacs and add mine instead then re-install uhoo integration. https://github.com/andrewleech/uhoo-homeassistant No guarantees I'll keep this stable long term, it's still where I do development, but does let the PR get easily tested for now!

@andrewleech thank you for your efforts on this. I've installed your integration and am getting this following error:

This error originated from a custom integration.

Logger: custom_components.uhoo
Source: custom_components/uhoo/init.py:55
Integration: uHoo (documentation, issues)
First occurred: 2:02:01 PM (10 occurrences)
Last logged: 2:10:15 PM

Error: an exception occurred while attempting to get latest data: 'temp'

Note that I've never had the uHoo integration working. I attempted to get it setup in HA after it was no longer working so perhaps there is another issue.

[andrewleech]

@dvhub I think there should be more details of the error / exception further down on either that log entry, or perhaps a second log entry? That snippet doesn't tell me enough unfortunately.

[dvhub]

@dvhub I think there should be more details of the error / exception further down on either that log entry, or perhaps a second log entry? That snippet doesn't tell me enough unfortunately.

I'll dig in deeper. I believe I need to enable debug logs in HACS.

[dvhub]

@dvhub I think there should be more details of the error / exception further down on either that log entry, or perhaps a second log entry? That snippet doesn't tell me enough unfortunately.

This is all that I see in the home-assistant.log:

2022-12-01 15:35:27.531 ERROR (MainThread) [custom_components.uhoo] Error: an exception occurred while attempting to get latest data: 'temp'
2022-12-01 15:35:27.534 ERROR (MainThread) [custom_components.uhoo] Error fetching uhoo data:

The two errors from Settings > System > Logs are:

---

This error originated from a custom integration.

Logger: custom_components.uhoo
Source: custom_components/uhoo/init.py:55
Integration: uHoo (documentation, issues)
First occurred: 3:35:27 PM (314 occurrences)
Last logged: 10:41:22 PM

Error: an exception occurred while attempting to get latest data: 'temp'

This error originated from a custom integration.

Logger: custom_components.uhoo
Source: helpers/update_coordinator.py:182
Integration: uHoo (documentation, issues)
First occurred: 3:35:27 PM (314 occurrences)
Last logged: 10:41:22 PM

Error fetching uhoo data:

[dvhub]

@dvhub I think there should be more details of the error / exception further down on either that log entry, or perhaps a second log entry? That snippet doesn't tell me enough unfortunately.

This is all that I see in the home-assistant.log:

2022-12-01 15:35:27.531 ERROR (MainThread) [custom_components.uhoo] Error: an exception occurred while attempting to get latest data: 'temp' 2022-12-01 15:35:27.534 ERROR (MainThread) [custom_components.uhoo] Error fetching uhoo data:

The two errors from Settings > System > Logs are:

This error originated from a custom integration.

Logger: custom_components.uhoo Source: custom_components/uhoo/init.py:55 Integration: uHoo (documentation, issues) First occurred: 3:35:27 PM (314 occurrences) Last logged: 10:41:22 PM

Error: an exception occurred while attempting to get latest data: 'temp'

This error originated from a custom integration.

Logger: custom_components.uhoo Source: helpers/update_coordinator.py:182 Integration: uHoo (documentation, issues) First occurred: 3:35:27 PM (314 occurrences) Last logged: 10:41:22 PM

Error fetching uhoo data:

I was unable to determine the root cause, but I did come up with a work around to my issue. I transferred the device to a new uHoo account and it worked without any issues. There must be something wrong with my original account. Thanks @andrewleech .

[scyto]

If you want to test it, you can remove this repo from hacs and add mine instead then re-install uhoo integration. https://github.com/andrewleech/uhoo-homeassistant No guarantees I'll keep this stable long term, it's still where I do development, but does let the PR get easily tested for now!

thank you, i just got my first uHoo and this worked perfectly (at least for initial setup i just did :-) )