Growatt_server integration not able to pull data from servers

[muppet3000]

The problem

The issue described (and fixed) here: #80950 has re-appeared as a result of changes/blocking on the Growatt Server end.
A new fix is required in the upstream library to fix this.
In the meantime the workaround outlined in #80950 can be used while this is fixed.

What version of Home Assistant Core has the issue?

2022.11.0

What was the last working version of Home Assistant Core?

N/A - Will affect all versions

What type of installation are you running?

Home Assistant Container

Integration causing the issue

Growatt_Server

Link to integration documentation on our website

https://www.home-assistant.io/integrations/growatt_server/

Diagnostics information

See #80950

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[home-assistant]

Hey there @indykoning, @JasperPlant, mind taking a look at this issue as it has been labeled with an integration (growatt_server) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of growatt_server can trigger bot actions by commenting:

• @home-assistant close Closes the issue.
• @home-assistant rename Awesome new title Change the title of the issue.
• @home-assistant reopen Reopen the issue.
• @home-assistant unassign growatt_server Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

---

growatt_server documentation
growatt_server source
_{^{(message by IssueLinks)}}

[muppet3000]

If other people are experiencing this issue can they please let me know on here?

[SACU-Student]

Mine is no longer updating...last update 2 hours ago.

Just for info, there was a very recent update to the shinephone android app...now version 7.1.0. (maybe they tweaked the user agent)

[muppet3000]

Mine is no longer updating...last update 2 hours ago.

Just for info, there was a very recent update to the shinephone android app...now version 7.1.0. (maybe they tweaked the user agent)

Thanks - I'll take a look at the network traffic from the new version of the app to see if there's anything special happening there.
In the meantime I'll submit a short-term fix/patch for the next hotfix release of HA that will get it working again (until they block it again).
Follow the hotfix/hack info on the other ticket to change the User-Agent field to something random again to make it work in the meantime.

[mjooris]

yup same issue here. i am still running with the workaround from last time but it stopped 2 hours ago again :(

[muppet3000]

yup same issue here. i am still running with the workaround from last time but it stopped 2 hours ago again :(

All you need to do to make it work again is to change the 'User-Agent' string which has something like Linux Dalvic.... in it to any other string and it will start working again, but there's no guarantee for how long until they decide to block it again.

[mjooris]

indeed, got it working with: Mozilla/5.0 (X11; CrOS x86_64 8172.45.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.64 Safari/537.36

thx!

[bigfloor]

We should reverse engineer the Shine phone app to see wat they are using there. Will try to SSL decrypt the traffic from it and do a packet capture....

[muppet3000]

We should reverse engineer the Shine phone app to see wat they are using there. Will try to SSL decrypt the traffic from it and do a packet capture....

There's no need, I already know what it's using - it's using the information from your local device. I have a plan on how to fix/resolve it, but I need to actually implement it.

In the meantime people will just have to use the same hack from last time.

[muppet3000]

indeed, got it working with: Mozilla/5.0 (X11; CrOS x86_64 8172.45.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.64 Safari/537.36

thx!

Remember that the application is hitting the Android endpoint, so this will almost definitely get blocked at some point because it's pretending to be a Web Browser

[timbba80]

I'm stuck with the workaround mentioned in the initial post. I have HA running in Raspberry 4 (Home Assistant OS, no docker / virtual environment) and growatt integration is in use. When I go to terminal and type "pip show growattServer", I'm getting an response "package(s) not found". Also when searching for files named as "__init__py", none of them refers to growattServer.

Any ideas how to proceed?

[bigfloor]

I activated SSL Decryption on my FW, that works without breaking the Shine app (which is actually bad, as they therefore do not do any certificate pinning).
And weird that they are using the info from the local device, cause them the integration should work if we mimic an iPhone for example.
Really curious to see what the underlying issue is
PS: Really thinking about setting up an extra RPi with a RS-485 to TTL converter, hook that up to my Growatt inverter and have that send info back to HomeAssistant, no more dependencies

[bigfloor]

I'm stuck with the workaround mentioned in the initial post. I have HA running in Raspberry 4 (Home Assistant OS, no docker / virtual environment) and growatt integration is in use. When I go to terminal and type "pip show growattServer", I'm getting an response "package(s) not found". Also when searching for files named as "__init__py", none of them refers to growattServer.

Any ideas how to proceed?

Do this first:
docker exec -it homeassistant bash

Works for my HA running on ESXi, with the 'SSH & Web Terminal' add-on installed

[SACU-Student]

thanks bigfloor !

I'm stuck with the workaround mentioned in the initial post. I have HA running in Raspberry 4 (Home Assistant OS, no docker / virtual environment) and growatt integration is in use. When I go to terminal and type "pip show growattServer", I'm getting an response "package(s) not found". Also when searching for files named as "__init__py", none of them refers to growattServer.
Any ideas how to proceed?

Do this first: docker exec -it homeassistant bash

Works for my HA running on ESXi, with the 'SSH & Web Terminal' add-on installed

That worked for me - I added the user agent from my android phone:

Mozilla/5.0 (Linux; Android 13; Pixel 6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Mobile Safari/537.36

and all is working again!

[PMizz13]

Same problem here since about 6 hours ago. I'm struggling to figure out how to find and edit the file for the work around. Running HA OS on Pi4 so I'll just wait and cross my fingers for a fix again. >_<

[timbba80]

@bigfloor thank you, this helped me as well!

Generally; freaking annoying that Growatt is acting in this way. Not sure why they do this (is there any rationale on this). To me as an basic user, who wants to have centralized information repostitory (Home Assistant), these changes do cause additional effort and harm. And if there is a provider who sees centralized solutions (such as home assistant) as an negative solution and wants people to use their own ecosystem instead....those organizations are not living in 2022. If I've had known this when making the deal for solar panels + inverter, I would had turned down the offer.

[muppet3000]

Hi everyone, going to try and respond to everyone in one mega post, rather than quoting everyone's messages I'm just going to tag the people.

Firstly, I know what the issues is - it's the same as last time, they've just blocked the new User-Agent value. My intention is to submit a short-term fix/hack/workaround via a PR to the HA codebase this afternoon so that everyone can get their systems up and running again at least for the weekend.

Secondly, I have a longer term plan as to how I can fix this, it will just take time for me to implement i.e. I can't do it in the background while I'm at work today.

Thirdly, I have personal commitments all this weekend (it's my Daughter's birthday) so I will be lynched if I spend all my time writing code, therefore I will be slow to respond and there won't be a 'proper' fix until some time next week, so please be patient, especially since there will be a Dev build that will work around the problem.

Now onto all of the individual posts that require a response:
@bigfloor - Thank you for your input, the user agent is specified by the phone, it's what's used to identify it on the server end, it 100% makes sense to use the Phone's ID for this. If you're interested in the solution I'm planning on implementing, please contact me directly via email and I'll kick off a chain when I have a moment. (This offer extends to anyone who's interested in my proposed implementation).
RE: Alternative implementations for this - There's lots of ideas for this over on the forums here, feel free to take a look as there already lots of alternative options if that's the way you want to go: https://community.home-assistant.io/t/anyone-experience-with-connecting-a-growatt-solar-inverter/60430/257

@SACU-Student - Again, this is spoofing a mobile web-browser, not an android app, slightly different but will get it working at least in the short term

@PMizz13 - I'll update as soon as I've had my PR merged with a work-around

@timbba80 - It's unlikely that they're wanting to stop people using HomeAssistant altogether, it's more likely that they're seeing an increased load on their servers on account of everyone's HomeAssistant servers polling them for info every minute. That volume of requests soon adds up. Therefore, what (at least I think) they've done is implement some checking to drop anything not coming from their Android/iPhone app by inspecting the header field. The CORRECT solution (if I were them) would be to increase the processing power on their backend to keep up with the load.

NOTE - I will not close this ticket until the 'proper' solution is implemented, the hotfix/hack/workaround is just temporary.

[timbba80]

Same problem here since about 6 hours ago. I'm struggling to figure out how to find and edit the file for the work around. Running HA OS on Pi4 so I'll just wait and cross my fingers for a fix again. >_<

1. If not installed already, install SSH terminal (settings -> add-ons -> add on store -> search with "SSH" keyword. You will find two options, "terminal & ssh" and "ssh & web terminal". I'm using "ssh & web terminal".
2. Once installed, start the SSH terminal and open it
3. In the SSH command prompt, type docker exec -it homeassistant bash
4. Type pip show growattServer. Take note of the "location"
5. type cd location* *replace location with the actual location from step 4
6. type cd growattServer
7. type vi __init__.py
8. go to the row which states headers = .......... It should be row nr. 30 (you can see current row at the bottom of the screen in vi
9. press i button on your keyboard (this enables edit mode in vi)
10. replace the current content after 'User-Agent':) as an example with the content mentioned above. As an example:
    headers = {'User-Agent': 'Mozilla/5.0 (X11; CrOS x86_64 8172.45.0)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Mobile Safari/537.36'} NOTE! As mentioned by @muppet3000, this might be only a short term fix as it's spoofing a web-browser instead of android app. So might be that they block this as well at some point.
11. Once you've done, press esc button (disables edit mode in vi)
12. type :x (saves the changes and quits vi)
13. type exit
14. type exit
15. reboot home assistant

If you want to quit VI without saving the changes (as an example if you did make an mistake and want to start from beginning), after pressing esc at step 11, type :q! (will close vi without saving the changes).

[dannytsang]

Minor changes to @timbba80 steps (I hope you don't mind) and thank you for writing up the step by step.

Changed step 7. to include the underscores in the file name and a shortcut in vi for :wq is :x (yes I'm that lazy 😉 ).

Same problem here since about 6 hours ago. I'm struggling to figure out how to find and edit the file for the work around. Running HA OS on Pi4 so I'll just wait and cross my fingers for a fix again. >_<

1. If not installed already, install SSH terminal (settings -> add-ons -> add on store -> search with "SSH" keyword. You will find two options, "terminal & ssh" and "ssh & web terminal". I'm using "ssh & web terminal".

2. Once installed, start the SSH terminal and open it

3. In the SSH command prompt, type "docker exec -it homeassistant bash"

4. Type "pip show growattServer". Take note of the "location"

5. type "cd location*" *replace location with the actual location from row 4

6. type "cd growattServer"

7. type vi __init__.py

8. go to the row which states "headers = .........". It should be row nr. 30 (you can see current row at the bottom of the screen in vi

9. press "i" button on your keyboard (this enables edit mode in vi)

10. replace the current content after 'User-Agent':) as an example with the content mentioned above.  As an example:
    `headers` = {'User-Agent': 'Mozilla/5.0 (X11; CrOS x86_64 8172.45.0)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Mobile Safari/537.36'} NOTE! As mentioned by @muppet3000, this might be only a short term fix as it's spoofing a web-browser instead of android app. So might be that they block this as well at some point.

11. Once you've done, press esc button (disables edit mode in vi)

12. type :x (saves the changes and quits vi)

13. type exit

14. type exit

15. reboot home assistant

If you want to quit VI without saving the changes (as an example if you did make an mistake and want to start from beginning), after pressing esc at step 11, type :q! (will close vi without saving the changes).

[timbba80]

@dannytsang good catch! I'm quite a newbie when it comes to github, and didn't realize that adding two underscores makes them disappear.. I've modified the original content now with your inputs.

[dsl10007]

A suggestion would be to make a list of 50 different strings and just randomly select one, every time it polls for data

[muppet3000]

So it doesn't look like I'm going to be allowed to merge the fix that I was proposing for this, I have to merge it into the upstream library and then merge that in, which will take some time as I don't own the release process for the upstream library.

You have the workaround though, that's all I can do for now. As soon as there's an upstream release though I'll get the PR updated for here and in the meantime I'll keep thinking about the best way to implement a PROPER fix.

Enjoy your weekends!

[DutchKillerbee]

Thanks @muppet3000 for all your work!
Fix works for me.

[Coooogz]

For what it is worth, mine is working ok and I have not applied any work around

Same here to be fair. It did stop working for a bit but when I upgraded to 2022.11.4 it kicked into life again. So perhaps just needed a reload?

[Saentist]

Guess something changed on Growatt CDN, my also start to work.

[delchrys]

It's not because I want to use grott, because I don't know if it's compatible with HAOS.. But thought maybe the different approach would be easy to integrate

[muppet3000]

Thanks for the continuing updates everyone. I sent a detailed to email to Growatt yesterday asking for some proper engagement.

FWIW The test script I have that exclusively uses the up-stream library fails when I use the User-Agent field that it comes with by default, so no idea why it's working for some of you and not others. Let's not try and debug that here, that's something for me and Indy to discuss in the other repo.

I've also been working with Indy on a solution that will work in the upstream library that prevents us from having to keep releasing fixes into/via HA as well.
Clearly Growatt are doing some active development and keep trying to change things on their end, if they engage I'll let you know the results.

As I've said before, I'll continue to support this integration and the upstream library so long as it makes sense to do so. Separately I'll investigate solutions like Grott and work out if it's possible to pivot the integration in that direction. In the meantime though I'll update this ticket (and other Growatt ones as & when I'm working on them).

[Saentist]

@muppet3000
Try to set UA to "home assistance client" +"mail address" (from client setup).
Ugly solution Growatt to collect "smart" telemetry, maby they can rethink.

[Sven4597]

I don't Update to the 2022 11 04 an it still work

[timaseth]

Out of nowhere, the integration started working again after a random reload!
(I am on the latest HA release.)

PS: @muppet3000
While the integration was down, I installed a NodeRed Growatt node https://flows.nodered.org/node/node-red-contrib-growatt.
This kept functioning without missing a beat during the entire period of 'outage'. Perhaps their code-base can point at a more stable solution? Just saying.

[Sampledelic]

My growatt integration works again.
My fix was to upgrading to "Home Assistant 2022.11.4", rebooting my wifi dongle and restart the "Growatt server integration"

I'm a very happy user again, thanks everybody for all your effort!

[thoompje]

Here everything is broken again....

[Saentist]

@thoompje
do you restart system not integration?

[thoompje]

@thoompje

do you restart system not integration?

I restarted the hardware and then it was not working. After a reload of the intergration, after the restart it is working again.

[Sampledelic]

@thoompje do you restart system not integration?

I've restart the system as effect of the "Home Assitant" upgrade to the latest release and after restarted the integration before it worked.

[Aad3229]

Updated to the latest release (Home Assistant 2022.11.4 in docker). Restarted HA and reloaded Growatt integration but still al entities are "Unavailable"

[muppet3000]

Has anyone bothered to read the description or the comments in this thread?

I'm not sure why everyone thinks it's going to have magically fixed itself. I'm still waiting on the merge in the upstream library, at which point I can release a fix here.

In the meantime (as I've said in the description and multiple times in the comments), you need to follow the same work-around that people applied last time (but with a different User-Agent) until the upstream library is fixed.
Here's a specific link to how to change it for the Docker instance: #80950 (comment)
For those of you not running the docker specific instance scroll further through those comments and you'll see other people with different installations and how they fixed it.

As soon as the upstream library is released I'll make the change in this library and then it'll start working again.

[aherzer69]

Hello Muppet3000, thank you for the quick feedback! I really appreciate your effort! Thank you for that!!!! Now I fixed it with your attached link. I have seen it before (fighting to fix since weeks), but I was not able to realize that it was the solution :-) Have a great weekend & Best regards Axel Powered by Mailbutler <https://www.mailbutler.io/?utm_source=watermark&utm_medium=email&utm_campaign=watermark-variant-primary> - still your inbox, but smarter.

…

Am 26.11.2022 um 21:26 schrieb muppet3000 ***@***.***>: Has anyone bothered to read the description or the comments in this thread? I'm not sure why everyone thinks it's going to have magically fixed itself. I'm still waiting on the merge in the upstream library, at which point I can release a fix here. In the meantime (as I've said in the description and multiple times in the comments), you need to follow the same work-around that people applied last time (but with a different User-Agent) until the upstream library is fixed. Here's a specific link to how to change it for the Docker instance: #80950 (comment) <#80950 (comment)> For those of you not running the docker specific instance scroll further through those comments and you'll see other people with different installations and how they fixed it. As soon as the upstream library is release I'll release the change here as well. — Reply to this email directly, view it on GitHub <#81951 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJ2TDCWFETTJDGOU74KAZMLWKJW7LANCNFSM6AAAAAAR5LWJUU>. You are receiving this because you commented.

[malosaa]

For me after the last message i sended, it started to work again by itself and no problems yet.
Don't know why.
But i see some good posts above for a temp fix, thanks for that. just in case i need it.

muppet3000 thanks for updating us :)

[Aad3229]

Changed the Docker instance as in comment #80950, restarted HA and now it is working again.

[thoompje]

I restarted the hardware, homeassistant docker and everything is unavailable again. Also tried to reload the intergration but still no luck. Anyone else with this problem? Or does anyone has a solution for this? Getting me crazy...

[muppet3000]

I restarted the hardware, homeassistant docker and everything is unavailable again. Also tried to reload the intergration but still no luck. Anyone else with this problem? Or does anyone has a solution for this? Getting me crazy...

Have you read my comment from yesterday? #81951 (comment)

[muppet3000]

For those still following this bug, it narrowly missed out on the 2022.11.5 release, but will be available in the latest Dev build.