Values in Home Assistant becomes unavalible for a split second #13
Comments
|
Try monitoring the stick's verbose logs, e.g. |
|
Ok, will do! I have the command running and logging now, will report back if I find anything. |
|
Ok, that was a quick and lucky troubleshooting! Unfortunately it seems my "99% sure it's not WiFi" above turned out to be wrong I guess.... I based this on Home Assistant not detecting the stick as offline via ping but that does not ping often enough to detect this short connection timeout.
Those 5 lost pings is enough for Home Assistant to declare the device (or values) Unavailable.... I see nothing odd in the logs from the stick during the same time frame, normal reporting with normal values included and then a warning that the connection was dropped, obviously since the stick looses connection for a short period of time. So I guess WiFi needs to be more stable but that was not within the scope of this project as you mentioned here #12 already, any idea why this might happens roughly every 15 minutes? Only thing that comes to my mind is this comment libretiny-eu/libretiny#113 (comment) where it's mentioned that there is a Does your stick loose a few pings every ~15 minutes @hn ? |
|
Here are roughly 6 hours of Pingplotter towards the Solis stick, I would say it looses connection for a short period of time every 15 minutes, almost like clockwork. (click for larger image) Red lines = lost pings
I will migrate back to my solution with the Pi later tonight since I want more stability in monitoring my inverter. I will still keep an close eye on https://github.com/hn/ginlong-solis and https://github.com/kuba2k2/libretiny (since I really want to use this project with my Solis stick, otherwise I have no other use for it really) in case there are any updates to the WiFi "stability" and these connection drops every 15 minutes, would be interesting to know if anyone else sees them or if it's some hardware issue with my stick. |
|
Today I am dumping more logs but this time over serial so I can see what happens when the stick goes missing, so far I only have output what happens when stick looses WiFi and then stays disconnect for X amount of time, so this is not one of those short 5 second disconnects, will report back once I have that captured as well. This happened 15 minutes after last disconnect so the "timer" from previous post is the same, the stick is working normally reporting "WiFi Signal", it gets no other data obviously since it's not connected to the inverter, this goes on for 15 minutes and then all of a sudden it just reboots I guess and then instantly tries to reconnect to WiFi but fails, over and over and over again, I just included 2 attempts but the last part just repeats until it manages to connect, anyway here is the output from serial [D][sensor:093]: 'WiFi Signal': Sending state -50.00000 dBm with 0 decimals of accuracy Initializing WIFI ... |
|
And here is the serial output when the stick just drops 5 pings and returns, included some more logs so will use Pastebin this time As I see it the same thing happens as when the stick is gone for longer periods of time, the only difference is that it actually manages to connect to WiFi on first attempt. So there are 2 problems as I see it
Really curious if this is only my stick or if someone else has this issue, quick way to find out is to just ping the stick and see if it drops out every 15 minutes, hopefully someone else can try this to see if they have the same issue. |
|
My S3 stick has stable WiFi (once initially connected) from morning to evening (power-on ... off). Random thoughts:
|
I will look into that
I very much doubt it, I have stable WiFi on all other devices in my house that use same SSID and same APs, also this 15 minute reboot timer occurs even in the "xmodem transfer (UART boot mode)" on my stick and in that made there is no WiFi, also the timer is just to perfect, if this was something external I think I would see some variation, but it's spot on, I can count 900 seconds and it will reboot precisely after 15 minutes from first ping.
Might be, I'm far to casual to be able to know or determine that. I will look into our first point regarding WiFi reboot_timeout and API reboot_timeout, hopefully I will find something out from that. |
|
Sorry for what probably is a super basic question....
Does not work for me [D][ota:147]: Starting OTA Update from 192.168.0.197... I have a OTA password in the secrets file as instructed when replacing the main application, I can't seem to figure out with Google how to use that password when running the update via OTA 😞 I re-complied the firmware on the same computer and all, just edited some timeout's in the |
|
For the OTA password problem, please see #7 (comment) and libretiny-eu/libretiny#142 (I never had these problems) Maybe you can help @kuba2k2 to fix OTA password-, ltchiptool flashing-, and WiFi problems by contributing to the corresponding LibreTiny issues. |
|
Thanks, will check those links out. Happy to help in any way I can! |
|
(disabled OTA for now for easier re-flashing) I have now tried Neither makes any difference, stick reboots after 15 minutes, just like before. |
|
Maybe you want to try to revert to an older version of LibreTiny-ESPhome, e.g. this is the exact version I'm currently using: |
|
Just as a sanity check I restored the default Solis firmware to the stick to check how i behaves. As long as it's missing WiFi it reboots every 15 minutes just like running it with LibreTiny firmware. Tested this by powering down APs so it could not connect to any SSID. Once it has WiFi it becomes stable, not a single reboot over close to 3 hours hours now, verified with ping, I can also see it's online timer on the Solis cloud
So.... I can only assume there is some underlying WiFi issue in the firmware compiled with LibreTiny for my stick, if there are any hardware revisions of these sticks that differentiates them I have no clue, I have a date printed on the PCB of the stick that says "2021-05-15" just next to the WiFi antenna on the same side as the LED's are. I will also try and build new FW with the version you suggested above! |
|
Just wanna make sure I did it correctly. I created a new directory called "Lower_Version" Got this message I then moved in the solis-esphome-emw3080.yaml / solis-modbus-inv.yaml / secrets.yaml and then ran the compile in the new "Lower_Version" folder, figured all the other steps are already completed
RAM: [== ] 22.5% (used 59048 bytes from 262144 bytes) And then like before flashed the stick with python2, booting the stick the version changed in Home Assistant to
Booting the stick shows the following lines regarding version
So I assume I did everything correctly? |
|
Assuming my post above declares I did everything correct the results are unfortunately the same, no difference att all, reboots every 15 minutes just like before. Only stable state this stick has is with original Solis firmware and WiFi connected, otherwise it reboots every 15 minutes on Solis firmware also, maybe there is some clue in that finding, I guess a ticket over at LibreTiny is the next step? |
|
Random thoughts II:
|
Sure thing, can try disabling mDNS and putting a static IP in the configuration of the stick and OTA update it. Regarding your other thoughts, HA: not sure how since the stick reboots after 15 minutes with or without WiFi (except when using Solis firmware) = HA can't interfere when without WiFi and it still reboots, I can shut down HA at a more convenient point to test this also. WiFi: Reboots with or without WiFi so not sure how that could affect it, also, all other devices in my network be it WiFi or cable are stable and without issues. Network: Nothing is ever certain but I have no other issues in my network with other devices. I work as a network technician so I am quite certain my network is ok, I really hope I don't end up having to explain myself if this turns out to be network related 😄 but I honestly don't see how it would be the network. I don't have other ESPhome devices yet but I have bought some Wemos D1 mini and a Pro actually for future projects, I will try and take the time today / tonight to set one up with some Dallas DS18B20 sensors or something. |
Did not make any difference, up to 4 reboots now unfortunately. I hope to try out a Wemos D1 mini later tonight if I get the time, got some IRL stuff I need to handle now. |
|
I have not yet tried the other Wemos D1, did not have time yesterday, this morning I just did a quick experiment with
Showed nothing new, it just reboots after 15 minutes without any odd message in the log
Slowed down the stick sooooo much that after 15 minutes it had not even arrived at the |
|
So, running
Delayed the reboot to ~30 minutes instead of 15, I see nothing out of the ordinary when the reboot occurs, here is a pastebin of it Looks exactly like all the other reboots to me. What I do find interesting is that it was delayed significantly when the stick was occupied with I now also have the Wemos D1 programmed and pinging to see if that stays online, same SSID and same AP as the Solis stick, will report back later. |
|
Wemos D1 Mini Stable and working for 2+ hours Wemos D1 Mini Stable and working for 2+ hours So we now also know that a Wemos D1 Mini with ESPhome is stable and working in my network, both with really simple configuration and with this project (I did not wire it up with a RS485-to-serial adapter but I don't see that making any difference really for this test since it would not be connected to the inverter anyway), it's only the Solis stick that is rebooting every 15 minutes, I'm personally out of ideas to test right now... |
|
Puhhh, that's all I know about it too. Probably now is the time to ask at LibreTiny or ESPhome. |
|
Yeah same, I need to take the time to write a good post and summarize all tests that have been done, it's somewhat spread out in a few tickets here already. I saw LibreTiny just released a new version so might try that also, I tried to find some change log to see if there are any changes that could help but could not find one. I guess we can close this ticket, otherwise it will be auto-closed sooner or later, I will update here if / when a solution is found, many thanks for all the help so far @hn ! EDIT Found a "change log" 😄 https://github.com/kuba2k2/libretiny/commits/master |
|
Just FYI, the new version doesn't have any important changes, it mostly updates some OTA code (not related to encryption issues) and reworks parts of the GPIO functions. |
|
Ah ok @kuba2k2 |
|
I assume that you've disabled all timeouts (api, wifi, etc) in esphome config.
Why is that? Maybe there's a hardware device inside (another chip, kind of a "watchdog") that reboots the device? |
|
Thanks to @kuba2k2's helpful comment, I took a look at a second (newer) S3 stick (which is supposedly more similar/equal to @Belaial is using): Breaking news, it has a slightly different board layout :)
Especially the 6-pin IC in the middle of the picture is
They covered all of the PCB with some protective coating which makes it hard to identify this IC (it might have "RCENG" printed on it?) If it really is a hardware watchdog, the question is how to reset the watchdog timer. PIN 12 (PA_0) has not been used in the old PCB layout and might be connected to this IC. Maybe it is sufficient to toggle PA_0 every X seconds to reset the WDT? @Belaial You could configure an ESPhome gpio-switch for this pin and toggle it periodically: I'm busy during the next days, don't expect any news soon. |
|
Hi, sorry for late reply, ended up getting a few beers after work and time just went by 😄 Anyway, I have this very same IC as pictured above by @hn (also great thinking @kuba2k2)
I will now test the suggestions made by @hn regarding PA00, will update you once I know more! |
|
I can now HAPPILY report that the stick is stable and no reboots for 3+ hours, not even a single missed ping! 😄 That solved he reboot issue, which I guess turned out to be a hardware watchdog timer on newer versions of the stick, again great thinking @kuba2k2 and many many thanks for all the troubleshooting along the way @hn (I was thinking along these lines here #13 (comment) but never got around to compare my PCB to the image @hn have uploaded here https://github.com/hn/ginlong-solis/blob/master/solis-wifi-stick-s3-pcb-front.jpg) I think the |
Manually added b25f978 1+ hour of stable connection, 0 missed pings so b25f978 looks good to me 👍 |
|
Happy to help! Like I said here #9 (comment) "Either we get it to accept custom firmware or the stick dies, whatever comes first 👍 " So happy we finally found all the issues and resolved them! 😄 |
|
I feel they were like "damn it, these realtek chips have such a crappy wifi, let's add a hardware watchdog to put them in order". |
Hi,
I have only spent about 1,5 days running this project on my Solis stick now but one of my findings so far is that about every 15 minutes or so the readings in Home Assistant for a split second becomes Unavailable and then goes back just as fast, I have tried to get a recording of this to show for demonstration purposes but no luck yet with that.
This seems to happen on most if not all the values I read out but the value I know it happens for and that causes issues for me is Active Power, the reason I know this is that I spent hours trying to figure out why one of my automation's stopped working when I changed from my Pi running https://github.com/incub77/solis2mqtt to reading stuff out with the Solis stick, the automation is simple, if Active Power is above 750W for 20 minutes it should turn on a power outlet, I noticed this had not happened for many hours of Active Power being greater than 750W, when I was looking at this I noticed with my eyes how Active Power all of a sudden was Unavailable and then back to whatever the actual power was, since this seems to happen around every ~15 minutes or so my automation of 20 minutes won't trigger since the value has not been above 750 for 20 minutes with these small Unavailable breaks in between, I could just lower the timer on my automation and be done with it but I figured it's best to report it if it might be something that could be solved.
I am 99% sure this is not related to my reportings regarding WiFi here #12 since I see the stick online via ping at the same time as values becomes Unavailable.
The only "proof" I can show of this issue now is based on the "Logbook" in Home Assistant, however for some reason the Active Power does not get logged there so I will show history of the value for Country Code instead
Also here it can be seen as small gaps in the timeline
Now this is for Country Code as and example, even more values does this at the same time
And then they return within the same second or 1-2 seconds after.
Any ideas what could be causing this?
I can just lower the timers on my automation but hopefully we can figure out the issue and resolve it.
The text was updated successfully, but these errors were encountered: