espixelstick stops functioning after about 5min

[stijn-dhondt]

--------- Instructions --------
Please provide answers directly below each section.
--------- Instructions ---------

ESPixelStick Firmware Version
4.0-ci5943922254

Hardware Version
quinled quad ETH
but have same issues with Uno and Octa

Binary release or compiled yourself?
release

Operating System (and version)
Windows

Web Browser (and version)
Version 114.0.5735.199 (Official build) (64-bits)

Access Point
Ubiquity unifi UAP-AC-Pro

Describe the bug
ESP 32 boots comes as available controller in FPP player but after about 5min controller goes offline and does not respond to anything anymore. Until i power cycly the device it stay's offline.
As from that point the same issue just repaets.

even if i send ping to the device so it stays awake it just stops functioning.
Also no response on the console just no more comms..

[patdelaney]

In order to try and reproduce the issue What version of xlights are you using?
This is the version I am running and it's pretty solid for over a week
4.0-ci5943922254

What version of xlights are you using?
What version of FPP are you using?
Is the Digquad connect via WiFi, ETH or both?

[stijn-dhondt]

@patdelaney
Xlights 2023.13 on mac
FPP 7.1 git 5f5cfcc
I already tested the firmware on following Quin boards:

• Dig Uno wifi only
• Dig Quad eth and wifi
• Dig Octa eth and wifi

all dropping all of the communication within less than 5min.
Don't think Xlights and FPP are the issue here it's really the ESP that completely stops responding.
Even the console does not output anything...

[rledonne]

I believe I am having a similar issue that may be the same thing. I added comments as a discussion before seeing this.
#659

It will stay connected until you open Tools | Test in xLights and start to send it data. Then it goes offline and doesn't reconnect until power cycled.

Edit to add versions:
I have tried multiple different recent CI versions of ESPixelstick firmware
Xlights 2023.13
FPP 7.0

[rledonne]

Adding an update.
I noticed that the network connection will drop randomly and not reconnect after a few minutes, regardless of whether I send data or not. Sending data to it just makes it happen faster.

If I add a fseq to the sd card and play it from the firmware, it will continue playing and looping after the network connection drops.

I am available to troubleshoot, I just don't know how to get any useful information since it a wifi device, and no logging, or maybe I am missing it.

[stijn-dhondt]

Adding an update. I noticed that the network connection will drop randomly and not reconnect after a few minutes, regardless of whether I send data or not. Sending data to it just makes it happen faster.

If I add a fseq to the sd card and play it from the firmware, it will continue playing and looping after the network connection drops.

I am available to troubleshoot, I just don't know how to get any useful information since it a wifi device, and no logging, or maybe I am missing it.

Turned off DDP and did not solve the issue on all 3 controllers so DDP is not the source of the issue here.
Seem that we really are looking at a similar issue with your update now. Great to not be alone here ;-)

[stijn-dhondt]

@patdelaney are you able to reproduce the issue?

[stijn-dhondt]

found version 4.0-ci3677767017 on an older system and this one seems to work normal... but that's one of 12 dec 2022.

[MartinMueller2003]

Please try this version: https://github.com/MartinMueller2003/ESPixelStick/blob/main/dist.zip

Please leave the ESP connected to the flashtool. Do whatever it is you normally do to cause it to disconnect. Save the logs and attach them to this issue.

[stijn-dhondt]

Log.txt

@MartinMueller2003, Logs attached... I started the esp arround 18:56:30 and it went offline from wifi today at 19:03:50 but no logs entries appeared at that time...

[MartinMueller2003]

Interesting that the wired interface came up but never got an IP address. That may be causing an internal routing conflict. Can you post your network config screen?

[stijn-dhondt]

@MartinMueller2003,

This is the last config i tested, before i had DHCP turned on with the same issue, and i also tested with the same ip on the ethernet interface which should be an issue anyway but in all 3 cases there was nothing connected to the ethernet interface...

[MartinMueller2003]

I never tested With nothing plugged into the Ethernet port and the IP set to static.

[rledonne]

I will add mine as well....
AEplus Log.txt

This was after a fresh flash using the above zip.
It flashed fine, connected to wifi, and watching it in the router, was connected for about 30 seconds before it dropped. During that 30 seconds, I clicked the output to lights in xlights, and started a sequence.

[stijn-dhondt]

@MartinMueller2003 just had 1 time the exact same issue in the logging as @rledonne! other times the console just stays silent...

[rledonne]

This one was flashed and nothing was done after it was flashed yesterday afternoon. It just sat connected to the computer. It looks like it dropped overnight and never tried to reconnect.

AEplus Log2.txt

[MartinMueller2003]

Now that is MUCH more interesting. These three lines are odd:
This line indicates the web page asked for the files needed to display and started working
16:19:31: [ 32710][E][vfs_api.cpp:104] open(): /littlefs/www/esps.css does not exist, no permits for creation

This line indicates there was a problem working with the AP. This one is VERY odd because the ESP should have stopped transferring data right away.
16:19:50: [WiFiDrv] WiFi Lost the connection to the AP

And this is when the ESP finally gave up interacting with the AP.
03:03:17: [WiFiDrv] WiFi Entering State: Connection Failed

Now to try and replicate what is going on. FYI: I ran mine overnight last night with no issues.

[rledonne]

Sorry, I just remembered this.
I think the line 16:19:31: [ 32710][E][vfs_api.cpp:104] open(): /littlefs/www/esps.css does not exist, no permits for creation happened when I did go to the Network Settings page to make sure it was connected and since you had asked for a screenshot earlier, but there was nothing really there, just the DHCP box and AP Reboot checked.

It did sit with "16:19:50: " in the serial output while waiting. My guess is when it dropped around 3am, it finished that line with [WiFiDrv] WiFi Lost the connection to the AP

[MartinMueller2003]

The open error is actually ok. The Web browser asks for the css file and then asks for the compressed version. We only have the compressed verion so the first call fails.

[rledonne]

Please let me know if there is any other way that I can help.

[MartinMueller2003]

This is just very odd. I just did a 24 hour run with no issues.

[stijn-dhondt]

just a quick update!
I started using the latest version and had similar issues.
I removed the FPP 7.1 from the network for now just to eliminate possible sources and all 3 controllers (Dig Uno, Quad and octa remaining online for now. so it sees that at first site FPP is causing the issue.

Will downgrade FPP to a stable version later today. @rledonne which fpp version are you running?

[MartinMueller2003]

Excellent. That is something I can work with.

[stijn-dhondt]

Excellent. That is something I can work with.

@MartinMueller2003 6.3.2 seems to be stable except for when the controllers are connected trough ethernet they show up in FPP with ip 0.0.0.0

[MartinMueller2003]

Added FPP 7.1 to my network (I stopped at 5.3 because I don't agree with the new hat license) and I see the heap dropping regularly. After about 5 minutes the heap drops to a level where no network traffic gets through the system. Only happens when FPP 7 is sending updates. Time to start debugging :)

[MartinMueller2003]

Please try the dist.zip found here: https://drive.google.com/drive/folders/1bGrkXSMoGjWtX5iWspPFRogM2G6c1VXu?usp=sharing

[MartinMueller2003]

That would be good but the code is common so if it works on one it should work on the others. Martin On Tuesday, September 26, 2023 at 03:00:12 AM EDT, stijn-dhondt ***@***.***> wrote: @MartinMueller2003 just tested a dig quad trough wifi and ethernet and working fine now! Do you want me to test a dig uno and Octa also? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[rledonne]

I tried the new zip from yesterday, and it looks much improved.
It stayed connected for 30 minutes and then I tried sending DDP data from xLights, and it stayed connected.
I moved the ESP32 off of the USB cord and back into the controller. It connected, pushed outputs from xLights. It updated in the controller and stayed connected. I played a sequence and it remained connected and made lights blink accordingly.
I booted up FPP, and played a sequence from there using multisync, and it also stayed connected.

I don't know what was changed, but it seems to be solid now (at least for today).

[MartinMueller2003]

I fixed a memory leak in the file close process. It was possible to miss closing a file.

[stijn-dhondt]

Still have some strange issues but give me some time to come back trying to pinpoint it first...

[stijn-dhondt]

@MartinMueller2003 Looking good for now! Controller has been online the whole night. Can you let us know when the update will be available in CI build?

[MartinMueller2003]

Fixed in: Fixed memory leak
#668