Error: Task SysControl timed out! #102

SupportKraatje · 2022-09-03T09:07:07Z

Used to work fine, just stopped......

2022-09-03 09:03:11 N: System boot, last reset reason: SDIO_RESET
2022-09-03 09:03:11 N: HW rev: 2, FW ver.: 2.3.5
2022-09-03 09:03:15 N: WiFi: connection successful
2022-09-03 09:03:15 N: WiFi info:
2022-09-03 09:03:15 N: Mode:STA
2022-09-03 09:03:15 N: Status:3
2022-09-03 09:03:16 N: IP:192.168.1.135
2022-09-03 09:03:16 N: Setup: Virtual remote ID: 205,194,36
2022-09-03 09:03:16 N: Webserver: started
2022-09-03 09:03:16 N: mDNS: started
2022-09-03 09:03:16 N: Hostname: nrg-itho-c224
2022-09-03 09:03:16 N: Setup: done
2022-09-03 09:03:51 N: Error: Task SysControl timed out!
2022-09-03 09:03:59 N: System boot, last reset reason: SDIO_RESET
2022-09-03 09:03:59 N: HW rev: 2, FW ver.: 2.3.5
2022-09-03 09:04:03 N: WiFi: connection successful
2022-09-03 09:04:03 N: WiFi info:
2022-09-03 09:04:03 N: Mode:STA
2022-09-03 09:04:03 N: Status:3
2022-09-03 09:04:03 N: IP:192.168.1.135
2022-09-03 09:04:04 N: Setup: Virtual remote ID: 205,194,36
2022-09-03 09:04:04 N: Webserver: started
2022-09-03 09:04:04 N: mDNS: started
2022-09-03 09:04:04 N: Hostname: nrg-itho-c224
2022-09-03 09:04:04 N: Setup: done

arjenhiemstra · 2022-09-06T09:03:46Z

I suspect the firmware of your itho devices crashed. It seems that some itho devices access the I2C interface without checking if there is activity on the interface leading to a crash. I'm looking into it.
For now a powercylce of your itho should 'fix' the issue. Also lowering the update speed from the standard 5 sec to a more moderate 10 sec. will also help.

TD-er · 2022-09-06T09:55:04Z

I suspect the firmware of your itho devices crashed. It seems that some itho devices access the I2C interface without checking if there is activity on the interface leading to a crash. I'm looking into it.

Or.... the I2C bus is stuck.
This may happen more often than you'd think.
What I do in ESPEasy is this:

Set timeout on I2C object
On timeout or when starting a cycle, detect if either SDA or SCL is pulled low for quite a long time
reverse SDA and SCL and send something (doesn't really matter what) until SDA and SCL are high again (of course take timeout into account)
revert SDA and SCL pins again and restart the I2C bus.

arjenhiemstra · 2022-09-06T10:30:44Z

Looking into this indeed. Thing is with this particular situation, this is a multi master setup where (at least from the itho side) there seems (for some firmware versions?) no control logic present in the itho firmware before a transmit is initiated and when there is a collision it is crashing itself.

Luckily there is a regular pattern, at least for CVE devices, where the itho firmware sends something on the I2C bus. The idea now is to implement some control code to check for this pattern and only send something on the bus after x amount of time until y amount of time before the next I2C communication is to be expected. But it needs to be tested for other itho devices as well.

TD-er · 2022-09-06T10:35:55Z

Do you also have devices on the bus which may use clock stretching?

SupportKraatje · 2022-09-06T10:42:03Z

No, only login with the webbrower (now standalone, will connect it later to domoticz) Van: TD-er ***@***.***> Verzonden: Tuesday, 6 September 2022 12:36 Aan: arjenhiemstra/ithowifi ***@***.***> CC: Support at kraatje ***@***.***>; Author ***@***.***> Onderwerp: Re: [arjenhiemstra/ithowifi] Error: Task SysControl timed out! (Issue #102) Do you also have devices on the bus which may use clock stretching? — Reply to this email directly, view it on GitHub<#102 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ASLRU6FGUAQ6U6T56LKUNPLV44NBNANCNFSM6AAAAAAQD2FPVQ>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>

arjenhiemstra · 2022-09-06T10:43:01Z

Do you also have devices on the bus which may use clock stretching?

I expect that indeed. It has been al long time ago but I remember during my initial reverse engineering efforts I needed to adjust some things around clock stretching to get it to work reliable.

arjenhiemstra · 2022-09-06T10:46:16Z

btw @SupportKraatje could you please provide the details of your itho device after a power cycle?
This info can be found on the itho settings page.
ie.:

TD-er · 2022-09-06T10:49:20Z

OK, so then it might not be useful to use the statistical approach you suggested.
It is better to wait to see if both lines are "up" before you initiate an I2C transfer.
If one of them is "down" for prolonged amount of time, you can assume one of the devices that may try to clock stretch is either waiting for something or actually hanging.
Most often either the master or the sensor missed 1 pulse somewhere.

By temporary swapping SDA and SCL and simply initiating something like a single I2C scan call, or making some call to a non-existing device, you will "push" that hanging device out of its waiting state.

There have to be some pulses on the data and clock to get it to work and depending on where things get stuck, you might have to send various sets of data with different amount of pulses on the data pin.

Luckily there is no need for the I2C clock to be at a fixed frequency, so that's why this may work to unlock the I2C bus when stuck.

SupportKraatje · 2022-09-07T06:22:28Z

btw @SupportKraatje could you please provide the details of your itho device after a power cycle? This info can be found on the itho settings page. ie.:

Itho device type: CVE-Silent
Itho fw version: 27

SupportKraatje · 2022-09-11T16:40:39Z

ok the 10 seconds interval did not help. Hangs again, no fan readings or control (always 0)

sciurius · 2022-09-27T06:24:57Z

FWIW, I seem to be running into the same issue with 2.4.0-alpha8.

2275 I: System boot, last reset reason: POWERON_RESET
2482 I: HW rev: 2, FW ver.: 2.4.0-alpha8
6247 I: WiFi: connection successful
6543 I: WiFi info:
6640 I: Mode:STA
6750 I: Status:3
6874 I: IP:192.168.1.115
7010 I: Setup: Virtual remotes, start ID: 224,139,139
7205 I: Setup: remotes configfile loaded
7377 I: Setup: init of CC1101 RF module successful
7805 I: Setup: remotes configfile loaded
7982 I: MQTT: connected, System config: 1
8358 I: Webserver: started
8536 I: mDNS: started
8679 I: Hostname: nrgitho
8829 I: Setup: done

arjenhiemstra · 2022-09-27T06:26:20Z

FWIW, I seem to be running into the same issue with 2.4.0-alpha8.

2275 I: System boot, last reset reason: POWERON_RESET
2482 I: HW rev: 2, FW ver.: 2.4.0-alpha8
6247 I: WiFi: connection successful
6543 I: WiFi info:
6640 I: Mode:STA
6750 I: Status:3
6874 I: IP:192.168.1.115
7010 I: Setup: Virtual remotes, start ID: 224,139,139
7205 I: Setup: remotes configfile loaded
7377 I: Setup: init of CC1101 RF module successful
7805 I: Setup: remotes configfile loaded
7982 I: MQTT: connected, System config: 1
8358 I: Webserver: started
8536 I: mDNS: started
8679 I: Hostname: nrgitho
8829 I: Setup: done

What is your itho model and firmware version?

sciurius · 2022-09-27T07:34:29Z

Same: CVS-E fw 27.

For testing purposes only! Changes since 2.4.2: Fix: lastCmd.source points sometimes point to bogus mem location Fix: Added I2C unblock function, tries to reset I2C bus and logs result (for issues: #111, #102, #61)

tjoen · 2022-10-19T12:27:20Z

I also still encounter this issue.
I'm already running the latest firmware ( Current firmware version: 2.4.2 ).

Indeed, I already figured out that powercycling the unit makes it work again.
Itho device type: CVE-Silent
Itho fw version: 27

I also noticed last time when it hung that the SysControl task memory had 0 bytes free.
Might be useful in finding a more permanent solution?

arjenhiemstra · 2022-10-21T18:58:42Z

I also noticed last time when it hung that the SysControl task memory had 0 bytes free.

That's useful info, thanks!

Please try https://github.com/arjenhiemstra/ithowifi/releases/tag/Version-2.4.3-beta2
I made some changes to the bus check and added logging to get an idea how many times a collision happens.

tjoen · 2022-10-22T08:39:05Z

Brilliant, I will install it now Arjen.

arjenhiemstra · 2022-10-27T08:16:10Z

I just released version 2.4.3 which addresses this further. In some situations the add-on could still access the I2C bus while another master was active. The latest code changes in 2.4.3 should prevent this or at least further lower the change of this happening.

stale · 2022-12-26T09:29:00Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

sciurius · 2022-12-26T11:53:20Z

I still get it occasionally

2022-12-22 16:25:40 I: Mem free: 103032, Mem low: 65624, Mem block: 44020
2022-12-22 21:46:59 I: Warning: Task SysControl timed out!
2022-12-22 22:54:32 I: System boot, last reset reason: SDIO_RESET
2022-12-22 22:54:32 I: HW rev: 2, FW ver.: 2.4.3

The reboot was manual (power flip) since the unit did not respond.

arjenhiemstra · 2023-01-05T21:44:37Z

Could you try the latest 2.4.4-beta4 firmware and report how it works for you? If you have an itho with built-in humidity sensor please enable the i2c safeguard feature (if your add-on hw revision supports it)

sciurius · 2023-01-05T21:58:31Z

Up and running. Enabled the I2C safeguard. Stay tuned...

arjenhiemstra · 2023-02-17T20:07:15Z

2.5.0 has been released as new stable release, including fixes for the described issue.

https://github.com/arjenhiemstra/ithowifi/releases/tag/2.5.0

arjenhiemstra · 2023-02-21T07:10:29Z

2.5.0 has been released as new stable release, including fixes for the described issue.

https://github.com/arjenhiemstra/ithowifi/releases/tag/2.5.0

I've found a serious issue with the fail save firmware update, please do not install 2.5.0 and if you have already, wait with further updates until 2.5.1 comes out.

stale · 2023-04-22T07:43:00Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

sciurius · 2023-04-22T08:02:22Z

Is it?

arjenhiemstra mentioned this issue Oct 8, 2022

init failed - please power cycle the itho unit - (Warning: I2C timeout) #111

Closed

stale bot added the stale closed No reaction on issue for a long time label Dec 26, 2022

stale bot removed the stale closed No reaction on issue for a long time label Dec 26, 2022

stale bot added the stale closed No reaction on issue for a long time label Apr 22, 2023

stale bot closed this as completed Apr 29, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error: Task SysControl timed out! #102

Error: Task SysControl timed out! #102

SupportKraatje commented Sep 3, 2022

arjenhiemstra commented Sep 6, 2022

TD-er commented Sep 6, 2022

arjenhiemstra commented Sep 6, 2022 •

edited

Loading

TD-er commented Sep 6, 2022

SupportKraatje commented Sep 6, 2022 via email

arjenhiemstra commented Sep 6, 2022

arjenhiemstra commented Sep 6, 2022 •

edited

Loading

TD-er commented Sep 6, 2022

SupportKraatje commented Sep 7, 2022

SupportKraatje commented Sep 11, 2022

sciurius commented Sep 27, 2022

arjenhiemstra commented Sep 27, 2022

sciurius commented Sep 27, 2022

tjoen commented Oct 19, 2022 •

edited

Loading

arjenhiemstra commented Oct 21, 2022

tjoen commented Oct 22, 2022

arjenhiemstra commented Oct 27, 2022

stale bot commented Dec 26, 2022

sciurius commented Dec 26, 2022

arjenhiemstra commented Jan 5, 2023

sciurius commented Jan 5, 2023

arjenhiemstra commented Feb 17, 2023

arjenhiemstra commented Feb 21, 2023

stale bot commented Apr 22, 2023

sciurius commented Apr 22, 2023

Error: Task SysControl timed out! #102

Error: Task SysControl timed out! #102

Comments

SupportKraatje commented Sep 3, 2022

arjenhiemstra commented Sep 6, 2022

TD-er commented Sep 6, 2022

arjenhiemstra commented Sep 6, 2022 • edited Loading

TD-er commented Sep 6, 2022

SupportKraatje commented Sep 6, 2022 via email

arjenhiemstra commented Sep 6, 2022

arjenhiemstra commented Sep 6, 2022 • edited Loading

TD-er commented Sep 6, 2022

SupportKraatje commented Sep 7, 2022

SupportKraatje commented Sep 11, 2022

sciurius commented Sep 27, 2022

arjenhiemstra commented Sep 27, 2022

sciurius commented Sep 27, 2022

tjoen commented Oct 19, 2022 • edited Loading

arjenhiemstra commented Oct 21, 2022

tjoen commented Oct 22, 2022

arjenhiemstra commented Oct 27, 2022

stale bot commented Dec 26, 2022

sciurius commented Dec 26, 2022

arjenhiemstra commented Jan 5, 2023

sciurius commented Jan 5, 2023

arjenhiemstra commented Feb 17, 2023

arjenhiemstra commented Feb 21, 2023

stale bot commented Apr 22, 2023

sciurius commented Apr 22, 2023

arjenhiemstra commented Sep 6, 2022 •

edited

Loading

arjenhiemstra commented Sep 6, 2022 •

edited

Loading

tjoen commented Oct 19, 2022 •

edited

Loading