[Bug] Harvester Crashes on plot refresh - Unrecoverable without restart

[jRazz]

What happened?

Good day. I will be inserting a couple of screenshots and the relevant logfile entries.
I have my farm machine and full node running with 7675 local plots and a second machine and harvester feeding 2604 more plots. Both running Windows 10 Pro, Intel Core i9 10850K, 128GB RAM, Chia+database on fast boot SSD, etc.
It would appear that at the exact moment that the harvester on the full node was performing a "_plot_refresh_callback", windows performed a routine automatic clock sync and rolled back the system clock by about 2.27 seconds, resulting in the duration for the plot refresh being reported as a negative value (normally it takes 0.28 seconds). It would seem that the harvester did not know how to handle this negative value anomaly, and broke down and proceeded to report that I have 0 local plots henceforth. Remote harvester was still being farmed though. Also, I can confirm that the harvester was indeed not farming my local plots anymore, despite continuing to run, because my EC on the pool dropped sharply. That is what alerted me to the issue, as the GUI thought it was still farming all my plots.
I took the screenshot and restarted my Chia full node client (did not restart remote harvester). The farm returned to normal operation after the restart.

Version

1.7.0

What platform are you using?

Windows

What ui mode are you using?

GUI

Relevant log output

2023-04-12T10:39:47.798 harvester chia.harvester.harvester: INFO     15 plots were eligible for farming e014f54c73... Found 0 proofs. Time: 0.38996 s. Total 7675 plots
2023-04-12T10:39:55.249 harvester chia.harvester.harvester: INFO     _plot_refresh_callback: event started, loaded 0, removed 0, processed 0, remaining 7675, duration: 0.00 seconds, total plots: 7675
2023-04-12T10:39:53.935 harvester chia.harvester.harvester: INFO     _plot_refresh_callback: event done, loaded 0, removed 0, processed 7675, remaining 0, duration: -1.59 seconds, total plots: 7675
2023-04-12T10:39:53.937 harvester chia.plotting.manager   : ERROR    _refresh_callback raised: Value -1 does not fit into uint64 with the traceback: Traceback (most recent call last):
  File "chia\plotting\manager.py", line 202, in _refresh_task
  File "chia\harvester\harvester.py", line 124, in _plot_refresh_callback
  File "chia\plot_sync\sender.py", line 295, in sync_done
  File "chia\util\struct_stream.py", line 69, in __init__
ValueError: Value -1 does not fit into uint64

2023-04-12T10:39:54.312 full_node chia.full_node.full_node: INFO       Finished signage point 50/64: CC: 3b0459943db3943a93a411a0d8dd543fbee7b4068b9deb8a524a01cdf941eb01 RC: ab0787249b0549214f21ab7e0e3917666a3cbb71f141b0367141524ffb108574 
2023-04-12T10:39:54.319 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming e014f54c73... Found 0 proofs. Time: 0.00000 s. Total 0 plots

[MumfMeisterT]

Wow, that's really crazy! We'll look into this. thx.

One of the best bug reports ever!!!**

[jRazz]

Hello again. So, this issue has occurred again. Same circumstance. Windows clock sync rolled back my system clock by about 1.5 seconds exactly in the middle of a plot refresh. This time, running Chia 2.1.1 Giga25. It is the same machine as last time, but last time it was running my full node (and remote harvester had kept on farming), and this time it is the remote harvester and the full node, which is now on another machine, continued farming. Just FYI, in case it might be relevant.

Version

Chia 2.1.1 Giga25

What platform are you using?

Windows

What ui mode are you using?

Windows Powershell

Relevant Log output:

2024-01-31T11:47:27.547 harvester chia.harvester.harvester: INFO     16 plots were eligible for farming 7dcbc86ab4... Found 0 proofs. Time: 3.60832 s. Total 10529 plots
2024-01-31T11:47:30.820 harvester chia.harvester.harvester: INFO     _plot_refresh_callback: event started, loaded 0, removed 0, processed 0, remaining 10529, duration: 0.00 seconds, total plots: 10529
2024-01-31T11:47:30.471 harvester chia.harvester.harvester: INFO     _plot_refresh_callback: event done, loaded 0, removed 0, processed 10529, remaining 0, duration: -1.05 seconds, total plots: 10529
2024-01-31T11:47:30.471 harvester chia.plotting.manager   : ERROR    _refresh_callback raised: Value -1 does not fit into uint64 with the traceback: Traceback (most recent call last):
  File "C:\GIGAHO~2\Farmer\chia\plotting\manager.py", line 255, in _refresh_task
  File "C:\GIGAHO~2\Farmer\chia\harvester\harvester.py", line 168, in _plot_refresh_callback
  File "C:\GIGAHO~2\Farmer\chia\plot_sync\sender.py", line 302, in sync_done
  File "C:\GIGAHO~2\Farmer\chia\util\struct_stream.py", line 73, in __init__
ValueError: Value -1 does not fit into uint64

2024-01-31T11:47:31.814 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming 7dcbc86ab4... Found 0 proofs. Time: 0.00000 s. Total 0 plots
2024-01-31T11:47:40.652 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming 7dcbc86ab4... Found 0 proofs. Time: 0.00000 s. Total 0 plots