Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WARNING] --- Your harvester appears to be offline! No events for the past 901 seconds. (keep_alive_monitor.py:83) #243

Closed
springjools opened this issue Jun 22, 2021 · 3 comments

Comments

@springjools
Copy link

WARNING] --- Your harvester appears to be offline! No events for the past 901 seconds. (keep_alive_monitor.py:83)

I get this error every 5 minutes, even if the harvester is up and running. Sometimes it helps to kill the chiadog process, rm debug.log.offset and restart everything.

Environment:

  • OS: Linux uranian 5.10.39-v8+ #1421 SMP PREEMPT Tue May 25 11:04:26 BST 2021 aarch64 GNU/Linux
  • Python version: Python 3.7.3
  • PIP version: pip 18.1 from /usr/lib/python3/dist-packages/pip (python 3.7)
  • Chia version: 1.1.7
@niebochod
Copy link

I had the same problem after sudden PC reboot, an garbage has been written into logs and log consumer has been confused and throwing exception at start:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\site-packages\decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\site-packages\retry\api.py", line 74, in retry_decorator
    logger)
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\site-packages\retry\api.py", line 33, in __retry_internal
    return f()
  File "D:\ProgramData\chiadog\src\chia_log\log_consumer.py", line 77, in _consume_loop
    for log_line in Pygtail(self._expanded_log_path, read_from_end=True, offset_file=self._offset_path):
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\site-packages\pygtail\core.py", line 87, in __init__
    [int(line.strip()) for line in offset_fh]
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\site-packages\pygtail\core.py", line 87, in <listcomp>
    [int(line.strip()) for line in offset_fh]
ValueError: invalid literal for int() with base 10: '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'

It looks like the consumer thread is crashed, so for chiadog it looks like there is no changes in the chia logs, so offline state assumed?

@pieterhelsen
Copy link
Collaborator

I had the same problem after sudden PC reboot, an garbage has been written into logs and log consumer has been confused and throwing exception at start:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\site-packages\decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\site-packages\retry\api.py", line 74, in retry_decorator
    logger)
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\site-packages\retry\api.py", line 33, in __retry_internal
    return f()
  File "D:\ProgramData\chiadog\src\chia_log\log_consumer.py", line 77, in _consume_loop
    for log_line in Pygtail(self._expanded_log_path, read_from_end=True, offset_file=self._offset_path):
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\site-packages\pygtail\core.py", line 87, in __init__
    [int(line.strip()) for line in offset_fh]
  File "C:\Users\lukas\AppData\Local\Programs\Python\Python37\lib\site-packages\pygtail\core.py", line 87, in <listcomp>
    [int(line.strip()) for line in offset_fh]
ValueError: invalid literal for int() with base 10: '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'

It looks like the consumer thread is crashed, so for chiadog it looks like there is no changes in the chia logs, so offline state assumed?

This is a known bug for which a fix is in the pipeline.
Meanwhile, deleting the debug.log.offset file from the Chiadog directory will fix this issue.
More here:
#241

Pull request here:
#214

@martomi
Copy link
Owner

martomi commented Jul 14, 2021

Bugfix is now released to main in release 0.6.1

@martomi martomi closed this as completed Jul 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants