List index out of range in phone_stats_home

[ulope]

Description

Recently we've started seeing the following exception appear in our homeserver logs every couple of hours:

2019-12-20 00:35:19,441 synapse.metrics.background_process_metrics(214) [ERROR   ]: phone_stats_home-68 Background process 'phone_stats_home' threw an exception
Traceback (most recent call last):
  File "/synapse-venv/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: 1530
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/synapse-venv/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: 1530
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/synapse-venv/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: 1530
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/synapse-develop/synapse/metrics/background_process_metrics.py", line 212, in run
    return (yield result)
  File "/synapse-venv/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/opt/synapse-develop/synapse/app/homeserver.py", line 525, in phone_stats_home
    old = stats_process[0]
IndexError: list index out of range

The index in the StopIteration exception varies between different occurrences of the message.

Steps to reproduce

¯\_(ツ)_/¯

Version information

• Homeserver: https://transport03.raiden.network (Separate Raiden Network testing federation)
• Version: 1.5.1
• Install method: Custom docker-compose setup (https://github.com/raiden-network/raiden-service-bundle)
• Platform: Linux / Docker

[richvdh]

any exceptions from performance_stats_init on startup?

[ulope]

Indeed:

2019-12-11 12:35:19,455 twisted(172) [CRITICAL]:  Unhandled Error
Traceback (most recent call last):
  File "/opt/synapse-develop/synapse/app/_base.py", line 114, in run
    run_command()
  File "/synapse-venv/lib/python3.7/site-packages/twisted/internet/base.py", line 1283, in run
    self.mainLoop()
  File "/synapse-venv/lib/python3.7/site-packages/twisted/internet/base.py", line 1292, in mainLoop
    self.runUntilCurrent()
  File "/opt/synapse-develop/synapse/metrics/__init__.py", line 434, in f
    ret = func(*args, **kwargs)
--- <exception caught here> ---
  File "/synapse-venv/lib/python3.7/site-packages/twisted/internet/base.py", line 913, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File "/opt/synapse-develop/synapse/util/__init__.py", line 97, in wrapped_callback
    callback(*args, **kwargs)
  File "/opt/synapse-develop/synapse/app/homeserver.py", line 588, in performance_stats_init
    (int(hs.get_clock().time(), resource.getrusage(resource.RUSAGE_SELF)))
builtins.TypeError: 'resource.struct_rusage' object cannot be interpreted as an integer

[richvdh]

oh, this was fixed in Synapse 1.6 by #6418.