Lots of errors after upgrade to 1.37

[Vaelio]

Description

I just updated synapse to 1.37 and after the restart the logs show errors such as this one (the host mentioned in the error is not the only one throwing errors):

2021-06-29 16:26:05,304 - synapse.access.http.8008 - 389 - INFO - PUT-10005 - 144.76.200.170 - 8008 - {rita.moe} Processed request: 0.032sec/-0.000sec (0.002sec, 0.004sec) (0.011sec/0.013sec/4) 11B 200 "PUT /_matrix/federation/v1/send/1624641365353 HTTP/1.1" "Synapse/1.36.0" [0 dbevts]                                                                                                                                                                                                                                              
2021-06-29 16:26:05,314 - synapse.metrics.background_process_metrics - 216 - ERROR - stats.notify_new_event-1484 - Background process 'stats.notify_new_event' threw an exception                                                                                     
Traceback (most recent call last):                                                                                       
  File "/usr/local/lib/python3.8/dist-packages/synapse/metrics/background_process_metrics.py", line 214, in run         
    return await maybe_awaitable(func(*args, **kwargs))                                                                          
  File "/usr/local/lib/python3.8/dist-packages/synapse/handlers/stats.py", line 74, in process                    
    await self._unsafe_process()                                                                                            
  File "/usr/local/lib/python3.8/dist-packages/synapse/handlers/stats.py", line 127, in _unsafe_process                    
    await self.store.bulk_update_stats_delta(                                                                 
  File "/usr/local/lib/python3.8/dist-packages/synapse/storage/databases/main/stats.py", line 398, in bulk_update_stats_delta
    await self.db_pool.runInteraction(                                                                                    
  File "/usr/local/lib/python3.8/dist-packages/synapse/storage/database.py", line 681, in runInteraction                   
    result = await self.runWithConnection(                                                                           
  File "/usr/local/lib/python3.8/dist-packages/synapse/storage/database.py", line 770, in runWithConnection                     
    return await make_deferred_yieldable(                                                                                        
  File "/usr/local/lib/python3.8/dist-packages/twisted/python/threadpool.py", line 250, in inContext                              
    result = inContext.theWork()                                                                                              
  File "/usr/local/lib/python3.8/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>                                             
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)                                                                     
  File "/usr/local/lib/python3.8/dist-packages/twisted/python/context.py", line 122, in callWithContext                             
    return self.currentContext().callWithContext(ctx, func, *args, **kw)                                                             
  File "/usr/local/lib/python3.8/dist-packages/twisted/python/context.py", line 85, in callWithContext                                       
    return func(*args,**kw)                                                                                                                 
  File "/usr/local/lib/python3.8/dist-packages/twisted/enterprise/adbapi.py", line 306, in _runWithConnection                        
    compat.reraise(excValue, excTraceback)                                                                                         
  File "/usr/local/lib/python3.8/dist-packages/twisted/python/compat.py", line 464, in reraise                                 
    raise exception.with_traceback(traceback)                                                                                                              
  File "/usr/local/lib/python3.8/dist-packages/twisted/enterprise/adbapi.py", line 297, in _runWithConnection                                                               
    result = func(conn, *args, **kw)                                                                                                                 
  File "/usr/local/lib/python3.8/dist-packages/synapse/storage/database.py", line 765, in inner_func                                         
    return func(db_conn, *args, **kwargs)                                                                                                         
  File "/usr/local/lib/python3.8/dist-packages/synapse/storage/database.py", line 549, in new_transaction                              
    r = func(cursor, *args, **kwargs)                                                                                                    
  File "/usr/local/lib/python3.8/dist-packages/synapse/storage/databases/main/stats.py", line 382, in _bulk_update_stats_delta_txn                     
    self._update_stats_delta_txn(                                                                                                                
  File "/usr/local/lib/python3.8/dist-packages/synapse/storage/databases/main/stats.py", line 486, in _update_stats_delta_txn                          
    self._upsert_with_additive_relatives_txn(                                   
  File "/usr/local/lib/python3.8/dist-packages/synapse/storage/databases/main/stats.py", line 571, in _upsert_with_additive_relatives_txn
    current_row[key] += val          
TypeError: unsupported operand type(s) for +=: 'NoneType' and 'int'

To me, it looks like it's just an incompatibility of some sort with older synapse but i wanted to make sure (also, i have not yet found any service interruption with my own users)

Steps to reproduce

I just had to upgrade from previous version

Version information

• Version: 1.37

• Install method: pip

• Platform: Ubuntu on AWS, i use sqlite database

[richvdh]

looks like somehow a NULL value has got into either room_stats_historical or user_stats_historical :/

[richvdh]

(worth noting that #9721 removes the code that updates those tables, so this will go away of its own accord)

[Vaelio]

looks like somehow a NULL value has got into either room_stats_historical or user_stats_historical :/

Do you want me to check that out or should we just wait the next release for the code to get removed ?

Thanks for your reply :)

[richvdh]

oh this might be due to the new knock stuff (#6739). We'll have a look.

[Vaelio]

FYI, I upgraded to 1.37.1 and the problem persists.
Also, i dont think its any critical since there's no loss of functionality. (however i'd say there is one to two errors per second witch spams the logs pretty hard)
Let me know if you need any infos anyway :)