Internal server error on room join

[MayeulC]

Description

I've had this stuck invite for a while in my client. It might have been due to using synapse janitor scripts more than a year ago, then receiving a message in a room I had left. Subsequently, I asked to be re-invited in that room.

Accepting the invitation triggers the following output in synapse 1.30.1:

2021-04-06 14:13:43,821 - synapse.http.server - 98 - ERROR - POST-19184- Failed handle request via 'JoinRoomAliasServlet': <XForwardedForRequest at 0x7ff3f6027550 method='POST' uri='/_matrix/client/r0/join/!BJWjCfMxUuIScQPqFU%3Amatrix.org' clientproto='HTTP/1.0' site='8008'>
Traceback (most recent call last):
  File "/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/sy
napse/http/server.py", line 259, in _async_render_wrapper    callback_return = await self._async_render(request)
  File "/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/synapse/http/server.py", line 447, in _async_render
    callback_return = await raw_callback_return
  File "/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/synapse/rest/client/v1/room.py", line 309, in on_POST
    third_party_signed=content.get("third_party_signed", None),  File "/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/synapse/handlers/room_member.py", line 350, in update_membership
    require_consent=require_consent,
  File "/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/synapse/handlers/room_member.py", line 557, in update_membership_locked    requester, remote_room_hosts, room_id, target, content
  File "/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/synapse/handlers/room_member.py", line 1101, in _remote_join
    remote_room_hosts, room_id, user.to_string(), content  File "/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/synapse/handlers/federation.py", line 1490, in do_invite_join    origin, room_id, auth_chain, state, event, room_version_obj
  File "/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/synapse/handlers/federation.py", line 2146, in _persist_auth_tree
    for e in itertools.chain(auth_events, state)
  File "/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/synapse/handlers/federation.py", line 3020, in persist_events_and_notify
    event_and_contexts, backfilled=backfilled  File "/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/synapse/storage/persist_events.py", line 263, in persist_events    defer.gatherResults(deferreds, consumeErrors=True)
twisted.internet.defer.FirstError: FirstError[#0, [Failure instance: Traceback: <class 'psycopg2.errors.UniqueViolation'>: ERREUR:  la valeur d'une clé dupliquée rompt la contrainte unique « event_reference_hashes_event_id_algorithm_key »
DETAIL:  La clé « (event_id, algorithm)=($L4lY_Vgeomi7X0-gXccrsWTHEwqypK5tgVrFqs0e86M, sha256) » existe déjà./opt/yunohost/matrix-synapse/lib/python3.7/site-packages/twisted/internet/defer.py:662:_runCallbacks/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/twisted/internet/defer.py:1514:gotResult/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/twisted/internet/defer.py:1443:_inlineCallbacks/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/twisted/python/failure.py:500:throwExceptionIntoGenerator
--- <exception caught here> ---/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/synapse/storage/persist_events.py:172:handle_queue_loop
/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/synapse/storage/persist_events.py:325:persisting_queue/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/synapse/storage/persist_events.py:537:_persist_events
/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/synapse/storage/databases/main/events.py:183:_persist_events_and_state_updates
/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/synapse/storage/database.py:668:runInteraction/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/synapse/storage/database.py:744:runWithConnection
/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/twisted/python/threadpool.py:238:inContext
/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/twisted/python/threadpool.py:255:<lambda>/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/twisted/python/context.py:118:callWithContext
/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/twisted/python/context.py:83:callWithContext/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/twisted/enterprise/adbapi.py:293:_runWithConnection
/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/twisted/python/deprecate.py:298:deprecatedFunction/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/twisted/python/compat.py:403:reraise
/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/twisted/enterprise/adbapi.py:284:_runWithConnection/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/synapse/storage/database.py:738:inner_func
/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/synapse/storage/database.py:532:new_transaction/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/synapse/logging/utils.py:70:wrapped
/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/synapse/storage/databases/main/events.py:400:_persist_events_txn
/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/synapse/storage/databases/main/events.py:1512:_update_metadata_tables_txn
/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/synapse/storage/databases/main/events.py:1632:_store_event_reference_hashes_txn
/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/synapse/storage/database.py:896:simple_insert_many_txn/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/synapse/storage/database.py:275:execute_batch
/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/synapse/storage/database.py:320:_do_execute/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/synapse/storage/database.py:275:<lambda>
/opt/yunohost/matrix-synapse/lib/python3.7/site-packages/psycopg2/extras.py:1209:execute_batch
]]

Steps to reproduce

• Have some predetermined condition in the database (sorry, I can't help here)
• Notice there is an invite
• Accept the invite

I would expect accepting it to succeed, or to fail in the worst case.

Version information

• Homeserver: https://mayeul.net

• Version: 1.30.1

• Install method: yunohost "app"

• Platform: yunohost

[anoadragon453]

Hey @MayeulC, we definitely have seen people run into this exact issue due to running synapse_janitor in the past: #8323 (comment)

The solution is to delete the offending rows from event_reference_hashes using the SQL in the link above. 🙂

[MayeulC]

Wow, excellent, that fixed it. After removing them, I still had the invite, but was able to join.

I somehow didn't find that issue before reporting, so thank you a lot for the pointer! 🎉