Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

room $ROOM for event $EVENT is unknown #7615

Closed
brandoncurtis opened this issue Jun 1, 2020 · 25 comments
Closed

room $ROOM for event $EVENT is unknown #7615

brandoncurtis opened this issue Jun 1, 2020 · 25 comments
Assignees
@erikjohnston
Labels
z-bug (Deprecated Label)

Comments

@brandoncurtis
Copy link

brandoncurtis commented Jun 1, 2020 •
edited
Loading

Description

Synapse log is full of these warnings:

2020-06-01 10:25:04,210 - synapse.storage.data_stores.main.events_worker - 592 - WARNING - notify_app_services-329- Room !xPRJVyOhRymgpkarYZ:matrix.org for event $15015719735288253CjVIO:matrix.org is unknown

I suspect that this may be causing problems with Synapse reporting events to a self-hosted bridge; that may be a separate issue, this issue is just about dealing with this warning.

Speculatively related:

Steps to reproduce

  • invite an appservice bot user to a channel
  • send a message in that channel
  • receive error messages

Version information

  • Homeserver: personal (not matrix.org)

If not matrix.org:

  • Version:
{
    "python_version": "3.6.8",
    "server_version": "1.13.0"
}
  • Install method: installed with pipenv
  • Platform: Ubuntu 18.04
@Josue-T
Copy link

Josue-T commented Jun 1, 2020 •
edited
Loading

I confirm that I've the same issue with version 1.14.0 on debian 9. Linked to mautrix/whatsapp#193

@Josue-T Josue-T mentioned this issue Jun 1, 2020
Closed
@brandoncurtis
Copy link
Author

@Josue-T I can confirm that deleting the events associated with this broken room stopped the errors and allowed events to flow to appservice services, thereby allowing my bridge bot to join rooms and function again.

Here's the code where this error is being thrown:

synapse/synapse/storage/data_stores/main/events_worker.py

Lines 626 to 632 in e5c67d0

if not room_version_id:
# this should only happen for out-of-band membership events
if not internal_metadata.get("out_of_band_membership"):
logger.warning(
"Room %s for event %s is unknown", d["room_id"], event_id
)
continue

I connected to the Synapse postgres db and verified that the room mentioned in the error message was not present in the rooms table:

sudo su - postgres
psql
\c synapse
SELECT * FROM rooms;

I also verified that none of the entries in the state_groups_state table referenced this room_id:

SELECT * FROM state_groups_state WHERE room_id = '!xPRJVyOhRymgpkarYZ:matrix.org';

then I went ahead and deleted the offending events:

DELETE FROM events WHERE room_id = '!xPRJVyOhRymgpkarYZ:matrix.org';

Restarted synapse, no errors!
Restarted the broken bridge and it functioned fine.

I do NOT recommend cavalier database edits. You'll probably want to wait for an official response and fix, but I thought I'd post what I've found here in case it's useful to someone else.

@brandoncurtis brandoncurtis mentioned this issue Jun 2, 2020
Closed
@erikjohnston
Copy link
Member

@brandoncurtis I don't suppose you have a copy of the event in question? I'm wondering if we've somehow managed to store an out_of_band_membership without marking it as such

@erikjohnston erikjohnston added z-bug (Deprecated Label) info-needed labels Jun 2, 2020
@brandoncurtis
Copy link
Author

My bad, I have destroyed the events. Perhaps @Josue-T can help?

This homeserver has been updated through many years and many event bugs, so I would not be surprised if this was a side-effect of previous database surgery.

This room_id had no entries in state_groups_state:

synapse=# SELECT * FROM state_groups_state WHERE room_id = '!xPRJVyOhRymgpkarYZ:matrix.org';                                                                                                                       

 state_group | room_id | type | state_key | event_id 
-------------+---------+------+-----------+----------
(0 rows)

@erikjohnston
Copy link
Member

Ah, never mind. Could you check for the room in local_current_membership too please? That may tell us if its an invite or whatever. While you're there, it might also be worth checking in room_memberships and state_events table for that event. And maybe for the room in room_stats_current, room_stats_historical and room_stats_state?

(I'm not sure whether to be happy or sad that an event can leave its mark on so many tables...)

@Josue-T
Copy link

Josue-T commented Jun 4, 2020

Hello,

I've theses event type in the event table:

m.room.create
m.room.member
m.room.power_levels
m.room.join_rules
m.room.history_visibility
m.room.guest_access
m.room.name
m.room.member
m.room.member
m.room.member

In the table local_current_membership and room_memberships I've only me as user and the whatsapp_bot as invite

The content of some part of state_events:

type state_key prev_state
m.room.create   NULL
m.room.member @user:domain.tld NULL
m.room.power_levels   NULL
m.room.join_rules   NULL
m.room.history_visibility   NULL
m.room.guest_access   NULL
m.room.name   NULL
m.room.member @whatsappbot:domain.tld NULL

The content of some part of room_stats_current:

room_id current_state_events joined_members invited_members left_members banned_members local_users_in_room completed_delta_stream_id
!YglmofrMgwoolbYsMx:domain.tld 8 1 1 0 0 1 362876

The content of some part of room_stats_historical:

room_id end_ts bucket_size current_state_events joined_members invited_members left_members banned_members local_users_in_room total_events total_event_bytes
!YglmofrMgwoolbYsMx:domain.tld 1590883200000 86400000 8 1 1 0 0 1 8 6361
!YglmofrMgwoolbYsMx:domain.tld 1591315200000 86400000 8 1 1 0 0 1 2 2232

The content of some part of room_stats_state:

room_id name canonical_alias join_rules history_visibility encryption avatar guest_access is_federatable topic
!YglmofrMgwoolbYsMx:domain.tld Whatsapp bot NULL invite shared NULL NULL can_join True NULL

Hope that It will help to find the bug.

@erikjohnston
Copy link
Member

Oh, that's very interesting. So this is an entirely local room? Could you check what the room_version column is for the rooms in the room table please? I don't suppose you have the entry of the problematic event in the events and event_json table?

@Josue-T
Copy link

Josue-T commented Jun 5, 2020 •
edited
Loading

On riot the room version is 5.

I did some more investigation and as I see the problematic event are not linked to the problematic room but to an other room !XqBunHwQIXUiqCaoxq:matrix.org. The strange thing is that this room is not in the room table.

I don't suppose you have the entry of the problematic event in the events and event_json table?

I can find the problematic event int the event table and in the event_json table.

This is my synapse log:

2020-06-05 11:52:23,349 - synapse.access.http.8008 - 311 - INFO - OPTIONS-77727- xx:xx:xx::xx - 8008 - {None} Processed request: 0.003sec/-0.000sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 2B 200 "OPTIONS /_matrix/client/r0/profile/%40whatsappbot%3Adomain.tld HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0" [0 dbevts]
2020-06-05 11:52:23,446 - synapse.access.http.8008 - 311 - INFO - GET-77728- xx:xx:xx::xx - 8008 - {None} Processed request: 0.058sec/0.002sec (0.005sec, 0.002sec) (0.007sec/0.041sec/2) 98B 200 "GET /_matrix/client/r0/profile/%40whatsappbot%3Adomain.tld HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0" [0 dbevts]
2020-06-05 11:52:23,533 - synapse.access.http.8008 - 311 - INFO - OPTIONS-77729- xx:xx:xx::xx - 8008 - {None} Processed request: 0.003sec/-0.000sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 2B 200 "OPTIONS /_matrix/client/r0/rooms/!YglMOfrMgwOOlbYsMx%3Adomain.tld/invite HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0" [0 dbevts]
2020-06-05 11:52:25,965 - synapse.access.http.8008 - 311 - INFO - POST-77730- xx:xx:xx::xx - 8008 - {@user:domain.tld} Processed request: 2.392sec/0.012sec (0.059sec, 0.015sec) (0.019sec/0.468sec/10) 2B 200 "POST /_matrix/client/r0/rooms/!YglMOfrMgwOOlbYsMx%3Adomain.tld/invite HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0" [0 dbevts]
2020-06-05 11:52:26,263 - synapse.federation.transport.server - 406 - INFO - PUT-77731- Received txn 1590298974935 from cmaboite.me. (PDUs: 0, EDUs: 1)
2020-06-05 11:52:26,301 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1416420931595BRDPc:jki.re is unknown
2020-06-05 11:52:26,310 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1439805849916cbSEx:matrix.org is unknown
2020-06-05 11:52:26,312 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14307829793565cxWOU:matrix.org is unknown
2020-06-05 11:52:26,314 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1416420910554DEyOl:matrix.org is unknown
2020-06-05 11:52:26,317 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1423520288129iSnro:matrix.org is unknown
2020-06-05 11:52:26,320 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1416420910538EkGDl:matrix.org is unknown
2020-06-05 11:52:26,324 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1438168069192256fVYnp:matrix.org is unknown
2020-06-05 11:52:26,326 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1440776481135125ENxKQ:matrix.org is unknown
2020-06-05 11:52:26,331 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $143544325157690WPOnr:matrix.org is unknown
2020-06-05 11:52:26,336 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1416420910522mQfXQ:matrix.org is unknown
2020-06-05 11:52:26,338 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $143679141068830nmOIg:matrix.org is unknown
2020-06-05 11:52:26,340 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1438965068161Yznuf:enconn.fr is unknown
2020-06-05 11:52:26,343 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14301303755606YYPjb:matrix.org is unknown
2020-06-05 11:52:26,345 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $143756384113352eBEbk:matrix.org is unknown
2020-06-05 11:52:26,348 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1422889053877NDXTP:matrix.org is unknown
2020-06-05 11:52:26,350 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $143550163473031vyTLF:matrix.org is unknown
2020-06-05 11:52:26,352 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14241200161876Kwyps:matrix.org is unknown
2020-06-05 11:52:26,355 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1416420931530AvRJH:jki.re is unknown
2020-06-05 11:52:26,357 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1424077226212qJMYW:matrix.org is unknown
2020-06-05 11:52:26,359 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14296387653702KHmdA:matrix.org is unknown
2020-06-05 11:52:26,361 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $142375629822gFxyM:jki.re is unknown
2020-06-05 11:52:26,363 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1416420914880oFczF:jki.re is unknown
2020-06-05 11:52:26,365 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14229703262125euXEv:matrix.org is unknown
2020-06-05 11:52:26,367 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1416420913198wPxtf:matrix.org is unknown
2020-06-05 11:52:26,369 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1416420914858UvPkK:jki.re is unknown
2020-06-05 11:52:26,371 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1416420913137QDxRk:matrix.org is unknown
2020-06-05 11:52:26,373 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14247963497668FsZSQ:matrix.org is unknown
2020-06-05 11:52:26,375 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14359356456575KFENf:matrix.org is unknown
2020-06-05 11:52:26,377 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14400972745cfkZh:mtrx.me is unknown
2020-06-05 11:52:26,379 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1416420915228Byipk:matrix.org is unknown
2020-06-05 11:52:26,381 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $143621577316243nnJpT:matrix.org is unknown
2020-06-05 11:52:26,382 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1416420910545FDbdr:matrix.org is unknown
2020-06-05 11:52:26,384 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $142316567653uFnap:jki.re is unknown
2020-06-05 11:52:26,386 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $143983625614685edzmR:matrix.org is unknown
2020-06-05 11:52:26,388 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14229213501730PHsnA:matrix.org is unknown
2020-06-05 11:52:26,390 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14192190793qHYoR:matrix.org is unknown
2020-06-05 11:52:26,392 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1416420910533PjyPo:matrix.org is unknown
2020-06-05 11:52:26,394 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1436142616100350HupcV:matrix.org is unknown
2020-06-05 11:52:26,396 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1437063008168960yVuaA:matrix.org is unknown
2020-06-05 11:52:26,397 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1437129987190447sDZvJ:matrix.org is unknown
2020-06-05 11:52:26,399 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1426624670851jPIbA:matrix.org is unknown
2020-06-05 11:52:26,401 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14230695081sutcl:erikjohnston.com is unknown
2020-06-05 11:52:26,402 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1416421015341kLOpD:matrix.org is unknown
2020-06-05 11:52:26,404 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14320437948vHCSh:testmatrix1.jki.re is unknown
2020-06-05 11:52:26,406 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14273591061010PsYjy:matrix.org is unknown
2020-06-05 11:52:26,408 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14226594140wFAqd:matrix.org is unknown
2020-06-05 11:52:26,409 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1420631992187xqQNO:matrix.org is unknown
2020-06-05 11:52:26,411 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14278400474523vWFYp:matrix.org is unknown
2020-06-05 11:52:26,413 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14229712432141claSN:matrix.org is unknown
2020-06-05 11:52:26,415 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14261361403059PYiva:matrix.org is unknown
2020-06-05 11:52:26,417 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14189830041267gNtmw:matrix.org is unknown
2020-06-05 11:52:26,419 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1416420914590avyez:jki.re is unknown
2020-06-05 11:52:26,420 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1416420911042txvso:matrix.org is unknown
2020-06-05 11:52:26,422 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14310875342eENLP:hveem.no is unknown
2020-06-05 11:52:26,424 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14229713842146ZAjhF:matrix.org is unknown
2020-06-05 11:52:26,426 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1416420923842GbLym:jki.re is unknown
2020-06-05 11:52:26,428 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14239186781073zEOfL:matrix.org is unknown
2020-06-05 11:52:26,430 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $143960794628731bVxmU:matrix.org is unknown
2020-06-05 11:52:26,431 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1438966636237383fXNQH:matrix.org is unknown
2020-06-05 11:52:26,435 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1416420930650ZDaIU:matrix.org is unknown
2020-06-05 11:52:26,437 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $143776226249569DvHoE:matrix.org is unknown
2020-06-05 11:52:26,439 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14291807974089hxMme:matrix.org is unknown
2020-06-05 11:52:26,440 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $142513682490hojBq:port1024.net is unknown
2020-06-05 11:52:26,442 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14292447074817IKTqr:matrix.org is unknown
2020-06-05 11:52:26,444 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14189829381263XLZgc:matrix.org is unknown
2020-06-05 11:52:26,453 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1419254862222VBMnw:matrix.org is unknown
2020-06-05 11:52:26,455 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14212537682914RscpP:matrix.org is unknown
2020-06-05 11:52:26,457 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14392964774231sEhqA:matrix.org is unknown
2020-06-05 11:52:26,459 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14303233541ncpHh:matrix.org is unknown
2020-06-05 11:52:26,462 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1416420910692FTaSj:jki.re is unknown
2020-06-05 11:52:26,463 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14242457004304yTpaI:matrix.org is unknown
2020-06-05 11:52:26,466 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14284078663134Vszfs:matrix.org is unknown
2020-06-05 11:52:26,468 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14205342164942MYPnJ:matrix.org is unknown
2020-06-05 11:52:26,470 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14194248451407xZVvE:matrix.org is unknown
2020-06-05 11:52:26,472 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1416420925146WUNCs:jki.re is unknown
2020-06-05 11:52:26,474 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $143783747048btmYk:main.f-t.so is unknown
2020-06-05 11:52:26,476 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $144069329652YfjwY:matrix.pin13.net is unknown
2020-06-05 11:52:26,478 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1423732972933TDArl:matrix.org is unknown
2020-06-05 11:52:26,480 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14214144884579TNCdy:matrix.org is unknown
2020-06-05 11:52:26,482 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $143549914772235OtagP:matrix.org is unknown
2020-06-05 11:52:26,484 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1416420918249zVeZL:matrix.org is unknown
2020-06-05 11:52:26,486 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14321493192769zVkfr:matrix.org is unknown
2020-06-05 11:52:26,488 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1437578408305ussTo:matrix.org is unknown
2020-06-05 11:52:26,490 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1416420910529mzqgm:matrix.org is unknown
2020-06-05 11:52:26,492 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1423870384840HEwwn:matrix.org is unknown
2020-06-05 11:52:26,494 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14359384297461Lvjne:matrix.org is unknown
2020-06-05 11:52:26,496 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14392962774171hsohJ:matrix.org is unknown
2020-06-05 11:52:26,498 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1426838849588MDZgT:matrix.org is unknown
2020-06-05 11:52:26,500 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14229713842147qGCCG:matrix.org is unknown
2020-06-05 11:52:26,502 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $142367712494mQFAy:matrix.org is unknown
2020-06-05 11:52:26,504 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $1416420910562CZaRR:matrix.org is unknown
2020-06-05 11:52:26,506 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $143646008199241oRZsK:matrix.org is unknown
2020-06-05 11:52:26,508 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14274632213691tlxFu:matrix.org is unknown
2020-06-05 11:52:26,510 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14189831211271lNNzt:matrix.org is unknown
2020-06-05 11:52:26,512 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $14267797422096PtNcC:matrix.org is unknown
2020-06-05 11:52:26,514 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $143639585277775fiDbW:matrix.org is unknown
2020-06-05 11:52:26,516 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $143871846325235YXXPl:matrix.org is unknown
2020-06-05 11:52:26,519 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $143940527710428VmXDs:matrix.org is unknown
2020-06-05 11:52:26,521 - synapse.storage.data_stores.main.events_worker - 630 - WARNING - notify_app_services-2957- Room !XqBunHwQIXUiqCaoxq:matrix.org for event $143947208740394OxCNC:matrix.org is unknown
2020-06-05 11:52:27,011 - synapse.access.http.8008 - 311 - INFO - GET-77726- xx:xx:xx::xx - 8008 - {@user:domain.tld} Processed request: 8.824sec/0.003sec (0.096sec, 0.013sec) (0.040sec/0.622sec/5) 1433B 200 "GET /_matrix/client/r0/sync?filter=27&timeout=30000&since=s362971_6475929_3776_203400_77677_3_675_34561_3 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0" [0 dbevts]
2020-06-05 11:52:27,022 - synapse.access.https.8448 - 311 - INFO - PUT-77731- ::ffff:82.64.159.13 - 8448 - {cmaboite.me} Processed request: 0.781sec/0.003sec (0.029sec, 0.005sec) (0.024sec/0.401sec/3) 12B 200 "PUT /_matrix/federation/v1/send/1590298974935 HTTP/1.1" "Synapse/1.12.4" [0 dbevts]
2020-06-05 11:52:27,179 - synapse.access.http.8008 - 311 - INFO - OPTIONS-77732- xx:xx:xx::xx - 8008 - {None} Processed request: 0.004sec/-0.000sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 2B 200 "OPTIONS /_matrix/client/r0/sync?filter=27&timeout=30000&since=s362972_6475930_3776_203400_77677_3_675_34561_3 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0" [0 dbevts]
2020-06-05 11:52:28,418 - synapse.access.http.8008 - 311 - INFO - OPTIONS-77734- xx:xx:xx::xx - 8008 - {None} Processed request: 0.004sec/-0.000sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 2B 200 "OPTIONS /_matrix/client/r0/rooms/!YglMOfrMgwOOlbYsMx%3Adomain.tld/read_markers HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0" [0 dbevts]

@Josue-T
Copy link

Josue-T commented Jun 5, 2020

I also verified that none of the entries in the state_groups_state table referenced this room_id:

SELECT * FROM state_groups_state WHERE room_id = '!xPRJVyOhRymgpkarYZ:matrix.org';

@brandoncurtis I've a very big number of event in this table linked to this problematic room.

@erikjohnston
Copy link
Member

The strange thing is that this room is not in the room table.

So this is why you're seeing the error, the problem is that we really should have it in there. Have you upgraded synapse since the room was created?

@Josue-T
Copy link

Josue-T commented Jun 5, 2020

Now I found which room is:

Some part of room_stats_state table:

room_id name canonical_alias join_rules history_visibility encryption avatar guest_access is_federatable topic
!XqBunHwQIXUiqCaoxq:matrix.org #matrix-dev:matrix.org #matrix-dev:matrix.org public world_readable NULL mxc://jki.re/aIcNgIBIBRimXoYOXFOSBpgC NULL True Dev chat for everyone building on top of Matrix | http://github.com/matrix-org | Current synapse version: v0.19.2

@Josue-T
Copy link

Josue-T commented Jun 5, 2020

Have you upgraded synapse since the room was created?

The room !XqBunHwQIXUiqCaoxq:matrix.org was created probably since many year ago so, yes I upgraded synapse since this.

@erikjohnston
Copy link
Member

Sorry, that was badly phrased, I guess I have you upgraded since the first time that room ID appeared on your server (i.e. since you first joined/were invited)?

Basically, not too long ago we added some background updates to populate a new rooms.room_version column in the database, and I'm wondering if that somehow went wrong.

@Josue-T
Copy link

Josue-T commented Jun 5, 2020 •
edited
Loading

I guess I have you upgraded since the first time that room ID appeared on your server (i.e. since you first joined/were invited)?

Yes

Is it a solution to just add this room in the rooms table ?

@erikjohnston
Copy link
Member

Yeah, that should fix

@erikjohnston
Copy link
Member

This sounds similar to #6902

@Josue-T
Copy link

Josue-T commented Jun 8, 2020 •
edited
Loading

@brandoncurtis I've tried to fix the issue as described here #7615 (comment) . I've no more warning but the bot still broken. Any idea ?
I've tried with mautrix-telegram and mautrix-whatsapp and I've the same issue.

@erikjohnston
Copy link
Member

@Josue-T Sorry for dropping the ball on this a bit. Did you add the room to the rooms table? If it's not working but you're no longer seeing the warning that sounds like a separate issue. Is it still broken?

@Josue-T
Copy link

Josue-T commented Aug 6, 2020

@Josue-T Sorry for dropping the ball on this a bit. Did you add the room to the rooms table? If it's not working but you're no longer seeing the warning that sounds like a separate issue. Is it still broken?

Hello, it's quite old so I don't really remember what I finally did. As I remember I dropped all problematic event as @brandoncurtis described and if fixed the issue.

@erikjohnston
Copy link
Member

Thanks! Sorry again for the delay. I'm going to close this for now then, we've fixed some bugs in this area around not correctly populating the rooms table recently. Please shout if it happens again though!

@madduck
Copy link

madduck commented Oct 19, 2022

I have an instance of this problem right now:

Exception: Room !gLEsEYdYNEBYbhPBYY:matrix.aventer.biz for event $150643241032TIbKb:matrix.aventer.biz is unknown

There are 3 events in the state_groups_state table:

state_group |                room_id                 |        type         |          state_key          |                event_id                
-------------+----------------------------------------+---------------------+-----------------------------+----------------------------------------
      250088 | !gLEsEYdYNEBYbhPBYY:matrix.aventer.biz | m.room.create       |                             | $14248996051100afgSi:matrix.org
      250088 | !gLEsEYdYNEBYbhPBYY:matrix.aventer.biz | m.room.member       | @REDACTED:matrix.aventer.biz | $1505733554113ScvTC:matrix.aventer.biz
      250088 | !gLEsEYdYNEBYbhPBYY:matrix.aventer.biz | m.room.power_levels |                             | $14757458252642826qqnrS:matrix.org
(3 rows)

I cannot really afford to wait to delete these events, but if the problem reoccurs, is there anything we can contribute or try?

@madduck
Copy link

madduck commented Oct 19, 2022

Removing the state_groups_state rows, as well as the events from the events and event_edges tables didn't fix the problem. Now I get the same error for a different event/room.

I've added the room to the rooms table, but the exception persists.

@madduck
Copy link

madduck commented Oct 19, 2022

I am also failing to get an appservice (maurix-signal) to work, and it may be related.

@DMRobertson
Copy link
Contributor

DMRobertson commented Oct 19, 2022 •
edited
Loading

Removing the state_groups_state rows, as well as the events from the events and event_edges tables didn't fix the problem. Now I get the same error for a different event/room.

I've added the room to the rooms table, but the exception persists.

@madduck You should NOT manually make edits to the Synapse database at all. Doing so without knowing what you're doing is highly unsafe and we cannot support anyone who has done so without our guidance.

Your best bet is to use the Delete room admin API with "purge": true and "block": false to remove your server from any problematic rooms, then try to rejoin them. (Ask in the Synapse Admins room if you need assistance using the admin API.) If the problem persists after doing so, please raise a new issue.

@madduck
Copy link

madduck commented Oct 19, 2022

@DMRobertson Thanks for that, that seems to have solved the problem (though the appservice still doesn't get sent any events). And yeah, I am aware that messing with the database is bad. Mind you, I did so only on a copy, would never do it on a live system.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@erikjohnston erikjohnston

Labels
z-bug (Deprecated Label)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@madduck @richvdh @brandoncurtis @Josue-T @erikjohnston @DMRobertson