This repository has been archived by the owner on Apr 26, 2024. It is now read-only.
500 errors when leaving or kicking in a room #16191
Labels
A-Database
DB stuff like queries, migrations, new/remove columns, indexes, unexpected entries in the db
A-Leave
Leaving a room
S-Minor
Blocks non-critical functionality, workarounds exist.
T-Defect
Bugs, crashes, hangs, security vulnerabilities, or other reported issues.
X-Needs-Info
This issue is blocked awaiting information from the reporter
Comments
DMRobertson
changed the title
Sounds like Relevant source: synapse/synapse/handlers/message.py Lines 1241 to 1265 in 40901af This suggests that your What does the following SQL give you? |
DMRobertson
added
S-Minor
This is likely element-hq/element-web#24392
I'm afraid there's nothing in your logs that demonstrates this; can you see any errors or warnings corresponding to federated rooms? E.g. grep logs for |
DMRobertson
added
the
X-Needs-Info
|
couldnt find any events with that grep the output was the last minute of logs after I've went into matrix.org rooms and scrolled up (which still haven't gotten any new messages through since thursday the 17th) |
the output of the SQL query is I didn't think I got any errors when porting from sqlite3 but maybe I missed something idk is there a way to like factory reset it or something? |
Try
If this was the case, I doubt Synapse would start; double check what tables you can see by running |
this is the output
sorry haven't really had to go into any of my db containers so this is kind of new to me, I've |
|
I tried to send a message to the synapse admin room and it failed this is the logs: |
You haven't managed to get into the database correctly. I think you'll have to read up on how to do this with the postgres container that you're running. E.g. it might be that you need to set a search path, or login as the right role. I think the best place to get advice is a) the documentation for your postgres docker image, and b) the Synapse Admins room. (Since it seems that your homeserver is inoperable at the moment, you can sign up for an account on the Matrix.org homeserver in order to reach the support room.) |
managed to figure it out (just had to switch db sry big noob) here is the output of and the output of the original SQL query is now |
|
That suggests that Synapse doesn't know about the latest set of events in that room. Again, this shouldn't normally happen. My best guess is that the porting from sqlite to postgres went wrong. Do you have any logs from the port_db script? Do you have a backup of the sqlite DB that you can rerun the query on? |
Are the logs from the port_db script saved anywhere or would I have had to save it after it ran cos if that the case then I dont have any logs, however I do still have the homeserver.db and the homeserver.db.snapshot files |
Unfortunately not. In that case, can you load up the most recent backup of the database ( |
Sry I'm probably doing something obviously wrong this is the output (I've done |
is there a way for me to start again with a blank database? |
|
Ahh, I've just noticed a mistake of mine. The room ID in the query has SELECT e.event_id FROM event_forward_extremities AS f
INNER JOIN events AS e USING (event_id)
WHERE f.room_id = '!sRHNtYGJcSmVtNOtMu:matrix.tazy.xyz'
ORDER BY e.depth DESC
LIMIT 10;on postgres please? |
still give the same output: however doing the same query on the sqlite3 homeserver.db I get the following output: Thanks for helping me with this |
|
That's surprising. It suggests that your sqlite DB knew about the room in question, but your postgres DB doesn't. Have you tried to purge this room using the admin API after porting to postgres? Can you additionally run SELECT count(*) FROM current_state_events
WHERE room_id = '!sRHNtYGJcSmVtNOtMu:matrix.tazy.xyz' AND type = 'm.room.member' AND state_key LIKE '%:matrix.tazy.xyz';against postgres and sqlite please? |
No im unsure how to do that ill give it a go but out rn managed to get the sql qurries run tho I ran the following curl command for the admin api
output for sqlite3 ill edit when i get back to pc |
|
It sounds like the porting process errored or didn't finish somehow then. You can try purging the room (https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#version-2-new-version) which will delete everything your server knows about this room. Be sure you want to do this: if there are no other homeservers in that room, that room's history will be gone forever. |
Thanks tbh I haven't really used matrix that much I did join a few rooms but haven't really sent that many messages to them I was hoping to migrate from discord but none of my friends want to so its just kind of sat there for a while too. I went back and edited the last comment I'll repeat I ran the following curl command for the admin api |
|
wrap the URL in quotes. |
edit: I don't think the purge goes at the end of the url but in the header so I've modified to I made a python script using Matrix Tutorial #5 on youtube import requests
host = "<Internal-IP>:8008"
access_token = "Admin User Access Token"
headers = {
"Authorization": "Bearer "+access_token,
"purge": "true"
}
r = requests.delete("http://"+host+"/_synapse/admin/v2/rooms/!GtIfdsfQtQIgbQSxwJ:archlinux.org/", headers=headers)the output of this is BTW I'm using the internal room ID for the archlinux room |
import requests
host = "127.0.0.1:8008"
access_token = ""
headers = {
"Authorization": "Bearer "+access_token,
}
body = {
"purge": True
}
r = requests.delete("http://"+host+"/_synapse/admin/v2/rooms/!GtIfdsfQtQIgbQSxwJ:archlinux.org", headers=headers, json=body)
print(r.json())This python script seems to have worked got this response |
|
I can't send any messages to any local rooms This is logs from an attempt to send a message to a local room with my main admin account when I tried to purge the sememcraft room (dm between my main tazy account and sememcraft account) I get the following error |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Description
Constant failures when doing things like trying to leave rooms, removing users from rooms, notifications not going away after viewing messages, and federated rooms not loading consistently I get the following error:
and if you go to the link this is the raw data
Also federated rooms like the synapse admin room doesn't federate properly like ill open element and it'll load and I click on that room and the last message was from a week ago
Steps to reproduce
Removing users from rooms
Trying to leave rooms
Homeserver
https://matrix.tazy.xyz
Synapse Version
{"server_version":"1.90.0","python_version":"3.11.4"}
Installation Method
Docker (matrixdotorg/synapse)
Database
postgresql:latest
Workers
Single process
Platform
I have synapse running in a docker host with multiple other containers mostly activity pub instances, lemmy, peertube, mastodon but also wireguard and jellyfin, I have 6Gb RAM dedicated and 4cpus. I'm also running a discord bridge (thats kind of working some of the time) and a signal bridge I also attempted slinding sync but i dont think its working it infinitely syncs when I try to sign in on element x android. I'm accessing synapse through Nginx Proxy Manager with the following custom config in the advanced tab:
there is no custom config for the custom locations tab within nginx proxy manager just the advanced tab
Configuration
I haven't really done much with the config file other than the bridges previously mentioned.
Relevant log output
2023-08-26 22:39:25,320 - synapse.access.http.8008 - 465 - INFO - PUT-145812 - 162.158.174.153 - 8008 - {gnulinux.club} Processed request: 1.242sec/0.000sec (0.000sec, 0.000sec) (0.000sec/0.002sec/2) 11B 200 "PUT /_matrix/federation/v1/send/1692614275113 HTTP/1.1" "Synapse/1.89.0" [0 dbevts] 2023-08-26 22:39:26,098 - synapse.util.caches.response_cache - 266 - INFO - GET-145815 - [sync]: using completed cached result for [(@sememcraft:matrix.tazy.xyz, 0, 's651813_17764_2_1505734_11581_222_8653_2237374_0_1', '{"room":{"rooms":["!sRHNtYGJcSmVtNOtMu:matrix.tazy.xyz"],"timeline":{"types":["m.room.message"],"not_senders":["@sememcraft:matrix.tazy.xyz"]}}}', False, 'GCCDYUWTJA', None)] 2023-08-26 22:39:26,098 - synapse.access.http.8008 - 465 - INFO - GET-145815 - 162.158.74.52 - 8008 - {@sememcraft:matrix.tazy.xyz} Processed request: 0.002sec/0.000sec (0.003sec, 0.000sec) (0.000sec/0.000sec/0) 254B 200 "GET /_matrix/client/r0/sync?filter=%7B%22room%22%3A%7B%22rooms%22%3A%5B%22%21sRHNtYGJcSmVtNOtMu%3Amatrix.tazy.xyz%22%5D%2C%22timeline%22%3A%7B%22types%22%3A%5B%22m.room.message%22%5D%2C%22not_senders%22%3A%5B%22%40sememcraft%3Amatrix.tazy.xyz%22%5D%7D%7D%7D&since=s651813_17764_2_1505734_11581_222_8653_2237374_0_1&access_token=<redacted> HTTP/1.1" "Java/17.0.8" [0 dbevts] 2023-08-26 22:39:26,703 - synapse.http.server - 134 - ERROR - POST-145817 - Failed handle request via 'RoomMembershipRestServlet': <XForwardedForRequest at 0x7f8f2c376850 method='POST' uri='/_matrix/client/r0/rooms/!sRHNtYGJcSmVtNOtMu%3Amatrix.tazy.xyz/kick' clientproto='HTTP/1.1' site='8008'> Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/synapse/http/server.py", line 320, in _async_render_wrapper callback_return = await self._async_render(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/http/server.py", line 532, in _async_render callback_return = await raw_callback_return ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/rest/client/room.py", line 1072, in on_POST return await self._do(request, requester, room_id, membership_action, None) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/rest/client/room.py", line 1045, in _do await self.room_member_handler.update_membership( File "/usr/local/lib/python3.11/site-packages/synapse/handlers/room_member.py", line 627, in update_membership result = await self.update_membership_locked( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/handlers/room_member.py", line 1088, in update_membership_locked return await self._local_membership_update( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/handlers/room_member.py", line 436, in _local_membership_update ) = await self.event_creation_handler.create_event( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/handlers/message.py", line 711, in create_event event, unpersisted_context = await self.create_new_client_event( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/util/metrics.py", line 113, in measured_func r = await func(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/handlers/message.py", line 1264, in create_new_client_event builder.type == EventTypes.Create or prev_event_ids ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: Attempting to create a non-m.room.create event with no prev_events 2023-08-26 22:39:26,704 - synapse.access.http.8008 - 465 - INFO - POST-145817 - 172.69.79.219 - 8008 - {@tazy:matrix.tazy.xyz} Processed request: 0.009sec/0.000sec (0.001sec, 0.000sec) (0.001sec/0.005sec/4) 55B 500 "POST /_matrix/client/r0/rooms/!sRHNtYGJcSmVtNOtMu%3Amatrix.tazy.xyz/kick HTTP/1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/116.0" [0 dbevts] 2023-08-26 22:39:27,614 - synapse.federation.transport.server.federation - 105 - INFO - PUT-145818 - Received txn 1692289308920 from startuptools.org. (PDUs: 0, EDUs: 1) 2023-08-26 22:39:27,620 - synapse.access.http.8008 - 465 - INFO - PUT-145818 - 172.71.102.173 - 8008 - {startuptools.org} Processed request: 0.009sec/0.000sec (0.000sec, 0.000sec) (0.000sec/0.006sec/3) 11B 200 "PUT /_matrix/federation/v1/send/1692289308920 HTTP/1.1" "Synapse/1.90.0" [0 dbevts] 2023-08-26 22:39:28,097 - synapse.util.caches.response_cache - 266 - INFO - GET-145819 - [sync]: using completed cached result for [(@sememcraft:matrix.tazy.xyz, 0, 's651813_17764_2_1505734_11581_222_8653_2237374_0_1', '{"room":{"rooms":["!sRHNtYGJcSmVtNOtMu:matrix.tazy.xyz"],"timeline":{"types":["m.room.message"],"not_senders":["@sememcraft:matrix.tazy.xyz"]}}}', False, 'GCCDYUWTJA', None)] 2023-08-26 22:39:28,098 - synapse.access.http.8008 - 465 - INFO - GET-145819 - 162.158.74.52 - 8008 - {@sememcraft:matrix.tazy.xyz} Processed request: 0.002sec/0.000sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 254B 200 "GET /_matrix/client/r0/sync?filter=%7B%22room%22%3A%7B%22rooms%22%3A%5B%22%21sRHNtYGJcSmVtNOtMu%3Amatrix.tazy.xyz%22%5D%2C%22timeline%22%3A%7B%22types%22%3A%5B%22m.room.message%22%5D%2C%22not_senders%22%3A%5B%22%40sememcraft%3Amatrix.tazy.xyz%22%5D%7D%7D%7D&since=s651813_17764_2_1505734_11581_222_8653_2237374_0_1&access_token=<redacted> HTTP/1.1" "Java/17.0.8" [0 dbevts] 2023-08-26 22:39:29,157 - synapse.federation.transport.server.federation - 105 - INFO - PUT-145820 - Received txn 1692305566051 from matrix.elgatopanzon.io. (PDUs: 0, EDUs: 1) 2023-08-26 22:39:29,161 - synapse.access.http.8008 - 465 - INFO - PUT-145820 - 162.158.87.46 - 8008 - {matrix.elgatopanzon.io} Processed request: 0.004sec/0.000sec (0.000sec, 0.000sec) (0.000sec/0.002sec/2) 11B 200 "PUT /_matrix/federation/v1/send/1692305566051 HTTP/1.1" "Synapse/1.90.0" [0 dbevts] 2023-08-26 22:39:29,339 - synapse.access.http.8008 - 465 - INFO - GET-145763 - 162.158.74.93 - 8008 - {@sememcraft:matrix.tazy.xyz} Processed request: 30.002sec/0.000sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 254B 200 "GET /_matrix/client/r0/sync?filter=1&timeout=30000&since=s651813_17764_2_1505734_11581_222_8653_2237374_0_1 HTTP/1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36" [0 dbevts] 2023-08-26 22:39:30,102 - synapse.util.caches.response_cache - 266 - INFO - GET-145823 - [sync]: using completed cached result for [(@sememcraft:matrix.tazy.xyz, 0, 's651813_17764_2_1505734_11581_222_8653_2237374_0_1', '{"room":{"rooms":["!sRHNtYGJcSmVtNOtMu:matrix.tazy.xyz"],"timeline":{"types":["m.room.message"],"not_senders":["@sememcraft:matrix.tazy.xyz"]}}}', False, 'GCCDYUWTJA', None)] 2023-08-26 22:39:30,103 - synapse.access.http.8008 - 465 - INFO - GET-145823 - 162.158.74.52 - 8008 - {@sememcraft:matrix.tazy.xyz} Processed request: 0.003sec/0.000sec (0.003sec, 0.000sec) (0.000sec/0.000sec/0) 254B 200 "GET /_matrix/client/r0/sync?filter=%7B%22room%22%3A%7B%22rooms%22%3A%5B%22%21sRHNtYGJcSmVtNOtMu%3Amatrix.tazy.xyz%22%5D%2C%22timeline%22%3A%7B%22types%22%3A%5B%22m.room.message%22%5D%2C%22not_senders%22%3A%5B%22%40sememcraft%3Amatrix.tazy.xyz%22%5D%7D%7D%7D&since=s651813_17764_2_1505734_11581_222_8653_2237374_0_1&access_token=<redacted> HTTP/1.1" "Java/17.0.8" [0 dbevts] 2023-08-26 22:39:30,753 - synapse.federation.transport.server.federation - 105 - INFO - PUT-145824 - Received txn 1692878711860 from cqmn.sh. (PDUs: 0, EDUs: 1) 2023-08-26 22:39:30,760 - synapse.access.http.8008 - 465 - INFO - PUT-145824 - 198.41.231.19 - 8008 - {cqmn.sh} Processed request: 0.008sec/0.000sec (0.000sec, 0.000sec) (0.000sec/0.006sec/2) 11B 200 "PUT /_matrix/federation/v1/send/1692878711860 HTTP/1.1" "Synapse/1.90.0" [0 dbevts] 2023-08-26 22:39:30,835 - synapse.crypto.keyring - 687 - INFO - PerspectivesKeyFetcher-8985 - Requesting keys [_FetchKeyRequest(server_name='matrix.saltyoldgeek.com', minimum_valid_until_ts=1693089570834, key_ids=['ed25519:a_Ubco'])] from notary server matrix.org 2023-08-26 22:39:31,062 - synapse.federation.transport.server.federation - 105 - INFO - PUT-145826 - Received txn 1692791195543 from aidoru.agency. (PDUs: 0, EDUs: 1) 2023-08-26 22:39:31,066 - synapse.access.http.8008 - 465 - INFO - PUT-145826 - 172.71.126.57 - 8008 - {aidoru.agency} Processed request: 0.004sec/0.000sec (0.000sec, 0.000sec) (0.000sec/0.002sec/2) 11B 200 "PUT /_matrix/federation/v1/send/1692791195543 HTTP/1.1" "Synapse/1.90.0" [0 dbevts]Anything else that would be useful to know?
I recently migrated from SQLite3 to postgres so I'm not sure if I fucked up with that and if I have how would I fix it?
The text was updated successfully, but these errors were encountered: