This repository has been archived by the owner on Apr 26, 2024. It is now read-only.
synapse.api.errors.StoreError: 404: No row found during background process rotate_notifs
#14454
Labels
A-Push
Issues related to push/notifications
O-Uncommon
Most users are unlikely to come across this or unexpected workflow
S-Major
Major functionality / product severely impaired, no satisfactory workaround.
T-Defect
Bugs, crashes, hangs, security vulnerabilities, or other reported issues.
X-Needs-Info
This issue is blocked awaiting information from the reporter
Comments
|
Do you have purging enabled on your homeserver? |
|
No, I don't have retention/purging enabled. |
|
@lsrmg that stack trace doesn't seem to match v1.71.0, in that the line numbers don't line up. Is it possible the running synapse was different from the one that was installed? Secondly, that stack trace is for a background process, which while not ideal shouldn't cause problems with |
erikjohnston
added
X-Needs-Info
|
Thank you for you quick answer! I send you the full logs to @erikj:jki.re. You're right, I restored the working state and must have copied the wron snipped. Today I installed 1.71.0 and secured the right log: And the error on the client-side (Element 1.11.14): |
|
Same issue present in synapse 1.72 for me |
DMRobertson
changed the title
rotate_notifs
DMRobertson
changed the title
rotate_notifssynapse.api.errors.StoreError: 404: No row found during background process rotate_notifs
|
@lsrmg do you have logs? |
|
I took a look at the logs you sent my teammate and while I do see the error from the background process, I am not seeing anything related to |
|
Looking at the debug logs provided, I do see Looking closer, it looks like It appears that the query that is failing is with the message which afaict originates from this chunk of code here: which seems to have been added in #13877. @clokep, I think this is part of the threading work? You are more familiar with it than I, do you have any thoughts on what might be the issue? |
|
Also just noting that the debug logs I looked at were from Synapse v1.72.0, not 1.71.0. |
It was not part of threading -- this process existed earlier. (Threading just made it a bit more complicated!)
I've only seen this be an issue when message retention is enabled. I'm not sure beyond that. |
|
I've stared at this a bit and can't see why the row in @lsrmg which database engine are you using (Postgres or SQLite)? Have you done anything 'unusual' to the database at some point in time (e.g. restored from a backup; ported from SQLite to Postgres using the portdb script; etc). Please would you be able to run for me? . |
|
@reivilibre I'm using Postgres version 13.8 At some point in time I migrated the DB between two servers (same synapse version) via db_dump and rebuild the indexes (from this). Output from |
|
Is that output being cut off for some reason (e.g. it's too tall)? I would expect many more entries, particularly recent ones. I'm also surprised that none of the names contain a One option is you can use |
|
You're right, it was cut off. Thanks for the tip! |
reivilibre
removed
the
X-Needs-Info
|
Can you try running the following sql and report back on whether that fixes things? BEGIN TRANSACTION;
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ;
SELECT MAX(stream_ordering) FROM event_push_summary;
INSERT INTO event_push_summary_stream_ordering(lock, stream_ordering) VALUES ('X', (SELECT MAX(stream_ordering) FROM event_push_summary));
COMMIT; |
squahtx
added
A-Push
|
Thanks for getting into it. I ran the sql and the answer was: |
|
In that case, please try: INSERT INTO event_push_summary_stream_ordering(lock, stream_ordering) VALUES('X', 0);How long has your Synpase deployment been around, out of interest? |
|
That sql worked. We are using this synapse since early 2020. The error is gone. I rolled back the server to a state before the sql statement and the error imminently appeared again. I also tested updating to 1.73.0 (after sql), but the "404: No row found" it still present for /sync-Requests. Shutting the services down I noticed exceptions for every worker that I didn't noticed before: Also there was an other error I've never seen before: I don't know if its related to this problem, but I also got this error (before and after sending your sql statement): Thank you for your help! |
If possible, could you post the new, full stack trace for that error?
This one is expected while shutting down a deployment with workers.
This one's a known issue. It should only affect Prometheus metrics.
Is your |
I'm sorry but due to vacation this is not possible until mid-February... I'll get back to you.
Yes it is. (I don‘t use federation, the server is running in a company network only) |
|
Today I had the chance to update to v1.75.0rc1 and collect the logs. I noticed that an issue #14669 resolved a problem that looks like mine, but for me the error still is present after updating. On client-side the /sync request isn't working and preventing me to update since 1.71.0. Before the error is also in the logs but don't prevent clients from syncing. |
This corresponds to synapse/synapse/storage/databases/main/event_push_actions.py Lines 1480 to 1486 in 24a97b3 Which suggests your synapse/synapse/storage/schema/main/delta/40/event_push_summary.sql Lines 30 to 38 in 0d1d3e0 And this row is never deleted; it is only ever updated here: synapse/synapse/storage/databases/main/event_push_actions.py Lines 1802 to 1805 in 24a97b3 But what's even more confusing is that there should be a row here, because you ran some SQL to make one according to this comment here. To confirm this, can you run \d event_push_summary_stream_ordering
SELECT count(*) FROM event_push_summary_stream_ordering; |
I rolled back the server after trying to insert the SQL, because /sync wasn‘t working even with the row inserted. I‘m not sure what’s going on and why since synapse v1.71.0 the /sync endpoint refused to work for me. Information: In the meantime I had to enable message-retention on the server for privacy reasons. |
|
Good news: The problem has solved, but I don't fully understand what happend. I have inserted the mentioned line into the "event_push_summary_stream_ordering"-table and noticed a higher CPU-load on my background-worker. I waited 10mins for the load to normalize and installed the synapse 1.75 update. The last time i tested this approach unsuccessful, I updated synapse and than inserted the line into the table afterwards and instantly tested the client where /sync was failing with "no row found" and I rolled back the server. In the end I don't know if I just had to wait some time. Thank you all for helping out! I guess @squahtx and @DMRobertson where right with pointing to event_push_summary_stream_ordering. Or 1.75.0 fixed something else failing for me. @mi-gri If you got the same Problem try inserting the row mentioned here |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Description
For some versions of matrix-synapse we're getting the following error in our homeserver-logs. We didn't noticed any issues while using the service at all, everythin was running fine except for the presence-status since the introduction of incremental
/syncrequests some versions ago.With the new synapse version 1.71.0 the
/syncrequests from all clients keep failing with the message "No row found".It seems to me that there are some null-values in the database where they not ment to be.
Steps to reproduce
Homeserver
on-premise/local network
Synapse Version
1.71.0
Installation Method
Debian packages from packages.matrix.org
Platform
Debian 11
Relevant log output
Anything else that would be useful to know?
There are two issues like this in #14219 and #14300.
The text was updated successfully, but these errors were encountered: