Make e2e backup versions numeric in the DB #4113
Conversation
We were doing max(version) which does not do what we wanted on a column of type TEXT.
| @@ -236,6 +246,7 @@ def _get_e2e_room_keys_version_info_txn(txn): | |||
| ), | |||
| ) | |||
| result["auth_data"] = json.loads(result["auth_data"]) | |||
| result["version"] = str(result["version"]) | |||
There was a problem hiding this comment.
Can we not just keep this as an int?
There was a problem hiding this comment.
It's a string in the API, ie. when json encoded for the client it needs to be "2" rather than 2, so we need to stringify it at some point. I don't really mind whether we model them as int throughout the whole of synapse & convert to str at the last minute for marshalling or just keep them as ints at the database layer.
There was a problem hiding this comment.
Is there a reason to then not have them as strings in the database layer?
There was a problem hiding this comment.
It currently does a MAX() at the db layer to get the largest version, then increments it in python, which breaks when the column type is a string because "10" < "2" which is the bug this fixes.
There was a problem hiding this comment.
why does the API have it as a string when JSON has numbers?
There was a problem hiding this comment.
#e2e-dev:matrix.org has the API / spec discussion: this one was: https://matrix.to/#/!uewiilduiDRfPomIha:matrix.org/$15397878202555374QpBig:matrix.org
synapse/handlers/e2e_room_keys.py
Outdated
| @@ -138,7 +138,7 @@ def upload_room_keys(self, user_id, version, room_keys): | |||
| else: | |||
| raise | |||
|
|
|||
| if version_info['version'] != version: | |||
| if str(version_info['version']) != version: | |||
There was a problem hiding this comment.
looks like get_e2e_room_keys_version_info returns a str here anyway?
There was a problem hiding this comment.
(though it would be helpful if its docstring actually said as much)
synapse/storage/prepare_database.py
Outdated
| @@ -25,7 +25,7 @@ | |||
|
|
|||
| # Remember to update this number every time a change is made to database | |||
| # schema files, so the users will be informed on server restarts. | |||
| SCHEMA_VERSION = 51 | |||
| SCHEMA_VERSION = 52 | |||
There was a problem hiding this comment.
someone (@erikjohnston ?) remind me what our policy is for when we have to bump this? is it written down somewhere?
There was a problem hiding this comment.
My reason for bumping it here is that the original e2e room keys tables were added in 51 and this changes obviously needs to happen after that.
There was a problem hiding this comment.
well, that seems a good reason. Another good reason is that we are already up to 52 in develop; I'm not sure why it is being shown as a change here.
There was a problem hiding this comment.
ah, probably because I branched off before that
|
There is now a test: matrix-org/sytest#523 |
…rsions_are_numbers
Features -------- - Include flags to optionally add `m.login.terms` to the registration flow when consent tracking is enabled. ([\#4004](#4004), [\#4133](#4133), [\#4142](#4142), [\#4184](#4184)) - Support for replacing rooms with new ones ([\#4091](#4091), [\#4099](#4099), [\#4100](#4100), [\#4101](#4101)) Bugfixes -------- - Fix exceptions when using the email mailer on Python 3. ([\#4095](#4095)) - Fix e2e key backup with more than 9 backup versions ([\#4113](#4113)) - Searches that request profile info now no longer fail with a 500. ([\#4122](#4122)) - fix return code of empty key backups ([\#4123](#4123)) - If the typing stream ID goes backwards (as on a worker when the master restarts), the worker's typing handler will no longer erroneously report rooms containing new typing events. ([\#4127](#4127)) - Fix table lock of device_lists_remote_cache which could freeze the application ([\#4132](#4132)) - Fix exception when using state res v2 algorithm ([\#4135](#4135)) - Generating the user consent URI no longer fails on Python 3. ([\#4140](#4140), [\#4163](#4163)) - Loading URL previews from the DB cache on Postgres will no longer cause Unicode type errors when responding to the request, and URL previews will no longer fail if the remote server returns a Content-Type header with the chartype in quotes. ([\#4157](#4157)) - The hash_password script now works on Python 3. ([\#4161](#4161)) - Fix noop checks when updating device keys, reducing spurious device list update notifications. ([\#4164](#4164)) Deprecations and Removals ------------------------- - The disused and un-specced identicon generator has been removed. ([\#4106](#4106)) - The obsolete and non-functional /pull federation endpoint has been removed. ([\#4118](#4118)) - The deprecated v1 key exchange endpoints have been removed. ([\#4119](#4119)) - Synapse will no longer fetch keys using the fallback deprecated v1 key exchange method and will now always use v2. ([\#4120](#4120)) Internal Changes ---------------- - Fix build of Docker image with docker-compose ([\#3778](#3778)) - Delete unreferenced state groups during history purge ([\#4006](#4006)) - The "Received rdata" log messages on workers is now logged at DEBUG, not INFO. ([\#4108](#4108)) - Reduce replication traffic for device lists ([\#4109](#4109)) - Fix `synapse_replication_tcp_protocol_*_commands` metric label to be full command name, rather than just the first character ([\#4110](#4110)) - Log some bits about room creation ([\#4121](#4121)) - Fix `tox` failure on old systems ([\#4124](#4124)) - Add STATE_V2_TEST room version ([\#4128](#4128)) - Clean up event accesses and tests ([\#4137](#4137)) - The default logging config will now set an explicit log file encoding of UTF-8. ([\#4138](#4138)) - Add helpers functions for getting prev and auth events of an event ([\#4139](#4139)) - Add some tests for the HTTP pusher. ([\#4149](#4149)) - add purge_history.sh and purge_remote_media.sh scripts to contrib/ ([\#4155](#4155)) - HTTP tests have been refactored to contain less boilerplate. ([\#4156](#4156)) - Drop incoming events from federation for unknown rooms ([\#4165](#4165))
We were doing max(version) which does not do what we wanted
on a column of type TEXT.
Fixes #4169