From 053ecae4db7c51c7fa3d56f5ea2b1289d73fa10d Mon Sep 17 00:00:00 2001 From: Andrew Conrad Date: Tue, 14 Nov 2017 15:09:50 -0600 Subject: [PATCH 01/24] Mention SyTest in the README, after Development Signed-off-by: Andrew Conrad --- README.rst | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/README.rst b/README.rst index 9da8c7f7a8d7..cda0794e5629 100644 --- a/README.rst +++ b/README.rst @@ -883,6 +883,17 @@ This should end with a 'PASSED' result:: PASSED (successes=143) +Running the Integration Tests +============================= + +Synapse is accompanied by `SyTest `_, +a Matrix homeserver integration testing suite, which uses HTTP requests to +access the API as a Matrix client would. It is able to run Synapse directly from +the source tree, so installation of the server is not required. + +Testing with SyTest is recommended for verifying that changes related to the +Client-Server API are functioning correctly. See the `installation instructions +`_ for details. Building Internal API Documentation =================================== From 4102468da98543ddb899857b67f5666ae4cfae92 Mon Sep 17 00:00:00 2001 From: Jeremy Cline Date: Tue, 16 Jan 2018 16:13:28 -0500 Subject: [PATCH 02/24] Note that Synapse is available in Fedora Signed-off-by: Jeremy Cline --- README.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.rst b/README.rst index ef1eed45841f..ef27b92ddd1a 100644 --- a/README.rst +++ b/README.rst @@ -354,6 +354,10 @@ https://matrix.org/docs/projects/try-matrix-now.html (or build your own with one Fedora ------ +Synapse is in the Fedora repositories as ``matrix-synapse``:: + + sudo dnf install matrix-synapse + Oleg Girko provides Fedora RPMs at https://obs.infoserver.lv/project/monitor/matrix-synapse From cb2c7c0669c7f4d4cc41d65a7f2328589fa04fc5 Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Wed, 14 Mar 2018 16:09:20 +0000 Subject: [PATCH 03/24] Update CHANGES.rst WIP, need to add most recent PRs --- CHANGES.rst | 91 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 91 insertions(+) diff --git a/CHANGES.rst b/CHANGES.rst index a7ed49e105a5..0bcd5a7177a5 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -8,6 +8,97 @@ This release also begins the process of renaming a number of the metrics reported to prometheus. See `docs/metrics-howto.rst `_. +Changes in synapse v0.27.0-rc1 (2018-03-14) +=========================================== + +Features: + * Add support for a remote media repository backed by S3 (PR 2867, 2777, 2783, 2789, 2791, 2804, 2812,2814, 2857, 2868, 2767) + * Add shiney new purge API. New implementation is:- + * Faster + * Supports clearing by timestamp + * Supports deleting of local events + * Is transactional +(PR #2858,2867,2882, 2946,2962,2943) + + * Let homeservers specify a whitelist for the format of 3PIDs that users are allowed to register with or add to their HS accounts. (PR #2813) + * Add /room/{id}/event/{id} to synapse (PR #2766) + * Add an admin route to get all the media in a room (PR #2818) Thanks to @turt2live! + * Add federation_domain_whitelist option (PR #2820,2821) + + +Changes: + + * Major refactor to move computation out of the main process and into a series of worker processes with a view to taking advantage of multicore machines. New workers are:- + * Event Creation + * Push Actions + +With more to follow + + (PR #2892, #2893, #2894, #2896, #2897, #2898, #2899, #2900, #2901, #2902, #2903, #2904, #2913, #2920, #2921, #2922, #2923, #2924, #2925, #2926, #2947, #2847, #2854, #2872, #2873, #2874, #2928, #2929, #2934, #2856) + + * Use StateResolutionHandler to resolve state in persist_events (PR #2864, #2871, #2802, #2835, #2836, #2841, #2842, #2849) + * Adapt the default config to bind on both IPv4 and IPv6 on all platforms (PR #2435) Thanks to @silkeh! + * Allow use of higher versions of saml2 (PR #2695) Thanks to @okurz! + * Better logging when login can't find a 3pid (PR #2744) + * add ?ts massaging for ASes (PR #2754) + * Remove 'verbosity'/'log_file' from generated cfg (PR #2755) + * Make indentation of generated log config consistent (PR #2762) + * Remove dead code related to default thumbnails (PR #2764) + * Update http request metrics before calling servlet (PR #2770) + * Do bcrypt hashing in a background thread (PR #2773) + * When using synctl with workers, don't start the main synapse automatically (PR #2774) + * Make Counter render floats (PR #2778) + * Store state groups separately from events (PR #2784) + * Reorganise request and block metrics (PR #2785) + * Metrics for number of RDATA commands received (PR #2786) + * Metrics for events processed in appservice and fed sender (PR #2787) + * Optimise LoggingContext creation and copying (PR #2792) + * Track db txn time in millisecs (PR #2793) + * Track DB scheduling delay per-request (PR #2794, #2795) + * Sanity checking for user ids (PR #2797) + * better exception logging in callbackmetrics (PR #2809) + * Fix bugs in block metrics (PR #2810) + * Add some comments about the reactor tick time metric (PR #2816) + * Use a connection pool for the SimpleHttpClient (PR #2817) + * Remove unused/bitrotted MemoryDataStore (PR #2828) + * Make it possible to run tests against postgres (PR #2829) + * Factor out get_db_conn to HomeServer base class (PR #2830) + * Logging and metrics for the http pusher (PR #2833) + * Improve exception handling in persist_event (PR #2834) + * montoring metrics for number of cache evictions (PR #2844) + * Update pynacl dependency to 1.2.1 or higher (PR #2888) Thanks to @bachp! + * Remove ability for AS users to call /events and /sync (PR #2948) + * use bcrypt.checkpw (PR #2949) Thanks to @krombel! + * Factor run_in_background out from preserve_fn (PR #2961) + * Add a metric which increments when a request is received (PR #2965) + * Improve caching for read_marker API (PR #2927) + + +Synapse 0.27.0 begins the process of rationalising metric names. To enable a graceful migration path, this release just adds new names for the metrics being renamed. A future release will remove the old ones. For more info see [here](https://github.com/matrix-org/synapse/blob/develop/docs/metrics-howto.rst#block-and-response-metrics-renamed-for-0270) + + + +Bug fixes: + + * synapse/config/password_auth_providers: Fixed bracket typo (PR #2683) Thanks to @seckrv! + * Check missing fields in event_from_pdu_json (PR #2745) + * Fix templating error with unban permission message (PR #2761) Thanks to @turt2live! + * Fix flaky test_rooms UTs (PR #2765) + * Fix publicised groups GET API (singular) over federation (PR #2772) + * Fix a logcontext leak in persist_events (PR #2790) + * Fix 'NoneType' object has no attribute 'writeHeaders' (PR #2796) + * fix SQL when searching all users (PR #2803) + * Fix server 500 on public rooms call when no rooms exist (PR #2827) + * Fix SQL for user search (PR #2831) + * Fix sql error in quarantine_media (PR #2837) + * Handle url_previews with no content-type (PR #2845) + * Add missing yield during 3pid signature checks (PR #2933) + * Fix race in sync when joining room (PR #2944) + * Fix slow event search, switch back from GIST to GIN indexes (PR #2769, #2848) + * Fix scary-looking dns resolution errors (PR #2838) + + + Changes in synapse v0.26.0 (2018-01-05) ======================================= From a492b17fe2cbbb10687d9a7cbe9eda67b0562656 Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Wed, 14 Mar 2018 16:18:09 +0000 Subject: [PATCH 04/24] Update CHANGES.rst clean formatting --- CHANGES.rst | 128 ++++++++++++++++++++++++---------------------------- 1 file changed, 60 insertions(+), 68 deletions(-) diff --git a/CHANGES.rst b/CHANGES.rst index 0bcd5a7177a5..4d1b4ba13c21 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -12,66 +12,58 @@ Changes in synapse v0.27.0-rc1 (2018-03-14) =========================================== Features: - * Add support for a remote media repository backed by S3 (PR 2867, 2777, 2783, 2789, 2791, 2804, 2812,2814, 2857, 2868, 2767) - * Add shiney new purge API. New implementation is:- +* Add support for a remote media repository backed by S3 (PR 2867, 2777, 2783, 2789, 2791, 2804, 2812,2814, 2857, 2868, 2767) +* Add shiney new purge API. New implementation is:- * Faster * Supports clearing by timestamp * Supports deleting of local events - * Is transactional -(PR #2858,2867,2882, 2946,2962,2943) + * Is transactional (PR #2858,2867,2882, 2946,2962,2943) - * Let homeservers specify a whitelist for the format of 3PIDs that users are allowed to register with or add to their HS accounts. (PR #2813) - * Add /room/{id}/event/{id} to synapse (PR #2766) - * Add an admin route to get all the media in a room (PR #2818) Thanks to @turt2live! - * Add federation_domain_whitelist option (PR #2820,2821) +* Let homeservers specify a whitelist for the format of 3PIDs that users are allowed to register with or add to their HS accounts. (PR #2813) +* Add /room/{id}/event/{id} to synapse (PR #2766) +* Add an admin route to get all the media in a room (PR #2818) Thanks to @turt2live! +* Add federation_domain_whitelist option (PR #2820,2821) Changes: - * Major refactor to move computation out of the main process and into a series of worker processes with a view to taking advantage of multicore machines. New workers are:- - * Event Creation - * Push Actions - -With more to follow - - (PR #2892, #2893, #2894, #2896, #2897, #2898, #2899, #2900, #2901, #2902, #2903, #2904, #2913, #2920, #2921, #2922, #2923, #2924, #2925, #2926, #2947, #2847, #2854, #2872, #2873, #2874, #2928, #2929, #2934, #2856) - - * Use StateResolutionHandler to resolve state in persist_events (PR #2864, #2871, #2802, #2835, #2836, #2841, #2842, #2849) - * Adapt the default config to bind on both IPv4 and IPv6 on all platforms (PR #2435) Thanks to @silkeh! - * Allow use of higher versions of saml2 (PR #2695) Thanks to @okurz! - * Better logging when login can't find a 3pid (PR #2744) - * add ?ts massaging for ASes (PR #2754) - * Remove 'verbosity'/'log_file' from generated cfg (PR #2755) - * Make indentation of generated log config consistent (PR #2762) - * Remove dead code related to default thumbnails (PR #2764) - * Update http request metrics before calling servlet (PR #2770) - * Do bcrypt hashing in a background thread (PR #2773) - * When using synctl with workers, don't start the main synapse automatically (PR #2774) - * Make Counter render floats (PR #2778) - * Store state groups separately from events (PR #2784) - * Reorganise request and block metrics (PR #2785) - * Metrics for number of RDATA commands received (PR #2786) - * Metrics for events processed in appservice and fed sender (PR #2787) - * Optimise LoggingContext creation and copying (PR #2792) - * Track db txn time in millisecs (PR #2793) - * Track DB scheduling delay per-request (PR #2794, #2795) - * Sanity checking for user ids (PR #2797) - * better exception logging in callbackmetrics (PR #2809) - * Fix bugs in block metrics (PR #2810) - * Add some comments about the reactor tick time metric (PR #2816) - * Use a connection pool for the SimpleHttpClient (PR #2817) - * Remove unused/bitrotted MemoryDataStore (PR #2828) - * Make it possible to run tests against postgres (PR #2829) - * Factor out get_db_conn to HomeServer base class (PR #2830) - * Logging and metrics for the http pusher (PR #2833) - * Improve exception handling in persist_event (PR #2834) - * montoring metrics for number of cache evictions (PR #2844) - * Update pynacl dependency to 1.2.1 or higher (PR #2888) Thanks to @bachp! - * Remove ability for AS users to call /events and /sync (PR #2948) - * use bcrypt.checkpw (PR #2949) Thanks to @krombel! - * Factor run_in_background out from preserve_fn (PR #2961) - * Add a metric which increments when a request is received (PR #2965) - * Improve caching for read_marker API (PR #2927) +* Major refactor to move computation out of the main process and into a series of worker processes with a view to taking advantage of multicore machines. New workers are XXXXXX, (PR #2892, #2893, #2894, #2896, #2897, #2898, #2899, #2900, #2901, #2902, #2903, #2904, #2913, #2920, #2921, #2922, #2923, #2924, #2925, #2926, #2947, #2847, #2854, #2872, #2873, #2874, #2928, #2929, #2934, #2856) +* Use StateResolutionHandler to resolve state in persist_events (PR #2864, #2871, #2802, #2835, #2836, #2841, #2842, #2849) +* Adapt the default config to bind on both IPv4 and IPv6 on all platforms (PR #2435) Thanks to @silkeh! +* Allow use of higher versions of saml2 (PR #2695) Thanks to @okurz! +* Better logging when login can't find a 3pid (PR #2744) +* add ?ts massaging for ASes (PR #2754) +* Remove 'verbosity'/'log_file' from generated cfg (PR #2755) +* Make indentation of generated log config consistent (PR #2762) +* Remove dead code related to default thumbnails (PR #2764) +* Update http request metrics before calling servlet (PR #2770) +* Do bcrypt hashing in a background thread (PR #2773) +* When using synctl with workers, don't start the main synapse automatically (PR #2774) +* Make Counter render floats (PR #2778) +* Store state groups separately from events (PR #2784) +* Reorganise request and block metrics (PR #2785) +* Metrics for number of RDATA commands received (PR #2786) +* Metrics for events processed in appservice and fed sender (PR #2787) +* Optimise LoggingContext creation and copying (PR #2792) +* Track db txn time in millisecs (PR #2793) +* Track DB scheduling delay per-request (PR #2794, #2795) +* Sanity checking for user ids (PR #2797) +* better exception logging in callbackmetrics (PR #2809) +* Fix bugs in block metrics (PR #2810) +* Add some comments about the reactor tick time metric (PR #2816) +* Use a connection pool for the SimpleHttpClient (PR #2817) +* Remove unused/bitrotted MemoryDataStore (PR #2828) +* Make it possible to run tests against postgres (PR #2829) +* Factor out get_db_conn to HomeServer base class (PR #2830) +* Logging and metrics for the http pusher (PR #2833) +* Improve exception handling in persist_event (PR #2834) +* montoring metrics for number of cache evictions (PR #2844) +* Update pynacl dependency to 1.2.1 or higher (PR #2888) Thanks to @bachp! +* Remove ability for AS users to call /events and /sync (PR #2948) +* use bcrypt.checkpw (PR #2949) Thanks to @krombel! +* Factor run_in_background out from preserve_fn (PR #2961) +* Add a metric which increments when a request is received (PR #2965) +* Improve caching for read_marker API (PR #2927) Synapse 0.27.0 begins the process of rationalising metric names. To enable a graceful migration path, this release just adds new names for the metrics being renamed. A future release will remove the old ones. For more info see [here](https://github.com/matrix-org/synapse/blob/develop/docs/metrics-howto.rst#block-and-response-metrics-renamed-for-0270) @@ -80,22 +72,22 @@ Synapse 0.27.0 begins the process of rationalising metric names. To enable a gra Bug fixes: - * synapse/config/password_auth_providers: Fixed bracket typo (PR #2683) Thanks to @seckrv! - * Check missing fields in event_from_pdu_json (PR #2745) - * Fix templating error with unban permission message (PR #2761) Thanks to @turt2live! - * Fix flaky test_rooms UTs (PR #2765) - * Fix publicised groups GET API (singular) over federation (PR #2772) - * Fix a logcontext leak in persist_events (PR #2790) - * Fix 'NoneType' object has no attribute 'writeHeaders' (PR #2796) - * fix SQL when searching all users (PR #2803) - * Fix server 500 on public rooms call when no rooms exist (PR #2827) - * Fix SQL for user search (PR #2831) - * Fix sql error in quarantine_media (PR #2837) - * Handle url_previews with no content-type (PR #2845) - * Add missing yield during 3pid signature checks (PR #2933) - * Fix race in sync when joining room (PR #2944) - * Fix slow event search, switch back from GIST to GIN indexes (PR #2769, #2848) - * Fix scary-looking dns resolution errors (PR #2838) +* synapse/config/password_auth_providers: Fixed bracket typo (PR #2683) Thanks to @seckrv! +* Check missing fields in event_from_pdu_json (PR #2745) +* Fix templating error with unban permission message (PR #2761) Thanks to @turt2live! +* Fix flaky test_rooms UTs (PR #2765) +* Fix publicised groups GET API (singular) over federation (PR #2772) +* Fix a logcontext leak in persist_events (PR #2790) +* Fix 'NoneType' object has no attribute 'writeHeaders' (PR #2796) +* fix SQL when searching all users (PR #2803) +* Fix server 500 on public rooms call when no rooms exist (PR #2827) +* Fix SQL for user search (PR #2831) +* Fix sql error in quarantine_media (PR #2837) +* Handle url_previews with no content-type (PR #2845) +* Add missing yield during 3pid signature checks (PR #2933) +* Fix race in sync when joining room (PR #2944) +* Fix slow event search, switch back from GIST to GIN indexes (PR #2769, #2848) +* Fix scary-looking dns resolution errors (PR #2838) From fb647164f2a008cdba56577ad0f41c4b4c1147d1 Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Wed, 14 Mar 2018 16:20:36 +0000 Subject: [PATCH 05/24] Update CHANGES.rst --- CHANGES.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/CHANGES.rst b/CHANGES.rst index 4d1b4ba13c21..53a97a769755 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -14,10 +14,10 @@ Changes in synapse v0.27.0-rc1 (2018-03-14) Features: * Add support for a remote media repository backed by S3 (PR 2867, 2777, 2783, 2789, 2791, 2804, 2812,2814, 2857, 2868, 2767) * Add shiney new purge API. New implementation is:- - * Faster - * Supports clearing by timestamp - * Supports deleting of local events - * Is transactional (PR #2858,2867,2882, 2946,2962,2943) + * Faster + * Supports clearing by timestamp + * Supports deleting of local events + * Is transactional (PR #2858,2867,2882, 2946,2962,2943) * Let homeservers specify a whitelist for the format of 3PIDs that users are allowed to register with or add to their HS accounts. (PR #2813) * Add /room/{id}/event/{id} to synapse (PR #2766) From c33c1ceddd5da8195b38059dce31255209075ba2 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 14 Mar 2018 11:09:08 -0600 Subject: [PATCH 06/24] OCD: Make the event_creator routes regex a code block All the others are code blocks, so this one should be to (currently it is a blockquote). Signed-off-by: Travis Ralston --- docs/workers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/workers.rst b/docs/workers.rst index dee04bbf3e47..80f8d2181ae2 100644 --- a/docs/workers.rst +++ b/docs/workers.rst @@ -230,7 +230,7 @@ file. For example:: ``synapse.app.event_creator`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Handles non-state event creation. It can handle REST endpoints matching: +Handles non-state event creation. It can handle REST endpoints matching:: ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/send From 5ccb57d3ff6e3affd32cc796f625b2378483d871 Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Wed, 14 Mar 2018 17:12:58 +0000 Subject: [PATCH 07/24] Update CHANGES.rst --- CHANGES.rst | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/CHANGES.rst b/CHANGES.rst index 53a97a769755..beac2117d3dd 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -1,16 +1,17 @@ Unreleased ========== + + +Changes in synapse v0.27.0-rc1 (2018-03-14) +=========================================== + synctl no longer starts the main synapse when using ``-a`` option with workers. A new worker file should be added with ``worker_app: synapse.app.homeserver``. This release also begins the process of renaming a number of the metrics reported to prometheus. See `docs/metrics-howto.rst `_. - -Changes in synapse v0.27.0-rc1 (2018-03-14) -=========================================== - Features: * Add support for a remote media repository backed by S3 (PR 2867, 2777, 2783, 2789, 2791, 2804, 2812,2814, 2857, 2868, 2767) * Add shiney new purge API. New implementation is:- @@ -27,7 +28,7 @@ Features: Changes: -* Major refactor to move computation out of the main process and into a series of worker processes with a view to taking advantage of multicore machines. New workers are XXXXXX, (PR #2892, #2893, #2894, #2896, #2897, #2898, #2899, #2900, #2901, #2902, #2903, #2904, #2913, #2920, #2921, #2922, #2923, #2924, #2925, #2926, #2947, #2847, #2854, #2872, #2873, #2874, #2928, #2929, #2934, #2856) +* Major refactor to move computation out of the main process and into a series of worker processes with a view to taking advantage of multicore machines. See more [here](https://github.com/matrix-org/synapse/blob/master/docs/workers.rst), (PR #2892-#2904, #2913, #2920 - #2926, #2947, #2847, #2854, #2872, #2873, #2874, #2928, #2929, #2934, #2856, #2976 - #2984, #2987 - #2989, #2991 - #2993, #2995) * Use StateResolutionHandler to resolve state in persist_events (PR #2864, #2871, #2802, #2835, #2836, #2841, #2842, #2849) * Adapt the default config to bind on both IPv4 and IPv6 on all platforms (PR #2435) Thanks to @silkeh! * Allow use of higher versions of saml2 (PR #2695) Thanks to @okurz! @@ -64,12 +65,9 @@ Changes: * Factor run_in_background out from preserve_fn (PR #2961) * Add a metric which increments when a request is received (PR #2965) * Improve caching for read_marker API (PR #2927) +* Add Measure block for persist_events (PR #2975) -Synapse 0.27.0 begins the process of rationalising metric names. To enable a graceful migration path, this release just adds new names for the metrics being renamed. A future release will remove the old ones. For more info see [here](https://github.com/matrix-org/synapse/blob/develop/docs/metrics-howto.rst#block-and-response-metrics-renamed-for-0270) - - - Bug fixes: * synapse/config/password_auth_providers: Fixed bracket typo (PR #2683) Thanks to @seckrv! From 10fdcf561d614235ba8f6ef3f980ac16ca0d7df7 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Wed, 14 Mar 2018 17:30:17 +0000 Subject: [PATCH 08/24] Fix up rst formatting --- CHANGES.rst | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/CHANGES.rst b/CHANGES.rst index beac2117d3dd..42fe461b2439 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -13,12 +13,14 @@ This release also begins the process of renaming a number of the metrics reported to prometheus. See `docs/metrics-howto.rst `_. Features: + * Add support for a remote media repository backed by S3 (PR 2867, 2777, 2783, 2789, 2791, 2804, 2812,2814, 2857, 2868, 2767) -* Add shiney new purge API. New implementation is:- - * Faster - * Supports clearing by timestamp - * Supports deleting of local events - * Is transactional (PR #2858,2867,2882, 2946,2962,2943) +* Add shiney new purge API. New implementation is: + + * Faster + * Supports clearing by timestamp + * Supports deleting of local events + * Is transactional (PR #2858,2867,2882, 2946,2962,2943) * Let homeservers specify a whitelist for the format of 3PIDs that users are allowed to register with or add to their HS accounts. (PR #2813) * Add /room/{id}/event/{id} to synapse (PR #2766) @@ -28,7 +30,7 @@ Features: Changes: -* Major refactor to move computation out of the main process and into a series of worker processes with a view to taking advantage of multicore machines. See more [here](https://github.com/matrix-org/synapse/blob/master/docs/workers.rst), (PR #2892-#2904, #2913, #2920 - #2926, #2947, #2847, #2854, #2872, #2873, #2874, #2928, #2929, #2934, #2856, #2976 - #2984, #2987 - #2989, #2991 - #2993, #2995) +* Major refactor to move computation out of the main process and into a series of worker processes with a view to taking advantage of multicore machines. See more at `docs/workers.rst `_ (PR #2892-#2904, #2913, #2920 - #2926, #2947, #2847, #2854, #2872, #2873, #2874, #2928, #2929, #2934, #2856, #2976 - #2984, #2987 - #2989, #2991 - #2993, #2995) * Use StateResolutionHandler to resolve state in persist_events (PR #2864, #2871, #2802, #2835, #2836, #2841, #2842, #2849) * Adapt the default config to bind on both IPv4 and IPv6 on all platforms (PR #2435) Thanks to @silkeh! * Allow use of higher versions of saml2 (PR #2695) Thanks to @okurz! From 2059b8573f997e3210e0fa995d03f9168f8aeba6 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Wed, 14 Mar 2018 18:11:21 +0000 Subject: [PATCH 09/24] Update CHANGES.rst --- CHANGES.rst | 86 +++++++++++++++-------------------------------------- 1 file changed, 24 insertions(+), 62 deletions(-) diff --git a/CHANGES.rst b/CHANGES.rst index 42fe461b2439..38cd3bec65d1 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -14,80 +14,42 @@ reported to prometheus. See `docs/metrics-howto.rst `_ for full details (PR #2858, #2867, #2882, #2946, #2962, #2943) +* Add support for whitelisting 3PIDs that users can register. (PR #2813) +* Add ``/room/{id}/event/{id}`` API (PR #2766) +* Add an admin API to get all the media in a room (PR #2818) Thanks to @turt2live! +* Add ``federation_domain_whitelist`` option (PR #2820, #2821) Changes: -* Major refactor to move computation out of the main process and into a series of worker processes with a view to taking advantage of multicore machines. See more at `docs/workers.rst `_ (PR #2892-#2904, #2913, #2920 - #2926, #2947, #2847, #2854, #2872, #2873, #2874, #2928, #2929, #2934, #2856, #2976 - #2984, #2987 - #2989, #2991 - #2993, #2995) -* Use StateResolutionHandler to resolve state in persist_events (PR #2864, #2871, #2802, #2835, #2836, #2841, #2842, #2849) -* Adapt the default config to bind on both IPv4 and IPv6 on all platforms (PR #2435) Thanks to @silkeh! -* Allow use of higher versions of saml2 (PR #2695) Thanks to @okurz! -* Better logging when login can't find a 3pid (PR #2744) -* add ?ts massaging for ASes (PR #2754) -* Remove 'verbosity'/'log_file' from generated cfg (PR #2755) -* Make indentation of generated log config consistent (PR #2762) -* Remove dead code related to default thumbnails (PR #2764) -* Update http request metrics before calling servlet (PR #2770) -* Do bcrypt hashing in a background thread (PR #2773) +* Continue to factor out processing from main process and into worker processes. See updated `docs/workers.rst `_ (PR #2892 - #2904, #2913, #2920 - #2926, #2947, #2847, #2854, #2872, #2873, #2874, #2928, #2929, #2934, #2856, #2976 - #2984, #2987 - #2989, #2991 - #2993, #2995, #2784) +* Ensure state cache is used when persisting events (PR #2864, #2871, #2802, #2835, #2836, #2841, #2842, #2849) +* Change the default config to bind on both IPv4 and IPv6 on all platforms (PR #2435) Thanks to @silkeh! +* No longer require a specific version of saml2 (PR #2695) Thanks to @okurz! +* Remove ``verbosity``/``log_file`` from generated config (PR #2755) +* Add and improve metrics and logging (PR #2770, #2778, #2785, #2786, #2787, #2793, #2794, #2795, #2809, #2810, #2833, #2834, #2844, #2965, #2927, #2975, #2790, #2796, #2838) * When using synctl with workers, don't start the main synapse automatically (PR #2774) -* Make Counter render floats (PR #2778) -* Store state groups separately from events (PR #2784) -* Reorganise request and block metrics (PR #2785) -* Metrics for number of RDATA commands received (PR #2786) -* Metrics for events processed in appservice and fed sender (PR #2787) -* Optimise LoggingContext creation and copying (PR #2792) -* Track db txn time in millisecs (PR #2793) -* Track DB scheduling delay per-request (PR #2794, #2795) -* Sanity checking for user ids (PR #2797) -* better exception logging in callbackmetrics (PR #2809) -* Fix bugs in block metrics (PR #2810) -* Add some comments about the reactor tick time metric (PR #2816) -* Use a connection pool for the SimpleHttpClient (PR #2817) -* Remove unused/bitrotted MemoryDataStore (PR #2828) -* Make it possible to run tests against postgres (PR #2829) -* Factor out get_db_conn to HomeServer base class (PR #2830) -* Logging and metrics for the http pusher (PR #2833) -* Improve exception handling in persist_event (PR #2834) -* montoring metrics for number of cache evictions (PR #2844) +* Minor performance improvements (PR #2773, #2792) +* Use a connection pool for non-federation outbound connections (PR #2817) +* Make it possible to run unit tests against postgres (PR #2829) * Update pynacl dependency to 1.2.1 or higher (PR #2888) Thanks to @bachp! * Remove ability for AS users to call /events and /sync (PR #2948) -* use bcrypt.checkpw (PR #2949) Thanks to @krombel! -* Factor run_in_background out from preserve_fn (PR #2961) -* Add a metric which increments when a request is received (PR #2965) -* Improve caching for read_marker API (PR #2927) -* Add Measure block for persist_events (PR #2975) - +* Use bcrypt.checkpw (PR #2949) Thanks to @krombel! Bug fixes: -* synapse/config/password_auth_providers: Fixed bracket typo (PR #2683) Thanks to @seckrv! -* Check missing fields in event_from_pdu_json (PR #2745) -* Fix templating error with unban permission message (PR #2761) Thanks to @turt2live! -* Fix flaky test_rooms UTs (PR #2765) +* Fix broken ``ldap_config`` config option (PR #2683) Thanks to @seckrv! +* Fix error message when user is not allowed to unban (PR #2761) Thanks to @turt2live! * Fix publicised groups GET API (singular) over federation (PR #2772) -* Fix a logcontext leak in persist_events (PR #2790) -* Fix 'NoneType' object has no attribute 'writeHeaders' (PR #2796) -* fix SQL when searching all users (PR #2803) -* Fix server 500 on public rooms call when no rooms exist (PR #2827) -* Fix SQL for user search (PR #2831) -* Fix sql error in quarantine_media (PR #2837) -* Handle url_previews with no content-type (PR #2845) -* Add missing yield during 3pid signature checks (PR #2933) -* Fix race in sync when joining room (PR #2944) +* Fix user directory when using ``user_directory_search_all_users`` config option (PR #2803, #2831) +* Fix error on ``/publicRooms`` when no rooms exist (PR #2827) +* Fix bug in quarantine_media (PR #2837) +* Fix url_previews when no Content-Type is returned from URL (PR #2845) +* Fix rare race in sync API when joining room (PR #2944) * Fix slow event search, switch back from GIST to GIN indexes (PR #2769, #2848) -* Fix scary-looking dns resolution errors (PR #2838) From 7d26591048ac5ab6263ddb24a840c58edfd85f50 Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Thu, 15 Mar 2018 10:33:24 +0000 Subject: [PATCH 10/24] Update CHANGES.rst --- CHANGES.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/CHANGES.rst b/CHANGES.rst index 38cd3bec65d1..a50d3ea5ba23 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -6,12 +6,13 @@ Unreleased Changes in synapse v0.27.0-rc1 (2018-03-14) =========================================== -synctl no longer starts the main synapse when using ``-a`` option with workers. -A new worker file should be added with ``worker_app: synapse.app.homeserver``. +The common case for running Synapse is not to run separate workers, but for those that do, synctl no longer starts the main synapse when using ``-a`` option with workers. A new worker file should be added with ``worker_app: synapse.app.homeserver``. This release also begins the process of renaming a number of the metrics reported to prometheus. See `docs/metrics-howto.rst `_. +Note, old metrics will be removed from the 0.28.0 release. + Features: * Add ability for ASes to override message send time (PR #2754) From 7367a4a8239a04047fa648694c08f3d1c63dd3ee Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Thu, 15 Mar 2018 10:33:52 +0000 Subject: [PATCH 11/24] Update CHANGES.rst --- CHANGES.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGES.rst b/CHANGES.rst index a50d3ea5ba23..48791cec3397 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -6,7 +6,7 @@ Unreleased Changes in synapse v0.27.0-rc1 (2018-03-14) =========================================== -The common case for running Synapse is not to run separate workers, but for those that do, synctl no longer starts the main synapse when using ``-a`` option with workers. A new worker file should be added with ``worker_app: synapse.app.homeserver``. +The common case for running Synapse is not to run separate workers, but for those that do, be aware that synctl no longer starts the main synapse when using ``-a`` option with workers. A new worker file should be added with ``worker_app: synapse.app.homeserver``. This release also begins the process of renaming a number of the metrics reported to prometheus. See `docs/metrics-howto.rst `_. From b29d1abab6ca42f693e3b83ab31440fee95d00f2 Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Thu, 15 Mar 2018 10:34:15 +0000 Subject: [PATCH 12/24] Update CHANGES.rst --- CHANGES.rst | 5 ----- 1 file changed, 5 deletions(-) diff --git a/CHANGES.rst b/CHANGES.rst index 48791cec3397..931ecb4d31a3 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -1,8 +1,3 @@ -Unreleased -========== - - - Changes in synapse v0.27.0-rc1 (2018-03-14) =========================================== From 068c21ab102b22aeac7fc9aa1f4d2d229a807872 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Thu, 15 Mar 2018 10:36:31 +0000 Subject: [PATCH 13/24] CHANGES.rst: reword metric deprecation --- CHANGES.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/CHANGES.rst b/CHANGES.rst index 931ecb4d31a3..e25327f28fa0 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -5,8 +5,7 @@ The common case for running Synapse is not to run separate workers, but for thos This release also begins the process of renaming a number of the metrics reported to prometheus. See `docs/metrics-howto.rst `_. - -Note, old metrics will be removed from the 0.28.0 release. +Note that the v0.28.0 release will remove the deprecated metric names. Features: From 0ad5125814dc18a79423740ac54f96e16a427758 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Thu, 15 Mar 2018 11:05:42 +0000 Subject: [PATCH 14/24] Update purge_history_api.rst clarify that `purge_history` will not purge state --- docs/admin_api/purge_history_api.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/admin_api/purge_history_api.rst b/docs/admin_api/purge_history_api.rst index ea2922da5c7c..2da833c82762 100644 --- a/docs/admin_api/purge_history_api.rst +++ b/docs/admin_api/purge_history_api.rst @@ -16,9 +16,11 @@ including an ``access_token`` of a server admin. By default, events sent by local users are not deleted, as they may represent the only copies of this content in existence. (Events sent by remote users are -deleted, and room state data before the cutoff is always removed). +deleted.) -To delete local events as well, set ``delete_local_events`` in the body: +Room state data (such as joins, leaves, topic) is always preserved. + +To delete local message events as well, set ``delete_local_events`` in the body: .. code:: json From ddb00efc1ddec646d02e8def6053003f04d077d7 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Thu, 15 Mar 2018 14:41:30 +0000 Subject: [PATCH 15/24] Bump version number --- synapse/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/synapse/__init__.py b/synapse/__init__.py index ef8853bd2423..d72d88b47dee 100644 --- a/synapse/__init__.py +++ b/synapse/__init__.py @@ -16,4 +16,4 @@ """ This is a reference implementation of a Matrix home server. """ -__version__ = "0.26.0" +__version__ = "0.27.0-rc1" From 926ba76e23ea9d55638baff541cdfaeb9e01ac47 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Thu, 15 Mar 2018 23:43:31 +0000 Subject: [PATCH 16/24] Replace ujson with simplejson --- synapse/api/filtering.py | 2 +- synapse/handlers/e2e_keys.py | 2 +- synapse/handlers/message.py | 6 +++--- synapse/http/server.py | 5 ++--- synapse/replication/tcp/commands.py | 2 +- synapse/rest/client/v1/room.py | 2 +- synapse/rest/client/v2_alpha/sync.py | 2 +- synapse/rest/media/v1/preview_url_resource.py | 2 +- synapse/storage/account_data.py | 2 +- synapse/storage/background_updates.py | 2 +- synapse/storage/deviceinbox.py | 12 ++++++------ synapse/storage/devices.py | 2 +- synapse/storage/end_to_end_keys.py | 2 +- synapse/storage/event_push_actions.py | 2 +- synapse/storage/events.py | 4 ++-- synapse/storage/receipts.py | 2 +- synapse/storage/room.py | 2 +- synapse/storage/roommember.py | 2 +- synapse/storage/schema/delta/25/fts.py | 4 ++-- synapse/storage/schema/delta/27/ts.py | 2 +- synapse/storage/schema/delta/31/search_update.py | 4 ++-- synapse/storage/schema/delta/33/event_fields.py | 4 ++-- synapse/storage/search.py | 2 +- synapse/storage/tags.py | 2 +- synapse/storage/transactions.py | 2 +- 25 files changed, 37 insertions(+), 38 deletions(-) diff --git a/synapse/api/filtering.py b/synapse/api/filtering.py index 83206348e549..db43219d240e 100644 --- a/synapse/api/filtering.py +++ b/synapse/api/filtering.py @@ -17,7 +17,7 @@ from synapse.types import UserID, RoomID from twisted.internet import defer -import ujson as json +import simplejson as json import jsonschema from jsonschema import FormatChecker diff --git a/synapse/handlers/e2e_keys.py b/synapse/handlers/e2e_keys.py index 668a90e4958a..ce2c87e40069 100644 --- a/synapse/handlers/e2e_keys.py +++ b/synapse/handlers/e2e_keys.py @@ -13,7 +13,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -import ujson as json +import simplejson as json import logging from canonicaljson import encode_canonical_json diff --git a/synapse/handlers/message.py b/synapse/handlers/message.py index 21f1717dd2c9..d7413833ed86 100644 --- a/synapse/handlers/message.py +++ b/synapse/handlers/message.py @@ -35,7 +35,7 @@ import logging import random -import ujson +import simplejson logger = logging.getLogger(__name__) @@ -561,8 +561,8 @@ def handle_new_client_event( # Ensure that we can round trip before trying to persist in db try: - dump = ujson.dumps(unfreeze(event.content)) - ujson.loads(dump) + dump = simplejson.dumps(unfreeze(event.content)) + simplejson.loads(dump) except Exception: logger.exception("Failed to encode content: %r", event.content) raise diff --git a/synapse/http/server.py b/synapse/http/server.py index 25466cd2929a..f1e9002e4d55 100644 --- a/synapse/http/server.py +++ b/synapse/http/server.py @@ -36,7 +36,7 @@ import collections import logging import urllib -import ujson +import simplejson logger = logging.getLogger(__name__) @@ -370,8 +370,7 @@ def respond_with_json(request, code, json_object, send_cors=False, if canonical_json or synapse.events.USE_FROZEN_DICTS: json_bytes = encode_canonical_json(json_object) else: - # ujson doesn't like frozen_dicts. - json_bytes = ujson.dumps(json_object, ensure_ascii=False) + json_bytes = simplejson.dumps(json_object) return respond_with_json_bytes( request, code, json_bytes, diff --git a/synapse/replication/tcp/commands.py b/synapse/replication/tcp/commands.py index 171227cce2ef..9633404f736f 100644 --- a/synapse/replication/tcp/commands.py +++ b/synapse/replication/tcp/commands.py @@ -19,7 +19,7 @@ """ import logging -import ujson as json +import simplejson as json logger = logging.getLogger(__name__) diff --git a/synapse/rest/client/v1/room.py b/synapse/rest/client/v1/room.py index 75b735b47dbc..80989731fa98 100644 --- a/synapse/rest/client/v1/room.py +++ b/synapse/rest/client/v1/room.py @@ -29,7 +29,7 @@ import logging import urllib -import ujson as json +import simplejson as json logger = logging.getLogger(__name__) diff --git a/synapse/rest/client/v2_alpha/sync.py b/synapse/rest/client/v2_alpha/sync.py index a0a8e4b8e46c..eb91c0b2934c 100644 --- a/synapse/rest/client/v2_alpha/sync.py +++ b/synapse/rest/client/v2_alpha/sync.py @@ -33,7 +33,7 @@ import itertools import logging -import ujson as json +import simplejson as json logger = logging.getLogger(__name__) diff --git a/synapse/rest/media/v1/preview_url_resource.py b/synapse/rest/media/v1/preview_url_resource.py index 40d2e664ebcf..a413cb6226f7 100644 --- a/synapse/rest/media/v1/preview_url_resource.py +++ b/synapse/rest/media/v1/preview_url_resource.py @@ -35,7 +35,7 @@ import re import fnmatch import cgi -import ujson as json +import simplejson as json import urlparse import itertools import datetime diff --git a/synapse/storage/account_data.py b/synapse/storage/account_data.py index 56a0bde5495c..40a2ad8d0529 100644 --- a/synapse/storage/account_data.py +++ b/synapse/storage/account_data.py @@ -18,7 +18,7 @@ from synapse.util.caches.descriptors import cached, cachedList, cachedInlineCallbacks -import ujson as json +import simplejson as json import logging logger = logging.getLogger(__name__) diff --git a/synapse/storage/background_updates.py b/synapse/storage/background_updates.py index 11a1b942f17e..8f3bff311a94 100644 --- a/synapse/storage/background_updates.py +++ b/synapse/storage/background_updates.py @@ -19,7 +19,7 @@ from twisted.internet import defer -import ujson as json +import simplejson as json import logging logger = logging.getLogger(__name__) diff --git a/synapse/storage/deviceinbox.py b/synapse/storage/deviceinbox.py index 548e795dafe1..a879e5bfc1a5 100644 --- a/synapse/storage/deviceinbox.py +++ b/synapse/storage/deviceinbox.py @@ -14,7 +14,7 @@ # limitations under the License. import logging -import ujson +import simplejson from twisted.internet import defer @@ -85,7 +85,7 @@ def add_messages_txn(txn, now_ms, stream_id): ) rows = [] for destination, edu in remote_messages_by_destination.items(): - edu_json = ujson.dumps(edu) + edu_json = simplejson.dumps(edu) rows.append((destination, stream_id, now_ms, edu_json)) txn.executemany(sql, rows) @@ -177,7 +177,7 @@ def _add_messages_to_local_device_inbox_txn(self, txn, stream_id, " WHERE user_id = ?" ) txn.execute(sql, (user_id,)) - message_json = ujson.dumps(messages_by_device["*"]) + message_json = simplejson.dumps(messages_by_device["*"]) for row in txn: # Add the message for all devices for this user on this # server. @@ -199,7 +199,7 @@ def _add_messages_to_local_device_inbox_txn(self, txn, stream_id, # Only insert into the local inbox if the device exists on # this server device = row[0] - message_json = ujson.dumps(messages_by_device[device]) + message_json = simplejson.dumps(messages_by_device[device]) messages_json_for_user[device] = message_json if messages_json_for_user: @@ -253,7 +253,7 @@ def get_new_messages_for_device_txn(txn): messages = [] for row in txn: stream_pos = row[0] - messages.append(ujson.loads(row[1])) + messages.append(simplejson.loads(row[1])) if len(messages) < limit: stream_pos = current_stream_id return (messages, stream_pos) @@ -389,7 +389,7 @@ def get_new_messages_for_remote_destination_txn(txn): messages = [] for row in txn: stream_pos = row[0] - messages.append(ujson.loads(row[1])) + messages.append(simplejson.loads(row[1])) if len(messages) < limit: stream_pos = current_stream_id return (messages, stream_pos) diff --git a/synapse/storage/devices.py b/synapse/storage/devices.py index bd2effdf34a6..712106b83a65 100644 --- a/synapse/storage/devices.py +++ b/synapse/storage/devices.py @@ -13,7 +13,7 @@ # See the License for the specific language governing permissions and # limitations under the License. import logging -import ujson as json +import simplejson as json from twisted.internet import defer diff --git a/synapse/storage/end_to_end_keys.py b/synapse/storage/end_to_end_keys.py index 2cebb203c6f1..ff8538ddf81e 100644 --- a/synapse/storage/end_to_end_keys.py +++ b/synapse/storage/end_to_end_keys.py @@ -17,7 +17,7 @@ from synapse.util.caches.descriptors import cached from canonicaljson import encode_canonical_json -import ujson as json +import simplejson as json from ._base import SQLBaseStore diff --git a/synapse/storage/event_push_actions.py b/synapse/storage/event_push_actions.py index 8efe2fd4bb66..575d710d5d21 100644 --- a/synapse/storage/event_push_actions.py +++ b/synapse/storage/event_push_actions.py @@ -21,7 +21,7 @@ from .stream import lower_bound import logging -import ujson as json +import simplejson as json logger = logging.getLogger(__name__) diff --git a/synapse/storage/events.py b/synapse/storage/events.py index d08f7571d7ab..bcca5632932a 100644 --- a/synapse/storage/events.py +++ b/synapse/storage/events.py @@ -38,7 +38,7 @@ import synapse.metrics import logging -import ujson as json +import simplejson as json # these are only included to make the type annotations work from synapse.events import EventBase # noqa: F401 @@ -56,7 +56,7 @@ def encode_json(json_object): if USE_FROZEN_DICTS: - # ujson doesn't like frozen_dicts + # simplejson doesn't like frozen_dicts return encode_canonical_json(json_object) else: return json.dumps(json_object, ensure_ascii=False) diff --git a/synapse/storage/receipts.py b/synapse/storage/receipts.py index 12b3cc7f5fb5..2c3aa33693b5 100644 --- a/synapse/storage/receipts.py +++ b/synapse/storage/receipts.py @@ -20,7 +20,7 @@ from twisted.internet import defer import logging -import ujson as json +import simplejson as json logger = logging.getLogger(__name__) diff --git a/synapse/storage/room.py b/synapse/storage/room.py index 23688430b71b..2051d8506d52 100644 --- a/synapse/storage/room.py +++ b/synapse/storage/room.py @@ -23,7 +23,7 @@ import collections import logging -import ujson as json +import simplejson as json import re logger = logging.getLogger(__name__) diff --git a/synapse/storage/roommember.py b/synapse/storage/roommember.py index 3e77fd3901df..c1ca2992850c 100644 --- a/synapse/storage/roommember.py +++ b/synapse/storage/roommember.py @@ -27,7 +27,7 @@ from synapse.types import get_domain_from_id import logging -import ujson as json +import simplejson as json logger = logging.getLogger(__name__) diff --git a/synapse/storage/schema/delta/25/fts.py b/synapse/storage/schema/delta/25/fts.py index 4269ac69ad10..e7351c3ae6b8 100644 --- a/synapse/storage/schema/delta/25/fts.py +++ b/synapse/storage/schema/delta/25/fts.py @@ -17,7 +17,7 @@ from synapse.storage.prepare_database import get_statements from synapse.storage.engines import PostgresEngine, Sqlite3Engine -import ujson +import simplejson logger = logging.getLogger(__name__) @@ -66,7 +66,7 @@ def run_create(cur, database_engine, *args, **kwargs): "max_stream_id_exclusive": max_stream_id + 1, "rows_inserted": 0, } - progress_json = ujson.dumps(progress) + progress_json = simplejson.dumps(progress) sql = ( "INSERT into background_updates (update_name, progress_json)" diff --git a/synapse/storage/schema/delta/27/ts.py b/synapse/storage/schema/delta/27/ts.py index 71b12a27312c..c0176c41eef6 100644 --- a/synapse/storage/schema/delta/27/ts.py +++ b/synapse/storage/schema/delta/27/ts.py @@ -16,7 +16,7 @@ from synapse.storage.prepare_database import get_statements -import ujson +import simplejson logger = logging.getLogger(__name__) diff --git a/synapse/storage/schema/delta/31/search_update.py b/synapse/storage/schema/delta/31/search_update.py index 470ae0c00559..fe6b7d196d40 100644 --- a/synapse/storage/schema/delta/31/search_update.py +++ b/synapse/storage/schema/delta/31/search_update.py @@ -16,7 +16,7 @@ from synapse.storage.prepare_database import get_statements import logging -import ujson +import simplejson logger = logging.getLogger(__name__) @@ -49,7 +49,7 @@ def run_create(cur, database_engine, *args, **kwargs): "rows_inserted": 0, "have_added_indexes": False, } - progress_json = ujson.dumps(progress) + progress_json = simplejson.dumps(progress) sql = ( "INSERT into background_updates (update_name, progress_json)" diff --git a/synapse/storage/schema/delta/33/event_fields.py b/synapse/storage/schema/delta/33/event_fields.py index 83066cccc99d..1e002f9db249 100644 --- a/synapse/storage/schema/delta/33/event_fields.py +++ b/synapse/storage/schema/delta/33/event_fields.py @@ -15,7 +15,7 @@ from synapse.storage.prepare_database import get_statements import logging -import ujson +import simplejson logger = logging.getLogger(__name__) @@ -44,7 +44,7 @@ def run_create(cur, database_engine, *args, **kwargs): "max_stream_id_exclusive": max_stream_id + 1, "rows_inserted": 0, } - progress_json = ujson.dumps(progress) + progress_json = simplejson.dumps(progress) sql = ( "INSERT into background_updates (update_name, progress_json)" diff --git a/synapse/storage/search.py b/synapse/storage/search.py index 479b04c63697..c19e4ea449e9 100644 --- a/synapse/storage/search.py +++ b/synapse/storage/search.py @@ -21,7 +21,7 @@ import logging import re -import ujson as json +import simplejson as json logger = logging.getLogger(__name__) diff --git a/synapse/storage/tags.py b/synapse/storage/tags.py index bff73f3f0414..982a50052073 100644 --- a/synapse/storage/tags.py +++ b/synapse/storage/tags.py @@ -17,7 +17,7 @@ from synapse.util.caches.descriptors import cached from twisted.internet import defer -import ujson as json +import simplejson as json import logging logger = logging.getLogger(__name__) diff --git a/synapse/storage/transactions.py b/synapse/storage/transactions.py index 8f61f7ffae02..f825264ea9cc 100644 --- a/synapse/storage/transactions.py +++ b/synapse/storage/transactions.py @@ -23,7 +23,7 @@ from collections import namedtuple import logging -import ujson as json +import simplejson as json logger = logging.getLogger(__name__) From ba48755d56ba134590f7b5dcb5da44371639fc4c Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Thu, 15 Mar 2018 23:57:26 +0000 Subject: [PATCH 17/24] Bump version and changelog --- CHANGES.rst | 8 ++++++++ synapse/__init__.py | 2 +- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/CHANGES.rst b/CHANGES.rst index dcf9adc95c70..38ba4249b873 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -1,3 +1,11 @@ +Changes in synapse v0.26.1 (2018-03-15) +======================================= + +Bug fixes: + +* Fix bug where an invalid event caused server to stop functioning correctly, + due to parsing and serializing bugs in ujson library. + Changes in synapse v0.26.0 (2018-01-05) ======================================= diff --git a/synapse/__init__.py b/synapse/__init__.py index ef8853bd2423..80efad3a6538 100644 --- a/synapse/__init__.py +++ b/synapse/__init__.py @@ -16,4 +16,4 @@ """ This is a reference implementation of a Matrix home server. """ -__version__ = "0.26.0" +__version__ = "0.26.1" From 5b631ff41a3230d8b6ed662e5aeb406c86989f9f Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Fri, 16 Mar 2018 00:07:08 +0000 Subject: [PATCH 18/24] Remove wrong comment --- synapse/storage/events.py | 1 - 1 file changed, 1 deletion(-) diff --git a/synapse/storage/events.py b/synapse/storage/events.py index bcca5632932a..73658a99276b 100644 --- a/synapse/storage/events.py +++ b/synapse/storage/events.py @@ -56,7 +56,6 @@ def encode_json(json_object): if USE_FROZEN_DICTS: - # simplejson doesn't like frozen_dicts return encode_canonical_json(json_object) else: return json.dumps(json_object, ensure_ascii=False) From 0092cf38ae95173519ffcc403b20b4fd2dec5c8d Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Fri, 16 Mar 2018 00:11:58 +0000 Subject: [PATCH 19/24] Newline --- CHANGES.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGES.rst b/CHANGES.rst index 38ba4249b873..3b32928238ba 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -6,6 +6,7 @@ Bug fixes: * Fix bug where an invalid event caused server to stop functioning correctly, due to parsing and serializing bugs in ujson library. + Changes in synapse v0.26.0 (2018-01-05) ======================================= From 38f952b9bc96cf72d96bab3510e8f428a8247542 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Mon, 19 Mar 2018 09:27:36 +0000 Subject: [PATCH 20/24] spell out not to massively increase bcrypt rounds --- synapse/config/registration.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/synapse/config/registration.py b/synapse/config/registration.py index 336959094b3e..c5384b3ad400 100644 --- a/synapse/config/registration.py +++ b/synapse/config/registration.py @@ -77,7 +77,9 @@ def default_config(self, **kwargs): # Set the number of bcrypt rounds used to generate password hash. # Larger numbers increase the work factor needed to generate the hash. - # The default number of rounds is 12. + # The default number is 12 (which equates to 2^12 rounds). + # N.B. that increasing this will exponentially increase the time required + # to register or login - e.g. 24 => 2^24 rounds which will take >20 mins. bcrypt_rounds: 12 # Allows users to register as guests without a password/email/etc, and From 9a0d783c113ae74c55e409d33219cd77f3662b9f Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Mon, 19 Mar 2018 11:35:53 +0000 Subject: [PATCH 21/24] Add comments --- synapse/util/caches/lrucache.py | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/synapse/util/caches/lrucache.py b/synapse/util/caches/lrucache.py index a4bf8fa6ae6d..1c5a982094a0 100644 --- a/synapse/util/caches/lrucache.py +++ b/synapse/util/caches/lrucache.py @@ -154,11 +154,18 @@ def cache_get(key, default=None, callbacks=[]): def cache_set(key, value, callbacks=[]): node = cache.get(key, None) if node is not None: + # We sometimes store large objects, e.g. dicts, which cause + # the inequality check to take a long time. So let's only do + # the check if we have some callbacks to call. if node.callbacks and value != node.value: for cb in node.callbacks: cb() node.callbacks.clear() + # We don't bother to protect this by value != node.value as + # generally size_callback will be cheap compared with equality + # checks. (For example, taking the size of two dicts is quicker + # than comparing them for equality.) if size_callback: cached_cache_len[0] -= size_callback(node.value) cached_cache_len[0] += size_callback(value) From 1a3aa957ca1164268c9bbbd564491c9d46fbb7d2 Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Mon, 19 Mar 2018 15:11:00 +0000 Subject: [PATCH 22/24] Update CHANGES.rst --- CHANGES.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/CHANGES.rst b/CHANGES.rst index e25327f28fa0..55358287a021 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -1,3 +1,10 @@ +Changes in synapse v0.27.0-rc2 (2018-03-19) +=========================================== +Bugs: + +* Fix bug introduced in v0.27.0-rc1 that causes much increased memory usage in state cache (PR #3005) + + Changes in synapse v0.27.0-rc1 (2018-03-14) =========================================== From c384705ee877ba40e1a736b68ebe07434662cbaa Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Mon, 19 Mar 2018 15:11:58 +0000 Subject: [PATCH 23/24] Update __init__.py bump version --- synapse/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/synapse/__init__.py b/synapse/__init__.py index d72d88b47dee..71b409ba8b50 100644 --- a/synapse/__init__.py +++ b/synapse/__init__.py @@ -16,4 +16,4 @@ """ This is a reference implementation of a Matrix home server. """ -__version__ = "0.27.0-rc1" +__version__ = "0.27.0-rc2" From f5aa027c2f9d836df7b98da6c1c5fee6b22d0a3d Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Tue, 20 Mar 2018 15:06:22 +0000 Subject: [PATCH 24/24] Update CHANGES.rst rearrange ordering of releases to match chronology --- CHANGES.rst | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/CHANGES.rst b/CHANGES.rst index 788150bcfcb0..bac78d3c1c5e 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -1,10 +1,21 @@ Changes in synapse v0.27.0-rc2 (2018-03-19) =========================================== -Bugs: + +Pulls in v0.26.1 + +Bug fixes: * Fix bug introduced in v0.27.0-rc1 that causes much increased memory usage in state cache (PR #3005) + + +Changes in synapse v0.26.1 (2018-03-15) +======================================= + +Bug fixes: + * Fix bug where an invalid event caused server to stop functioning correctly, - due to parsing and serializing bugs in ujson library. (pulled in from 0.26.1) + due to parsing and serializing bugs in ujson library (PR #3008) + Changes in synapse v0.27.0-rc1 (2018-03-14) =========================================== @@ -55,16 +66,6 @@ Bug fixes: * Fix slow event search, switch back from GIST to GIN indexes (PR #2769, #2848) -Changes in synapse v0.26.1 (2018-03-15) -======================================= - -Bug fixes: - -* Fix bug where an invalid event caused server to stop functioning correctly, - due to parsing and serializing bugs in ujson library. - - - Changes in synapse v0.26.0 (2018-01-05) =======================================