Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

When purging, don't de-delta state groups we're about to delete #2214

Merged
merged 3 commits into from
May 11, 2017

Conversation

richvdh
Copy link
Member

@richvdh richvdh commented May 10, 2017

... because that's (a) pointless (b) slow and (c) eats all my disk space just when I'm trying to free disk space

@richvdh richvdh force-pushed the rav/hurry_up_purge branch from bcba88a to 8e345ce Compare May 10, 2017 17:44
for i in xrange(0, len(state_groups_to_delete), 100)
]
for chunk in chunks:
for i in xrange(0, len(state_rows), 100):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems odd to dedupe state_groups_to_delete and then use the non-deduped list here

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the set is for easy lookup at line 2101 rather than deduping. We can't easily divide a set into chunks, hence the use of the list here. There shouldn't be any dups because the query has a GROUP BY.

new_state_edges.extend(row["state_group"] for row in rows)
new_state_edges.extend(
row["state_group"] for row in rows
if row["state_group"] not in state_groups_to_delete
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you comment this please? It took me a few moments to realise what was going on

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup will do more comments

Copy link
Member

@erikjohnston erikjohnston left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(see comments)

@erikjohnston erikjohnston assigned richvdh and unassigned erikjohnston May 11, 2017
Try to make this clearer with more comments and some variable renames
@richvdh
Copy link
Member Author

richvdh commented May 11, 2017

@erikjohnston ptal

@richvdh richvdh assigned erikjohnston and unassigned richvdh May 11, 2017
Copy link
Member

@erikjohnston erikjohnston left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@richvdh richvdh merged commit 29ded77 into develop May 11, 2017
psaavedra added a commit to psaavedra/synapse that referenced this pull request May 19, 2017
Changes in synapse v0.21.0 (2017-05-18)
=======================================

No changes since v0.21.0-rc3

Changes in synapse v0.21.0-rc3 (2017-05-17)
===========================================

Features:

* Add per user rate-limiting overrides (PR matrix-org#2208)
* Add config option to limit maximum number of events requested by ``/sync``
  and ``/messages`` (PR matrix-org#2221) Thanks to @psaavedra!

Changes:

* Various small performance fixes (PR matrix-org#2201, matrix-org#2202, matrix-org#2224, matrix-org#2226, matrix-org#2227, matrix-org#2228,
  matrix-org#2229)
* Update username availability checker API (PR matrix-org#2209, matrix-org#2213)
* When purging, don't de-delta state groups we're about to delete (PR matrix-org#2214)
* Documentation to check synapse version (PR matrix-org#2215) Thanks to @hamber-dick!
* Add an index to event_search to speed up purge history API (PR matrix-org#2218)

Bug fixes:

* Fix API to allow clients to upload one-time-keys with new sigs (PR matrix-org#2206)

Changes in synapse v0.21.0-rc2 (2017-05-08)
===========================================

Changes:

* Always mark remotes as up if we receive a signed request from them (PR matrix-org#2190)

Bug fixes:

* Fix bug where users got pushed for rooms they had muted (PR matrix-org#2200)

Changes in synapse v0.21.0-rc1 (2017-05-08)
===========================================

Features:

* Add username availability checker API (PR matrix-org#2183)
* Add read marker API (PR matrix-org#2120)

Changes:

* Enable guest access for the 3pl/3pid APIs (PR matrix-org#1986)
* Add setting to support TURN for guests (PR matrix-org#2011)
* Various performance improvements (PR matrix-org#2075, matrix-org#2076, matrix-org#2080, matrix-org#2083, matrix-org#2108,
  matrix-org#2158, matrix-org#2176, matrix-org#2185)
* Make synctl a bit more user friendly (PR matrix-org#2078, matrix-org#2127) Thanks @APwhitehat!
* Replace HTTP replication with TCP replication (PR matrix-org#2082, matrix-org#2097, matrix-org#2098,
  matrix-org#2099, matrix-org#2103, matrix-org#2014, matrix-org#2016, matrix-org#2115, matrix-org#2116, matrix-org#2117)
* Support authenticated SMTP (PR matrix-org#2102) Thanks @DanielDent!
* Add a counter metric for successfully-sent transactions (PR matrix-org#2121)
* Propagate errors sensibly from proxied IS requests (PR matrix-org#2147)
* Add more granular event send metrics (PR matrix-org#2178)

Bug fixes:

* Fix nuke-room script to work with current schema (PR matrix-org#1927) Thanks
  @zuckschwerdt!
* Fix db port script to not assume postgres tables are in the public schema
  (PR matrix-org#2024) Thanks @jerrykan!
* Fix getting latest device IP for user with no devices (PR matrix-org#2118)
* Fix rejection of invites to unreachable servers (PR matrix-org#2145)
* Fix code for reporting old verify keys in synapse (PR matrix-org#2156)
* Fix invite state to always include all events (PR matrix-org#2163)
* Fix bug where synapse would always fetch state for any missing event (PR matrix-org#2170)
* Fix a leak with timed out HTTP connections (PR matrix-org#2180)
* Fix bug where we didn't time out HTTP requests to ASes  (PR matrix-org#2192)

Docs:

* Clarify doc for SQLite to PostgreSQL port (PR matrix-org#1961) Thanks @benhylau!
* Fix typo in synctl help (PR matrix-org#2107) Thanks @HarHarLinks!
* ``web_client_location`` documentation fix (PR matrix-org#2131) Thanks @matthewjwolff!
* Update README.rst with FreeBSD changes (PR matrix-org#2132) Thanks @feld!
* Clarify setting up metrics (PR matrix-org#2149) Thanks @encks!
@richvdh richvdh deleted the rav/hurry_up_purge branch October 9, 2017 17:10
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants