Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Reduce federation replication traffic #2115

Merged
merged 9 commits into from
Apr 12, 2017

Conversation

erikjohnston
Copy link
Member

This is mainly done by moving the calculation of where to send presence updates from the presence handler to the transaction queue, so we only need to send the presence event (and not the destinations) across the replication connection. Before we were duplicating by sending the full state across once per destination.

This is mainly done by moving the calculation of where to send presence
updates from the presence handler to the transaction queue, so we only
need to send the presence event (and not the destinations) across the
replication connection. Before we were duplicating by sending the full
state across once per destination.
@erikjohnston erikjohnston force-pushed the erikj/dedupe_federation_repl branch from dffee64 to 29574fd Compare April 10, 2017 15:49
continue

host = get_domain_from_id(user_id)
hosts_and_states.append(([host], local_states))
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The above is copied verbatim from _get_interested_parties

state.user_id: state for state in states
})

preserve_fn(self._attempt_new_transaction)(destination)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The above is copied verbatim from the old send_presence function

@@ -224,17 +227,95 @@ def _send_pdu(self, pdu, destinations):
self._attempt_new_transaction, destination
)

def send_presence(self, destination, states):
if not self.can_send_to(destination):
@preserve_fn
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this an acceptable thing to do style wise?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think so. I think a comment saying # the caller should not yield on this on this line would help.

Copy link
Member

@richvdh richvdh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks broadly plausible, but I had a bit of a nightmare trying to understand what was going on. A lot of that is because of poor comments in the existing code, so I had to go digging to understand things. Accordingly, I've made a load of requests that you comment things better; I realise that some of them aren't changing, but in the interests of trying to make work in this area less awful in the first place I'd appreciate it if we could take the opportunity to improve things.

I think moving get_interested_parties out would really help.

@@ -224,17 +227,95 @@ def _send_pdu(self, pdu, destinations):
self._attempt_new_transaction, destination
)

def send_presence(self, destination, states):
if not self.can_send_to(destination):
@preserve_fn
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think so. I think a comment saying # the caller should not yield on this on this line would help.

@@ -187,18 +190,14 @@ def send_edu(self, destination, edu_type, content, key=None):

self.notifier.on_new_replication_data()

def send_presence(self, destination, states):
def send_presence(self, states):
"""As per TransactionQueue"""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you doc the type of states anyway, please

"""Sends state updates to remote servers.

Args:
hosts_to_states (dict): Mapping `server_name` -> `[UserPresenceState]`
hosts_to_states (list): list(state)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

list(UserPresenceState), no?

states, calculate_remote_hosts=False
)
room_ids_to_states, users_to_states, _ = parties
parties = yield self._get_interested_parties(states)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since we're changing the signature of _get_interested_parties anyway, it would be a good time to rename it to reflect the fact it is being used outside PresenceHandler.

see below: I think _get_interested_parties needs to move anyway.


host = get_domain_from_id(user_id)
hosts_to_states.setdefault(host, []).extend(local_states)

# TODO: de-dup hosts_to_states, as a single host might have multiple
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

think this comment is dead

@@ -78,6 +78,7 @@ def __init__(self, hs):
self.pending_edus_by_dest = edus = {}

# Presence needs to be separate as we send single aggragate EDUs
self.pending_presence = {}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you document what all three of these fields mean and their types?

@preserve_fn
@defer.inlineCallbacks
def send_presence(self, states):
"""Send the new presence states to the appropriate destinations.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Start sending" ... "if we are not already sending updates" or something

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you didn't like my suggestion?

# hosts in those rooms.
room_ids_to_states = {}
users_to_states = {}
for state in states.itervalues():
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please let's not c&p get_interested_parties here. How about moving this lump of code to the store or something?

(For super-mega-bonus points, make it a separate PR which precedes this one...)

for u in plist:
users_to_states.setdefault(u, []).append(state)

hosts_and_states = []
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment on what this is going to be, please.


hosts_and_states = []
for room_id, states in room_ids_to_states.items():
local_states = filter(lambda s: self.is_mine_id(s.user_id), states)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

again, why are we filtering, and why would updates for remote users be getting in here? (and could we filter before deriving room_ids_to_states and users_to_states to avoid some work? happy if you want to leave this for now to avoid changing too much at once)

@richvdh richvdh assigned erikjohnston and unassigned richvdh Apr 11, 2017
@erikjohnston erikjohnston force-pushed the erikj/dedupe_federation_repl branch from 39d527d to 414522a Compare April 11, 2017 14:33
@erikjohnston
Copy link
Member Author

I've moved the get_interested_* functions out into standalone functions in the handlers/presence.py file, as it doesn't feel like the sort of thing that belongs in the storage layer

@erikjohnston erikjohnston assigned richvdh and unassigned erikjohnston Apr 11, 2017
# hosts in those rooms.
room_ids_to_states = {}
users_to_states = {}
for state in states.itervalues():
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we not use get_interested_parties here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heh, for some reason I got it stuck in my head that they were different

@preserve_fn
@defer.inlineCallbacks
def send_presence(self, states):
"""Send the new presence states to the appropriate destinations.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you didn't like my suggestion?

# to be sent out by user_id. Entries here get processed and put in
# pending_presence_by_dest
self.pending_presence = {}
# Map of destination -> user_id -> UserPresenceState of pending presence
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could do with a blank line here

@@ -78,7 +78,18 @@ def __init__(self, hs):
self.pending_edus_by_dest = edus = {}

# Presence needs to be separate as we send single aggragate EDUs
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not quite sure what this comment means, any more

each row the list of UserPresenceState should be sent to each
destination
"""
hosts_and_states = [] # Final result to return
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

now that you have an (excellent) description of the return value of the function, this probably doesn't really need a comment, but it's harmless enough

@@ -669,7 +641,7 @@ def _push_to_remotes(self, states):
"""Sends state updates to remote servers.

Args:
hosts_to_states (list): list(state)
hosts_to_states (list(UserPresenceState))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/hosts_to_states/states/. sorry for not spotting that one before

@richvdh
Copy link
Member

richvdh commented Apr 12, 2017

I've moved the get_interested_* functions out into standalone functions in the handlers/presence.py file, as it doesn't feel like the sort of thing that belongs in the storage layer

Fine by me. Looks much better now.

@richvdh richvdh assigned erikjohnston and unassigned richvdh Apr 12, 2017
@erikjohnston erikjohnston assigned richvdh and unassigned erikjohnston Apr 12, 2017
Copy link
Member

@richvdh richvdh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@richvdh richvdh removed their assignment Apr 12, 2017
@erikjohnston erikjohnston merged commit 247c736 into develop Apr 12, 2017
psaavedra added a commit to psaavedra/synapse that referenced this pull request May 19, 2017
Changes in synapse v0.21.0 (2017-05-18)
=======================================

No changes since v0.21.0-rc3

Changes in synapse v0.21.0-rc3 (2017-05-17)
===========================================

Features:

* Add per user rate-limiting overrides (PR matrix-org#2208)
* Add config option to limit maximum number of events requested by ``/sync``
  and ``/messages`` (PR matrix-org#2221) Thanks to @psaavedra!

Changes:

* Various small performance fixes (PR matrix-org#2201, matrix-org#2202, matrix-org#2224, matrix-org#2226, matrix-org#2227, matrix-org#2228,
  matrix-org#2229)
* Update username availability checker API (PR matrix-org#2209, matrix-org#2213)
* When purging, don't de-delta state groups we're about to delete (PR matrix-org#2214)
* Documentation to check synapse version (PR matrix-org#2215) Thanks to @hamber-dick!
* Add an index to event_search to speed up purge history API (PR matrix-org#2218)

Bug fixes:

* Fix API to allow clients to upload one-time-keys with new sigs (PR matrix-org#2206)

Changes in synapse v0.21.0-rc2 (2017-05-08)
===========================================

Changes:

* Always mark remotes as up if we receive a signed request from them (PR matrix-org#2190)

Bug fixes:

* Fix bug where users got pushed for rooms they had muted (PR matrix-org#2200)

Changes in synapse v0.21.0-rc1 (2017-05-08)
===========================================

Features:

* Add username availability checker API (PR matrix-org#2183)
* Add read marker API (PR matrix-org#2120)

Changes:

* Enable guest access for the 3pl/3pid APIs (PR matrix-org#1986)
* Add setting to support TURN for guests (PR matrix-org#2011)
* Various performance improvements (PR matrix-org#2075, matrix-org#2076, matrix-org#2080, matrix-org#2083, matrix-org#2108,
  matrix-org#2158, matrix-org#2176, matrix-org#2185)
* Make synctl a bit more user friendly (PR matrix-org#2078, matrix-org#2127) Thanks @APwhitehat!
* Replace HTTP replication with TCP replication (PR matrix-org#2082, matrix-org#2097, matrix-org#2098,
  matrix-org#2099, matrix-org#2103, matrix-org#2014, matrix-org#2016, matrix-org#2115, matrix-org#2116, matrix-org#2117)
* Support authenticated SMTP (PR matrix-org#2102) Thanks @DanielDent!
* Add a counter metric for successfully-sent transactions (PR matrix-org#2121)
* Propagate errors sensibly from proxied IS requests (PR matrix-org#2147)
* Add more granular event send metrics (PR matrix-org#2178)

Bug fixes:

* Fix nuke-room script to work with current schema (PR matrix-org#1927) Thanks
  @zuckschwerdt!
* Fix db port script to not assume postgres tables are in the public schema
  (PR matrix-org#2024) Thanks @jerrykan!
* Fix getting latest device IP for user with no devices (PR matrix-org#2118)
* Fix rejection of invites to unreachable servers (PR matrix-org#2145)
* Fix code for reporting old verify keys in synapse (PR matrix-org#2156)
* Fix invite state to always include all events (PR matrix-org#2163)
* Fix bug where synapse would always fetch state for any missing event (PR matrix-org#2170)
* Fix a leak with timed out HTTP connections (PR matrix-org#2180)
* Fix bug where we didn't time out HTTP requests to ASes  (PR matrix-org#2192)

Docs:

* Clarify doc for SQLite to PostgreSQL port (PR matrix-org#1961) Thanks @benhylau!
* Fix typo in synctl help (PR matrix-org#2107) Thanks @HarHarLinks!
* ``web_client_location`` documentation fix (PR matrix-org#2131) Thanks @matthewjwolff!
* Update README.rst with FreeBSD changes (PR matrix-org#2132) Thanks @feld!
* Clarify setting up metrics (PR matrix-org#2149) Thanks @encks!
@erikjohnston erikjohnston deleted the erikj/dedupe_federation_repl branch October 26, 2017 11:00
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants