Sync Freenode membership to Matrix

[ara4n]

In order to make the Freenode entries in the room directory coherent, and make the bridge more usable, and now that we have collapsed join/parts in Riot, we need to (at last) sync the membership lists of Freenode into Matrix.

[torarnv]

This would be very very welcome! I find myself talking to people in bridged IRC channels who have since left the room/channel, since the auto-completion doesnt know the person has left.

[devnoname120]

Syncing the membership would also slightly mitigate this: #348

[kegsay]

This hasn't been enabled yet because we have real concerns that this will take out Synapse on matrix.org. We're periodically re-evaluating this as load concerns are addressed: this may have to wait until we have an SSD backed Synapse. All the other IRC networks have membership syncing enabled.

[Croydon]

@kegsay Is there an estimated time frame when you will get an SSD backed Synapse?

[kegsay]

We're in the process of evaluating/testing providers. I don't have any ETA beyond "when it's ready".

[Croydon]

/me wonders if this is going to happen now :)

[albfan]

Any chances to get progress on this

[kegsay]

We now have matrix.org Synapse running on a faster machine, but I'm unconvinced this will be of any help. See #384 (comment) for examples of some problems we've had (and subsequent degradation in service) since we moved, albeit not related to the move directly. We need to monitor how things go for a while before revisiting this.

[Valodim]

I'd also like to point out that at least riot android does not yet have join/leave collapsing. I enabled irc->matrix membership sync on my bridge for a day, and user response was very negative about this~

[rodrigoaguilera]

Does anyone know why some rooms have the sync (and join/parts) enabled on freenode like #drupal but others do not like #drupal-18n ?
Is it based on popularity?
Can it be requested?

How can I help to make it possible (code, donations)?

[kegsay]

@rodrigoaguilera as of 09:48 BST we enabled membership syncing on Freenode which is probably why you saw some channels syncing and some not. This was followed by a subsequent restart at 13:10 BST to pull in rate limiting fixes to ease load on Synapse.

At our current best estimates, it will take tens of hours to fully sync membership lists as this is the first time we've done this ever and there's 1.5-2 years worth of nicks accrued on the Matrix-side. The bridge will be operable throughout this time frame.

[kegsay]

This issue will be closed when I can confirm that:

• We've caught up with the backlog.
• The "normal" join/part rate isn't going to cause issues for the bridge or matrix.org Synapse.

[torarnv]

Will this completely sync channel membership lists? So that any user not currently online will not be in the room member list?

Right now I'm seeing some channels with grayed out "Offline" users, and non-grayed out "Offline for 20d". Once the whole sync completes, would all these be "Online" users?

[kegsay]

You're referring to 2 different things:

• sync channel membership lists (i.e. do they appear on the member list)
• grayed out "Offline" users (i.e. are members on the member list online or offline)

We have enabled syncing of channel membership lists, that's all. Presence is unaffected and will remain showing incorrect information (this is partly because presence on matrix.org is disabled currently).

Even though they have incorrect presence information, if they are present on the membership list then they should be present on the IRC-side. There are a number of edge cases which can cause the number of reported users on the Matrix-side to be slightly too high:

• The bridge doesn't understand nick changes very well. If an IRC user changes their nick then says something, they will have 2 users Matrix-side (one with the old nick). This will be cleaned up when the bridge restarts. We're aware of this issue.
• The bridge is concurrently syncing membership lists along with processing normal traffic. This can create race conditions e.g where through normal traffic the user leaves the channel, but then the concurrently syncing membership lists incorrectly joins the user Matrix-side. This will eventually converge to the correct state, but it's likely that high join/part channels will show larger discrepancies in the member lists than channels with low amounts of join/parts.

Overall, the membership lists for freenode will be much more accurate for Matrix users going forwards (provided we don't need to rollback the changes due to matrix.org Synapse struggling).

[torarnv]

Ah, I see, thanks for the explanation.

As long as the Riot client's autocomplete feature uses the room member list, then all should be good and I'll stop unknowingly pinging people who are offline 😄

[kegsay]

Yeah, we're acutely aware how much of a UX issue it has been by not syncing membership lists. Hopefully this will be mostly resolved in about 13 hours.

We're also aware of the issue @Valodim points out: it's likely that enabling member list syncing on Freenode will be the impetus for the mobile apps to sprout support for collapsing joins/parts.

[ara4n]

finished as of about 4 hours ago!

[rodrigoaguilera]

This is great. Now I can't start recommending riot to everyone as replacement for their IRC client :)

[kegsay]

You stole my thunder @ara4n :)

I was going to close this after checking we can cope with the "normal" rate of joins/parts for freenode later tonight, but it's probably fine given how synapse mostly didn't die during the high leave rate.

[ArchangeGabriel]

@Valodim and others interested in Riot-Android part: element-hq/riot-android#1236