matrix-synapse: 1.19.3 -> 1.20.1

[Ma27]

Motivation for this change

https://github.com/matrix-org/synapse/releases/tag/v1.20.0
https://github.com/matrix-org/synapse/releases/tag/v1.20.1

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

[Ekleog]

Reading the changelog, I can't find mention of security fixes, and there's this backwards-compatibility warning.

So… as I guess synapse' retro-compat story in the federation is still as bad as before, I'd say maybe let's backport to 20.09 but not 20.03, thus keeping 20.03 stable?

[Ma27]

Ported to stable as e7c26a7.

[dali99]

1.20 rc1 has a fix for matrix-org/synapse#8220
without which logs are spammed with user directory errors. And crashing the user directory because a server sent an invalid display name.

That could count as a partial denial of service vulnerability 🤷

[Ma27]

cc @Ekleog what do you think of it?

[Ekleog]

@dali99 What do you mean by “crashing the user directory”? I can't find this in the issue you linked.

As for the part about synapse spamming logs… my synapse instance, on which we are fewer than 5 users, logged ~375k log lines yesterday using more or less the nixos-default configuration, without this bug happening (tested by grepping for InvalidTextRepresentation). Would this bug make the behavior significantly worse? Having tracebacks in the logs is something I remember synapse regularly doing (though I guess it happens less nowadays), so unless it'd increase by an order of magnitude I'm not sure it's (by itself and not counting that “crashing the user directory” issue) worth risking backwards-compatibility breakage so close to 20.03 EOL?

Also anyway, thank you for the report, regardless of the end result it's great you reported this!

[dali99]

The user directory background process, crashes, and immediately retries, not processing any updates, of course immediately crashing again. repeat multiple times per second. When the user directory crashes that means new users the server sees won't be processed, which means the user directory won't show them to you when you search for their usernames when you want to invite them or otherwise.

@Ekleog in the past 24 hours my synapse has logged 10 027 118 lines. And that is with a log level changed up from nixos default INFO to WARN. With this specific traceback occuring 2-4 times per second at the error log level.

Which is two orders of magnitude worse.

I have a suspicion that this bug is also what's causing my synapse to idle at 70% CPU.

As for where this is described in the issue I linked:

User directory gets stuck

The main issue is that the error path seems to continue throwing errors and stops making progress.

[Ekleog]

Thank you for your feedback!

I… guess this is worth backporting to 20.03, then? My problem with this is, reading the backwards-compatibility issue, it looks like clients that are even just 5 months old are currently incompatible with the new synapse release…

I guess I'll just yell at synapse in my head again and we can backport the update to 20.03, as FluffyChat doesn't look widely-used… if that's also the way you feel, @Ma27?

[Ma27]

Tbh I deferred the debate to this thread since I don't have a strong opinion about it (all of my systems are already using a recent version of release-20.09).

However we have people here experiencing this issue and I actually established a backport-by-default policy for matrix-synapse about a year ago, so I'm fine with it and would take care of it now.

While some clients seem to have issues with v1.20.0 now, we shouldn't forget that (1) the client violated the protocol and (2) the most famous example (fluffychat) has fixed the issue already[1].

[1] matrix-org/synapse#6766 (comment)

[Ma27]

Ported to 20.03 as 760c6ec now.