nixos/matrix-synapse: migrate to generators.toYAML and other cleanup

[sumnerevans]

Motivation for this change

• The current method of generating the synapse config involves manually creating the YAML. This PR converts it to use the generators.toYAML function which is more maintainable.
• This PR adds the ability to use a Nix attrset to add additional settings to Synapse.
• This PR adds a toFloat library function to assist in converting strings to floats (this is modeled after toInt).
• This PR changes moves the rate limiting configurations to submodules, and converts them to the correct types.
• This PR converts the verbose and bcrypt_rounds options to ints
• This PR removes the user_creation_max_duration option (not found in synapse codebase at all)
• This PR removes the uploads_path option (Removed here: Remove vestiges of uploads_path config matrix-org/synapse#9462)
• This PR removes the tls_dh_params_path option (synapse_admin_mau:current metric is not updated when not limiting usage matrix-org/synapse#4229)
• This PR removes the expire_access_token option (Remove non-functional 'expire_access_token' setting matrix-org/synapse#5782)

I am doing all of these deprecations at once to try and avoid mkChangedOptionModule chains.

Note to Reviewers

• I recommend going through each commit one-by-one.

Manual testing performed

Test 1

I tested using the following config:

Test 1 Nix config

services.matrix-synapse = {
  enable = true;
  enable_metrics = true;
  enable_registration = false;
  database_type = "sqlite3";
  server_name = "localhost";
  max_upload_size = "250M";
  listeners = [
    {
      port = 8008;
      bind_address = "::1";
      type = "http";
      tls = false;
      x_forwarded = true;
      resources = [
        {
          names = [ "client" "federation" ];
          compress = false;
        }
      ];
    }
  ];
};

• I ensured that Synapse runs without error.
• I inspected the systemd service config to ensure that everything looked right. This is the ExecStart line:

  ExecStart=/nix/store/8gbwdy0m7hda76mjycs7pifyry172kci-matrix-synapse-1.34.0/bin/homeserver \
    --config-path /nix/store/pihg9l7jgy2ii9hs8w4h8kn97n73xv60-homeserver.yaml --config-path /nix/store/47xi81pm06p6cpm211wl68cxac4zl0z8-extra-homeserver.yaml \
    --keys-directory /var/lib/matrix-synapse
  

• I looked at the generated homeserver.yaml file to ensure that everything looked correct. Here is the generated config (piped through jq for clarity):

Generated config (piped through `jq`)

{
  "account_threepid_delegates": {},
  "allow_guest_access": false,
  "app_service_config_files": [],
  "bcrypt_rounds": 12,
  "database": {
    "args": {
      "database": "/var/lib/matrix-synapse/homeserver.db"
    },
    "name": "sqlite3"
  },
  "dynamic_thumbnails": false,
  "enable_metrics": true,
  "enable_registration": false,
  "enable_registration_captcha": false,
  "event_cache_size": "10K",
  "key_refresh_interval": "1d",
  "listeners": [
    {
      "bind_address": "::1",
      "port": 8008,
      "resources": [
        {
          "compress": false,
          "names": [
            "client",
            "federation"
          ]
        }
      ],
      "tls": false,
      "type": "http",
      "x_forwarded": true
    }
  ],
  "log_config": "/nix/store/2x89r9ji20wy2sggd017jfhlmn9ygsj3-log_config.yaml",
  "max_image_pixels": "32M",
  "max_upload_size": "250M",
  "media_store_path": "/var/lib/matrix-synapse/media",
  "no_tls": false,
  "perspectives": {
    "servers": {
      "matrix.org": {
        "verify_keys": {
          "ed25519:auto": {
            "key": "Noi6WqcDj0QmPxCNQqgezwTlBKrfqehY1u2FyWP9uYw"
          }
        }
      }
    }
  },
  "pid_file": "/run/matrix-synapse.pid",
  "rc_federation": {
    "concurrent": 3,
    "reject_limit": 50,
    "sleep_delay": 500,
    "sleep_limit": 10,
    "window_size": 1000
  },
  "rc_message": {
    "burst_count": 10,
    "per_second": 0.2
  },
  "recaptcha_private_key": "",
  "recaptcha_public_key": "",
  "recaptcha_siteverify_api": "https://www.google.com/recaptcha/api/siteverify",
  "redaction_retention_period": 7,
  "report_stats": false,
  "room_prejoin_state": {
    "additional_event_types": [],
    "disable_default_event_types": false
  },
  "server_name": "localhost",
  "signing_key_path": "/var/lib/matrix-synapse/homeserver.signing.key",
  "turn_shared_secret": "",
  "turn_uris": [],
  "turn_user_lifetime": "1h",
  "verbose": 0
}

Test 2 (deprecated conversions)

I tested using the following config:

Test 2 Nix config

  services.matrix-synapse = {
    enable = true;
    enable_metrics = true;
    enable_registration = false;

    rc_messages_per_second = "1.2";
    rc_message_burst_count = "11.0";
    federation_rc_window_size = "1024";
    federation_rc_sleep_limit = "42";
    federation_rc_sleep_delay = "404";
    federation_rc_reject_limit = "80";
    federation_rc_concurrent = "4";

    database_type = "sqlite3";
    server_name = "localhost";
    max_upload_size = "250M";
    listeners = [
      {
        port = 8008;
        bind_address = "::1";
        type = "http";
        tls = false;
        x_forwarded = true;
        resources = [
          {
            names = [ "client" "federation" ];
            compress = false;
          }
        ];
      }
    ];
  };

• The following warnings were shown:

  trace: warning: The option `services.matrix-synapse.federation_rc_concurrent' defined in `/etc/nixos/host-configurations/coruscant.nix' has been changed to `services.matrix-synapse.rc_federation.concurrent' that has a different type. Please read `services.matrix-synapse.rc_federation.concurrent' documentation and update your configuration accordingly.
  trace: warning: The option `services.matrix-synapse.federation_rc_reject_limit' defined in `/etc/nixos/host-configurations/coruscant.nix' has been changed to `services.matrix-synapse.rc_federation.reject_limit' that has a different type. Please read `services.matrix-synapse.rc_federation.reject_limit' documentation and update your configuration accordingly.
  trace: warning: The option `services.matrix-synapse.federation_rc_sleep_delay' defined in `/etc/nixos/host-configurations/coruscant.nix' has been changed to `services.matrix-synapse.rc_federation.sleep_delay' that has a different type. Please read `services.matrix-synapse.rc_federation.sleep_delay' documentation and update your configuration accordingly.
  trace: warning: The option `services.matrix-synapse.federation_rc_sleep_limit' defined in `/etc/nixos/host-configurations/coruscant.nix' has been changed to `services.matrix-synapse.rc_federation.sleep_limit' that has a different type. Please read `services.matrix-synapse.rc_federation.sleep_limit' documentation and update your configuration accordingly.
  trace: warning: The option `services.matrix-synapse.federation_rc_window_size' defined in `/etc/nixos/host-configurations/coruscant.nix' has been changed to `services.matrix-synapse.rc_federation.window_size' that has a different type. Please read `services.matrix-synapse.rc_federation.window_size' documentation and update your configuration accordingly.
  trace: warning: The option `services.matrix-synapse.rc_message_burst_count' defined in `/etc/nixos/host-configurations/coruscant.nix' has been changed to `services.matrix-synapse.rc_message.burst_count' that has a different type. Please read `services.matrix-synapse.rc_message.burst_count' documentation and update your configuration accordingly.
  trace: warning: The option `services.matrix-synapse.rc_messages_per_second' defined in `/etc/nixos/host-configurations/coruscant.nix' has been changed to `services.matrix-synapse.rc_message.per_second' that has a different type. Please read `services.matrix-synapse.rc_message.per_second' documentation and update your configuration accordingly.
  

• I ensured that Synapse runs without error.

• I looked at the generated homeserver.yaml file to ensure that everything looked correct. Here is the generated config (piped through jq for clarity):

Generated config (piped through `jq`)

{
  "account_threepid_delegates": {},
  "allow_guest_access": false,
  "app_service_config_files": [],
  "bcrypt_rounds": 13,
  "database": {
    "args": {
      "database": "/var/lib/matrix-synapse/homeserver.db"
    },
    "name": "sqlite3"
  },
  "dynamic_thumbnails": false,
  "enable_metrics": true,
  "enable_registration": false,
  "enable_registration_captcha": false,
  "event_cache_size": "10K",
  "key_refresh_interval": "1d",
  "listeners": [
    {
      "bind_address": "::1",
      "port": 8008,
      "resources": [
        {
          "compress": false,
          "names": [
            "client",
            "federation"
          ]
        }
      ],
      "tls": false,
      "type": "http",
      "x_forwarded": true
    }
  ],
  "log_config": "/nix/store/2x89r9ji20wy2sggd017jfhlmn9ygsj3-log_config.yaml",
  "max_image_pixels": "32M",
  "max_upload_size": "250M",
  "media_store_path": "/var/lib/matrix-synapse/media",
  "no_tls": false,
  "perspectives": {
    "servers": {
      "matrix.org": {
        "verify_keys": {
          "ed25519:auto": {
            "key": "Noi6WqcDj0QmPxCNQqgezwTlBKrfqehY1u2FyWP9uYw"
          }
        }
      }
    }
  },
  "pid_file": "/run/matrix-synapse.pid",
  "rc_federation": {
    "concurrent": 4,
    "reject_limit": 80,
    "sleep_delay": 404,
    "sleep_limit": 42,
    "window_size": 1024
  },
  "rc_message": {
    "burst_count": 11,
    "per_second": 1.2
  },
  "recaptcha_private_key": "",
  "recaptcha_public_key": "",
  "recaptcha_siteverify_api": "https://www.google.com/recaptcha/api/siteverify",
  "redaction_retention_period": 7,
  "report_stats": false,
  "room_prejoin_state": {
    "additional_event_types": [],
    "disable_default_event_types": false
  },
  "server_name": "localhost",
  "signing_key_path": "/var/lib/matrix-synapse/homeserver.signing.key",
  "turn_shared_secret": "",
  "turn_uris": [],
  "turn_user_lifetime": "1h",
  "verbose": 10
}

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

[sumnerevans]

@ajs124 how did you inspect the test machine and find the contents of the files?

[ajs124]

@ajs124 how did you inspect the test machine and find the contents of the files?

I evaluated the test from master and with this PR applied. Then I used nix-store -qR | grep homeserver.yaml to get the config file derivations and then realized those drvs.

[ajs124]

I'm not sure I'm qualified to approve this, although I guess it seems fine?

[ajs124]

@infinisil (or somebody else), could you maybe approve this addition to lib/strings.nix

[ajs124]

This PR removes the user_creation_max_duration option (not found in synapse codebase at all)

Was removed in matrix-org/synapse@1c4f05d

[sumnerevans]

This PR removes the user_creation_max_duration option (not found in synapse codebase at all)

Was removed in matrix-org/synapse@1c4f05d

Thanks for finding this! I've updated the commit message.

[infinisil]

Instead of doing this, set the defaults in the config section by assigning to services.matrix-synapse directly with mkDefault's. Check out https://nixos.org/manual/nixos/stable/index.html#ex-settings-nix-representable for an example.

[sumnerevans]

I don't think I understand what you mean. I don't think I am setting any defaults here... it's just collapsing all of the individual options that already exist on the module into a single attrset so that it can be converted to YAML.

[infinisil]

Oh right. Still should be done this way though. E.g. see the how the ZNC module provides support for legacy options (ignore the mkOverride 900 bit):

nixpkgs/nixos/modules/services/networking/znc/options.nix

Lines 231 to 261 in 7de613d

	services.znc.config = let
	c = cfg.confOptions;
	# defaults here should override defaults set in the non-legacy part
	mkDefault = mkOverride 900;
	in {
	LoadModule = mkDefault c.modules;
	Listener.l = {
	Port = mkDefault c.port;
	IPv4 = mkDefault true;
	IPv6 = mkDefault true;
	SSL = mkDefault c.useSSL;
	URIPrefix = c.uriPrefix;
	};
	User.${c.userName} = {
	Admin = mkDefault true;
	Nick = mkDefault c.nick;
	AltNick = mkDefault "${c.nick}_";
	Ident = mkDefault c.nick;
	RealName = mkDefault c.nick;
	LoadModule = mkDefault c.userModules;
	Network = mapAttrs (name: net: {
	LoadModule = mkDefault net.modules;
	Server = mkDefault "${net.server} ${optionalString net.useSSL "+"}${toString net.port} ${net.password}";
	Chan = optionalAttrs net.hasBitlbeeControlChannel { "&bitlbee" = mkDefault {}; } //
	listToAttrs (map (n: nameValuePair "#${n}" (mkDefault {})) net.channels);
	extraConfig = if net.extraConf == "" then mkDefault null else net.extraConf;
	}) c.networks;
	extraConfig = [ c.passBlock ];
	};
	extraConfig = optional (c.extraZncConf != "") c.extraZncConf;
	};

The reason it should be done this way is because otherwise if other modules try to access config.services.matrix-synapse.settings, they won't have the full view of the set options, but rather only a partial one that doesn't include the ones from yamlConfig here. And in general recursiveUpdate should be avoided in modules, because the module system already supports merging by itself, and a much better version of it.

[infinisil]

Also check out #75584, which shows how you can use pkgs.formats.yaml for both the type and the generator in one.

And the filterAttrs here looks like it only filters top-level null's out, is that intended like this? And is there a problem with leaving in the nulls?

[mweinelt]

Superseded by #158605.