Update pyproject.toml to be compatible with other standard Python packaging tools

[anoadragon453]

The previous fields were all poetry-specific. Since then, these have moved into a standard set of "project metadata" attributes (PEP 621). These are compatible with other standard Python packaging tools.

The motivation for this change was to have cibuildwheel pick up on the minimum python requirement when choosing Python versions to build wheel for (see #19119 (comment)). It won't recognise tool.poetry.dependencies.python, but it will pick up project.requires-python.

We can't just have a [project] field with requires-python however (as name is required, etc.). So we convert everything at once.

Merging into the release-v1.142 branch as this is currently blocking release assets from being built.

We can see that cibuildwheel is now picking up the correct python version requirements in a CI run of this PR. Whereas previously, it found no Python version requirements.

This PR requires an equivalent Sytest PR, which bumps the poetry version to 2.x: matrix-org/sytest#1417. That PR must be merged and new matrixdotorg/sytest-synapse images pushed to registries before this PR will pass.

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
  • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
• Code style is correct (run the linters)

[reivilibre]

seems reasonable if it appears to work

[anoadragon453]

Now that I see how many knock-on effects this has had, I'm going to defer this to 1.143.0 as while it will fix the cibuildwheel release CI, #19119 would also do so with a less invasive change.

[anoadragon453]

Quite a few things changed (including the base branch, as this is now slated for Synapse 1.143.0), so re-requesting review @reivilibre. I needed to migrate the project dependencies as well (else poetry check failed).

Review commit-by-commit from 44d2717 to start from where you last left off.

[anoadragon453]

(requesting review from the rest of the team, as @reivilibre is currently away.)

[MadLittleMods]

Prior art but I'm confused why we support pyOpenSSL = ">=16.0.0" if we actually need version 17.0 for things to work. Seems like we should bump our minimum version.

[anoadragon453]

Good point. It looks like most distros have moved on from such an ancient version as well, supporting at least 20.0.01: https://repology.org/project/python%3Apyopenssl/versions

I can bump this and remove the specific sed line in a separate PR.

[MadLittleMods]

To verify this, we can look at the output of the trial-olddeps job,

https://github.com/element-hq/synapse/actions/runs/19237403138/job/55126824973?pr=19137#step:7:28

Patched pyproject.toml

Patched pyproject.toml
  + cat pyproject.toml
  [project]
  name = "matrix-synapse"
  version = "1.142.0rc4"
  description = "Homeserver for the Matrix decentralised comms protocol"
  readme = "README.rst"
  authors = [
      { name = "Matrix.org Team and Contributors", email = "packages@matrix.org" }
  ]
  requires-python = ">=3.10.0,<4.0.0"
  license = "AGPL-3.0-or-later OR LicenseRef-Element-Commercial"
  classifiers = [
      "Development Status :: 5 - Production/Stable",
      "Topic :: Communications :: Chat",
  ]
  
  # Mandatory Dependencies
  dependencies = [
      # we use the TYPE_CHECKER.redefine method added in jsonschema 3.0.0
      "jsonschema==3.0.0",
      # 0.25.0 is the first version to support Python 3.14.
      # We can remove this once https://github.com/python-jsonschema/jsonschema/issues/1426 is fixed
      # and included in a release.
      "rpds-py==0.25.0",
      # We choose 2.0 as a lower bound: the most recent backwards incompatible release.
      # It seems generally available, judging by https://pkgs.org/search/?q=immutabledict
      "immutabledict==2.0",
      # We require 2.1.0 or higher for type hints. Previous guard was == 1.1.0
      "unpaddedbase64==2.1.0",
      # We require 2.0.0 for immutabledict support.
      "canonicaljson==2.0.0,<3.0.0",
      # we use the type definitions added in signedjson 1.1.
      "signedjson==1.1.0,<2.0.0",
      # validating SSL certs for IP addresses requires service_identity 18.1.
      "service-identity==18.1.0",
      # Twisted 18.9 introduces some logger improvements that the structured
      # logger utilises
      # Twisted 19.7.0 moves test helpers to a new module and deprecates the old location.
      # Twisted 21.2.0 introduces contextvar support.
      # We could likely bump this to 22.1 without making distro packagers'
      # lives hard (as of 2025-07, distro support is Ubuntu LTS: 22.1, Debian stable: 22.4,
      # RHEL 9: 22.10)
      "Twisted[tls]==21.2.0",
      "treq==21.5.0",
      # Twisted has required pyopenssl 16.0 since about Twisted 16.6.
      "pyOpenSSL==17.0.0",
      "PyYAML==5.3",
      "pyasn1==0.1.9",
      "pyasn1-modules==0.0.7",
      "bcrypt==3.1.7",
      # 10.0.1 minimum is mandatory here because of libwebp CVE-2023-4863.
      # Packagers that already took care of libwebp can lower that down to 5.4.0.
      "Pillow==10.0.1",
      # We use SortedDict.peekitem(), which was added in sortedcontainers 1.5.2.
      # 2.0.5 updates collections.abc imports to avoid Python 3.10 incompatibility.
      "sortedcontainers==2.0.5",
      "pymacaroons==0.13.0",
      "msgpack==0.5.2",
      "phonenumbers==8.2.0",
      # we use GaugeHistogramMetric, which was added in prom-client 0.4.0.
      # `prometheus_client.metrics` was added in 0.5.0, so we require that too.
      # We chose 0.6.0 as that is the current version in Debian Buster (oldstable).
      "prometheus-client==0.6.0",
      # we use `order`, which arrived in attrs 19.2.0.
      # Note: 21.1.0 broke `/sync`, see https://github.com/matrix-org/synapse/issues/9936
      "attrs==19.2.0,!=21.1.0",
      "netaddr==0.7.18",
      # Jinja 2.x is incompatible with MarkupSafe==2.1. To ensure that admins do not
      # end up with a broken installation, with recent MarkupSafe but old Jinja, we
      # add a lower bound to the Jinja2 dependency.
      "Jinja2==3.0",
      # 3.2.0 updates collections.abc imports to avoid Python 3.10 incompatibility.
      "bleach==3.2.0",
      # pydantic 2.12 depends on typing-extensions==4.14.1
      "typing-extensions==4.14.1",
      # We enforce that we have a `cryptography` version that bundles an `openssl`
      # with the latest security patches.
      "cryptography==3.4.7",
      # ijson 3.1.4 fixes a bug with "." in property names
      "ijson==3.1.4",
      "matrix-common==1.3.0,<2.0.0",
      # We need packaging.verison.Version(...).major added in 20.0.
      "packaging==20.0",
      "pydantic==2.8;python_version < '3.14'",
      "pydantic==2.12;python_version == '3.14'",
  
      # This is for building the rust components during "poetry install", which
      # currently ignores the `build-system.requires` directive (c.f.
      # https://github.com/python-poetry/poetry/issues/6154). Both `pip install` and
      # `poetry build` do the right thing without this explicit dependency.
      #
      # This isn't really a dev-dependency, as `poetry install --without dev` will fail,
      # but the alternative is to add it to the main list of deps where it isn't
      # needed.
      "setuptools_rust==1.3",
  
      # This is used for parsing multipart responses
      "python-multipart==0.0.9",
  ]
  
  [project.optional-dependencies]
  matrix-synapse-ldap3 = ["matrix-synapse-ldap3==0.1"]
  postgres = [
  ]
  saml2 = ["pysaml2==4.5.0"]
  oidc = ["authlib==0.15.1"]
  url-preview = ["lxml==4.6.3"]
  sentry = ["sentry-sdk==0.7.2"]
  opentracing = ["jaeger-client==4.2.0", "opentracing==2.2.0"]
  jwt = ["authlib"]
  # hiredis is not a *strict* dependency, but it makes things much faster.
  # (if it is not installed, we fall back to slow code.)
  redis = ["txredisapi==1.4.7", "hiredis"]
  # Required to use experimental `caches.track_memory_usage` config option.
  cache-memory = ["pympler"]
  test = ["parameterized==0.7.4", "idna==2.5"]
  
  # The duplication here is awful. I hate hate hate hate hate it. However, for now I want
  # to ensure you can still `pip install matrix-synapse[all]` like today. Two motivations:
  # 1) for new installations, I want instructions in existing documentation and tutorials
  #    out there to still work.
  # 2) I don't want to hard-code a list of extras into CI if I can help it. The ideal
  #    solution here would be something like https://github.com/python-poetry/poetry/issues/3413
  #
  # NB: the strings in this list must be *package* names, not extra names.
  # Some of our extra names _are_ package names, which can lead to great confusion.
  #
  # NB2: This is primarily a poetry limitation. pip has allowed placing extras
  # names since 2021. `uv` also supports this. So we could eliminate this if we
  # switched to `uv`.
  all = [
      # matrix-synapse-ldap3
      "matrix-synapse-ldap3==0.1",
      # postgres
      # saml2
      "pysaml2==4.5.0",
      # oidc and jwt
      "authlib==0.15.1",
      # url-preview
      "lxml==4.6.3",
      # sentry
      "sentry-sdk==0.7.2",
      # opentracing
      "jaeger-client==4.2.0", "opentracing==2.2.0",
      # redis
      "txredisapi==1.4.7", "hiredis",
      # cache-memory
      "pympler",
      # omitted:
      #   - test: it's useful to have this separate from dev deps in the olddeps job
  ]
  
  [project.urls]
  repository = "https://github.com/element-hq/synapse"
  documentation = "https://element-hq.github.io/synapse/latest"
  "Issue Tracker" = "https://github.com/element-hq/synapse/issues"
  
  [project.scripts]
  synapse_homeserver = "synapse.app.homeserver:main"
  synapse_worker = "synapse.app.generic_worker:main"
  synctl = "synapse._scripts.synctl:main"
  
  export_signing_key = "synapse._scripts.export_signing_key:main"
  generate_config = "synapse._scripts.generate_config:main"
  generate_log_config = "synapse._scripts.generate_log_config:main"
  generate_signing_key = "synapse._scripts.generate_signing_key:main"
  hash_password = "synapse._scripts.hash_password:main"
  register_new_matrix_user = "synapse._scripts.register_new_matrix_user:main"
  synapse_port_db = "synapse._scripts.synapse_port_db:main"
  synapse_review_recent_signups = "synapse._scripts.review_recent_signups:main"
  update_synapse_database = "synapse._scripts.update_synapse_database:main"
  
  
  [tool.towncrier]
      package = "synapse"
      filename = "CHANGES.md"
      directory = "changelog.d"
      issue_format = "[\\#{issue}]([https://github.com/element-hq/synapse/issues/{issue}](https://github.com/element-hq/synapse/issues/%7Bissue%7D))"
  
      [[tool.towncrier.type]]
          directory = "feature"
          name = "Features"
          showcontent = true
  
      [[tool.towncrier.type]]
          directory = "bugfix"
          name = "Bugfixes"
          showcontent = true
  
      [[tool.towncrier.type]]
          directory = "docker"
          name = "Updates to the Docker image"
          showcontent = true
  
      [[tool.towncrier.type]]
          directory = "doc"
          name = "Improved Documentation"
          showcontent = true
  
      [[tool.towncrier.type]]
          directory = "removal"
          name = "Deprecations and Removals"
          showcontent = true
  
      [[tool.towncrier.type]]
          directory = "misc"
          name = "Internal Changes"
          showcontent = true
  
  [tool.ruff]
  line-length = 88
  target-version = "py310"
  
  [tool.ruff.lint]
  # See https://beta.ruff.rs/docs/rules/#error-e
  # for error codes. The ones we ignore are:
  #  E501: Line too long (black enforces this for us)
  #  E731: do not assign a lambda expression, use a def
  #
  # flake8-bugbear compatible checks. Its error codes are described at
  # https://beta.ruff.rs/docs/rules/#flake8-bugbear-b
  #  B023: Functions defined inside a loop must not use variables redefined in the loop
  ignore = [
      "B023",
      "E501",
      "E731",
  ]
  select = [
      # pycodestyle
      "E",
      "W",
      # pyflakes
      "F",
      # isort
      "I001",
      # flake8-bugbear
      "B0",
      # flake8-comprehensions
      "C4",
      # flake8-2020
      "YTT",
      # flake8-slots
      "SLOT",
      # flake8-debugger
      "T10",
      # flake8-pie
      "PIE",
      # flake8-executable
      "EXE",
      # flake8-logging
      "LOG",
      # flake8-logging-format
      "G",
      # pyupgrade
      "UP006",
      "UP007",
      "UP045",
  ]
  extend-safe-fixes = [
      # pyupgrade rules compatible with Python == 3.9
      "UP006",
      "UP007",
      # pyupgrade rules compatible with Python == 3.10
      "UP045",
  ]
  
  [tool.ruff.lint.isort]
  combine-as-imports = true
  section-order = ["future", "standard-library", "third-party", "twisted", "first-party", "testing", "local-folder"]
  known-first-party = ["synapse"]
  
  [tool.ruff.lint.isort.sections]
  twisted = ["twisted", "OpenSSL"]
  testing = ["tests"]
  
  [tool.ruff.format]
  quote-style = "double"
  indent-style = "space"
  skip-magic-trailing-comma = false
  line-ending = "auto"
  
  [tool.maturin]
  manifest-path = "rust/Cargo.toml"
  module-name = "synapse.synapse_rust"
  
  [tool.poetry]
  packages = [
      { include = "synapse" },
  ]
  include = [
      { path = "AUTHORS.rst", format = "sdist" },
      { path = "book.toml", format = "sdist" },
      { path = "changelog.d", format = "sdist" },
      { path = "CHANGES.md", format = "sdist" },
      { path = "CONTRIBUTING.md", format = "sdist" },
      { path = "demo", format = "sdist" },
      { path = "docs", format = "sdist" },
      { path = "INSTALL.md", format = "sdist" },
      { path = "mypy.ini", format = "sdist" },
      { path = "scripts-dev", format = "sdist" },
      { path = "synmark", format="sdist" },
      { path = "sytest-blacklist", format = "sdist" },
      { path = "tests", format = "sdist" },
      { path = "UPGRADE.rst", format = "sdist" },
      { path = "Cargo.toml", format = "sdist" },
      { path = "Cargo.lock", format = "sdist" },
      { path = "rust/Cargo.toml", format = "sdist" },
      { path = "rust/build.rs", format = "sdist" },
      { path = "rust/src/**", format = "sdist" },
  ]
  exclude = [
      { path = "synapse/*.so", format = "sdist"}
  ]
  
  [tool.poetry.build]
  script = "build_rust.py"
  generate-setup-file = true
  
  [tool.poetry.group.dev.dependencies]
  # We pin development dependencies in poetry.lock so that our tests don't start
  # failing on new releases. Keeping lower bounds loose here means that dependabot
  # can bump versions without having to update the content-hash in the lockfile.
  # This helps prevents merge conflicts when running a batch of dependabot updates.
  ruff = "0.14.3"
  
  # Typechecking
  lxml-stubs = "==0.4.0"
  mypy = "*"
  mypy-zope = "*"
  types-bleach = "==4.1.0"
  types-jsonschema = "==3.2.0"
  types-netaddr = "==0.8.0.6"
  types-opentracing = "==2.4.2"
  types-Pillow = "==8.3.4"
  types-pyOpenSSL = "==20.0.7"
  types-PyYAML = "==5.4.10"
  types-requests = "==2.26.0"
  types-setuptools = "==57.4.0"
  
  # Dependencies which are exclusively required by unit test code. This is
  # NOT a list of all modules that are necessary to run the unit tests.
  # Tests assume that all optional dependencies are installed.
  # parameterized<0.7.4 can create classes with names that would normally be invalid
  # identifiers. trial really does not like this when running with multiple workers.
  parameterized = "==0.7.4"
  idna = "==2.5"
  
  # The following are used by the release script
  click = "==8.1.3"
  # GitPython was == 3.1.14; bumped to 3.1.20, the first release with type hints.
  GitPython = "==3.1.20"
  markdown-it-py = "==3.0.0"
  pygithub = "==1.59"
  + echo ::endgroup::
  # The following are executed as commands by the release script.
  twine = "*"
  # Towncrier min version comes from https://github.com/matrix-org/synapse/pull/3425. Rationale unclear.
  towncrier = "==18.6.0rc1"
  
  # Used for checking the Poetry lockfile
  tomli = "==1.2.3"
  
  
  [build-system]
  # The upper bounds here are defensive, intended to prevent situations like
  # https://github.com/matrix-org/synapse/issues/13849 and
  # https://github.com/matrix-org/synapse/issues/14079 where we see buildtime or
  # runtime errors caused by build system changes.
  # We are happy to raise these upper bounds upon request,
  # provided we check that it's safe to do so (i.e. that CI passes).
  requires = ["poetry-core==2.0.0,<=2.1.3", "setuptools_rust==1.3,<=1.11.1"]
  build-backend = "poetry.core.masonry.api"
  
  
  [tool.cibuildwheel]
  # Skip unsupported platforms (by us or by Rust).
  # See https://cibuildwheel.readthedocs.io/en/stable/options/#build-skip for the list of build targets.
  # We skip:
  #  - CPython 3.8: EOLed
  #  - musllinux i686: excluded to reduce number of wheels we build.
  #    c.f. https://github.com/matrix-org/synapse/pull/12595#discussion_r963107677
  skip = "cp38* *-musllinux_i686"
  # Enable non-default builds.
  # "pypy" used to be included by default up until cibuildwheel 3.
  enable = "pypy"
  
  # We need a rust compiler.
  #
  # We temporarily pin Rust to 1.82.0 to work around
  # https://github.com/element-hq/synapse/issues/17988
  before-all =  "sh .ci/before_build_wheel.sh"
  environment= { PATH = "$PATH:$HOME/.cargo/bin" }
  
  # For some reason if we don't manually clean the build directory we
  # can end up polluting the next build with a .so that is for the wrong
  # Python version.
  before-build = "rm -rf {project}/build"
  build-frontend = "build"
  test-command = "python -c 'from synapse.synapse_rust import sum_as_string; print(sum_as_string(1, 2))'"
  
  
  [tool.cibuildwheel.linux]
  # Wrap the repair command to correctly rename the built cpython wheels as ABI3.
  repair-wheel-command = "./.ci/scripts/auditwheel_wrapper.py -w {dest_dir} {wheel}"
  
  [tool.cibuildwheel.macos]
  # Wrap the repair command to correctly rename the built cpython wheels as ABI3.
  repair-wheel-command = "./.ci/scripts/auditwheel_wrapper.py --require-archs {delocate_archs} -w {dest_dir} {wheel}"

[MadLittleMods]

Assuming this is a bunch of copy-paste to the new format

[MadLittleMods]

Is this still true with Poetry when using the new PEP standards to define things?

(feel free to ignore as prior art and something to look into in the future)

[anoadragon453]

After some digging, I found https://peps.python.org/pep-0735/ which defines a standard for dependency groups. It also specifically covers our use case:

[dependency-groups]
test = ["pytest", "coverage"]
docs = ["sphinx", "sphinx-rtd-theme"]
typing = ["mypy", "types-requests"]
typing-test = [{include-group = "typing"}, {include-group = "test"}, "useful-types"]

and poetry supports this PEP since 2.2.0! Poetry's documentation for it is here. Our desired use case is discussed in https://python-poetry.org/docs/managing-dependencies/#including-dependencies-from-other-groups.

So this would allow us to specify all without needing to duplicate our dependency list 🥳 But it does mean we'd need to change our documentation to instead say poetry install matrix-synapse --with all.

And if we're going to update the documentation, might it make more sense to do that during a possible uv transition instead?

---

Something to figure out in a separate PR. For now, I've included this information in the comment: 79c49f7.

[MadLittleMods]

Similar comment as I made in #18432 (comment)

Is "Bug Tracker" idiomatic? Big projects doing this? Something that PyPI supports?

Is it really necessary over when we already have the homepage and repository links and issues are a click away

[anoadragon453]

It's freeform text, but PyPI will show a little "bug" icon for "Bug Tracker" or "Issue Tracker".

Big projects indeed do this, see i.e. PyTorch (source). Personally I find it convenient to go straight to the issues tab from PyPI (and, I suppose, it's useful if one is unfamiliar with GitHub).