Stabilize support for MSC2946: spaces summary

[clokep]

Once MSC2946 lands we should (at some point) switch to a stable endpoint and make some updates. There's few bits to this that we need to do:

Phase 1

• Support to the stable endpoint.
• Require limit to be > 0. (This has always been correct.)
• Ensure the endpoint is ratelimited.

Phase 2

• Remove the fallback logic of get_room_hierarchy to request the /spaces endpoint over federation.
• Remove the /spaces client and federation endpoints (and corresponding code in the handlers).
• Update the API responses to remove allowed_spaces (this was replaced with allowed_room_ids).
• Clean-up code around max-children, which is not used by the /hierarchy endpoint.
• Remove room_id from the serialized children_state, it is redundant and removed from the API response.

Phase 3

• Remove unstable /hierarchy client and federation endpoints.
• Remove the typo join_rules field from the response.

[clokep]

I think this work needs to be split into a few pieces.

Phase 1:

• Support the stable endpoint.
• Remove some of the legacy code (the /spaces endpoint? and surrounding bits?)
• Enforce the limit / ratelimiting.

Phase 2:

• Remove the unstable /hierarchy endpoint.

[clokep]

Remove some of the legacy code (the /spaces endpoint? and surrounding bits?)

It looks like these endpoints still get some usage on matrix.org, so might be best to just leave them for now and remove all the unstable code at once.

[clokep]

Client bugs for using the stable endpoint:

• Use the stable hierarcy endpoint from MSC2946 element-hq/element-ios#5144
• Use the stable hierarchy endpoint from MSC2946 element-hq/element-android#4462
• Use the stable hierarchy endpoint from MSC2946 element-hq/element-web#19738

The only other client I know of which implements spaces is Cinny, which uses the matrix-js-sdk.

See also element-hq/element-meta#84.

[clokep]

For phase 2 it is useful to know which version of clients first started using (unstable or stable) /hierarchy instead of /spaces.

• Element Android v1.2.1 (Feature/bca/space explore pagination element-hq/element-android#3926)
• matrix-ios-sdk v0.20.11 (Support pagination in the Space Summary API matrix-ios-sdk#1259)
  • Element iOS: v1.6.9 (https://github.com/vector-im/element-ios/blob/v1.6.9/Podfile#L16)
• matrix-js-sdk v12.4.0 (Add support for /hierarchy API and fall back to /spaces API if needed matrix-js-sdk#1818):
  • Element Web: v1.8.2 (https://github.com/vector-im/element-web/blob/v1.8.2/package.json#L64)
  • Cinny: v1.2.1 (https://github.com/ajbura/cinny/blob/v1.2.1/package.json#L27)

[clokep]

Note that I don't think Element iOS ever shipped a version using /spaces? I think before v1.6.9 it didn't even have spaces support.

[turt2live]

Using our opt-in analytics to determine "market share" of each Element platform (where values at or near zero aren't considered) for the Feb 17th - 22nd period, the following conclusions can be drawn. Note that scraping user agents is likely to be more reliable, however I think given the numbers below it's safe enough to move forward.

TLDR: All platforms are reporting versions which exceed the minimum outlined by previous comments.

• iOS (requires 1.6.9+ to be safe):
  • 1.7.0 is at 21.61%
  • 1.8.1 is at 77.04%
• Android (requires 1.2.1+ to be safe):
  • 1.3.15 is at 7.00%
  • 1.3.18 is at 12.65%
  • 1.4.0 is at 79.89%
• Web (requires 1.8.2+ to be safe):
  • No data (reporting error)
• Desktop (requires 1.8.2+ to be safe):
  • No data (reporting error)

While I don't have data on Element Web/Desktop, 1.8.2 was August 2021 and it's now February - if people haven't updated, I don't hold hope that they will by now.

[clokep]

Thanks for pulling those numbers @turt2live! I'll dust off my branch that removes /spaces support and go from there. 👍

I think we need to leave the unstable /hierarchy endpoint for now since the stable one isn't yet used in clients to my knowledge.

[richvdh]

I think we need to leave the unstable /hierarchy endpoint for now since the stable one isn't yet used in clients to my knowledge.

can we try to at least agree a timescale when it might be possible to change this? one month? two months? six months?

[clokep]

For phase 3 it is useful to know which version of clients first started using the stable /hierarchy endpoint:

• Element Android v1.4.6 (Changes room hierarchy endpoint to stable element-hq/element-android#5443)
• matrix-ios-sdk v0.23.2 (Use the stable hierarchy endpoint from MSC2946 #5144 matrix-ios-sdk#1415)
  • Element iOS: v1.8.11 (https://github.com/vector-im/element-ios/blob/v1.8.11/Podfile#L16)
• matrix-js-sdk v15.4.0 (Use v1 prefix for /hierarchy API, falling back to both previous variants matrix-js-sdk#2022):
  • Element Web: v1.9.9 (https://github.com/vector-im/element-web/blob/v1.9.9/package.json#L63
  • Cinny: v1.2.1 (https://github.com/ajbura/cinny/blob/v1.8.0/package.json#L31)

[clokep]

All known clients using the unstable /hierarchy endpoint have now been updated and had a release.

[clokep]

From taking a look at the latest client numbers it seems that ~93% of Android and ~97% of iOS users are on versions of Element which support the stable endpoint.

[clokep]

I put up #12851 to close this issue out! 🎉

[clokep]

Remove room_id from the serialized children_state, it is redundant and removed from the API response.

This was never done for some reason...even though I checked it off.

[clokep]

This is rather annoying since Synapse validates it over federation...and complement wants it. So I think we need to:

• Remove the validation.
• Update complement: Incorrectly tests the results of MSC2946 /hierarchy response complement#416.
• Remove the field.

[clokep]

Reopening due to #13365 (comment), and the subsequent reversion in #13501.

[clokep]

Reopening due to #13365 (comment), and the subsequent reversion in #13501.

The subsequent fix was in:

• matrix-ios-sdk v0.23.15 (Stop using the room ID from the space child state events. matrix-ios-sdk#1547)
  • Element iOS: v1.8.26 (https://github.com/vector-im/element-ios/blob/v1.8.26/Podfile#L19)

As of now ~90% of the iOS population is on v1.8.26 or higher, I think we'll want to wait a little bit longer since this causes a crash for those who have not updated.

[clokep]

As of today, ~94% of iOS users are on v1.8.26 or higher.