Release 2023-10-19 - (expected chart version 4.39.0)

[zebot]

[2023-10-19] (Chart Release 4.39.0)

Release notes

• New field for Supported protocols in Galley's MLS feature config

  Galley will refuse to start if the list supportedProtocols does not contain
  the value of the field defaultProtocol. Galley will also refuse to start if
  MLS migration is enabled and MLS is not part of supportedProtocols.

  The default value for supportedProtocols is:

  [proteus, mls]
  

(#3374)

API changes

• The JSON schema of NonConnectedBackends has changed to have its single field now called non_connected_backends. (WPB-3798 incorrect json field names #3518)

• Remove de-federation (to avoid a scalability issue). (Disable de-federation (to avoid running into a scalability issue). #3582)

• Replace the placeholder self conversation id with the qualified conversation id for welcome events. ([FS-1915] Add conversation id to welcome messages #3335)

• Add new endpoint DELETE /mls/key-packages/self/:client (Endpoint for deleting key packages #3295)

• Introduce an endpoint for deleting a subconversation ([FS-1334] Reset a Subconversation #2956, [FS-1558] Add a test: delete a subconversation as a conversation member #3119, [FS-1588] No proposals after deleting a subconversation #3123)

• Remove MLS endpoints from API v4 and finalise it (Finalise v4 #3545)

• Add new endpoint GET /conversations/one2one/:domain/:uid to fetch the MLS 1-1 conversation with another user (Add GET endpoint for MLS 1-1 conversations #3345)

• Introduce a subconversation GET endpoint (Add GET endpoint for subconversations #2869, [FS-1214] Add the last commit timestamp for a subconversation #2995)

• Add GET /conversations/:domain/:cid/subconversations/:id/groupinfo endpoint to fetch the group info object for a subconversation (Commit bundles for subconversations #2932)

• Introduce v5 development version (Introduce API v5 #3527)

• It is now possible to use PUT /conversation/:domain/:id/protocol to transition from Mixed to MLS (Mixed to MLS transition #3334)

• Report a failure to add remote users to an MLS conversation ([FS-1148] MLS-specific changes to adding members #3304)

• The key package API has gained a ciphersuite query parameter, which should be the hexadecimal value of an MLS ciphersuite, defaulting to 0x0001. The ciphersuite parameter is used by the claim and count endpoints. For uploads, the API is unchanged, and the ciphersuite is taken directly from the uploaded key package. (Support post-quantum MLS ciphersuite #3454)

• Add MLS migration feature config (MLS migration feature config #3299)

• Switch to MLS draft 20. The following endpoints are affected by the change:

  • All endpoints with message/mls content type now expect and return draft-20 MLS structures.
  • POST /conversations does not require creator_client anymore.
  • POST /mls/commit-bundles now expects a "stream" of MLS messages, i.e. a sequence of TLS-serialised messages, one after the other, in any order. Its protobuf interface has been removed.
  • POST /mls/welcome has been removed. Welcome messages can now only be sent as part of a commit bundle.
  • POST /mls/message does not accept commit messages anymore. All commit messages must be sent as part of a commit bundle. (MLS upgrade #3172)

• Key packages and leaf nodes with x509 credentials are now supported (Support x509 credentials #3532)

Features

• Add reason field to conversation.member-leave ([WPB-4547] Attach the reason for a member to leave a conversation to the leave event #3640)

• Support deleting a remote subconversation ([FS-1334] Reset a Remote Subconversation #2964)

• Introduce support for resetting a subconversation ([FS-1334] Reset a Subconversation #2956)

• Introduce a "mixed" conversation protocol type. A conversation of "mixed" protocol functions as a Proteus converation as well as a MLS conversations. It's intended to be used for migrating conversations from Proteus to MLS. (Introduce "mixed" protocol #3258)

• Added support for post-quantum ciphersuite 0xf031. Correspondingly, MLS groups with a non-default ciphersuite are now supported. The first commit in a group determines the group ciphersuite. (Support post-quantum MLS ciphersuite #3454)

• Remove conversation size limit for MLS conversations (MLS conversation limits #3468)

• Added support for MSL 1-1 conversations (WPB-1928: Add members to MLS one2one conversations #3360)

• MLS application messages for older epochs are now rejected (Reject stale application messages #3438)

• The public key in an x509 credential is now checked against that of the client (Validate public key in an x509 credential #3542)

• Add federated endpoints to get subconversations (Add federated endpoints to get subconversations #2952)

• Add Helm chart (rabbitmq-external) to interface RabbitMQ instances outside of the Kubernetes cluster. (Helm: Push rabbitmq-external to Helm repo #3626)

• Removing or kicking a user from a conversation also removes the user's clients from any subconversation. ([FS-1335] Remove clients from subconversations when user is removed from main conversation #2942)

• Add support for subconversations in POST /mls/commit-bundles (Commit bundles for subconversations #2932)

• Implement endpoint for leaving a subconversation ([FS-1336] Leaving subconversations #2969, [FS-1534] A subconversation leaver gets a remove proposal #3080, [FS-1534] The subconversation non-creator gets a remove proposal when leaving #3085, skip pending proposals included in external commit #3107)

Bug fixes and other updates

• Fix nix derivations for rust packages ([fix] some fixes related to nix #3628)

• Ensure benchmarking dependencies are provided by nix development environment ([fix] some fixes related to nix #3628)

• Disable a guest user from creating a group conversation ([WPB-1908] Unverified user creating conversation #3622)

• Adding users to a conversation now enforces that all federation domains that will be in the conversation are federated with each other. (Wpb 3842 federation completeness check #3514)

• Fix ES migration script. (Fix ES migration script. #3558)

• Fixed add user to conversation when one of the other participating backends is offline (WPB-4629 impossible to add users to a conversation if one of the members is from an offline backend #3585)

• Create a new http2 connection in every federator client request instead of using a shared connection. (federator-client: Use a new http2 connection on every request #3602)

• list-clients returns with partial success even if one of the remote backends is unreachable (WPB-4835 call-between-web-and-android-cant-be-established-when-one-backend-is-not-available #3611)

• Defederation notifications, federation.delete and federation.connectionRemoved, now deduplicate the user list so that we don't send them more notifications than required. (WPB-3916: Filtering out duplicate members when sending defederation notifications #3515)

• Fix bug where notifications for MLS messages were not showing up in all notification streams of clients (Fix MLS message notification bug #3610)

• Map the MLS self-conversation creator's key package reference in Brig ([FS-1488] Fix self-conversation creator key package mapping #3055)

• This fixes a bug where a remote member is removed from a conversation while their backend is unreachable, and the backend does not receive the removal notification once it is reachable again. ([WPB-3664] Bug fix: Notify remote backends of their users removed from conversation when reachable again #3537)

• Welcome messages are not sent anymore to the creator of an MLS group on the first commit (Welcome messages should not be sent to creator #3392)

Documentation

• Fix: support api versions other than v0 in swagger docs. (WPB-1103: Adding relative URLs to swagger docs. #3619)

• Updating the route documentation from Swagger 2 to OpenAPI 3. (WPB-4240: Migrate from swagger2 to openapi3 #3570)

• Elaborate on internal user creation in prod ([WPB-4556] document internal user creation #3596)

• Adding a testing config entry to the PR guidelines. (WPB-5017: Adding a test config file to the PR guidelines. #3624)

Internal changes

• remove leaving clients immediately from subconversations ([FS-1564] no messages sent to clients who left subconv #3096)

• Servantify internal end-points: brig/teams ([feat] improve type safety for Named, servantify brig internal route #3634)

• add conversation type to group ID serialisation (add conversation type to group ID serialisation #3344)

• Do not cache federation remote configs on non-brig services (WPB-3797 do not cache federation remote domain config #3612)

• JSON derived schemas have been changed to no longer pre-process record fields to drop prefixes that were required to disambiguate fields.
  Prefix processing still exists to drop leading underscores from field names, as we are using prefixed field names with makeLenses.
  Code has been updated to use OverloadedRecordDot with the changed field names. (WPB-3798 incorrect json field names #3518)

• Updating the route documentation library from swagger2 to openapi3.

  This also introduced a breaking change in how we track what federation calls each route makes.
  The openapi3 library doesn't support extension fields, and as such tags are being used instead in a similar way. (WPB-4240: Migrate from swagger2 to openapi3 #3570)

• • Extending the information returned in errors for Federator. Paths and response bodies, if available, are included in error logs.
  • Prometheus metrics for outgoing and incoming federation requests added. They can be enabled by setting metrics.serviceMonitor.enabled, like in other charts. ([WPB-4406] federator improve logging #3556)

• CLI tool to consume messages from a RabbitMQ queue (WPB-4748 create a way to inspect background notifications queue #3589, WPB-5069 Docs for rabbitmq-consumer #3655)

• Removed user and client threshold fields from mls migration feature. (Remove user & client thresholds from MLS migration #3364)

• Include timestamp in s3 upload path for test logs (WPB-4910 include build timestamp in s3 upload path for test logs #3621)

• Migrating the following routes to the Servant API form.

  POST /provider/services
  GET /provider/services
  GET /provider/services/:sid
  PUT /provider/services/:sid
  PUT /provider/services/:sid/connection
  DELETE /provider/services/:sid
  GET /providers/:pid/services
  GET /providers/:pid/services/:sid
  GET /services
  GET /services/tags
  GET /teams/:tid/services/whitelisted
  POST /teams/:tid/services/whitelist (WPB-633 Servantify Brig/Provider.Service API #3554)

• Provider API has been migrated to servant ([WPB-664] Servantify brig Provider API #3547)

• background-worker: Get list of domains from RabbitMQ instead of brig for pushing backend notifications (Fix federationStrategy 'allowAll' #3588)

• Avoid including MLS application messages in the sender client's event stream. (Avoid MLS messages in sender client's event stream #3379)

• Avoid empty pushes when chunking pushes in galley (Avoid empty pushes from galley #3646)

• Introduce a Galley DB table for subconversations (Add GET endpoint for subconversations #2869)

• Support mapping MLS group IDs to subconversations (Add GET endpoint for subconversations #2869)

• change version and conversation type to 16 bit in group ID serialisation (alter group ID serialisation #3353)

• Brig does not perform key package ref mapping anymore. Claimed key packages are simply removed from the mls_key_packages table. The mls_key_package_refs table is now unused, and will be removed in the future. (MLS upgrade #3172)

• Add intermediate "mixed" protocol for migrating from Proteus to MLS (Refactor protocol update as conversation actions #3292)

• • Do not perform client checks for add and remove proposals in mixed conversations
  • Restrict protocol updates to team conversations
  • Disallow MLS application messages in mixed conversations
  • Send remove proposals when users leave mixed conversations (FS-1901: Missing bits for mixed protocol state #3303)

• New cron job to save data usable to watch the progress of the Proteus to MLS migration in S3 bucket.

  IMPORTANT: This cron job is not meant for general use! It can leak data about one team to other teams. (Save MLS migration statistics in S3 bucket #3579)

• Subconversations are now created on their first commit (Create subconversations on first commit #3355)

• Propagate messages in MLS subconversations (FS-902: Fix empty client set & set subconv field in message propagation #2937)

• Move some MLS tests to new integration suite (Port MLS test framework to new integration suite #3286)

• Check validity of notification IDs in the notification API (Check validity of notification IDs #3550)

• stern: Optimize RAM usage of /i/users/meta-info (stern: Optimize RAM usage of /i/users/meta-info #3522)

• Additional integration test for federated connections ([WPB-3799] cannot fetch conversation details after connection request #3538)

• The bot API is now migrated to servant ([WPB-662] servantify brig provider bot api #3540)

• rusty-jwt-tools is upgraded to version 0.5.0 (WPB-4361 upgrade jwt tools in wire server #3572)

• Refactored schema version tracking from manually managed to automatic. ([WPB-3138] Moved gundeck schema migrations to gundeck lib. #3643)

• Avoid unnecessary error logs on service shutdown (Cleanup error logs on shutdown #3592)

• Introduce an effect for subconversations (Add GET endpoint for subconversations #2869)

• Via the update path update the key package of the committer in epoch 0 of a subconversation ([FS-901] Update the key package reference mapping of the creator of the subconversation #2975)

• Add more tests for joining a subconversation ([FS-901] Tests for joining subconversation #2974)

• Added /tools/db/repair-brig-clients-table to clean up after the fix in [WPB-3888] Make sure cassandra updates do not re-introduce removed content. #3504 (Add repair-brig-clients-table to clean up after the fix in #3504. #3507)

• Distinguish between update and upsert cassandra commands (follow-up to [WPB-3888] Make sure cassandra updates do not re-introduce removed content. #3504) (Fix involuntary cassandra upserts #3513)

• Truncate galley.mls_group_member_client table and drop galley.member_client table.

  The data in mls_group_member_client could contain nulls from client testing in prod. So, its OK to truncate it.
  The member_client table is unused. (galley: Cleanup garbage data in mls_group_member_client table and drop unused member_client table #3648)

• All integration tests can generate XML reports.

  To generate the report in brig-integration, galley-integration,
  cargohold-integration, gundeck-integration, stern-integration and the new
  integration suite pass --xml=<outfile> to generate the XML file.

  For spar-integration and federator-integration pass -f junit and set
  JUNIT_OUTPUT_DIRECTORY and JUNIT_SUITE_NAME environment variables. The XML
  report will be generated at $JUNIT_OUTPUT_DIRECTORY/junit.xml.

  (integration-tests: Allow generating tests results in ant xml format #3568, integration-tests: Allow uploading XML results to S3 #3633)

Federation changes

• Add subconversation ID to onMLSMessageSent request payload. (Add subconversation ID to RemoteMLSMessage #3270)

• Derive group ID from qualified conversation ID and, if applicable,
  subconversation ID.

  Retire mapping from group IDs to conversation IDs. (group_id_conv_id)

  Remove federation endpoints

  • on-new-remote-conversation,
  • on-new-remote-subconversation, and
  • on-delete-mls-conversation
    which were used to synchronise the group to conversation mapping. (Derive group ID from qualified conversation ID and, if applicable, subconversation ID #3309)

• Introduce an endpoint for resetting a remote subconversation ([FS-1334] Reset a Remote Subconversation #2964)

• Split federation endpoint into on-new-remote-conversation and on-new-remote-subconversation
  Call on-new-remote-subconversation when a new subconversation is created
  Call on-new-remote-subconversation for all existing subconversations when a new backend gets involved
  Call on-new-remote-subconversation when a subconversation is reset (extend on-new-remote-conversation by the subconv ID #2997)

• federator: Allow setting TCP connection timeout for HTTP2 requests

  The helm chart defaults it to 5s which should be best for most installations. (federator: Allow setting TCP connection timeout for HTTP2 requests #3595)

• Constrain which federation endpoints can be used via the queueing federation client ([WPB-3867] Use queueing federation client for a federation API endpoint #3629)

• There is a breaking change in the "on-mls-message-sent" federation endpoint due to queueing. Now that there is retrying because of queueing, the endpoint can no longer respond with a list of unreachable users. ([WPB-3867] Use queueing federation client for a federation API endpoint #3629)

• Remote MLS messages get queued via RabbitMQ ([WPB-4984] Queue remote MLS messages #3635)

[CLAassistant]

All committers have signed the CLA.