Spec

Hey all,

Once again it's been a little while since we've done a spec release (sorry; we're aiming for quarterly releases from here on out), but this time we have some pretty big news: we've got an all-new spec platform and a new versioning scheme. The new spec platform has been needed for a long time to make better sense of what Matrix is, and as part of getting that out the door we reduced the number of "Matrix versions" to just one.

Huge thanks to Will Bamberg for building it out for us, anoa for working out the deployment details, and everyone for testing it all. They talk at length about what this specification even is and about the platform itself on Matrix Live S6E19. It's the single greatest improvement to the spec we've seen to date.

The new versioning scheme presents the whole specification as a single document, making it easier to check compatibility between implementations and the spec itself. Previously, a grid would have to be drawn to show whether Server-Server r0.1.4 is compatible with Client-Server r0.6.1 - while obvious at release time, the historical context gets lost quite easily. Now, with a single version number, the entire specification is compatible across a single version number: servers implementing Matrix 1.1 will be compatible with clients implementing v1.1, and vice versa. For the specification itself, this means we no longer have to carefully point and update links between the APIs, as they'll instead be versioned together.

Changing the versioning of the specification does have an impact on the Client-Server API in particular, however. You may have noticed that Client-Server APIs are currently versioned at "r0". By removing rX.Y.Z versioning, all of the endpoints suddenly don't have a version to reference. All endpoints across the specification are now versioned individually to allow for breaking changes at the endpoint level. They no longer require the whole specification to be listed as a breaking change: a v1 endpoint can get additions/changes which are backwards compatible, but if we need to change format (for example) then it'll get bumped up to v2, leaving v1 in its last known state.

For the Client-Server API, a slight complication is that v1 and v2 (alpha) are already versions familiar to those that have been around for a while - to avoid confusing those people, existing Client-Server API endpoints will start at v3. New endpoints introduced after v1.1 will start at v1 instead.

It's been well over a year since the last release, which means there's a whole lot of features and changes to cover. Here we'll try to cover the things most clients/servers will have to worry about, but we do still recommend reading through the changelog included below. Overall, 28 MSCs have been merged through this release, but some have had to be excluded in the interest of getting the spec release out there. Many of them are expected to be in the next anticipated release (which should hopefully be next quarter).

Clients: There's a lot of stuff

In Matrix 1.1, client developers get all sorts of new features to play with. Clients which support end-to-end encryption should explore key backups, cross-signing, SSSS, and breaking changes to verification. Quite a lot of this stuff has existed for a while, but has made it into the specification formally now. As an added bonus, the emoji for SAS verification have been translated (contribute here).

Knocking has also landed in the spec (thanks Sorunome for leading the charge on this!), opening up the ability for rooms to optionally allow people to request invites to join. This can be helpful for semi-private rooms where it can be easier to approve/deny requests compared to finding someone's MXID and manually inviting them. This does require a v7 room to work, however.

Thanks again to Sorunome, Message Spoilers have been officially included in Matrix. People can now safely discuss the ending to the latest movie without being banned for spoilers. Though, as a new feature, there's a chance that the spoiler text still gets included in the message: clients should update as soon as possible to avoid their users accidentally getting banned for spoiling the conclusion to the Spaces saga ;)

There's a few other smaller improvements/additions, and of course the regular "various clarifications and bug fixes" to take a look at. For a sample checklist, check out element-web's issue on the subject.

Servers: Knock knock, it's a new room version

Room version 7 has landed, bringing forth the ability for users to knock on rooms (requesting an invite to join). The changes are largely scoped to using the reserved keywords for join rules and membership, and are described through the auth rules. Thankfully, the changes over v6 are minimally invasive so should be quick to implement.

Additionally, the cross-signing bits have been included in the API responses and EDU definitions. Matthew's blog post from last year (it really has been that long...) describes cross-signing and the history of its implementation.

As per usual, there's also various specification errors corrected to aid understanding. Synapse has an exhaustive issue to detail what servers might need to do.

PS: Room versions 8 and 9 are also out there, but will be included in a future spec release.

The full changelog

We haven't mentioned identity servers, bridges, etc in this post but they have changes too! Below is the whole changelog, the entire year and a bit of it. Thank you to everyone who has submitted MSCs, and congratulations on getting them released. If we forgot yours, please mention it in #matrix-spec:matrix.org so we can apologize and correct.

PS: The MSC process is how changes to Matrix are made, and you (yes, you) can propose those changes too. Check out the Matrix Live episode where Matthew talks about how this process works, and how we avoid blocking clients from implementing their proposals behind the relatively slow spec release cycles.

Client-Server API

Breaking Changes

• Document curve25519-hkdf-sha256 key agreement method for SAS verification, and deprecate old method as per MSC2630. (#2687)
• Add m.key.verification.ready and m.key.verification.done to key verification framework as per MSC2366. (#3139)

Deprecations

• Deprecate starting verifications that don't start with m.key.verification.request as per MSC3122. (#3199)

New Endpoints

• Add key backup (/room_keys/*) endpoints as per MSC1219. (#2387, #2639)
• Add POST /keys/device_signing/upload and POST /keys/signatures/upload as per MSC1756. (#2536)
• Add /knock endpoint as per MSC2403. (#3154)
• Add /login/sso/redirect/{idpId} as per MSC2858. (#3163)

Removed Endpoints

• Remove unimplemented m.login.oauth2 and m.login.token user-interactive authentication mechanisms as per MSC2610 and MSC2611. (#2609)

Backwards Compatible Changes

• Document how clients can advise recipients that it is withholding decryption keys as per MSC2399. (#2399)
• Add cross-signing properties to the response of POST /keys/query as per MSC1756. (#2536)
• Document Secure Secret Storage and Sharing as per MSC1946 and MSC2472. (#2597)
• Add a device_id parameter to login fallback as per MSC2604. (#2709)
• Added a common set of translations for SAS Emoji. (#2728)
• Added support for reason on all membership events and related endpoints as per MSC2367. (#2795)
• Add a 404 M_NOT_FOUND error to push rule endpoints as per MSC2663. (#2796)
• Make reason and score parameters optional in the content reporting API as per MSC2414. (#2807)
• Allow guests to get the list of members for a room as per MSC2689. (#2808)
• Add support for spoilers as per MSC2010 and MSC2557, and color attribute as per MSC2422. (#3098)
• Add <details> and <summary> to the suggested HTML subset as per MSC2184. (#3100)
• Add key verification using in-room messages as per MSC2241. (#3139, #3150)
• Add information about using SSSS for cross-signing and key backup. (#3147)
• Add key verification method using QR codes as per MSC1544. (#3149)
• Document how clients can simplify usage of Secure Secret Storage as per MSC2874. (#3151)
• Add support for knocking, as per MSC2403. (#3154, #3254)
• Multiple SSO providers are possible through m.login.sso as per MSC2858. (#3163)
• Add device_id to /account/whoami response as per MSC2033. (#3166)
• Downgrade identity server discovery failures to FAIL_PROMPT as per MSC2284. (#3169)
• Re-version all endpoints to be v3 as a starting point instead of r0 as per MSC2844. (#3421)

Spec Clarifications

• Fix issues with age and unsigned being shown in the wrong places. (#2591)
• Fix definitions for room version capabilities. (#2592)
• Fix various typos throughout the specification. (#2594, #2599, #2809, #2878, #2885, #2888, #3116, #3339)
• Clarify link to OpenID Connect specification. (#2605)
• Clarify the behaviour of SSO login and UI-Auth. (#2608)
• Remove spurious room_id from /sync examples. (#2629)
• Reorganize information in Push Notifications module for clarity. (#2634)
• Improve consistency and clarity of event schema titles. (#2647)
• Fix schema issues in m.key.verification.accept and secret storage. (#2653)
• Reword "UI Authorization" to "User-Interactive Authentication" to be more clear. (#2667)
• Fix schemas for push rule actions to represent their alternative object form. (#2669)
• Fix usage of highlight tweak for consistency. (#2670)
• Clarify the behaviour of state for /sync with lazy-loading. (#2754)
• Clarify description of m.room.redaction event. (#2814)
• Mark messages as a required JSON body field in PUT /_matrix/client/r0/sendToDevice/{eventType}/{txnId} calls. (#2928)
• Correct examples of client_secret request body parameters so that they do not include invalid characters. (#2985)
• Fix example MXC URI for m.presence. (#3091)
• Clarify that event bodies are untrusted, as per MSC2801. (#3099)
• Fix the maximum event size restriction (65535 bytes -> 65536). (#3127)
• Update Access-Control-Allow-Headers recommendation to fit CORS specification. (#3225)
• Explicitly state that replacement_room is a room ID in m.room.tombstone events. (#3233)
• Clarify that all request bodies are required. (#3238, #3332)
• Add missing titles to some scheams. (#3330)
• Add User-Interactive Authentication fields to cross-signing APIs as per MSC1756. (#3331)
• Mention that a canonical alias event should be added when a room is created with an alias. (#3337)
• Add an 'API conventions' section to the Appendices. (#3350)
• Clarify the documentation around the pagination tokens used by /sync, /rooms/{room_id}/messages, /initialSync, /rooms/{room_id}/initialSync, and /notifications. (#3353)
• Remove the inaccurate 'Pagination' section. (#3366)
• Clarify how redacted_because is meant to work. (#3411)
• Remove extraneous mimetype from EncryptedFile examples, as per MSC2582. (#3412)
• Describe how MSC2844 affects the /versions endpoint. (#3420)
• Fix documentation errors around threepid_creds. (#3471)

Server-Server API

New Endpoints

• Add /make_knock and /send_knock endpoints as per MSC2403. (#3154)

Backwards Compatible Changes

• Add cross-signing information to GET /user/keys and GET /user/devices/{userId}, m.device_list_update EDU, and a new m.signing_key_update EDU as per MSC1756. (#2536)
• Add support for knocking, as per MSC2403. (#3154)

Spec Clarifications

• Specify that GET /_matrix/federation/v1/make_join/{roomId}/{userId} can return a 404 if the room is unknown. (#2688)
• Fix various typos throughout the specification. (#2888, #3116, #3128, #3207)
• Correct the /_matrix/federation/v1/user/devices/{userId} response which actually returns "self_signing_key" instead of "self_signing_keys". (#3312)
• Explain the reasons why <hostname> TLS certificate is needed rather than <delegated_hostname> for SRV delegation. (#3322)
• Tweak the example PDU diagram to better demonstrate situations with multiple prev_events. (#3340)

Application Service API

Spec Clarifications

• Fix various typos throughout the specification. (#2888)

Identity Service API

New Endpoints

• Add GET /_matrix/identity/versions API as per MSC2320. (#3101)

Removed Endpoints

• The v1 identity service API has been removed in favour of the v2 API, as per MSC2713. (#3170)

Spec Clarifications

• Fix various typos throughout the specification. (#2888)
• Clarify that some identifiers must be case folded prior to processing, as per MSC2265. (#3167, #3176)
• Describe how MSC2844 affects the /versions endpoint. (#3459)

Push Gateway API

Spec Clarifications

• Clarify where to get information about the various parameter values for the notify endpoint. (#2763)

Room Version 6, Client-Server r0.6.1, and Federation r0.1.4 spec releases

Hey all,

It's been a little while since we've done a spec release, so here we are with Room Version 6, Client-Server r0.6.1, and Federation r0.1.4.

Room Version 6 (and the associated Federation r0.1.4 release) is largely something for implementations to worry about. It contains new event authorisation rules, changes to the redaction algorithm, and stricter compliance for JSON.

Client-Server r0.6.1 contains a number of clarifications as well as SSO support for authorisation, "soft logout" to avoid needlessly destroying e2e history, and new ways to publish aliases within rooms.

If you're wondering where all the E2E-by-default related MSCs are - we're doing final iterations based on the real-world feedback from the E2E-by-default launch a few weeks ago, and they are then expected to land in the upcoming Client-Server r0.7.

Here's all the MSCs that got merged since the last release:

• MSC2240: Room version 6
• MSC2457: Invalidating devices during password modification
• MSC2454: Support UI auth for SSO
• MSC2451: Remove query_auth federation endpoint
• MSC2432: Updated semantics for publishing room aliases
• MSC2324: Facilitating early releases of software dependent on spec
• MSC2313: Moderation policies as rooms
• MSC1466: Soft Logout
• MSC2540: Stricter event validation: JSON compliance
• MSC2209: Alter auth rules to check notifications in m.room.power_levels
• MSC1802: Remove the '200' value from some federation responses

Client-Server r0.6.1 changelog

New Endpoints

• Added /rooms/{roomId}/aliases for retrieving local aliases for a room. (#2562)

Backwards Compatible Changes

• Added data structures for defining moderation policies in rooms per MSC2313. (#2434)
• Optionally invalidate other access tokens during password modification per MSC2457. (#2523)
• Add User-Interactive Authentication for SSO-backed homeserver per MSC2454. (#2532)
• Add soft-logout support per MSC1466. (#2546)
• Replaced legacy room alias handling with a more sustainable solution per MSC2432. (#2562)

Spec Clarifications

• List available enum values for the room versions capability. (#2245)
• Fix various spelling errors throughout the specification. (#2351, #2415, #2453, #2524, #2553, #2569)
• Minor clarifications to token-based User-Interactive Authentication. (#2369)
• Minor clarification for what the user directory searches. (#2381)
• Fix key export format example to match the specification. (#2430)
• Clarify the IV data type for encrypted files. (#2492)
• Fix the .m.rule.contains_user_name default push rule to set the highlight tweak. (#2519)
• Clarify that an event_id is returned when sending events. (#2525)
• Fix some numbers in the specification to match their explanation text. (#2554)
• Move redaction algorithm into the room version specifications. (#2563)
• Clarify signature object structures for encryption. (#2566)
• Clarify which events are created as part of /createRoom. (#2571)
• Remove claims that the homeserver is exclusively responsible for profile information in membership events. (#2574)

Server-Server (Federation) r0.1.4 changelog

New Endpoints

• Add new POST /publicRooms endpoint for filtering the room directory. (#2305)
• Add new v2 /send_join and /send_leave endpoints per MSC1802. (#2547)

Removed Endpoints

• Remove the unused query_auth API per MSC2451. (#2470)

Spec Clarifications

• Move auth event selection to a more obvious location. (#2392)
• Fix typo in Request Authentication python example. (#2510)
• Clarify which fields are required on the key server endpoints. (#2527)
• Clarify the limits of prev_events and auth_events for PDUs. (#2538)
• Clarify which events are targeted by backfill. (#2559)
• Fix the response format of the /send endpoint. (#2560)
• Clarify signature object structures for encryption. (#2566)
• Clarify the server names to use when signing requests. (#2570)
• Clarify the state/auth chain requirements for /send_join. (#2575)
• Fix various spelling errors throughout the specification. (#2577)

Hey all,

For the last several months the team has been working on tightening up privacy in Matrix, and with the 1.4 release of Synapse and Riot quite a lot has been done in the area. One of the remaining pieces was to release all the specification changes to help other client/server implementations achieve the same goals, and now we've done that.

The Client-Server r0.6.0 and Identity Service r0.3.0 spec releases both cover the privacy improvements added through a number of MSCs in the last few months. Of particular note is that identity servers are now expected to support terms of service endpoints, which requires authentication that clients might need to worry about - check the spec changelogs for details.

The full changelog for the Client-Server r0.6.0 release is:

• Breaking Changes

  • Add id_access_token as a required request parameter to a few endpoints which require an id_server parameter as part of MSC2140. (#2255)

• New Endpoints

  • Add POST /account/3pid/unbind for removing a 3PID from an identity server. (#2282)

• Backwards Compatible Changes

  • Add M_USER_DEACTIVATED error code. (#2234)
  • Remove bind_msisdn and bind_email from /register now that the identity server's bind endpoint requires authentication. (#2279)
  • Add m.identity_server account data for tracking the user's preferred identity server. (#2281)
  • Deprecate id_server and make it optional in several places. (#2310)

• Spec Clarifications

  • Add missing format fields to m.room.message$m.notice schema. (#2125)
  • Remove "required" designation from the url field of certain m.room.message msgtypes. (#2129)
  • Fix various typos throughout the specification. (#2131, #2136, #2148, #2215)
  • Clarify the distinction between m.key.verification.start and its m.sas.v1 variant. (#2132)
  • Fix link to Olm signing specification. (#2133)
  • Clarify the conditions for the .m.rule.room_one_to_one push rule. (#2152)
  • Clarify the encryption algorithms supported by the device of the device keys example. (#2157)
  • Clarify that /rooms/:roomId/event/:eventId returns a Matrix error. (#2204)
  • Add a missing state_key check on .m.rule.tombstone. (#2223)
  • Fix the m.room_key_request action value, setting it from cancel_request to request_cancellation. (#2247)
  • Clarify that the submit_url field is without authentication. (#2341)
  • Clarify the expected phone number format. (#2342)
  • Clarify that clients should consider not requesting URL previews in encrypted rooms. (#2343)
  • Add missing information on how filters are meant to work with /context. (#2344)
  • Clarify what the keys are for rooms in /sync. (#2345)

The full changelog for the Identity Service r0.3.0 release is:

• New Endpoints

  • Add /account, /account/register, and /account/logout to authenticate with the identity server. (#2255)
  • Add endpoints for accepting and handling terms of service. (#2258)
  • Add /hash_details and a new /lookup endpoint for performing hashed association lookups. (#2287)

• Backwards Compatible Changes

  • Deprecate the v1 API in favour of an authenticated v2 API. (#2254)