matrix.org homeserver

Greetings from Element's backend/SRE team, who run the matrix.org homeserver on behalf of the Matrix.org Foundation.

Recently users of the matrix.org homeserver began seeing problems where rooms would simply stop working. Operations such as sending a new message, or joining the room as a new member, would fail for mysterious reasons. Where an error message was shown at all, it tended to be something cryptic like "No create event in auth events".

After a couple of weeks of hard work by a team of Element staff including backend developers and systems engineers, we were able to repair almost all of the affected rooms. Although we're still investigating exactly what went wrong and checking that everything is now working as it should, we'd like to share some details about what we know and the work we've done to date.

We'll be diving into some quite technical details. Hopefully you'll find it interesting learning a bit about how Synapse works, how Postgres works, and the work we sometimes find ourselves doing to keep the matrix.org homeserver running.

🔗TL;DR

Let's start with a high-level summary.

The matrix.org homeserver is backed by a large PostgreSQL database instance. Parts of an index on one of tables in this database had become corrupted. We are unsure exactly what caused this corruption, but believe it happened at least a year ago, and likely significantly longer.

The nature of this corruption was such that it had little or no effect at first. However, a background maintenance task which removes old, unreferenced data from this table recently started working on the corrupted region. Due to the corrupt index, the maintenance task incorrectly removed active data from the table, in effect corrupting rooms.

Having identified the problem, we rebuilt the corrupted index, and then restored the data that had been incorrectly removed, from database backups.

Continue reading…

🔗TL;DR

As we need to take more concrete steps to improve the financial situation of the Foundation, we will be rolling out a freemium offer for the matrix.org homeserver users. The alternative is to turn off the server, which we want to avoid doing. The goal is for the most active users to support the cost of the service. Free users will have limits on how they can use the service (mostly around media). The change can be supported by any client with limited to no development. Premium plans will be rolled out over the summer, and we will be iterating on the exact scope in the first few weeks. The Homeserver Terms and Privacy Policy will be updated accordingly and deployed in the coming weeks.

Continue reading…

We're thrilled to announce that the migration of matrix.org to the Matrix Authentication Service (MAS) is complete and went according to plan - having been running for over 24h in our brave new world, we’re declaring the migration a success! As of Monday April 7th 07:30 UTC, matrix.org is running on Matrix’s next-generation auth system based on OAuth 2.0/OpenID Connect.

This is no mean feat - the migration shifted all 45M access tokens and 110M users from Synapse to MAS in under 30 minutes (thanks in part to MAS’s cheeky use of the x86-64-v2 architecture; who knew that database migrations can be SIMD-accelerated?) - and represents the culmination of over 4 years of work to move Matrix to a modern authentication standard. Many thanks go to Element for funding, Hugh, Olivier and many other contributors who helped me make Next Gen Auth happen!

Continue reading…

On Monday 7th of April 2025 at 7am UTC, we will migrate the Matrix.org homeserver's authentication system over to MAS (Matrix Authentication Service) in order to benefit from Next-generation authentication.

The migration will involve up to one hour of downtime.

MSC3861 (Next-generation auth for Matrix, based on OAuth 2.0/OpenID Connect (OIDC)) and its dependent MSCs have progressed sufficiently that the Foundation is confident in MAS and the new next-generation auth APIs. Specifically, all the MSCs are now in or have passed Final Comment Period (FCP) with disposition to merge! 🎉

We expect the MSCs to finish FCP and get merged into the next spec release. The full list of core Next-gen Auth MSCs is:

• MSC3861: Next-generation auth for Matrix, based on OAuth 2.0/OIDC
• MSC2964: Usage of OAuth 2.0 authorization code grant and refresh token grant
• MSC2965: OAuth 2.0 Authorization Server Metadata
• MSC2966: Usage of OAuth 2.0 Dynamic Client Registration in Matrix
• MSC2967: API scopes
• MSC4254: Usage of RFC7009 Token Revocation for Matrix client logout

This is incredibly exciting, reflecting 4 years of work on next-generation auth, and brings with it a new account management interface, additional security, and a better registration experience.

Continue reading…

The Matrix.org homeserver will see changes related to authentication in Q1 2025. The team will turn off guest account access on Matrix.org on January 16th and roll out Matrix Authentication Service (MAS) to embrace Matrix 2.0 after February 10. Client developers need to ensure their clients support the required changes.

🔗What is MAS

Matrix Authentication Service is Matrix's next-generation authentication stack. It allows for more flexible authentication journeys without requiring client developers to support every one of them.

You can find all the technical details in Quentin's Matrix Conf talk, Harder Better Faster Stronger Authentication with OpenID Connect.

Continue reading…

We will be decommissioning the sliding sync proxy next week (21/11/2024) and Element are removing client support in mid-January (17/01/2025).

Sliding Sync is designed to provide a significantly faster and more scalable sync experience in our clients. The initial implementation was first prototyped in Element Web backed by an entirely experimental server proxy. The implementation had half an eye on low bandwidth use cases, and the prototype led to MSC3575. We then realised that a simpler approach would be beneficial, and reused the same (experimental) proxy concept to facilitate beta testing with Element X, this time making it available on matrix.org. In doing so, we learned valuable lessons, leading to a refined and simplified API design in MSC4186. The proxy itself was only ever considered as a temporary arrangement to aid speed of development, rather than being a long term solution.

Simplified Sliding Sync MSC4186 (also known as native sliding sync), has since been implemented in Synapse, with encouraging results. Now that we don’t expect the API shape to change significantly, we recommend homeserver developers to implement MSC4186 natively.

The Matrix.org Foundation does not have the resources to keep up maintenance of the proxy service or its codebase, and plans to decommission the proxy from Mid-November and archive the sliding-sync repo.

Recognising that the community needs time to adopt sliding sync natively, Element will keep client support for the old API (MSC3575) until the 17th of January, 2025.

Continue reading…

Hello everyone,

The Trust & Safety team has been working hard to get MSC3916 in the hands of users, and we’re nearly there with Matrix 1.11 being released last week. This fixes a long-standing design flaw in Matrix where media (images, avatars, files, etc) can be accessed without authentication if the URL is known. Matrix 1.11 fixes this by requiring authentication on these URLs, removing the ability for users to treat homeservers as CDNs for hosting arbitrary Matrix content for arbitrary users.

Rolling this feature out to the entire public federation is a bit tricky, particularly when considering the user safety and privacy benefits which Matrix 1.11 brings. Developers are encouraged to support authenticated media quickly to give server admins the ability to freeze unauthenticated media access on their servers. Media uploaded or cached before the freeze will remain accessible on the unauthenticated endpoints, but any media uploaded or cached after the freeze will only be available through the authenticated endpoints.

This freeze reduces the amount of breakage users will see. We’re aware of links being shared outside the context of a room already, and breaking those would be pretty disappointing for those users. We also don’t want to encourage that capability going forwards due to the space it takes up and the anonymous nature of the requests. Users who keep their clients updated should see no impact when their servers implement their freeze, but may find themselves unable to copy/paste media URLs to their friends.

Matrix 1.11 recommends that all servers evaluate their local ecosystem to determine when would be best to implement the freeze, and that the freeze should happen before Matrix 1.12 is released in August 2024. For the matrix.org homeserver, we anticipate most of our users to have updated clients in July, putting our freeze date in August.

Developers, and those curious, are encouraged to review the Matrix 1.11 blog post for details on the changes they’ll need to make in July to have near-zero matrix.org user impact, and for information about the recommended freeze approach.

🔗Timeline for matrix.org homeserver

To assist developers and other server admins in testing their implementations, we will be updating the beta.matrix.org homeserver to enact the freeze as soon as code is available for that. We expect this to happen in July 2024. The matrix.org (non-beta) homeserver’s freeze will be started on August 28th, 2024 September 4th, 2024 during normal UK business hours.

Update August 14, 2024: Most of the ecosystem has already updated to support authenticated media with only a few bug fixes pending release. To give a little bit more buffer for these bug fixes to roll out, we've moved our scheduled date to September 4th, 2024.

All media uploaded and cached prior to the freeze will remain accessible on the unauthenticated endpoints and authenticated endpoints. Media uploaded and cached after the freeze will only be available through the authenticated endpoints, not the unauthenticated ones.

🔗Developer support

The team is making themselves available in the #matrix-client-developers:matrix.org and #matrix-homeserver-developers:matrix.org rooms on Matrix to help support developers in implementing this feature. Client, server, and bridge authors are welcome to visit those rooms to get help in figuring out what needs to happen to support authenticated media. Further resources are also available in the Matrix 1.11 blog post.

For questions about the rollout itself, the freeze date, or the (beta.)matrix.org homeserver, please visit #foundation-office:matrix.org on Matrix.

We look forward to seeing the ecosystem working towards a safer, authenticated, experience for users.

Thank you,

Travis Ralston & the whole Matrix.org Foundation team