Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://spec.matrix.org/proposals.
For those familiar with Travis' weekly task lists of MSCs for the Spec Core Team to review in the Office of the Matrix Spec Core Team room, a new weekly list is now being posted in the Matrix Spec & Docs Authoring room. This list is aimed at technical writers who can help by converting MSC authors' words into PRs against the spec text itself.
This is the final step for getting an MSC integrated into a new release of the Matrix spec, and anyone can try their hand at it! It would also very much help the Spec Core Team by freeing up more bandwidth for review of the MSC backlog, as well as push forward the protocol itself. Thank you!
If you have any questions, feel free to ask them in the relevant Matrix rooms.
This MSC proposes expanding the set of suggested, interpreted HTML tags in Matrix clients to include additional tags related to tables. With them, more control over table rendering is possible. The proposal itself includes one such (albeit fairly arbitrary) example
The proposal is well-written and straight-forward, so do feel free to have a look if the subject interests you!
This year again the Matrix Foundation and Community will have plenty of opportunities to meet at FOSDEM! Together with our awesome community, we’re organising a FOSDEM Fringe Event before FOSDEM itself, we will have a booth to meet everyone and spread the word about Matrix, and a devroom to go more in depth on Matrix topics.
We’re excited to announce the timeline for our first ever elections as we take the next big step in open governance for Matrix. We’re also introducing two new membership tiers to increase community representation by including open source projects and foundations on the Governing Board.
I confess I'm awfully chuffed with myself as we return from the holiday break. I just completed a successful migration of my main Matrix account from managed hosting to a homeserver that I run for myself on a virtual private server (VPS).
The whole experience has been illuminating, and there are some specific details that are timely for people like me who needed to migrate off of Element Matrix Services (EMS) as they pivot to focus on enterprise.
Thus, this blog post. I'm going to share my experience in hopes that it'll help some folks with that migration!
2023 has been a pivotal year for Matrix, with huge changes landing both organisationally and technically to prepare the protocol for future generations. The ecosystem has once again gone from strength to strength, with active users (based on Synapse opt-in phone-home reporting) doubling across the public network, and more projects building on Matrix than we can count (look out for the “This Year in Matrix” community wrap-up blog post) - and more organisations than we can track joining Matrix for all their secure decentralised communication needs.
On the governance side, we are in an incredibly exciting new era with Josh joining the Matrix.org Foundation as its first ever Managing Director (and employee!), with a mandate to cement sustainable funding for Matrix as an independent foundation, governed by the forthcoming elected open Governance Board. Now, Matrix needs funding more than ever - but rather than turning the entirety of this post into a plea for donations, I’m going to let Josh fly the flag in the coming weeks. Meanwhile, if you want Matrix to keep existing (especially if you’re an organisation who builds on Matrix) please join the Foundation and donate.
On the technical side: the theme of the year has been one of focus: extreme, overdue, focus.
Over the years, it’s fair to say that the core team has tried to strike a balance between building the core foundational technology of Matrix (the spec, a stable server implementation, client SDKs, end-to-end encryption, VoIP, etc)... and long-term forward-looking projects designed to futureproof Matrix (e.g. Account Portability, P2P Matrix, Dendrite, Hydrogen) and/or inspire developers to build on Matrix for more than just chat (e.g. Third Room, Applications Beyond Chat). In retrospect, this was wildly optimistic: we underestimated the amount of remaining work needed to polish the foundational tech to mainstream quality - and despite Matrix uptake going through the roof, this hasn’t translated into sufficient funding to have the luxury to support folks to proactively work on next-gen projects (or foundational projects, for that matter).
So, this year, we’ve ended up focused on one thing: getting the foundational Matrix featureset to better-than-mainstream quality, performance and stability. We’ve dubbed the overall initiative Matrix 2.0, and kicked it off at FOSDEM 2023 with our Matrix 2.0: How we’re making Matrix go vooooom main-stage talk.
Matrix 2.0 isn’t (yet) an actual versioned release of the Matrix specification - instead, it describes the various foundational projects needed to get quality, performance and stability up to and beyond that of today’s mainstream messaging apps. These projects are:
Sliding Sync (MSC3575): the ability to instantly log in, launch and sync Matrix clients no matter how large or busy the account.
Native E2EE Group VoIP (MSC3898 + TBA): scalable video and voice conferencing and calling built natively on Matrix and so benefiting from Matrix’s end-to-end encryption.
Native OIDC (MSC3861): replacing Matrix’s historical authentication mechanisms with industry-standard Open ID Connect (giving us two factor authentication, multi-factor auth, passkeys, and radically simplifying auth implementations for both client and server developers).
Faster Remote Room Joins (MSC3902): letting servers rapidly join rooms on other servers by incrementally participating in the room.
Over the course of the year Matrix 2.0 has gone from the initial demo on stage at FOSDEM to concrete implementations which users can play with today as announced in our Matrix 2.0: The Future of Matrix post in September. Since then, we’ve been busy polishing away. On Sliding Sync, the proxy has pretty much stabilised - although the protocol itself can and should be simplified before we think seriously about native implementations (in practice, having the server track room list ordering gets very fiddly when only clients can really determine the final ordering, due to E2EE). Element X and matrix-rust-sdk has been the main implementation driving forwards Sliding Sync and much of the other Matrix 2.0 work, for those itching to play with it.
On Native Group VoIP: we’ve gone through many iterations over the year - starting off with Full Mesh calling (good for ~7 users per call); then switching to the experimental waterfall Selective Forwarding Unit (SFU) to provide scalable but not-E2EE conferencing; and then switching to a hybrid solution using LiveKit to provide an E2EE-capable scalable SFU, but with the signalling and encryption all handled by Matrix. Element Call is the main implementation driving forwards the underlying Matrix work here, and Element Call Beta 3 showed off the new LiveKit based implementation in July - which was then integrated with Element X complete with end-to-end encryption in November. There’s still some polishing remaining here, with a new layout engine in the wings for Element Call, and enabling full encrypted-per-sender conferencing by default in both Element Web and Element X, but it really feels like the hardest work is behind us now: the core team has been successfully doing all of its collaboration on Element Call for months now, like so:
On Open ID Connect: things are also shaping up well. This will be the first time that we’ve replaced a large chunk of the Matrix spec with something else, and in order to manage your account in Matrix 2.0-native clients like Element X homeserver admins will need to migrate their authentication to the new OIDC World using matrix-authentication-service (MAS). There’s a great blog post from September which explains what this will entail - and since then, we even have the beginnings of syn2mas: a migration script to migrate from Synapse-managed accounts to MAS-managed accounts (warning: still experimental). The Matrix.org homeserver hasn’t been migrated yet (as we need to support social login first), but an increasing number of standalone Matrix servers are going OIDC-native, so arguably the migration has already begun! We’ll keep https://areweoidcyet.com updated as the project progresses.
Finally, the core of Faster Remote Room Joins (FRRJ) shipped in Synapse back in February. There’s still some major speedups that FRRJ could unlock, but the other tracks of Matrix 2.0 have been taking priority.
So: Matrix 2.0 is palpably on the horizon - all that remains is polish on the example clients (Element X & Element Call), full support for migrating to OIDC, and landing the MSCs into the spec. For instance, Element X just added read receipts and (early) E2EE backup support in the last few days - the gap is closing! It’s worth noting that significant amounts of this work has been funded by BWI for BwMessenger and BundesMessenger: huge thanks to BWI for supporting core Matrix development by contracting Element.
Encryption stability received a huge amount of attention this year. It turns out that reliable end-to-end encryption is surprisingly tricky in a decentralised environment, and historically we’ve been playing on hard mode by implementing three entirely separate implementations of the Matrix layer of encryption between matrix-js-sdk, matrix-ios-sdk and matrix-android-sdk2, each with their own bugs - more than tripling the costs of development, audits, and maintenance by the Foundation (quite ignoring the independent implementations from the community in mtxclient, libquotient, matrix-dart-sdk, trixnity etc).
So a huge project has been underway to converge on a single auditable codebase for the core team’s E2EE implementation so that any bugs or future features can be resolved in a single place. That codebase is matrix-rust-sdk’s matrix-sdk-crypto crate (and our underlying vodozemac double ratchet implementation) - and we’re proud to say that we are using it for encryption in matrix-rust-sdk itself (as showcased by Element X and Fractal 5), matrix-ios-sdk and matrix-android-sdk2 (as used in the old Element iOS & Android apps), and have now merged it in matrix-js-sdk too (available for new logins on develop.element.io). The process of rustifying the encryption in Element Web and the old iOS & Android apps has been nicknamed “Element R”.
The process of switching matrix-js-sdk to use Rust encryption has been particularly gruelling, requiring compiling matrix-sdk-crypto down to WASM as matrix-rust-sdk-crypto-wasm and then doing heart surgery to replace the old JS crypto implementation… while also needing to extensively loop from WASM back into the browser to use IndexedDB for storage, all while outperforming the old implementation. It’s tantalisingly close now: while develop.element.io has it turned on by default for new logins, there are still a few remaining performance edge cases to be chased down related to online backup before we migrate everyone to it. The remaining blocking issues can be found on GitHub for those interested in tracking progress.
matrix-crypto-sdk is already manifestly more reliable than the old implementations (in terms of the chances of hitting infamous Unable To Decrypt errors) - and now that we are so close to converging on it everywhere, the race is on to ensure that any remaining defects get flushed out for once and for all. One of the new initiatives here is called complement-crypto - a full end-to-end torture testing suite specifically for matrix-crypto-sdk. You can read all about it in the announcement post a few weeks ago, but suffice it to say it’s a super exciting project which stress-tests both matrix-rust-sdk and matrix-js-sdk (with the new rust crypto implementation) against federated Synapse containers in order to test E2EE under the most horrible failure modes imaginable. It’s already picked up some elusive bugs which have plagued us for literally years, and it looks set to be the main framework by which we will hunt down and kill any remaining issues. See the Test hitlist for the full scope we’re targeting.
Now that everyone’s (almost) converged on matrix-sdk-crypto, the next big project for the Crypto Team is going to be improving the E2EE usability (at last!). The big news here is that we’re shifting to Trust On First Use (TOFU) for user trust. Specifically: this means that we will only encrypt messages to devices whose owner has explicitly cross-signed them (essentially trusting the owner by default). You will still be able to explicitly verify that other users are not being impersonated (via QR scan or emoji comparison), but this should improve the default behaviour to be much more secure. Alongside TOFU will come other radical simplifications of the E2EE UX (both around login, self-verification, cross-verification and backup), so watch this space: the game is afoot to finally fix Matrix’s E2EE usability, now we can make all the changes in one place!
Finally, work continues to progress at matrix-dmls on supporting a decentralised dialect of Messaging Layer Security (MLS, RFC9420) on top of Matrix as an alternative to our normal Olm/Megolm encryption, with recent work focused on making it play nice with matrix-sdk-crypto. https://arewemlsyet.com is the place to track updates (although it’s a bit overdue for an update).
Faced with limited funding and the decision to focus exclusively on stability, reliability and performance, there have inevitably been some major changes impacting the core team.
One of the biggest changes is that Element (the company formed by the core Matrix team back in 2017 to try to fund our work on Matrix) can no longer financially afford to donate its work on Synapse and other server components to the Matrix Foundation under the permissive Apache licence. Instead, Element is continuing development under the copyleft AGPLv3 licence at github.com/element-hq/synapse going forwards. This is to let Element sell AGPL exceptions to commercial Matrix vendors in order to fund their underlying Matrix development: you can read more about it at Element’s announcement - or you can listen to this week’s Matrix Live for a firsthand explanation:
The other major change is that we’ve had no choice but pause development on the majority of the core team’s next-generation Matrix projects. We had high hopes of being able to secure dedicated funding for Third Room (especially after the awesome Tech Preview 2: Creator Update in June), but the interested parties did not come through, and the team has now disbanded. Meanwhile, P2P Matrix and Low Bandwidth Matrix is on hiatus until there’s dedicated funding - and Account Portability work is also temporarily paused in favour of commercial Element work, despite the fantastic progress made recently with Pseudo IDs (MSC4014) and Cryptographic identifiers (MSC4080). Given P2P Matrix and Account Portability were the main projects driving Dendrite development recently, this may also cause a slow-down in Dendrite development, although Dendrite itself will still be maintained.
Needless to say, this is far from an ideal situation: we sent up distress flares loud and clear at the beginning of last year’s holiday update; and we’ve now had to shrink to focus exclusively on the core projects. However, we’re optimistic that the tighter focus in the medium term will help us get back to the point where we can resume the longer-term projects - assuming that organisations (and individuals) dependent on Matrix sign up to support the project.
Despite the downsides of 2023, right now we’re feeling distinctly optimistic: Matrix 2.0 clients like Element X already outperform the best proprietary mainstream options by many metrics - and focusing purely on improving the foundations is only going to improve that. We may not have taken the most direct route to get to where we are today, but it genuinely feels like 2024 will be the year where Matrix overtakes the incumbents.
Talking of which, there’s just one last thing to mention: amidst the economic challenges of 2023, one future-facing core team Matrix project has survived: our work around the EU Digital Markets Act (DMA). The DMA is leading antitrust legislation from the European Commission, which aims to stop big centralised tech companies (so called ‘gatekeepers’) from carelessly suppressing innovation, competition and consumer choice by trapping users inside their walled gardens.
Needless to say, we’ve been tracking the DMA closely throughout its gestation, and we’re now in the final sprint: in March 2024, messaging services which have been identified as gatekeepers will have to open their networks to allow interoperability with requesting messaging services (while preserving E2EE, if they’re encrypted). So far, only WhatsApp and Facebook Messenger have been identified as gatekeeper services (Apple is doing everything they can to wriggle out of it). However, it looks like WhatsApp is taking it seriously, which could prove very interesting indeed.
As Matrix, we’ve taken a two-pronged approach: on one side, showcasing how Matrix as it stands today can already bridge existing chat systems together as a highest-common-denominator protocol (including preserving E2EE, if they happen to already use the Double Ratchet). On the other side, we’ve also been contributing significantly to MIMI, the IETF Working Group dedicated to standardising a lowest-common-denominator protocol specifically for DMA interoperability.
2023 has involved a lot of work on MIMI, participating in the Design Team alongside Phoenix, Wire, Cisco, Google and Wickr - and at IETF 118 in Prague in November we collectively proposed the first draft of the protocol (see also the architecture presentation for context). MIMI has ended up taking some inspiration from Matrix (and Linearized Matrix, a simplified dialect we proposed which does away with full-mesh decentralisation), while focusing very tightly on the specific problems of server-to-server interop between existing communication services - leaning on MLS (where available) for synchronising state across the services, while also leaving the door open for using the Double Ratchet to provide an on-ramp for pragmatic bridging to today’s services (including Matrix!).
MIMI’s tight focus means that it doesn’t currently provide conversation history, arbitrary state events, or decentralised conversations - and is focused purely on server-server communication. However, there’s certainly a world where Matrix could evolve to be compatible with MIMI - such a hypothetical Matrix 3.0 would effectively layer Matrix’s richer existing semantics (decentralised conversations, extensible state events, state resolution, group VoIP etc) on top of MIMI’s subset of functionality. It’ll be interesting to see how this plays out. Meanwhile, we’ll continue to provide both Matrix for pragmatic DMA interoperability for today - and participate in MIMI for IETF-track interoperability.
Anyhow: hopefully it’s clear that 2024 is going to be a super interesting year for Matrix - whether that’s simply by nailing Matrix 2.0, or whatever excitements DMA interoperability brings. And if you’re relying on Matrix: please donate.
Meanwhile - have a fantastic end of the year; and thanks once again for flying Matrix.
(Please ignore, that I never know, what year number to attach to such a "Year In")
You may remember I organized a small blog post last year to collect stories from the different community projects and what they did in the year and maybe some sneak peaks at the next year. If not, you can find it here or on the Matrix.org blog.
Anyway, enough about 2022, I now encourage you to talk about 2023 and beyond! If you have interesting stuff to report about your projects or projects you have been involved in in 2023, feel free to join #year-in-2023:neko.dev and talk about it! The usual TWIM rules apply there, just that we talk about a whole year and it may involve lots of manual editing on my side, so don't try to break it. Also please be positive in your news and lets try to end 2023 with a bang!
And please share this with projects you want to hear about. :)
As 2023 winds down and I find myself in the thick of planning for 2024, I’d like to start preparing all of us in the Matrix ecosystem for what is to come.
Next year will mark a number of important milestones in the history and evolution of Matrix: the protocol will mark its 10th birthday, we’ll see key initiatives in the spec cross the finish line, and we’ll seat the first ever community-elected Governing Board.
The election of our first Governing Board is what I’d like to focus on today, because it is a huge milestone on the path to an independent, self-sustaining, and self-governing ecosystem. When we celebrate Matrix’s 20th birthday, we’ll look back and our history will be divided much the same way it is in other ecosystems: before and after incorporating a foundation, and before and after introducing community governance.
Let’s talk about what the Governing Board is, why it matters, and how to get involved!
I'll start with a heads-up that the Foundation is going to clean-up all the libera.chat aliases on Matrix. You might want to check the rooms under your control to update the alias and the matrix.to links to it.
The Matrix.org Foundation has taken down the bridge with the Libera Chat network. This only prevented messages from making it across the bridges, for Matrix users to appear on the IRC side, and for new IRC users to appear on the Matrix side.
As part of our work to remove the bridge leftovers, we have removed the ghosts in Matrix rooms and demoted the Libera Chat appservice user. We will now remove the aliases from the rooms, and strongly encourage you to make sure you update the links to your Matrix room if they relied on a matrix.to link that contains :libera.chat
This Tuesday we released Synapse 1.98.0rc1. Its promotion to the 1.98.0 release proper is planned for the coming Tuesday (12th Dec). We expect v1.98.0 to be the final release of 2023, as the team takes a break over the holiday period.
