15.06.2023 15:21
—
General
—
Thib
Last update: 15.06.2023 14:16
Hello federation,
TLDR: New website is coming tomorrow, your RSS reader might be disoriented during the switch.
That's right, after several months of studying, designing and implementing: we're finally going to deploy the new matrix.org website on June 16! Let's have a look back at the why and how.
I also need to thank Jonas Platte not only for his technical expertise but also for his kind support and patience. Thanks to MTRNord as well for kickstarting the project, and to the various designers involved.
Back in September 2022 we launched the very first public technology preview of Third Room - our entirely open source, open standards-based platform for creating decentralised multiparty spatial apps and virtual worlds on top of Matrix.
The mission of Third Room is to ensure that a truly open and equitable platform exists for powering shared 3D environments - providing an alternative to the closed walled gardens of the bigger vendors, and generally safeguard against a repeat of the fragmented dystopia that has plagued instant messaging and VoIP systems. In short, just as Matrix aims to be the missing secure communication layer of the open Web, Third Room aims to be the spatial collaboration layer.
Today, we’re incredibly excited to announce Third Room Technology Preview 2: The Creator Update. As more and more 3D hardware enters the market, the race is on to provide tools to developers and creators so they can build on an open, vendor-agnostic platform - and in this update we’ve focused on building out the scripting, editing and authoring capabilities of Third Room to provide a solid platform for building and running collaborative 3D apps of any kind. Check out the new release at https://thirdroom.io.
As a reminder: the Third Room team is a tiny band formed by Robert, Nate and Ajay and operates outside of all the rest of our work on Matrix: the other 97% of our effort goes into making the core of Matrix amazing (particularly the underpinnings for Element X and the next generation of Matrix clients). However, Matrix is about more than just chat and VoIP, and Third Room provides an excellent showcase of Matrix’s abilities as a general purpose communication fabric.
The DMA is new antitrust/competition regulation from Europe which came into force in November, whose objective is to make digital markets more competitive by forcing gatekeepers (i.e. large tech companies) to reconsider some of their anti-competitive or self-preferencing practices. Gatekeepers are defined as companies which have a clear position of influence in a given market (based on revenue / market cap / number of users thresholds), and “an entrenched and durable position”. The process for designating which companies count as gatekeepers will start in May 2023.
The DMA touches upon different key topics, from self-preferencing behaviour to app store management practices - but most importantly includes interoperability for “number-independent interpersonal communication services” (NIICS), otherwise known as chat and voice/video calling and conferencing services (social media was left out for now).
This particular workshop was focused on the latter: interoperability between messaging services, with the aim of getting the different stakeholders of the industry in the same place to discuss how the legislation could be implemented. The whole idea is to figure out a practical way in which WhatsApp could interoperate with iMessage, Google Messages and others, creating an interoperable communication network where users are no longer locked into communication silos and pick their preferred service provider without compromising on who they can talk to.
About 900 people participated online, and around 80 people were present in person: the maximum the room could hold. It was particularly fun to see representatives from the whole industry turning up in person, including folks from XMPP, MIMI (the new IETF working group on messaging interoperability), MLS, us from Matrix obviously (alongside Matrix ecosystem representatives from Beeper and NeoChat!) - all together with the Body of European Regulators for Electronic Communications (BEREC), civil society representatives (like the Federation of German Consumer Organisations (VZBV) and European Digital Rights (EDRi)), mobile network operators, local network agencies, and obviously some of those who are likely to be designated as gatekeepers, such as Meta, Apple and Google.
Note: This was originally posted on https://blog.neko.dev/posts/matrix-year-in-review-2022.html , which also includes some small info boxes about each projects, which got lost in the translation.
As we send off 2022 with a bang, it makes sense to look back on what we did this year and where we want to go next year. In its holiday special post, the Matrix Foundation has been focusing on the core team's work. This post is focusing on the achievements of the community outside of the Matrix Foundation.
I tried to reach out to as many people I have seen do "stuff" on Matrix and have them write something they would see fitting for a year in review. Now, most people have better things to do between christmas and new years, so I hope you can excuse if some projects are missing. Also I probably forgot like half of the interesting people... HOWEVER! I hope you still enjoy what everyone wrote up. And don't forget to check out the official Matrix Holiday Update 2022 and of course read you weekly TWIM to keep up to date with any cool projects you find.
2022 has been a rollercoaster of a year for Matrix.
On one hand, the network has doubled in size (44.1M to 80.3M visible matrix IDs). The wider world is having a grand awakening to the importance of decentralisation thanks to the situation at Twitter. We’ve seen an amazing number of major new players entering the Matrix ecosystem: Reddit appears to be building out new Chat functionality using Matrix; TeamSpeak announced Matrix-based chat in TS5; Discourse is working on adding Matrix support; Thunderbird launched Matrix support; Governments from Luxembourg to Ukraine have launched their own Matrix-powered chat infrastructure; and hundreds of other organisations ranging from startups to massive private & public sector entities are betting on the protocol. The European Parliament has used Matrix as a proof-point for the viability for communication interoperability between gatekeepers in the Digital Markets Act. FOSDEM 2022 ran smoothly via Matrix with over 23,000 attendees, making it the world's largest open source conference (with 70% of attendees using their own servers!). Sweden has published case studies on the benefits of Matrix for messaging interoperability. Meanwhile existing players like Germany’s BWI have expanded their scope to providing Matrix messaging to the whole German State; Automattic is busy building Matrix plugins for Wordpress; Rocket.Chat launched federation via Matrix, Gematik has been busy progressing their TI Messenger initiative for interoperable messaging within Germany’s healthcare industry, and Tchap in France is continuing to expand.
On the other hand, only a handful of these initiatives have resulted in funding reaching the core Matrix team. This is directly putting core Matrix development at risk. We are witnessing a classic tragedy of the commons. We’ve released all the foundational code of Matrix as permissively-licensed open source and got it to the point that anyone can successfully run it at scale themselves. The network is expanding exponentially. But in return, it transpires that the vast majority of these commercial deployments fail to contribute financially to the Matrix Foundation - whether by donating directly or supporting indirectly by working with Element, who fund the vast majority of core Matrix development today.
In short: folks love the amazing decentralised encrypted comms utopia of Matrix. But organisations also love that they can use it without having to pay anyone to develop or maintain it. This is completely unsustainable, and Element is now literally unable to fund the entirety of the Matrix Foundation on behalf of everyone else - and has had to lay off some of the folks working on the core team as a result.
The only viable solution to this is for organisations building on Matrix to contribute to sharing the costs of maintaining Matrix’s core projects. We made a proposal to address this a few weeks ago, which we’ll iterate on further in the new year to find an approach which both empowers the community and encourages organisations to participate. In the interim, if you are an organisation who’s building on Matrix and you want the project to continue to flourish, please mail [email protected] to discuss how you can support the foundations that you are depending on.
As a reminder, the work the Foundation does today for the benefit of the Matrix includes:
Developing a reference push notification server (sygnal)
Developing a reference identity directory server (sydent)
Procuring and publishing independent public audits of Matrix's encryption and wider stack
Publishing the matrix.org website and blog
Publishing the weekly "Matrix Live" video podcast
Publishing the weekly "This Week In Matrix" news letter
Organising regular meetups (e.g. "Open Tech Will Save Us")
Promoting Matrix at open source conferences
Running the matrix.org homeserver
Moderating the matrix.org project rooms
Running free public bridges to networks such as IRC networks and XMPP.
This list is not remotely exhaustive (turns out there are over 240 projects in the matrix.org GitHub org!) but it serves to illustrate the sheer scale of work that the Foundation performs today. Keeping the core team funded to work on Matrix as our day job is critical for Matrix’s long-term success, and so we really hope that organisations depending on Matrix (or passing philanthropists who appreciate Matrix’s value) will drop a line to [email protected] and help keep the show on the road.
Turbocharging Matrix
Aside from the nightmares of funding open source software, 2022 has very much been a year of building - focusing on implementing a step change in Matrix’s performance and usability: ensuring that the protocol can punch its weight (and more!) against centralised proprietary alternatives. After all, Matrix clients need to be at least as good as the centralised alternatives in order to get widespread uptake.
This work has ended up taking many forms: on the server-side, Synapse sprouted Rust support to accelerate its hot paths, starting with push rule evaluation. It’s super exciting to see Synapse performance heading into a new era, building on the foundations of what’s become a very mature and stable homeserver implementation.
Meanwhile work is in the final stages on “Faster Joins”, finally letting servers rapidly join rooms over federation by synchronising only the minimal subset of state needed to join, rather than proactively synchronising the room’s full current state. Faster joins became available for testing in Synapse in October, and since then the team has been working through making it support workers and addressing the various edge cases and bugs that have shown up during testing. Current join performance is a roughly 25x speedup on large rooms, although we’re confident that we can improve this even more, and we’re aiming to land it in time for FOSDEM at the beginning of Feb.
On the client-side, the work to transform Matrix client performance has centred around “Sliding Sync” - our entirely new API for synchronising the minimal data to a client needed for it to render its UI, thus making login, launch and sync instant. Sliding sync (originally called “sync v3”) has been a long time in the making; the API has gone through countless iterations as we worked away throughout 2022 implementing it in real-life clients, and adding all the extensions (MSC3884, MSC3885) needed to get to parity with sync v2. The wait has been well worth it, though: support in Element Web is in the final stages of development - and moreover the next-generation Element X mobile clients will only speak Sliding Sync.
Element X itself is shaping up to be a showcase of just how snappy and performant Matrix can be: built on matrix-rust-sdk, it uses native Swift UI on iOS/macOS and Jetpack Compose on Android to couple together the best possible platform-native user experience with the ultimate underlying native-code SDK implementation, backed by sliding sync. The goal is to be at least as snappy as Telegram, iMessage or WhatsApp (we’ve taken to counting the frames in screen recordings to compare things like time-to-launch and time-to-load scrollback). Element X is currently in late alpha on iOS, and the hope is to enter public beta in time for FOSDEM. You can see a sneak peek here of the iPad-style layout (running under macOS) though!
Finally, in terms of usability, there have been leaps and bounds forwards across Matrix - particularly with Element’s mobile UI being entirely refreshed by the design team in September as a stepping stone to the forthcoming final Element X design. Any remaining UX quirks should be flushed out with Element X, but the visuals are already a clear step forwards towards an excellent alternative to the centralised encumbents.
Encryption
We had great plans for E2EE in Matrix this year; starting off in a huge rush to get vodozemac finished and audited as our shiny new native-Rust implementation of Olm/Megolm. The plan was then to integrate vodozemac into matrix-rust-sdk’s crypto crate, and then replace the various old fragmented E2EE implementations across matrix-js-sdk, matrix-ios-sdk, matrix-android-sdk2 and matrix-rust-sdk itself with One True audited implementation - with audits booked with Least Authority to get further assurance around matrix-rust-sdk-crypto, matrix-rust-sdk itself and finally the full stack (Element X + Synapse).
Unfortunately, things went sideways when security researchers from Royal Holloway University London & elsewhere got in touch to explain that they had found some nasty implementation vulnerabilities in the venerable matrix-js-sdk implementation. So, we had no choice but to pause “Element R” - the project to converge matrix-{js,ios,android}-sdk on matrix-rust-sdk-crypto, and instead start analysing and addressing the issues across all current shipping Matrix clients in order to resolve them as rapidly as possible. Ironically, it turned out in the end that only matrix-{js,ios,android}-sdk were affected - all other independent implementations, including matrix-rust-sdk, were okay. As such, the Element R work would have protected us from these vulnerabilities had it been ready, and failing that it would have let us solve them in a single place. Instead, Element R ended up getting pushed back for months while we worked through the various issues in triplicate across the legacy SDKs, while also checking all the other client implementations we could find, and dealing with additional issues which the RHUL researchers discovered as they dug deeper. Eventually we finished the analysis and agreed a coordinated disclosure at the end of September.
(EDIT: to be clear, we are very grateful to the security researchers for discovering and disclosing the vulns responsibly to us. The frustration here stems from the irony that if we'd finished the matrix-rust-sdk-crypto rewrite a few months earlier, we'd have mitigated the severe vulns - but instead, the rewrite got pushed back even further. It's obviously our fault though, not the researchers'.)
Since then, work has been split three ways: firstly, Element R work has resumed - and in fact Element R on iOS is pretty much usable as of today, other than needing some work to support E2EE push notifications (which will also be required for Element X). Element R on Android is very close too, and meanwhile Element R on Web decrypted its first event on Dec 19th! We’re hoping to get Element R in production on all platforms by Feb.
Secondly, we’ve been addressing other points raised by the RHUL researchers to ensure that malicious servers cannot add malicious devices or users to conversations, rather than warning as we do today. This is not a trivial problem to solve, but we’re making progress via MSC3917 (Cryptographically Constrained Room Membership) and MSC3834 (Opportunistic user key pinning (TOFU)). However, this work is effectively blocked on Element R landing first, as there’s no way we’re going to fix this in triplicate on the legacy SDKs.
Thirdly, we’ve been pushing ahead on implementing Decentralised MLS as a next-generation encryption protocol for Matrix to potentially replace Olm and Megolm. This work was badly disrupted by RHUL mitigations, but we’re making good progress again - you can follow all the details over at https://arewemlsyet.com. Matrix over DMLS is currently in alpha, but the aim is to start beta testing Decentralised MLS in 2023.
Finally, we’ve been working hard on completely reworking the overall UX of how E2EE should work in Matrix clients - specifically, requiring users to cross-sign their devices in order to use E2EE, and so end up in a much higher trust world (alongside Trust On First Use). Can’t wait to finally simplify the E2EE UX!
All new features
It’s not all been performance and stability work this year - there have been some large areas of feature work happening too.
One of the most visible projects has been Threads, which launched in beta in April, and subsequently has undergone huge amounts of polish to improve performance, notification semantics, unread behaviour and thread-aware read receipts. The end result is feeling great now, and threads exited beta in Element Mobile on Dec 20th. Web narrowly missed the window due to a final ‘stuck notification’ bug which is still being tracked down, but will follow shortly afterwards and then threads will be finally out of beta!
Another big project in 2022 has been to create a general purpose Rich Text Editor to provide WYSIWYG (What You See Is What You Get) message composition for Matrix clients. This has ended up being a very ambitious project to define all the core editing semantics in a shared rust library, with platform-specific bindings to link it into the editing UI available on Web, iOS & Android. The end result lives at https://github.com/matrix-org/matrix-rich-text-editor - and you can play with it by enabling it in labs on Element Web/iOS/Android or experiment with the live demo. The core behaviour is feeling excellent, although predictably some of the fine detail is very fiddly to get right. It’s almost there, though, and thanks to its built-in rust test harness generator(!) we are confident we’ll catch and control all the edge cases, and this should form an incredibly strong platform for all future rich text editing requirements in Matrix (and beyond!). This work was very kindly sponsored by one of Element’s public sector customers in order to get Element to parity with Teams - thank you!
Location Sharing was another feature which landed in 2022 - powered by MSC3488 and MSC3489, and implemented in matrix-{js,ios,android}-sdk in Element Web/iOS/Android, letting users share static and live locations and view them on an OpenStreetMap compatible map tileserver of their server’s choice. The Live Location Sharing is controversial in that it stores location data in the room history (and as such is hidden behind a labs flag on Element), but should eventually be replaced by MSC3672 to share locations are custom ephemeral events instead (once custom EDUs land) in the spec. Around the same time, Polls also went live thanks to MSC3381 - it’s worth noting that both Location Sharing and Polls are excellent examples of “extensible events” in the wild: ensuring that clients which understand the custom event type will render them appropriately, but letting other clients fall back to showing them as simple timeline events.
Open ID Connect
The transition to using Open ID Connect for Matrix authentication has been progressing steadily throughout 2022 - with Third Room being the first OIDC-native Matrix client, closely followed by Element X. matrix-authentication-service now exists as a basic OIDC identity provider suitable for being linked into Synapse, and meanwhile Third Room demonstrates how you can integrate Keycloak as a third party IDP (complete with reCAPTCHA and guest access!). The team also went on a very exciting detour to figure out how to perform login-and-E2EE-setup in a single operation by scanning a QR code (MSC3906), and how it might integrate into OIDC in future.
Element X looks set to be the showcase for native OIDC in a typical Matrix client going forwards, so watch this space to see how it feels!
2022 was the year that Matrix finally got native multiparty VoIP. After launching Element Call Beta 1 in March followed by Beta 2 in June, we’ve been busy embedding Element Call as a “matryoshka” widget into Element Web, using it to replace Jitsi in powering video rooms and video calls. You can read all about this in detail in the summer blog post.
Meanwhile, lots of progress is underway on Waterfall - the name we picked for the Pion-based decentralised Selective Forwarding Unit (i.e. conferencing focus) contributed by Sean DuBois earlier in the year, including adding simulcast support to support large scale conferences.
There’s only one catch, which is that Element Call is still in (very very late) beta, thanks to a handful of bugs which have been hard to track down, which has in turn kept all the other dependencies (embedded Element Call; video rooms etc) in beta too. However, we think we’re pretty much there now - which is perfect timing given how Waterfall is coming together, meaning that both stable and scalable native Matrix conferences are on the horizon!
Even better, the plan is for Element X to rely entirely on embedding Element Call for VoIP - so we should be able to jump forwards pretty rapidly to having excellent native multiparty VoIP and video rooms on mobile as well as on Web. So, once Element Call exits beta, everything should follow. Just for a change, we’re aiming to get this done by the end of January - but there are a lot of unknown unknowns still flying around, so watch this space…
IETF & MIMI
Another massive new initiative this year has been the process of proposing Matrix to the IETF as a candidate for use in interoperable instant messaging standardisation. The MIMI (More Instant Messaging Interoperability) working group emerged earlier in the year within IETF as an initiative to define how MLS could be used to interoperate between different instant messaging silos - as shortly to be required by the Digital Markets Act.
One of the things that MIMI aims to do is to define a common application layer protocol to exchange messages. At first CPIM was proposed (an ancient message format that looks a lot like email) - and then an entirely new JSON message format proposal emerged which looks somewhat Matrix (but isn’t). At this point it became obvious that we should throw our hat into the ring and encourage MIMI to use Matrix rather than reinvent it, and so we set about proposing Matrix as at least the message format and message transport layer of the stack. It’s quite surreal to see Matrix starting to fly around as IETF Drafts!
The next step here is to re-express the relevant bits of the current Matrix spec as self-contained IETF Drafts (rather than backreferencing the current spec from the drafts). The idea is that the normal Matrix spec will continue to evolve much as it always has, but we’ll effectively donate a Long Term Supported dialect of it to IETF which can then evolve according to IETF process and be immortalise as RFCs for use in MIMI. We’ll then backport those changes into spec.matrix.org in order to avoid fragmentation, while retaining the same ability we have to rapidly iterate and extend Matrix with MSCs. This work is well under way (taking opportunity to use Extensible Events from the outset!), and we should see explosions of further IETF Drafts emanating from Travis as 2023 progresses.
Trust & Safety
2022 saw a real uptick in spam and abuse across Matrix, and there have been some valiant attempts to improve our moderation tooling over the course of the year. Unfortunately it hasn’t come together as rapidly as we might have hoped, however, and we’ve seen several large communities give up on Matrix and move back to Discord thanks in part to needing better anti-abuse mechanisms.
In 2023 we’re resetting our trust & safety work, with Mjolnir dev returning to its original development team, and we’ll be working as tactically as possible to ensure that all communities on Matrix can easily block abuse using whatever mechanisms they need.
P2P & Dendrite
Meanwhile, Dendrite (our second generation homeserver implementation) development has continued at pace throughout the year. According to sytest we are now at 93% client-server API compliance with 577 out of 620 tests passing, and the server-server API compliance is at 97% with 111 out of 114 tests passing! None of the missing tests are showstoppers, so it’s fair to say that Dendrite is very nearly ready for primetime.
The interesting plot twist is that Dendrite development has ended up increasingly focusing on embedded matrix server use cases - particularly to power Peer-to-Peer Matrix, where clients require a server to be embedded within them. So while Synapse has ended up increasingly focusing on large-scale deployments, Dendrite has ended up pursuing smaller instances (which is ironic, given originally it was meant to be the other way round!).
P2P Matrix work has been progressing well too - you can follow the blow-by-blow updates over at https://arewep2pyet.com. After a lot of back and forth evaluating hard-state routing versus soft-state routing in Pinecone, we’ve ended up converging on soft-state routing (which is chattier, but easier to reason about in terms of mitigating attacks). However, the chattiness means that it doesn’t scale as well as one might hope - so we’re now working on a “tiered” approach where separate Pinecone networks can be tiered together into one inter-network, giving us scalability at the expense of being slightly less decentralised. It’s fair to say that the journey here has been pretty frustrating in its twists and turns, and sadly Neil Alexander chose to move on a few months ago. However, Devon has stepped up to fill his shoes as primary Pinecone and P2P wrangler, and is making amazing progress on the remaining work - firstly implementing Store and Forward relaying in Dendrite so that today’s Pinecone networks can exchange messages even if the recipient node is offline. Next up will be bridging P2P Matrix with today’s Matrix network - and then working on tiering to provide the scalability we need. The expectation is that today’s serverside Dendrite instances will effectively turn into static pinecone peers, store and forwarding messages on behalf of P2P nodes, and providing tiering between respective pinecone subnets.
Hydrogen & Chatterbox
Development on Hydrogen as a super-lightweight progressive-web-app Matrix client has also been progressing throughout the year (with a few detours to help out with end-to-end testing via trafficlight both for the benefit of Hydrogen and other clients).
The biggest change has been Hydrogen sprouting a separate SDK layer, letting the engine be embedded into other webapps in order to add noninvasive Matrix messaging with as minimal a footprint as possible. This was showcased in Element’s chatterbox offering in July - providing an open source chatbox which can be trivially embedded into existing sites, and also powers the Chatrix wordpress plugin that Automattic is working on.
Hydrogen also added independent support for MSC3401 multiparty voice/video calling (albeit on a branch), letting us showcase heterogeneous Element Call <-> Hydrogen group calling and prove MSC3401 as fit for purpose as a true open interoperable call signalling - and in turn Hydrogen SDK, complete with the multiparty voice/video calling, powers the Matrix engine within Third Room - our metaverse-on-Matrix platform.
We’re looking forwards to Hydrogen continuing to reach full feature parity with Element over the next year, and popping up in increasingly unexpected places as everyone’s favourite embedded Matrix client!
Third Room
Finally, it’s hard to believe that Third Room, our Matrix-based open platform for decentralised realtime spatial collaboration, barely existed at the beginning of the year. Third Room serves to demonstrate that Matrix is way more than just chat and VoIP, but can power the spatial communication layer of the open web. This has ended up driving forwards a tonne of new capabilities for Matrix - showcasing native OIDC auth; scalable multiparty VoIP in Hydrogen SDK, efficient binary-diffed file storage, and more recently has been defining how to store extensible behaviour for Matrix rooms as WASM objects stored in the Matrix room itself.
Third Room itself is a Hydrogen-based Matrix client, which lets you view Matrix rooms as interactive multiparty 3D environments (using MSC3815) - with the world defined as glTF blobs stored in the Matrix room, and the ability to script and customise any aspect of that world using WASM blobs stored in Matrix rooms, which execute on the participating clients, exposing a new scenegraph API called WebSceneGraph in order to manipulate the glTF that makes up the world. We also expect to see a variant of Matrix’s normal widget API to be exposed to these WASM blobs, introducing the concept of sandboxed clientside widgets, bots or other integrations - letting users customise and extend Matrix without ever having to run serverside bots again.
The intention is to provide a platform which can be used to build any kind of interactive realtime spatial multiparty app in an open standardised, decentralised, end-to-end encrypted way - whether that’s for gaming, social, or professional activity such as building “digital twins” for manufacturing, agriculture, smart cities, search & rescue, etc. You can read more about the vision at thirdroom.io/preview, or via press coverage at TheNewStack or Golem. We were also incredibly flattered to be invited to present Third Room at SIGGRAPH Asia a few weeks ago. The official recording has yet to emerge, but you can find a cheeky bootleg here.
We launched Tech Preview 1 of Third Room at the end of September, and since then all of the work has been around building out WebSceneGraph and the WASM scripting environment - letting users build their own functionality in JS via QuickJS or C (and in future Rust or Zig too). We’ve also been working on making the networking (via Matrix WebRTC-negotiated data channels) more robust, switching to an ‘authoritative’ simulation model rather than having each client run its own physics simulation, in order to kick the hard problem of decentralised physics simulations down the road a bit further. We’re also adding in a much-needed ‘discover’ page to help users find new rooms and explore everything that’s possible in the platform. And finally, we’re adding WebXR support so that folks can use ThirdRoom with VR and AR hardware if they so desire. All this should culminate in Tech Preview 2, due in the coming weeks.
So there you have it: it’s been a mixed year for Matrix, but at least the project itself is moving forwards faster than ever, for now. If you look back at the predictions from last year’s holiday blog post you’ll see that most of them even came true. This year, we’ll keep the predictions simple: our plans for 2023 are to ensure that the Foundation is well funded, ship all of the step-change improvements in performance and usability which are currently in beta as rapidly as possible - and demonstrate for once and for all that Matrix can indeed punch its weight against the proprietary centralised alternatives.
If you can afford it, please consider donating to the Matrix.org Foundation to support our work. The most efficient way to support us is to donate via donorbox. Our Patreon is not going anywhere, so if you wish to keep supporting it there we're happy to count you in our supporters.
TL;DR: you can now officially join the Matrix Foundation as an organisational
or individual member in order to sustainably support core Matrix development,
help steer the direction of the protocol and how best to fund it! Organisations
can join by filling this form and we will get back in touch, or individuals can
now donate directly here (as a more efficient alternative to Patreon, which
remains online for Patrons used to it).
Hi all,
Only two days late for Giving Tuesday (and 4 years late on the Foundation
scale), we are super excited to announce that we are finally expanding the
Matrix Foundation into a fully-fledged non-profit fund-raising organisation to
help support the core Matrix development and the wider open source Matrix
ecosystem!
Has your organisation been using Matrix to communicate via the open source
server and clients and you want to ensure that improvements and features still
keep coming? Become a member, and feed into the roadmap.
Has your company been building on top of Matrix, making the most of its openness
and flexibility, but you’ve never figured out how to contribute back in order
to ensure the resilience of the tech which you rely on for the success of your
business? Become a member, and ensure the core development is funded and that
Matrix is here to thrive.
Are you about to be designated a gatekeeper by the EU, and have to rapidly figure out how to implement DMA-compliant interoperability (complete with end-to-end encryption compatibility) into your services? Become a member, and discover how Matrix can solve all your problems, while providing your input too.
Are you looking to implement DMA interoperability but don't fall into the gatekeeper designation? Become a member, and help ensure Matrix fits your needs too!
Are you a non-profit organisation with a mission to provide secure and sovereign communications to those who need it the most, providing an alternative to the current players? Become a member, and help us take our mission forward.
Are you an individual who strongly supports the mission of Matrix and wants to see it thrive and become the open backbone of the world’s communications? Become a member, and support the future of Matrix.
In short, this finally gives the wider world a way to contribute concretely to the significant costs of funding folks to work fulltime on core Matrix development - which now covers over 243(!) projects in github.com/matrix-org. Core Matrix work ranges from:
managing the Matrix spec itself
maintaining the reference client SDKs and encryption and getting them independently audited
maintaining example server implementations in the form of Synapse and Dendrite
writing the test suite
publishing the matrix.org website
promoting awareness of Matrix
running the matrix.org homeserver
and so much more...
In exchange for supporting the Foundation, and beyond providing certainty that our work can continue, organisations and individuals who contribute financially will be able to directly provide input to the trajectory of the core Matrix developments by becoming official members of the Matrix.org Foundation.
Introducing Foundation Memberships and the Governing Board
Practically speaking, this means that we are going to create a "Governing Board" to the Matrix.org Foundation: a new team, made up of members voted by the overall membership, a subset of the Guardians and a subset of the Spec Core Team. The Governing Board will have the responsibility of determining how Foundation funds are distributed and used, how the Spec Core Team roadmap is prioritised, how to best grow Matrix awareness, etc.
In other words, we are literally expanding the day-to-day steering of the direction of the Matrix Foundation to the wider community. In order to run the Governing Board and the overall work of the Foundation, we are also hiring an Executive Director, so please get in touch via [email protected] if you’re a non-profit foundation-running expert! In the interim, while this search proceeds, Matthew and Amandine will fill the role with the support of the other Guardians of Matrix. The Guardians themselves retain their existing function as the Foundation's non-executive board - responsible for safeguarding the overall mission of Matrix, appointing the membership of the Spec Core Team, approving membership applications, and defining the overall changes we’re outlining here.
Membership comes at various levels, each with different rewards:
Ability to vote in the appointment of up to 2 ‘community representatives’ to the Foundation's governing board.
Name on the Matrix.org website
Silver member: between £2,000 and £80,000 per year, depending on organisation size
Ability to vote on the appointment of up to 2 ‘Silver representative’ to the Foundation's governing board
Supporter logo on the front page of the new Matrix.org website
Gold member: £200,000 / year, adds:
Ability to vote on the appointment of up to 3 ‘Gold representatives’ to the Foundation's governing board.
Press release announcing the sponsorship
1 original post on the Matrix.org blog per year
Participation in the internal Spec Core Team room
Larger logo on the front page of Matrix.org
Platinum member: £500,000 / year, adds:
Ability to vote on the appointment of up to 5 ‘platinum representatives’ to the Foundation's governing board.
1 sponsored Matrix Live episode per year
Largest logo on the front page of Matrix.org
As the activities of the Foundation increase, we expect to add more benefits to this list, for example discounts on sponsoring a future Matrix Conference, or similar.
Governing Board elections will occur yearly, with the first election planned towards the end of this year once we’ve gathered together the first wave of members and candidates have proposed themselves for election.
For anyone building on Matrix, memberships are a no-brainer as they will ensure the perpetuity and future-proof-ness of the standard. But for anyone supporting the mission of Matrix, this membership can be key to define its future.
Meanwhile, with Matrix looking increasingly integral to implementing interoperable communication for compliance with the EU’s Digital Markets Act (whether that’s as pure Matrix, or as part of the IETF MIMI initiative), this is an incredible opportunity for the organisations impacted by DMA to get a front-row seat within the Foundation to ensure that Matrix thrives and solves the challenges posed by the act. To get involved, please apply via the membership application form.
So, why is this all happening?
Since 2017, core Matrix development has been funded primarily by Element - the company founded by the team who created Matrix. Over the years, Element has put tens of millions of dollars into Matrix - which in turn has come from both selling Matrix hosting (EMS), on-premise Matrix solutions, and VC investment in Element. To put it in perspective, even though there are over 5000 contributors to github.com/matrix-org - over 90% of the actual committed lines of code come from Element employees. Similarly, while we are enormously thankful for the past and existing generous donations from the wider Matrix community, today they only come to $6,000 a month, relative to the $400,000 a month that Element has been funding.
Over the last year, we’ve seen a palpable shift within the Matrix ecosystem. Matrix is growing faster than ever. Synapse has improved immeasurably, using less RAM than ever and even sprouting Rust to optimise its hot paths. Element has improved immeasurably too, with an entirely new design on mobile, and tons of new features including threads, voice messages, location share, video rooms and more. Monthly active users reported via Synapse’s phone-home stats have almost doubled and are growing at their fastest ever rate. The number of servers has increased equivalently. We hear about major new commercial Matrix deployments almost every day. However, while usage is going through the roof - we haven’t seen a matching increase in players looking to support the project.
In fact, we’ve seen the opposite: commercial vendors forking the protocol while trying to break up the core team. Matrix tenders lost to “preferred” vendors who know absolutely nothing about Matrix. Vendors selling Matrix hosting or services without contributing anything back to the project at all. Organisations with huge amounts of money (governments, $$B massive enterprises) have enthusiastically launched proprietary Matrix solutions by building on the liberally-licensed Apache reference Matrix implementations… while contributing back nothing. Now, we are enormously grateful for the commercial Matrix deployments who do actually work with Element to fund core development or contribute code themselves - but this is very clearly the minority. Obviously it’s great to see folks building on Matrix - but it’s rather galling if it ends up with insufficient funding trickling down to the core Matrix team to be able to build the foundational technology that everyone else is relying on.
Element in particular has staffed up in order to both support the core Matrix ecosystem (the spec, Synapse, Dendrite, Client SDKs, Encryption implementations etc) as well as Element-specific work (the Element apps, EMS, the Element Enterprise Installer, etc). As a result, Element employs roughly twice as many developers as you might expect, and while Matrix is here to stay, this turns out not to be sustainable for Element unless the wider ecosystem helps support the foundational work
We believe the world needs secure decentralised communication more than ever, and ensuring the Foundation can distribute funding to those contributing to the core platform (be that Element or individuals or other organisations) is key to that. So: please consider this a call to arms - if you believe the world needs Matrix, and particularly if you depend on it for your business, please join the Foundation and participate!
Fill in the form to apply for membership, or if you are not acting on behalf of an organisation, you can go straight to our donation page.
Matthew, Amandine & the whole Matrix.org Foundation
This year, the Matrix.org Foundation is excited to host the first ever
Matrix.org Foundation and Community devroom in person at FOSDEM. Half a day of
talks, demos and workshops around Matrix itself and projects built on top of
Matrix.
We encourage people working on the Matrix protocol or building on it in an open
source project to submit a proposal! Note that companies are welcome to talk
about the Matrix details of their open source projects, but marketing talks are
not welcome.
This call for participation is only about the physical devroom. A separate CfP
will be issued for the online devroom once there are more details about it.
Key dates are:
Conference dates 4-5 February, 2023 In person
Matrix Devroom date: Sunday 5 morning in person, online devroom to be
announced
Submission deadline: Monday 5th December
Announcement of selected talks: Thursday 15th December
You must be available in person to present your talk for the physical devroom.
Talk Details
The talks can follow one of thee two formats for the physical devroom
20 min talk + 10 min Q&A, for topics that can be covered briefly
50 min talk + 10 min Q&A for more complex subjects which need more focus
We strongly encourage you to prepare a demo when it makes sense, so people can
actually see what your work looks like in practice!
Of course, the proposal must respect the FOSDEM terms as well:
The conference language is English. All content must relate to Free and Open
Source Software. By participating in the event you agree to the publication of
your recordings, slides and other content provided under the same licence as
all FOSDEM content (CC-BY).
We expect to receive more requests than we have slots available. The devroom
organisers will be reviewing the proposals and accepting them based on the
potential positive impact the project has on Matrix (as defined in by the
Mission section of https://matrix.org/foundation).
If a project proposal has been turned down, it doesn't mean we don't believe it
has good potential. Maintainers are invited to join the #twim:matrix.org
Matrix room to give it some visibility.
As of Synapse 1.69, we consider "faster remote room joins" to be ready for testing by server admins.
There are a number of caveats, which I'll come to, but first: this is an important step in a project which we've been working on for 9 months. Most people who use Matrix will be familiar with the pain of joining a large room over federation: typically you are just faced with a spinner, which is eventually replaced by a cryptic error. If you're lucky, the room eventually pops up in your room list of its own accord. The whole experience is one of the longest-standing open issues in Synapse.
At the end of each year it’s been traditional to do a big review of everything that the Matrix core team got up to that year, and announcing our predictions for the next. You can see the last edition in 2021 here - and if you’re feeling nostalgic you can head down memory lane with the 2020, 2019, 2018 ones etc too.
This year is turning out to be slightly different, however. Our plans for 2022 are particularly ambitious: to force a step change in improving Matrix’s performance and usability so that we firmly transition from our historical “make it work” and “make it work right” phases into “making it fast”. Specifically: to succeed, Matrix has to succeed in powering apps which punch their weight in terms of performance and usability against the proprietary centralised alternatives of WhatsApp, Discord, Slack and friends.
We’ve seen an absolute tonne of work happening on this so far this year… and somehow the end results all seem to be taking concrete shape at roughly the same time, despite summer traditionally being the quietest point of the year. The progress is super exciting and we don’t want to wait until things are ready to enthuse about them, and so we thought it’d be fun to do a spontaneous Summer Special gala blog post so that everyone can follow along and see how things are going!
We have always focused on first making Matrix “work right” before we make it “work fast” - sometimes to a fault. After all: the longer you build on a given architecture the harder it becomes to swap it out down the line, and the core architecture of Matrix has remained essentially the same since we began in 2014 - frankly it’s amazing that the initial design has lasted for as long as it has.
Over the years we’ve done a lot of optimisation work on the core team implementations of that original architecture - whether that’s Synapse or matrix-{js,react,ios,android}-sdk and friends: for instance Synapse uses 5-10x less RAM than it used to (my personal federated server is only using 145MB of RAM atm! 🤯) and it continues to speed up in pretty much every new release (this PR looks to give a 1000x speedup on calculating push notification actions, for instance!). However, there are some places where Matrix’s architecture itself ends up being an embarrassingly slow bottleneck: most notably when rapidly syncing data to clients, and when joining rooms for the first time over federation. We’re addressing these as follows…
Historically, /sync always assumed that the client would typically want to know about all the conversations its user is in - much as an IRC client or XMPP client is aware of all your current conversations. This provided some nice properties - such as automatically enabling perfect offline support, simplifying client and server development, and making features like “jump to room” and “tab complete” work instantly given the data is all client-side. In the early days of Matrix, when nobody was yet a power user, this wasn’t really a problem - but as users join more conversations and join bigger rooms, it’s become one of Matrix’s biggest performance bottlenecks. In practice, logging into a large account (~4000 rooms) can take ~10 minutes and hundreds of megabytes of network traffic, which is clearly ridiculous. Worse: if you go offline for a day or so, the incremental sync to catch back up can take minutes to calculate (and can even end up being worse than an initial sync).
To fix this, we started work on Sliding Sync (MSC3575) in 2021: a complete reimagining of the /sync API used by Matrix clients to receive data from their homeserver. In Sliding Sync, we only send the client the data it needs to render its UI. Most importantly, we only tell it about the subset of rooms which it is visible in the scroll window of its room list (or that it needs to display notifications about). As the user scrolls around the room list, they slide the window up and down - hence the name “sliding sync”. Sliding Sync was originally called Sync v3, given it’s our 3rd iteration of the sync API - it got renamed Sliding Sync given the current sync API confusingly ended up with a prefix of /v3.
Back in December our work on Sliding Sync was still pretty early: we had the initial MSC, an experimental proxy that converted the existing sync v2 API into Sliding Sync, and a simple proof-of-concept web client to exercise it. Since then, however, there has been spectacular progress:
MSC3575 has undergone some bigiterations as we converge on the optimal API shape.
The sliding-sync proxy has matured to be something which we’re now running in stealth against matrix.org for those dogfooding the API
We added the concept of extensions to split out how to sync particular classes of data (to avoid the API becoming a monolithic monster) - specifically:
Account Data
End-to-end Encryption
To-device messages
Ephemeral events (to be done)
Presence (to be done)
We added support for spaces!
We implemented it in matrix-js-sdk (which merged a few weeks ago!)
But most importantly, we’ve also been busy implementing Sliding Sync in Element Web itself so we can start using it for real. Now, this is still a work in progress, but as of today it’s just getting to the point where one can experiment with it as a daily driver (although it’s definitely alpha and we’re still squishing bugs like crazy!) - and we can see just how well it really performs in real life.
For instance, here’s a video of my account (4055 rooms, redacted for privacy) logging in on an entirely fresh browser via Sliding Sync - the actual sync takes roughly 1 second (at 00:18 in the video). And if we’d started the sync operation while the user is setting up E2E encryption, it would have completed in the background before they even got to the main screen, giving instant login(!). Given my account typically takes ~10 minutes to initial sync (plus even more time for encryption to sync up), this is at least a real-life 600x improvement. Moreover, the sync response is only 20KB (a ~5000x improvement) - a huge win for low-bandwidth Matrix situations.
Then, having logged in, the client subsequently launches pretty much instantly, no matter how long you’ve been offline. Total launch time is roughly 4 seconds, most of which is loading the app’s assets - which in turn could well be improved by progressively loading the app. It could also be sped up even more if we cached state locally - currently the implementation simply reloads from the server every time the app launches rather than maintaining a local cache.
As you can see, this is palpably coming together, but there’s still a bunch of work to be done before we can encourage folks to try it, including:
Switching the RoomList to be fully backed by sliding sync (currently the v2 roomlist is jury-rigged up to the sliding sync API, causing some flakey bugs such as duplicate rooms)
Spec and hook up typing / receipts / presence extensions
Hook up favourites, low_priority, knocked and historical rooms
Adding back in loading room members
Apply quality-of-service to to-device messages so we prioritise ones relevant to the current sliding window
Sync encrypted rooms in the background to search for notifications (and for indexing).
More local caching to speed up operations which now require checking the server (e.g. Ctrl/Cmd-K room switching)
We also need to determine whether it’s viable to run the sliding-sync proxy against matrix.org for general production use, or whether we’ll need native support in Synapse before we can turn it on by default for everyone. But these are good problems to have!!
Meanwhile, over in the land of Rust, we’ve been making huge progress in maturing and stabilising matrix-rust-sdk and exercising it in Element X: the codename for the next generation of native Element mobile apps. Most excitingly, we literally just got the first (very alpha) cut of Sliding Sync working in matrix-rust-sdk and hooked up to Element X on iOS - you can see Ștefan’s demo from last week here:
matrix-rust-sdk itself is now getting a steady stream of releases - including long-awaited official node bindings, providing excellent and performant encryption support via the newly audited vodozemac Rust implementation of Olm. It’s also great to see loads of major contributions to matrix-rust-sdk from across the wider Matrix community - particularly from Ruma, Fractal, Famedly and others - thank you!! As a result the SDK is shaping up to be much more healthy and heterogeneous than the original matrix-{js,ios,android}-sdk projects.
On Element X itself: matrix-rust-sdk is being used first on iOS in Element X iOS - aiming first for launching a stable “barbecue” feature set (i.e. personal messaging) asap, followed by adding on “banquet” features (i.e. team collaboration) such as spaces and threads afterwards. We’ve shamelessly misappropriated the barbecue & banquet terminology from Tobias Bernard’s excellent blog post “Banquets and Barbecues” - although, ironically, unlike the post, our plan is still to have a single app which incrementally discloses the banquet functionality as the user’s barbecue starts to sprawl. We’ve just published the brand new development roadmap for Element X from the rust-sdk perspective on GitHub. Above all else, the goal of Element X is to be the fastest mobile messenger out there in terms of launch and sync time, thanks to Sliding Sync. Not just for Matrix - but the fastest messenger, full stop :D Watch this space to see how we do!
Finally: Element is getting a major redesign of the core UI on both iOS and Android - both for today’s Element and Element X. I’m not going to spoil the final result (which is looking amazing) given it’ll have a proper glossy launch in a few weeks, but you can get a rough idea based on the earlier design previewed by Amsha back in June:
In addition to the upcoming overall redesign, Element also landed a complete rework of the login and registration flows last week on iOS and Android - you can see all about it over on the Element blog.
In terms of performance, the other area that we’re reworking at the protocol level is room joins.
One of the most glaring shortcomings of Matrix happens when a new server admin excitedly decides to join the network, installs a homeserver, tries to join a large room like #matrix:matrix.org, and then looks on in horror as it takes 10+ minutes to join the room, promptly despairs of Matrix being slow and complains bitterly about it all over HN and Reddit :)
The reason for the current behaviour is that the Matrix rooms are replicated between the servers who participate in them - and in the initial design of the protocol we made that replication atomic. In other words, a new server joining a room picks a server from which to acquire the room (typically the one in the room’s alias), and gets sent a copy of all the state events (i.e. structural data) about the room, as well as the last 20 or so messages. For a big room like Matrix HQ, this can be massive - right now, there are 79,969 state events in the room - and 126,510 auth_chain events (i.e. the events used to justify the existence of the state events). The reason there are so many is typically that the act of a user joining or parting the room is described by a state event - and in the naive implementation, the server needs to know all current state events in the room (e.g. including parted users), in order to keep in sync with the other servers in the room and faithfully authorise each new event which they receive for that room.
However, each event is typically around 500 bytes in size, and so the act of joining a big room could require generating, transmitting, receiving, authenticating and storing up to 100MB of JSON 😱. This is why joining big rooms for the first time is so painfully slow.
Happily, there is an answer: much as Sliding Sync lets clients synchronise with the bare minimum of data required to render their UI, we’ve created MSC3706 (and its precursor MSC2775) in order to rework the protocol to let servers receive the bare minimum of state needed to join a room in order to participate. Practically speaking, we only really care about events relevant to the users who are currently participating in the room; the 40,000 other lurkers can be incrementally synced in the background so that our membership list is accurate - but it shouldn’t block us from being able to join or read (or peek) the room. We already have membership lazyloading in the client-server API to support incrementally loaded membership data, after all.
The problem with this change is that Synapse was written from the outset to assume that each room’s state should be atomically complete: in other words, room state shouldn’t incrementally load in the background. So the work for Faster Joins has been an exercise in auditing the entirety of Synapse for this assumption, and generally reviewing and hardening the whole state management system. This has been loads of work that has been going on since the end of last year - but the end is in sight: you can see the remaining issues here.
As of right now, faster joins work (although aren’t enabled by default) - with the main proviso that you can’t speak in the room yet until the background sync has completed, and the new implementation has not yet been optimised. However, thanks to all the preparation work, this should be relatively straightforward, so the end is in sight on this one too.
In terms of performance: right now, joining Matrix HQ via the unoptimised implementation of faster joins completes on a fresh server in roughly 30 seconds - so a ~25x improvement over the ~12 minutes we’ve seen previously. However, the really exciting news is that this only requires synchronising 45 state events and 107 auth_chain events to the new server - a ~1400x improvement! So there should be significant scope for further optimising the calculation of these 152 events, given 30 seconds to come up with 152 events is definitely on the high side. In our ideal world, we’d be getting joins down to sub-second performance, no matter how big the room is - once again, watch this space to see how we do.
Finally, alongside faster remote joins, we’re also working on faster local joins. This work overlaps a bit with the optimisation needed to speed up the faster remote join logic - given we are seeing relatively simple operations unexpectedly taking tens of seconds in both instances. Some of this is needing to batch database activity more intelligently, but we also have some unknown pauses which we’re currently tracking down. Profiling is afoot, as well as copious Jaeger and OpenTracing instrumentation - the hunt is on!
All the work above describes some pretty bold changes to speed up Matrix and improve usability - but in order to land these changes with confidence, avoiding regressions both now and in future, we have really levelled up our testing this year.
Looking at matrix-react-sdk as used by Element Web/Desktop: all PRs made to matrix-js-sdk must now pass 80% unit test coverage for new code (measured using Sonarqube, enforced as a GitHub PR check). All matrix-react-sdk PRs must be accompanied by a mix of unit tests, end-to-end tests (via Cypress) and screenshot tests (via percy.io). All regressions (in both nightly and stable) are retro’d to ensure fixed things stay fixed (usually via writing new tests), and we have converted fully to typescript for full type safety.
Concretely, since May, we’ve increased js-sdk unit test coverage by ~10% globally, increased react-sdk coverage by ~17%, and added ever more Cypress integration tests to cover the broad strokes. Cypress now completely replaces our old Puppeteer-based end-to-end tests, and Sliding Sync work in matrix-react-sdk is being extensively tested by Cypress from the outset (the Sliding Sync PR literally comes with a Cypress test suite).
In mobile land, the situation is more complex given our long-term strategy is to deprecate matrix-ios-sdk and matrix-android-sdk2 in favour of matrix-rust-sdk. matrix-rust-sdk has always had excellent coverage, and in particular, adopting the crypto module in the current matrix-{js,ios,android}-sdk will represent a night and day improvement for quality (not to mention perf!). We’ll also be adopting PR checks, and screenshot testing for the mobile SDKs.
On the backend, we continue to build out test cases for our new integration tester Complement (in Golang), alongside the original sytest integration test suite (in Perl). In particular, we can now test Synapse in worker mode. The intention with Complement is that it should be homeserver agnostic so that any homeserver implementation can benefit. Indeed the project was initiated by Kegan wearing his Dendrite hat.
Finally, we’ve had a huge breakthrough with true multi-client end-to-end testing in the form of Michael Kaye’s brand new Traffic Light project. For the first time, we can fully test things like cross signing and verification and VoIP calls end-to-end across completely different platforms and different clients. It’s early days yet, but this really will be a game changer, especially for crypto and VoIP.
Next up, we will turn our attention to a performance testing framework so that we can reliably track performance improvements and regressions in an automated fashion - heavily inspired by Safari’s Page Load Test approach. This will be essential as we build out new clients like Element X.
All the stuff above is focused on improving the core performance and usability of Matrix - but in parallel we have also been making enormous progress on entirely new features and capabilities. The following isn’t a comprehensive list, but we wanted to highlight a few of the areas where new development is progressing at a terrifying rate…
2022 is turning out to be the year that Matrix finally gets fully native voice/video conferencing. After speccing MSC3401 at the end of last year, Element Call Beta 1 launched as a reference implementation back in March, followed by enabling E2EE, spatial audio and walkie-talkie mode in Element Call Beta 2 in June.
However, the catch was that Element Call beta 1 and 2 only ever implemented “full mesh” conferencing - where every participant calls every other participant simultaneously, limiting the size of the conference to ~7 participants on typical hardware, and wasting lots of bandwidth (given you end up sending the same upstream multiple times over for all the other participants). Element Call has been working surprisingly well in spite of this, but the design of MSC3401 was always to have “foci” (the plural of ‘focus’ - i.e. conference servers) to optionally sit alongside homeservers in order to aggregate the participating calls, a bit like this:
With foci, clients only need to send their upstream to their local focus, rather than replicating it across all the other participants - and the focus can then fan it out to other foci or clients as required. In fact, if no other clients are even watching your upstream, then your client can skip sending an upstream to its focus entirely!
Most importantly, foci are decentralised, just like Matrix: there is no single conference server as a single point of control or failure responsible for powering the group call - users connect to whichever focus is closest to them, and so you automatically get a standards-based heterogeneous network-split-resilient geographically distributed cascading conferencing system with end-to-end-encryption, powered by a potentially infinite number of different implementations. To the best of our knowledge, this is the first time someone’s proposed an approach like this for decentralised group calling (watch out, Zoom, we’re coming for you!)
Now, the VoIP team have been busy polishing Element Call (e.g. chasing down end-to-end encryption edge cases and reliability), and also figuring out how to embed it into Element and other Matrix clients as a quick way to get excellent group VoIP (more on that later). As a result, work on building out foci for scalable conferencing had to be pushed down the line.
But in the last few months this completely changed, thanks to an amazing open source contribution from Sean DuBois, project lead over at Pion - the excellent Golang WebRTC implementation. Inspired by our initial talk about MSC3401 at CommCon, Sean independently decided to see how hard it’d be to build a Selective Forwarding Unit (SFU) focus that implemented MSC3401 semantics using Pion - and published it at https://github.com/sean-der/sfu-to-sfu (subsequently donated to github.com/matrix-org). In many ways this was a flag day for Matrix: it’s the first time that a core MSC from the core team has been first implemented from outside the core team (let alone outside the Matrix community!). It’s the VoIP equivalent of Synapse starting off life as a community contribution rather than being written by the core team.
Either way: Sean’s SFU work has opened the floodgates to making native Matrix conferencing actually scale, with Šimon Brandner and I jumping in to implement SFU support in matrix-js-sdk… and as of a few weeks ago we did the first ever SFU-powered Matrix call - which worked impressively well for 12 participants!
Now, this isn’t released yet, and there is still work to be done, including:
We actually need to select the subset of streams we care about from the focus
We need to support thumbnail streams as well as high-res streams
We need rate control to ensure clients on bad connections don’t get swamped
We need to hook up cascading between foci (although the SFU already supports it!)
We need E2EE via insertable streams
Faster signalling for switching between streams
You can see the full todo list for basic and future features over on GitHub. However, we’re making good progress thanks to Šimon’s work and Sean’s help - but with any luck beta 3 of Element Call might showcase SFU support!
Meanwhile it’s worth noting that Element Call is not the only MSC3401 implementation out there - the Hydrogen team has added native support to Hydrogen SDK too (skipping over the old 1:1 calling), so expect to see Element <-> Hydrogen calling in the near future. The Hydrogen implementation is also what powers Third Room (see below…)
Elsewhere on VoIP, we’ve also been hard at work figuring out how to embed Element Call into Matrix clients in general, starting with Element Web, iOS & Android. Given MSC3401 is effectively a superset of native 1:1 Matrix VoIP calling, we’d ideally like to replace the current 1:1-only VoIP implementation in Element with an embedded instance of Element Call (not least so we don’t have to maintain it in triplicate over Web/iOS/Android, and because WebRTC-in-a-webview really isn’t very different to native WebRTC). To do this efficiently however, the embedded Element Call needs to share the same underlying Matrix client as the parent Element client (otherwise you end up wasting resources and devices and E2EE overhead between the two). Effectively Element Call ends up needing to parasite off the parent’s client. We call this approach “matryoshka embedding”, given it resembles nested Russian dolls. 🪆
In practice, we do this by extending the Widget API to let Matrix clients within the widget share the parent’s Matrix client for operations such as sending and receiving to-device messages and accessing TURN servers (c.f. MSC3819 and MSC3846). This in turn has been implemented in the matrix-widget-api helper library for widget implementers - and then a few days ago Robin demonstrated the world’s first ever matryoshka embedded Element Call call, looking like this:
Note that the MSC3401 events are happening in the actual room where the widget has been added, sent by the right users from Element Web rather than from Element Call, and so are subject to all the normal Matrix access control and encryption semantics. This is a huge step forwards from embedding Jitsi widgets, where the subsequent call membership and signalling happens in an entirely separate system (XMPP via Prosody, ironically) - instead: this is proper native Matrix calling at last.
Moreover, the same trick could be used to efficiently embed other exotic Matrix clients such as Third Room or TheBoard - giving the user the choice either to use the app standalone or within the context of their existing Matrix client. Another approach could be to use OIDC scopes to transparently log the embedded client in using the parent’s identity; this has the advantage of no code changes being needed on the embedded client - but has the disadvantage that you needlessly end up running two Matrix clients for the same account side by side, and adding another device to your account, which isn’t ideal for a performance sensitive app like Element Call or Third Room.
Matryoshka embedding isn’t live yet, but between scalable calls via SFU and native Element Call in Element Web/iOS/Android, the future is looking incredibly exciting for native Matrix VoIP. We hope to finish embedding Element Call in Element Web/iOS/Android in Sept/Oct - and if we get lucky perhaps the SFU will be ready too and then Element Call can exit beta!
Finally, we also added Video Rooms to Element Web - adding the user interface for an “always on” video room that you can hop into whenever you want. You can read about it over on the Element blog - the initial implementation uses Jitsi, but once Element Call and Matryoshka embedding is ready, we’ll switch over to using Element Call instead (and add Voice Rooms too!)
Just as MSC3401 and Element Call natively adds decentralised voice/video conferences to boring old textual Matrix chatrooms, MSC3815 and Third Room go the whole enchilada and adds a full decentralised 3D spatial collaboration environment into your Matrix room - letting you turn your Matrix rooms into a full blown interconnected virtual world.
I can’t overstate how exciting this is: one of the key origins of Matrix was back in Oct 2013 when Amandine and myself found ourselves in Berlin after TechCrunch Disrupt, debating why Second Life hadn’t been more successful - and wondering what you’d have to do to build an immersive 3D social environment which would be as positive and successful as a wildly popular chat network. Our conclusion was that the first key ingredient you’d need would be a kick-ass open decentralised communication protocol to build it on - providing truly open communication primitives that anyone could build on, much like the open web… and that was what got us thinking about how to build Matrix.
Fast forward 9 years, and Third Room is making spectacular progress in building out this dream, thanks to the incredibly hard work of Robert, Nate and Ajay. The goal of Third Room is to be an open platform layered directly on Matrix for spatial collaboration of any kind: effectively a blank canvas to let folks create freeform collaborative 3D (and in future 2D, VR or AR) experiences, either by using existing assets or building their own user-generated content and functionality. Just like the open web itself, this unlocks a literally infinite range of possibilities, but some of the obvious uses include: spatial telepresence, social VR, 3D visualisation of GIS or weather data, 3D simulated environments, search and rescue and disaster response operations (imagine streaming LIDAR from a drone surveying hurricane devastation into Third Room, where you can then overlay and collaborate on GIS data in realtime), and of course 3D gaming in all its various forms.
Now, we’re hoping to give Third Room a proper launch in a few weeks, so I’m not going to spoil too much right now - but the final pieces which are currently coming together include:
Finalising the initial version of Manifold, the multi-threaded game engine which powers Third Room (built on Three.JS, bitECS and Rapier), using SharedArrayBuffers as triple-buffers to synchronise between the various threads. See this update for a bit more detail on how the engine works.
Finalising the Matrix client interface itself, powered by Hydrogen SDK in order to be as lightweight as possible
Adding in full spatial audio and game networking via MSC3401 and Hydrogen SDK (currently full mesh, but will go SFU as soon as SFUs land!)
Adding in animated avatars (currently using Mixamo animations)
Adding in name tags and object labels
Adding in 3D Tile support in order to incrementally load 3D map tiles à la Google Earth
Building an asset pipeline from Unity and Blender through to the glTF assets which Third Room uses.
Initial framework for an in-world direct-manipulation editor
Lightmap support for beautiful high-performance static lighting and shadows
Full post-processing pipeline (bloom, depth-of-field, anti-aliasing etc)
Integrating with OIDC for login, registration, and account management (see OIDC below)
As a quick teaser - here’s an example of a Unity asset exported into Third Room, showing off lightmaps (check out the light and shadows cast by the strip lighting inside, or the shadow on the ground outside). Ignore the blurry HDR environment map of Venice in the background, which is just there to give the metals something to reflect. Check out the stats on the right-hand side: on Robert’s M1 Macbook Pro we’re getting a solid 60fps at 2000x1244px, with 13.12ms of unused gametime available for every 16.67ms frame, despite already showing a relatively complicated asset!
Meanwhile, here are some shots of Robert and Nate chasing each other around the UK City demo environment (also exported from Unity), showing off blended Mixamo animations and throwing around crates thanks to the Rapier physics engine.
And don't forget, it's just a Matrix client - with no infrastructure required other than a normal Matrix server:
As you can see, we are rapidly approaching the point where we’ll need support from technical artists to help create beautiful scenes and avatars and assets in order to make it all come to life - especially once the Blender and Unity pipelines, and/or the Third Room editor are finished. If you’re interested in getting involved come chat at #thirdroom-dev:matrix.org!
Back in the real world, a recent new project that we haven’t spoken about much yet is adding consistent WYSIWYG (What You See Is What You Get) editing to the message composer in matrix-{react,ios,android}-sdk as used by Element Web/iOS/Android - as well as publishing the resulting WYSIWYG editor for the greater glory of the wider ecosystem.
This is a bit of a contentious area, because we’ve tried several times over the years to add a rich text editor to matrix-react-sdk - firstly with the Draft.jsimplementation by Aviral (which we abandoned after Facebook de-staffed Draft), and then later with a Slateimplementation by me (which we abandoned thanks to the maintenance burden of keeping up with Slate’s API changes). Finally, burnt by the experience with third party solutions, Bruno wrote his own editor called CIDER, which was a great success and is what Element Web uses today to author messages including ‘pills’ for structured rooms/users etc… but this deliberately didn’t provide full WYSIWYG functionality. Meanwhile, Slack added WYSIWYG, forced it on, and screwed it up - and apps like WhatsApp and Discord seem to get by fine without WYSIWYG.
However, given that users are now used to WYSIWYG in Teams and Slack, we’ve now decided to have another go at it, inspired by CIDER’s success - and with the novel twist that the heavy lifting of modelling and versioning the document and handling Unicode + CJK voodoo will be provided by a cross-platform native library written in Rust, ensuring that matrix-{react,ios,android}-sdk (and in future matrix-rust-sdk-based apps like Element X) all have precisely the same consistent semantics, and we don’t spend our lives fixing per-platform WYSIWYG bugs unless it really is a platform-specific issue with the user interface provided on that particular platform.
The project is fairly young but developing fast, and lives over at https://github.com/matrix-org/matrix-wysiwyg (better name suggestions welcome ;) - we’re aiming to get it into clients by the end of October. The editor itself is not Matrix specific at all, so it’ll be interesting to see if other projects pick it up at all - and meanwhile, if we’ve done a good job, it’ll be interesting to see if this can be used to power Matrix-backed collaborative-editing solutions in future…
Update: we should have mentioned that the WYSIWYG editor project is being built out by staff at Element, who very kindly have been sponsored to work on it by one of Element's Big Public Sector Customers in order to get to parity with Teams. Thank you!!
On the other hand, a project we recently yelled about a lot is Matrix’s transition to Open ID Connect for standards-based authentication and account management. We announced this at the end of the year and the project has built up huge momentum subsequently, culminating with the release of https://areweoidcyet.com last week to track the progress and remaining work.
Our plan is to use native OIDC in production for the first time to provide all the login, registration and account management for Third Room when it launches in a few weeks (using a branded Keycloak instance as the identity provider, for convenience). After all, the last thing we wanted to do was to waste time building fiddly Matrix-specific login/registration UI in Third Room when we’re about to move to OIDC! This will be an excellent case study to see how it works, and how it feels, and inform the rest of the great OIDC experiment and proposed migration.
Meanwhile, the Next Generation team has continued to focus on their mission to make Dendrite as efficient and usable as possible. Within recent months, Dendrite has matured dramatically, with a considerable list of bugs fixed, performance significantly improved and new features added - push notifications, history visibility and presence to name a few notable additions.
Neil Alexander, Kegan and Till have continued to streamline the Dendrite architecture and to refactor areas of the codebase which have long needed attention, as well as moving from Kafka to NATS JetStream, an all-new caching model and some other fairly major architectural changes. We’ve also seen an increase of code contributions from the community and outside organisations, which is exciting, and the gomatrixserverlib library which underpins much of Dendrite is also seeing more active development and attention thanks to its use in the Complement integration testing suite.
With the most recent 0.9.3 release, we are proud to announce that Dendrite nowpasses 90% of Client-Server API tests and 95% of Server-Server API tests and has support for all specced room versions in use today. We have a growing community of users who are (quite successfully) trialling using Dendrite homeservers day-to-day, as well as our own public dendrite.matrix.org homeserver, which is open to the public for registration for anyone who wants to experiment with Dendrite without running their own deployment.
Dendrite plays an important role in our future strategy as it is also the homeserver implementation used for embedded homeservers, P2P development and experimentation. In addition to being able to scale up, we have also successfully scaled down, with the Element P2P demos proving that an embedded Dendrite homeserver can run comfortably on an iOS or Android device.
Research on the Pinecone overlay network for P2P Matrix has also continued, with Devon and Neil having experimented with a number of protocol iterations and spent considerable time bringing the Pinecone Simulator up to scratch to help us to test our designs more rapidly. Our work in this area is helping us to form a better direction and strategy for P2P Matrix as a whole, which is moving more towards a hybridised model with the current Matrix federation — a little different to our original vision, but will hopefully result in a much smoother transition path for existing users whilst solving some potential scaling problems. The arewep2pyet.com site is a living page which contains a high level overview of our goals and all the progress being made.
Comparing all of the above with the predictions for 2022 section of the end-of-year blog post, we’re making very strong progress in a tonne of areas - and the list above isn’t comprehensive. For instance, we haven’t called out all the work that the Trust & Safety team are doing to roll out advanced moderation features by default to all communities - or the work that Eric has been doing to close the remaining gap between Gitter and Matrix by creating new static archives throughout Matrix. Hydrogen has also been beavering away to provide a tiny but perfectly formed web client suitable for embedding, including the new embeddable Hydrogen SDK. We haven’t spoken about the work that the Cryptography team have been doing to adopt vodozemac and matrix-rust-sdk-crypto throughout matrix-{js,ios,android}-sdk, or improve encryption stability and security throughout. We’ve also not spoken about the new initiative to fix long-term chronic bugs (outside of the work above) in general - or all the work being done around Digital Markets Act interoperability…
Other things left on the menu for this year include getting Threads out of beta: we’ve had a bit of an adventure here figuring out how to get the right semantics for notification badges and unread state in rooms with threads (especially if you use a mix of clients which support and don’t support threads), and once that’s done we’ll be returning to Spaces (performance, group permissions etc).
Looking through this post (and congratulations if you’re still reading it at this point ;P), it really feels that Matrix is on the verge of shifting into a new phase. Much as MacOS X started off as a promising but distinctly unoptimised operating system, and then subsequently got unrecognisably faster year by year (even on the same hardware!) as Apple diligently worked away optimising the kernel… similarly: we are now landing the architectural changes to completely transform how Matrix performs.
Between protocol changes like Sliding Sync, Faster Joins, native OIDC and native VoIP conferencing all landing at roughly the same time - and alongside new implementations like matrix-rust-sdk and vodozemac, let alone Third Room - it feels unquestionably like we have an unrecognisable step change on the horizon. Our aim is to land as much of this as possible by the end of the year, and if we pull it off, I’m tempted to suggest we call the end result Matrix 2.0.
TL;DR: we’ve just launched areweoidcyet.com to track the project to adopt OpenID Connect (OIDC) as the authentication method used by Matrix. It has a load of useful resources (FAQs, status etc.) so do please check it out!
Hey folks,
As you may know, there is a proposal and project afoot to change the way authentication is done in Matrix…
Currently Matrix uses a custom authentication protocol baked into the Matrix spec. This poses a number of drawbacks. To overcome these drawbacks the project proposes migrating to use the industry standard authentication protocol OpenID Connect (OIDC) instead.
In terms of why this is a good idea: MSC3861 has all the details - please check it out!
The bottom line is that Matrix should focus on being a decentralised communication protocol - not an authentication protocol… and by adopting a dedicated authentication protocol we can benefit from all sorts of goodies such as easy 2FA and MFA, passwordless-auth via WebAuthn, Login via QR-code, alternative CAPTCHAs and so much more.
In support of this the proposal extends to the Matrix.org Foundation joining the OpenID Foundation as a non-profit member to support the work that the OpenID Foundation is doing to build a robust and audited ecosystem for open authentication.
Whilst this project proposes a significant change to the Matrix ecosystem that would take some time to migrate to, we believe that it will better support the continued growth and adoption of Matrix in the years to come.
Today we are launching the areweoidcyet.com website which is packed with information and resources on the project:
What? Why? When?
MSC proposals
Status of homeservers, clients, auth servers (OIDC Providers/OPs)
A client implementation guide
Links to the Matrix OIDC Playground environment where you can try out the latest progress
