General

Hi all,

Just over a week ago we had the honour of using Matrix to host FOSDEM: the world's largest free & open source software conference. It's taken us a little while to write up the experience given we had to recover and catch up on business as usual... but better late than never, here's an overview of what it takes to run a ~30K attendee conference on Matrix!

First of all, a quick (re)introduction to Matrix for any newcomers: Matrix is an open source project which defines an open standard protocol for decentralised communication. The global Matrix network makes up at least 28M Matrix IDs spread over around 60K servers. For FOSDEM, we set up a fosdem.org server to host newcomers, provided by Element Matrix Services (EMS) - Element being the startup formed by the Matrix core team to help fund Matrix development.

The most unique thing about Matrix is that conversations get replicated across all servers whose users are present in the conversation, so there's never a single point of control or failure for a conversation (much as git repositories get replicated between all contributors). And so hosting FOSDEM in Matrix meant that everyone already on Matrix (including users bridged to Matrix from IRC, XMPP, Slack, Discord etc) could attend directly - in addition to users signing up for the first time on the FOSDEM server. Therefore the chat around FOSDEM 2021 now exists for posterity on all the Matrix servers whose users who participated; and we hope that the fosdem.org server will hang around for the benefit of all the newcomers for the foreseeable so they don't lose their accounts!

Talking of which: the vital stats of the weekend were as follows:

• We saw almost 30K local users on the FOSDEM server + 4K remote users from elsewhere in Matrix.
• There were 24,826 guests (read-only invisible users) on the FOSDEM server.
• There were 8,060 distinct users actively joined to the public FOSDEM rooms...
• ...of which 3,827 registered on the FOSDEM server. (This is a bit of an eye-opener: over 50% of the actively participating attendees for FOSDEM were already on Matrix!)
• These numbers don't count users who were viewing the livestreams directly, but only those who were attending via Matrix.

Given last year's FOSDEM had roughly 8,500 in-person attendees at the Université libre de Bruxelles, this feels like a pretty good outcome :)

Graphwise, local user activity on the FOSDEM server looked like this:

How was it built?

There were four main components on the Matrix side:

1. A horizontally-scalable Matrix server deployment (Synapse hosted in EMS)
2. A Jitsi cluster for the video conferencing, used to host all the Q&A sessions, hallway sessions, stands, and other adhoc video conferences
3. An elastically scalable Jibri cluster used to livestream the Jitsi conferences both to the official FOSDEM livestreams and to provide a local preview of the conference on Matrix (to avoid the Jitsis getting overloaded with folks who just want to view)
4. conference-bot - a Matrix bot which orchestrated the overall conference on Matrix, written from scratch for FOSDEM by TravisR, consuming the schedule from FOSDEM and maintaining all the necessary rooms with the correct permissions, widgets, invites, etc.

Architecturally, it looked like this:

On the clientside, we made heavy use of widgets: the ability to embed arbitrary web content as iframes into Matrix chatrooms. (Widgets currently exist as a set of proposals for the Matrix spec, which have been preemptively implemented in Element.)

For instance, the conference-bot created Matrix rooms for all the FOSDEM devrooms with a predefined widget for viewing the official FOSDEM livestream for that room, pointing at the appropriate HLS stream at stream.fosdem.org - which looked like this:

Each devroom also had a schedule widget available on the righthand side, visualising the schedule of that room - huge thanks to Hato and Steffen and folks at Nordeck for putting this together at the last minute; it enormously helped navigate the devrooms (and even had a live countdown to help you track where you were at in the schedule!)

Each devroom was also available via IRC on Freenode via a dedicated bridge (#fosdem-...) and via XMPP.

The bot also created rooms for each and every talk at FOSDEM (all 666 of them), as the space where the speaker and host could hang out in advance; watch the talk together, and then broadcast the Q&A session. At the end of the talk slot, the bot then transformed the talk room into a 'hallway' for the talk, and advertised it to the audience in the devroom, so folks could pose follow-on questions to the speaker as so often happens in real life at FOSDEM. The speaker's view of the talk rooms looked like this:

On the right-hand side you can see a "scoreboard" - a simple widget which tracked which messages in the devroom had been most upvoted, to help select questions for the Q&A session. On the left-hand side you can see a hybrid Jitsi/livestream widget used to coordinate between the speaker & host. By default, the widget showed the local livestream of the video call - if you clicked 'join conference' you'd join the Jitsi itself. This stopped view-only users from overloading the Jitsi once the room became public.

The widgets themselves were hosted by the bot (you can see them at https://github.com/matrix-org/conference-bot/tree/main/web). Meanwhile the chat.fosdem.org webclient itself ended up being identical to mainline Element Web 1.7.19, other than FOSDEM branding and being configured to hook the 'video call' button up to the hybrid Jitsi/livestream widget rather than a plain Jitsi.

Meanwhile, for conferencing we hosted an off-the-shelf Jitsi cluster sized to ~100 concurrent conferences, and for the Jibri livestreaming we set up an elastic scalable cluster using AWS Auto Scaling Groups. Jibri is essentially a Chromium which views the Jitsi webapp, running in a headless X server whose framebuffer and ALSA audio is hooked up to an ffmpeg process which livestreams to the appropriate destination - so we chose to run a separate VM for every concurrent livestream to keep them isolated from each other. The Jibri ffmpegs compressed the livestream to RTMP and relayed it to our nginx, which in turn relayed it to FOSDEM's livestreaming infrastructure for use in the official stream, as well as relaying it back to the local video preview in the Matrix livestream/video widget.

Here's a screengrab of the Jitsi/Jibri Grafana dashboard during the first day of the conference, showing 46 concurrent conferences in action, with 25 spare jibris in the scaling group cluster ready for action if needed :)

There was also an explosion of changes to Element itself to try to make things go as smoothly as possible. Probably the most important one was implementing Social Login - giving single-click registration for attendees who were happy to piggyback on existing identity providers (GitHub, GitLab, Google, Apple and Facebook) rather then signing up natively in Matrix:

This was a real epic to get together (and is also an important part of achieving parity between Gitter and Element) - and seems to have been surprisingly successful for FOSDEM. Almost 50% of users who signed up on the FOSDEM server did so via social login! We should also be turning it on for the matrix.org server this week.

Finally, on the Matrix server side, we ran a cluster of synapse worker processes (1 federation inbound, reader and sender, 1 pusher, 1 initial sync worker, 10 synchrotrons, 1 event persister, 1 event creator, 4 general purpose client readers, 1 typing worker and 1 user directory) within Kubernetes on EMS. These were hooked up for horizontal scalability as follows:

The sort of traffic we saw (from day 2) looked like this:

How did it go?

Overall, people seem to have had a good time. Some folks have even been kind enough to call it the best online event they've been to :) This probably reflects the fact that FOSDEM rocks no matter what - and that Matrix is an inherently social medium, built by and for open source communities (after all, the whole Matrix ecosystem is developed over Matrix!). Also, Matrix being an open network means that folks could join from all over, so the social dynamics already present in Matrix spilled over into FOSDEM - and we even saw a bunch of people spin up their own servers to participate; literally sharing the hosting responsibility themselves. Finally, having critical infrastructure rooms available such as #beerevent:fosdem.org, #cafe:fosdem.org and #food-trucks:fosdem.org probably helped as well.

That said, we did have some production incidents which impacted the event. The most serious one was on Saturday morning, where it transpired that some of the endpoints hosted on the main Synapse process were taking way more CPU than we'd anticipated - most importantly the /groups endpoints which handle traffic relating to communities (the legacy way of defining groups of rooms in Matrix). One of the last things we'd done to prepare for the conference on Friday night was to create a +fosdem:fosdem.org community which spanned all 1000 public rooms in the conference, as well as add the +staff:fosdem.org community to all of those rooms - and unfortunately we didn't anticipate how popular these would be. As a result we had to do some emergency rebalancing of endpoints, spinning up new workers and reconfiguring the loadbalancer to relieve load off the main process.

Ironically the Matrix server was largely working okay during this timeframe, given event-sending no longer passes through the main process - but the most serious impact was that the conference bot was unable to boot due to hitting a wide range of endpoints on startup as it syncs with the conference, some of which were timing out. This in turn impacted widgets, which had been hosted by the bot for convenience, meaning that the Jitsi conferences for stands and talk Q&A were unavailable (even though the Jitsi/Jibri cluster was fine). This was solved by lunchtime on Saturday: we are really sorry for folks whose Q&As or conferences got caught in this. On the plus side, we spotted that many affected rooms just added their own widgets for their own Jitsis or BBBs to continue with minimal distraction - effectively manually taking over from the bot.

The other main incident was briefly first thing on Sunday morning, where two Jibri livestreams ended up trying to broadcast video to the same RTMP URL (potentially due to a race when rapidly removing and re-adding the jitsi/livestream widget for one of the stands). This caused a cascading failure which briefly impacted all RTMP streams, but was solved within about 30 minutes. We also had a more minor problem with the active speaker recognition malfunctioning in Jitsi on Sunday (apparently a risk when using SCTP rather than Websockets as a transport within Jitsi) - this was solved around lunchtime. Again, we're really sorry if you were impacted by this. We've learned a lot from the experience, and if we end up doing this again we will make sure these failure modes don't repeat!

Other things we'd change if we have the chance to do it again include:

• Providing a to-the-second countdown via a widget in the talk room so the speaker & host can see precisely when they're going 'live' in the devroom (and when precisely they're going to be cut in favour of the next talk)
• Providing a scratch-pad of some kind in the talk room so the host & speaker can track which questions they want to answer, and which they've already answered
• Keep the questions scoreboard and scratchpad visible to the speaker/host after their Q&A has finished so they can keep answering the questions in the per-talk room, and advertise the per-talk room more effectively.
• Use Spaces rather than Communities to group the rooms together and automatically provide a structured room directory! (Like this!)
• Use threads (once they land!) to help structure conversations in the devroom (perhaps these could even replace the hallway rooms?)
• Make the schedule widgets easier to find, and have more of them around the place
• Make room directory easier to find.
• Give the option of recording the video in the per-talk and stands for posterity
• Provide more tools to stands to help organise demos etc.

So, there you have it. We hope that this shows that it's possible to host successful large-scale conferences on Matrix using an entirely open source stack, and we hope that other events will be inspired to go online via Matrix! We should give a big shout out to HOPE, who independently trailblazed running conferences on Matrix last year and inspired us to make FOSDEM work.

If you want to know more, we also did a talk about FOSDEM-on-Matrix in this month's Open Tech Will Save Us meetup and the Building Massive Virtual Communities on Matrix talk at FOSDEM went into more detail too. Our historical Taking FOSDEM online via Matrix blog has been somewhat overtaken by events but gives further context still.

Finally, huge thanks to FOSDEM for letting Matrix host the social side of the conference. This was a big bet, starting from scratch with our offer to help back in September, and we hope it paid off. Also, thanks to all the folks at Element who bust a gut to pull it together - and to the FOSDEM organisers, who were a real pleasure to work with.

Let's hope that FOSDEM 2022 will be back in person at ULB - but whatever happens, the infrastructure we built this year will be available if ever needed in future.

Imagine you could physically step into your favourite FOSS projects’ chatrooms, mailing lists or forums and talk in person to other community members, contributors or committers? Imagine you could see project leads show off their latest work in front of a packed audience, and then chat and brainstorm with them afterwards (and maybe grab a beer)? Imagine, as a developer, you could suddenly meet a random subset of your users, to hear and understand their joys and woes in person?

This is FOSDEM, Europe’s largest Free and Open Source conference, where every year thousands of people (last year, ~8,500) take over the Solbosch campus of the Université Libre de Bruxelles in Belgium for a weekend and turn it into both a cathedral and bazaar for FOSS, with over 800+ talks organised over 50+ tracks, hundreds of exhibitor stands, and the whole campus generally exploding into a physical manifestation of the Internet. The event is completely non-commercial and volunteer run, and is a truly unique and powerful (if slightly overwhelming!) experience to attend. Ever since we began Matrix in 2014, FOSDEM has been the focal point of our year as we’ve rushed to demonstrate our latest work and catch up with the wider community and sync with other projects.

This year, things are of course different. Thankfully FOSDEM 2020 snuck in a few weeks before the COVID-19 pandemic went viral, but for FOSDEM 2021 on Feb 6/7th the conference will inevitably happen online. When this was announced a few months back, we reached out to FOSDEM to see if we could help: we’d just had a lot of fun helping HOPE go online, and meanwhile a lot of the work that’s gone into Matrix and Element in 2020 has been around large-scale community collaboration due to COVID - particularly thanks to all the development driven by Element’s German Education work. Meanwhile, we obviously love FOSDEM and want it to succeed as much as possible online - and we want to attempt to solve the impossible paradox of faithfully capturing the atmosphere and community of an event which is “online communities, but in person!”... but online.

And so, over the last few weeks we’ve been hard at work with the FOSDEM team to figure out how to make this happen, and we wanted to give an update on how things are shaping up (and to hopefully reassure folks that things are on track, and that devrooms don’t need to make their own plans!).

Firstly, FOSDEM will have its own dedicated Matrix server at fosdem.org (hosted by EMS along with a tonne of Jitsi’s) acting as the social backbone for the event. Matrix is particularly well suited for this, because:

• We’re an open standard comms protocol with an open network run under a non-profit foundation with loads of open source implementations (including the reference ones): folks can jump on board and participate via their own servers, clients, bridges, bots etc.
• We provide official bridges through to IRC and XMPP (and most other chat systems), giving as much openness and choice as possible - if folks want to participate via Freenode and XMPP they can!
• We’re built with large virtual communities in mind (e.g. Mozilla, KDE, Matrix itself) - for instance, we’ve worked a lot on moderation recently.
• We’ve spent a lot of time improving widgets recently: these give the ability to embed arbitrary webapps into chatrooms - letting you add livestreams, video conferences, schedules, Q&A dashboards etc, augmenting a plain old chatroom into a much richer virtual experience that can hopefully capture the semantics and requirements of an event like FOSDEM.

We’re currently in the middle of setting up the server with a dedicated Element as the default client, but what we’re aiming for is:

• Attendees can lurk as read-only guests in devrooms without needing to set up accounts (or they can of course use their existing Matrix/IRC/XMPP accounts)
• Every devroom and track will have its own chatroom, where the audience can hang out and view the livestream of that particular devroom (using the normal FOSDEM video livestream system). There’ll also be a ‘backstage’ room per track for coordination between the devroom organisers and the speakers.
• The talks themselves will be prerecorded to minimise risk of disaster, but each talk will have a question & answer session at the end which will be a live Jitsi broadcast from the speaker and a host who will relay questions from the devroom.
• Each talk will have a dedicated room too, where after the official talk slot the audience can pop in and chat to the speaker more informally if they’re available (by text and/or by moderated jitsi). During the talk, this room will act as the ‘stage’ for the speaker & host to watch the livestream and conduct the question & answer session.
• Every stand will also have its own chatroom and optional jitsi+livestream, as will BOFs or other adhoc events, so folks can get involved both by chat and video, to get as close to the real event as possible (although it’s unlikely we’ll capture the unique atmospheric conditions of K building, which may or may not be a bug ;)
• There’ll also be a set of official support, social etc rooms - and of course folks can always create their own! Unfortunately folks will have to bring their own beer though :(
• All of this will be orchestrated by a Matrix bot (which is rapidly taking shape over at https://github.com/matrix-org/conference-bot), responsible for orchestrating the hundreds of required rooms, setting up the right widgets and permissions, setting up bridges to IRC & XMPP, and keeping everything in sync with the official live FOSDEM schedule.

N.B. This is aspirational, and is all still subject to change, but that said - so far it’s all coming together pretty well, and hopefully our next update will be opening up the rooms and the server so that folks can get comfortable in advance of the event.

Huge thanks go to the FOSDEM team for trusting us to sort out the social/chat layer of FOSDEM 2021 - we will do everything we can to make it as successful and as inclusive as we possibly can! :)

P.S. We need help!

FOSDEM is only a handful of weeks away, and we have our work cut out to bring this all together in time. There are a few areas where we could really do with some help:

• Folks on XMPP often complain that the Bifröst Matrix<->XMPP bridge doesn’t support MAMs - meaning that if XMPP users lose connection, they lose scrollback. We’re not going to have time to fix this ourselves in time, so this would be a great time for XMPP folks who grok xmpp.js to come get involved and help to ensure the best possible XMPP experience! (Similarly on other bifrost shortcomings).
• It’d be really nice to be able to render nice schedule widgets for each devroom, and embed the overall schedule in the support rooms etc. The current HTML schedules at https://fosdem.org/2021/schedule/day/saturday/ and (say) https://fosdem.org/2021/schedule/room/vcollab/ don’t exactly fit - if someone could write a thing which renders them at (say) 2:5 aspect ratio so they can fit nicely down the side of a chatroom then that could be awesome!
• While we’ll bridge all the official rooms over to Freenode, it’d be even nicer if people could just hop straight into any room on the FOSDEM server (or beyond) via IRC - effectively exposing the whole thing as an IRC network for those who prefer IRC. We have a project to do this: matrix-ircd, but it almost certainly needs more love and polish before it could be used for something as big as this. If you like Rust and know Matrix, please jump in and get involved!
• If you just want to follow along or help out, then we’ve created a general room for discussion over at #fosdem-matrix:fosdem.org. It’d be awesome to have as many useful bots & widgets as possible to help things along.

Hi all,

Over the years it’s become a tradition to write an end-of-year wrap-up on Christmas Eve, reviewing all the things the core Matrix team has been up over the year, and looking forwards to the next (e.g. here’s last year’s edition). These days there’s so much going on in Matrix it’s impossible to cover it all (and besides, we now have This Week In Matrix and better blogging in general to cover events as they happen). So here’s a quick overview of the highlights:

Looking back at our plans for 2020 in last year’s wrap-up, amazingly it seems we pretty much achieved what we set out to do. Going through the bulletpoints in order:

• We turned on End-to-end Encryption by default.
• We have a dedicated team making major improvements to First-Time User Experience in Element (as of the last few months; hopefully you’ve been noticing the improvements!)
• RiotX became Element Android and shipped.
• Communities have been completely reinvented as Spaces (MSC1772) and while in alpha currently, they should ship in Jan.
• Synapse scalability is fixed: we now shard horizontally by event - and Synapse is now pretty much entirely async/await!
• Dendrite Beta shipped, as did the initial P2P Matrix experiments, which have subsequently continued to evolve significantly (although we haven’t implemented MSC1228 or MSC2787 portable accounts yet). Check out the Dendrite end-of-year update for more.
• MLS experiments are in full swing - we got the first MLS messages passing over Matrix a few days ago, and Decentralised MLS work is back on the menu after an initial sprint in May.
• There’s been a valiant mission to improve Bridge UX in the form of MSC2346 and its implementations in Element Web, although this has ended up failing to get to the top of the todo list (sorry Half-Shot! :/)
• Spec progress has improved somewhat, and we are very excited to have welcomed Will Bamberg (formerly MDN) to support the spec from a professional tech writer perspective, with the all-new engine landing any day now! We’re still experimenting with ways to ensure the spec gets enough time allocated to keep up with the backlog, however - particularly community contributions.
• ...and in terms of Abuse/Reputation - we properly kicked off our anti-abuse work and launched a first PoC implementation in the depths of Cerulean last week.

Perhaps more interesting is the stuff we didn’t predict (or at least didn’t want to pre-announce ;) for 2020:

• Riot, Modular and New Vector got unified at last behind a single name: Element; hopefully the shock has worn off by now :)
• Mozilla joined Matrix in force, turning off Moznet IRC in favour of going full Matrix.
• We welcomed Gitter into the heart of the Matrix ecosystem (with Element acquiring Gitter from Gitlab in order to ensure Gitter’s Matrix integration acts as a reference for integrating future chat silos into Matrix) - with native Matrix support in Gitter going live shortly afterwards.
• Automattic launched itself into the Matrix ecosystem with an investment in Element, and since then we’ve been working on getting Matrix better integrated and available to them (although all of Element’s Matrix-for-governments activity has ended up delaying this a bit). If you want to work for Automattic on integrating Matrix, they’re hiring!
• We previewed Cerulean as a super-exciting proof-of-concept client, demonstrating how social media could work on Matrix, with native threading, profiles-as-rooms, decentralised reputation, and (shortly) peeking-over-federation.
• We completely rewrote matrix.to and relaunched it as a much more capable and friendly permalink redirection service; a precursor to finally getting matrix:// URLs everywhere!
• We certainly didn’t predict that the “how to install Synapse” video tutorial published at the beginning of the COVID-19 pandemic would end up with 25.5K views (and counting…)

Then, there’s whole new waves of exciting stuff going on. The most obvious has to be the amount of Government uptake we’ve seen with Matrix this year, following on from France embracing Matrix across the public sector last year. Firstly the German armed forces announced their transition to Matrix, and then the German states of Schleswig-Holstein and Hamburg announced a mammoth 500K user Matrix deployment for education and public administration. Meanwhile, North Rhine Westphalia (the biggest state in Germany) launched their own Matrix-powered messager for education; loads of different universities have rolled out Matrix for collaboration - and we hear Famedly is making good progress with Matrix-powered healthcare messaging solutions. Finally, outside of Germany, we’re seeing the first official deployments in the UK government and US federal government - we’ll share details where possible (but sometimes big deployments of encrypted communication systems want to remain discreet). It’s incredibly exciting to see Matrix spreading across the public sector and education, and we’re hoping this will follow a similar pattern to how the Internet, email or indeed the Web first developed: a mix of high profile public sector deployments, complemented by a passionate grass-roots technical community, eventually spreading to span the rest of society :).

Another exciting thing which emerged this year is the amazing academic work that Karlsruhe Institute of Technology’s Decentralized Systems and Network Services Research Group has been conducting on Matrix. This really came on the radar back in June when their Matrix Decomposition: Analysis of an Access Control Approach on Transaction-based DAGs without Finality paper was published - a truly fascinating analysis of how state resolution works in Matrix, and how we manage to preserve access control within rooms without using blockchain-style ‘sealed blocks’ (and has helped fix a few nasty bugs!). I’m not sure any of us realised that Matrix’s state resolution counts as a new field of research, but it’s been great to follow along with their independent work. Most recently, and even more excitingly, they’re circulating a preview of their Analysis of the Matrix Event Graph Replicated Data Type paper - a deep analysis of the properties of Matrix DAGs themselves. We highly recommend reading the papers (what better way to spend the holiday break!). To give a taste, the final paragraph of the paper concludes:

2020 has also seen the arrival and maturation of a whole new generation of Matrix clients - Hydrogen is really impressive as an experimental next-generation Web (and Mobile Web) client; an account with 3000 rooms that uses 1.4GB of RAM on Element Web uses 14MB of RAM on Hydrogen and launches instantly, complete with excellent E2EE implementation. It even works on MSIE! The whole app, including dependencies, is about 70KB of code (200KB including Olm). Meanwhile, matrix-rust-sdk is coming along well, providing a general purpose native library for writing excellent native Matrix clients. Fractal merged initial matrix-rust-sdk a few weeks ago, and we’ll be experimenting with switching to it in Element iOS and Element Android (for its e2ee) in the coming year. It’s not inconceivable to think of a world where matrix-rust-sdk ends up being the no-brainer official SDK for native apps, and Hydrogen’s SDK becomes the no-brainer official SDK for JS apps.

Meanwhile, in the community, there’s been so much activity it’s untrue. But on the subject of maturing apps, it’s been incredibly exciting to see NeoChat emerge as an official KDE Matrix client (built on libQuotient and Kirigami, forked from Spectral), FluffyChat going from strength to strength; Nheko continuing to mature impressively; Mirage appearing out of nowhere as a fully featured desktop client; Fractal merging matrix-rust-sdk etc. On the serverside, Conduit was the big community story of the year - with an incredibly fast Rust + Sled server appearing out of the blue, with viable federation coming up on the horizon. The best bet for an overview of all things community is to checkout the TWIM backlogs however - there’s simply way too much stuff to mention it all here.

Obviously, no 2020 wrap-up post would be complete without acknowledging the COVID-19 pandemic - which increased focus on Matrix and other remote collaboration technology more than anyone could have predicted (especially given all the privacy missteps from Zoom, Teams and others). One of the highlights of the year was seeing the HOPE (Hackers On Planet Earth) conference shift their entire proceedings over to Matrix - turning the conference into a 10 day television station of premium hacking content, with Matrix successfully providing the social glue to preserve a sense of community despite going virtual. Similarly, we’re incredibly excited that FOSDEM 2021 is highly likely to run primarily via Matrix (with bridges to IRC and XMPP, of course) - our work is going to be cut out for us in January to ensure the amazing atmosphere of FOSDEM is preserved online for the >8,500 participants and ~800 talks. And if any other event organisers are reading this - please do reach out if you’re interested in going online via Matrix: we want Matrix to be the best possible ecosystem for online communities, including virtual events, and we’ll be happy to help :)

Talking of FOSDEM, a really fun bit of work which landed in Element this year was to (finally!) polish Widgets: the ability to embed arbitrary webapps into Matrix chatrooms. This includes being able to embed widgets in the RightPanel on Element Web, the LeftPanel too, add as many as you like to a room, resize them(!), and generally build much more sophisticated dashboards of additional content. Modal and fullscreen widgets are coming too, as are ways to simplify and unify access control. It turns out that these have arrived in the nick of time for events like FOSDEM, where we’re expecting to very heavily use widgets to embed video streams, video conferences, schedules, and generally automate the workflow of the conference via adding in web UIs as widgets wherever necessary. The work for this has been driven by the various German education deployments, where the same tricks are invaluable for automating online learning experiences. We originally wrote Widgets back in 2017 as a proof-of-concept to try to illustrate how chatrooms could be used to host proper custom UIs, and it's fantastic to see that dream finally come of age.

Finally, it’s been really exciting to see major progress in recent months on what’s essentially a whole new evolution of Matrix. Two years ago, a quiet patch during the Christmas holidays gave birth to a whole bunch of wild science fiction Matrix Spec Changes: MSC1772: Spaces (groups as rooms), MSC1769: Profiles as rooms, MSC1767: Extensible events, MSC1776: Peeking over /sync, MSC1777: Peeking over federation, etc. This was in part trying to ensure that we had something to look forward to when we emerged from the tunnel of launching Matrix 1.0, and in part trying to draw a coherent high-level sketch of what the next big wave of Matrix features could look like. Inevitably the MSCs got stuck in limbo for ages while we exited beta, launched Matrix 1.0, turned on E2EE by default etc - but in the latter half of this year they’ve hit the top of the todo list and it’s been incredibly exciting to see entirely new features landing once again. Implementation for Spaces is in in full swing and looking great; Profiles-as-rooms are effectively being trialled in Cerulean; Peeking over /sync has landed in Dendrite and peeking over federation is in PR (and unlocks all sorts of desirable features for using rooms more generically than we have today, including Spaces). Only Extensible events remains in limbo for now (we have enough to handle getting the others landed!)

Of these, Spaces has turned out to be exciting in wholly unexpected ways. While prototyping the UX for how best to navigate hierarchies of spaces, we had a genuine epiphany: the ability for anyone to define and share arbitrary hierarchies of rooms makes Matrix effectively a global decentralised hierarchical file system (where the ‘files’ are streams of realtime data, but can obviously store static content too). The decentralised access controls that KIT DSN wrote about could literally be file-system style access controls; enforcing access on a global decentralised hierarchy. We obviously have shared hierarchical filesystems today thanks to Dropbox and Google Drive, but these of course are centralised and effectively only store files - whereas Spaces could potentially scale to the whole web. In fact, you could even think of Spaces as flipping Matrix entirely on its head: the most defining building block going forwards could be the Spaces themselves rather than the rooms and events - just as directories are intrinsic to how you navigate a conventional filesystem. How has Matrix got this far without the concept of folders/directories?!

Right now these thoughts are just overexcited science fiction, but the potential really is mindblowing. It could give us a global read/write web for organising any arbitrary realtime data - with the social controls via ACLs to delegate and crowdsource curation of hierarchies however folks choose. The Matrix.org Foundation could seed a ‘root’ hierarchy, go curate all the rooms we know about into some Linnean-style taxonomy, delegate curation of the various subspaces to moderators from the community, and hey presto we’ve reinvented USENET… but with modern semantics, and without the rigid governance models. Hell, we could just mount (i.e. bridge) USENET straight into it. And any other hierarchical namespace of conversations you can think of - Google Groups, Stackoverflow, Discourse, IMAP trees…

Of course, the initial Spaces implementation is going to be focused of on letting communities publish their existing rooms, and users organise their own rooms, rather than managing an infinite ever-expanding global space hierarchy - but given we’ve been designing Spaces to support government (and inter-government) scales of Spaces, it’s not inconceivable to think we could use it to navigate gigantic public shared Spaces in the longer term.

Anyway, enough Space scifi - what’s coming up in 2021?

2021

Our current hit list is:

• Spaces - see above :)
• Social Login - we’re going to be making Single Sign On (SSO) a proper first-class citizen in Matrix (and Synapse and Element) in the coming weeks, and enabling it on the matrix.org homeserver, so users can do single-click logins via Github/Gitlab/Google and other SSO providers. Obviously this means your Matrix identity will be beholden to your identity provider (IdP), but this may well be preferable for many users who just want a single-click way to enter Matrix and don’t care about being tied to a given IdP.
• VoIP - we have a lot of work in flight at the moment to make 1:1 VoIP super robust. Some of it has already landed in Element, but the rest will land in the coming weeks - and then we’re hoping to revisit Matrix-native group voice/video.
• Voice messaging - we’re hoping to finally add voice messaging to Element (and Matrix)
• Location sharing - ...and this too.
• **P2P **- Lots of P2P work on the horizon, now Dendrite is increasingly stable. First of all we need to iterate more on Pinecone, our pre-alpha next-generation P2P overlay network - and then sort out account portability, and privacy-preserving store-and-forward. We’re hoping to see the live P2P Matrix network turn on this year, however, and ideally see homeservers (probably Dendrite) multihoming by default on both today’s Matrix as well as the P2P network, acting as gateways between the two.
• Threads - Cerulean is excellent proof for how threading could work in Matrix; we just need to get it implemented in Element!
• Peeking - Peeking is going to become so much more important for participating in non-chat rooms, such as Spaces, Profiles, Reputation feeds, etc. We’ll finish it in Dendrite, and then implement it in Synapse too.
• **Decentralised Reputation **- Cerulean has the first implementation of decentralised reputation for experimentation purposes, and we’ll be working solidly on it over the coming year to empower users to counter abuse by applying their own subjective reputation feeds to their content.
• **Incremental Signup **- Once upon a time, Element (Riot) had the ability to gradually sign-up without the user even really realising they’d signed up. We want to bring it back - perhaps this will be the year?
• DMLS - with the first MLS messages flowing over Matrix, we want to at least provide MLS as an option alongside Megolm for encryption. It should be radically more performant in larger rooms (logarithmic rather than linear complexity), but lacks deniability (the assurance that you cannot prove a user said something in retrospect, in order to blackmail them or similar), and is still unproven technology. We’ll aim to prove it in 2021.
• E2EE improvements - We improved E2EE immeasurably in 2020; turning it on by default, adding cross-signing, QR code verification etc. But usability and reliability can still be improved. We’ll be looking at further simplifying the UX, and potentially combining together your login password and recovery/security passphrase so you only have one password to remember going forwards.
• Hydrogen - We’ll keep polishing Hydrogen, bringing it towards feature parity with Element, ensure its SDK is available for other clients, and start seeing how we can use it in Element itself. For instance, the Spaces-aware RoomList in Element may well end up stealing alien technology from Hydrogen.
• matrix-rust-sdk - Similarly, we’ll keep polishing matrix-rust-sdk; stealing inspiration from Hydrogen’s state model, and start migrating bits of the native mobile Element apps to use it.
• The Spec - get Will’s new spec website live, and get improving all the surrounding material too.

I’m sure I’m missing lots here, but these are the ones which pop immediately to mind. You can also check Element's public roadmap, which covers all the core Matrix work donated by Element (as well as everything else Element is getting up to).

As always, huge huge thanks goes to the whole Matrix community for flying Matrix and keeping the dream alive and growing faster than ever. It’s been a rough year, and we hope that you’ve survived it intact (and you have our sincere sympathies if you haven’t). Let’s hope that 2021 will be a massive improvement, and that the whole Matrix ecosystem will continue to prosper in the new year.

-- Matthew, Amandine, and the whole Matrix team.

Hi all,

We have a bit of an unexpected early Christmas present for you today…

Alongside all the normal business-as-usual Matrix stuff, we’ve found some time to do a mad science experiment over the last few weeks - to test the question: “Is it possible to build a serious Twitter-style decentralised microblogging app using Matrix?”

It turns out the answer is a firm “yes” - and as a result we’d like to present a very early sneak preview of Cerulean: a highly experimental new microblogging app for Matrix, complete with first-class support for arbitrarily nested threading, with both Twitter-style (“vertical”) and HN/Reddit-style (“horizontal”) layout… and mobile web support!

Cerulean is unusual in many ways:

• It’s (currently) a very minimal javascript app - only 2,500 lines of code.
• It has zero dependencies (other than React).
  • This is to show just how simple a fairly sophisticated Matrix client can be...
  • ...and so the code can be easily understood by folks unfamiliar with Matrix...
  • ...and so we can iterate fast while figuring out threading...
  • ...and because none of the SDKs support threading yet :D
• It relies on MSC2836: Threading - our highly experimental Matrix Spec Change to extend relationships (as used by reaction & edit aggregations) to support free-form arbitrary depth threading.
• As such, it only works on Dendrite, as that’s where we’ve been experimenting with implementing MSC2836. (We’re now running an official public Dendrite server instance at dendrite.matrix.org though, which makes it easy to test - and our test Cerulean instance https://cerulean.matrix.org points at it by default).

This is **very much a proof of concept. **We’re releasing it today as a sneak preview so that intrepid Matrix experimenters can play with it, and to open up the project for contributions! (PRs welcome - it should be dead easy to hack on!). Also, we give no guarantees about data durability: both Cerulean and dendrite.matrix.org are highly experimental; do not trust them yet with important data; we reserve the right to delete it all while we iterate on the design.

What can it do?

So for the first cut, we’ve implemented the minimal features to make this something you can just about use and play with for real :)

• Home view (showing recent posts from folks you follow)
• Timeline view (showing the recent posts or replies from a given user)
• Thread view (showing a post and its replies as a thread)
• Live updating (It’s Matrix, after all! We’ve disabled it for guests though.)
• Posting plain text and images
• Fully decentralised thanks to Matrix (assuming you’re on Dendrite)
• Twitter-style “Vertical” threading (replies form a column; you indent when someone forks the conversation)
• HN/Reddit/Email-style “Horizontal” threading (each reply is indented; forks have the same indentation)
• Basic Registration & Login
• Guest support (slightly faked with non-guest users, as Dendrite’s guest support isn’t finished yet)
• Super-experimental proof-of-concept support for decentralised reputation filtering(!)

Obviously, there’s a huge amount of stuff needed for parity with a proper Twitter-style system:

• Configurable follows. Currently the act of viewing someone’s timeline automatically follows them. This is because Dendrite doesn’t peek over federation yet (but it’s close), so you have to join a room to view its contents - and the act of viewing someone’s timeline room is how you follow them in Cerulean.
• Likes (i.e. plain old Matrix reactions, although we might need to finally sort out federating them as aggregations rather than individually, if people use them like they use them on Twitter!)
• Retweets (dead easy)
• Pagination / infinite scrolling (just need to hook it up)
• Protect your posts (dead easy; you just switch your timeline room to invite-only!)
• Show (some) replies to messages in the Home view
• Show parent and sibling context as well as child context in the Thread view
• Mentions (we need to decide how to notify folks when they’re mentioned - perhaps Matrix’s push notifications should be extended to let you subscribe to keywords for public rooms you’re not actually in?)
• Notifications (although this is just because Dendrite doesn’t do notifs yet)
• Search (again, just needs to be implemented in Dendrite - although how do you search beyond the data in your current homeserver? Folks are used to global search)
• Hashtags (it’s just search, basically)
• Symlinks (see below)
• Figure out how to handle lost unthreaded messages (see below)
• Offline support? (if we were using a proper Matrix SDK, we’d hopefully get this for free, but currently Cerulean doesn’t store any state locally at all).

How does it work?

Every message you send using Cerulean goes into two Matrix rooms, dubbed the "timeline" room and the "thread" room. The "timeline" room (with an alias of #@matthew:dendrite.matrix.org or whatever your matrix id is) is a room with all of your posts and no one else's. The "thread" room is a normal Matrix room which represents the message thread itself. Creating a new "Post" will create a new "thread" room. Replying to a post will join the existing "thread" room and send a message into that room. MSC2836 is used to handle threading of messages in the "thread” room - the replies refer to their parent via an m.relationship field in the event.

These semantics play nicely with existing Matrix clients, who will see one room per thread and a flattened chronological view of the thread itself (unless the client natively supports MSC2836, but none do yet apart from Cerulean). However, as Cerulean only navigates threaded messages with an m.reference relationship (eg it only ever uses the new /event_relationships API rather than /messages to pull in history), normal messages sent by Matrix into a thread or timeline room will not yet show up in Cerulean.

In this initial version, Cerulean literally posts the message twice into both rooms - but we’re also experimenting with the idea of adding “symlinks” to Matrix, letting the canonical version of the event be in the timeline room, and then the instance of the event in the thread room be a ‘symlink’ to the one in the timeline. This means that the threading metadata could be structured in the thread room, and let the user do things like turn their timeline private (or vice versa) without impacting the threading metadata. We could also add an API to both post to timeline and symlink into a thread in one fell swoop, rather than manually sending two events. It’d look something like this:

We also experimented with cross-room threading (letting Bob’s timeline messages directly respond to Alice’s timeline messages and vice versa), but it posed some nasty problems - for instance, to find out what cross-room replies a message has, you’d need to store forward references somehow which the replier would need permission to create. Also, if you didn’t have access to view the remote room, the thread would break. So we’ve punted cross-room threading to a later MSC for now.

Needless to say, once we’re happy with how threading works at the protocol level, we’ll be looking at getting it into the UX of Element and mainstream Matrix chat clients too!

What’s with the decentralised reputation button?

Cerulean is very much a test jig for new ideas (e.g. threading, timeline rooms, peeking), and we’re taking the opportunity to also use it as an experiment for our first forays into publishing and subscribing to reputation greylists; giving users the option to filter out content by default they might not want to see… but doing so on their own terms by subscribing to whatever reputation feed they prefer, while clearly visualising the filtering being applied. In other words, this is the first concrete experimental implementation of the work proposed in the second half of Combating Abuse in Matrix without Backdoors. This is super early days, and we haven’t even published a proto-MSC for the event format being used, but if you’re particularly interested in this domain it’s easy enough to figure out - just head over to #nsfw:dendrite.matrix.org (warning: not actually NSFW, yet) and look in /devtools to see what’s going on.

So, there you have it - further evidence that Matrix is not just for Chat, and a hopefully intriguing taste of the shape of things to come! Please check out the demo at https://cerulean.matrix.org or try playing with your own from https://github.com/matrix-org/cerulean, and then head over to #cerulean:matrix.org and let us know what you think! :)

One of the goals on the core team is to reduce friction for new users joining Matrix. A challenge we regularly face is all the factors which make Matrix flexible and powerful as an open, secure decentralised protocol also increase the difficulty of getting started.

As one example— to create a healthy, vibrant ecosystem and put power back into the hands of end users, it’s critical multiple clients exist, and that users ultimately have control over which one to use. However, needing to choose a client before getting going is counter-intuitive, and adds cognitive load which proprietary services simply don’t have.

Striking this balance is tricky, and one we’re aiming to improve with the latest version of Matrix.to.

What's Matrix.to?

Matrix.to is a simple URL redirection service which lets users share links to rooms, users and communities. It isn’t tied to any client, and for end users it serves as a landing page for many as their first touch to Matrix. And until the matrix:// URI scheme is deployed commonplace (we finally registered it with IANA a few months ago!) it is the only client-agnostic way to link to content in Matrix.

It’s privacy preserving, with no information stored server side, with all calculation done in client-side JavaScript. It started life years ago as a temporary minimum viable hack by Matthew which then hung around untouched for years being… minimally viable. But we’ve recently given it a huge upgrade in usability and functionality at last!

What's new?

We’ve re-written matrix.to from scratch, giving it a visual upgrade and refocused the experience around:

1. Helping new users find the best client for them to easily get going. By default, links will prioritise showing clients which are compatible with the platform the link is being viewed from, including mobile platforms.

2. Optionally, remembering your preferred client for future visits. This also includes deeplinking into native apps for Element Desktop & Element Mobile (and in future, other clients too, of course).

3. Fetching useful previews. One of our main observations when testing with new users is a lack of confidence trying out a new service without personalised, contextual information on the rooms or people they’re heading to. Matrix.to now displays room metadata like avatars, names and topics directly (fetched via matrix.org by default, asking for permission).

What's next?

We’ve baked in the ability to specify clients and deployments within links, allowing sharing to give the option of a specific destination to guarantee a predictable experience. For instance, Mozilla might share matrix.to links which recommend using chat.mozilla.org, if you don’t already have a preferred Matrix client configured. We’ve yet to implement this feature in Element, but we’ll be researching and experimenting with different implementations soon and will recommend best practises when we have them.

We plan to evolve matrix.to over time, including eventually evolving it to better support the Matrix URI scheme. As before, you can find the source code on GitHub and please go ahead and submit pull requests to get your Matrix client listed.

Let us know what you think over in the matrix.to room, linked to via matrix.to!

It's been a year since Dendrite development picked up again and it's certainly been a busy one at that! We started off 2020 by sprinting to complete the FOSDEM P2P demo and, since then, we have continued to develop Dendrite into a more featureful and stable homeserver.

In October, we moved Dendrite into beta, and have since released a number of releases. We've also seen quite a lot of interest from the community, so I'm here today to write about some of the fun things that have been going on in Dendrite-land.

Announcing dendrite.matrix.org

I'm happy to announce that we've finally deployed our own public Dendrite instance at dendrite.matrix.org! It's running the latest Dendrite code and is open for registration, so if you have been looking for an opportunity to play with Dendrite without hosting your own deployment, here's your chance!

There are still bugs and missing features, but overall the server is quite usable, so please feel free to register and try it and let us know how you get on.

This is the first deployment that we've built for any kind of scale, so we are cautious of the fact that there may be performance bottlenecks still. That said, over the last few weeks, a number of performance-improving changes have been merged, including:

• Around 20x performance improvement on the State Resolution v2 algorithm, which is used in room versions 2 and above
• Significantly reducing the amount of time spent recalculating event reference hashes and event IDs in the roomserver and sync API, reducing CPU usage
• Optimised memory usage and reduced database hits in the federation sender, which helps particularly in large rooms with lots of resident servers

We're optimistic that running this deployment will help us to identify scaling pain points and to make Dendrite leaner in the long run. Feel free to sign up and give it a spin! :-)

Features

Since the beginning of the year, we've added a number of new features, including but not limited to:

• Room versions support for all currently specced versions (v1-v6), including support for State Resolution v2
• SQLite storage support in addition to PostgreSQL, largely useful for the P2P demos
• Support for database schema upgrades, making updating Dendrite significantly easier
• Early end-to-end encryption support, including device list syncing and send-to-device messages, although with key backup and cross-signing still to come
• A number of federation features, including invites, retries and backing off unreachable homeservers
• User-interactive authentication (UIA), which is used in password changes and deleting devices from your device list
• Support for local pagination, backfilling over federation and fetching missing events
• Redaction of events both locally and over federation
• Entirely new microservices for managing server signing keys, E2E keys, user and device management
• Lots of great contributions from the community - including all of Read Receipts (thanks to S7evinK) and Read Markers (thanks to Lesterpig)!

...and of course entered Beta in October!

Spec compliance

Of course, Dendrite also needs to be able to fulfill the promise of being a usable next-generation Matrix homeserver at the same time as being a sci-fi development platform. We have spent much of the last year working specifically on this. Today, Dendrite's Sytest compliance sits at:

• 59% compliance for Client-Server APIs, up from 33% in May 2020
• 83% compliance for Server-Server APIs, up from 26% in May 2020

As you can see, these are significant leaps in the numbers of tests passing against Dendrite.

Experimental MSCs

We have been increasingly trying to use Dendrite for the development and testing of some new Matrix feature proposals. Recently we've seen early support added for Peeking (MSC2753) and there is work in progress on Peeking over Federation (MSC2444).

Peeking enables temporarily subscribing to a room for real-time events without joining the room. This will only be possible with rooms that are world-readable, but it reduces the overhead of looking into a room significantly as there is no need to update the room state for each peeking user/device.

In addition to that, we've also been working on Threading (MSC2836) support, which is the gateway to building some pretty new and interesting Matrix experiences. Twitter-like or Reddit-like social prototypes like this have traditionally been difficult to build on top of Matrix as the m.reference relation type from MSC1849 had never really been fleshed out.

Threading adds m.relationship fields for embedding these relationships, and also specifies an additional /event_relationships API endpoint for finding other events related to a given event in either direction. This makes it possible to build threads.

P2P Matrix

Dendrite has also been our primary development platform for P2P Matrix. This year we have released multiple P2P Matrix demos, including:

• p2p.riot.im, which uses libp2p rendezvous and embeds a full Dendrite homeserver into an in-browser Service Worker
• Element iOS P2P, available on TestFlight, which embeds a full Dendrite homeserver into the Element iOS app, initially using QUIC over Yggdrasil as a transport for federation traffic, but with more recent versions using QUIC over the experimental Pinecone protocol
• dendrite-demo-libp2p, a standalone binary which formed the basis of the FOSDEM 2020 P2P demo, using libp2p and local multicast discovery as a transport for federation traffic
• dendrite-demo-yggdrasil, another standalone binary like above, but using QUIC over Yggdrasil connectivity instead of libp2p as the transport for federation traffic

Each experiment teaches us more about potential issues that need to be resolved in order to bring P2P Matrix closer to being reality, and we are continuing to use Dendrite for this work. We'll be announcing more information in the New Year about our latest efforts and the Pinecone routing scheme that we are developing.

It's also worth highlighting that all of the other experimental work taking place right now, including Threading and Peeking, also work over P2P!

What's next

We'll be taking a short break for Christmas, but will then be continuing work on Dendrite in 2021, with the main aims being to add new features, improve spec compliance further, fix bugs and eventually exit beta. We'll also be continuing further experimental work in the P2P and Threading areas, as well as supporting the development of new MSCs such as Portable Identities (MSC2787).

We'd like to say thank you for the community support and interest, and also to send out a special thanks to our community contributors who have contributed a number of fixes and features in recent months! We always welcome code contributions via GitHub if you are an interested developer.

As always, stay tuned for more Dendrite updates either by joining us in #dendrite:matrix.org or by getting your regular dose of This Week in Matrix!

— Neil Alexander and Kegan

Hi all,

It’s been just over 2 months since we revealed that Gitter was going to join Matrix - and we are incredibly proud to announce that Gitter has officially turned on true native Matrix connectivity: all public Gitter rooms are now available natively via Matrix, and all Gitter users now natively exist on Matrix. So, if you wanted to join the official Node.js language support room at https://gitter.im/nodejs/node from Matrix, just head over to #nodejs_node:gitter.im and *boom*, you’re in!

This means Gitter is now running a Matrix homeserver at gitter.im which exposes all the active public rooms - so if you go to the the room directory in Element (for instance) and select gitter.im as a homeserver, you can jump straight in:

Once you’re in, you can chat back and forth transparently between users on the Gitter side and the Matrix side, and you no longer have the ugly “Matrixbot” user faking the messages back and forth - these are ‘real’ users talking directly to one another, and every public msg in every public room is now automatically exposed into Matrix.

So, suddenly all the developer communities previously living only in Gitter (Scala, Node, Webpack, Angular, Rails and thousands of others) are now available to anyone anywhere on Matrix - alongside communities bridged from Freenode and Slack; the native Matrix communities for Mozilla, KDE, GNOME communities etc. We’re hopeful that glueing everything together via Matrix will usher in a new age of open and defragmented dev collaboration, a bit like we used to have on IRC, back in the day.

This is also great news for mobile Gitter users - as the original mobile Gitter clients have been in a holding pattern for over a year, and native Matrix support for Gitter means they are now officially deprecated in favour of Element (or indeed any other mobile Matrix client).

What features are ready?

Now, this is the first cut of native Matrix support in Gitter: much of the time since Gitter joined Element has been spent migrating stuff over from Gitlab to Element, and it’s only really been a month of work so far in hooking up Matrix. As a result: all the important features work, but there’s also stuff that’s yet to land:

Features ready today:

• Ability to join rooms from Matrix via #org_repo:gitter.im
• Bridging Edits, Replies (mapped to Threads on Gitter), Deletes, File transfer
• Bridging Markdown & Emoji

What remains:

• Ability to send/receive Direct Messages
• Ability to plumb existing Matrix rooms into Gitter natively
• Ability to view past Gitter history from Matrix. This is planned thanks to https://github.com/matrix-org/matrix-doc/pull/2716
• Synchronising the full Gitter membership list to Matrix. Currently the membership syncs incrementally as people speak
• Turning off the old Gitter bridge
• Bridging emotes (/me support) (almost landed!)
• Bridging read receipts
• Synchronising room avatars
• Bridge LaTeX

Stuff we’re not planning to support:

• Ability to join arbitrary rooms on Matrix from Gitter. This could consume huge resources on Gitter, and we’re not in a rush to mirror all of Matrix into Gitter. This will get addressed when Gitter merges with Element into a pure Matrix client.
• Bridging Reactions. Gitter doesn’t have these natively today, and rather than adding them to Gitter, we’d rather work on merging Gitter & Element together.

For more details, we strongly recommend checking out the native Matrix epic on Gitlab for the unvarnished truth straight from the coal-face!

How do you make an existing chat system talk Matrix?

In terms of the work which has gone into this - Gitter has been an excellent case study of how you can easily plug an existing large established chat system into Matrix.

At high level, the core work needed was as simple as:

• Add ‘virtual users’, so remote Matrix users can be modelled and represented in Gitter correctly: https://gitlab.com/gitterHQ/webapp/-/merge_requests/2027/diffs.
  • This can be accomplished by simply adding a virtualUser property to your chat message/post/tweet schema which holds the mxid, displayName, and avatar as an alternative to your author field. Then display the virtualUser whenever available over the author.
• Add an application service to Gitter to bridge traffic in & out of Matrix: https://gitlab.com/gitterHQ/webapp/-/merge_requests/2041/diffs
  • This "application service" comes pre-packaged for you in many cases, so for example you can simply drop in a library like matrix-appservice-bridge in a Node.js application, and all of the Matrix talking complexity is handled for you.
• Polish it!

In practice, Eric (lead Gitter dev) laid out the waypoints of the full journey:

1. First big step was to add the concept of virtual users to Gitter. We could also have created a new Gitter user for every new matrix ID that appears, but tagging them as virtual users is a bit cleaner.
2. Figuring out how to balance the Matrix traffic coming into/out of Gitter.
   1. Spreading the inbound load comes for free via our existing load-balancer setup (ELB) where we already have 8 webapp servers running the various services of gitter.im. We just run the Matrix bridge on those servers alongside each web and api process, and then the load-balancer’s matrix.gitter.im spreads out to the servers.
   2. Events from Matrix then hit the load balancer and reach one of the servers (no duplication when processing events).
   3. If something on Gitter happens, the action occurs on one server and we just propagate it over to Matrix (no duplication or locking needed).
3. We have realtime websockets and Faye subscriptions already in the app which are backed by Mongoose database hooks whenever something changes. We just tapped into the same thing to be able to bridge across new information to Matrix as we receive it on Gitter.
4. Hooking up the official Matrix bridging matrix-appservice-bridge library to use Gitter’s existing MongoDB for storage instead of nedb.
5. Figuring out how to namespace the mxids of the gitter users:
   1. It’s nice to have the mxid as human readable as possible instead of just the numerical userId in your service.
   2. But if people can change their username in your service, you can’t change your mxid on Matrix. In the future, we’ll have portable accounts in Matrix to support this (MSC2787) but sadly these are still vapourware at this point.
   3. If you naively just switch the user’s mxid when they rename their username, then you could end up leaking conversation history between mxids(!)
   4. So we went with @username-userid:gitter.im for the Matrix ID to make it a bit more human readable but also unique so any renames can happen without affecting anything.
6. For room aliases, we decided to change our community/room URI syntax to underscores for the room aliases, #community_room:gitter.im
7. Figuring out how to bridge features correctly;
   1. Emoji - mapping between :shortcode: and unicode emojis
   2. Mapping between Gitter threaded conversations <-> Matrix replies
   3. Mapping between Matrix mentions and Gitter mentions
8. Keeping users and room data in sync
   1. We haven’t gotten there yet, but the data comes through the same Mongoose hook and we can update the bridged data as they change on the Gitter end.

Meanwhile, the Matrix side of gitter.im is hosted by Element Matrix Services and is a plain old Synapse, talking through to Gitter via the Application Service API. An alternative architecture would be to have got Gitter directly federating with Matrix by embedding a “homeserver library” into it (e.g. embedding Dendrite). However, given Dendrite is still beta and assumes it is storing its data itself (rather than persisting in an existing backend such as Gitter’s mongodb), we went for the simpler option to start with.

It’s been really interesting to see how this has played out week by week in the Gitter updates in This Week in Matrix: you can literally track the progress and see how the integration came to life between Oct 9, Oct 23, Nov 6, Nov 27 and finally Dec 4.

Huge thanks go to Eric Eastwood, the lead dev of Gitter and mastermind behind the project - and also to Half-Shot and Christian who’ve been providing all the support and review from the Matrix bridging team.

What’s next?

First and foremost we’re going to be working through the “What remains” section of the list above: killing off the old bridge, sorting out plumbed rooms, hooking up DMs, importing old Gitter history into Matrix, etc. This should then give us an exceptionally low impedance link between Gitter & Matrix.

Then, as per our original announcement, the plan is:

In the medium/long term, it’s simply not going to be efficient for the combined Element/Gitter team to split our efforts maintaining two high-profile Matrix clients. Our plan is instead to merge Gitter’s features into Element (or next generations of Element) itself and then - if and only if Element has achieved parity with Gitter - we expect to upgrade the deployment on gitter.im to a Gitter-customised version of Element. The inevitable side-effect is that we’ll be adding new features to Element rather than Gitter going forwards.

Now, that means implementing some features in Matrix/Element to match...

• Instant live room peeking (less than a second to load the webapp into a live-view of a massive room with 20K users!!)
• Seamless onboarding thanks to using GitLab & GitHub for accounts
• Curated hierarchical room directory
• Magical creation of rooms on demand for every GitLab and GitHub project ever
• GitLab/GitHub activity as a first-class citizen in a room’s side-panel
• Excellent search-engine-friendly static content and archives
• KaTeX support for Maths communities
• Threads!

...and this work is in full swing:

• We have a proposal for fast peeking (via lazy-loading state over federation) at MSC2775 and the new peek APIs at MSC2753 and MSC2444 (and even implemented by Dendrite)
• Social Login for seamless onboarding via GitLab, GitHub & friends is almost landed in Element: https://github.com/matrix-org/matrix-react-sdk/pull/5426
• Spaces gives a curated hierarchical room directory: https://github.com/vector-im/roadmap/issues/2
• KaTeX support landed in Element Web a few weeks ago thanks to akissinger and thosgood
• Threads are on the horizon, thanks to MSC2836.

The only bits which aren’t already progressing yet are tighter GL/GH integration, and better search engine optimised static archives.

So, the plan is to get cracking on the rest of the feature parity, then merge Gitter & Element together - and meanwhile continue getting the rest of the world into Matrix :)

We live in exciting times: open standards-based interoperable communication is on the rise again, and we hope Gitter’s new life in Matrix is the beginning of a new age of cross-project developer collaboration, at last escaping the fragmentation we’ve suffered over the last few years.

Finally, please do give feedback via Gitter or Matrix (or mail!) on the integration and where you’d like to see it go next!

Thanks for flying Matrix and Gitter,

-- The Matrix Team

UPDATE: Nov 9th 2020

Not only are UK/US/AU/NZ/CA/IN/JP considering mandating backdoors, but it turns out that the Council of the European Union is working on it too, having created an advanced Draft Council Resolution on Encryption as of Nov 6th, which could be approved by the Council as early as Nov 25th if it passes approval. This doesn't directly translate into EU legislation, but would set the direction for subsequent EU policy.

Even though the Draft Council Resolution does not explicitly call for backdoors, the language used...

Competent authorities must be able to access data in a lawful and targeted manner

...makes it quite clear that they are seeking the ability to break encryption on demand: i.e. a backdoor.

Please help us spread the word that backdoors are fundamentally flawed - read on for the rationale, and an alternative approach to combatting online abuse.

Hi all,

Last Sunday (Oct 11th 2020), the UK Government published an international statement on end-to-end encryption and public safety, co-signed by representatives from the US, Australia, New Zealand, Canada, India and Japan. The statement is well written and well worth a read in full, but the central point is this:

We call on technology companies to [...] enable law enforcement access to content in a readable and usable format where an authorisation is lawfully issued, is necessary and proportionate, and is subject to strong safeguards and oversight.

In other words, this is an explicit request from seven of the biggest governments in the world to mandate a backdoor in end-to-end encrypted (E2EE) communication services: a backdoor to which the authorities have a secret key, letting them view communication on demand. This is big news, and is of direct relevance to Matrix as an end-to-end encrypted communication protocol whose core team is currently centred in the UK.

Now, we sympathise with the authorities’ predicament here: we utterly abhor child abuse, terrorism, fascism and similar - and we did not build Matrix to enable it. However, trying to mitigate abuse with backdoors is, unfortunately, fundamentally flawed.

• Backdoors necessarily introduce a fatal weak point into encryption for everyone, which then becomes the ultimate high value target for attackers. Anyone who can determine the secret needed to break the encryption will gain full access, and you can be absolutely sure the backdoor key will leak - whether that’s via intrusion, social engineering, brute-force attacks, or accident. And even if you unilaterally trust your current government to be responsible with the keys to the backdoor, is it wise to unilaterally trust their successors? Computer security is only ever a matter of degree, and the only safe way to keep a secret like this safe is for it not to exist in the first place.

• End-to-end encryption is nowadays a completely ubiquitous technology; an attempt to legislate against it is like trying to turn back the tide or declare a branch of mathematics illegal. Even if Matrix did compromise its encryption, users could easily use any number of other approaches to additionally secure their conversations - from PGP, to OTR, to using one-time pads, to sharing content in password-protected ZIP files. Or they could just switch to a E2EE chat system operating from a jurisdiction without backdoors.

• Governments protect their own data using end-to-end encryption, precisely because they do not want other governments being able to snoop on them. So not only is it hypocritical for governments to argue for backdoors,** it immediately puts their own governmental data at risk of being compromised**. Moreover, creating infrastructure for backdoors sets an incredibly bad precedent to the rest of the world - where less salubrious governments will inevitably use the same technology to the massive detriment of their citizens’ human rights.

• Finally, in Matrix’s specific case: Matrix is an encrypted decentralised open network powered by open source software, where anyone can run a server. Even if the Matrix core team were obligated to add a backdoor, this would be visible to the wider world - and there would be no way to make the wider network adopt it. It would just damage the credibility of the core team, push encryption development to other countries, and the wider network would move on irrespectively.

In short, we need to keep E2EE as it is so that it benefits the 99.9% of people who are good actors. If we enforce backdoors and undermine it, then the bad 0.1% percent simply will switch to non-backdoored systems while the 99.9% are left vulnerable.

We’re not alone in thinking this either: the GDPR (the world-leading regulation towards data protection and privacy) explicitly calls out robust encryption as a necessary information security measure. In fact, the risk of US governmental backdoors explicitly caused the European Court of Justice to invalidate the Privacy Shield for EU->US data. The position of the seven governments here (alongside recent communications by the EU commissioner on the ‘problem’ of encryption) is a significant step back on the protection of the fundamental right of privacy.

So, how do we solve this predicament for Matrix?

Thankfully: there is another way.

This statement from the seven governments aims to protect the general public from bad actors, but it clearly undermines the good ones. What we really need is something that empowers users and administrators to identify and protect themselves from bad actors, without undermining privacy.

What if we had a standard way to let users themselves build up and share their own views of whether other users, messages, rooms, servers etc. are obnoxious or not? What if you could visualise and choose which filters to apply to your view of Matrix?

Just like the Web, Email or the Internet as a whole, there is literally no way to unilaterally censor or block content in Matrix. But what we can do is provide first-class infrastructure to let users (and room/community moderators and server admins) make up their own mind about who to trust, and what content to allow. This would also provide a means for authorities to publish reputation data about illegal content, providing a privacy-respecting mechanism that admins/mods/users can use to keep illegal content away from their servers/clients.

The model we currently have in mind is:

• Anyone can gather reputation data about Matrix rooms / users / servers / communities / content, and publish it to as wide or narrow an audience as they like - providing their subjective score on whether something in Matrix is positive or negative in a given context.
• This reputation data is published in a privacy preserving fashion - i.e. you can look up reputation data if you know the ID being queried, but the data is stored pseudonymised (e.g. indexed by a hashed ID).
• Anyone can subscribe to reputation feeds and blend them together in order to inform how they filter their content. The feeds might be their own data, or from their friends, or from trusted sources (e.g. a fact-checking company). Their blended feed can be republished as their own.
• To prevent users getting trapped in a factional filter bubble of their own devising, we’ll provide UI to visualise and warn about the extent of their filtering - and make it easy and fun to shift their viewpoint as needed.
• Admins running servers in particular jurisdictions then have the option to enforce whatever rules they need on their servers (e.g. they might want to subscribe to reputation feeds from a trusted source such as the IWF, identifying child sexual abuse content, and use it to block it from their server).
• This isn’t just about combating abuse - but the same system can also be used to empower users to filter out spam, propaganda, unwanted NSFW content, etc on their own terms.

This forms a relative reputation system. As uncomfortable as it may be, one man’s terrorist is another man’s freedom fighter, and different jurisdictions have different laws - and it’s not up to the Matrix.org Foundation to play God and adjudicate. Each user/moderator/admin should be free to make up their own mind and decide which reputation feeds to align themselves with. That is not to say that this system would help users locate extreme content - the privacy-preserving nature of the reputation data means that it’s only useful to filter out material which would otherwise already be visible to you - not to locate new content.

In terms of how this interacts with end-to-end-encryption and mitigating abuse: the reality is that the vast majority of abuse in public networks like Matrix, the Web or Email is visible from the public unencrypted domain. Abusive communities generally want to attract/recruit/groom users - and that means providing a public front door, which would be flagged by a reputation system such as the one proposed above. Meanwhile, communities which are entirely private and entirely encrypted typically still have touch-points with the rest of the world - and even then, the chances are extremely high that they will avoid any hypothetical backdoored servers. In short, investigating such communities requires traditional infiltration and surveillance by the authorities rather than an ineffective backdoor.

Now, this approach may sound completely sci-fi and implausibly overambitious (our speciality!) - but we’ve actually started successfully building this already, having been refining the idea over the last few years. MSC2313 is a first cut at the idea of publishing and subscribing to reputation data - starting off with simple binary ban rules. It’s been implemented and in production for over a year now, and is used to maintain shared banlists used by both matrix.org and mozilla.org communities. The next step is to expand this to support a blendable continuum of reputation data (rather than just binary banlists), make it privacy preserving, and get working on the client UX for configuring and visualising them.

Finally: we are continuing to hire a dedicated Reputation Team to work full time on building this (kindly funded by Element). This is a major investment in the future of Matrix, and frankly is spending money that we don’t really have - but it’s critical to the long-term success of the project, and perhaps the health of the Internet as a whole. There’s nothing about a good relative reputation system which is particularly specific to Matrix, after all, and many other folks (decentralised and otherwise) are clearly in desperate need of one too. We are actively looking for funding to support this work, so if you’re feeling rich and philanthropic (or a government wanting to support a more enlightened approach) we would love to hear from you at [email protected]!

Here’s to a world where users have excellent tools to protect themselves online - and a world where their safety is not compromised by encryption backdoors.

-- The Matrix.org Core Team

*Comments at HN, lobste.rs, and r/linux, LWN

Hi all,

We are ridiculously excited to announce that Gitter is joining the Matrix ecosystem and will become the first major existing chat platform to switch to natively speaking Matrix!

If you’re reading this from the Gitter community and have no idea what Matrix is: we’re an open source project that provides an open protocol for secure, decentralised communication - effectively the missing real-time communication layer of the open Web. The open Matrix network has more than 20M users on it and is growing fast (adding another 1.7M or so with the arrival of Gitter!)

Gitter is easily one of the best developer community chat systems out there, used by the communities of some massive projects (Node, TypeScript, Angular, Scala etc) and is a custodian of huge archives of knowledge via their chat logs. Gitter is unique in specifically focusing on developers: their tagline is literally “Where developers come to talk” (unlike Slack, which has barely any community features - or Discord, with its ban on unofficial clients, where developers are a bit of an afterthought relative to the gamers). With Gitter natively joining Matrix, we’re super excited to see the global developer community converging on the open Matrix network - and Gitter’s community rooms should see a huge new lease of life as they’re properly made natively available to the wider network as first class citizens :)

We’ve always had a bit of a crush on Gitter ever since we ended up opposite each other in the exhibition hall at TechCrunch Disrupt Europe 2014 - particularly when they demoed us not only their sexy webapp but also their official IRC server bridge at irc.gitter.im :D Over the years we’ve been gently nudging them to consider fully embracing Matrix, but perhaps understandably they’ve been busy focusing on their own stuff. However, earlier this year, our friends at GitLab (who acquired Gitter in 2017) reached out to explore the opportunity of Gitter becoming a core part of Matrix rather than a non-core project at GitLab… and we’ve jumped on that opportunity to bring Gitter fully into Matrix.

In practice, the way this is happening is that Element (the company founded by the Matrix core team to fund Matrix development) is acquiring Gitter from GitLab, with a combined Gitter and Element dev team focusing on giving Gitter a new life in Matrix! You can read about it from the Element angle over on the Element blog.

Practically speaking, we have a pretty interesting plan here, which we’d like to be very transparent about given it’s a little unusual:

At first, Gitter will keep running as it always has - needless to say, we will be doing everything we can to delight the Gitter community and keep the service in good shape.

Then we’re going to build out native Matrix connectivity - running a dedicated Matrix homeserver on gitter.im with a new bridge direct into the heart of Gitter; letting all Gitter rooms be available to Matrix directly as (say) #angular_angular:gitter.im, and bridging all the historical conversations into Matrix via MSC2716 or similar. We will of course do this entirely as open source, just as Gitter itself is open source thanks to GitLab releasing it under the MIT license in 2017. The plan is to comprehensively document our progress as the flagship worked example case study of “how do you make an existing chat system talk Matrix.”

This will of course replace the old and creaky matrix-appservice-gitter bridge we’ve been running since 2016. Gitter users will also be able to talk to other users elsewhere in the open Matrix network - e.g. DMing them, and (possibly) joining arbitrary Matrix rooms. Effectively, Gitter will have become a Matrix client.

Now we come to the interesting bit. Gitter has some really nice features which are sorely lacking in Element today:

• Instant live room peeking (less than a second to load the webapp into a live-view of a massive room with 20K users!!)
• Seamless onboarding thanks to using GitLab & GitHub for accounts
• Curated hierarchical room directory
• Magical creation of rooms on demand for every GitLab and GitHub project ever
• GitLab/GitHub activity as a first-class citizen in a room’s side-panel
• Excellent search-engine-friendly static content and archives
• KaTeX support for Maths communities
• Threads!

...and we promise to do everything in our power to preserve and honour these features at all costs and continue to give the Gitter community the experience they’ve come to know and love.

However: in the medium/long term, it’s simply not going to be efficient for the combined Element/Gitter team to split our efforts maintaining two high-profile Matrix clients. Our plan is instead to merge Gitter’s features into Element (or next generations of Element) itself and then - if and only if Element has achieved parity with Gitter based on the above list - we expect to upgrade the deployment on gitter.im to a Gitter-customised version of Element. The inevitable side-effect is that we’ll be adding new features to Element rather than Gitter going forwards.

In practice, the main outcome in the end should be Element having benefited massively from levelling up with Gitter - and Gitter benefiting massively from all the goodies which Element and Matrix brings, including:

• E2E Encryption
• Reactions
• Constantly improving native iOS & Android clients (which should be a welcome alternative to Gitter’s natives ones, which are already being deprecated)
• VoIP and conferencing
• All the alternative clients, bots, bridges and servers in Matrix
• The full open standard Matrix API
• Widgets (embedding webapps into rooms!)
• ...and of course participation in the wider decentralised Matrix network.

So, there you have it. It’s a new era for Gitter - and we look forward to reinvigorating Gitter’s communities over the coming months. We hope Gitter users will be blown away by the features arriving from Matrix… and we hope that Element users will be ecstatic with the performance and polish work that Gitter-parity will drive us towards. Imagine having guest access in Element that can launch and load a massive room in less than a second!

Finally, we would like to explicitly reassure the Gitter community again that we love and understand Gitter (it was one of the very first ever bridges we wrote for Matrix, for instance) - and we will be doing everything we can to not screw up our responsibility in looking after it. Please, please let us know if you have any concerns or if we ever fall short on this.

Any questions, come talk to us on #gitter:matrix.org - which is bridged with https://gitter.im/matrix-org/gitter. Exciting times ahead!

- Matthew, Amandine, and the whole Matrix, Element and Gitter teams.

Matthew and Amandine model 2014-vintage Matrix & Gitter swag in celebration :D

Bonus update - The Changelog Interview!

Sid Sijbrandij (CEO at GitLab) and Matthew had a chance to sit down with The Changelog to talk about Gitter's Big Adventure - so tune in to hear the story first hand! Warning: contains non-ironic use of the word "synergy" :D

The Changelog 414: Gitter's Big Adventure - Listen on Changelog.com

Hi all,

Regular readers of TWIM may be familiar with the Decentralized Systems and Network Services Research Group at Karlsruhe Institute of Technology, who have been busy over the last few years analysing Matrix from an independent academic point of view. The work started in 2018 with Florian Jacob’s DSN Traveler spidering project, resulting in the Glimpse of the Matrix paper analysing Matrix’s scale and room/server distribution (at least as it was back then).

Last week, they released an entirely new paper: Matrix Decomposition: Analysis of an Access Control Approach on Transaction-based DAGs without Finality by Florian Jacob, Luca Becker, Jan Grashöfer and Hannes Hartenstein, presented at ACM SACMAT ‘20.

Now, the new paper is an absolutely fascinating deep dive analysis into State Resolution v2 - the algorithm at the heart of Matrix which defines how servers merge together their potentially conflicting copies of a given room, such that everyone ends up eventually with a consistent view… even in the face of bad actors. This means that Matrix effectively implements a decentralised access control system - ensuring that users stay banned, and only users with permission can ban, etc. You can see the slides below, and read the full paper here. The video of Florian’s talk from SACMAT should be published shortly.

To give some context from the Matrix side: designing and implementing State Resolution v2 back in 2018 was a bit of a mission. Our original v1 implementation had some bugs which meant that the result of the merge could unexpectedly favour historical state over the current state (so called ‘state resets’) - thus giving an attacker a way to maliciously revert the state of the room. In v2 we thought much more carefully about the algorithm, considering state present in one version of the room but not the other as a conflict, separating and applying access control events from regular events, and adding additional ordering of the state in the room by considering events in the context of their authorisation chain (the ‘auth DAG’). The end result is that we feel confident in v2 State Res, and we haven’t seen any problems with it in the wild since we shipped it in July 2018.

However: state resolution is not intuitive at first - for instance, when you merge two versions of a room together, you treat the state events as unordered sets… even though they are ordered in the context of the room DAG. The reason is that state res needs to work even if you don’t have a copy of the whole room DAG (otherwise you’d have to download way too much data to participate in a large room). Another example is the sequence in which orderings are then applied to the state events - and how that interacts with re-authorising those events, to stop malicious ones creeping in. In the core team, we’ve end up describing it several different ways to try to help folks understand: first Erik’s original MSC1442, then uhoreg’s literary Haskell implementation, then the terse reference version in the Spec itself, and most recently Neil Alexander’s State Resolution v2 for the Hopelessly Unmathematical.

As a result we are very excited and happy that Florian and the DSN team have now published the first ever independent in-depth analysis of the algorithm, particularly in the context of decentralised access control (i.e. enforcing bans, power levels, etc). We’re pleasantly surprised that apparently “To the best of our knowledge, Matrix is the only system that implements access control based on an eventually consistent partial order without finality and without a consensus algorithm”.

Even better, the DSN team found some remaining thinkos in Synapse’s implementation and the Matrix specification, which could have caused resolution results to diverge from other implementations, specifically:

1. we weren’t enforcing integers in JSON to be within range [-2⁵³+1, 2⁵³-1], fixed in https://github.com/matrix-org/synapse/pull/7381 and MSC2540
2. we forgot to include the notification field when authing power level events, fixed in https://github.com/matrix-org/synapse/issues/7501 and MSC2209 (thanks to Luca from DSN for the MSC!)
3. we forgot to spec the limit that one should apply to the number of parents of an event in the DAG (fixed in https://github.com/matrix-org/matrix-doc/pull/2538)
4. we missed that moderators could set server ACLs which could let them undermine room admins (fixed in https://github.com/matrix-org/synapse/pull/6834).

All of these have now been fixed in Synapse and the latest versions of the spec (room v6), and we’d like to sincerely thank Florian and Luca for rapidly and responsibly disclosing the issues to us. In other words: this research is directly improving Matrix, and it’s even more exciting that the stated future work for the DSN team is to work on a formal verification for the security of Matrix’s authorisation rules and state resolution. This stuff is tough, as anyone who’s played with TLA⁺ will know, and we are incredibly glad that the research community is helping out to formalise and hopefully prove that State Res v2 is as good as we think it is.

We should stress that DSN’s work is completely independent of The Matrix.org Foundation or anyone else building on the protocol; we’re just writing about it here because we think it’s incredibly cool and deserves the attention of the whole Matrix ecosystem.

Thanks again to Florian and the team - we look forward to seeing what comes next!