Hi all,
Over the years it’s become a tradition to write an end-of-year wrap-up on Christmas Eve, reviewing all the things the core Matrix team has been up over the year, and looking forwards to the next (e.g. here’s last year’s edition). These days there’s so much going on in Matrix it’s impossible to cover it all (and besides, we now have This Week In Matrix and better blogging in general to cover events as they happen). So here’s a quick overview of the highlights:
Looking back at our plans for 2020 in last year’s wrap-up, amazingly it seems we pretty much achieved what we set out to do. Going through the bulletpoints in order:
Perhaps more interesting is the stuff we didn’t predict (or at least didn’t want to pre-announce ;) for 2020:
Then, there’s whole new waves of exciting stuff going on. The most obvious has to be the amount of Government uptake we’ve seen with Matrix this year, following on from France embracing Matrix across the public sector last year. Firstly the German armed forces announced their transition to Matrix, and then the German states of Schleswig-Holstein and Hamburg announced a mammoth 500K user Matrix deployment for education and public administration. Meanwhile, North Rhine Westphalia (the biggest state in Germany) launched their own Matrix-powered messager for education; loads of different universities have rolled out Matrix for collaboration - and we hear Famedly is making good progress with Matrix-powered healthcare messaging solutions. Finally, outside of Germany, we’re seeing the first official deployments in the UK government and US federal government - we’ll share details where possible (but sometimes big deployments of encrypted communication systems want to remain discreet). It’s incredibly exciting to see Matrix spreading across the public sector and education, and we’re hoping this will follow a similar pattern to how the Internet, email or indeed the Web first developed: a mix of high profile public sector deployments, complemented by a passionate grass-roots technical community, eventually spreading to span the rest of society :).
Another exciting thing which emerged this year is the amazing academic work that Karlsruhe Institute of Technology’s Decentralized Systems and Network Services Research Group has been conducting on Matrix. This really came on the radar back in June when their Matrix Decomposition: Analysis of an Access Control Approach on Transaction-based DAGs without Finality paper was published - a truly fascinating analysis of how state resolution works in Matrix, and how we manage to preserve access control within rooms without using blockchain-style ‘sealed blocks’ (and has helped fix a few nasty bugs!). I’m not sure any of us realised that Matrix’s state resolution counts as a new field of research, but it’s been great to follow along with their independent work. Most recently, and even more excitingly, they’re circulating a preview of their Analysis of the Matrix Event Graph Replicated Data Type paper - a deep analysis of the properties of Matrix DAGs themselves. We highly recommend reading the papers (what better way to spend the holiday break!). To give a taste, the final paragraph of the paper concludes:
2020 has also seen the arrival and maturation of a whole new generation of Matrix clients - Hydrogen is really impressive as an experimental next-generation Web (and Mobile Web) client; an account with 3000 rooms that uses 1.4GB of RAM on Element Web uses 14MB of RAM on Hydrogen and launches instantly, complete with excellent E2EE implementation. It even works on MSIE! The whole app, including dependencies, is about 70KB of code (200KB including Olm). Meanwhile, matrix-rust-sdk is coming along well, providing a general purpose native library for writing excellent native Matrix clients. Fractal merged initial matrix-rust-sdk a few weeks ago, and we’ll be experimenting with switching to it in Element iOS and Element Android (for its e2ee) in the coming year. It’s not inconceivable to think of a world where matrix-rust-sdk ends up being the no-brainer official SDK for native apps, and Hydrogen’s SDK becomes the no-brainer official SDK for JS apps.
Meanwhile, in the community, there’s been so much activity it’s untrue. But on the subject of maturing apps, it’s been incredibly exciting to see NeoChat emerge as an official KDE Matrix client (built on libQuotient and Kirigami, forked from Spectral), FluffyChat going from strength to strength; Nheko continuing to mature impressively; Mirage appearing out of nowhere as a fully featured desktop client; Fractal merging matrix-rust-sdk etc. On the serverside, Conduit was the big community story of the year - with an incredibly fast Rust + Sled server appearing out of the blue, with viable federation coming up on the horizon. The best bet for an overview of all things community is to checkout the TWIM backlogs however - there’s simply way too much stuff to mention it all here.
Obviously, no 2020 wrap-up post would be complete without acknowledging the COVID-19 pandemic - which increased focus on Matrix and other remote collaboration technology more than anyone could have predicted (especially given all the privacy missteps from Zoom, Teams and others). One of the highlights of the year was seeing the HOPE (Hackers On Planet Earth) conference shift their entire proceedings over to Matrix - turning the conference into a 10 day television station of premium hacking content, with Matrix successfully providing the social glue to preserve a sense of community despite going virtual. Similarly, we’re incredibly excited that FOSDEM 2021 is highly likely to run primarily via Matrix (with bridges to IRC and XMPP, of course) - our work is going to be cut out for us in January to ensure the amazing atmosphere of FOSDEM is preserved online for the >8,500 participants and ~800 talks. And if any other event organisers are reading this - please do reach out if you’re interested in going online via Matrix: we want Matrix to be the best possible ecosystem for online communities, including virtual events, and we’ll be happy to help :)
Talking of FOSDEM, a really fun bit of work which landed in Element this year was to (finally!) polish Widgets: the ability to embed arbitrary webapps into Matrix chatrooms. This includes being able to embed widgets in the RightPanel on Element Web, the LeftPanel too, add as many as you like to a room, resize them(!), and generally build much more sophisticated dashboards of additional content. Modal and fullscreen widgets are coming too, as are ways to simplify and unify access control. It turns out that these have arrived in the nick of time for events like FOSDEM, where we’re expecting to very heavily use widgets to embed video streams, video conferences, schedules, and generally automate the workflow of the conference via adding in web UIs as widgets wherever necessary. The work for this has been driven by the various German education deployments, where the same tricks are invaluable for automating online learning experiences. We originally wrote Widgets back in 2017 as a proof-of-concept to try to illustrate how chatrooms could be used to host proper custom UIs, and it's fantastic to see that dream finally come of age.
Finally, it’s been really exciting to see major progress in recent months on what’s essentially a whole new evolution of Matrix. Two years ago, a quiet patch during the Christmas holidays gave birth to a whole bunch of wild science fiction Matrix Spec Changes: MSC1772: Spaces (groups as rooms), MSC1769: Profiles as rooms, MSC1767: Extensible events, MSC1776: Peeking over /sync, MSC1777: Peeking over federation, etc. This was in part trying to ensure that we had something to look forward to when we emerged from the tunnel of launching Matrix 1.0, and in part trying to draw a coherent high-level sketch of what the next big wave of Matrix features could look like. Inevitably the MSCs got stuck in limbo for ages while we exited beta, launched Matrix 1.0, turned on E2EE by default etc - but in the latter half of this year they’ve hit the top of the todo list and it’s been incredibly exciting to see entirely new features landing once again. Implementation for Spaces is in in full swing and looking great; Profiles-as-rooms are effectively being trialled in Cerulean; Peeking over /sync has landed in Dendrite and peeking over federation is in PR (and unlocks all sorts of desirable features for using rooms more generically than we have today, including Spaces). Only Extensible events remains in limbo for now (we have enough to handle getting the others landed!)
Of these, Spaces has turned out to be exciting in wholly unexpected ways. While prototyping the UX for how best to navigate hierarchies of spaces, we had a genuine epiphany: the ability for anyone to define and share arbitrary hierarchies of rooms makes Matrix effectively a global decentralised hierarchical file system (where the ‘files’ are streams of realtime data, but can obviously store static content too). The decentralised access controls that KIT DSN wrote about could literally be file-system style access controls; enforcing access on a global decentralised hierarchy. We obviously have shared hierarchical filesystems today thanks to Dropbox and Google Drive, but these of course are centralised and effectively only store files - whereas Spaces could potentially scale to the whole web. In fact, you could even think of Spaces as flipping Matrix entirely on its head: the most defining building block going forwards could be the Spaces themselves rather than the rooms and events - just as directories are intrinsic to how you navigate a conventional filesystem. How has Matrix got this far without the concept of folders/directories?!
Right now these thoughts are just overexcited science fiction, but the potential really is mindblowing. It could give us a global read/write web for organising any arbitrary realtime data - with the social controls via ACLs to delegate and crowdsource curation of hierarchies however folks choose. The Matrix.org Foundation could seed a ‘root’ hierarchy, go curate all the rooms we know about into some Linnean-style taxonomy, delegate curation of the various subspaces to moderators from the community, and hey presto we’ve reinvented USENET… but with modern semantics, and without the rigid governance models. Hell, we could just mount (i.e. bridge) USENET straight into it. And any other hierarchical namespace of conversations you can think of - Google Groups, Stackoverflow, Discourse, IMAP trees…
Of course, the initial Spaces implementation is going to be focused of on letting communities publish their existing rooms, and users organise their own rooms, rather than managing an infinite ever-expanding global space hierarchy - but given we’ve been designing Spaces to support government (and inter-government) scales of Spaces, it’s not inconceivable to think we could use it to navigate gigantic public shared Spaces in the longer term.
Anyway, enough Space scifi - what’s coming up in 2021?
Our current hit list is:
I’m sure I’m missing lots here, but these are the ones which pop immediately to mind. You can also check Element's public roadmap, which covers all the core Matrix work donated by Element (as well as everything else Element is getting up to).
As always, huge huge thanks goes to the whole Matrix community for flying Matrix and keeping the dream alive and growing faster than ever. It’s been a rough year, and we hope that you’ve survived it intact (and you have our sincere sympathies if you haven’t). Let’s hope that 2021 will be a massive improvement, and that the whole Matrix ecosystem will continue to prosper in the new year.
-- Matthew, Amandine, and the whole Matrix team.