General

Hi all,

If you’re reading this - congratulations; you made it through another year :) Every winter we sit down and review Matrix’s progress over the last twelve months, and look forward to the next - for it’s all too easy to get lost in the day-to-day development and fail to realise how much the overall project is evolving, especially when it’s one as large and ambitious as Matrix!

Looking back at 2021, it’s unbelievable how much stuff has been going on in the core team (as you can tell by the length of this post - sorry!). There’s been a really interesting mix of activity too - between massive improvements to the core functionality and baseline features that Matrix provides, and also major breakthroughs on next generation work. But first, let’s check out what’s been happening in the wider ecosystem…

Continue reading…

Hi folks,

Big news today: Element, the startup founded by the team who created Matrix, just raised $30M of Series B funding in order to further accelerate Matrix development and improve Element, the flagship Matrix app. The round is led by our friends at Protocol Labs and Metaplanet, the fund established by Jaan Tallinn (co-founder of Skype and Kazaa). Both Protocol Labs and Metaplanet are spectacularly on board our decentralised communication quest, and you couldn't really ask for a better source of funding to help take Matrix to the next level. Thank you for believing in Matrix and leading Element's latest funding!

You can read all about it from the Element perspective over at the Element Blog, but suffice it to say that this is enormous news for the Matrix ecosystem as a whole. In addition to transforming the Element app, on the Matrix side this means that there is now concrete funding secured to:

• finish building out P2P Matrix and get it live (including finishing Dendrite, given our P2P work builds on Dendrite!)

• implement native decentralised E2EE voip/video conferencing for Matrix

• fully build out our relative decentralised reputation system in order to combat abuse in Matrix.

Obviously this is in addition to all the normal business-as-usual work going on in terms of:

• getting Spaces out of beta
• adding Threading to Element (yes, it's finally happening!)
• speeding up room joins over federation
• creating 'sync v3' to lazy-load all content and make the API super-snappy
• lots of little long-overdue fun bits and pieces (yes, custom emoji, we're looking at you).

If you're wondering whether Protocol Labs' investment means that we'll be seeing more overlap between IPFS and Matrix, then yes - where it makes tech sense to do so, we're hoping to work more closely together; for instance collaborating with the libp2p team on our P2P work (we still need to experiment properly with gossipsub!), or perhaps giving MSC2706 some attention. However, there are no plans to use cryptocurrency incentives in Matrix or Element any time soon.

So, exciting times ahead! We'd like to inordinately thank everyone who has supported Matrix over the years - especially our Patreon supporters, whose donations pay for all the matrix.org infrastructure while inspiring others to open their cheque books; the existing investors at Element (especially Notion and Automattic, who have come in again on this round); all the large scale Matrix deployments out there which are effectively turning Matrix into an industry (hello gematik!) - and everyone who has ever run a Matrix server, contributed code, used the spec to make their own Matrix-powered creation, or simply chatted on Matrix.

Needless to say, Matrix wouldn't exist without you: the protocol and network would have fizzled out long ago were it not for all the people supporting it (the matrix.org server can now see over 35.5M addressable users on the network!) - and meanwhile the ever-increasing energy of the community and the core team combines to keep the protocol advancing forwards faster than ever.

We will do everything we possibly can to succeed in creating the long-awaited secure communication layer of the open Web, and we look forward to large amounts of Element's new funding being directed directly into core Matrix development :)

thanks for flying Matrix,

Matthew, Amandine & the whole Matrix core team.

Hi folks,

We’re incredibly excited to officially announce that the national agency for the digitalisation of the healthcare system in Germany (gematik) has selected Matrix as the open standard on which to base all its interoperable instant messaging standard - the TI-Messenger.

gematik has released a concept paper that explains the initiative in full.

🔗TL;DR

With the TI-Messenger, gematik is creating a nationwide decentralised private communication network - based on Matrix - to support potentially more than 150,000 healthcare organisations within Germany’s national healthcare system. It will provide end-to-end encrypted VoIP/Video and messaging for the whole healthcare system, as well as the ability to share healthcare based data, images and files.

Initially every healthcare provider (HCP) with an HBA (HPC ID card) will be able to choose their own TI-Messenger provider. The homesever for HCP accounts will be hosted by the provider’s datacentre. The homeserver for institutions can be hosted by TI-Messenger providers, or on-premise.

Each organisation and individual will therefore retain complete ownership and control of their communication data - while being able to share it securely within the healthcare system with end-to-end encryption by default. All servers in the Matrix-based private federation will be hosted within Germany.

Needless to say, security is key when underpinning the entire nation’s healthcare infrastructure and safeguarding sensitive patient data. As such, the entire implementation will be accredited by BSI (Federal Office for Information Security) and BfDI (Federal Commissioner for Data Protection and Freedom of Information).

🔗The full context...

Germany’s digital care modernisation law (“Digitale Versorgung und Pflege Modernisierungs Gesetz” or DVPMG), which came into force in June 2021, spells out the need for an instant messaging solution.

The urgency has increased by a significant rise in the use of instant messaging and video conferencing within the healthcare system - for instance, the amount of medical practices using messenger services doubled in 2020 compared to 2018 (much of this using insecure messaging solutions).

gematik, majority-owned by Germany’s Federal Ministry of Health, is responsible for the standardised digital transformation of Germany’s healthcare sector. It focuses on improving efficiency and introducing new ways of working by setting, testing and certifying healthcare technology including electronic health cards, electronic patient records and e-prescriptions.

TI-Messenger is gematik’s technical specification for an interoperable secure instant messaging standard. The healthcare industry will be able to build a wide range of apps based on TI-Messenger specifications knowing that, being built on Matrix, all those apps will interoperate.

More than 150,000 organisations - ranging from local doctors to clinics, hospitals, and insurance companies - can potentially standardise on instant messaging thanks to gematik’s TI-Messenger initiative.

🔗The road to interoperability

By 1 October 2021, TI-Messenger will initially specify how communication should work in practice between healthcare professionals (HCPs). Physicians will be able to find and communicate with each other via TI-Messenger approved apps - specifications include secure authentication mechanisms with electronic health professional cards (eHBAs), electronic institution cards (SMC-B) and a central FHIR directory. The first compliant apps for HCPs are expected to be licensed by Q2 2022.

Eric Grey (product manager for TI-Messenger at gematik), reckons there will initially be around 10-15 TI-Messenger compliant Matrix-based apps for HCP communications available from different vendors.

Healthcare professionals will be able to choose a TI-Messenger provider, who will be hosting their personal accounts and provide the messenger-client.

Healthcare organisations will choose a TI-Messenger provider to build the dedicated homeserver infrastructure (on prem or in a data center), provide the client and ongoing support.

🔗What does this mean for the Matrix community?

Matrix is already integral to huge parts of the public sector; from the French government’s Tchap platform, to Bundeswehr’s use of BwMessenger and adoption by universities and schools across Europe.

Germany’s healthcare system standardising on Matrix takes this to entirely the next level - and we can’t wait to see the rest of Europe (and the world!) converge on Matrix for healthcare!

We'll have more info about TI-Messenger on this week's Matrix Live, out on Friday - stay tuned!

Last week the UK government published a draft of the proposed Online Safety Bill, after having initially introduced formal proposals for said bill in early 2020. With this post we aim to shed some light on its potential impacts and explain why we think that this bill - despite having great intentions - may actually be setting a dangerous precedent when it comes to our rights to privacy, freedom of expression and self determination.

The proposed bill aims to provide a legal framework to address illegal and harmful content online. This focus on “not illegal, but harmful” content is at the centre of our concerns - it puts responsibility on organisations themselves to arbitrarily decide what might be harmful, without any legal backing. The bill itself does not actually provide a definition of harmful, instead relying on service providers to assess and decide on this. This requirement to identify what is “likely to be harmful” applies to all users, children and adults. Our question here is - would you trust a service provider to decide what might be harmful to you and your children, with zero input from you as a user?

Additionally, the bill incentivises the use of privacy-invasive age verification processes which come with their own set of problems. This complete disregard of people’s right to privacy is a reflection of the privileged perspectives of those in charge of the drafting of this bill, which fails to acknowledge how actually harmful it would be for certain groups of the population to have their real life identity associated with their online identity.

Our view of the world, and of the internet, is largely different from the one presented by this bill. Now, this categorically does not mean we don’t care about online safety (it is quite literally our bread and butter) - we just fundamentally disagree with the approach taken.

Whilst we sympathise with the government’s desire to show action in this space and to do something about children’s safety (everyone’s safety really), we cannot possibly agree with the methods.

Back in October of 2020 we presented our proposed approach to online safety - ironically also in response to a government proposal, albeit about encryption backdoors. In it, we briefly discussed the dangers of absolute determinations of morality from a single cultural perspective:

As uncomfortable as it may be, one man’s terrorist is another man’s freedom fighter, and different jurisdictions have different laws - and it’s not up to the Matrix.org Foundation to play God and adjudicate.

We now find ourselves reading a piece of legislation that essentially demands these determinations from tech companies. The beauty of the human experience lies with its diversity and when we force technology companies to make calls about what is right or wrong - or what is “likely to have adverse psychological or physical impacts” on children - we end up in a dangerous place of centralising and regulating relative morals. Worst of all, when the consequence of getting it wrong is criminal liability for senior managers what do we think will happen?

Regardless of how omnipresent it is in our daily lives, technology is still not a solution for human problems. Forcing organisations to be judge and jury of human morals for the sake of “free speech” will, ironically, have severe consequences on free speech, as risk profiles will change for fear of liability.

Forcing a “duty of care” responsibility on organisations which operate online will not only drown small and medium sized companies in administrative tasks and costs, it will further accentuate the existing monopolies by Big Tech. Plainly, Big Tech can afford the regulatory burden - small start-ups can’t. Future creators will have their wings clipped from the offset and we might just miss out on new ideas and projects for fear of legal repercussions. This is a threat to the technology sector, particularly those building on emerging technologies like Matrix. In some ways, it is a threat to democracy and some of the freedoms this bill claims to protect.

These are, quite frankly, steps towards an authoritarian dystopia. If Trust & Safety managers start censoring something as natural as a nipple on the off chance it might cause “adverse psychological impacts” on children, whose freedom of expression are we actually protecting here?

More specifically on the issue of content moderation: the impact assessment provided by the government alongside this bill predicts that the additional costs for companies directly related to the bill will be in the billions, over the course of 10 years. The cost for the government? £400k, in every proposed policy option. Our question is - why are these responsibilities being placed on tech companies, when evidently this is a societal problem?

We are not saying it is up to the government to single-handedly end the existence of Child Sexual Abuse and Exploitation (CSAE) or extremist content online. What we are saying is that it takes more than content filtering, risk assessments and (faulty) age verification processes for it to end. More funding for tech literacy organisations and schools, to give children (and parents) the tools to stay safe is the first thing that comes to mind. Further investment in law enforcement cyber units and the judicial system, improving tech companies’ routes for abuse reporting and allowing the actual judges to do the judging seems pretty sensible too. What is absolutely egregious is the degradation of the digital rights of the majority, due to the wrongdoings of a few.

Our goal with this post is not to be dramatic or alarmist. However, we want to add our voices to the countless digital rights campaigners, individuals and organisations that have been raising the alarm since the early days of this bill. Just like with coercive control and abuse, the degradation of our rights does not happen all at once. It is a slippery slope that starts with something as (seemingly) innocuous as mandatory content scanning for CSAE content and ends with authoritarian surveillance infrastructure. It is our duty to put a stop to this before it even begins.

Hi all,

As many know, over the years we've experimented with how to let users locate and curate sets of users and rooms in Matrix. Back in Nov 2017 we added 'groups' (aka 'communities') as a custom mechanism for this - introducing identifiers beginning with a + symbol to represent sets of rooms and users, like +matrix:matrix.org.

However, it rapidly became obvious that Communities had some major shortcomings. They ended up being an extensive and entirely new API surface (designed around letting you dynamically bridge the membership of a group through to a single source of truth like LDAP) - while in practice groups have enormous overlap with rooms: managing membership, inviting by email, access control, power levels, names, topics, avatars, etc. Meanwhile the custom groups API re-invented the wheel for things like pushing updates to the client (causing a whole suite of problems). So clients and servers alike ended up reimplementing large chunks of similar functionality for both rooms and groups.

And so almost before Communities were born, we started thinking about whether it would make more sense to model them as a special type of room, rather than being their own custom primitive. MSC1215 had the first thoughts on this in 2017, and then a formal proposal emerged at MSC1772 in Jan 2019. We started working on this in earnest at the end of 2020, and christened the new way of handling groups of rooms and users as... Spaces!

Spaces work as follows:

• You can designate specific rooms as 'spaces', which contain other rooms.
• You can have a nested hierarchy of spaces.
• You can rapidly navigate around that hierarchy using the new 'space summary' (aka space-nav) API - MSC2946.
• Spaces can be shared with other people publicly, or invite-only, or private for your own curation purposes.
• Rooms can appear in multiple places in the hierarchy.
• You can have 'secret' spaces where you group your own personal rooms and spaces into an existing hierarchy.

Today, we're ridiculously excited to be launching Space support as a beta in matrix-react-sdk and matrix-android-sdk2 (and thus Element Web/Desktop and Element Android) and Synapse 1.34.0 - so head over to your nearest Element, make sure it's connected to the latest Synapse (and that Synapse has Spaces enabled in its config) and find some Space to explore! #community:matrix.org might be a good start :)

The beta today gives us the bare essentials: and we haven't yet finished space-based access controls such as setting powerlevels in rooms based on space membership (MSC2962) or limiting who can join a room based on their space membership (MSC3083) - but these will be coming asap. We also need to figure out how to implement Flair on top of Spaces rather than Communities.

This is also a bit of a turning point in Matrix's architecture: we are now using rooms more and more as a generic way of modelling new features in Matrix. For instance, rooms could be used as a structured way of storing files (MSC3089); Reputation data (MSC2313) is stored in rooms; Threads can be stored in rooms (MSC2836); Extensible Profiles are proposed as rooms too (MSC1769). As such, this pushes us towards ensuring rooms are as lightweight as possible in Matrix - and that things like sync and changing profile scale independently of the number of rooms you're in. Spaces effectively gives us a way of creating a global decentralised filesystem hierarchy on top of Matrix - grouping the existing rooms of all flavours into an epic multiplayer tree of realtime data. It's like USENET had a baby with the Web!

For lots more info from the Element perspective, head over to the Element blog. Finally, the point of the beta is to gather feedback and fix bugs - so please go wild in Element reporting your first impressions and help us make Spaces as awesome as they deserve to be!

Thanks for flying Matrix into Space;

Matthew & the whole Spaces (and Matrix) team.

Hi all,

Over the years it’s become a tradition to write an end-of-year wrap-up on Christmas Eve, reviewing all the things the core Matrix team has been up over the year, and looking forwards to the next (e.g. here’s last year’s edition). These days there’s so much going on in Matrix it’s impossible to cover it all (and besides, we now have This Week In Matrix and better blogging in general to cover events as they happen). So here’s a quick overview of the highlights:

Continue reading…

Hi all,

We have a bit of an unexpected early Christmas present for you today…

Alongside all the normal business-as-usual Matrix stuff, we’ve found some time to do a mad science experiment over the last few weeks - to test the question: “Is it possible to build a serious Twitter-style decentralised microblogging app using Matrix?”

It turns out the answer is a firm “yes” - and as a result we’d like to present a very early sneak preview of Cerulean: a highly experimental new microblogging app for Matrix, complete with first-class support for arbitrarily nested threading, with both Twitter-style (“vertical”) and HN/Reddit-style (“horizontal”) layout… and mobile web support!

Cerulean is unusual in many ways:

• It’s (currently) a very minimal javascript app - only 2,500 lines of code.
• It has zero dependencies (other than React).
  • This is to show just how simple a fairly sophisticated Matrix client can be...
  • ...and so the code can be easily understood by folks unfamiliar with Matrix...
  • ...and so we can iterate fast while figuring out threading...
  • ...and because none of the SDKs support threading yet :D
• It relies on MSC2836: Threading - our highly experimental Matrix Spec Change to extend relationships (as used by reaction & edit aggregations) to support free-form arbitrary depth threading.
• As such, it only works on Dendrite, as that’s where we’ve been experimenting with implementing MSC2836. (We’re now running an official public Dendrite server instance at dendrite.matrix.org though, which makes it easy to test - and our test Cerulean instance https://cerulean.matrix.org points at it by default).

This is **very much a proof of concept. **We’re releasing it today as a sneak preview so that intrepid Matrix experimenters can play with it, and to open up the project for contributions! (PRs welcome - it should be dead easy to hack on!). Also, we give no guarantees about data durability: both Cerulean and dendrite.matrix.org are highly experimental; do not trust them yet with important data; we reserve the right to delete it all while we iterate on the design.

🔗What can it do?

So for the first cut, we’ve implemented the minimal features to make this something you can just about use and play with for real :)

• Home view (showing recent posts from folks you follow)
• Timeline view (showing the recent posts or replies from a given user)
• Thread view (showing a post and its replies as a thread)
• Live updating (It’s Matrix, after all! We’ve disabled it for guests though.)
• Posting plain text and images
• Fully decentralised thanks to Matrix (assuming you’re on Dendrite)
• Twitter-style “Vertical” threading (replies form a column; you indent when someone forks the conversation)
• HN/Reddit/Email-style “Horizontal” threading (each reply is indented; forks have the same indentation)
• Basic Registration & Login
• Guest support (slightly faked with non-guest users, as Dendrite’s guest support isn’t finished yet)
• Super-experimental proof-of-concept support for decentralised reputation filtering(!)

Obviously, there’s a huge amount of stuff needed for parity with a proper Twitter-style system:

• Configurable follows. Currently the act of viewing someone’s timeline automatically follows them. This is because Dendrite doesn’t peek over federation yet (but it’s close), so you have to join a room to view its contents - and the act of viewing someone’s timeline room is how you follow them in Cerulean.
• Likes (i.e. plain old Matrix reactions, although we might need to finally sort out federating them as aggregations rather than individually, if people use them like they use them on Twitter!)
• Retweets (dead easy)
• Pagination / infinite scrolling (just need to hook it up)
• Protect your posts (dead easy; you just switch your timeline room to invite-only!)
• Show (some) replies to messages in the Home view
• Show parent and sibling context as well as child context in the Thread view
• Mentions (we need to decide how to notify folks when they’re mentioned - perhaps Matrix’s push notifications should be extended to let you subscribe to keywords for public rooms you’re not actually in?)
• Notifications (although this is just because Dendrite doesn’t do notifs yet)
• Search (again, just needs to be implemented in Dendrite - although how do you search beyond the data in your current homeserver? Folks are used to global search)
• Hashtags (it’s just search, basically)
• Symlinks (see below)
• Figure out how to handle lost unthreaded messages (see below)
• Offline support? (if we were using a proper Matrix SDK, we’d hopefully get this for free, but currently Cerulean doesn’t store any state locally at all).

🔗How does it work?

Every message you send using Cerulean goes into two Matrix rooms, dubbed the "timeline" room and the "thread" room. The "timeline" room (with an alias of #@matthew:dendrite.matrix.org or whatever your matrix id is) is a room with all of your posts and no one else's. The "thread" room is a normal Matrix room which represents the message thread itself. Creating a new "Post" will create a new "thread" room. Replying to a post will join the existing "thread" room and send a message into that room. MSC2836 is used to handle threading of messages in the "thread” room - the replies refer to their parent via an m.relationship field in the event.

These semantics play nicely with existing Matrix clients, who will see one room per thread and a flattened chronological view of the thread itself (unless the client natively supports MSC2836, but none do yet apart from Cerulean). However, as Cerulean only navigates threaded messages with an m.reference relationship (eg it only ever uses the new /event_relationships API rather than /messages to pull in history), normal messages sent by Matrix into a thread or timeline room will not yet show up in Cerulean.

In this initial version, Cerulean literally posts the message twice into both rooms - but we’re also experimenting with the idea of adding “symlinks” to Matrix, letting the canonical version of the event be in the timeline room, and then the instance of the event in the thread room be a ‘symlink’ to the one in the timeline. This means that the threading metadata could be structured in the thread room, and let the user do things like turn their timeline private (or vice versa) without impacting the threading metadata. We could also add an API to both post to timeline and symlink into a thread in one fell swoop, rather than manually sending two events. It’d look something like this:

We also experimented with cross-room threading (letting Bob’s timeline messages directly respond to Alice’s timeline messages and vice versa), but it posed some nasty problems - for instance, to find out what cross-room replies a message has, you’d need to store forward references somehow which the replier would need permission to create. Also, if you didn’t have access to view the remote room, the thread would break. So we’ve punted cross-room threading to a later MSC for now.

Needless to say, once we’re happy with how threading works at the protocol level, we’ll be looking at getting it into the UX of Element and mainstream Matrix chat clients too!

🔗What’s with the decentralised reputation button?

Cerulean is very much a test jig for new ideas (e.g. threading, timeline rooms, peeking), and we’re taking the opportunity to also use it as an experiment for our first forays into publishing and subscribing to reputation greylists; giving users the option to filter out content by default they might not want to see… but doing so on their own terms by subscribing to whatever reputation feed they prefer, while clearly visualising the filtering being applied. In other words, this is the first concrete experimental implementation of the work proposed in the second half of Combating Abuse in Matrix without Backdoors. This is super early days, and we haven’t even published a proto-MSC for the event format being used, but if you’re particularly interested in this domain it’s easy enough to figure out - just head over to #nsfw:dendrite.matrix.org (warning: not actually NSFW, yet) and look in /devtools to see what’s going on.

So, there you have it - further evidence that Matrix is not just for Chat, and a hopefully intriguing taste of the shape of things to come! Please check out the demo at https://cerulean.matrix.org or try playing with your own from https://github.com/matrix-org/cerulean, and then head over to #cerulean:matrix.org and let us know what you think! :)

One of the goals on the core team is to reduce friction for new users joining Matrix. A challenge we regularly face is all the factors which make Matrix flexible and powerful as an open, secure decentralised protocol also increase the difficulty of getting started.

As one example— to create a healthy, vibrant ecosystem and put power back into the hands of end users, it’s critical multiple clients exist, and that users ultimately have control over which one to use. However, needing to choose a client before getting going is counter-intuitive, and adds cognitive load which proprietary services simply don’t have.

Striking this balance is tricky, and one we’re aiming to improve with the latest version of Matrix.to.

🔗What's Matrix.to?

Matrix.to is a simple URL redirection service which lets users share links to rooms, users and communities. It isn’t tied to any client, and for end users it serves as a landing page for many as their first touch to Matrix. And until the matrix:// URI scheme is deployed commonplace (we finally registered it with IANA a few months ago!) it is the only client-agnostic way to link to content in Matrix.

It’s privacy preserving, with no information stored server side, with all calculation done in client-side JavaScript. It started life years ago as a temporary minimum viable hack by Matthew which then hung around untouched for years being… minimally viable. But we’ve recently given it a huge upgrade in usability and functionality at last!

🔗What's new?

We’ve re-written matrix.to from scratch, giving it a visual upgrade and refocused the experience around:

1. Helping new users find the best client for them to easily get going. By default, links will prioritise showing clients which are compatible with the platform the link is being viewed from, including mobile platforms.

2. Optionally, remembering your preferred client for future visits. This also includes deeplinking into native apps for Element Desktop & Element Mobile (and in future, other clients too, of course).

3. Fetching useful previews. One of our main observations when testing with new users is a lack of confidence trying out a new service without personalised, contextual information on the rooms or people they’re heading to. Matrix.to now displays room metadata like avatars, names and topics directly (fetched via matrix.org by default, asking for permission).

🔗What's next?

We’ve baked in the ability to specify clients and deployments within links, allowing sharing to give the option of a specific destination to guarantee a predictable experience. For instance, Mozilla might share matrix.to links which recommend using chat.mozilla.org, if you don’t already have a preferred Matrix client configured. We’ve yet to implement this feature in Element, but we’ll be researching and experimenting with different implementations soon and will recommend best practises when we have them.

We plan to evolve matrix.to over time, including eventually evolving it to better support the Matrix URI scheme. As before, you can find the source code on GitHub and please go ahead and submit pull requests to get your Matrix client listed.

Let us know what you think over in the matrix.to room, linked to via matrix.to!

It's been a year since Dendrite development picked up again and it's certainly been a busy one at that! We started off 2020 by sprinting to complete the FOSDEM P2P demo and, since then, we have continued to develop Dendrite into a more featureful and stable homeserver.

In October, we moved Dendrite into beta, and have since released a number of releases. We've also seen quite a lot of interest from the community, so I'm here today to write about some of the fun things that have been going on in Dendrite-land.

🔗Announcing dendrite.matrix.org

I'm happy to announce that we've finally deployed our own public Dendrite instance at dendrite.matrix.org! It's running the latest Dendrite code and is open for registration, so if you have been looking for an opportunity to play with Dendrite without hosting your own deployment, here's your chance!

There are still bugs and missing features, but overall the server is quite usable, so please feel free to register and try it and let us know how you get on.

This is the first deployment that we've built for any kind of scale, so we are cautious of the fact that there may be performance bottlenecks still. That said, over the last few weeks, a number of performance-improving changes have been merged, including:

• Around 20x performance improvement on the State Resolution v2 algorithm, which is used in room versions 2 and above
• Significantly reducing the amount of time spent recalculating event reference hashes and event IDs in the roomserver and sync API, reducing CPU usage
• Optimised memory usage and reduced database hits in the federation sender, which helps particularly in large rooms with lots of resident servers

We're optimistic that running this deployment will help us to identify scaling pain points and to make Dendrite leaner in the long run. Feel free to sign up and give it a spin! :-)

🔗Features

Since the beginning of the year, we've added a number of new features, including but not limited to:

• Room versions support for all currently specced versions (v1-v6), including support for State Resolution v2
• SQLite storage support in addition to PostgreSQL, largely useful for the P2P demos
• Support for database schema upgrades, making updating Dendrite significantly easier
• Early end-to-end encryption support, including device list syncing and send-to-device messages, although with key backup and cross-signing still to come
• A number of federation features, including invites, retries and backing off unreachable homeservers
• User-interactive authentication (UIA), which is used in password changes and deleting devices from your device list
• Support for local pagination, backfilling over federation and fetching missing events
• Redaction of events both locally and over federation
• Entirely new microservices for managing server signing keys, E2E keys, user and device management
• Lots of great contributions from the community - including all of Read Receipts (thanks to S7evinK) and Read Markers (thanks to Lesterpig)!

...and of course entered Beta in October!

🔗Spec compliance

Of course, Dendrite also needs to be able to fulfill the promise of being a usable next-generation Matrix homeserver at the same time as being a sci-fi development platform. We have spent much of the last year working specifically on this. Today, Dendrite's Sytest compliance sits at:

• 59% compliance for Client-Server APIs, up from 33% in May 2020
• 83% compliance for Server-Server APIs, up from 26% in May 2020

As you can see, these are significant leaps in the numbers of tests passing against Dendrite.

🔗Experimental MSCs

We have been increasingly trying to use Dendrite for the development and testing of some new Matrix feature proposals. Recently we've seen early support added for Peeking (MSC2753) and there is work in progress on Peeking over Federation (MSC2444).

Peeking enables temporarily subscribing to a room for real-time events without joining the room. This will only be possible with rooms that are world-readable, but it reduces the overhead of looking into a room significantly as there is no need to update the room state for each peeking user/device.

In addition to that, we've also been working on Threading (MSC2836) support, which is the gateway to building some pretty new and interesting Matrix experiences. Twitter-like or Reddit-like social prototypes like this have traditionally been difficult to build on top of Matrix as the m.reference relation type from MSC1849 had never really been fleshed out.

Threading adds m.relationship fields for embedding these relationships, and also specifies an additional /event_relationships API endpoint for finding other events related to a given event in either direction. This makes it possible to build threads.

🔗P2P Matrix

Dendrite has also been our primary development platform for P2P Matrix. This year we have released multiple P2P Matrix demos, including:

• p2p.riot.im, which uses libp2p rendezvous and embeds a full Dendrite homeserver into an in-browser Service Worker
• Element iOS P2P, available on TestFlight, which embeds a full Dendrite homeserver into the Element iOS app, initially using QUIC over Yggdrasil as a transport for federation traffic, but with more recent versions using QUIC over the experimental Pinecone protocol
• dendrite-demo-libp2p, a standalone binary which formed the basis of the FOSDEM 2020 P2P demo, using libp2p and local multicast discovery as a transport for federation traffic
• dendrite-demo-yggdrasil, another standalone binary like above, but using QUIC over Yggdrasil connectivity instead of libp2p as the transport for federation traffic

Each experiment teaches us more about potential issues that need to be resolved in order to bring P2P Matrix closer to being reality, and we are continuing to use Dendrite for this work. We'll be announcing more information in the New Year about our latest efforts and the Pinecone routing scheme that we are developing.

It's also worth highlighting that all of the other experimental work taking place right now, including Threading and Peeking, also work over P2P!

🔗What's next

We'll be taking a short break for Christmas, but will then be continuing work on Dendrite in 2021, with the main aims being to add new features, improve spec compliance further, fix bugs and eventually exit beta. We'll also be continuing further experimental work in the P2P and Threading areas, as well as supporting the development of new MSCs such as Portable Identities (MSC2787).

We'd like to say thank you for the community support and interest, and also to send out a special thanks to our community contributors who have contributed a number of fixes and features in recent months! We always welcome code contributions via GitHub if you are an interested developer.

As always, stay tuned for more Dendrite updates either by joining us in #dendrite:matrix.org or by getting your regular dose of This Week in Matrix!

— Neil Alexander and Kegan

Hi all,

It’s been just over 2 months since we revealed that Gitter was going to join Matrix - and we are incredibly proud to announce that Gitter has officially turned on true native Matrix connectivity: all public Gitter rooms are now available natively via Matrix, and all Gitter users now natively exist on Matrix. So, if you wanted to join the official Node.js language support room at https://gitter.im/nodejs/node from Matrix, just head over to #nodejs_node:gitter.im and *boom*, you’re in!

This means Gitter is now running a Matrix homeserver at gitter.im which exposes all the active public rooms - so if you go to the the room directory in Element (for instance) and select gitter.im as a homeserver, you can jump straight in:

Once you’re in, you can chat back and forth transparently between users on the Gitter side and the Matrix side, and you no longer have the ugly “Matrixbot” user faking the messages back and forth - these are ‘real’ users talking directly to one another, and every public msg in every public room is now automatically exposed into Matrix.

So, suddenly all the developer communities previously living only in Gitter (Scala, Node, Webpack, Angular, Rails and thousands of others) are now available to anyone anywhere on Matrix - alongside communities bridged from Freenode and Slack; the native Matrix communities for Mozilla, KDE, GNOME communities etc. We’re hopeful that glueing everything together via Matrix will usher in a new age of open and defragmented dev collaboration, a bit like we used to have on IRC, back in the day.

This is also great news for mobile Gitter users - as the original mobile Gitter clients have been in a holding pattern for over a year, and native Matrix support for Gitter means they are now officially deprecated in favour of Element (or indeed any other mobile Matrix client).

🔗What features are ready?

Now, this is the first cut of native Matrix support in Gitter: much of the time since Gitter joined Element has been spent migrating stuff over from Gitlab to Element, and it’s only really been a month of work so far in hooking up Matrix. As a result: all the important features work, but there’s also stuff that’s yet to land:

Features ready today:

• Ability to join rooms from Matrix via #org_repo:gitter.im
• Bridging Edits, Replies (mapped to Threads on Gitter), Deletes, File transfer
• Bridging Markdown & Emoji

What remains:

• Ability to send/receive Direct Messages
• Ability to plumb existing Matrix rooms into Gitter natively
• Ability to view past Gitter history from Matrix. This is planned thanks to https://github.com/matrix-org/matrix-doc/pull/2716
• Synchronising the full Gitter membership list to Matrix. Currently the membership syncs incrementally as people speak
• Turning off the old Gitter bridge
• Bridging emotes (/me support) (almost landed!)
• Bridging read receipts
• Synchronising room avatars
• Bridge LaTeX

Stuff we’re not planning to support:

• Ability to join arbitrary rooms on Matrix from Gitter. This could consume huge resources on Gitter, and we’re not in a rush to mirror all of Matrix into Gitter. This will get addressed when Gitter merges with Element into a pure Matrix client.
• Bridging Reactions. Gitter doesn’t have these natively today, and rather than adding them to Gitter, we’d rather work on merging Gitter & Element together.

For more details, we strongly recommend checking out the native Matrix epic on Gitlab for the unvarnished truth straight from the coal-face!

🔗How do you make an existing chat system talk Matrix?

In terms of the work which has gone into this - Gitter has been an excellent case study of how you can easily plug an existing large established chat system into Matrix.

At high level, the core work needed was as simple as:

• Add ‘virtual users’, so remote Matrix users can be modelled and represented in Gitter correctly: https://gitlab.com/gitterHQ/webapp/-/merge_requests/2027/diffs.
  • This can be accomplished by simply adding a virtualUser property to your chat message/post/tweet schema which holds the mxid, displayName, and avatar as an alternative to your author field. Then display the virtualUser whenever available over the author.
• Add an application service to Gitter to bridge traffic in & out of Matrix: https://gitlab.com/gitterHQ/webapp/-/merge_requests/2041/diffs
  • This "application service" comes pre-packaged for you in many cases, so for example you can simply drop in a library like matrix-appservice-bridge in a Node.js application, and all of the Matrix talking complexity is handled for you.
• Polish it!

In practice, Eric (lead Gitter dev) laid out the waypoints of the full journey:

1. First big step was to add the concept of virtual users to Gitter. We could also have created a new Gitter user for every new matrix ID that appears, but tagging them as virtual users is a bit cleaner.
2. Figuring out how to balance the Matrix traffic coming into/out of Gitter.
   1. Spreading the inbound load comes for free via our existing load-balancer setup (ELB) where we already have 8 webapp servers running the various services of gitter.im. We just run the Matrix bridge on those servers alongside each web and api process, and then the load-balancer’s matrix.gitter.im spreads out to the servers.
   2. Events from Matrix then hit the load balancer and reach one of the servers (no duplication when processing events).
   3. If something on Gitter happens, the action occurs on one server and we just propagate it over to Matrix (no duplication or locking needed).
3. We have realtime websockets and Faye subscriptions already in the app which are backed by Mongoose database hooks whenever something changes. We just tapped into the same thing to be able to bridge across new information to Matrix as we receive it on Gitter.
4. Hooking up the official Matrix bridging matrix-appservice-bridge library to use Gitter’s existing MongoDB for storage instead of nedb.
5. Figuring out how to namespace the mxids of the gitter users:
   1. It’s nice to have the mxid as human readable as possible instead of just the numerical userId in your service.
   2. But if people can change their username in your service, you can’t change your mxid on Matrix. In the future, we’ll have portable accounts in Matrix to support this (MSC2787) but sadly these are still vapourware at this point.
   3. If you naively just switch the user’s mxid when they rename their username, then you could end up leaking conversation history between mxids(!)
   4. So we went with @username-userid:gitter.im for the Matrix ID to make it a bit more human readable but also unique so any renames can happen without affecting anything.
6. For room aliases, we decided to change our community/room URI syntax to underscores for the room aliases, #community_room:gitter.im
7. Figuring out how to bridge features correctly;
   1. Emoji - mapping between :shortcode: and unicode emojis
   2. Mapping between Gitter threaded conversations <-> Matrix replies
   3. Mapping between Matrix mentions and Gitter mentions
8. Keeping users and room data in sync
   1. We haven’t gotten there yet, but the data comes through the same Mongoose hook and we can update the bridged data as they change on the Gitter end.

Meanwhile, the Matrix side of gitter.im is hosted by Element Matrix Services and is a plain old Synapse, talking through to Gitter via the Application Service API. An alternative architecture would be to have got Gitter directly federating with Matrix by embedding a “homeserver library” into it (e.g. embedding Dendrite). However, given Dendrite is still beta and assumes it is storing its data itself (rather than persisting in an existing backend such as Gitter’s mongodb), we went for the simpler option to start with.

It’s been really interesting to see how this has played out week by week in the Gitter updates in This Week in Matrix: you can literally track the progress and see how the integration came to life between Oct 9, Oct 23, Nov 6, Nov 27 and finally Dec 4.

Huge thanks go to Eric Eastwood, the lead dev of Gitter and mastermind behind the project - and also to Half-Shot and Christian who’ve been providing all the support and review from the Matrix bridging team.

🔗What’s next?

First and foremost we’re going to be working through the “What remains” section of the list above: killing off the old bridge, sorting out plumbed rooms, hooking up DMs, importing old Gitter history into Matrix, etc. This should then give us an exceptionally low impedance link between Gitter & Matrix.

Then, as per our original announcement, the plan is:

In the medium/long term, it’s simply not going to be efficient for the combined Element/Gitter team to split our efforts maintaining two high-profile Matrix clients. Our plan is instead to merge Gitter’s features into Element (or next generations of Element) itself and then - if and only if Element has achieved parity with Gitter - we expect to upgrade the deployment on gitter.im to a Gitter-customised version of Element. The inevitable side-effect is that we’ll be adding new features to Element rather than Gitter going forwards.

Now, that means implementing some features in Matrix/Element to match...

• Instant live room peeking (less than a second to load the webapp into a live-view of a massive room with 20K users!!)
• Seamless onboarding thanks to using GitLab & GitHub for accounts
• Curated hierarchical room directory
• Magical creation of rooms on demand for every GitLab and GitHub project ever
• GitLab/GitHub activity as a first-class citizen in a room’s side-panel
• Excellent search-engine-friendly static content and archives
• KaTeX support for Maths communities
• Threads!

...and this work is in full swing:

• We have a proposal for fast peeking (via lazy-loading state over federation) at MSC2775 and the new peek APIs at MSC2753 and MSC2444 (and even implemented by Dendrite)
• Social Login for seamless onboarding via GitLab, GitHub & friends is almost landed in Element: https://github.com/matrix-org/matrix-react-sdk/pull/5426
• Spaces gives a curated hierarchical room directory: https://github.com/vector-im/roadmap/issues/2
• KaTeX support landed in Element Web a few weeks ago thanks to akissinger and thosgood
• Threads are on the horizon, thanks to MSC2836.

The only bits which aren’t already progressing yet are tighter GL/GH integration, and better search engine optimised static archives.

So, the plan is to get cracking on the rest of the feature parity, then merge Gitter & Element together - and meanwhile continue getting the rest of the world into Matrix :)

We live in exciting times: open standards-based interoperable communication is on the rise again, and we hope Gitter’s new life in Matrix is the beginning of a new age of cross-project developer collaboration, at last escaping the fragmentation we’ve suffered over the last few years.

Finally, please do give feedback via Gitter or Matrix (or mail!) on the integration and where you’d like to see it go next!

Thanks for flying Matrix and Gitter,

-- The Matrix Team