TL;DR: we shipped a major update (v0.1.1) to https://p2p.riot.im - fire up a desktop Chrome or Firefox in not-private-browsing mode and give it a go!

Hi folks,

As many know by now, a few of us have been working away since mid-December on experimenting with running Matrix in a peer-to-peer architecture - one where every user has absolute total autonomy and ownership of their conversations, because the only place their conversations exist is on the devices they own.

In some ways this is the logical end goal of Matrix: our aim has always been to empower users to have full control over their communication rather than being beholden to any given service provider, and in a P2P world we completely return power over secure communication to the people.

Why P2P?

P2P Matrix is about more than just letting users store their own conversations: it can also avoid dependencies on the Internet itself by working over local networks, mesh networks, or situations where the Internet has been cut off. Even more interestingly, without homeservers, there is nowhere for metadata to accumulate about who is talking to who, and when - which is a legitimate complaint about today’s Matrix network, given the homeservers of all users in a given conversation necessarily have to store that conversation’s metadata. P2P also lets us radically simplify signup for new users if they don’t have to pick a server to get going - and we avoid the unintentional centralisation of users piling onto public servers.

P2P also forces us to solve many of the hardest remaining problems in Matrix: e.g multi-homed accounts, given multi-device P2P requires your account to exist in multiple places. This in turn unlocks high availability and geo-redundancy for accounts on today’s Matrix network (imagine having a primary and backup homeserver that magically did the right thing!), as well as account portability, and thus also vhosting and load-balancing accounts between servers, and even improved GDPR compliance (for if your user IDs are ephemeral they are no longer personally identifying information baked into your Matrix rooms). We’ll also need better safety mechanisms to avoid folks exploiting the anonymous nature of the network for abuse, accelerating the work we’re already doing for today’s Matrix network.

The way we’ve been approaching P2P is the “hamfisted but genius” approach of taking homeservers and running them on the client, alongside or within your Matrix client - meaning that there are literally no changes required for any Matrix client to talk P2P Matrix, and so P2P Matrix can instantly benefit from all the work which has gone into Riot and other apps. As a result, P2P is also a huge motivator towards developing much smaller homeservers which can run efficiently clientside (e.g. Dendrite!) - which is of course great news for Matrix as a whole. It also forces us to develop more scalable routing algorithms (as you don’t want your client to have to talk to every other device in a room every time it sends a message!) and also spurs development of low bandwidth Matrix transports (as you don’t want the additional chatter of talking to multiple peers to consume all your bandwidth). Finally, it forces us to really ruggedize federation, given nodes are constantly appearing and disappearing, giving the federation much more of a stress test than we see with today’s relatively static homeservers.

P2P in Practice

So, P2P has been acting as fuel for a lot of our longer term Matrix work over the last few months. There have been three main experiments so far: at FOSDEM we showed off running our next-gen Dendrite homeserver running clientside using HTTP over libp2p as the transport. We also highlighted Timothée Floure’s project at EPFL experimenting with Synapse talking P2P CoAP over yggdrasil as the transport via a proxy.

Most recently, however, we’ve been experimenting with compiling Dendrite down to Web Assembly and running it embedded in Riot Web as a Service Worker, using HTTP over libp2p’s websocket transport (coordinated via a websocket rendezvous server). Architecturally, it looks like this:

Today, we’re shipping a major new alpha (v0.1.1) of this P2P demo up at https://p2p.riot.im (requires desktop Chrome or Firefox in non-private-browsing mode) - which hopefully should give a really usable and concrete taste of the shape of things to come.

The main features are:

• Your conversations are now persisted in your browser storage (via IndexedDB), meaning that as long as all the browsers participating in a given conversation don’t clear their local storage, rooms on the P2P network are here to stay!
• Your room directory lists all the aliases for all the rooms published by active nodes on the network. Moreover, we now automatically publish a local room alias whenever you join a public room, so that others will be able to discover that room via you, even if the server who originally created the alias has disappeared.
• Lots and lots of federation improvements between the nodes - for instance, when a node comes online, others should now automatically detect and send scrollback to it. Invites should work, and there should no longer be any unexpectedly redacted messages.

Needless to say, all the code for this is open source under the Apache license, and if you’re feeling particularly adventurous you can embed your very own P2P Dendrite into Riot Web by using the Dockerfile at https://github.com/matrix-org/dendrite/blob/master/build/docker/DendriteJS.Dockerfile or following the instructions at https://github.com/matrix-org/dendrite/blob/master/docs/p2p.md.

Please report bugs to https://github.com/matrix-org/dendrite/issues!

Finally, please understand that the demo is very likely not what the final version of P2P Matrix will look like - this is just one step in a series of experiments as we investigate the best paths forward :)

What’s next?

For the current demo, there’s still lots of stuff remaining, including:

• More federation debugging (and hooking in tardis and writing up everything we’ve learned about implementing federation in Dendrite!)
• Making the content repository work in-browser (gotta fill up those IndexedDBs with some GIFs!)
• Hooking up E2E Encryption APIs in Dendrite (not that it buys us much in a pure P2P world)
• WebRTC transports. Turns out that service workers aren’t allowed to speak WebRTC, so we’ll have to shim through to Riot to speak true peer-to-peer WebRTC data channels rather than relaying all the traffic through the websocket rendezvous server.
• Decentralised accounts for multidevice support - reviewing MSC1228 and getting Dendrite supporting multihoming accounts!
• Finishing all of Dendrite’s other remaining APIs.

Beyond this, there are some bigger picture questions left to be answered in future experiments.

Firstly: we do not yet have a solution for “store and forward” nodes which can relay messages on behalf of a room if all the participating devices are offline. A first cut will be to run a P2P-capable homeserver server-side for this, but then metadata will start to accumulate server-side for the conversations it hosts. A more interesting approach would be to use a store and forward system which obfuscates who is talking to who, such as a mixnet, and could even provide resistance to network traffic pattern analysis. This is very much an open area of research, but one we are getting into :D

Secondly: we want to experiment more with other transports, and find out which works best for Matrix. Libp2p has some really exciting new stuff in the form of Gossipsub v1.1 - a much smarter routing algorithm for pubsub traffic in libp2p, which David Dias gave us a VIP tour of at the first Open Tech Will Save Us meetup. So we’ll need to restructure our libp2p transport as pubsub to see how it works in practice. Separately, we also want to play with hooking up Yggdrasil (the encrypted overlay network) as a transport as a totally different approach - Yggdrasil will easily let us span different underlying network transports, but comes with different tradeoffs (e.g. no browser support yet). We also want to take a look at the DAT / hypercore / hyperswarm / Cabal ecosystem to see if there’s a match :)

Thirdly and finally: we obviously want to unify the new P2P Matrix network with today’s federated one. The ideal outcome here would be to have a hybrid model, where teams who want their users to have a dedicated homeserver (for availability, IT policies, etc) can continue to have one as they do today - but newbies who have just installed Riot would float around on P2P unless they decided to consciously put down roots on a server or two. Best of all, it would let us turn off the matrix.org homeserver: the best public homeserver is one you run yourself on your own phone ;) The approach we take for linking P2P and today’s Matrix will depend very much on the transport we select for P2P in the long run, but the likelihood is that today’s homeservers will sprout P2P gateways to link the networks.

Conclusion

So, there you have it. P2P Matrix exists, and is developing at an alarming speed - and pushing Dendrite development along with it. Most excitingly, there have been no changes yet to the Matrix spec for P2P at all; we’ve just swapped https for http-over-libp2p as the transport. So all of the work we’ve been doing making Dendrite work in a P2P world has directly translated into making Dendrite work on today’s Matrix too You can now stand up a Dendrite and have it federate pretty reliably with the wider Matrix network, although we’re still rushing through implementing APIs (we’re up to 35% passing sytest coverage - although that 35% does contain most of the important tests :)

Finally, in case you’re worried about why the Matrix core team is off chasing P2P dreams rather than improving Riot’s UX, or implementing Communities, or Extensible Profiles, or working through the MSC backlog etc... in practice only two people (ignoring Matthew) have been working on P2P - Neil Alexander (author of the original FOSDEM demo, Dendrite wrangler and Yggdrasil co-maintainer) and Kegan Dougal (of the original Matrix dev team, one of the original authors of Dendrite, and now wrangling the WASM P2P work too). Huge thanks to Kegan & Neil for pushing P2P forwards - and huge thanks to everyone else on the core team and the wider community for keeping today’s Matrix advancing too!

Hope this has given a tempting glimpse of the shape of things to come. Honestly we never thought we’d get as far as P2P when we started Matrix back in 2014, but it’s really fun to be finally catching up with the future :D

-- Matthew

P.S. You can read more about this from Neil Alexander’s point of view over at his blog (including more thoughts on the potential Yggdrasil demo!)

P.P.S You can read the gory details of the P2P and WASM implementation from Kegan's point of view over at the Dendrite wiki.

P.P.P.S Comments over at HN

Hi all,

We’re very excited indeed to announce that Automattic, the creators of WordPress.com, are jumping head first into the Matrix ecosystem with a strategic investment of almost $5M into New Vector (the company which makes Riot and Modular.im, founded by the core Matrix team in 2017). More importantly, Matt Mullenweg (co-founder of WordPress and founder of Automattic) and the Automattic gang are committing to make the most of Matrix in their work going forwards!

This is huge news, not least because WordPress literally runs over 36% of the websites on today’s web - and the potential of bringing Matrix to all those users is incredible. Imagine if every WP site automatically came with its own Matrix room or community? Imagine if all content in WP automatically was published into Matrix as well as the Web? (This isn’t so far fetched an idea - turns out that Automattic already runs a XMPP bridge for wordpress.com over at im.wordpress.com!). Imagine there was an excellent Matrix client available as a WordPress plugin for embedding realtime chat into your site? Imagine if Tumblr (which is part of Automattic these days) became decentralised!?

In fact, if you’re a developer in either the Matrix or WordPress communities, now might be a good time to think about how to cross the streams.... not least because Automattic just opened up a role for a Matrix.org/WordPress Integrations Engineer! Quite aside from the investment, this shows Automattic is serious about Matrix - and we’d like to thank them for opening up jobs in these challenging times to further accelerate Matrix. Perhaps some day Matrix Engineer will be as common a career choice as Web Developer ;)

That said, it’s super early days for integration work, and there isn’t a concrete project to announce yet beyond the investment in New Vector (which is effectively an extension of the funding NV raised in October) and Automattic’s Job opening - but these are the sort of ideas we’ve been kicking around. And at the very least, we should expect to see Automattic’s communities migrating over to Matrix in the coming months.

It’s been loads of fun working with Matt and the team on this: we see a huge overlap in terms of a genuine love for the open web, open source and open standards. It’s also no coincidence that Matt (independently of Automattic) donated substantially to Matrix via Patreon back in 2017 when we needed it the most. We’re also looking forward to benefiting from Automattic’s experience in sustainably and responsibly funding and growing open source projects in general - WordPress.com is an excellent example of how one can support development of a project like WordPress without compromising its open source nature.

So, we’d like to formally welcome WordPress and the rest of the Automattic family into Matrix. It’s incredibly exciting times, and we can’t wait to see what will come of the partnership! And meanwhile, if any other massive open source organisations want to join Automattic and Mozilla in leaping into Matrix, you know where to find us… :D

Huge thanks go to Matt for believing in Matrix - watch this space for updates.

• Matthew, Amandine & the Matrix Team.

Hi all,

As of today, Matrix is end-to-end encrypted by default for private conversations.

Three years have passed since we first announced End-to-end Encryption in Matrix and started to beta test it in Riot - and after an enormous amount of polishing and refinement on its user experience, we are finally declaring it out of beta and enabling it by default for all new private conversations in Riot. As Riot is currently the most common Matrix client, this means that Matrix as a whole should now be considered end-to-end encrypted by default for DMs and invite-only rooms.

Work on E2EE in Matrix has progressed in waves since we first shipped it - including:

• adding keysharing (letting you share encryption keys between your devices to improve reliability)
• making Riot Web's encryption resilient to running concurrently in multiple tabs
• adding online key backup (so you don't lose all your history if you lose all your devices)
• making encryption resilient to restoring the app from a backup
• adding interactive key verification via emoji to make the verification process easier.

However, our goal was always to enable E2EE by default for all private rooms, which means having feature parity between unencrypted and E2EE Matrix so that we can enable encryption without any negative impact on usability. The high-level remaining items were significant:

• Cross-signing: verifying your own logins so others don’t have to.
• Adding QR codes for even better verification UX, to make cross-signing as painless as possible.
• Replacing the old prototype UI for E2EE with final polished UI/UX.
• Ability to support non-E2EE clients.
• Ability to search encrypted rooms.
• Ability to view file indexes in encrypted rooms.
• Fixing the remaining “Unable to decrypt” errors.

Over the last few months the Riot team has been almost entirely focused on implementing solutions to these items - and we're finally at the point where the switch can be flipped and as of Riot Web/Desktop 1.6, Riot iOS 0.11.1 and RiotX Android 0.19, all new private rooms will be encrypted by default; completing the transition we began at FOSDEM 2020 when we landed cross-signing E2E-by-default in the development branches of Riot.

For full details, please go check out the massive deep dive over at the Riot blog - also featuring all the other recent progress in Riot!

Heads up that encrypted traffic is slightly heavier on the server than unencrypted (due to exchanging keys, verification traffic, and keybackup traffic), and so there is a risk that the already-over-popular Matrix.org server instance may feel a little hugged to death. However, unprecedented Synapse performance breakthroughs are on the horizon in the coming weeks which will fix this - and, of course, you can (and should!) be using your own instance anyway.

Thanks everyone for helping us test encryption over the years and getting us to this point: cross-signing provides a more secure way of tracking device trust than almost any other comms system out there, and we hope that you'll agree the improved UX has been worth the wait.

Next stop: Synapse performance, and rebuilding Riot's first time user experience!

thanks,

Matthew, Amandine & the Matrix Team.

(Comments over at HN)

Hi all,

Over the last few weeks there's been huge increase in interest from folks wanting the security and autonomy of running their remote collaboration services, rather than being at the mercy of traditional proprietary centralised apps. Meanwhile, the Matrix.org homeserver has been very overloaded (although we're at last making excellent progress in radically improving Synapse's performance) - so it's particularly important right now to help folks run their own servers.

Therefore we're very happy to announce that it's easier than ever before now to self-host your own video conferencing alongside Riot & Synapse: as of Riot/Web 1.5.15 (released last week), it's now a single config option to point Riot at a specific Jitsi rather than needing to hook up to an integration manager!

Meanwhile, over the last 18 months, it's got easier and easier to run your own Matrix deployments: the Debian packages are unrecognisably better now, and with .well-known URL support it's trivial to set up federation without needing to worry about complicated DNS, TLS or load balancer configurations.

So, to try to show off just how smooth this has become, we thought we'd do a run-through video showing installing Synapse, Riot & Jitsi on a completely fresh Debian install. It's (almost) filmed in a single shot, and takes about 20 minutes from beginning to end.

Please note that this does assume you're pretty familiar with Linux system administration. If you're not, then we'd recommend using a Matrix hosting provider such as Modular.im (which directly supports development of the core team), Ungleich.ch, or StartupStack.

Finally, while the video shows how to install on Debian via Debian packages, there are many many other environments and architectures (e.g. installing under Docker) - this is just one relatively easy way to skin the cat. Perhaps there will be other 'speed-run' videos in future :)

If you want to follow along at home without listening to the video (and I can't blame you if you do ;) the high level steps are as follows:

Debian & DNS

• Take one fresh Debian 10 install.
• Point the DNS for your domain to it. You should use separate subdomains for the various services as a hygiene measure to make cross-site scripting attacks less effective. In this example, we set up DNS for:
  • dangerousdemos.net (general website, and for hosting a .well-known path to advertise the Matrix service)
  • matrix.dangerousdemos.net (Synapse)
  • riot.dangerousdemos.net (Riot/Web)
  • jitsi.dangerousdemos.net (Jitsi video conferencing)
  • In practice, we used a *.dangerousdemos.net wildcard DNS record for the three subdomains in this instance.

Nginx and LetsEncrypt

• Install nginx as a webserver: apt-get update && apt -y install nginx
• Go to /etc/nginx/sites-enabled and copy the vhost configuration block from the bottom of default to new files called dangerousdemos.net, matrix.dangerousdemos.net, and riot.dangerousdemos.net. We don't set up jitsi.dangerousdemos.net at this point as the jitsi installer handles it for us.
  • Rename the server_name field in the new files to match the hostname of each host, and point root to an appropriate location per domain (e.g. /var/www/dangerousdemos.net for the main domain, or /var/www/riot.dangerousdemos.net/riot for riot)
  • For the Synapse domain (matrix.dangerousdemos.net here), you should replace the contents of the location block with proxy_pass http://localhost:8008; - telling nginx to pass the traffic through to synapse, which listens by default for plaintext HTTP traffic on port 8008. (N.B. do not put a trailing slash on the URL here, otherwise nginx will mangle the forwarded URLs.)
• Enable TLS via LetsEncrypt on nginx, by: apt install -y python3-certbot-nginx && certbot --nginx -d dangerousdemos.net -d riot.dangerousdemos.net -d matrix.dangerousdemos.net (or whatever your domains are).
• You should be able to go to https://dangerousdemos.net at this point and see a page with valid HTTPS.

Synapse

• Then, install Synapse via Debian packages using the instructions at https://github.com/matrix-org/synapse/blob/master/INSTALL.md#debianubuntu (see below). If you're not on Debian, keep an eye out for all the other OSes we support too!
  • You should specify the server name to be the domain you want in your matrix IDs - i.e. dangerousdemos.net in this example.
  • Please report anonymous aggregate stats to us so we can gauge uptake and help justify funding for Matrix!

sudo apt install -y lsb-release wget apt-transport-https
sudo wget -O /usr/share/keyrings/matrix-org-archive-keyring.gpg https://packages.matrix.org/debian/matrix-org-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/matrix-org-archive-keyring.gpg] https://packages.matrix.org/debian/ $(lsb_release -cs) main" |
    sudo tee /etc/apt/sources.list.d/matrix-org.list
sudo apt update
sudo apt install matrix-synapse-py3

• You should now be able to go to https://matrix.dangerousdemos.net and see a valid "It works! Synapse is running" page.

• Then, you should enable registration on your synapse by switching enable_registration: true in /etc/matrix-synapse/homeserver.yaml and restarting synapse via systemctl restart matrix-synapse.

• Now you need to tell the rest of Matrix how to find your server. The easiest way to do this is to publish a file at https://dangerousdemos.net/.well-known/matrix/server which tells everyone the hostname and port where they can find the synapse for dangerousdemos.net - in this instance, it's matrix.dangerousdemos.net:443:

mkdir -p /var/www/dangerousdemos.net/.well-known/matrix
cd /var/www/dangerousdemos.net/.well-known/matrix
echo '{ "m.server": "matrix.dangerousdemos.net:443" }' > server

• Alternatively, you could advertise the server via DNS, if you don't have write access to /.well-known on your main domain. However, to prove you are allowed to host the Matrix traffic for dangerousdemos.net, you would have to configure nginx to use the dangerousdemos.net TLS certificate for the matrix.dangerousdemos.net vhost (i.e. the "wrong" one), and in general we think that /.well-known is much easier to reason about. In this case you would advertise the server with an SRV record like this:

_matrix._tcp.dangerousdemos.net. 300    IN  SRV 10 5 443 matrix.dangerousdemos.net.

Riot/Web

• Then, install Riot/Web. Grab the latest .tgz release from https://github.com/vector-im/riot-web/releases. You should check its GnuPG signature too:

mkdir /var/www/riot.dangerousdemos.net
cd /var/www/riot.dangerousdemos.net
wget https://github.com/vector-im/riot-web/releases/download/v1.5.15/riot-v1.5.15.tar.gz

# check its GnuPG signature (particularly advisable, given Riot is what stores
# your end-to-end encryption keys)
apt install -y gnupg
wget https://github.com/vector-im/riot-web/releases/download/v1.5.15/riot-v1.5.15.tar.gz.asc

# grab the signing key for the riot releases repository, ideally from a keyserver...
gpg --keyserver keyserver.ubuntu.com --search-keys [email protected]

# ...and/or you can grab or cross-check the signing key from packages.riot.im
wget https://packages.riot.im/riot-release-key.asc
gpg --import riot-release-key.asc

gpg --verify riot-v1.5.15.tar.gz.asc
# hopefully this will report "Good signature", even though it won't know to trust the riot release key.

# you could also choose to explicitly trust the key by editing it, entering 'trust' and then '5' for ultimate trust.
gpg --edit-key 74692659bda3d940

tar -xzvf riot-v1.5.15.tar.gz
ln -s riot-v1.5.15 riot
chown www-data:www-data -R riot
cd riot
cp config.sample.json config.json

• You then tweak the config.json to change the base_url of the homeserver to be https://matrix.dangerousdemos.net (i.e. where to find the Client Server API for your server), and change the server_name to be dangerousdemos.net (i.e. the name of your server).
• You should then be able to go to https://riot.dangerousdemos.net, register for an account, sign in, and talk to the rest of Matrix!

Jitsi

• Finally, we install Jitsi so you can run your own video conferencing. We take the instructions from Jitsi's quick install guide:

echo 'deb https://download.jitsi.org stable/' >> /etc/apt/sources.list.d/jitsi-stable.list
wget -qO -  https://download.jitsi.org/jitsi-key.gpg.key | sudo apt-key add -
apt-get update
apt-get -y install jitsi-meet

• We give the installer the hostname jitsi.dangerousdemos.net. Make sure this DNS is already set up, otherwise the installer will fail!

• The installer magically detects you have nginx installed and adds in an appropriate vhost!

• We select a self-signed certificate for now, and then upgrade it to LetsEncrypt after the fact with /usr/share/jitsi-meet/scripts/install-letsencrypt-cert.sh.

  • Alternatively, you could have specified manual certificates, and then used certbot alongside the rest of nginx to create a certificate for jitsi.dangerousdemos.net - both work.

• You should now be able to go to https://jitsi.dangerousdemos.net and use the Jitsi directly.

• Finally, and this is the cool new bit: you can now point Riot to use the new Jitsi by going to its config.json at /var/www/riot.dangerousdemos.net/riot/config.json and changing the preferredDomain of the jitsi block from https://jitsi.riot.im to your own self-hosted https://jitsi.dangerousdemos.net.

• You then refresh your Riot/Web, and you should be all set to use Jitsi from within your new Riot - as Riot/Web 1.5.15 and later has the ability to natively embed Jitsi straight into the app without needing to use an integration manager.

Conclusion

Matrix nowadays provides an excellent alternative to the centralised solutions. It gives:

• Full autonomy over how to host and store your own conversations
• Full freedom to talk to anyone else on the wider global Matrix network (or indeed anyone else bridged into Matrix)
• Full privacy via full end-to-end-encryption for chats, file transfer and 1:1 voice/video calls (when enabled)
• Full transparency by being 100% open source (as well as benefiting from the overall open source community)

Hopefully this gives some confidence that it's pretty easy to run your own fully functional Matrix instance these days. If not, then hopefully someone will do a similar one to show off Docker! And if that's still too scary, please take a look at a hosting services like Modular.im.

(Comments over at HN and here too)

Hi all,

Heads up that yesterday at 12:00 ET, the Mozilla IRC network was switched off after over 22 years of valiant service, and the mozilla.org Matrix instance is now in full production. You can get at it via the Riot instance at https://chat.mozilla.org, by pointing your client at https://mozilla.modular.im, or by joining rooms on the mozilla.org server over federation via its room directory.

We'd like to thank Mozilla again for putting their faith in Matrix, and are determined to do everything we can to ensure we're a more than worthy successor to IRC; we have big boots to fill :)

We've been gathering a huge amount of invaluable FTUE (first time user experience) feedback from the commentary in #synchronicity:mozilla.org - and we're in the process of implementing it over the coming weeks. In particular, we've already implemented alphabetic room ordering, custom theming support, and done a bunch more SSO work.

The immediate priorities include:

• Fixing a regression in jumps/jank when scrolling (fix PRed to develop today)
• Enabling Mozilla IAM SSO authentication on remaining "interactive user auth" flows (e.g. managing devices)
• Fixing the UX around selecting server when browsing the room directory.
• Fixing notification defaults, behaviour and settings UX
• Better educating users to connect to the mozilla.modular.im if using a random app.

Finally, it's worth noting that the matrix-ircd project is seeing some commits again, many thanks to jplatte from the Ruma project - so if you are currently despairing the demise of moznet, never fear: you may yet be able to connect to the Mozilla matrix server via IRC (authing via Mozilla IAM, of course) and pretend that none of this newfangled Matrix stuff exists :D

Please keep the feedback coming in #synchronicity:mozilla.org - we're gathering it all up into Github (under the mozilla label) as well as a high level Google Doc to help collate everything.

thanks,

- The Matrix Core Team.

(Comments over at HN)

Hi all,

Synapse 1.11 landed on Friday (sorry for running late on blogging the release notes!)

The main change is to introduce an experimental API MSC2432 for managing aliases for rooms on your local server. In Synapse 1.10 we removed support for m.room.aliases events, which were a way to try to track which aliases a room had on the room itself (but were vulnerable to abuse). In this release we've re-added the ability to query which aliases a given server has for the room, giving visibility for managing aliases, without having them spray all over the room state itself. Riot/Web 1.5.10 supports the new API, giving a way to manage aliases on your local server while we finish off the remaining work to improve alias safety & maintenance.

We've also changed the default power levels for new rooms so that room upgrades and ACLs require you to be an Admin (PL100), and invites in public rooms now require you to be a moderator (PL50).

Get the new release from github or any of the sources mentioned at https://github.com/matrix-org/synapse/blob/master/INSTALL.md.

Synapse 1.11.0 (2020-02-21)

Improved Documentation

• Small grammatical fixes to the ACME v1 deprecation notice. (#6944)

Synapse 1.11.0rc1 (2020-02-19)

Features

• Admin API to add or modify threepids of user accounts. (#6769)
• Limit the number of events that can be requested by the backfill federation API to 100. (#6864)
• Add ability to run some group APIs on workers. (#6866)
• Reject device display names over 100 characters in length to prevent abuse. (#6882)
• Add ability to route federation user device queries to workers. (#6873)
• The result of a user directory search can now be filtered via the spam checker. (#6888)
• Implement new GET /_matrix/client/unstable/org.matrix.msc2432/rooms/{roomId}/aliases endpoint as per MSC2432. (#6939, #6948, #6949)
• Stop sending m.room.aliases events when adding / removing aliases. Check alt_aliases in the latest m.room.canonical_alias event when deleting an alias. (#6904)
• Change the default power levels of invites, tombstones and server ACLs for new rooms. (#6834)

Bugfixes

• Fixed third party event rules function on_create_room's return value being ignored. (#6781)
• Allow URL-encoded User IDs on /_synapse/admin/v2/users/<user_id>[/admin] endpoints. Thanks to @NHAS for reporting. (#6825)
• Fix Synapse refusing to start if federation_certificate_verification_whitelist option is blank. (#6849)
• Fix errors from logging in the purge jobs related to the message retention policies support. (#6945)
• Return a 404 instead of 200 for querying information of a non-existent user through the admin API. (#6901)

Updates to the Docker image

• The deprecated "generate-config-on-the-fly" mode is no longer supported. (#6918)

Improved Documentation

• Add details of PR merge strategy to contributing docs. (#6846)
• Spell out that the last event sent to a room won't be deleted by a purge. (#6891)
• Update Synapse's documentation to warn about the deprecation of ACME v1. (#6905, #6907, #6909)
• Add documentation for the spam checker. (#6906)
• Fix worker docs to point /publicised_groups API correctly. (#6938)
• Clean up and update docs on setting up federation. (#6940)
• Add a warning about indentation to generated configuration files. (#6920)
• Databases created using the compose file in contrib/docker will now always have correct encoding and locale settings. Contributed by Fridtjof Mund. (#6921)
• Update pip install directions in readme to avoid error when using zsh. (#6855)

Deprecations and Removals

• Remove m.lazy_load_members from unstable_features since lazy loading is in the stable Client-Server API version r0.5.0. (#6877)

Internal Changes

• Add type hints to SyncHandler. (#6821)
• Refactoring work in preparation for changing the event redaction algorithm. (#6823, #6827, #6854, #6856, #6857, #6858)
• Fix stacktraces when using ObservableDeferred and async/await. (#6836)
• Port much of synapse.handlers.federation to async/await. (#6837, #6840)
• Populate rooms.room_version database column at startup, rather than in a background update. (#6847)
• Reduce amount we log at INFO level. (#6833, #6862)
• Remove unused get_room_stats_state method. (#6869)
• Add typing to synapse.federation.sender and port to async/await. (#6871)
• Refactor _EventInternalMetadata object to improve type safety. (#6872)
• Add an additional entry to the SyTest blacklist for worker mode. (#6883)
• Fix the use of sed in the linting scripts when using BSD sed. (#6887)
• Add type hints to the spam checker module. (#6915)
• Convert the directory handler tests to use HomeserverTestCase. (#6919)
• Increase DB/CPU perf of _is_server_still_joined check. (#6936)
• Tiny optimisation for incoming HTTP request dispatch. (#6950)

Hi all,

We're just back from an incredible time at FOSDEM 2020 - Europe's biggest Free & Open Source Software conference. Huge huge thanks to everyone who came to our talks (sorry if you couldn't get in :/), came to talk to us at the stand, or flagged us down to give feedback, chase PRs, file bugs, or just say thanks. Thanks also to FOSDEM to accepting all of our talks this year, and to the FOSDEM organisers for pulling together yet another amazing event :)

We ended up with three talks:

• Making and Breaking Matrix's E2E Encryption
• The Path to Peer-to-Peer Matrix
• Crossing the Bifröst - Bridging All The Things with Matrix

We'll do a proper blog write-up on enabling E2E encryption by default, cross-signing, and all the other E2E encryption work that's been going on once we ship the stable release - but as of Saturday(!) it has landed on Riot/Web Develop, RiotX/Android (0.14.2) and Riot/iOS develop TestFlight, but we're still debugging and we need a bit longer before cutting the final releases.

So, until then, please take a look at the videos if you missed the livestream or weren't at the event!

The FOSDEM video recording lost the slides for the bridging talk - but luckily the presentation itself is a Matrix client and so you can view the slides yourself right here!.

Here's to FOSDEM 2021!

A few years ago, back when Matrix was originally implementing end-to-end encryption, we asked Moxie (the project lead for Signal) whether he’d ever consider connecting Signal (then TextSecure) to Matrix. After all, one of Matrix’s goals is to be an interoperability layer between other communication silos, and one of the reasons for us using Signal’s Double Ratchet Algorithm for Matrix’s encryption was to increase our chances of one day connecting with other apps using the same algorithm (Signal, WhatsApp, Google Allo, Skype, etc). Moxie politely declined, and then a few months later wrote “The ecosystem is moving” to elaborate his thoughts on why he feels he “no longer believes that it is possible to build a competitive federated messenger at all.”

At the time we didn’t respond via a blog post; instead we ended up talking it through a few times in person to see how misaligned we really were. The conclusion was that we agreed to disagree and Moxie said he’d be happy to be proved wrong, and wished us good luck. However, the subject has come up again thanks to Moxie’s talk on the same subject at 36C3 last week, and we keep getting asked to write a formal response on the Matrix side. So, here’s an attempt to do so. (Moxie didn’t want the 36C3 talk recorded, and I haven’t watched it, so this is responding to the original blog post).

From my perspective, the main points proposed in ‘The ecosystem is moving’ boil down to:

• Decentralised systems are harder to design and build than centralised ones, as coordination is harder if you don’t have a single authority to trust.

• Decentralised systems are harder and slower to evolve than centralised ones, as you can’t force participants to rapidly roll out (or even agree on) new features.

• Users in federated systems tend to coalesce around the best/biggest server that the bulk of people use - which means that server typically gets to see a disproportionate amount of communication metadata (who’s talking to who, and when), and has disproportionate power over the network, which could bully others away from running their own deployments.

• If users don’t trust their app provider, they can always go switch apps, which gives them freedom.

• Open systems are less secure because you have no control over the quality of the implementations - if anyone can bring their own client or server to the table, all it takes is one bad implementation to compromise everyone in the vicinity.

Now, all of these points are valid to some extent.

It’s absolutely true that decentralised systems are harder than centralised ones. Prior to Matrix we built centralised comms systems - we literally can do a side-by-side comparison for the same team to see how easily and fast we built our centralised comms system relative to Matrix. Empirically It took us around 6 times longer to get to the same feature-set with Matrix.

It’s also true that decentralised systems are harder to evolve than centralised ones - you can’t just push out a given feature with a single app update, but you have to agree and publish a public spec, support incremental migration, and build governance processes and community dynamics which encourage everyone to implement and upgrade. This is hard, but not impossible: we’ve spent loads of time and money on Matrix’s governance model and spec process to get it right. It’s still not perfect, but we haven’t seen much fragmentation so far, and when we’re pushing out a feature empirically we can and do go just as fast as the centralised alternatives. (E2E by default is a bit of a special case because we’ve had to go and reimplement many features users take for granted today in an E2E-capable manner, but we’re sprinting to get it done in the coming weeks). A bigger problem is that there are hundreds of spec change proposals which folks would like to see in the protocol, and finding a way to manage expectations and parallelise spec progress is hard - something we’re looking to improve in 2020 (although still figuring out how!)

It’s also fair that in a multi-server federated model, users naturally tend to sign up on the most prominent server(s) (e.g. the matrix.org homeserver in the case of Matrix). In practice, the matrix.org homeserver currently makes up about 35% of the visible Matrix network by active users. It’s also true that Matrix servers currently store metadata about who’s talking to who, and when, as a side-effect of storing and relaying messages on behalf of their users. And without an adequate protocol governance system in place, a large server could start pushing around smaller ones in terms of protocol behaviour. In practice, we’re looking into solving metadata protection in Matrix by experimenting with hybrid P2P / Client Server models - letting users store their metadata purely clientside if they so desire, and potentially obfuscating who’s talking to who via mixnets of blinded store & forward servers (more about this coming up at FOSDEM). Combined with nomadic accounts, this would let us eventually turn off the matrix.org server entirely and eliminate the pseudo-centralisation effect - the default ‘server’ would be the one running on your client.

It’s true that if a user doesn’t trust (say) Telegram, they are free to go switch to Signal or WhatsApp or whatever instead… at the massive expense of having to persuade all their friends to install yet another app, and fragmenting their conversation history across multiple apps.

Finally, it’s also true that because anyone can develop a Matrix client or server and connect to the global network, there’s a risk of bad quality implementations in the wild. There are many forks of Riot on the app stores - we simply can’t vouch for whether they are secure. Similarly there are Matrix clients whose E2E encryption is partial, missing, or unreviewed. And there are a wide range of different Matrix servers run by different people with different agendas in different locations, which may be more or less trustworthy.

HOWEVER: all of this completely ignores one critical thing - the value of freedom. Freedom to select which server to use. Freedom to run your own server (perhaps invisibly in your app, in a P2P world). Freedom to pick which country your server runs in. Freedom to select how much metadata and history to keep. Freedom to choose which apps to use - while still having the freedom to talk to anyone you like (without them necessarily installing yet another app). Freedom to connect your own functionality - bots, bridges, integrations etc. Freedom to select which identifiers (if any) to use to register your account. Freedom to extend the protocol. Freedom to write your own client, or build whole new as-yet-unimagined systems on top.

It’s true that if you’re writing a messaging app optimised for privacy at any cost, Moxie’s approach is one way to do it. However, this ends up being a perversely closed world - a closed network, where unofficial clients are banned, with no platform to build on, no open standards, and you end up thoroughly putting all your eggs in one basket, trusting past, present & future Signal to retain its values, stay up and somehow dodge compromise & censorship… despite probably being the single highest value attack target on the ‘net.

Quite simply, that isn’t a world I want to live in.

We owe the entire success of the Internet (let alone the Web) to openness, interoperability and decentralisation. To declare that openness, interoperability and decentralisation is ‘too hard’ and not worth the effort when building a messaging solution is to throw away all the potential of the vibrancy, creativity and innovation that comes from an open network. Sure, you may end up with a super-private messaging app - but one that starts to smell alarmingly like a walled garden like Facebook’s Internet.org initiative, or an AOL keyword, or Google’s AMP.

So, we continue to gladly take up Moxie’s challenge to prove him wrong - to show that it’s both possible and imperative to create an open decentralised messaging platform which (if you use reputable apps and servers) can be as secure and metadata-protecting as Signal… and indeed more so, given you can run your server off the grid, and don’t need to register with a phone number, and in future may not even need a server at all.

--Matthew

(Comments over at HN)

Hi all,

We've just released Synapse 1.7.3 - an important bug fix to address a class of failures due to malformed events. We've seen this in the wild over the last few days, so we'd recommend updating as soon as possible, especially if you are having problems federating.

Get the new release from github or any of the sources mentioned at https://github.com/matrix-org/synapse/blob/master/INSTALL.md.

The changelog since 1.7.2 is:

Synapse 1.7.3 (2019-12-31)

This release fixes a long-standing bug in the state resolution algorithm.

Bugfixes

• Fix exceptions caused by state resolution choking on malformed events. (#6608)

Hi all,

Every year we do an annual wrap-up and retrospective of all the things happening in the Matrix core team - if you’re feeling particularly curious or bored you can check out the 2015, 2016, 2017 and 2018 editions for context. The idea is to look at the bigger picture trends in Matrix outside of the weekly TWIM posts to get an idea of the stuff which we made progress on, and the stuff which still remains.

That said, it’s hard to know where to start - Matrix accelerated more than ever before in 2019, and there’s been progress on pretty much all battlefronts. So as a different format, let’s take the stuff we said we had planned for 2019 from the end of last year’s update and see what we actually achieved...

2019: the immediate priorities

So, our immediate priorities for 2019 were:

• r0 spec releases across the board (aka Matrix 1.0)
• Implementing them in Synapse

✅ Well, unless you’ve been floating in a sensory deprivation tank for the last year, hopefully you spotted that Matrix (as a protocol) finally exited beta - starting off with the announcement at FOSDEM in February of the first stable release of the Server-Server API, alongside the Synapse 0.99.x series as we began the process of migrating to the 1.0 APIs.

Specifically this meant killing off self-signed certificates, adding .well-known server discovery and implementing room version semantics so we could upgrade the underlying room version algorithm to fix the residual flaws. This culminated in June with the official release of Matrix 1.0 - now including the remaining APIs and a stable release of Synapse 1.0. The emphasis was on addressing all the main pre-1.0 design flaws rather than adding features or performance, but with 1.0 out the door at last we’ve been able to make progress there too.

• Landing the Riot redesign

✅ The full redesign of Riot’s UI on Web/Desktop landed shortly after FOSDEM in Feb with The Big 1.0. Cosmetically we got most of the new look & feel in place, and have had very positive feedback overall - although some of the UX thinkos of the old app remain and coming up on the radar for fixing.

• Finalising the Matrix.org Foundation

✅ This happened too, coincident with releasing Matrix 1.0 in June - read all about it at https://matrix.org/foundation. So far the governance and legal infrastructure the Foundation provides has helped the project significantly, and while it was a mammoth task to organise, we’re very glad it’s here! Huge thanks go out to Jon, Ross and Jutta for agreeing to join the foundation as Guardians - they have been excellent in patiently listening to the various dramas of the year and ensuring Matrix’s neutrality and that we keep an even keel.

• Landing all the new E2E encryption UX and features

The good news on E2E encryption is that we’ve been making solid progress throughout the year - the bad news is that we are still yet to turn it on by default. Progress updates for the various pieces of the puzzle are as follows:

• ✅ The final UX is pretty much locked down (after several iterations as we try to tread the balance between trustworthiness and security) - here’s a sneak preview of what we’re aiming at.

• 🏗 Cross-signing is the single biggest remaining piece of work in progress - letting users attest to the trustworthiness of their own devices, so you only ever have to trust a given user once rather than trusting all their devices individually. We gave a very early demo of an experimental implementation back at FOSDEM in Feb, inspired by some of the initial spec proposal at MSC1680 (MSC = Matrix Spec Change, our process for evolving Matrix).

  However, having played with it a bit, MSC1680 turned out to be too generic and complicated (it worked by the user signing a device with any other of their devices, building a twisted maze of which device vouched for which) - and we replaced it with MSC1756, which shifts the model to be the simpler “the user has a key, which they use to sign their devices”. However, this in turn requires more infrastructure - you need somewhere secure to store your signing key, which prompted MSC1946 - Secure Secret Storage & Sharing (SSSS): the ability to sync your signing key between devices by storing it (encrypted, of course) on the server.

  Meanwhile, it also became obvious that the primitives for key verification needed to be improved too: introducing verification by emoji comparison (MSC1267) and QR codes (MSC1543), and switching key verification to be performed in the context of a DM (MSC2241) so that you can see your verification history, find verifications, and easily dip in and out of verifying users as needed.

  Whilst everyone else was panicking about Matrix 1.0 and associated baggage, Uhoreg was off in the wilderness plugging his way through all of this - iterating on the design, speccing it and implementing it in synapse and matrix-js-sdk, complete with a test jig to demonstrate it all working (part 1 and part 2). Over the last few months the rest of the team has joined him though, and we’ve been frantically working away implementing it all on both Riot/Web, iOS & RiotX/Android. For instance - here’s verification happening in DM between Riot/Web & RiotX a few weeks ago, and here’s a very early (unskinned) cut of verification happening in Riot/Web’s RightPanel a few days ago.

  We were hoping to get cross-signing ready for the end of 2019, but in practice we’re now sprinting to get it done by FOSDEM 2020 in Feb - not least because we have a main-stage talk proposed to tell everyone how we landed it and turned on E2E by default... ;)

• ✅ Support for non-E2E clients. The last thing we want is to make it impossible to write a simple Matrix client, or to suddenly excommunicate (hah) all the existing Matrix bots & bridges which haven’t implemented E2E. To this end, poljar created pantalaimon - our very own Matrix daemon, which can sit in the background and offload all your E2EE from your Matrix client by acting as a transparent Matrix proxy which magically encrypts everything. Built on matrix-nio and asyncio python3, We use it in production today for running various bots and it works excellently.

• ✅ Support for search in E2E rooms. Hot off the heels of pantalaimon’s success, poljar also created seshat - a native library for clientside indexing encrypted Matrix events written in Rust, powered by the tantivy full-text search engine. (pantalaimon also has support for indexing via tantivy, which involved contributing python bindings for tantivy, but we ended up going with Rust so we could embed it natively in as many Matrix clients as possible). Seshat is particularly cool in that the indexes themselves are encrypted in on disk - and in future could even be synced between clients using SSSS so you don’t have to reindex your messages every time you log in on a new device. Seshat is implemented behind a labs flag on Riot/Desktop and it will ship as soon Riot/Desktop’s build pipeline is fully updated to support native modules (which will also unlock other goodies, such as using faster/safer native E2E primitives, safer key storage, and Discord-style keyboard-shortcuts for VoIP).

• 🏗 Fixing “unable to decrypt” errors. We’ve done big sprint over the last month or so to track down the final straggling causes of unable to decrypt errors. Some of these are legitimate bugs (e.g. https://github.com/matrix-org/synapse/issues/6399) - but many are artefacts of the current architecture: for instance, if the sender has no way to know your device was in the room when it encrypted a message, you won’t be able to decrypt. We’re addressing this by improving better error messages and feedback so the user isn’t surprised by what’s going on (aiming for Jan) - and in future we’ll have to revisit E2E’s fundamentals to ensure that it’s impossible to receive a message without also receiving the key to decrypt it.

• ✅ Support for push notifications in E2E rooms. This is kinda solved right now by having all clients get (silently) pushed whenever they receive a message in an E2E room with push enabled, and relying on the client to be woken up by the push in order to decrypt the message in order to display the push notification. However, this is battery intensive, and we could probably do better - but this isn’t a blocker for going live.

• 🏗 Support for FilePanel and NotifPanel in E2E rooms. Seshat should fix this by indexing all your messages (and so tracking whether they contain pushes or files, and populating up your local view of your file & notif panels respectively) - just need to ensure it’s hooked up.

...and that’s where things stand right now on E2E by default. We’ll start turning it on by default for private rooms as soon as the UX has landed (probably starting first with new DMs and private rooms, prompting the user in case they want to opt out - and then migrating existing ones). It’s worth noting that we have poured a lot of work into E2E encryption now - often to the detriment of the rest of Matrix; our rich featureset and decentralisation has combined to make this a tough nut to crack, but the end is in sight. Thanks to all for your patience and support while we’ve been working through this.

That takes us to the end of the stuff we planned to prioritise in 2019 - but what about the more speculative medium-term stuff which was on the menu this time last year?

2019: the medium-term priorities

• Reworking and improving Communities/Groups.

We have some really promising UX work and a fairly early spec proposal (MSC1772), but work in earnest hasn’t kicked off yet. It’s going to be one of the next big projects though.

• Reactions.

✅ Riot now has Reactions! 🎉🎉🎉 The only remaining work is to finish the remaining rough edges of the spec proposal (MSC1849) and actually land them in the Matrix spec proper.

• E2E-encrypted Search

✅ Seshat exists! (see above)

• Filtering. (empowering users to filter out rooms & content they're not interested in).

✅ We’ve ended up thinking lots in 2019 about empowering users to filter content. The main impetus has been to ensure that users and communities can filter out abuse (on their own terms), and also to start building infrastructure which can be used for folks to share their own filters. Over the last few months, this has started to take concrete form - with the arrival of MSC2313 “Moderation policies as rooms”, and Mjolnir - a bot you can run to enforce moderation policies on your rooms. It’s all quite early, but we expect a lot more work in this space over the coming year (and it’s wryly amusing that Twitter has also woken up to it being an interesting problem needing to be solved.)

• Extensible events

Sorry folks; no progress here since a flurry of spec work (MSC1767) back in Jan 2019. The good news is that the spec proposal seems to be relatively well received. The bad news is that we haven’t had bandwidth to finish reviewing it, implementing it and migrating it anywhere. It blocks a bunch of really useful stuff in Matrix, and there are users willing to pay for it (via New Vector) - we’ll get to it as soon as we can.

• Editable messages.

✅ These landed too and are a thing of joy! Just need to merge MSC1849.

• Extensible Profiles (we've actually been experimenting with this already).

Similar to Extensible Events, there was a flurry of spec work (MSC1769) back in Jan, but little progress since. This will also unlock a lot of really useful features - e.g. custom status, custom profile data, social timeline rooms etc. We’ll likely get to it shortly after communities work.

• Threading.

🏗 So we actually landed label-based threading (MSC2326) in Synapse 1.6, but it’s not exposed in Riot yet (or elsewhere). It doesn’t have quite the same semantics as Slack-style threading; the idea is to filter down your room based on which messages are tagged as part of a given topic. However, it’s very powerful, and it’ll be fun to add it to Riot at some point in 2020. Meanwhile, better-than-label-based-threading is also on the cards, although slightly lower priority than some of the other stuff in this section.

• Landing the Riot/Android rewrite

🏗 As you probably know, RiotX is a full rewrite of Riot/Android in Kotlin using modern AndroidX and Jetpack idioms - and it entered beta back in June. Since then we’ve been frantically working away on both playing catch-up with the old app… as well as implementing all the new stuff (reactions, edits, new E2E verification, cross-signing etc) which makes no sense to waste time adding in Riot/Android, but also pushes out the timeline on RiotX itself.

We’re currently sprinting to try to get RiotX ready for FOSDEM in February - hopefully users will have felt the app starting to really stabilise over the last few months (it even supports breadcrumbs now!)

• Considering whether to do a similar overhaul of Riot/iOS

🏗 It’s cheating a bit, but Manu (the lead developer on Riot/iOS and delivery manager of Riot/Mobile in general) has been hacking on an entirely new client called Messagerie in his spare time, using SwiftUI. The idea of throwing away the whole UI layer and replacing it with the latest best practices sounds suspiciously like RiotX - it’ll be interesting to see how RiotX/iOS takes shape next year!

• Scaling synapse via sharding the master process

We ended up bottlenecked on IO rather than CPU in 2019, and as a result we worked on splitting synapse’s database across multiple database instances on a per-table granularity. However, the master process itself doesn’t shard yet; so we’re now bottlenecked on CPU and need to get on and do this asap to unlock further Synapse scalability for mega-monolithic-deployments like the Matrix.org homeserver.

• Bridge UI for discovery of users/rooms and bridge status

🏗 There’s been a bit of movement in the last few weeks on this, but nothing concrete yet.

• Bandwidth-efficient transports

✅ We finished the 100bps CoAP transport proof-of-concept for Matrix, demoed it at FOSDEM and shipped it in March. However, we haven’t progressed it much further; it really needs a corporate sponsor who wants to fund work to finish it off and bake it properly into Matrix. If you’re interested, please get in touch.

• Bandwidth-efficient routing

🏗 We also did a bunch of related work on bandwidth-efficient routing, which sadly hasn’t been released yet. However, it’s interesting to note that the Decentralized Systems and Network Services Research Group at Karlsruhe Institute of Technology’s Institute of Telematics has been looking into this space too - c.f. their A Glimpse of the Matrix paper, which ponders very similar problems.

• Getting Dendrite to production.

🏗 Dendrite work has been bubbling away in the background thanks to Anoa, Brendan, cnly (our GSoC dendrite contributor) and others. Inevitably most of our bandwidth has gone into getting Synapse to 1.0 and making sure it’s fit for purpose, but we want and need to keep Dendrite alive for next-generation purposes - and in fact New Vector is hiring new people to work on it in 2020.

• Inline widgets (polls etc)

🏗 We have an MSC (MSC2192), but not an implementation.

• Improving VoIP over Matrix.

Very little progress here, frustratingly. Jitsi has been upgraded and conference calls should kick ass these days (let us know if they don’t), but 1:1 needs a lot of love. Hopefully we’ll get to it in 2020.

• Adding more bridges, and improving the current ones.

Lots of bridging progress in 2020 - all new puppeting Slack support; huge fixes to the IRC bridge (including shifting to Postgres at last); Bifrost (the XMPP bridge) progressed too, and there’s been loads of community bridging work around WhatsApp, Discord and others.

• Account portability
• Replacing MXIDs with public keys

We’ve just started looking at implementing these seriously via MSC1228 (as of last week) - expect progress in 2020.

So that sums up progress on the medium term menu - as you can see, a bunch actually happened; a bunch made progress; a few didn’t happen at all.

2019: the longer-term priorities

Finally, on the longer term radar:

• Shared-code cross-platform client SDKs (e.g. sharing a native core library between matrix-{js,ios,android}-sdk)

No progress here. Instead, all three main platforms have continued to write and maintain their own platform-specific SDKs for now. Seshat however will be the first piece of native rust code shared across all 3 platforms - let’s see how that goes first...

• Matrix daemons (e.g. running an always-on client as a background process in your OS which apps can connect to via a lightweight CS API)

✅ Pantalaimon lives!

• Push notifications via Matrix (using a daemon-style architecture)

No progress here, unless you count the CoAP low-bandwidth work. However, Bubu (also Riot/Android Fdroid maintainer) has been working on a project called OpenPush which looks to help in this space (albeit not built on Matrix, but could be used by Matrix). There are a few other related projects. If someone wants to build this on top of Matrix + CoAP please get in touch asap!

• Clientside homeservers (i.e. p2p matrix) - e.g. compiling Dendrite to WASM and running it in a service worker.

🏗 Work is actually happening on this currently. Dendrite has successfully compiled to WASM and runs, and we’ve had it (almost) talking HTTP tunnelled over libp2p as part of P2P Matrix experiments. In 2020 we’re going to be investing a lot in P2P Matrix - to give users full control of their communication without even having to run a server, and also to simplify onboarding and account portability enormously. We have a talk about this accepted for FOSDEM 2020 (The Path to P2P Matrix) and we’re actively (frantically) hacking on Dendrite to make it happen - keep an eye out for how things develop!

• Experimenting with MLS for E2E Encryption

🏗 Now that E2E-by-default has entered the “it works! let’s land it in Riot asap” phase, Uhoreg has had some time to start thinking about the longer term future of encryption in Matrix. MLS (Messaging Layer Security) is the IETF’s initiative to define a standard mechanism for end-to-end-encrypted group chats, which has some major algorithmic improvements over Olm/Megolm and the Double Ratchet Algorithm as used by Signal. The catch is that it doesn’t work at all well with decentralisation - however, we’ve been working with them to try to ensure MLS can work in a decentralised world. More recently, uhoreg has had a chance to think a lot more about this and we’re working on a proposal for Decentralised MLS which builds on plain MLS while also giving the semantics needed for Matrix. It’s all very experimental at this point (and the proof-of-concept implementation is written in Julia!) - but looks promising. We’ll share more asap, and will certainly be investing more time in this in 2020..

• Storing and querying more generic data structures in Matrix (e.g. object trees; scene graphs)

Sadly no progress here :(

• Alternate use cases for VR, IoT, etc.

...and none here either.

So, of all the myriad things on our radar for 2019 (as of Dec 2018), hopefully this gives some idea of where we hit the mark.

2019: the unpredictable bits

However, there’s also a tonne of other stuff which happened which wasn’t explicitly on the radar. On the synapse side, we finished fully migrating from Python 2 to Python 3, and started using asyncio and all the latest Python 3 goodies! We finally implemented configurable history retention for servers and rooms! We even implemented self-destructing messages in Synapse (not that Riot exposes them yet). And there has been loads of optimisation and performance work since 1.0 landed in June.

On the ops side, we overhauled all our ops processes and security after the Matrix.org datacenter breach in April, throwing away our legacy infrastructure and rebuilding it properly - and subsequently have been expanding our ops team from one dedicated ops person to four. We also found ourselves having to do another emergency datacenter migration back in November when the old one was unable to reliably service IO for our database cluster.

We also spent a bunch time after shipping Matrix 1.0 working on tightening up Matrix’s privacy model - particularly around third party identity servers, integration managers, and making sure that folks self-hosting Matrix don’t accidentally depend on use 3rd party services without realising it. If you missed out on the fun at the time, you can read all about it here and here. This ended up being way more work than we expected, but we’re very glad to have sorted it out now.

Meanwhile, mainstream uptake of Matrix has properly taken off, with the French Government launching Tchap (their fork of Riot), now with hundreds of thousands of daily active users. The German Government revealed today that they are also formally trialling Matrix, starting with the Bundeswehr (Ministry of Defense); we’ve been helping them out with the deployment too. It is not an exaggeration to suggest that we could end up with an official cross-government Matrix network, publicly federated with the wider Internet, for self-hosted encrypted decentralised instant messaging. In fact Ulrich Kelber, the Bundesdatenschutzbeauftragte (Federal Data Protection Commissioner) for Germany pointed out: “You could even set up a privacy-friendly messenger service in cooperation with France, which in the medium term could represent a real alternative to existing products on the market as a pan-European solution”.

Alongside all this, Mozilla announced they are replacing the Moznet IRC network with Matrix; KDE joined Matrix in Feb, Wikimedia is getting set up on their server, and more and more massive players (including the largest in the world) keep getting in touch to find out how they can best get onboard Matrix - it’s incredibly exciting. It also means that we were able to raise capital to keep folks employed to work on Matrix fulltime via New Vector and scale up Modular.im as a paid hosting platform - which massively helps support core Matrix development.

2020

All that remains now is to make some predictions for 2020. Our main priorities are:

• Get E2E enabled for private rooms by default (see above).

• Riot First-time User Experience (FTUE). While we redesigned Riot’s UI in 2019, there are still far too many weird gotchas which trip over new users. Starting in October we began a shift to completely change how Riot development works - transitioning the project to being led by the UX design team rather than the dev team, and ensuring that the design team considers the app holistically across all 3 platforms. Above all else, our priority is to make it kick ass for normal non-technical mainstream users - not just for opensourcey wizards. This is a tough ask, but we believe it’s literally make-or-break for the project in the long term if Matrix is ever to become as prevalent as Slack or WhatsApp, and we are throwing everything we have at it. The second that E2E is on by default, the entirety of the Riot teams will be focusing on the mission to clear our FTUE backlog.

• RiotX. We’re shipping RiotX on Android as fast as we can - currently users on Riot/Android are left high and dry and we need to do everything we can to finish RiotX and get them upgraded as rapidly as possible.

• Communities. Off the back of FTUE comes the importance of grouping rooms & users together into communities in a much better way than we have today. This will be up next.

• Synapse: shard the master by user/room to avoid being it being bottlenecked on CPU. We also need to apply smarter queue management on federation traffic to better reduce the memory footprint (and so eliminate complexity limits on small-footprint hosted servers!) - and we also desperately need to speed up joins.

• Dendrite & P2P Matrix: the plan currently is to use Dendrite as the basis for our P2P Matrix experiments. In practice this means making it federate using MSC1228-semantics (no point in wasting time implementing the ‘legacy’ key management), and then experiment with hooking it up to various P2P transports (e.g. our low-bandwidth CoAP transport) and discovery systems (e.g. mDNS; libp2p; etc). How we go about actually getting it into production depends entirely on how well the experiment goes; we could evolve Matrix to be hybrid CS/P2P; we could treat it as a new protocol and bridge to it; who knows. Watch this space...

• MLS: figure out our plan for next-generation E2E - for better scaling, and better reliability, and what (if anything) we should do with MLS.

• Bridges: loads of work on the horizon to put a better UX on Bridging. Bridge stability has improved enormously over the last year (thanks Half-Shot!) but we need to transition from being robust but ugly to being robust and polished...

• Spec: we need to work out how to go faster on reviewing MSCs (both our own and from the wider community). While the governance process in general feels healthier than it’s ever been, empirically we’re not exactly burning through the MSC backlog - and this is in part that MSC work is squeezed in alongside the other dayjob stuff everyone’s working on. Finding the right balance between sculpting spec and sculpting code is tough, but we’re going to try to improve it in 2020.

• Abuse / Reputation: we want to empower users to make their own minds up about what content they want to see and not see on Matrix (or what they want to host or not host on their servers / communities / rooms). Mjolnir is a good start, but we’ll be continuing to work on this throughout the year.

Meanwhile, all the things listed above that we didn’t get to in 2019 are of course still options on the menu too.

So there you have it. I’ve not even tried to talk about the amazing stuff that the wider Matrix community has been up to - whether that’s amazing new clients like Ditto (React Native!) or Nio! (SwiftUI), or new bridges like mautrix-facebook and mautrix-hangouts, or even poljar’s secret rewrite of weechat-matrix in Rust; your best bet there is to skim through TWIM. Huge undying thanks go out though to everyone who builds on Matrix and keeps the ecosystem maturing and growing (especially while we’re scurrying around shoring up the foundations) - there’s simply no point in Matrix as a protocol without the vibrant community building on top.

All told, it’s been a bit of an epic year (both in terms of wins and fails), and all that remains is to thank everyone who continues to use Matrix (particularly our Patreon supporters) for their ongoing support and for helping the project accelerate forwards. More than ever before, the world needs free and open communication open to all; the age of proprietary communication silos may be coming to an end - consigned to live alongside AOL CDs and Compuserve IDs in the history books. With your support, Matrix can provide a decent mainstream yet decentralised alternative - and we’ll do everything we can to make that happen in 2020.

Happy holidays!

Matthew, Amandine & the whole Matrix.org team.