The Matrix Spring Special!

It's been 3 months since the Matrix Holiday Special and once again we've all been too busy writing code to put anything that detailed on the blog. So without further a do here's a quick overview of how things have progressed so far in 2016!

🔗Home servers

🔗Synapse

Work on Synapse (our reference homeserver) has been primarily focused on improving performance. This may sound boring, but there's been a huge amount of improvement here since synapse 0.12 was released on Jan 4. Synapse 0.13 on Feb 10 brought huge CPU savings thanks to a whole fleet of caching and other optimisation work - the best way of seeing the difference here is to look at the load graph of the server that hosts matrix.org's synapse+postgres over the last few months:

Ignoring the unrelated blip during March, you can see an enormous step change in system load (which had a matching decrease in actual CPU usage) at the beginning of Feb when the 0.13 optimisations landed on matrix.org :)

Meanwhile, Synapse 0.14 is due any day now with 0.14.0-rc2 released on Wednesday. Here, the focus has been all about memory optimisation - anyone who's run a Synapse seriously will be aware that it can be a memory hog thanks to aggressively caching as much state and history in RAM as possible to avoid hitting the database and keeping everything responsive. 0.14 should improve memory usage just as dramatically as 0.13 improved CPU utilisation - introducing a quick-and-dirty SYNAPSE_CACHE_FACTOR environment variable that lets admins dial down the aggressiveness of the caching (at the expense of performance), but more interestingly implementing string interning and ensuring that events are cached by ID rather than duplicated across multiple caches in order to make memory usage more efficient. It's too early to have impressive looking graphs, and there are still a few memory spikes being tracked down before we release 0.14, but we're hoping for at least a 50% reduction in memory footprint.

Featurewise the highlights include: server-generated unread notification & highlight counts and push badge support, lots of support and refinements for guest access and 3rd party ID invites. Meanwhile we've finally fixed some of the most embarrassing long-standing missing features such as letting folks logout serverside(!), delete aliases and determine whether rooms should be published in the room directory or not.

Finally, Synapse is now part of FreeBSD Ports thanks to Brendan Molloy, and NixOS thanks to Robin Lambertz! Huge thanks to them for contributing the packages to the respective OSes and to all the other synapse package maintainers out there!

It's incredibly exciting to see Synapse's maturity improving and hitting the optimisation stage of its life; huge kudos to Erik for spearheading the optimisation work. We strongly recommend folks upgrade to 0.14 when it's available; it's never been a better time to run a homeserver! :D

🔗Dendron

Meanwhile, Dendron (our next generation homeserver) development has been progressing interestingly: we finished an initial spike to get a Golang skeleton server in place, albeit one that delegates most of the endpoints through to Synapse. In fact, matrix.org itself has been running via Dendron since February!

The whole point of Dendron is to provide an architecture where we can split apart the various endpoints that Synapse provides today, re-implementing them where appropriate in Golang, and critically letting the endpoints scale horizontally with clusters of backend servers abstracted by the single Dendron API facade. As a result, most of the Dendron work has actually ended up going into restructuring Synapse such that multiple Synapses can be run in a cluster behind a single Dendron, allowing us to horizontally scale API endpoints at last. This takes the form of adding cluster replication support to Synapse. This is still work-in-progress as we go through fixing up more and more state to be replicable (replicatable?) between synapses - hopefully it should land in the Synapse 0.15 timeframe. And then we enter a very very interesting new world of horizontally scalable homeservers...

🔗Ruma

Ruma has also seen some progress over the last few months - Ruma is an independent Rust language homeserver project led by Jimmy Cuadra, and whilst in early development still (currently focusing on the user login and registration system) shows a lot of promise. Lots of work has ended up going into the required Rust dependencies rather than the Matrix code itself, but if you're interested in Rust then please drop by #ruma:matrix.org or #ruma on Freenode and say hi!

🔗Clients

Whilst homeserver development is mainly all about performance and scaling work currently, the client side of the Matrix ecosystem is the polar opposite - with lots of rapid progress on exciting new clients happening from all over the community.

🔗Perpetually Talking Online (PTO)

PTO has evolved enormously since Torrie Fischer first revealed it at the end of 2015. PTO is an independent project that acts as a Matrix client that exposes an IRC server interface - effectively turning any Matrix homeserver into an ircd; letting folks hook their favourite IRC clients directly into Matrix and use it as an enormous decentralised IRC network. (N.B. this is not to be confused with matrix-appservice-irc, which acts as a server-side bridge between Matrix rooms and IRC channels.) Obviously you lose some of the Matrix specific features (read receipts, typing notifs, VoIP, etc) but there's clearly a huge benefit for the IRC community to be able to use Matrix as if it were an IRC network.

There have been three releases so far, with the v0.3.0 ("Carburetor") release in March being tantalisingly close to being usable for everyday purposes. We actually have pto.matrix.org all set up and ready to go as an IRC frontend for the matrix.org homeserver and once issue #60 is resolved we'll be turning it on :)

There's one catch though - XChat was never quite built to handle the hundreds of rooms that we've got used to Matrix supporting... :D

Come hang out in #pto:oob.systems if you're interested in PTO!

🔗Quaternion

Quaternion is a new Qt/QML/C++ desktop client created by Felix Rohrbach. It's a fairly early alpha but still quite usable and in very active development. #quaternion:matrix.org is the place to talk all things Quaternion :)

🔗matrix-glib-sdk

Meanwhile, over on the GTK side of the world, Gergely Polonkai has been been making great progress on his matrix-glib-sdk Glib client SDK for Matrix. The end goal here is to implement a full Telepathy plugin for Matrix on top of the SDK. Originally written in C, but now shifted to Vala, the SDK is in very active development and now implements all(?) of the Matrix client-server API - a snapshot of the work-in-progress SDK API docs can be found at http://gergely.polonkai.eu/matrix-glib-sdk. Next up is a formal release and building out clients on top!

🔗matrix-react-sdk, matrix-ios-sdk, matrix-android-sdk and Vector

Finally, huge amounts of time and effort have continued to be pumped into the official matrix-react-sdk, matrix-ios-sdk and matrix-android-sdk - driven substantially by requirements for Vector, the FOSS Matrix-powered collaboration app that we've been helping with:

The best way of seeing what's been going on here is probably by considering Vector itself, which is currently in formal beta (0.4.1 for web, 0.1.2 for iOS and #116 on Android). The big news includes:

• Beta iOS and Android apps. These are early beta but feedback is very much appreciated - the Android beta can be downloaded from Jenkins; if you want to help beta iOS via TestFlight, come ask on #ios:matrix.org.
• Guest access. Anyone can jump straight into Matrix by going to http://vector.im without even having to sign up for an account. Guests are quite restricted on what they can do (and can only join rooms which explicitly have guest access enabled), but this is a *huge* improvement in getting folks using Matrix.
• Ability to jump to any message ever - e.g. when clicking through search results or when permalinking a message... using precisely the same UI that you use when chatting. Permalinks are awesome. If you want to randomly jump back in time to the first weeks of #matrix:matrix.org, now you can...
• Read Markers, scrolling that remembers the scroll offset per-room, and the ability to jump to unread messages
• Synchronised missed notification and missed highlighted notification information per-room
• Badge counts for unread notifications
• Entirely reworked Room Settings
• Entirely reworked User Settings, including push notification configuration
• Entirely reworked Room Directory
• Lots of performance improvements
• Much improved inviting by email
• Much improved reliability on video conferencing
• Closing literally hundreds and hundreds of bugs...

All that remains right now is yet more bugfixing and incorporating feedback from the current betas! Please give as much feedback as possible in #vector:matrix.org :)

🔗Bridges & Bots

Bridges, bots, and other integrations and application services have inevitably taken slightly lower priority whilst we've been focusing on the core server and client bits of the ecosystem. However, as of March we've started a major new project to get these moving again, starting with a big update to the IRC Bridge. This is due to be released next week, but you can get a sneak peek at what's going into the release at the commit log. Highlights include the ability to persist nicks; connect via IPv6; improve formatted message handling; actually feed error messages from IRC back to Matrix; and much much more.

matrix-appservice-verto also got some love, which means that multiway video conferencing powered by FreeSWITCH now works reliably. The quality still could be improved, but the unreliable call setup that plagued earlier versions is now fixed.

In the next few months we're expecting to see a lot more activity on bridges & bots... watch this space :)

Update Sat March 26:

Totally forgot to mention a few of the key new bridges which have been contributed by the community this year - particularly interesting are the Rocket.Chat<->Matrix bridge written by Sing-Li over at Rocket.Chat which provides basic bridging between the awesome Rocket.Chat collaboration app and the wider Matrix ecosystem. It's early days, but this is incredibly promising for 'hardcoded' bridging between specific rooms - it just needs Rocket.Chat to support 'virtual' users and will then be seamless federation.

Similarly, matrix-appservice-gitter is a Gitter<->Matrix bridge built by Leonerd on top of the matrix-appservice-bridge Node library. Again, it's early days but is working well for 'hardcoded' bridging - supporting dynamic rooms and users is next on the todo list :)

🔗The Spec

We started our formal release process for the spec just before Christmas with r0.0.0 - and released r0.0.1 in January with minor clarifications and updates. In practice the spec feels quite stable right now, although things have moved on a bit since January and r0.0.2 is definitely overdue at this point.

In the meantime, you can always get the very latest bleeding edge copy of the spec via the speculator. We've also added an initial cut at a spec for the Identity Service at last.

🔗Events

We've been focusing on writing code than evangelising Matrix recently, although we did get out to FOSDEM 2016 and TADHack Mini Japan and WebRTC Conference and Enterprise Connect 2016 where we showed off Matrix & Vector in the WebRTC Real World Innovation showcase.

🔗GSoC

We are incredibly grateful to have been accepted as an organisation into Google Summer of Code 2016! The last two weeks have been the window for students to propose projects to us that they could work on over the course of the summer, and it's been fascinating to meet the GSoCers and see a whole new community pop up on Matrix and advise and mentor applicants through their proposals. At the last count we've received 35 proposals, many inspired by our list of ideas, including some really impressive candidates - many thanks to all the students who have applied to us. We don't know yet how many slots Google will allocate to us, but one way or another we're really looking forward to helping the GSoCers make the most out of their summer of Matrix! All GSoC discussion is happening in #gsoc:matrix.org.

🔗What's next?

In no particular order, the urgent stuff that still remains includes:

• Continuing to polish synapse and build out dendron-based clustering
• Building as many bridges, bots and other integrations as possible
• The matrix.to URL-handler service: having client-agnostic https://matrix.to/#matrix:matrix.org URLs to help with sharing matrix room aliases etc
• End-to-end crypto. No progress since December; we need to get back to it asap.
• Exiting Vector from beta
• Finishing the server-to-server API specification
• Improving the security model for access_tokens
• Editable messages
• Pinned, tagged, and 'liked' messages
• Threading
• Decentralised accounts
• Decentralised reputation

In practice, Bridging and E2E crypto is likely to get the most attention first (beyond the standard ongoing polishing). There's obviously a significant amount of work there, but we expect to see benefits pretty quickly throughout Matrix - especially from bridging. Hopefully it's true to say that the next few months should be quite transformational :D

Anyway, thanks for reading this sprawling update and for supporting Matrix. And please come say hi in #matrix:matrix.org if you have any questions :)

• Matthew, Amandine & the Matrix.org team.