It's been 3 months since the Matrix Holiday Special and once again we've all been too busy writing code to put anything that detailed on the blog. So without further a do here's a quick overview of how things have progressed so far in 2016!
Work on Synapse (our reference homeserver) has been primarily focused on improving performance. This may sound boring, but there's been a huge amount of improvement here since synapse 0.12 was released on Jan 4. Synapse 0.13 on Feb 10 brought huge CPU savings thanks to a whole fleet of caching and other optimisation work - the best way of seeing the difference here is to look at the load graph of the server that hosts matrix.org's synapse+postgres over the last few months:
Ignoring the unrelated blip during March, you can see an enormous step change in system load (which had a matching decrease in actual CPU usage) at the beginning of Feb when the 0.13 optimisations landed on matrix.org :)
Meanwhile, Synapse 0.14 is due any day now with 0.14.0-rc2 released on Wednesday. Here, the focus has been all about memory optimisation - anyone who's run a Synapse seriously will be aware that it can be a memory hog thanks to aggressively caching as much state and history in RAM as possible to avoid hitting the database and keeping everything responsive. 0.14 should improve memory usage just as dramatically as 0.13 improved CPU utilisation - introducing a quick-and-dirty SYNAPSE_CACHE_FACTOR environment variable that lets admins dial down the aggressiveness of the caching (at the expense of performance), but more interestingly implementing string interning and ensuring that events are cached by ID rather than duplicated across multiple caches in order to make memory usage more efficient. It's too early to have impressive looking graphs, and there are still a few memory spikes being tracked down before we release 0.14, but we're hoping for at least a 50% reduction in memory footprint.
Featurewise the highlights include: server-generated unread notification & highlight counts and push badge support, lots of support and refinements for guest access and 3rd party ID invites. Meanwhile we've finally fixed some of the most embarrassing long-standing missing features such as letting folks logout serverside(!), delete aliases and determine whether rooms should be published in the room directory or not.
Finally, Synapse is now part of FreeBSD Ports thanks to Brendan Molloy, and NixOS thanks to Robin Lambertz! Huge thanks to them for contributing the packages to the respective OSes and to all the other synapse package maintainers out there!
It's incredibly exciting to see Synapse's maturity improving and hitting the optimisation stage of its life; huge kudos to Erik for spearheading the optimisation work. We strongly recommend folks upgrade to 0.14 when it's available; it's never been a better time to run a homeserver! :D
Meanwhile, Dendron (our next generation homeserver) development has been progressing interestingly: we finished an initial spike to get a Golang skeleton server in place, albeit one that delegates most of the endpoints through to Synapse. In fact, matrix.org itself has been running via Dendron since February!
The whole point of Dendron is to provide an architecture where we can split apart the various endpoints that Synapse provides today, re-implementing them where appropriate in Golang, and critically letting the endpoints scale horizontally with clusters of backend servers abstracted by the single Dendron API facade. As a result, most of the Dendron work has actually ended up going into restructuring Synapse such that multiple Synapses can be run in a cluster behind a single Dendron, allowing us to horizontally scale API endpoints at last. This takes the form of adding cluster replication support to Synapse. This is still work-in-progress as we go through fixing up more and more state to be replicable (replicatable?) between synapses - hopefully it should land in the Synapse 0.15 timeframe. And then we enter a very very interesting new world of horizontally scalable homeservers...
Whilst homeserver development is mainly all about performance and scaling work currently, the client side of the Matrix ecosystem is the polar opposite - with lots of rapid progress on exciting new clients happening from all over the community.
There have been three releases so far, with the v0.3.0 ("Carburetor") release in March being tantalisingly close to being usable for everyday purposes. We actually have pto.matrix.org all set up and ready to go as an IRC frontend for the matrix.org homeserver and once issue #60 is resolved we'll be turning it on :)
There's one catch though - XChat was never quite built to handle the hundreds of rooms that we've got used to Matrix supporting... :D
Come hang out in #pto:oob.systems if you're interested in PTO!
Meanwhile, over on the GTK side of the world, Gergely Polonkai has been been making great progress on his matrix-glib-sdk Glib client SDK for Matrix. The end goal here is to implement a full Telepathy plugin for Matrix on top of the SDK. Originally written in C, but now shifted to Vala, the SDK is in very active development and now implements all(?) of the Matrix client-server API - a snapshot of the work-in-progress SDK API docs can be found at http://gergely.polonkai.eu/matrix-glib-sdk. Next up is a formal release and building out clients on top!
Finally, huge amounts of time and effort have continued to be pumped into the official matrix-react-sdk, matrix-ios-sdk and matrix-android-sdk - driven substantially by requirements for Vector, the FOSS Matrix-powered collaboration app that we've been helping with:
The best way of seeing what's been going on here is probably by considering Vector itself, which is currently in formal beta (0.4.1 for web, 0.1.2 for iOS and #116 on Android). The big news includes:
All that remains right now is yet more bugfixing and incorporating feedback from the current betas! Please give as much feedback as possible in #vector:matrix.org :)
Bridges, bots, and other integrations and application services have inevitably taken slightly lower priority whilst we've been focusing on the core server and client bits of the ecosystem. However, as of March we've started a major new project to get these moving again, starting with a big update to the IRC Bridge. This is due to be released next week, but you can get a sneak peek at what's going into the release at the commit log. Highlights include the ability to persist nicks; connect via IPv6; improve formatted message handling; actually feed error messages from IRC back to Matrix; and much much more.matrix-appservice-verto also got some love, which means that multiway video conferencing powered by FreeSWITCH now works reliably. The quality still could be improved, but the unreliable call setup that plagued earlier versions is now fixed.
In the next few months we're expecting to see a lot more activity on bridges & bots... watch this space :)Update Sat March 26:
Totally forgot to mention a few of the key new bridges which have been contributed by the community this year - particularly interesting are the Rocket.Chat<->Matrix bridge written by Sing-Li over at Rocket.Chat which provides basic bridging between the awesome Rocket.Chat collaboration app and the wider Matrix ecosystem. It's early days, but this is incredibly promising for 'hardcoded' bridging between specific rooms - it just needs Rocket.Chat to support 'virtual' users and will then be seamless federation.
Similarly, matrix-appservice-gitter is a Gitter<->Matrix bridge built by Leonerd on top of the matrix-appservice-bridge Node library. Again, it's early days but is working well for 'hardcoded' bridging - supporting dynamic rooms and users is next on the todo list :)
We started our formal release process for the spec just before Christmas with r0.0.0 - and released r0.0.1 in January with minor clarifications and updates. In practice the spec feels quite stable right now, although things have moved on a bit since January and r0.0.2 is definitely overdue at this point.
We've been focusing on writing code than evangelising Matrix recently, although we did get out to FOSDEM 2016 and TADHack Mini Japan and WebRTC Conference and Enterprise Connect 2016 where we showed off Matrix & Vector in the WebRTC Real World Innovation showcase.
We are incredibly grateful to have been accepted as an organisation into Google Summer of Code 2016! The last two weeks have been the window for students to propose projects to us that they could work on over the course of the summer, and it's been fascinating to meet the GSoCers and see a whole new community pop up on Matrix and advise and mentor applicants through their proposals. At the last count we've received 35 proposals, many inspired by our list of ideas, including some really impressive candidates - many thanks to all the students who have applied to us. We don't know yet how many slots Google will allocate to us, but one way or another we're really looking forward to helping the GSoCers make the most out of their summer of Matrix! All GSoC discussion is happening in #gsoc:matrix.org.
In no particular order, the urgent stuff that still remains includes:
In practice, Bridging and E2E crypto is likely to get the most attention first (beyond the standard ongoing polishing). There's obviously a significant amount of work there, but we expect to see benefits pretty quickly throughout Matrix - especially from bridging. Hopefully it's true to say that the next few months should be quite transformational :D
Anyway, thanks for reading this sprawling update and for supporting Matrix. And please come say hi in #matrix:matrix.org if you have any questions :)