We have recently been made aware of a critical security issue in Synapse. Full disclosure of the issue and patch will be made at 2016-07-08 13:00 UTC. We are coordinating with package maintainers to ensure that patched versions of the packages will be available at that time.

If you run your own Synapse please be prepared to upgrade as soon as the patched versions are released.

Thank you for your time, patience and understanding while we resolve this issue, The Matrix Team

Signed pre-disclosure notice

Hi folks - another few months have gone by and once again the core Matrix team has ended up too busy hacking away on the final missing pieces of the Matrix jigsaw puzzle to have been properly updating the blog; sorry about this. The end is in sight for the current crunch however, and we expect to return to regular blog updates shortly! Meanwhile, rather than letting news stack up any further, here's a quick(?) attempt to summarise all the things which have been going on!

🔗Synapse 0.16.1 released!

This one's a biggy: in the mad rush during June to support the public debut for Vector, we made a series of major Synapse releases which apparently we forgot to tell anyone about (sorry!). The full changelog is at the bottom of the post as it's huge, but the big features are:

• Huge performance improvements, including adding write-thru event caches and improving caching throughout, and massive improvements to the speed of the room directory API.
• Add support for inline URL previewing!
• Add email notifications!
• Add support for LDAP authentication (thanks to Christoph Witzany)
• Add support for JWT authentication (thanks to Niklas Riekenbrauck)
• Add basic server-side ignore functionality and abuse reporting API
• Add ability to delegate /publicRooms API requests to a list of secondary homeservers
• Lots and lots and lots of bug fixes.

If you haven't upgraded, please do asap from https://github.com/matrix-org/synapse!

There's also been a huge amount of work going on behind the scenes on horizontal scalability for Synapse.  We haven't drawn much attention to this yet (or documented it) as it's still quite experimental and in flux, but the main change is to add the concept of application-layer replication to Synapse - letting you split the codebase into multiple endpoints which can then be run in parallel, each replicating their state off the master synapse process.  For instance, right now the Matrix.org homeserver is actually running off three different processes: the main synapse; another specific to calculating push notifications and another specific to serving up the /sync endpoint.  These three are then abstracted behind the dendron layer (which also implements the /login endpoint). The idea is that one can then run multiple instances of the /sync and pusher (and other future) endpoints to horizontally scale.  For now, they share a single database writer, but in practice this has improved our scalability and performance on the Matrix.org HS radically.

In future we'll actually document how to run these, as well as making it easy to spin up multiple concurrent instances - in the interim if you find you're hitting performance limits running high-traffic synapses come talk to us about it on #matrix-dev:matrix.org.  And the longer term plan continues to be to switch out these python endpoint implementations in future for more efficient implementations.  For instance, there's a golang implementation of the media repository currently in development which could run as another endpoint cluster.

🔗Vector released!

Much has been written about this elsewhere, but Web, iOS and Android versions of the Vector clients were finally released to the general public on June 9th at the Decentralised Web Summit in San Francisco.  Vector is a relatively thin layer on top of the matrix-react-sdk, matrix-ios-sdk and matrix-android-sdk Matrix.org client SDKs which showcases Matrix's collaboration and messaging capabilities in a mass-market usable app.  There's been huge amounts of work here across the SDKs for the 3 platforms, with literally thousands of issues resolved.  You can find the full SDK changelogs on github for React, iOS and Android.  Some of the more interesting recent additions to Vector include improved room notifications, URL previews, configurable email notifications, and huge amounts of performance stability work.

Future work on Vector is focused on showcasing end-to-end encryption, providing a one-click interface for adding bots/integrations & bridges to a room, and generally enormously improving the UX and polish.  Meanwhile, there's an F-Droid release for Android landing any day now.

If you haven't checked it out recently, it's really worth a look :)

🔗Matrix Spec 0.1.0

In case you didn't notice, we also released v0.1.0 of the Matrix spec itself in May - this is a fairly minor update which improves the layout of the document somewhat (thanks to a PR from Jimmy Cuadra) and a some bugfixes.  You can see the full changelog here. We're overdue a new release since then (albeit again with relatively minor changes).

🔗Google Summer of Code

We're in the middle of the second half of GSoC right now, with our GSoC students Aviral and Half-Shot hacking away on Vector and Microblogging projects respectively.  There's a lot of exciting stuff coming out of this - Aviral contributing Rich Text Editing, Emoji autocompletion, DuckDuckGo and other features into Vector (currently on branches, but will be released soon) and Half-Shot building a Twitter bridge as part of his Matrix-powered microblogging system.  Watch this space for updates!

🔗Ruma

Lots of exciting stuff has been happening recently over at Ruma.io - an independent Matrix homeserver implementation written in Rust.  Over the last few weeks Jimmy and friends have got into the real meat of implementing events and the core of the Matrix CS API, and as of the time of writing they're the topmost link on HackerNews!  There's a lot of work involved in writing a homeserver, but Ruma is looking incredibly promising and the feedback from their team has been incredibly helpful in keeping us honest on the Matrix spec and ensuring that it's fit for purpose for 3rd party server implementers.

Also, Ruma just released some truly excellent documentation as a high-level introduction to Matrix (thanks to Leah and Jimmy) - much better than anything we have on the official Matrix.org site.  Go check it out if you haven't already!

🔗End to End Encryption

There has been LOADS of work happening on End to End encryption: finalising the core 1:1 "Olm" cryptographic ratchet; implementing the group "Megolm" ratchet (which shares a single ratchet over all the participants of a room for scalability); fully hooking Olm into matrix-js-sdk and Vector-web at last, and preparing for a formal and published-to-the-public 3rd party security audit on Olm which will be happening during July.

This deserves a post in its own right, but the key thing to know is that Olm is almost ready - and indeed the work-in-progress E2E UX is even available on the develop branch of vector if you enable E2E in the new 'Labs' section in User Settings.  Olm itself is usable only for 'burn after reading' strictly PFS messages, but Megolm integration with Vector & Synapse will follow shortly afterwards which will finally provide the E2E nirvana we've all been waiting for :)

🔗Decentralised Web Summit

Matrix had a major presence as a sponsor at the first ever Decentralised Web Summit hosted by the Internet Archive in San Francisco back in June.  This was a truly incredible event - with folks gathering from across the world to discuss, collaborate and debate on ensure that the web is not fragmented or trapped into proprietary silos - with the likes of Tim Berners-Lee, Vint Cerf and Brewster Kahle in attendance.  We ran a long 2 hour workshop on Matrix and showed off Vector to anyone and everyone - and meanwhile the organisers were kind enough to promote Matrix as the main decentralised chat interface for the conference itself (bridged with their Slack).  A full writeup of the conference really merits a blog post in its own right, but the punchline is that you could genuinely tell that this is the beginning of a new era of the internet - whether it's using Merkle DAGs (like Matrix) or Blockchain or similar technologies: we are about to see a major shift in the balance of power on the internet back towards its users.

We strongly recommend checking out the videos which have all been published at Decentralised Web Summit, including lightning talks introducing both Matrix and Vector, and digging into as many of the projects advertised as possible.  It was particularly interesting for us to get to know Tim Berners-Lee's latest project at MIT: Solid - which shares quite a lot of the same goals as Matrix, and subsequently seeing Tim pop up on Matrix via Vector.  We're really looking forward to working out how Matrix & Solid can complement each other in future.

🔗Matrix.to

Not the most exciting thing ever, but heads up that there's a simple site up at https://matrix.to to provide a way of doing client-agnostic links to content in Matrix.  For instance, rather than linking specifically into an app like Vector, you can now say https://matrix.to/#/#matrix:matrix.org to go there via whatever app you choose.  This is basically a bootstrapping process towards having proper mx:// URLs in circulation, but given mx:// doesn't exist yet, https://matrix.to hopefully provides a useful step in the right direction :)

PRs very welcome at https://github.com/matrix-org/matrix.to.

🔗Bridges and Bots

Much of the promise of Matrix is the ability to bridge through to other silos, and we've been gradually adding more and more bridging capabilities in.

For instance, the IRC bridge has had a complete overhaul to add in huge numbers of new features and finally deployed for Freenode a few weeks ago:

New Features:

• Nicks set via !nick will now be preserved across bridge restarts.
• EXPERIMENTAL: IRC clients created by the bridge can be assigned their own IPv6 address.
• The bridge will now send connection status information to real Matrix users via the admin room (the same room !nickcommands are issued).
• Added !help.
• The bridge will now fallback to body if the HTML content contains any unrecognised tags. This makes passing Markdown from Matrix to IRC much nicer.
• The bridge will now send more metrics to the statsd server, including the join/part rate to and from IRC.
• The config option matrixClients.displayName is now implemented.

Bug fixes:

• Escape HTML entities when sending from IRC to Matrix. This prevents munging occurring between IRC formatting and textual < element > references, whereby if you sent a tag and some colour codes from IRC it would not escape the tag and therefore send invalid HTML to Matrix.
• User IDs starting with - are temporarily filtered out from being bridged.
• Deterministically generate the configuration file.
• Recognise more IRC error codes as non-fatal to avoid IRC clients reconnecting unnecessarily.
• Add a 10 second timeout to join events injected via the MemberListSyncer to avoid HOL blocking.
• 'Frontier' Matrix users will be forcibly joined to IRC channels even if membership list syncing I->M is disabled. This ensures that there is always a Matrix user in the channel being bridged to avoid losing traffic.
• Cache the /initialSync request to avoid hitting this endpoint more than once, as it may be very slow.
• Indexes have been added to the NeDB .db files to improve lookup times.
• Do not recheck if the bridge bot should part the channel if a virtual user leaves the channel: we know it shouldn't.
• Refine what counts as a "request" for metrics, reducing the amount of double-counting as requests echo back from the remote side.
• Fixed a bug which caused users to be provisioned off their user_id even if they had a display name set.

Meanwhile, a Gitter bridge is in active development (and in testing with the Neovim community on Gitter/Matrix/Freenode), although lacking documentation so far.

Finally, NEB - the Matrix.org bot framework is currently being ported from Python to Golang to act as a general Go SDK for rapidly implementing new bot capabilities.

There's little point in all of the effort going into bridges and bots if it's too hard for normal users to deploy them, so on the Vector side of things there's an ongoing project to build a commercial-grade bot/bridge hosted service offering for Matrix which should make it much easier for non-sysadmins to quickly add their own bots and bridges into their rooms.  There's nothing to see yet, but we'll be yelling about it when it's ready!

🔗Conclusion

I'm sure there's a lot of stuff missing from the quick summary above - suffice it to say that the Matrix ecosystem is growing so fast and so large that it's pretty hard to keep track of everything that's going on.  The big remaining blockers we see at this point are:

• End-to-end Encryption roll-out
• Polishing UX on Vector - showing that it's possible to build better-than-Slack quality UX on top of Matrix
• Bots, Integrations and Bridges - making them absolutely trivial to build and deploy, and encouraging everyone to write as many as they can!
• Improving VoIP, especially for conferencing, especially on Mobile
• Threading
• Editable messages
• Synapse scaling and stability - this is massively improved, but there's still work to be done.  Meanwhile projects like Ruma give us hope for light at the end of the Synapse tunnel!
• Spec refinements - there are still a lot of open spec bugs which we need to resolve so we can declare the spec (and thus Matrix!) out of beta.
• More clients - especially desktop ones; helping out with Quaternion, Tensor, PTO, etc.

...and then all the pieces of the jigsaw will finally be in place, and Matrix should hopefully fulfil its potential as an invaluable, open and decentralised data fabric for the web.

Thanks, once again, to everyone who's been supporting and using Matrix - whether it's by hanging out in the public chatrooms, running your own server, writing your own clients, bots, or servers, or just telling your friends about the project.  The end of the beginning is in sight: thanks for believing in us, and thank you for flying Matrix.

Matthew, Amandine & the Matrix Team.

🔗Appendix: The Missing Synapse Changelogs

🔗Changes in synapse v0.16.1 (2016-06-20)

Bug fixes:

• Fix assorted bugs in /preview_url (PR #872)
• Fix TypeError when setting unicode passwords (PR #873)

Performance improvements:

• Turn use_frozen_events off by default (PR #877)
• Disable responding with canonical json for federation (PR #878)

🔗Changes in synapse v0.16.1-rc1 (2016-06-15)

Features: None

Changes:

• Log requester for /publicRoom endpoints when possible (PR #856)
• 502 on /thumbnail when can't connect to remote server (PR #862)
• Linearize fetching of gaps on incoming events (PR #871)

Bugs fixes:

• Fix bug where rooms where marked as published by default (PR #857)
• Fix bug where joining room with an event with invalid sender (PR #868)
• Fix bug where backfilled events were sent down sync streams (PR #869)
• Fix bug where outgoing connections could wedge indefinitely, causing push notifications to be unreliable (PR #870)

Performance improvements:

• Improve /publicRooms performance (PR #859)

🔗Changes in synapse v0.16.0 (2016-06-09)

NB: As of v0.14 all AS config files must have an ID field.

Bug fixes:

• Don't make rooms published by default (PR #857)

🔗Changes in synapse v0.16.0-rc2 (2016-06-08)

Features:

• Add configuration option for tuning GC via gc.set_threshold (PR #849)

Changes:

• Record metrics about GC (PR #771, #847, #852)
• Add metric counter for number of persisted events (PR #841)

Bug fixes:

• Fix 'From' header in email notifications (PR #843)
• Fix presence where timeouts were not being fired for the first 8h after restarts (PR #842)
• Fix bug where synapse sent malformed transactions to AS's when retrying transactions (Commits310197b, 8437906)

Performance Improvements:

• Remove event fetching from DB threads (PR #835)
• Change the way we cache events (PR #836)
• Add events to cache when we persist them (PR #840)

🔗Changes in synapse v0.16.0-rc1 (2016-06-03)

Version 0.15 was not released. See v0.15.0-rc1 below for additional changes.

Features:

• Add email notifications for missed messages (PR #759, #786, #799, #810, #815, #821)
• Add a url_preview_ip_range_whitelist config param (PR #760)
• Add /report endpoint (PR #762)
• Add basic ignore user API (PR #763)
• Add an openidish mechanism for proving that you own a given user_id (PR #765)
• Allow clients to specify a server_name to avoid 'No known servers' (PR #794)
• Add secondary_directory_servers option to fetch room list from other servers (PR #808, #813)

Changes:

• Report per request metrics for all of the things using request_handler (PR #756)
• Correctly handle NULL password hashes from the database (PR #775)
• Allow receipts for events we haven't seen in the db (PR #784)
• Make synctl read a cache factor from config file (PR #785)
• Increment badge count per missed convo, not per msg (PR #793)
• Special case m.room.third_party_invite event auth to match invites (PR #814)

Bug fixes:

• Fix typo in event_auth servlet path (PR #757)
• Fix password reset (PR #758)

Performance improvements:

• Reduce database inserts when sending transactions (PR #767)
• Queue events by room for persistence (PR #768)
• Add cache to get_user_by_id (PR #772)
• Add and use get_domain_from_id (PR #773)
• Use tree cache for get_linearized_receipts_for_room (PR #779)
• Remove unused indices (PR #782)
• Add caches to bulk_get_push_rules* (PR #804)
• Cache get_event_reference_hashes (PR #806)
• Add get_users_with_read_receipts_in_room cache (PR #809)
• Use state to calculate get_users_in_room (PR #811)
• Load push rules in storage layer so that they get cached (PR #825)
• Make get_joined_hosts_for_room use get_users_in_room (PR #828)
• Poke notifier on next reactor tick (PR #829)
• Change CacheMetrics to be quicker (PR #830)

🔗Changes in synapse v0.15.0-rc1 (2016-04-26)

Features:

• Add login support for Javascript Web Tokens, thanks to Niklas Riekenbrauck (PR #671,#687)
• Add URL previewing support (PR #688)
• Add login support for LDAP, thanks to Christoph Witzany (PR #701)
• Add GET endpoint for pushers (PR #716)

Changes:

• Never notify for member events (PR #667)
• Deduplicate identical /sync requests (PR #668)
• Require user to have left room to forget room (PR #673)
• Use DNS cache if within TTL (PR #677)
• Let users see their own leave events (PR #699)
• Deduplicate membership changes (PR #700)
• Increase performance of pusher code (PR #705)
• Respond with error status 504 if failed to talk to remote server (PR #731)
• Increase search performance on postgres (PR #745)

Bug fixes:

• Fix bug where disabling all notifications still resulted in push (PR #678)
• Fix bug where users couldn't reject remote invites if remote refused (PR #691)
• Fix bug where synapse attempted to backfill from itself (PR #693)
• Fix bug where profile information was not correctly added when joining remote rooms (PR #703)
• Fix bug where register API required incorrect key name for AS registration (PR #727)

The original promise of the Internet was to be an interoperable platform for distributing data. However, we have since increasingly seen our data fragmented and trapped in a number of proprietary silos. Matrix hopes to fix this by being a federated, open standard for data exchange that any service can use.

The Decentralized Web Summit is a meetup for anyone interested in building the Decentralized Web, which aims to make the Web open, secure and free of censorship by distributing data, processing, and hosting across millions of computers around the world, with no centralized control. It takes place at the Internet Archive, San Francisco, CA on Wednesday June 8 and Thursday June 9, 2016.

Matrix will be represented at the event, and we hope to also host a workshop or a talk about Matrix.

The meetup has a Slack room set up for pre-meetup conversations - you can also access this room via Matrix: #decentralizedweb-general:matrix.org

We are looking forward to interesting people and interesting conversations at the first Decentralized Web Summit!

Last week I went to Kamailio World 2016 in Berlin to meet fellow VoIP-developers and tell them all about Matrix. It's a fairly small conference, which is actually quite nice as it means you get to talk to almost everyone. A lot of people were interested in Matrix - both new and familiar faces - in fact, some of them heard about Matrix a year ago at Kamailio World 2015 and were interested in hearing what progress we've made since.

As always, Matrix participated in James Body's dangerous demos session - and I also gave a 30min talk on Matrix and recent updates to a full room on the first day of the conference.

Several people mentioned that Matrix could be interesting to their project, either as a glue between services, or for adding text-based chat to VoIP apps. I hope to see some of you in Matrix at some point - please join us in #matrix:matrix.org and say hi! It's also a good place to ask questions and discuss how Matrix can work with your project. Auf Wiedersehen!

Congratulations to Aviral Dasgupta and Will "Half-Shot" Hunt who will be working with Matrix for their Google Summer of Code projects!

As mentioned, picking two projects out of all our proposals was no easy task. However, we now look forward to getting started, and we are sure Aviral and Half-Shot will help make Matrix even better over the next few months!

Aviral will be developing a flexible plugin system to facilitate integrating various services such as github/trello/duckduckgo with Matrix. Meanwhile, Half-Shot will be looking at adding features on top of Matrix - infact, he's already built a MPD DJ bot and started working on a .NET SDK. Aviral too, has been committing various enhancements already.

According to Google's GSoC timeline we are currently in the "Community Bonding" phase, which lasts till May 22, 2016 - which is when the projects formally kick off.

We're looking forward to seeing what awesome things Aviral and Half-Shot come up with!

As previously announced, Matrix is participating in Google Summer of Code (GSoC) 2016. We have had a lot of interest: lots of people joining Matrix to talk to us about their project ideas and a total of 38 project proposals. We have even had some code contributions to our various projects from people who discovered Matrix via GSoC!

It's our first year as a GSoC mentoring organisation and we were only allocated two project slots. This means that we had the tough decision of choosing between some really good projects - and that means a lot of you who applied will unfortunately be left feeling disappointed. Selecting our two projects was very difficult, and we talked it over until we all agreed. Please remember that not being picked does not mean that your proposal was bad.

If you missed out on a GSoC slot this year, that doesn't have to stop you from contributing, either by hacking on your own project or contributing to an existing Matrix project. It's a great way to hone your programming skills and we'll be more than happy to help out and support you - find us in #matrix:matrix.org and #matrix-dev:matrix.org.

All the best from the Matrix team and good luck to everyone in their summer projects, whether GSoC or not!

TADHack-mini took place in east London over the weekend with 88 people in attendance. There were $8k in prizes, and five different services to hack on. This time, we didn't have too many people using Matrix in their hack, but we did have an excellent idea and implementation called Babelonio, who won our prize: a Phantom X Hexapod Mk3!

Babelonio adds speech-to-text and translation via Google translate to Matrix, via the Vector client and a Chrome extension. This is quite nice, because it means you don't have to run a custom client. And by using Google translate, you immediately get access to a lot of languages (although sometimes the translation doesn't quite work, as you can see in the presentation video). The project was done by Steven Bakker, Timo Uelen and Bart Uelen. You can see the presentation and demo of the hack here.

A good write-up of all the hacks and winners can be found on the TADHack blog.

Thanks to the TADHack organisers for another fine event, and also to everyone who came over to chat about Matrix - and again congratulations to the Babelonio team:

It's soon time for the 2nd TADHack-mini London. The event starts at 10am on Saturday April 9th and hacking continues until the projects are pitched, starting at 1pm on Sunday April 10th. As you can see by the many previous TADHacks, every hackathon brings interesting and impressive projects, so we are again expecting great things!

As usual, there are great prizes to be won - worth around $5k in total. This time, we will award the best Matrix-related hack a PhantomX AX Metal Hexapod Mark III from Trossen Robotics, a build-it-yourself hexapod robot kit! The robot is built on an entirely open source platform, complete with 3D cad models of the robot, open software, and schematics for the electronics.

If you're planning to attend TADHack-mini London: see you there! If not - why aren't you? Consider spending a day and a half hacking on some cool technologies - it could be well worth your time!

You can be one step ahead by getting acquainted with the Matrix C-S API or the AS API. And if you have any questions - or want to discuss potential hacks - please come talk to us in #matrix:matrix.org!

We just released Synapse 0.14.0 - a major update which incorporates lots of work on making Synapse more RAM efficient. There's still a lot of room for further improvements, but the main headlines are reducing the resident memory footprint dramatically by interning strings and deduplicating events across the many different caches. It also adds a much-needed SYNAPSE_CACHE_FACTOR environment variable that can be used to globally decrease or increase the sizing of all of Synapse's various caches (with an associated slow-down or speed-up in performance). Quite how improved the new memory footprint seems to very much depend on your own use case, but it's certainly a step in the right direction.

For more details on recent Synapse performance work (and a general state of the union for the whole Matrix ecosystem), check out our Spring update.

Get all new synapse from https://github.com/matrix-org/synapse - we recommend upgrading (or installing!) asap :)

Full changelog follows:

🔗Changes in synapse v0.14.0 (2016-03-30)

No changes from v0.14.0-rc2

🔗Changes in synapse v0.14.0-rc2 (2016-03-23)

Features:

• Add published room list API (PR #657)

Changes:

• Change various caches to consume less memory (PR #656, #658, #660, #662, #663, #665)
• Allow rooms to be published without requiring an alias (PR #664)
• Intern common strings in caches to reduce memory footprint (#666)

Bug fixes:

• Fix reject invites over federation (PR #646)
• Fix bug where registration was not idempotent (PR #649)
• Update aliases event after deleting aliases (PR #652)
• Fix unread notification count, which was sometimes wrong (PR #661)

🔗Changes in synapse v0.14.0-rc1 (2016-03-14)

Features:

• Add event_id to response to state event PUT (PR #581)
• Allow guest users access to messages in rooms they have joined (PR #587)
• Add config for what state is included in a room invite (PR #598)
• Send the inviter's member event in room invite state (PR #607)
• Add error codes for malformed/bad JSON in /login (PR #608)
• Add support for changing the actions for default rules (PR #609)
• Add environment variable SYNAPSE_CACHE_FACTOR, default it to 0.1 (PR #612)
• Add ability for alias creators to delete aliases (PR #614)
• Add profile information to invites (PR #624)

Changes:

• Enforce user_id exclusivity for AS registrations (PR #572)
• Make adding push rules idempotent (PR #587)
• Improve presence performance (PR #582, #586)
• Change presence semantics for last_active_ago (PR #582, #586)
• Don't allow m.room.create to be changed (PR #596)
• Add 800x600 to default list of valid thumbnail sizes (PR #616)
• Always include kicks and bans in full /sync (PR #625)
• Send history visibility on boundary changes (PR #626)
• Register endpoint now returns a refresh_token (PR #637)

Bug fixes:

• Fix bug where we returned incorrect state in /sync (PR #573)
• Always return a JSON object from push rule API (PR #606)
• Fix bug where registering without a user id sometimes failed (PR #610)
• Report size of ExpiringCache in cache size metrics (PR #611)
• Fix rejection of invites to empty rooms (PR #615)
• Fix usage of bcrypt to not use checkpw (PR #619)
• Pin pysaml2 dependency (PR #634)
• Fix bug in /sync where timeline order was incorrect for backfilled events (PR #635)

It's been 3 months since the Matrix Holiday Special and once again we've all been too busy writing code to put anything that detailed on the blog. So without further a do here's a quick overview of how things have progressed so far in 2016!

🔗Home servers

🔗Synapse

Work on Synapse (our reference homeserver) has been primarily focused on improving performance. This may sound boring, but there's been a huge amount of improvement here since synapse 0.12 was released on Jan 4. Synapse 0.13 on Feb 10 brought huge CPU savings thanks to a whole fleet of caching and other optimisation work - the best way of seeing the difference here is to look at the load graph of the server that hosts matrix.org's synapse+postgres over the last few months:

Ignoring the unrelated blip during March, you can see an enormous step change in system load (which had a matching decrease in actual CPU usage) at the beginning of Feb when the 0.13 optimisations landed on matrix.org :)

Meanwhile, Synapse 0.14 is due any day now with 0.14.0-rc2 released on Wednesday. Here, the focus has been all about memory optimisation - anyone who's run a Synapse seriously will be aware that it can be a memory hog thanks to aggressively caching as much state and history in RAM as possible to avoid hitting the database and keeping everything responsive. 0.14 should improve memory usage just as dramatically as 0.13 improved CPU utilisation - introducing a quick-and-dirty SYNAPSE_CACHE_FACTOR environment variable that lets admins dial down the aggressiveness of the caching (at the expense of performance), but more interestingly implementing string interning and ensuring that events are cached by ID rather than duplicated across multiple caches in order to make memory usage more efficient. It's too early to have impressive looking graphs, and there are still a few memory spikes being tracked down before we release 0.14, but we're hoping for at least a 50% reduction in memory footprint.

Featurewise the highlights include: server-generated unread notification & highlight counts and push badge support, lots of support and refinements for guest access and 3rd party ID invites. Meanwhile we've finally fixed some of the most embarrassing long-standing missing features such as letting folks logout serverside(!), delete aliases and determine whether rooms should be published in the room directory or not.

Finally, Synapse is now part of FreeBSD Ports thanks to Brendan Molloy, and NixOS thanks to Robin Lambertz! Huge thanks to them for contributing the packages to the respective OSes and to all the other synapse package maintainers out there!

It's incredibly exciting to see Synapse's maturity improving and hitting the optimisation stage of its life; huge kudos to Erik for spearheading the optimisation work. We strongly recommend folks upgrade to 0.14 when it's available; it's never been a better time to run a homeserver! :D

🔗Dendron

Meanwhile, Dendron (our next generation homeserver) development has been progressing interestingly: we finished an initial spike to get a Golang skeleton server in place, albeit one that delegates most of the endpoints through to Synapse. In fact, matrix.org itself has been running via Dendron since February!

The whole point of Dendron is to provide an architecture where we can split apart the various endpoints that Synapse provides today, re-implementing them where appropriate in Golang, and critically letting the endpoints scale horizontally with clusters of backend servers abstracted by the single Dendron API facade. As a result, most of the Dendron work has actually ended up going into restructuring Synapse such that multiple Synapses can be run in a cluster behind a single Dendron, allowing us to horizontally scale API endpoints at last. This takes the form of adding cluster replication support to Synapse. This is still work-in-progress as we go through fixing up more and more state to be replicable (replicatable?) between synapses - hopefully it should land in the Synapse 0.15 timeframe. And then we enter a very very interesting new world of horizontally scalable homeservers...

🔗Ruma

Ruma has also seen some progress over the last few months - Ruma is an independent Rust language homeserver project led by Jimmy Cuadra, and whilst in early development still (currently focusing on the user login and registration system) shows a lot of promise. Lots of work has ended up going into the required Rust dependencies rather than the Matrix code itself, but if you're interested in Rust then please drop by #ruma:matrix.org or #ruma on Freenode and say hi!

🔗Clients

Whilst homeserver development is mainly all about performance and scaling work currently, the client side of the Matrix ecosystem is the polar opposite - with lots of rapid progress on exciting new clients happening from all over the community.

🔗Perpetually Talking Online (PTO)

PTO has evolved enormously since Torrie Fischer first revealed it at the end of 2015. PTO is an independent project that acts as a Matrix client that exposes an IRC server interface - effectively turning any Matrix homeserver into an ircd; letting folks hook their favourite IRC clients directly into Matrix and use it as an enormous decentralised IRC network. (N.B. this is not to be confused with matrix-appservice-irc, which acts as a server-side bridge between Matrix rooms and IRC channels.) Obviously you lose some of the Matrix specific features (read receipts, typing notifs, VoIP, etc) but there's clearly a huge benefit for the IRC community to be able to use Matrix as if it were an IRC network.

There have been three releases so far, with the v0.3.0 ("Carburetor") release in March being tantalisingly close to being usable for everyday purposes. We actually have pto.matrix.org all set up and ready to go as an IRC frontend for the matrix.org homeserver and once issue #60 is resolved we'll be turning it on :)

There's one catch though - XChat was never quite built to handle the hundreds of rooms that we've got used to Matrix supporting... :D

Come hang out in #pto:oob.systems if you're interested in PTO!

🔗Quaternion

Quaternion is a new Qt/QML/C++ desktop client created by Felix Rohrbach. It's a fairly early alpha but still quite usable and in very active development. #quaternion:matrix.org is the place to talk all things Quaternion :)

🔗matrix-glib-sdk

Meanwhile, over on the GTK side of the world, Gergely Polonkai has been been making great progress on his matrix-glib-sdk Glib client SDK for Matrix. The end goal here is to implement a full Telepathy plugin for Matrix on top of the SDK. Originally written in C, but now shifted to Vala, the SDK is in very active development and now implements all(?) of the Matrix client-server API - a snapshot of the work-in-progress SDK API docs can be found at http://gergely.polonkai.eu/matrix-glib-sdk. Next up is a formal release and building out clients on top!

🔗matrix-react-sdk, matrix-ios-sdk, matrix-android-sdk and Vector

Finally, huge amounts of time and effort have continued to be pumped into the official matrix-react-sdk, matrix-ios-sdk and matrix-android-sdk - driven substantially by requirements for Vector, the FOSS Matrix-powered collaboration app that we've been helping with:

The best way of seeing what's been going on here is probably by considering Vector itself, which is currently in formal beta (0.4.1 for web, 0.1.2 for iOS and #116 on Android). The big news includes:

• Beta iOS and Android apps. These are early beta but feedback is very much appreciated - the Android beta can be downloaded from Jenkins; if you want to help beta iOS via TestFlight, come ask on #ios:matrix.org.
• Guest access. Anyone can jump straight into Matrix by going to http://vector.im without even having to sign up for an account. Guests are quite restricted on what they can do (and can only join rooms which explicitly have guest access enabled), but this is a *huge* improvement in getting folks using Matrix.
• Ability to jump to any message ever - e.g. when clicking through search results or when permalinking a message... using precisely the same UI that you use when chatting. Permalinks are awesome. If you want to randomly jump back in time to the first weeks of #matrix:matrix.org, now you can...
• Read Markers, scrolling that remembers the scroll offset per-room, and the ability to jump to unread messages
• Synchronised missed notification and missed highlighted notification information per-room
• Badge counts for unread notifications
• Entirely reworked Room Settings
• Entirely reworked User Settings, including push notification configuration
• Entirely reworked Room Directory
• Lots of performance improvements
• Much improved inviting by email
• Much improved reliability on video conferencing
• Closing literally hundreds and hundreds of bugs...

All that remains right now is yet more bugfixing and incorporating feedback from the current betas! Please give as much feedback as possible in #vector:matrix.org :)

🔗Bridges & Bots

Bridges, bots, and other integrations and application services have inevitably taken slightly lower priority whilst we've been focusing on the core server and client bits of the ecosystem. However, as of March we've started a major new project to get these moving again, starting with a big update to the IRC Bridge. This is due to be released next week, but you can get a sneak peek at what's going into the release at the commit log. Highlights include the ability to persist nicks; connect via IPv6; improve formatted message handling; actually feed error messages from IRC back to Matrix; and much much more.

matrix-appservice-verto also got some love, which means that multiway video conferencing powered by FreeSWITCH now works reliably. The quality still could be improved, but the unreliable call setup that plagued earlier versions is now fixed.

In the next few months we're expecting to see a lot more activity on bridges & bots... watch this space :)

Update Sat March 26:

Totally forgot to mention a few of the key new bridges which have been contributed by the community this year - particularly interesting are the Rocket.Chat<->Matrix bridge written by Sing-Li over at Rocket.Chat which provides basic bridging between the awesome Rocket.Chat collaboration app and the wider Matrix ecosystem. It's early days, but this is incredibly promising for 'hardcoded' bridging between specific rooms - it just needs Rocket.Chat to support 'virtual' users and will then be seamless federation.

Similarly, matrix-appservice-gitter is a Gitter<->Matrix bridge built by Leonerd on top of the matrix-appservice-bridge Node library. Again, it's early days but is working well for 'hardcoded' bridging - supporting dynamic rooms and users is next on the todo list :)

🔗The Spec

We started our formal release process for the spec just before Christmas with r0.0.0 - and released r0.0.1 in January with minor clarifications and updates. In practice the spec feels quite stable right now, although things have moved on a bit since January and r0.0.2 is definitely overdue at this point.

In the meantime, you can always get the very latest bleeding edge copy of the spec via the speculator. We've also added an initial cut at a spec for the Identity Service at last.

🔗Events

We've been focusing on writing code than evangelising Matrix recently, although we did get out to FOSDEM 2016 and TADHack Mini Japan and WebRTC Conference and Enterprise Connect 2016 where we showed off Matrix & Vector in the WebRTC Real World Innovation showcase.

🔗GSoC

We are incredibly grateful to have been accepted as an organisation into Google Summer of Code 2016! The last two weeks have been the window for students to propose projects to us that they could work on over the course of the summer, and it's been fascinating to meet the GSoCers and see a whole new community pop up on Matrix and advise and mentor applicants through their proposals. At the last count we've received 35 proposals, many inspired by our list of ideas, including some really impressive candidates - many thanks to all the students who have applied to us. We don't know yet how many slots Google will allocate to us, but one way or another we're really looking forward to helping the GSoCers make the most out of their summer of Matrix! All GSoC discussion is happening in #gsoc:matrix.org.

🔗What's next?

In no particular order, the urgent stuff that still remains includes:

• Continuing to polish synapse and build out dendron-based clustering
• Building as many bridges, bots and other integrations as possible
• The matrix.to URL-handler service: having client-agnostic https://matrix.to/#matrix:matrix.org URLs to help with sharing matrix room aliases etc
• End-to-end crypto. No progress since December; we need to get back to it asap.
• Exiting Vector from beta
• Finishing the server-to-server API specification
• Improving the security model for access_tokens
• Editable messages
• Pinned, tagged, and 'liked' messages
• Threading
• Decentralised accounts
• Decentralised reputation

In practice, Bridging and E2E crypto is likely to get the most attention first (beyond the standard ongoing polishing). There's obviously a significant amount of work there, but we expect to see benefits pretty quickly throughout Matrix - especially from bridging. Hopefully it's true to say that the next few months should be quite transformational :D

Anyway, thanks for reading this sprawling update and for supporting Matrix. And please come say hi in #matrix:matrix.org if you have any questions :)

• Matthew, Amandine & the Matrix.org team.