Another season has passed; the leaves are dropping from the trees in the northern hemisphere (actually, in the time it's taken us to finish this post, most of them have dropped :-/) and once again the Matrix team has been hacking away too furiously to properly update the blog. So without further delay here's an update on all things Matrix!

Synapse 0.18

Back in September, we forgot to properly announce the 0.18 release of Synapse! This is a major oversight given that 0.18 was a huge update with some critical performance improvements, but hopefully everyone has upgraded by now anyway. If not, there's never been a better time to run your own homeserver! The main improvement is that the Matrix room state updates are now stored as deltas in the database rather than snapshots, which reduces the size of the database footprint by around 5 - 7x. The first time you run synapse after upgrading to 0.18 it will go through your database deleting all the historical data, after which you can VACUUM the db to reclaim the freed diskspace.

You can tell when it's finished based on whether it's stopped logging about the 'background_deduplicate_state' task. There was a bug in 0.18.0 that meant this process was very slow (weeks) on sqlite DBs and chewed 100% CPU; this was fixed in 0.18.1, and subsequently we've also had 0.18.2 (various perf and bug fixes, and a new modular internal API for authentication) and the current release: 0.18.3 to address a major vulnerability on deployments using LDAP with obsolete versions (0.9.x) of the python ldap3 library - e.g. Debian Stable. Folks using the Debian Stable packages must upgrade immediately.

Other big changes in Synapse 0.18 were:

• Adding the final APIs required to support end-to-end encryption: specifically, a new store-and-forward API called "to device messaging", which lets messages be passed between specific devices outside the context of a room or a room DAG. This is used for exchanging authentication tokens and sensitive end-to-end key data between devices (e.g. when a new device joins a room and needs to be looped in) and is not intended for general messaging.
• Changing how remote directory servers are queried. Rather than constantly spidering them via the secondary_directory_servers option (which was causing a load crisis on the matrix.org server, as everyone else in Matrix kept polling it for directory updates), clients can now set a 'server' parameter on the publicRooms request to ask their server to proxy the request through to a specific remote server. Element (the app formally known as Riot/Web) implements this already. This is a stopgap until we have a proper global room discovery database of some kind.
• Adding pagination support to the room directory API. We now have enough rooms in Matrix that downloading the full list every time the user searches for a room was getting completely untenable - we now support paginating and searching the list. Riot (now Element) and Riot-Android (now Element Android) are using the new APIs already.
• Basic support for 'direct room' semantics. When you create a room you can now state the intent for that room to be a 1:1 with someone via the is_direct parameter.
• Making the /notifications API work - this lets clients download a full list of all the notifications a user has been recently sent (highlights, mentions etc)

Spec for all of these new APIs are currently making their way into the official matrix spec; you can see the work in progress at https://matrix.org/speculator. Meanwhile, we're waiting for the last bits of the end-to-end encryption APIs to land there before releasing 0.3 of the Matrix spec, which should happen any day now.

To find out more and get upgraded if you haven't already, please check out the full changelog.

Synapse scalability

Something which we've been quietly adding over the last 6 months is support for running large synapse deployments like the Matrix.org homeserver. Matrix.org has around 500K accounts on it, 50k rooms, and relays around 500K messages per day and obviously the community expects it to have good performance and availability (even though we'd prefer if you ran your own server, for obvious reasons!)

The current scaling approach for this is called 'Workers' - where we've split out a whole bunch of different endpoints from the main Synapse process into child 'worker' processes which replicate their state from the master Synapse process. These workers are designed to scale horizontally, adding as many as you like to handle the traffic load. It's not full active/active horizontal scalability in that you're still limited by the performance of the master process and the database master you're writing to, but it's a great way to escape Python's global interpreter lock limiting processes effectively to a single core, and in practice it's a huge improvement and works pretty well as of Synapse 0.18.

You can read more about the architecture and how to run your Synapse in worker-mode over at https://github.com/matrix-org/synapse/blob/master/docs/workers.rst.

Starting a Riot (now Element)

Meanwhile, the biggest news in Matrixland has probably been the renaming of Vector as Riot (now Element) and the 'mass market' launch of Riot as a flagship Matrix client at the MoNage conference on Sept 19th in Boston. The reasons for renaming Vector have been done to death by now and hopefully folks have got over the shock, but the rationale is to have a more distinctive and memorable (and controversial!) name, which is more aligned with the idea of returning control of communication back to the people :) Amandine has the full story over at the Riot blog.

Riot (now Element) itself is a fairly thin layer on top of the official client Matrix SDKs, and so 95% of the work for Riot (now Element) took the form of updates to matrix-js-sdk, matrix-react-sdk, matrix-ios-sdk, matrix-ios-kit, matrix-android-sdk and synapse itself. There's been a tonne of changes here since June, but the main highlights are:

• End-to-end encryption support landed in matrix-js-sdk and matrix-react-sdk (and thus Riot/Web (now Element)) and in dev on matrix-ios-sdk and matrix-android-sdk using the Olm and Megolm ratchets. More about this later.
• Hosted integrations, bots and bridges! More about this later too.
• Direct Message UI landed in Riot/Web (now Element) to tag rooms which exist for contacting a specific user. These get grouped now as the 'People' list in Riot (now Element). It's in dev on Riot/iOS (now Element (iOS)) & Android (now Element Android).
• Entirely new UI for starting conversations with people - no more creating a room and then inviting; you just say "i want to talk with Bob".
• Entirely new UI for inviting people into a room - no more confusion between searching the membership list and inviting users.
• FilePanel UI in Riot/Web (now Element) to instantly view all the attachments posted in a room
• NotificationPanel UI in Riot/Web (now Element) to instantly view all your missed notifications and mentions in a single place
• "Volume control" UI to have finer grained control over per-room notification noisiness
• Entirely re-worked Room Directory navigator - lazy-loading the directory from the server, and selecting rooms via bridge and remote server
• It's very exciting to see a wider audience discovering Matrix through Riot (now Element) - and Riot (now Element)'s usage stats have been growing steadily since launch, but there's still a lot of room for improvement.

Stuff on the horizon includes:

• Formal beta-testing the full end-to-end encryption feature-set.
• Performance and optimisation work on all platforms - there are huge improvements to be had.
• Long-awaited poweruser features: 'dark' colour scheme; more whitespace-efficient layout; collapsing consecutive joins/parts...
• "Landing page" to help explain what's going on to new users and to show deployment-specific announcements and room lists.
• Support for arbitrary profile information.
• Threading.

Riot (now Element) releases are announced on #riot:matrix.org, the Riot blog and Twitter - keep your eyes peeled for updates!

End to End Encryption

Full cross-platform end-to-end encryption is incredibly close now, with the develop branches of iOS & Android SDKs and Riot (now Element) currently in internal testing as of Nov 7 - expect a Big Announcement very shortly.  We're very optimistic based on how the initial implementation on Riot/Web (now Element)has been behaving so far.

When E2E first landed on Riot/Web (now Element) in September we were missing mobile support, encrypted attachments, encrypted VoIP signalling, and the ability to retrieve encrypted history on new devices - as well as a formal audit of the underlying Olm and Megolm libraries. Since then things have progressed enormously with most of the core team working since September on filling in the gaps, as well as getting audited and fixing all the weird and wonderful edge cases that the audit showed up. All the missing stuff has been landing on the develop branches over the last few weeks, with encrypted attachments landing on web on Nov 10; encrypted VoIP landing on Nov 11; etc. Watch this space for news on the upcoming cross-platform public beta!

Hosted Integrations and introducing go-neb

One of the new features which arrived in Riot (now Element) is the ability to add "single click" integrations (i.e. bots, bridges, application services) into rooms from Riot/Web (now Element) by clicking the "Manage Integrations" button in Room Settings. These integrations are hosted for free by Riot (now Element) in its production infrastructure (codenamed Scalar), but all the actual bots/bridges/services themselves are normal opensource Matrix apps and you can of course run them yourself too.

The Bot integrations are all provided by go-neb - a complete rewrite in Golang and general reimagining of the old python NEB bot which old-timers will recall as the very first bot written for the Matrix ecosystem. Go-neb has effectively now become a general purpose golang bot/integration framework for Matrix, with the various different services implemented as plugins for Github, JIRA, Giphy, Guggy etc. Critically it supports authenticating Matrix users through to the remote service, letting normal Matrix users interact with Github and friends using their actual Github identity rather than via a bot user - this is a huge huge improvement over the original naive python NEB.

If you like Go and you like Matrix, we'd strongly suggest having a go (hah) at adding new services into go-neb: anything implemented against go-neb will also magically be hosted and available as part of the "Manage Integrations" interface in Riot (now Element), as well as being available to anyone else running their own go-nebs. For full details of the architecture and how to implement new plugins, go check out the full README.

If Matrix is to provide a good FOSS alternative to systems like Slack it's critical to have a large array of available integrations, so we really hope that the community will help us grow the list!

Building Bridges

There have been vast improvements to bridging over the last few months, including the ability to "plumb" bridges into arbitrary rooms (letting you link a single Matrix room through to multiple remote networks). Like go-neb, Riot (now Element) is providing free bridge hosting with the ability to add to rooms with a "single click" via the Manage Integrations button in Room Settings. For now, Riot (now Element) is hosting any bridges built on the matrix-appservice-bridge codebase.

In short, this means that any user can go and take an existing Matrix room and link it through to Slack, IRC, Gitter, and more.

matrix-appservice-irc

Huge amounts of work have gone into improving the IRC bridge - both adding new features to try to give the most IRC-friendly experience when bridging into IRC, as well as lots of maintenance and performance work to ensure that the matrix.org hosted bridges can scale to the large amounts of traffic we're seeing going through Freenode and others. We've also added hosted bridges for OFTC and Snoonet, and turned on connecting via IPv6 by default for networks which support it.

You can read the full changelogs for 0.5.0 and 0.6.0 at https://github.com/matrix-org/matrix-appservice-irc/blob/master/CHANGELOG.md, but the main highlights are:

matrix-appservice-irc 0.6.0

• Debouncing quits and netsplits: if IRC users quit there can be a window where they are shown as just offline rather than leaving the room, avoiding join/part spam and creating unnecessary state events in Matrix.
• Topic bridging: IRC topics can now be bridged to Matrix!
• Support custom SSL CAs (thanks to @Waldteufel)
• Support custom media repository URLs
• Support the ability to quit your IRC user from the network entirely
• Fix rate limiting for traffic from privileged IRC users and services

matrix-appservice-irc 0.5.0:

• Support throttling reconnections to IRC servers to avoid triggering abuse thresholds
• Support "Third party lookup": mapping from IRC users & rooms into Matrix IDs for discovery purposes
• Support rate-limiting membership entries to avoid triggering abuse thresholds
• Require permission of an IRC chanop when plumbing an IRC channel into a Matrix room
• Prevent routing loops by advisory m.room.bridging events
• Better error messages
• Sync chanmode +s correctly
• Fix IPv6 support

Next up is automating NickServ login, and generally continuing to make the IRC experience as good as we possibly can.

matrix-appservice-slack

Similarly, the Slack bridge has had loads of work. The main changes include:

• Ability to dynamically bridge ("plumb") rooms on request
• Add Prometheus monitoring metrics
• Ability to discover slack team tokens via OAuth2
• Sync avatars both ways

We're currently looking at shifting over to Slack's RTM (Real Time Messaging) API rather than using webhooks in order to get an even better fit with Slack and support bridging DMs, but the current setup is still very usable. For more details: https://github.com/matrix-org/matrix-appservice-slack.

matrix-appservice-gitter

The Gitter bridge has provided a lot of inspiration for the more recent work on the Slack bridge. Right now it provides straightforward bridging into Gitter rooms, albeit proxied via a 'matrixbot' user on the Gitter side. We're currently looking at letting also users authenticate using their Gitter credentials so they are bridged through to their 'real' Gitter user - watch this space. For more details: https://github.com/matrix-org/matrix-appservice-gitter.

Community updates

matrix-ircd

matrix-ircd is a rewrite of the old PTO project (pto.im): a Rust application that turns Matrix into a single great big decentralised IRC network. PTO itself has unfortunately been on hiatus and is rather bitrotted, so Erik from the core Matrix Team picked it up to see if it could be resurrected. This ended up turning into a complete rewrite (switching from mio to tokio etc), and the new project can be found at https://github.com/matrix-org/matrix-ircd.

matrix-ircd really is an incredibly promising way of getting folks onto Matrix, as it exposes the entirety of Matrix as a virtual IRC network. This means that IRC addicts can jack straight into Matrix, talking native IRC from their existing IRC clients - but interacting directly with Matrix rooms as if they were IRC channesls without going through a bridge. Obviously you lose all of the features and semantics which Matrix provides beyond IRC, but this is still a great way to get started.

The project is currently alpha but provides a good functioning base to extend, and Erik's explicitly asking for help from the Rust and Matrix community to fill in all the missing features. If you're interested in helping, please come talk on #matrix-ircd:matrix.org!.

matrix-appservice-gitter-twisted

Not to be confused with the Node-based matrix-appservice-gitter, matrix-appservice-gitter-twisted is an entirely separate project written in Python/Twisted by Remram (Remi Rampin) that has the opposite architecture: rather than bridging existing rooms into Matrix, matrix-appservice-gitter-twisted lets you provide your Gitter credentials and acts instead as a Gitter client, bridging your personal view of a Gitter room into a private Matrix room just for you.

This obviously has some major advantages (your actions on Gitter use your real Gitter account rather than a bot), and some disadvantages too (you can't use Matrix features when interacting with other Matrix users in the same room, and the Gitter channel itself is not decentralised into Matrix). However, it's a really cool example of how the other model can work - and within the core team, we've been arguing back and forth for ages now on whether normal bridges or "sidecar" bridges like this one are a more preferable architecture. Thanks to Remram's work we can try both side by side! Go check it out at https://github.com/remram44/matrix-appservice-gitter-twisted.

telematrix

Telematrix is Telegram<->Matrix bridge, written by Sijmen Schoon using python3 and asyncio. Right now it's a fairly early alpha hardcoded to bridge a specific Telegram channel into a specific Matrix room, but it works and in use and could be an excellent base for folks interested in a more comprehensive Matrix/Telegram bridge. Go check it out at https://github.com/SijmenSchoon/telematrix

Ruma

Meanwhile, the Ruma project to write a Matrix homeserver in Rust has been progressing steadily, with more and more checkboxes appearing on the status page, with significant new contributions from mujx and farodin91. The best way to keep track of Ruma is to read Jimmy's excellent This Week in Ruma updates and of course hang out on #ruma:matrix.org.

NaChat

An entirely new client on the block since the last update is NaChat, written by Ralith. NaChat is a pure cross-platform Qt/C++ desktop client written from the ground up, supporting local history synchronisation, excellent performance, native Qt theming, and generally being a lean and mean Matrix client machine. It's still alpha, but it's easy to build and a lot of fun to play with.

Please give a spin, encourage Ralith to finish the timeline-view-rewrite branch (which is probably the one you want to be running!), and come hang out on #nachat:matrix.org.

Quaternion

Meanwhile, the Quaternion Qt/QML desktop client and its libqmatrixclient library has been making sure and steady progress, with fxrh, kitsune, maralorn and others working away at it. The difference with NaChat here is using QML rather than native Qt widgets, and a focus on more advanced UX features like a custom infinite-scrolling scrollbar widget, unread message notifications, and read-up-to markers.  Recent developments include the first official release (0.0.1) on Sept 12, official Windows builds, lots of work on implementing better Read-up-to Markers, scrolling behaviour etc. Again, it's worth keeping a checkout of Quaternion handy and playing with the client - it's loads of fun!

Google Summer of Code 2016 Retrospective

The summer is long gone now, and along with it Google Summer of Code. This was the first year we've participated in GSoC, and it was an incredible experience - both judging all the applications, and then working with Aviral Dasgupta and Will Hunt (Half-Shot) who joined the core team as part of their GSoC endeavours.

Aviral's work has been widespread throughout Riot (now Element): adding consistent Emoji support throughout the app via Emoji One, implementing the beta Rich Text Editor (RTE) and all-new autocompletion UI, as well as a bunch of spec proposals for rich message semantics and an initial Slack Webhooks application service. You can read his wrap up at http://www.aviraldg.com/p/gsoc-2016-wrapup and use the code in Riot/Web (now Element) today. We're currently working on fixing the final issues on RTE and auto-complete and hope to enable them by default real soon now!

Meanwhile, Half-Shot's work ended up focusing on bridging through to Twitter and working on the Threading spec proposal for Matrix. You can find out all about the Twitter bridge at https://half-shot.github.io/matrix-appservice-twitter; it works incredibly well (arguably too well, given the amount of traffic it can bridge into Matrix! :S) - and we're currently working on hosting a version of it on matrix.org for all your tweeting needs. You can see Half-Shot's wrapup blog post over at https://half-shot.uk/gsoc16_evaulation.

Finally, as a bit of a wildcard, we discovered the other day that there was also another GSoC project using Matrix by Waqee Khalid, supported by the Berkman Center for Internet and Society at Harvard to switch Apache Wave (formerly Google Wave) over to using Matrix rather than XMPP for federation!  The implementation looks a little curious here, as Wave used XMPP as a blunt pubsub layer for synchronising protobuf deltas - and it looks like this implementation uses Matrix similarly, thus killing any interop with the rest of Matrix, which is a bit of a shame.  If anyone knows more about the project we'd love to hear though!

Either way, it's been a pleasure to work with the GSoC community and we owe Aviral and Half-Shot (and Waqee!) a huge debt of gratitude for spending their summers (and more!) hacking away improving Matrix. So, thanks Google for making GSoC possible and thanks to the GSoCers for all their contributions, effort & enthusiasm! Watch this space for updates on RTE, new-autocomplete and the twitter bridge going live...

Matrix in the news

Just in case you missed them, there have been a couple of high profile articles flying around about Matrix recently - we made the front cover of Linux Magazine in August with a comprehensive review of Matrix and Vector (now Riot (now Element)). Then when we launched Riot (now Element) itself we got a cautiously positive write-up from Mike Butcher at Techcrunch. We also wrote an guest column for Techcrunch about the importance of bringing power back to the people via decentralisation, which got a surprising amount of attention on HackerNews and elsewhere.

More recently, we were lucky enough to get an indepth video interview with Bryan Lunduke as part of his 'Linux & Whatnot' series, and also a write-up in NetworkWorld alongside Signal & Wire as part of Bryan's journeys in the land of encrypted messaging.

Huge thanks to everyone who's been nice enough to spread the word of Matrix!

Matrix In Real Life

Finally, we've been present at a slew of different events. In August we attended FOSSCON again in Philadelphia to give a general update on Matrix to the Freenode community...

It's @fossconNE time! Dave will be talking about Matrix at 1pm today. Come & say hi! pic.twitter.com/KtfTVRnAVn

— Matrix (@matrixdotorg) August 20, 2016

...and then Riot (now Element) was launched at Monage in Boston in September, with Matthew and Amandine respectively presenting Matrix and Riot (now Element):

Best #swag of #MoNage? The @RiotChat stand is getting mobbed :) pic.twitter.com/NltlfO74Y9

— Oisin Lunny (@oisinlunny) September 20, 2016

Whilst quite a small event, the quality of folks present was incredibly high - much fun was had comparing open communities to walled gardens with Nicola Greco from Tim Berners-Lee's Solid project...

.@ara4n & @AmandineLePape showing off the new Riot (now Element) at #MoNage in #Boston! pic.twitter.com/U0qSNjNLGs

— Riot (@RiotChat) September 21, 2016

...comparing notes with the founders of ICQ, hanging out with Alan from Wire...

A meeting of the messaging minds! @Wire and @matrixdotorg federating over a pint at #MoNage. pic.twitter.com/uTUvWrKRqp

— Oisin Lunny (@oisinlunny) September 21, 2016

...chatting to FireChat's CTO, catching up with Dan York from the Internet Society, etc.

Then in October we spoke about scaling Python/Twisted for Matrix at PyCon France in Rennes - this was really fun, albeit slightly embarrassing to be the only talk about Python/Twisted in a track otherwise entirely about Python 3 and asyncio :D That said, the talk seemed to be well received and it was fantastic to meet some of the enthusiastic French Python community and see folks in the audience who were already up and running on Matrix!

Lots of fun at @pyconfr today demoing Matrix, including a quick video conference with the audience & #TADHack London! pic.twitter.com/rwbA43X7wB

— Matrix (@matrixdotorg) October 15, 2016

The same weekend also featured TADHack Global - we were present at the London site; you can read all about it in our earlier blog post. There was a really high standard of hacks on Matrix this year, and it was incredibly hard to judge the hackathon. In most ways this is a good problem to have though!

Dramatic prep for @maffydub and yinyee's Matrix IoT demo with multiple ESP8266, proximity sensor, and @tropo ASR!! pic.twitter.com/eytG8QWFq6

— Matrix (@matrixdotorg) October 16, 2016

Meanwhile, coming up on the horizon we have TADSummit in Lisbon next week, where we'll be giving an update on Matrix to the global Telco Application Developer community, and then the week after we'll be in Israel as part of the Geektime Techfest, Devfest and Chatbot Summit. So if you're in Lisbon or Tel Aviv do give us a ping on Matrix and come hang out!

Matrixing for fun and profit!

If you've read this far, we're guessing you're hopefully quite interested in Matrix (or just skipping to the end ;).  Something we don't talk about as much as we should is that if you're interested in being paid to work on Matrix full time, we're always interested in expanding the core team.  Right now we're particularly looking for:

• Experienced front-end developers who can help build the next generation of matrix-react-sdk and vector-web
• Professional tech-writers to help keep The Spec and tutorials and other docs updated and as kick-ass as possible
• Backend Python/Twisted or Golang wizards to help us improve and evolve Synapse
• Mobile developers (especially Android) to help keep the mobile SDKs and apps evolving as quickly as possible
• Integration fiends who'd like to be paid to build more bridges, bots and other integrations for the overall ecosystem!

Most of the core team hangs out in London or Rennes (France), but we're also open to remote folks where it makes sense.  If this sounds interesting, please shoot us a mail to [email protected].  Obviously it helps enormously if we already know you from the Matrix community, and you have a proven FOSS track record.

Conclusion

Apologies once again for an overdue and overlong update, but hopefully this gives a good taste of how Matrix is progressing. Just to give a different datapoint: this graph is quite interesting - showing the volume of events per day sent by native (i.e. non-bridged) Matrix users visible to the matrix.org homeserver since we turned the service on back in 2014:

As you can see, things are accelerating quite nicely - and this is ignoring all the traffic in the rest of the Matrix ecosystem that happens not to be federated onto the matrix.org HS, not to mention the huge amounts of traffic due to bridging.

Our plans over the next few months are going to involve:

• Turning on end-to-end encryption by default for any rooms with private history - whilst ensuring it's as easy to write Matrix clients, bots and bridges as it ever was.
• Yet more scalability and performance work across the board, to ensure Synapse and the client SDKs can handle the growth curve we're seeing here
• Releasing 0.3.0 of the matrix spec itself.
• Making Riot (now Element)'s UX excellent.
• Editable messages.
• Threading.
• User groups, for applying permissions/invites etc to groups of users as well as individuals.
• Formalising the federation spec at last
• As many bots, bridges and other integrations as possible!
• Making VoIP/Video conferencing and calling awesome.
• More experiments with next-generation homeservers
• Starting to really think hard about decentralised identity and reputation/spam management
• ...and a few new things we don't want to talk about yet ;)

If you've got this far - congratulations! Thanks for reading, and thank you for supporting the Matrix ecosystem.

Now more than ever before we believe that it is absolutely critical to have a healthy and secure decentralised communications ecosystem on the 'net (whether that's Matrix, XMPP, Tox or whatever) - so thank you again for participating in our one :)  And if you don't already run your server, please grab a Synapse and have fun!

• Matthew, Amandine & the Matrix Team.

Hi folks - another few months have gone by and once again the core Matrix team has ended up too busy hacking away on the final missing pieces of the Matrix jigsaw puzzle to have been properly updating the blog; sorry about this. The end is in sight for the current crunch however, and we expect to return to regular blog updates shortly! Meanwhile, rather than letting news stack up any further, here's a quick(?) attempt to summarise all the things which have been going on!

Synapse 0.16.1 released!

This one's a biggy: in the mad rush during June to support the public debut for Vector, we made a series of major Synapse releases which apparently we forgot to tell anyone about (sorry!). The full changelog is at the bottom of the post as it's huge, but the big features are:

• Huge performance improvements, including adding write-thru event caches and improving caching throughout, and massive improvements to the speed of the room directory API.
• Add support for inline URL previewing!
• Add email notifications!
• Add support for LDAP authentication (thanks to Christoph Witzany)
• Add support for JWT authentication (thanks to Niklas Riekenbrauck)
• Add basic server-side ignore functionality and abuse reporting API
• Add ability to delegate /publicRooms API requests to a list of secondary homeservers
• Lots and lots and lots of bug fixes.

If you haven't upgraded, please do asap from https://github.com/matrix-org/synapse!

There's also been a huge amount of work going on behind the scenes on horizontal scalability for Synapse.  We haven't drawn much attention to this yet (or documented it) as it's still quite experimental and in flux, but the main change is to add the concept of application-layer replication to Synapse - letting you split the codebase into multiple endpoints which can then be run in parallel, each replicating their state off the master synapse process.  For instance, right now the Matrix.org homeserver is actually running off three different processes: the main synapse; another specific to calculating push notifications and another specific to serving up the /sync endpoint.  These three are then abstracted behind the dendron layer (which also implements the /login endpoint). The idea is that one can then run multiple instances of the /sync and pusher (and other future) endpoints to horizontally scale.  For now, they share a single database writer, but in practice this has improved our scalability and performance on the Matrix.org HS radically.

In future we'll actually document how to run these, as well as making it easy to spin up multiple concurrent instances - in the interim if you find you're hitting performance limits running high-traffic synapses come talk to us about it on #matrix-dev:matrix.org.  And the longer term plan continues to be to switch out these python endpoint implementations in future for more efficient implementations.  For instance, there's a golang implementation of the media repository currently in development which could run as another endpoint cluster.

Vector released!

Much has been written about this elsewhere, but Web, iOS and Android versions of the Vector clients were finally released to the general public on June 9th at the Decentralised Web Summit in San Francisco.  Vector is a relatively thin layer on top of the matrix-react-sdk, matrix-ios-sdk and matrix-android-sdk Matrix.org client SDKs which showcases Matrix's collaboration and messaging capabilities in a mass-market usable app.  There's been huge amounts of work here across the SDKs for the 3 platforms, with literally thousands of issues resolved.  You can find the full SDK changelogs on github for React, iOS and Android.  Some of the more interesting recent additions to Vector include improved room notifications, URL previews, configurable email notifications, and huge amounts of performance stability work.

Future work on Vector is focused on showcasing end-to-end encryption, providing a one-click interface for adding bots/integrations & bridges to a room, and generally enormously improving the UX and polish.  Meanwhile, there's an F-Droid release for Android landing any day now.

If you haven't checked it out recently, it's really worth a look :)

Matrix Spec 0.1.0

In case you didn't notice, we also released v0.1.0 of the Matrix spec itself in May - this is a fairly minor update which improves the layout of the document somewhat (thanks to a PR from Jimmy Cuadra) and a some bugfixes.  You can see the full changelog here. We're overdue a new release since then (albeit again with relatively minor changes).

Google Summer of Code

We're in the middle of the second half of GSoC right now, with our GSoC students Aviral and Half-Shot hacking away on Vector and Microblogging projects respectively.  There's a lot of exciting stuff coming out of this - Aviral contributing Rich Text Editing, Emoji autocompletion, DuckDuckGo and other features into Vector (currently on branches, but will be released soon) and Half-Shot building a Twitter bridge as part of his Matrix-powered microblogging system.  Watch this space for updates!

Ruma

Lots of exciting stuff has been happening recently over at Ruma.io - an independent Matrix homeserver implementation written in Rust.  Over the last few weeks Jimmy and friends have got into the real meat of implementing events and the core of the Matrix CS API, and as of the time of writing they're the topmost link on HackerNews!  There's a lot of work involved in writing a homeserver, but Ruma is looking incredibly promising and the feedback from their team has been incredibly helpful in keeping us honest on the Matrix spec and ensuring that it's fit for purpose for 3rd party server implementers.

Also, Ruma just released some truly excellent documentation as a high-level introduction to Matrix (thanks to Leah and Jimmy) - much better than anything we have on the official Matrix.org site.  Go check it out if you haven't already!

End to End Encryption

There has been LOADS of work happening on End to End encryption: finalising the core 1:1 "Olm" cryptographic ratchet; implementing the group "Megolm" ratchet (which shares a single ratchet over all the participants of a room for scalability); fully hooking Olm into matrix-js-sdk and Vector-web at last, and preparing for a formal and published-to-the-public 3rd party security audit on Olm which will be happening during July.

This deserves a post in its own right, but the key thing to know is that Olm is almost ready - and indeed the work-in-progress E2E UX is even available on the develop branch of vector if you enable E2E in the new 'Labs' section in User Settings.  Olm itself is usable only for 'burn after reading' strictly PFS messages, but Megolm integration with Vector & Synapse will follow shortly afterwards which will finally provide the E2E nirvana we've all been waiting for :)

Decentralised Web Summit

Matrix had a major presence as a sponsor at the first ever Decentralised Web Summit hosted by the Internet Archive in San Francisco back in June.  This was a truly incredible event - with folks gathering from across the world to discuss, collaborate and debate on ensure that the web is not fragmented or trapped into proprietary silos - with the likes of Tim Berners-Lee, Vint Cerf and Brewster Kahle in attendance.  We ran a long 2 hour workshop on Matrix and showed off Vector to anyone and everyone - and meanwhile the organisers were kind enough to promote Matrix as the main decentralised chat interface for the conference itself (bridged with their Slack).  A full writeup of the conference really merits a blog post in its own right, but the punchline is that you could genuinely tell that this is the beginning of a new era of the internet - whether it's using Merkle DAGs (like Matrix) or Blockchain or similar technologies: we are about to see a major shift in the balance of power on the internet back towards its users.

We strongly recommend checking out the videos which have all been published at Decentralised Web Summit, including lightning talks introducing both Matrix and Vector, and digging into as many of the projects advertised as possible.  It was particularly interesting for us to get to know Tim Berners-Lee's latest project at MIT: Solid - which shares quite a lot of the same goals as Matrix, and subsequently seeing Tim pop up on Matrix via Vector.  We're really looking forward to working out how Matrix & Solid can complement each other in future.

Matrix.to

Not the most exciting thing ever, but heads up that there's a simple site up at https://matrix.to to provide a way of doing client-agnostic links to content in Matrix.  For instance, rather than linking specifically into an app like Vector, you can now say https://matrix.to/#/#matrix:matrix.org to go there via whatever app you choose.  This is basically a bootstrapping process towards having proper mx:// URLs in circulation, but given mx:// doesn't exist yet, https://matrix.to hopefully provides a useful step in the right direction :)

PRs very welcome at https://github.com/matrix-org/matrix.to.

Bridges and Bots

Much of the promise of Matrix is the ability to bridge through to other silos, and we've been gradually adding more and more bridging capabilities in.

For instance, the IRC bridge has had a complete overhaul to add in huge numbers of new features and finally deployed for Freenode a few weeks ago:

New Features:

• Nicks set via !nick will now be preserved across bridge restarts.
• EXPERIMENTAL: IRC clients created by the bridge can be assigned their own IPv6 address.
• The bridge will now send connection status information to real Matrix users via the admin room (the same room !nickcommands are issued).
• Added !help.
• The bridge will now fallback to body if the HTML content contains any unrecognised tags. This makes passing Markdown from Matrix to IRC much nicer.
• The bridge will now send more metrics to the statsd server, including the join/part rate to and from IRC.
• The config option matrixClients.displayName is now implemented.

Bug fixes:

• Escape HTML entities when sending from IRC to Matrix. This prevents munging occurring between IRC formatting and textual < element > references, whereby if you sent a tag and some colour codes from IRC it would not escape the tag and therefore send invalid HTML to Matrix.
• User IDs starting with - are temporarily filtered out from being bridged.
• Deterministically generate the configuration file.
• Recognise more IRC error codes as non-fatal to avoid IRC clients reconnecting unnecessarily.
• Add a 10 second timeout to join events injected via the MemberListSyncer to avoid HOL blocking.
• 'Frontier' Matrix users will be forcibly joined to IRC channels even if membership list syncing I->M is disabled. This ensures that there is always a Matrix user in the channel being bridged to avoid losing traffic.
• Cache the /initialSync request to avoid hitting this endpoint more than once, as it may be very slow.
• Indexes have been added to the NeDB .db files to improve lookup times.
• Do not recheck if the bridge bot should part the channel if a virtual user leaves the channel: we know it shouldn't.
• Refine what counts as a "request" for metrics, reducing the amount of double-counting as requests echo back from the remote side.
• Fixed a bug which caused users to be provisioned off their user_id even if they had a display name set.

Meanwhile, a Gitter bridge is in active development (and in testing with the Neovim community on Gitter/Matrix/Freenode), although lacking documentation so far.

Finally, NEB - the Matrix.org bot framework is currently being ported from Python to Golang to act as a general Go SDK for rapidly implementing new bot capabilities.

There's little point in all of the effort going into bridges and bots if it's too hard for normal users to deploy them, so on the Vector side of things there's an ongoing project to build a commercial-grade bot/bridge hosted service offering for Matrix which should make it much easier for non-sysadmins to quickly add their own bots and bridges into their rooms.  There's nothing to see yet, but we'll be yelling about it when it's ready!

Conclusion

I'm sure there's a lot of stuff missing from the quick summary above - suffice it to say that the Matrix ecosystem is growing so fast and so large that it's pretty hard to keep track of everything that's going on.  The big remaining blockers we see at this point are:

• End-to-end Encryption roll-out
• Polishing UX on Vector - showing that it's possible to build better-than-Slack quality UX on top of Matrix
• Bots, Integrations and Bridges - making them absolutely trivial to build and deploy, and encouraging everyone to write as many as they can!
• Improving VoIP, especially for conferencing, especially on Mobile
• Threading
• Editable messages
• Synapse scaling and stability - this is massively improved, but there's still work to be done.  Meanwhile projects like Ruma give us hope for light at the end of the Synapse tunnel!
• Spec refinements - there are still a lot of open spec bugs which we need to resolve so we can declare the spec (and thus Matrix!) out of beta.
• More clients - especially desktop ones; helping out with Quaternion, Tensor, PTO, etc.

...and then all the pieces of the jigsaw will finally be in place, and Matrix should hopefully fulfil its potential as an invaluable, open and decentralised data fabric for the web.

Thanks, once again, to everyone who's been supporting and using Matrix - whether it's by hanging out in the public chatrooms, running your own server, writing your own clients, bots, or servers, or just telling your friends about the project.  The end of the beginning is in sight: thanks for believing in us, and thank you for flying Matrix.

Matthew, Amandine & the Matrix Team.

Appendix: The Missing Synapse Changelogs

Changes in synapse v0.16.1 (2016-06-20)

Bug fixes:

• Fix assorted bugs in /preview_url (PR #872)
• Fix TypeError when setting unicode passwords (PR #873)

Performance improvements:

• Turn use_frozen_events off by default (PR #877)
• Disable responding with canonical json for federation (PR #878)

Changes in synapse v0.16.1-rc1 (2016-06-15)

Features: None

Changes:

• Log requester for /publicRoom endpoints when possible (PR #856)
• 502 on /thumbnail when can't connect to remote server (PR #862)
• Linearize fetching of gaps on incoming events (PR #871)

Bugs fixes:

• Fix bug where rooms where marked as published by default (PR #857)
• Fix bug where joining room with an event with invalid sender (PR #868)
• Fix bug where backfilled events were sent down sync streams (PR #869)
• Fix bug where outgoing connections could wedge indefinitely, causing push notifications to be unreliable (PR #870)

Performance improvements:

• Improve /publicRooms performance (PR #859)

Changes in synapse v0.16.0 (2016-06-09)

NB: As of v0.14 all AS config files must have an ID field.

Bug fixes:

• Don't make rooms published by default (PR #857)

Changes in synapse v0.16.0-rc2 (2016-06-08)

Features:

• Add configuration option for tuning GC via gc.set_threshold (PR #849)

Changes:

• Record metrics about GC (PR #771, #847, #852)
• Add metric counter for number of persisted events (PR #841)

Bug fixes:

• Fix 'From' header in email notifications (PR #843)
• Fix presence where timeouts were not being fired for the first 8h after restarts (PR #842)
• Fix bug where synapse sent malformed transactions to AS's when retrying transactions (Commits310197b, 8437906)

Performance Improvements:

• Remove event fetching from DB threads (PR #835)
• Change the way we cache events (PR #836)
• Add events to cache when we persist them (PR #840)

Changes in synapse v0.16.0-rc1 (2016-06-03)

Version 0.15 was not released. See v0.15.0-rc1 below for additional changes.

Features:

• Add email notifications for missed messages (PR #759, #786, #799, #810, #815, #821)
• Add a url_preview_ip_range_whitelist config param (PR #760)
• Add /report endpoint (PR #762)
• Add basic ignore user API (PR #763)
• Add an openidish mechanism for proving that you own a given user_id (PR #765)
• Allow clients to specify a server_name to avoid 'No known servers' (PR #794)
• Add secondary_directory_servers option to fetch room list from other servers (PR #808, #813)

Changes:

• Report per request metrics for all of the things using request_handler (PR #756)
• Correctly handle NULL password hashes from the database (PR #775)
• Allow receipts for events we haven't seen in the db (PR #784)
• Make synctl read a cache factor from config file (PR #785)
• Increment badge count per missed convo, not per msg (PR #793)
• Special case m.room.third_party_invite event auth to match invites (PR #814)

Bug fixes:

• Fix typo in event_auth servlet path (PR #757)
• Fix password reset (PR #758)

Performance improvements:

• Reduce database inserts when sending transactions (PR #767)
• Queue events by room for persistence (PR #768)
• Add cache to get_user_by_id (PR #772)
• Add and use get_domain_from_id (PR #773)
• Use tree cache for get_linearized_receipts_for_room (PR #779)
• Remove unused indices (PR #782)
• Add caches to bulk_get_push_rules* (PR #804)
• Cache get_event_reference_hashes (PR #806)
• Add get_users_with_read_receipts_in_room cache (PR #809)
• Use state to calculate get_users_in_room (PR #811)
• Load push rules in storage layer so that they get cached (PR #825)
• Make get_joined_hosts_for_room use get_users_in_room (PR #828)
• Poke notifier on next reactor tick (PR #829)
• Change CacheMetrics to be quicker (PR #830)

Changes in synapse v0.15.0-rc1 (2016-04-26)

Features:

• Add login support for Javascript Web Tokens, thanks to Niklas Riekenbrauck (PR #671,#687)
• Add URL previewing support (PR #688)
• Add login support for LDAP, thanks to Christoph Witzany (PR #701)
• Add GET endpoint for pushers (PR #716)

Changes:

• Never notify for member events (PR #667)
• Deduplicate identical /sync requests (PR #668)
• Require user to have left room to forget room (PR #673)
• Use DNS cache if within TTL (PR #677)
• Let users see their own leave events (PR #699)
• Deduplicate membership changes (PR #700)
• Increase performance of pusher code (PR #705)
• Respond with error status 504 if failed to talk to remote server (PR #731)
• Increase search performance on postgres (PR #745)

Bug fixes:

• Fix bug where disabling all notifications still resulted in push (PR #678)
• Fix bug where users couldn't reject remote invites if remote refused (PR #691)
• Fix bug where synapse attempted to backfill from itself (PR #693)
• Fix bug where profile information was not correctly added when joining remote rooms (PR #703)
• Fix bug where register API required incorrect key name for AS registration (PR #727)

We just released Synapse 0.14.0 - a major update which incorporates lots of work on making Synapse more RAM efficient. There's still a lot of room for further improvements, but the main headlines are reducing the resident memory footprint dramatically by interning strings and deduplicating events across the many different caches. It also adds a much-needed SYNAPSE_CACHE_FACTOR environment variable that can be used to globally decrease or increase the sizing of all of Synapse's various caches (with an associated slow-down or speed-up in performance). Quite how improved the new memory footprint seems to very much depend on your own use case, but it's certainly a step in the right direction.

For more details on recent Synapse performance work (and a general state of the union for the whole Matrix ecosystem), check out our Spring update.

Get all new synapse from https://github.com/matrix-org/synapse - we recommend upgrading (or installing!) asap :)

Full changelog follows:

Changes in synapse v0.14.0 (2016-03-30)

No changes from v0.14.0-rc2

Changes in synapse v0.14.0-rc2 (2016-03-23)

Features:

• Add published room list API (PR #657)

Changes:

• Change various caches to consume less memory (PR #656, #658, #660, #662, #663, #665)
• Allow rooms to be published without requiring an alias (PR #664)
• Intern common strings in caches to reduce memory footprint (#666)

Bug fixes:

• Fix reject invites over federation (PR #646)
• Fix bug where registration was not idempotent (PR #649)
• Update aliases event after deleting aliases (PR #652)
• Fix unread notification count, which was sometimes wrong (PR #661)

Changes in synapse v0.14.0-rc1 (2016-03-14)

Features:

• Add event_id to response to state event PUT (PR #581)
• Allow guest users access to messages in rooms they have joined (PR #587)
• Add config for what state is included in a room invite (PR #598)
• Send the inviter's member event in room invite state (PR #607)
• Add error codes for malformed/bad JSON in /login (PR #608)
• Add support for changing the actions for default rules (PR #609)
• Add environment variable SYNAPSE_CACHE_FACTOR, default it to 0.1 (PR #612)
• Add ability for alias creators to delete aliases (PR #614)
• Add profile information to invites (PR #624)

Changes:

• Enforce user_id exclusivity for AS registrations (PR #572)
• Make adding push rules idempotent (PR #587)
• Improve presence performance (PR #582, #586)
• Change presence semantics for last_active_ago (PR #582, #586)
• Don't allow m.room.create to be changed (PR #596)
• Add 800x600 to default list of valid thumbnail sizes (PR #616)
• Always include kicks and bans in full /sync (PR #625)
• Send history visibility on boundary changes (PR #626)
• Register endpoint now returns a refresh_token (PR #637)

Bug fixes:

• Fix bug where we returned incorrect state in /sync (PR #573)
• Always return a JSON object from push rule API (PR #606)
• Fix bug where registering without a user id sometimes failed (PR #610)
• Report size of ExpiringCache in cache size metrics (PR #611)
• Fix rejection of invites to empty rooms (PR #615)
• Fix usage of bcrypt to not use checkpw (PR #619)
• Pin pysaml2 dependency (PR #634)
• Fix bug in /sync where timeline order was incorrect for backfilled events (PR #635)

It's been 3 months since the Matrix Holiday Special and once again we've all been too busy writing code to put anything that detailed on the blog. So without further a do here's a quick overview of how things have progressed so far in 2016!

Home servers

Synapse

Work on Synapse (our reference homeserver) has been primarily focused on improving performance. This may sound boring, but there's been a huge amount of improvement here since synapse 0.12 was released on Jan 4. Synapse 0.13 on Feb 10 brought huge CPU savings thanks to a whole fleet of caching and other optimisation work - the best way of seeing the difference here is to look at the load graph of the server that hosts matrix.org's synapse+postgres over the last few months:

Ignoring the unrelated blip during March, you can see an enormous step change in system load (which had a matching decrease in actual CPU usage) at the beginning of Feb when the 0.13 optimisations landed on matrix.org :)

Meanwhile, Synapse 0.14 is due any day now with 0.14.0-rc2 released on Wednesday. Here, the focus has been all about memory optimisation - anyone who's run a Synapse seriously will be aware that it can be a memory hog thanks to aggressively caching as much state and history in RAM as possible to avoid hitting the database and keeping everything responsive. 0.14 should improve memory usage just as dramatically as 0.13 improved CPU utilisation - introducing a quick-and-dirty SYNAPSE_CACHE_FACTOR environment variable that lets admins dial down the aggressiveness of the caching (at the expense of performance), but more interestingly implementing string interning and ensuring that events are cached by ID rather than duplicated across multiple caches in order to make memory usage more efficient. It's too early to have impressive looking graphs, and there are still a few memory spikes being tracked down before we release 0.14, but we're hoping for at least a 50% reduction in memory footprint.

Featurewise the highlights include: server-generated unread notification & highlight counts and push badge support, lots of support and refinements for guest access and 3rd party ID invites. Meanwhile we've finally fixed some of the most embarrassing long-standing missing features such as letting folks logout serverside(!), delete aliases and determine whether rooms should be published in the room directory or not.

Finally, Synapse is now part of FreeBSD Ports thanks to Brendan Molloy, and NixOS thanks to Robin Lambertz! Huge thanks to them for contributing the packages to the respective OSes and to all the other synapse package maintainers out there!

It's incredibly exciting to see Synapse's maturity improving and hitting the optimisation stage of its life; huge kudos to Erik for spearheading the optimisation work. We strongly recommend folks upgrade to 0.14 when it's available; it's never been a better time to run a homeserver! :D

Dendron

Meanwhile, Dendron (our next generation homeserver) development has been progressing interestingly: we finished an initial spike to get a Golang skeleton server in place, albeit one that delegates most of the endpoints through to Synapse. In fact, matrix.org itself has been running via Dendron since February!

The whole point of Dendron is to provide an architecture where we can split apart the various endpoints that Synapse provides today, re-implementing them where appropriate in Golang, and critically letting the endpoints scale horizontally with clusters of backend servers abstracted by the single Dendron API facade. As a result, most of the Dendron work has actually ended up going into restructuring Synapse such that multiple Synapses can be run in a cluster behind a single Dendron, allowing us to horizontally scale API endpoints at last. This takes the form of adding cluster replication support to Synapse. This is still work-in-progress as we go through fixing up more and more state to be replicable (replicatable?) between synapses - hopefully it should land in the Synapse 0.15 timeframe. And then we enter a very very interesting new world of horizontally scalable homeservers...

Ruma

Ruma has also seen some progress over the last few months - Ruma is an independent Rust language homeserver project led by Jimmy Cuadra, and whilst in early development still (currently focusing on the user login and registration system) shows a lot of promise. Lots of work has ended up going into the required Rust dependencies rather than the Matrix code itself, but if you're interested in Rust then please drop by #ruma:matrix.org or #ruma on Freenode and say hi!

Clients

Whilst homeserver development is mainly all about performance and scaling work currently, the client side of the Matrix ecosystem is the polar opposite - with lots of rapid progress on exciting new clients happening from all over the community.

Perpetually Talking Online (PTO)

PTO has evolved enormously since Torrie Fischer first revealed it at the end of 2015. PTO is an independent project that acts as a Matrix client that exposes an IRC server interface - effectively turning any Matrix homeserver into an ircd; letting folks hook their favourite IRC clients directly into Matrix and use it as an enormous decentralised IRC network. (N.B. this is not to be confused with matrix-appservice-irc, which acts as a server-side bridge between Matrix rooms and IRC channels.) Obviously you lose some of the Matrix specific features (read receipts, typing notifs, VoIP, etc) but there's clearly a huge benefit for the IRC community to be able to use Matrix as if it were an IRC network.

There have been three releases so far, with the v0.3.0 ("Carburetor") release in March being tantalisingly close to being usable for everyday purposes. We actually have pto.matrix.org all set up and ready to go as an IRC frontend for the matrix.org homeserver and once issue #60 is resolved we'll be turning it on :)

There's one catch though - XChat was never quite built to handle the hundreds of rooms that we've got used to Matrix supporting... :D

Come hang out in #pto:oob.systems if you're interested in PTO!

Quaternion

Quaternion is a new Qt/QML/C++ desktop client created by Felix Rohrbach. It's a fairly early alpha but still quite usable and in very active development. #quaternion:matrix.org is the place to talk all things Quaternion :)

matrix-glib-sdk

Meanwhile, over on the GTK side of the world, Gergely Polonkai has been been making great progress on his matrix-glib-sdk Glib client SDK for Matrix. The end goal here is to implement a full Telepathy plugin for Matrix on top of the SDK. Originally written in C, but now shifted to Vala, the SDK is in very active development and now implements all(?) of the Matrix client-server API - a snapshot of the work-in-progress SDK API docs can be found at http://gergely.polonkai.eu/matrix-glib-sdk. Next up is a formal release and building out clients on top!

matrix-react-sdk, matrix-ios-sdk, matrix-android-sdk and Vector

Finally, huge amounts of time and effort have continued to be pumped into the official matrix-react-sdk, matrix-ios-sdk and matrix-android-sdk - driven substantially by requirements for Vector, the FOSS Matrix-powered collaboration app that we've been helping with:

The best way of seeing what's been going on here is probably by considering Vector itself, which is currently in formal beta (0.4.1 for web, 0.1.2 for iOS and #116 on Android). The big news includes:

• Beta iOS and Android apps. These are early beta but feedback is very much appreciated - the Android beta can be downloaded from Jenkins; if you want to help beta iOS via TestFlight, come ask on #ios:matrix.org.
• Guest access. Anyone can jump straight into Matrix by going to http://vector.im without even having to sign up for an account. Guests are quite restricted on what they can do (and can only join rooms which explicitly have guest access enabled), but this is a *huge* improvement in getting folks using Matrix.
• Ability to jump to any message ever - e.g. when clicking through search results or when permalinking a message... using precisely the same UI that you use when chatting. Permalinks are awesome. If you want to randomly jump back in time to the first weeks of #matrix:matrix.org, now you can...
• Read Markers, scrolling that remembers the scroll offset per-room, and the ability to jump to unread messages
• Synchronised missed notification and missed highlighted notification information per-room
• Badge counts for unread notifications
• Entirely reworked Room Settings
• Entirely reworked User Settings, including push notification configuration
• Entirely reworked Room Directory
• Lots of performance improvements
• Much improved inviting by email
• Much improved reliability on video conferencing
• Closing literally hundreds and hundreds of bugs...

All that remains right now is yet more bugfixing and incorporating feedback from the current betas! Please give as much feedback as possible in #vector:matrix.org :)

Bridges & Bots

Bridges, bots, and other integrations and application services have inevitably taken slightly lower priority whilst we've been focusing on the core server and client bits of the ecosystem. However, as of March we've started a major new project to get these moving again, starting with a big update to the IRC Bridge. This is due to be released next week, but you can get a sneak peek at what's going into the release at the commit log. Highlights include the ability to persist nicks; connect via IPv6; improve formatted message handling; actually feed error messages from IRC back to Matrix; and much much more.

matrix-appservice-verto also got some love, which means that multiway video conferencing powered by FreeSWITCH now works reliably. The quality still could be improved, but the unreliable call setup that plagued earlier versions is now fixed.

In the next few months we're expecting to see a lot more activity on bridges & bots... watch this space :)

Update Sat March 26:

Totally forgot to mention a few of the key new bridges which have been contributed by the community this year - particularly interesting are the Rocket.Chat<->Matrix bridge written by Sing-Li over at Rocket.Chat which provides basic bridging between the awesome Rocket.Chat collaboration app and the wider Matrix ecosystem. It's early days, but this is incredibly promising for 'hardcoded' bridging between specific rooms - it just needs Rocket.Chat to support 'virtual' users and will then be seamless federation.

Similarly, matrix-appservice-gitter is a Gitter<->Matrix bridge built by Leonerd on top of the matrix-appservice-bridge Node library. Again, it's early days but is working well for 'hardcoded' bridging - supporting dynamic rooms and users is next on the todo list :)

The Spec

We started our formal release process for the spec just before Christmas with r0.0.0 - and released r0.0.1 in January with minor clarifications and updates. In practice the spec feels quite stable right now, although things have moved on a bit since January and r0.0.2 is definitely overdue at this point.

In the meantime, you can always get the very latest bleeding edge copy of the spec via the speculator. We've also added an initial cut at a spec for the Identity Service at last.

Events

We've been focusing on writing code than evangelising Matrix recently, although we did get out to FOSDEM 2016 and TADHack Mini Japan and WebRTC Conference and Enterprise Connect 2016 where we showed off Matrix & Vector in the WebRTC Real World Innovation showcase.

GSoC

We are incredibly grateful to have been accepted as an organisation into Google Summer of Code 2016! The last two weeks have been the window for students to propose projects to us that they could work on over the course of the summer, and it's been fascinating to meet the GSoCers and see a whole new community pop up on Matrix and advise and mentor applicants through their proposals. At the last count we've received 35 proposals, many inspired by our list of ideas, including some really impressive candidates - many thanks to all the students who have applied to us. We don't know yet how many slots Google will allocate to us, but one way or another we're really looking forward to helping the GSoCers make the most out of their summer of Matrix! All GSoC discussion is happening in #gsoc:matrix.org.

What's next?

In no particular order, the urgent stuff that still remains includes:

• Continuing to polish synapse and build out dendron-based clustering
• Building as many bridges, bots and other integrations as possible
• The matrix.to URL-handler service: having client-agnostic https://matrix.to/#matrix:matrix.org URLs to help with sharing matrix room aliases etc
• End-to-end crypto. No progress since December; we need to get back to it asap.
• Exiting Vector from beta
• Finishing the server-to-server API specification
• Improving the security model for access_tokens
• Editable messages
• Pinned, tagged, and 'liked' messages
• Threading
• Decentralised accounts
• Decentralised reputation

In practice, Bridging and E2E crypto is likely to get the most attention first (beyond the standard ongoing polishing). There's obviously a significant amount of work there, but we expect to see benefits pretty quickly throughout Matrix - especially from bridging. Hopefully it's true to say that the next few months should be quite transformational :D

Anyway, thanks for reading this sprawling update and for supporting Matrix. And please come say hi in #matrix:matrix.org if you have any questions :)

• Matthew, Amandine & the Matrix.org team.

Hi all,

Synapse 0.13 was released this afternoon, bringing a new wave of features, bug fixes and performance fixes. The main headlines include: huge performance increases (big catchup /syncs that were taking 20s now take 0.3s!), support for server-side per-room unread message and notification badge counts, ability for guest accounts to upgrade into fully-fledged accounts, change default push rules back to notifying for group chats, and loads of bug fixes. This release incorporates what-was 0.12.1-rc1.

Please note that on first launch after upgrading a pre-0.13 server to 0.13 or later, synapse will add a large database index which may take several minutes to complete. Whilst the index is added the service will be unresponsive.

Please get the new release from https://github.com/matrix-org/synapse and have fun!

Matthew

Full release notes:

Changes in synapse v0.13.1 (2016-02-10) =======================================

• Bump matrix-angular-sdk (matrix web console) dependency to 0.6.8 to pull in the fix for SYWEB-361 so that the default client can display HTML messages again(!)

Changes in synapse v0.13.0 (2016-02-10)

This version includes an upgrade of the schema, specifically adding an index to the events table. This may cause synapse to pause for several minutes the first time it is started after the upgrade.

Changes:

• Improve general performance (PR #540, #543. #544, #54, #549, #567)
• Change guest user ids to be incrementing integers (PR #550)
• Improve performance of public room list API (PR #552)
• Change profile API to omit keys rather than return null (PR #557)
• Add /media/r0 endpoint prefix, which is equivalent to /media/v1/ (PR #595)

Bug fixes:

• Fix bug with upgrading guest accounts where it would fail if you opened the registration email on a different device (PR #547)
• Fix bug where unread count could be wrong (PR #568)

Changes in synapse v0.12.1-rc1 (2016-01-29)

Features:

• Add unread notification counts in /sync (PR #456)
• Add support for inviting 3pids in /createRoom (PR #460)
• Add ability for guest accounts to upgrade (PR #462)
• Add /versions API (PR #468)
• Add event to /context API (PR #492)
• Add specific error code for invalid user names in /register (PR #499)
• Add support for push badge counts (PR #507)
• Add support for non-guest users to peek in rooms using /events (PR #510)

Changes:

• Change /sync so that guest users only get rooms they've joined (PR #469)
• Change to require unbanning before other membership changes (PR #501)
• Change default push rules to notify for all messages (PR #486)
• Change default push rules to not notify on membership changes (PR #514)
• Change default push rules in one to one rooms to only notify for events that are messages (PR #529)
• Change /sync to reject requests with a from query param (PR #512)
• Change server manhole to use SSH rather than telnet (PR #473)
• Change server to require AS users to be registered before use (PR #487)
• Change server not to start when ASes are invalidly configured (PR #494)
• Change server to require ID and as_token to be unique for AS's (PR #496)
• Change maximum pagination limit to 1000 (PR #497)

Bug fixes:

• Fix bug where /sync didn't return when something under the leave key changed (PR #461)
• Fix bug where we returned smaller rather than larger than requested thumbnails when method=crop (PR #464)
• Fix thumbnails API to only return cropped thumbnails when asking for a cropped thumbnail (PR #475)
• Fix bug where we occasionally still logged access tokens (PR #477)
• Fix bug where /events would always return immediately for guest users (PR #480)
• Fix bug where /sync unexpectedly returned old left rooms (PR #481)
• Fix enabling and disabling push rules (PR #498)
• Fix bug where /register returned 500 when given unicode username (PR #513)

Happy 2016 everyone!

To greet the new year, we bring you all new Synapse 0.12. The focus here has been a wide range of polishing, bugfixes, performance improvements and feature tweaks. The biggest news are that the 'v2' sync APIs are now production ready; the search APIs now work much better; 3rd party ID invites now work; and we now mount the whole client-server API under the /_matrix/client/r0 URI prefix, as per the r0.0.0 release of the Client Server API from a few weeks ago. The r0 release unifies what were previously the somewhat confusing mix of 'v1' and 'v2' APIs as a single set of endpoints which play nice together.

We highly recommend all homeservers upgrading to v0.12.0 as soon as possible. Get it now from https://github.com/matrix-org/synapse/ or via our shiny new Debian packages at https://matrix.org/packages/debian/.

Full changelog follows:

Changes in synapse v0.12.0 (2016-01-04)

• Expose /login under r0 (PR #459)

Changes in synapse v0.12.0-rc3 (2015-12-23)

• Allow guest accounts access to /sync (PR #455)
• Allow filters to include/exclude rooms at the room level rather than just from the components of the sync for each room. (PR #454)
• Include urls for room avatars in the response to /publicRooms (PR #453)
• Don't set a identicon as the avatar for a user when they register (PR #450)
• Add a display_name to third-party invites (PR #449)
• Send more information to the identity server for third-party invites so that it can send richer messages to the invitee (PR #446)
• Cache the responses to /initialSync for 5 minutes. If a client retries a request to /initialSync before the a response was computed to the first request then the same response is used for both requests (PR #457)
• Fix a bug where synapse would always request the signing keys of remote servers even when the key was cached locally (PR #452)
• Fix 500 when pagination search results (PR #447)
• Fix a bug where synapse was leaking raw email address in third-party invites (PR #448)

Changes in synapse v0.12.0-rc2 (2015-12-14)

• Add caches for whether rooms have been forgotten by a user (PR #434)
• Remove instructions to use --process-dependency-link since all of the dependencies of synapse are on PyPI (PR #436)
• Parallelise the processing of /sync requests (PR #437)
• Fix race updating presence in /events (PR #444)
• Fix bug back-populating search results (PR #441)
• Fix bug calculating state in /sync requests (PR #442)

Changes in synapse v0.12.0-rc1 (2015-12-10)

• Host the client APIs released as r0 by https://matrix.org/docs/spec/r0.0.0/client_server.html on paths prefixed by/_matrix/client/r0. (PR #430, PR #415, PR #400)
• Updates the client APIs to match r0 of the matrix specification.
  • All APIs return events in the new event format, old APIs also include the fields needed to parse the event using the old format for compatibility. (PR #402)
  • Search results are now given as a JSON array rather than a JSON object (PR #405)
  • Miscellaneous changes to search (PR #403, PR #406, PR #412)
  • Filter JSON objects may now be passed as query parameters to /sync (PR #431)
  • Fix implementation of /admin/whois (PR #418)
  • Only include the rooms that user has left in /sync if the client requests them in the filter (PR #423)
  • Don't push for m.room.message by default (PR #411)
  • Add API for setting per account user data (PR #392)
  • Allow users to forget rooms (PR #385)
• Performance improvements and monitoring:
  • Add per-request counters for CPU time spent on the main python thread. (PR #421, PR #420)
  • Add per-request counters for time spent in the database (PR #429)
  • Make state updates in the C+S API idempotent (PR #416)
  • Only fire user_joined_room if the user has actually joined. (PR #410)
  • Reuse a single http client, rather than creating new ones (PR #413)
• Fixed a bug upgrading from older versions of synapse on postgresql (PR #417)

Hi all,

We've been pretty bad at updating the blog over the last few months with all the progress that's been happening with Matrix.  Whilst Matrix rooms like #matrix:matrix.org and #matrix-dev:matrix.org have been very active (and our twitter account too), in general we've ended up spending way too much time actually writing software and not enough time talking about it, at least here. When a blog goes quiet it normally means that either the authors have got bored, or they're too busy building cool stuff to keep it updated. I'm happy to say that option 2 is the case here!

As a result, there's a huge backlog of really cool stuff we should have talked about. Hopes of writing an Advent Calendar series of blog posts also went out the window as we set Christmas as an arbitrary deadline for loads of work on Synapse, the Matrix Spec and matrix-react-sdk.

So, to try to break the impasse, here's a slightly unorthodox whistle-stop tour of all the amazing blogposts we would have written if we'd had time. And perhaps some of them will actually expand into full write-ups when we have more time to spare in the future :)

End to End Encryption Update

One of the great promises of Matrix is to provide End-to-end encryption as part of the baseline standard (configurable per-room). In practice, our progress has been a little non-linear - we started writing an Axolotl ratchet implementation in C++14 (with a pure C API) named Olm back in February, and then finished it off and wired a basic 1:1 proof-of-concept implementation into matrix-react-sdk in June. We then announced Olm back at the wonderful Jardin Entropique conference in Rennes:

You can read the full presentation that we gave at https://matrix.org/~matthew/2015-06-26 Matrix Jardin Entropique.pdf - and you can even play with a very basic test jig at https://matrix.org/~markjh/olm/javascript/demo.html which uses an emscripten compiled version of Olm in the browser to put the ratchet through its paces.

Things then stalled for a bit, but as of this month they're moving again, and if you're interested in the progress you can read all about it at:

• https://matrix.org/jira/browse/SPEC-162
• and a full draft spec at http://matrix.org/speculator/spec/drafts%2Fe2e/client_server.html#end-to-end-encryption
• the Olm git repo at http://matrix.org/git/olm/about/
• and the formal Olm spec at https://matrix.org/docs/spec/olm.html

The main stuff remaining is basically key management (in Synapse and the matrix spec), group conversation ratchets, and UX for wiring it properly into various Matrix clients. We expect to make progress on this over the next few months :)

Meanwhile, huge kudos to Tor who was crazy enough to add the basic 1:1 Olm ratchet to Weechat before we'd even finished writing our test jig!

Lean DUS

A few days after Jardin Entropique we made it to Lean DUS - a great tech meetup in Düsseldorf organised by Sipgate, who were kind enough to invite us to speak. This was a chance to give a full update on Matrix (as of July!) and talk some more about Olm and plans for end-to-end encryption. This one got recorded - and you can see it below. There's also an official page with full videos, slide deck and photos up at https://www.leandus.de/2015/08/weil-und-hodgson/.

Lean Dus #9 - End to end encryption for decentralised communication mit Matthew Hodgson from sipgate on Vimeo.

New Matrix Bridges

Somehow we've failed to blog about the amazing matrix-appservice-bridge Node framework which we've built as general purpose infrastructure for building Matrix Application Services which act as bridges between existing networks and comms solutions and Matrix. The architecture here looks something like this:

...and the goal is to end up with something like this:

matrix-appservice-bridge is still in development, but there are a bunch of really cool bridges already using it - and a great howto that shows how you can use it to write a Matrix{'<->'}Slack bridge in under 100 lines of code.

• the Matrix/Verto Bridge uses it to hook FreeSWITCH up to Matrix - currently used to provide multiway video and voice conferencing for Vector. It could be easily extended to do generic Matrix{'<->'}SIP or Matrix{'<->'}anything-that-FreeSWITCH-can-speak though.
• a basic Matrix/Slack Bridge, which works well enough for hardcoding bridges between specific Matrix and Slack rooms.
• matrix-appservice-respoke - a crazy experiment that bridges Asterisk to Matrix by implementing the Respoke API such that Asterisk can connect to Matrix using chan_respoke.
• matrix-appservice-purple - another crazy experiment that hooks libpurple up to matrix-appservice-bridge such that *any* network that libpurple can talk to can be bridged into Matrix. So far we've experimented with Lync, Skype and Facebook (and AIM(!)) and it works - but it needs a lot more love to be usable other than as a toy.

Meanwhile, there's also:

• the Matrix/IRC Bridge, which doesn't actually use it yet, but will do soon.
• an experimental Golang Matrix/Slack Bridge, which looks at whether we should be doing this all in Golang rather than Node.

As of right now our work on bridging has been on hiatus for a month or so, and we would love support from the community in advancing and extending the stuff we've built so far. Otherwise we'll get back to it ourselves in the new year.

Astricon 2015

We had a lot of fun in Orlando in October at Astricon 2015 - we put together matrix-appservice-respoke (see above) for our talk and Dangerous Demo in a desperate 24 hour hack and it even worked! The judges were kind enough to give us the "Swan Award" prize in the dangerous demo shoot-out for the glossiest demo :)

The slides for our 'Bridging Asterisk to the Matrix Ecosystem' talk are downloadable here.

Pidgin!

We also implemented a basic libpurple plugin for Matrix - adding Matrix support to any app like Pidgin or Bitlbee that uses libpurple. (You could in theory even use it with matrix-appservice-purple to bridge from Matrix to Matrix, but that'd be silly :). It supports basic functionality and uses the new 'v2' APIs for syncing to Matrix. Adventurous libpurplers can go check it out and experiment with it from https://github.com/matrix-org/purple-matrix - feedback welcome.

OpenWebRTC Support on iOS

We went and hooked OpenWebRTC up to matrix-ios-sdk so that iOS Matrix apps can use the OpenWebRTC stack from Ericsson Research for VoIP and Video. Apparently we haven't written it up fully yet, but you can find the code at https://github.com/matrix-org/matrix-ios-openwebrtc-wrapper for those interested in using OWR with Matrix!

Debian Repository for Synapse

We built a 3rd party Debian package repository for Synapse... and then forgot to tell anyone about it, other than buried in the Synapse readme! Well, it exists, and intrepid debianers should go check it out at http://matrix.org/packages/debian/.

TADSummit

In November we attended TADSummit in Lisbon - a great event for folks hacking on telco applications and the telcos themselves. Apparently we failed to do a writeup, but we had a wonderful time: highlights included sitting down with Maarten Ectors from Canonical to wrap up Synapse as an Ubuntu Snappy app such that anyone in the Ubuntu Core ecosystem can trivially run a Matrix homeserver, and demoing it as part of the Dangerous Demos track there. We also gave a 'Matrix: One Year In' talk to summarise what we got up to in 2015.

WebRTC Paris

Whilst on the subject of conferences that we forgot to write up - we just got back from WebRTC Paris, where we demoed the latest & greatest Matrix clients and bridges, hung out with the OpenWebRTC guys and gave another ecosystem update. You can see the slides at https://matrix.org/~daniel/Matrix- One-year Status Report.pdf.

New Clients and App Services

There have been a flurry of really interesting new clients and other projects which certainly deserve whole blog posts of their own!

There's Tensor from David A Roberts - a multiplatform native client written in QML that heavily leverages the matrix-js-sdk:

There's matrix.el from Ryan Rix - a native Matrix client for Emacs! You can read all about the whys and wherefores here.

There's also loads of cool stuff that Ryan's been doing with Matrix on his blog - including Polynomial - a decentralised webring built on Matrix (yes, webrings were and are cool, ok!?!), and his Matrix-powered Body Computing System. Also, some philosophical posts on the benefits of Matrix which give us some hope that we're on the right track!

Then there's Power Take-Off from Torrie Fischer - an early lets-IRC-clients-connect-to-Matrix project in Rust...

...and there's Morpheus from Christine Dodrill (Xena) - a Matrix client and bot framework for Haskell; part of a more over-arching IRC{'<->'}Matrix unification project. Xena also wrote a great call to arms for Matrix :)

Very recently there's the Ruma project from Jimmy Cuadra - an ambitious mission to build Matrix components (up to and including a homeserver) in Rust!

Other stuff includes a Hubot adaptor from davidar, Bender: an Elixir client and bot library from Dylan Griffith, Jon Frederickson's matrix-xmpp-bridge, rzr's guide to installing synapse on a minnowmax, and I'm sure many others we don't know about or have missed!

Finally, Tor has done an amazing job on weechat-matrix-protocol-script in implementing features like V2 Sync and E2E crypto faster than we've managed to add them in the official client SDKs!

Release Zero of the Matrix Specification

We have made some major improvements to the spec over the last few months: adding in feature profiles and spec modules to better structure the document, and most recently splitting it up explicitly into separate Client-Server, Server-Server and Application-Server APIs, each with a well-defined single global 'release' number for versioning. We started this with a 'r0.0.0' release of the Client-Server API, which consolidates the horrible mess of 'v1' and 'v2' APIs we had previously flying around into a single well-defined version of the spec. Meanwhile the spec is now entirely consolidated into a set of JSON schema and Swagger 2 API descriptors, with a bunch of ReStructured Text for the verbiage - you can find it all at https://github.com/matrix-org/matrix-doc.

The r0.0.0 changelog is exciting stuff - you can see it in its entirety at http://matrix.org/docs/spec/r0.0.0/client_server.html#changelog. Synapse itself will support the full r0 API set in 0.12, which will be released any day now.

We've also switched the Swagger-based API viewer over to Swagger 2.0: http://matrix.org/docs/api. We also rejigged the Matrix documentation website entirely, generating it via Jekyll and adding in a new guides section.

Also, we should have mentioned the existence of Speculator - a golang helper app (source here) which, as the name suggests, renders out copies of the spec as HTML from different branches and pull requests for ease of previewing.

Dendron

Over the last few months we've also started an entirely new project, codenamed Dendron. Dendron is the project to evolve Synapse from the current single-threaded Python/Twisted monolithic homeserver into something with a lot more type-safety, horizontal scalability and high availability. We've mainly been experimenting with different ways of doing this, but the current plan is to split Synapse itself up into multiple services which can each scale independently, and then rewrite some/all of them in languages with better type safety and/or performance or profiling tools.

Some folks may remember a survey that we posted a few months ago asking for the community's thoughts on what languages they'd like their ideal homeserver to be written in, from the perspectives of someone running it as well as hacking on it. Whilst we haven't (at all) based our decisions for Dendron purely on the survey, it was still quite an interesting exercise. And here are the results (maximum 'score' is 5, not 10):

The basic feedback was that from the existing community: folks dislike running Java or Node servers; are okayish with Python, but would prefer native or near-native code (be that C, Rust or Go). Meanwhile, for contributing code, there's slightly more interest in the (relatively) new shinies of Go and Rust. And of course, everyone wanted to plug their own special snowflake language in the 'Others' section, which was mainly a mix of Erlang, Elixir, Haskell, Lisp and Perl :)

This reinforced the choices we were looking at anyway - either Rust (for its safety), or Go (for its simplicity, python-likeness, and concurrency). (We'd also consider Java, but have to concede that the FOSS community doesn't like running it.)

So we looked at the dependencies that Synapse currently has, and the Rust equivalents, and concluded that the Rust ecosystem unfortunately isn't quite mature enough yet to reliably handle the rather large set of complicated deps that we need in a homeserver. Also, nobody on the core team is really a Rust guru yet. Meanwhile, we have at least one ex-Google Go expert in the core team, and in practice it has the edge in terms of maturity. So, right now, we're looking at switching chunks of Dendron to Go where it makes sense. (This is subject to change though depending on how we get on). You should expect to hear a lot more about Dendron in 2016 :)

matrix-react-sdk, Vector, and latest Matrix features

Last but not least: huge amounts of our time over the last year has gone into building matrix-react-sdk - a full set of glossy Web UI components for building super-high quality glossy apps based on Matrix, built on the matrix-js-sdk. This is basically a reaction against the original matrix-angular-sdk and Matrix Console app that we launched Matrix with back in 2014 - which had minimal attention to UI/UX and suffered from major performance problems; it was built purely as the fastest possible way we knew to prototype and demo Matrix in the first place. matrix-react-sdk however has been built for both performance and quality of UI/UX, as well as obviously using all the latest and greatest Matrix APIs. (In fact, the transition from matrix-angular-sdk to matrix-react-sdk is pretty similar to the Synapse to Dendron transition on the horizon - although the latter should be more incremental and less 'rewrite the world').

Meanwhile, as part of our commercial work at our day job (i.e. not as Matrix.org) we've been helping on a glossy FOSS app called Vector which is layered on top of matrix-react-sdk as a thin 'skin' layer of CSS and a few custom components. The intention for Vector is to ensure that Matrix has a flagship glossy client: which it clearly needs, in order to gain credibility and drive uptake of the Matrix standard, and also ensure that the Matrix standard actually does indeed fit the needs for a state-of-the-art collaboration tool.

matrix-react-sdk (and thus Vector) is still in incredibly heavy development - we're going to start a formal beta fairly soon, but as of right now it's still sprouting features and refinements on a daily basis. Meanwhile matrix-react-sdk's APIs are not remotely frozen (we entirely refactored it as recently as a few weeks ago), so not yet ready for use as a general purpose building block.

Some of the stuff going into react-sdk is incredibly cool - recent Matrix stuff that it shows off includes:

• Animated read receipts. If you haven't seen these, you haven't lived. They are a relatively new addition to the Matrix spec.
• Full server-side search. We now have full-text search in the Matrix spec, and implemented on synapse both on sqlite and postgres - and now in Vector too. Having good search over all of your chat history makes Matrix *so* much more usable.
• Video conferencing. We have full multi-way conferencing in Vector via matrix-appservice-verto and FreeSWITCH. The intention is to add this to the core Matrix spec (having first made it a bit more generic) - see the draft spec for details.
• 3rd party invites. You can now invite users into Matrix by email address as well as matrix ID, and it works. Vector implements this.
• Room tagging. You can now tag rooms as favourites, low priority, or with arbitrary namespaced metadata. Vector implements this through a swanky drag & drop UI.
• "V2" Sync API. Now part of the 'r0.0.0' spec, this lets Matrix support much smarter incremental and partial synchronisation patterns. Vector now implements this, meaning that browser windows sync much faster after being offline for a bit, and no longer hammer the user with stale desktop notifications.
• Accessing 'historical' rooms. Matrix now lets you keep track of rooms you've left, so you can view and search the conversation logs even after you've left the conversation. Vector now implements this (as of Monday!)
• Tab-complete that doesn't suck. This is a purely client-side feature which landed on Thursday!
• Roll-overable animated GIFs. 'nuff said.
• Markdown support. yay!
• Synchronised read and notification history. This hasn't landed yet (in vector or synapse or even the spec), but finally provides a way to keep read and notification state in sync in realtime across all your clients and a meaningful favicon 'badge' telling you how many notifications you missed!
• Guest access. This hasn't landed in Vector yet, but it's in the spec and Synapse. It will let folks use Matrix without having to create an account (at least for rooms which support 'guest access' from the public).

If you haven't given Vector a spin, it's well worth heading over to https://vector.im and taking a look.

There's also an Electron desktop version of Vector in progress, contributed by Steven Hammerton at https://github.com/stevenhammerton/vector-desktop (although it's currently stuck on an old release).

Epilogue

Okay, this has got a lot longer than it was meant to be - but hopefully makes up a bit for the lack of comms over the last few months whilst we've been drowning in work on Synapse, Vector, the Spec, Dendron, and everything else mentioned above.

2015 has been an epic year as we've taken Matrix from a very early beta to the advanced stage that it's at now. Obviously there's still a lot of stuff to do though. Right now we expect the focus in 2016 to be:

• Vector - making sure the Matrix protocol has a flagship glossy FOSS client that normal (non-geek) users can use.
• Dendron - making Synapse more reliable, scalable and maintainable.
• Bridging - wiring as much of the rest of the world into Matrix as accurately and efficiently as possible.
• Federation Spec - finishing and releasing the Server-Server API.
• End-to-end crypto - finishing it off.

...and obviously continuing to refine and extend the core of Matrix itself with features like threading, editable messages, and possibly even distributed accounts.

There are very fun and exciting times ahead. We'd just like to say a profound thank you to everyone who's supported Matrix this far and helped make this mission possible - whether it's by running clients/servers/services, or writing your own, or filing bugs and feedback on our code or the spec, or telling folks about the project, or paying us to work on it(!), or just by reading this blog post. Hopefully 2016 will be the year where online communication starts to open up and interoperate once again, rather than becoming ever more fragmented and closed.

Thanks for reading - and Merry Christmas, for those who celebrate :)

• Matthew, Amandine & the Matrix Team.

Hi all,

Just realised that the release of Synapse 0.10.0 on Sept 3rd 2015 was precisely one year from the initial launch of Matrix. As such, it feels only right to have a quick update on where we've got to so far, and where we expect things to go from here!

Back at the original launch, all we had was a very rough and ready Synapse homeserver, an early draft of the spec, and precisely one client - the Angular webclient, much of which was mainly written at the last minute on the plane to TechCrunch Disrupt SF (and has never quite recovered :S). From this initial seed it's been incredibly exciting and slightly scary to see how much things have advanced and grown - the big headlines for the past year (in roughly chronological order) include:

• Making Federation Work:
  • Switching all of federation over to SSL, using perspectives for key management
  • Crypto-signing all the events in a room's message graph to assert integrity
  • Sorting out 'power levels' and 'auth events' to allow totally decentralised kicks/bans/etc to work in an open federated environment
• Decentralised content repository and thumbnailing
• Reference mobile "Matrix Console" clients for iOS and Android
• Official client SDKs for iOS and Android - both at the API wrapping layer and the reusable UI component layer
• Push notifications for APNS and GCM (both on server & clients)
• Official client SDKs for JavaScript, Python and Perl
• Typing notifications
• The sytest integration test harness
• Proper WebRTC support for VoIP, including TURN.
• Application Services and Bots - actually letting Matrix defragment communications :)
  • Bridging to all of Freenode, Moznet and other IRC networks
  • Matrix<->SMS bridge from OpenMarket
  • SIP bridges via FreeSWITCH and Verto
  • Parrot Bebop Drone <-> Matrix bridge via Janus
  • ODB2 telemetry <->  Matrix bridge via Android SDK
  • Reusable bridging framework in Node
• Many iterations and refinements to the spec, including designing v2 of the client SDK
• Switching from Angular to React for all of our web-client development
• Customisable skins and embedding support for the matrix-react-sdk
• End-to-end encryption (not quite formally released yet, but it's written, specced and it works!)
• VoIP support on mobile (landed in Android; still experimenting with different WebRTC stacks on iOS)
• History ACLs
• Delivery reports
• Switching from access_tokens to macaroons for authentication (not yet released)
• Lots and lots of performance work on Synapse, as we've tried to get the most out of Twisted.

...and last but not least, the evolution of the #matrix:matrix.org community - including loads of 3rd party clients, SDKs and application services, synapse packaging and even experimental home servers :)

Overall the last year was an exercise in actually fleshing out the whole ecosystem of Matrix and getting it to a stable usable beta acceptable to early adopters. The plan for the next 12 months is then to make the transition from stable beta to a properly production grade environment that can be used to run large scale services used by normal end-users. In practice, this means:

• A major rearchitecture of Synapse.
  • Synapse currently has no support for horizontal scaling or clustering within a single instance, and many will have seen the performance problems we've hit with a relatively monolithic Twisted app architecture. Profiling deferreds in Twisted has been a particular nightmare.
  • During September we are starting the process of splitting Synapse apart into separate services (e.g. separating reading eventstreams from writing messages) both to allow horizontal scalability and to experiment with implementing the services in more efficient languages than Python/Twisted.
  • We will continue the normal Synapse release process in parallel with this work.
• Ensuring Matrix can support a genuinely excellent UX for normal end-users on glossy clients, and supporting glossy client development as required.  The days of Matrix being just for powerusers are numbered... :)
• Switching to use 3rd party IDs as the primary means of referring to users in Matrix, hiding matrix IDs as a feature for powerusers and developers.
• Finishing the spec. You may have noticed the spec has been quietly evolving over the last few months - finally gaining a versioning system, and with larger chunks of it being automatically generated from formal API spec descriptions. We will be finishing off and filling in the remaining holes.
• Improving the documentation (and FAQ!) on matrix.org in general by switching to a git-backed jekyll system for all the staticish content
• Replace the Angular-based reference webapp bundled with Synapse entirely with a matrix-react-sdk based reference app, and providing better examples and documentation for using it to embed Matrix functionality into existing websites.
• Moving to v2 of the client-server API. This fixes some significant limitations in the v1 API that everyone's been using all year, and should improve performance significantly for many use cases (especially when launching apps). The v1 API will hang around for a very very long time for backwards compatibility.
• Writing *lots* more bridges and integrations to other protocols, now we have a nice framework for rapidly developing them.
• General security audits and double-checking the security model.
• New features, including:
  • Multiway VoIP and Video conferencing, most likely using FreeSWITCH's new conferencing capabilities via an Application Service bridge (should be ready very shortly!)
  • Getting E2E crypto reviewed/audited and putting it live across the board.
  • Adding VoIP to iOS
  • Implementing delivery reports in all clients
  • Improving how invites work (ability to reject them; metadata about where they came from)
  • Search API
  • Improved file management
• ...and an awful lot of bug fixing as we work through the backlog we've accumulated on JIRA.

Hopefully this won't take up all year(!) and is just a beginning - there's a huge list of interesting ideas beyond this baseline which we'll be looking at assuming the core stuff above is on track. For instance, we need to work out how to decentralise the identity services that mapping 3rd party IDs to matrix IDs. We need to work out how to avoid spam. And it could be fascinating to start bridging more internet-of-things devices and ecosystems into Matrix, or decentralising user accounts between homeservers, or perhaps using Matrix for synchronising more sophisticated data structures than timelines and key-value state dictionaries...

Finally, we also want to save as much time as possible to help support the wider community in building out clients, services and servers on top of Matrix. Just like the web itself, Matrix is only as useful as the content and services built on top of it - and we will do everything we can to help the pioneers who are interested in colonising this brave new world :)

Huge thanks to everyone over the last year who have supported us - whether that's by creating an account and using the system, running a homeserver, hacking on top of the platform, contributing to the core project, enduring one of our presentations, or even paying for us to work on this. The coming year should prove incredibly interesting, and we hope you'll stay and bring along all your friends, family and colleagues for the ride as the adventure continues!

Matthew, Amandine & the whole Matrix.org team.

Hi folks,

Whilst the blog has been a bit quiet, we've actually had an incredibly busy summer refining Synapse, building the new matrix-react-sdk and example apps, building an entirely new matrix-appservice-bridge framework for rapidly creating Matrix<->other-protocol bridges, getting end-to-end encryption ready for primetime and lots more fun stuff as we keep chipping away to take Matrix out of beta. We'll write about all of these once they're ready, but right now the big news is that after 6 release candidates we have a major new update for Synapse out today - version 0.10.0. This also includes 0.9.4, which we never quite got around to releasing and ended up skipping from 0.9.4-rc1 straight to 0.10.0-rc1.

The release focuses mainly on performance, bugfixes, and infrastructure work to support forthcoming features like end-to-end encryption, read receipts, etc. Some of the more exciting new features include preset ACLs for room creation; history visibility ACLs; SAML2 single-sign-on login (courtesy of Ericsson, thanks guys!), filename support when sending files, support for specifying a canonical alias for a room, support for intermediary SSL certificates, etc.

The full changelog is below.

To upgrade, go read https://github.com/matrix-org/synapse/blob/master/UPGRADE.rst - to install for the first time, go to https://github.com/matrix-org/synapse/blob/master/README.rst.

Changes in synapse v0.10.0 (2015-09-03) =======================================

No change from release candidate.

Changes in synapse v0.10.0-rc6 (2015-09-02)

• Remove some of the old database upgrade scripts.
• Fix database port script to work with newly created sqlite databases.

Changes in synapse v0.10.0-rc5 (2015-08-27)

• Fix bug that broke downloading files with ascii filenames across federation.

Changes in synapse v0.10.0-rc4 (2015-08-27)

• Allow UTF-8 filenames for upload. (PR #259)

Changes in synapse v0.10.0-rc3 (2015-08-25)

• Add --keys-directory config option to specify where files such as certs and signing keys should be stored in, when using --generate-config or --generate-keys. (PR #250)

• Allow --config-path to specify a directory, causing synapse to use all \*.yaml files in the directory as config files. (PR #249)

• Add web_client_location config option to specify static files to be hosted by synapse under /_matrix/client. (PR #245)

• Add helper utility to synapse to read and parse the config files and extract the value of a given key. For example::

  $ python -m synapse.config read server_name -c homeserver.yaml localhost

  (PR #246)

Changes in synapse v0.10.0-rc2 (2015-08-24)

• Fix bug where we incorrectly populated the event_forward_extremities table, resulting in problems joining large remote rooms (e.g. #matrix:matrix.org)
• Reduce the number of times we wake up pushers by not listening for presence or typing events, reducing the CPU cost of each pusher.

Changes in synapse v0.10.0-rc1 (2015-08-21)

Also see v0.9.4-rc1 changelog, which has been amalgamated into this release.

General:

• Upgrade to Twisted 15 (PR #173)
• Add support for serving and fetching encryption keys over federation. (PR #208)
• Add support for logging in with email address (PR #234)
• Add support for new m.room.canonical_alias event. (PR #233)
• Change synapse to treat user IDs case insensitively during registration and login. (If two users already exist with case insensitive matching user ids, synapse will continue to require them to specify their user ids exactly.)
• Error if a user tries to register with an email already in use. (PR #211)
• Add extra and improve existing caches (PR #212, #219, #226, #228)
• Batch various storage request (PR #226, #228)
• Fix bug where we didn't correctly log the entity that triggered the request if the request came in via an application service (PR #230)
• Fix bug where we needlessly regenerated the full list of rooms an AS is interested in. (PR #232)
• Add support for AS's to use v2_alpha registration API (PR #210)

Configuration:

• Add --generate-keys that will generate any missing cert and key files in the configuration files. This is equivalent to running --generate-config on an existing configuration file. (PR #220)
• --generate-config now no longer requires a --server-name parameter when used on existing configuration files. (PR #220)
• Add --print-pidfile flag that controls the printing of the pid to stdout of the demonised process. (PR #213)

Media Repository:

• Fix bug where we picked a lower resolution image than requested. (PR #205)
• Add support for specifying if a the media repository should dynamically thumbnail images or not. (PR #206)

Metrics:

• Add statistics from the reactor to the metrics API. (PR #224, #225)

Demo Homeservers:

• Fix starting the demo homeservers without rate-limiting enabled. (PR #182)
• Fix enabling registration on demo homeservers (PR #223)

Changes in synapse v0.9.4-rc1 (2015-07-21)

General:

• Add basic implementation of receipts. (SPEC-99)
• Add support for configuration presets in room creation API. (PR #203)
• Add auth event that limits the visibility of history for new users. (SPEC-134)
• Add SAML2 login/registration support. (PR #201. Thanks Muthu Subramanian!)
• Add client side key management APIs for end to end encryption. (PR #198)
• Change power level semantics so that you cannot kick, ban or change power levels of users that have equal or greater power level than you. (SYN-192)
• Improve performance by bulk inserting events where possible. (PR #193)
• Improve performance by bulk verifying signatures where possible. (PR #194)

Configuration:

• Add support for including TLS certificate chains.

Media Repository:

• Add Content-Disposition headers to content repository responses. (SYN-150)

Hi all,

We seem to have done that thing again where we were too busy writing new stuff (E2E crypto, all-new React-based Web SDK, long-awaited iOS Console update) to remember that we'd released a new version of Synapse - sorry!

Synapse 0.9.3 was released on July 1st (actually on June 23rd as 0.9.3-rc1, but was released without changes). It's a fairly minor release but does provide some performance improvements and bug fixes - see below for details. Get it from http://github.com/matrix-org/synapse if you haven't already.

Changes in synapse v0.9.3 (2015-07-01) ======================================

General:

• Fix a memory leak in the notifier. (SYN-412)
• Improve performance of room initial sync. (SYN-418)
• General improvements to logging.
• Remove access_token query params from INFO level logging.

Configuration:

• Add support for specifying and configuring multiple listeners. (SYN-389)

Application services:

• Fix bug where synapse failed to send user queries to application services.