Category – Tech
51 posts tagged with "Tech" (See all categories)

Olm 3.0.0 released!

2018-10-25 — Tech — 

Olm 3.0.0 has been released, which features several big changes. It can be downloaded from https://git.matrix.org/git/olm/. The npm package for JavaScript can be downloaded from https://matrix.org/packages/npm/olm/olm-3.0.0.tgz

Python

The biggest change is the merge of poljar's improved Python bindings. These bindings should be much easier to use for Python programmers, and are used by Zil0's E2E support in the Matrix Python SDK.

Since the binding API has changed, existing Python code will need to be rewritten in order to work with this release.

poljar has also included comprehensive documentation for the new API.

CMake

mujx contributed support for building olm using CMake. This should allow for easier building on different platforms. Currently the library can be built using either make or CMake. In the future, make support may be removed.

JavaScript

The JavaScript bindings now use WebAssembly by default. WebAssembly is much faster than the previous asm.js build, and is supported by recent versions of the most popular browsers. For compatibility with browsers that do not support WebAssembly, the asm.js version is still provided.

Due to adding support for WebAssembly, the API had to be changed slightly. There is now an init function that must be called before the library can be used. This function will return a promise that will resolve once the library is ready to be used. The matrix-js-sdk has not yet been updated to do this, so users of matrix-js-sdk should continue using olm 2.x until it has been updated.

Key backups

The public key API has been updated to support the proposal for server-side key backups. More details on how to use these functions will be published in the future.

Matrix Spec Update August 2018

2018-09-03 — Tech — 

Introducing Client Server API 0.4, and the first ever stable IS, AS and Push APIs spec releases!

Hi folks,As many know, we've been on a massive sprint to improve the spec - both fixing omissions where features have been implemented in the reference servers but were never formalised in the spec, and fixing bugs where the spec has thinkos which stop us from being able to ratify it as stable and thus fit for purpose .In practice, our target has been to cut stable releases of all the primary Matrix APIs by the end of August - effectively declaring Matrix out of beta, at least at the specification level.  For context: historically only one API has ever been released as stable - the Client Server API, which was the result of a similar sprint back in Jan 2016. This means that the Server Server (SS) API, Identity Service (IS) API, Application Service (AS) API and Push Gateway API have never had an official stable release - which has obviously been problematic for those implementing them.However, as of the end of Friday Aug 31, we're proud to announce the first ever stable releases of the IS, AS and Push APIs!

To the best of our knowledge, these API specs are now complete and accurately describe all the current behaviour implemented in the reference implementations (sydent, synapse and sygnal) and are fit for purpose. Any deviation from the spec in the reference implementations should probably be considered a bug in the impl. All changes take the form of filling in spec omissions and adding clarifications to the existing behaviour in order to get things to the point that an independent party can implement these APIs without having to refer to anything other than the spec.This is the result of a lot of work which spans the whole Spec Core Team, but has been particularly driven by TravisR, who has taken the lead on this whole mission to improve the spec.  Huge thanks are due to Travis for his work here, and also massive thanks to everyone who has suffered endured reviewed his PRs and contributed to the releases.  The spec is looking unrecognisably better for it - and Matrix 1.0 is feeling closer than ever!Alongside the work on the IS/AS/Push APIs, there has also been a massive attempt to plug all the spec omissions in the Client Server API.  Historically the CS API releases have missed some of the newer APIs (and of course always miss the ones which postdate a given release), but we've released the APIs which /have/ been specified as stable in order to declare them stable.  However, in this release we've tried to go through and fill in as many remaining gaps as possible.The result is the release of Client Server API version 0.4. This is a huge update - increasing the size of the CS API by ~40%. The biggest new stuff includes fully formalising support for end-to-end encryption (thanks to Zil0!), versioning for rooms (so we can upgrade rooms to new versions of the protocol), synchronised read markers, user directories, server ACLs, MSISDN 3rd party ids, and .well-known server discovery (not that it's widely used yet), but for the full picture, best bet is to look at the changelog (now managed by towncrier!).  It's probably fair to say that the CS API is growing alarmingly large at this point - Chrome says that it'd be 223 A4 pages if printed. Our solution to this will be to refactor it somehow (and perhaps switch to a more compact representation of the contents).Some things got deliberately missed from the CS 0.4 release: particularly membership Lazy Loading (because we're still testing it out and haven't released it properly in the wild yet), the various GDPR-specific APIs (because they may evolve a bit as we refine them since the original launch), finalising ID grammars in the overall spec (because this is surprisingly hard and subtle and we don't want to rush it) and finally Communities (aka Groups), as they are still somewhat in flux.Meanwhile, on the Server to Server API, there has also been a massive amount of work.  Since the beginning of July it's tripled in size as we've filled in the gaps, over the course of >200 commits (>150 of which from Travis).  If you take a look at the current snapshot it's pretty unrecognisable from the historical draft; with the main changes being:
  • Adding the new State Resolution algorithm to address flaws in the original one.  This has been where much of our time has gone - see MSC1442 for full details.  Adopting the new algorithm requires rooms to be recreated; we'll write more about this in the near future when we actually roll it out.
  • Adding room versioning so we can upgrade to the new State Resolution algorithm.
  • Everything is now properly expressed as Swagger (OpenAPI), just like the CS API
  • Adding all the details for E2E encryption (including dependencies like to-device messaging and device-list synchronisation)
  • Improvements in specifying how to authorize inbound events over federation
  • Document federation APIs such as /event_auth and /query_auth and /get_missing_events
  • Document 3rd party invites over federation
  • Document the /user/* federation endpoints
  • Document Server ACLs
  • Document read receipts over federation
  • Document presence over federation
  • Document typing notifications over federation
  • Document content repository over federation
  • Document room directory over federation
  • ...and many many other minor bug fixes, omission fixes, and restructuring for coherency - see https://github.com/matrix-org/matrix-doc/issues/1464 for an even longer list :)
However, we haven't finished it all: despite our best efforts we're running slightly past the original target of Aug 31.  The current state of play for the r0 release overall (in terms of pending issues) is:...and you can see the full breakdown over at the public Github project dashboard.The main stuff we still have remaining on the Server/Server API at this point is:
  • Better specifying how we validate inbound events. See MSC1646 for details & progress.
  • Switching event IDs to be hashes. See MSC1640 for details and progress.
  • Various other remaining security considerations (e.g. how to handle malicious auth events in the DAG; how to better handle DoS situations).
  • Merging in the changes to authoring m.room.power_levels (as per MSC1304)
  • Formally specifying the remaining identifiers which lack a formal grammar - MSC1597 and particularly room aliases (MSC1608)
The plan here is to continue speccing and implementing these at top priority (with Travis continuing to work fulltime on spec work), and we'll obviously keep you up-to-date on progress.  Some of the changes here (e.g. event IDs) are quite major and we definitely want to implement them before speccing them, so we're just going to have to keep going as fast as we can. Needless to say we want to cut an r0 of the S2S API alongside the others asap and declare Matrix out of beta (at least at the spec level :)In terms of visualising progress on this spec mission it's interesting to look at the rate at which we've been closing PRs: this graph shows the total number of PRs which are in state ‘open' or ‘closed' on any given day:

...which clearly shows the original sprint to get the r0 of the CS API out the door at the end 2015, and then a more leisurely pace until the beginning of July 2018 since which the pace has picked up massively.  Other ways of looking at include the number of open issues...

...or indeed the number of commits per week…

...or the overall Github Project activity for August.  (It's impressive to see Zil0 sneaking in there on second place on the commit count, thanks to all his GSoC work documenting E2E encryption in the spec as part of implementing it in matrix-python-sdk!)

Anyway, enough numerology.  It's worth noting that all of the dev for r0 has generally followed the proposed Open Governance Model for Matrix, with the core spec team made up of both historical core team folk (erik, richvdh, dave & matthew), new core team folk (uhoreg & travis) and community folk (kitsune, anoa & mujx) working together to review and approve the changes - and we've been doing MSCs (albeit with an accelerated pace) for anything which we feel requires input from the wider community.  Once the Server/Server r0 release is out the door we'll be finalising the open governance model and switching to a slightly more measured (but productive!) model of spec development as outlined there.Meanwhile, Matrix 1.0 gets ever closer.  With (almost) all this spec mission done, our plan is to focus more on improving the reference implementations - particularly performance in Synapse, UX in matrix-{react,ios,android}-sdk as used by Riot (especially for E2E encryption), and then declare a 1.0 and get back to implementing new features (particularly Editable Messages and Reactions) at last.We'd like to thank everyone for your patience whilst we've been playing catch up on the spec, and hope you agree it's been worth the effort :)Matthew & the core spec team.

GSOC: Implementing End-to-End Encryption in the Matrix Python SDK

2018-08-01 — Tech — 

Following on from the previous post, we have an update from zil0 on his GSoC project, which entailed implementing E2E support in the Matrix Python SDK.


The goal of my project is to implement Matrix's end-to-end encryption protocol in Python, as part of matrix-python-sdk. My mentors are Richard van der Hoff (richvdh) and Hubert Chathi (uhoreg).

It was easy to get started on the project, since the simple parts came first (adding API calls), and then the whole process to follow is documented in an implementation guide, while there is also the reference implementation in JavaScript. And most importantly, the community is nice. :)

Some parts of the work consist in wrapping around the cryptographic primitives implemented in libolm (via Python bindings), in order to handle encrypted events. Others are less straightforward, such as tracking device lists of users, or finding the right way to persist keys and related data between startups.

An interesting aspect of this project is that I am working on a new part of the Python SDK, while also having to integrate with existing code, which is a cool balance between freedom and guidelines.

As the encryption documentation is a bit outdated and incomplete, one (fun) difficulty is to look for information across old issues, Gdocs and source code (and asking my mentor when in doubt). For anyone trying to implement E2E, it should be better by the end of the project, as I am currently working on documenting the missing bits.

I have had a great experience so far. Working on an open source project differs from my previous coding experiences, as people are actually going to use what I write! I have learnt to think about the best design from a usability point of view, discuss different approaches, and I had to write tests and document my code, which sadly is not something I do on personal projects. I enjoyed reviews, and the discussions they led to. And of course I have learnt quite an interesting lot about the E2E voodoo, along with some new Python tricks.

Currently, the implementation is in a working state. Some of the code is merged, and some is awaiting review. It is possible to try it here before everything is merged. The project will be finished in about one week, after some tidying up and when I release device verification and key sharing, which should be the last missing features compared to Riot.

Dendrite Progress Update

2018-07-30 — Tech — 

As you may know, for the last few months anoa (Andrew) and APWhiteHat have been working on Dendrite, the next generation Matrix homeserver, written in Go. We asked for an update on their progress, and Andrew provided the blog post below. Serious progress has been made on Dendrite this summer!


Hey everyone, my name is Andrew Morgan and I've been working full-time over the summer on Dendrite, our next-generation Matrix homeserver. Over the last two months, I've seen the project transform from a somewhat functioning toy server to a near-production-ready homeserver that is working towards complete feature support. I've appreciated the thought put into the project since day one, and enjoy the elegance of the multi-component design. Documentation is fairly decent at the moment, but comments are plentiful throughout the codebase, while the code itself tends towards simple and maintainable rather than complex and unmanageable.

Application Service Integration

The main focus of my time here has been on the implementation of application service support for Dendrite. Application services are external programs that act as privileged extensions to a homeserver, allowing such functionality as bots in rooms and bridges to third-party networks. Supporting application services requires a few different bits and pieces to be set up. Currently all planned features have a PR for them, with the bold items already merged:
  • Sending events to application services
  • Support user masquerading for events
  • Support editing event timestamps
  • Support room alias querying
  • Support user ID querying
  • Support third party lookup proxying
As you can see a decent portion of the functionality is already in master! The rest will hopefully follow after some further back and forth.

Google Summer of Code

I certainly haven't been going at this all on my own. Alongside extensive help from Erik, who's been mentoring me, our resident Google Summer of Code student, APWhiteHat, has been tackling feature after feature in Dendrite wherever he can find them. Application services received a good deal of help on client-server endpoint authentication side, however, APWhiteHat has mostly been focusing on federation and some other very useful pieces. While his GSoC period still has a week or so before its conclusion, he has so far implemented:
  • Idempotency to roomserver event processing to prevent duplication
  • Username auto generation
  • Tokens library based on macaroons
  • Lots of left-over federation stuff: state API & get missing events being the major ones
  • AS support to clientapi auth
  • Typing server: handling of PUT /typing by clientapi
  • More typing server stuff on its way
From my perspective, APWhiteHat was an excellent developer to work with. He asked good questions and was quick to answer any myself or the community had as well. His code reviews were also very comprehensive. I learned a lot from working with him and everyone else :)

OpenTracing and Prometheus Monitoring

Placing any large server into a production environment requires extensive monitoring capabilities in order to ensure operations are running smoothly. To that effect, Dendrite has been both the addition of OpenTracing and Prometheus support. Prometheus, also used heavily in Synapse, allows a homeserver operator to track a wide range of data including endpoint usage, resource management as well as user statistics over any given range of time.In Dendrite, we are taking this one step further by introducing OpenTracing, a language and platform-agnostic framework for tracking the journey of an endpoint call from incoming request to outgoing response, with every method, hierarchy change and database call in between. It will be immensely useful in tracking down performance issues, as well as providing insight into the most critical paths throughout the codebase and where we should focus most of our optimization efforts on. It also comes with a lovely dashboard courtesy of Jaeger:

Community

We've also seen some encouraging interest and development work from the community in the past couple months. While PR review from our own side is admittedly slow due to our focus on getting the foundational work in place, that hasn't stopped both old and new developers from sending in PRs and performing code reviews. A huge thank you to everyone involved! From this we've gotten API implementations and application service fixes from @turt2live, an end-to-end encryption implementation from @fadeAce, filtering support from @CromFr, and some PRs and numerous helpful review comments from @krombel.We've also started to see some people running Dendrite in live environments, which is incredibly exciting for us to see! While Dendrite is not considered production-ready yet (though it moves closer every day), if you are interested in giving it a go please consult the quickstart installation guide. We look forward to any feedback you may have!

Synapse 0.33.0 is here!!

2018-07-19 — Tech — 

Hi all,

We've just released Synapse 0.33.0!  This is a major performance upgrade which speeds up /sync (i.e. receiving messages) by a factor of almost 2x!  This has already made a massive difference to the CPU usage and snappiness of the matrix.org homeserver since we rolled it out a few days ago - you can see the drop in sync worker CPU just before midday on July 17th; previously we were regularly hitting the CPU ceiling (at which point everything grinds to a halt) - now we're back down hovering between 40% and 60% CPU (at the current load).  This is actually fixing a bug which crept in around Synapse 0.31, so please upgrade - especially if Synapse has been feeling slower than usual recently, and especially if you are still on Synapse 0.31.

Meanwhile we have a lot of new stuff coming on the horizon - a whole new algorithm for state resolution (watch this space for details); incremental state resolution (at last!) to massively speed up state resolution and mitigate extremities build up (and speed up the synapse master process, which is now the bottleneck again on the matrix.org homeserver); better admin tools for managing resource usage, and all the Python3 porting work (with associated speedups and RAM & GC improvements).  Fun times ahead!

The full changelog follows below; as always you can grab Synapse from https://github.com/matrix-org/synapse.   Thanks for flying Matrix!

Synapse 0.33.0 (2018-07-19)

Bugfixes

  • Disable a noisy warning about logcontexts. (#3561)

Synapse 0.33.0rc1 (2018-07-18)

Features

  • Enforce the specified API for report_event. (#3316)
  • Include CPU time from database threads in request/block metrics. (#3496#3501)
  • Add CPU metrics for _fetch_event_list. (#3497)
  • Optimisation to make handling incoming federation requests more efficient. (#3541)

Bugfixes

  • Fix a significant performance regression in /sync. (#3505#3521#3530#3544)
  • Use more portable syntax in our use of the attrs package, widening the supported versions. (#3498)
  • Fix queued federation requests being processed in the wrong order. (#3533)
  • Ensure that erasure requests are correctly honoured for publicly accessible rooms when accessed over federation. (#3546)

Misc

Security update: Synapse 0.31.2

2018-06-14 — Tech — 

Hi all,

On Monday (2018-06-11) we had an incident where #matrix:matrix.org was hijacked by a malicious user pretending to join the room immediately after its creation in 2014 and then setting an m.room.power_levels event ‘before' the correct initial power_level for the room.

Under normal circumstances this should be impossible because the initial m.room.power_levels for a room should be set before its m.room.join_rules event, meaning users who join the room are subject to its power levels. However, back before we'd even released Synapse, the first two rooms ever created in Matrix (#test:matrix.org and #matrix:matrix.org) were manually created and set the join_rules before the power_levels event, letting users join before the room's power_levels were defined, and so were vulnerable to this attack. We've since re-created #matrix:matrix.org - please re-/join the room if you haven't already!

As a defensive measure, we are releasing a security update of Synapse (0.31.2) today which changes the rules used to authenticate power_level events, such that we fail-safe rather than fail-deadly if the existing auth mechanisms fail. In practice this means changing the default power level required to set state to be 50 rather than 0 if there is no power_levels event present, thus meaning that only the room creator can set the initial power_levels event.

We are not aware of anyone abusing this (other than the old #matrix:matrix.org room) but we'd rather be safe than sorry, so would recommend that everyone upgrade as soon as possible.

This of course constitutes a change to the spec, so full technical details and ongoing discussion around the Matrix Spec Change proposal can be followed over at MSC1304.

EDIT: if you are aware of your server participating in rooms whose first power_levels event is deliberately set by a different user to their creator, please let us know asap (and don't upgrade!)

This work is all part of a general push to finalise and harden and fully specify the Server-Server API as we push towards a long-awaited stable release of Matrix!

As always, you can get the new update from https://github.com/matrix-org/synapse/releases/tag/v0.31.2 or from any of the sources mentioned at https://github.com/matrix-org/synapse.

thanks, and apologies for the inconvenience.

Changes in synapse v0.31.2 (2018-06-14)

SECURITY UPDATE: Prevent unauthorised users from setting state events in a room when there is no m.room.power_levels event in force in the room. (PR #3397)

Discussion around the Matrix Spec change proposal for this change can be followed at https://github.com/matrix-org/matrix-doc/issues/1304.

Introducing Matrix Specification Changes

2018-05-17 — Tech — 

Hi all,

We've been able to start investing more time in advancing the Matrix Specification itself over the last month or so thanks to Ben joining the core team (and should be able to accelerate even more with uhoreg joining in a few weeks!)  The first step in the new wave of work has been to provide much better infrastructure for the process of actually evolving the spec - whether that's from changes proposed by the core team or the wider Matrix Community.

So, without further ado, we'd like to introduce https://matrix.org/docs/spec/proposals - a dashboard for all the spec change proposals we've accumulated so far (ignoring most of the ones which have already been merged), as well as a clearer workflow for how everyone can help improve the Matrix spec itself.  Part of this is introducing a formal numbering system - e.g. MSC1228 stands for Matrix Spec Change 1228 (where 1228 is the ID of the Github issue on the github.com/matrix-org/matrix-doc/issues repository that tracks the proposal).

Please note that these are NOT like XEPs or RFCs - i.e. optional proposals or add-ons to the protocol; instead they are literally proposals for changes to the Matrix Spec itself.  Once merged into the spec, they are only of historical interest.

We've also created a new room: #matrix-spec:matrix.org to discuss specific spec proposal changes - please join if you want to track how proposals are evolving! (Conversation is likely to fork off into per-proposal rooms or overflow into #matrix-dev:matrix.org or #matrix-architecture:matrix.org depending on traffic levels, however).

Feedback would be much appreciated on this - so please head over to #matrix-spec:matrix.org and let us know how it feels and how it could do better.

This is also a major step towards properly formalising Matrix.org's governance model - hopefully the changes above are sufficient to improve the health of the evolution of the Spec as we work towards an initial stable release later this year, and then you should expect to see a spec proposal for formal governance once we've (at last!) exited beta :)

Huge thanks to Ben for putting this together, and thanks to everyone who's contributed so far to the spec - we're looking forward to working through the backlog of proposals and turning them all into merged spec PRs!!

Matthew

SECURITY UPDATE: Synapse 0.28.1

2018-05-01 — Tech — 

Hi all,

Many people will have noticed disruption in #matrix:matrix.org and #matrix-dev:matrix.org on Sunday, when a validation bug in Synapse was exploited which allowed a malicious event to be inserted into the room with 'depth' value that made the rooms temporarily unusable. Whilst a transient workaround was found at the time (thanks to /dev/ponies, kythyria and Po Shamil for the workaround and to Half-Shot for working on a proposed fix), we're doing an urgent release of Synapse 0.28.1 to provide a temporary solution which will mitigate the attack across all rooms in upgraded servers and un-break affected ones.  Meanwhile we have a full long-term fix on the horizon (hopefully) later this week.

This vulnerability has already been exploited in the wild; please upgrade as soon as possible.

Synapse 0.28.1 is available from https://github.com/matrix-org/synapse/releases/tag/v0.28.1 as normal.

The 'depth' parameter is used primarily as a way for servers to signal the intended cosmetic order of their events within a room (particularly when the room's message graph has gaps in it due to the server being offline, or due to users backfilling old disconnected chunks of conversation). This means that affected rooms may experience message ordering problems until a full long-term fix is provided, which we're working on currently (and tentatively involves no longer trusting 'depth' information from servers).  For full details you can see the proposal documents for the temporary fix in 0.28.1 and the options for the imminent long-term fix.

We'd like to acknowledge jzk for identifying the vulnerability, and Max Dor for providing feedback on the fixes.

As a general reminder, Synapse is still beta (as is the Matrix spec) and the federation API particularly is still being debugged and refined and is pre-r0.0.0. For the benefit of the whole community, please disclose vulnerabilities and exploits responsibly by emailing [email protected] or DMing someone from +matrix:matrix.org. Thanks.

Changes in synapse v0.28.1 (2018-05-01)

SECURITY UPDATE
  • Clamp the allowed values of event depth received over federation to be [0, 2^63 - 1]. This mitigates an attack where malicious events injected with depth = 2^63 - 1 render rooms unusable. Depth is used to determine the cosmetic ordering of events within a room, and so the ordering of events in such a room will default to using stream_ordering rather than depth (topological_ordering). This is a temporary solution to mitigate abuse in the wild, whilst a long solution is being implemented to improve how the depth parameter is used.Full details at https://docs.google.com/document/d/1I3fi2S-XnpO45qrpCsowZv8P8dHcNZ4fsBsbOW7KABI
  • Pin Twisted to <18.4 until we stop using the private _OpenSSLECCurve API.

Security update: Riot/Web 0.13.5 released - fixing XSS vulnerability

2018-02-09 — Tech — 

Hi all,

Heads up that we made an emergency release of Riot/Web 0.13.5 a few hours ago to fix a XSS vulnerability found and reported by walle303 - many thanks for disclosing it responsibly.

Please upgrade to Riot/Web 0.13.5 asap. If you're using riot.im/app or riot.im/develop this simply means hitting Refresh; otherwise please upgrade your Riot deployment as soon as possible. Alpine, Debian and Fedora/RPM packages are already updated - huge thanks to the maintainers for the fast turnaround.

The issue lies in the relatively obscure external_url feature, which lets bridges specify a URL for bridged events, letting Riot/Web users link through to the 'original' event (e.g. a twitter URL on a bridged tweet).  The option is hidden in a context menu and labelled "Source URL", and is only visible on events which have the external_url field set.  Unfortunately Riot/Web didn't sanitise the URL correctly, allowing a malicious URL to be injected - and this has been the case since the feature landed in Riot 0.9.0 (Nov 2016).

If you're not able to upgrade to Riot/Web 0.13.5 for some reason, then please do not click on the 'Source URL' feature on the event context menu:

Apologies for the inconvenience,

thanks,

Matthew

Synapse 0.26 released!

2018-01-05 — Tech — 

Hi folks,

Synapse 0.26 is here (with no changes since RC1 which we released just before Christmas).  It's a general maintenance release, albeit with a few new features but mainly lots of bugfixes and general refinements.  Enjoy!

As always, you can get it from https://github.com/matrix-org/synapse/releases/tag/v0.26.0.

Changes in synapse v0.26.0 (2018-01-05)

No changes since v0.26.0-rc1

Changes in synapse v0.26.0-rc1 (2017-12-13)

Features:

  • Add ability for ASes to publicise groups for their users (PR #2686)
  • Add all local users to the user_directory and optionally search them (PR#2723)
  • Add support for custom login types for validating users (PR #2729)
Changes:
  • Update example Prometheus config to new format (PR #2648) Thanks to@krombel!
  • Rename redact_content option to include_content in Push API (PR #2650)
  • Declare support for r0.3.0 (PR #2677)
  • Improve upserts (PR #2684#2688#2689#2713)
  • Improve documentation of workers (PR #2700)
  • Improve tracebacks on exceptions (PR #2705)
  • Allow guest access to group APIs for reading (PR #2715)
  • Support for posting content in federation_client script (PR #2716)
  • Delete devices and pushers on logouts etc (PR #2722)
Bug fixes:
  • Fix database port script (PR #2673)
  • Fix internal server error on login with ldap_auth_provider (PR #2678) Thanks to @jkolo!
  • Fix error on sqlite 3.7 (PR #2697)
  • Fix OPTIONS on preview_url (PR #2707)
  • Fix error handling on dns lookup (PR #2711)
  • Fix wrong avatars when inviting multiple users when creating room (PR #2717)
  • Fix 500 when joining matrix-dev (PR #2719)

Synapse 0.25 is out... as is Matrix Specification 0.3(!!!)

2017-11-15 — Tech — 

Hi all,

Today is a crazy release day here - not only do we have Synapse 0.25, but we've also made a formal release of the Matrix Specification (CS API) for the first time in 16 months!

Matrix CS API 0.3

Talking first about the spec update: the workflow of the Matrix spec is that new experimental features get added to an /unstable API prefix, and then whenever we release the Matrix spec, these get moved over to being part of the /r0 prefix (or whatever version we happen to be on).  We've been very constrained on manpower to work on the spec over the last ~18 months, but we've been keeping it up-to-date on a best effort basis, with a bit of help from the wider community.   As such, this latest release does not contain all the latest APIs (and certainly not experimental ones like Groups/Communities which are still evolving), but it does release all of the unstable ones which we've managed to document and which are considered stable enough to become part of the 'r0' prefix.  Going forwards, we're hoping that the wider community will help us fill in the remaining gaps (i.e. propose PRs against the matrix-org/matrix-doc repository to formalise the various spec drafts flying around the place) - and we're also hoping (if/when funding crisis is abated) to locate full-time folk to work on the spec.

The full changelog for 0.3 of the spec is:

  • Breaking changes:
    • Change the rule kind of .m.rule.contains_display_name from underride to override. This works with all known clients which support push rules, but any other clients implementing the push rules API should be aware of this change. This makes it simple to mute rooms correctly in the API (#373).
    • Remove /tokenrefresh from the API (#395).
    • Remove requirement that tokens used in token-based login be macaroons (#395).
  • Changes to the API which will be backwards-compatible for clients:
    • Add filename parameter to POST /_matrix/media/r0/upload (#364).
    • Document CAS-based client login and the use of m.login.token in /login (#367).
    • Make origin_server_ts a mandatory field of room events (#379).
    • Add top-level account_data key to the responses to GET /sync and GET /initialSync (#380).
    • Add is_direct flag to POST /createRoom and invite member event. Add 'Direct Messaging' module (#389).
    • Add contains_url option to RoomEventFilter (#390).
    • Add filter optional query param to /messages (#390).
    • Add 'Send-to-Device messaging' module (#386).
    • Add 'Device management' module (#402).
    • Require that User-Interactive auth fallback pages call window.postMessage to notify apps of completion (#398).
    • Add pagination and filter support to /publicRooms. Change response to omit fields rather than return null. Add estimate of total number of rooms in list. (#388).
    • Allow guest accounts to use a number of endpoints which are required for end-to-end encryption. (#751).
    • Add key distribution APIs, for use with end-to-end encryption. (#894).
    • Add m.room.pinned_events state event for rooms. (#1007).
    • Add mention of ability to send Access Token via an Authorization Header.
    • New endpoints:
      • GET /joined_rooms (#999).
      • GET /rooms/{roomId}/joined_members (#999).
      • GET /account/whoami (#1063).
      • GET /media/{version}/preview_url (#1064).
  • Spec clarifications:
    • Add endpoints and logic for invites and third-party invites to the federation spec and update the JSON of the request sent by the identity server upon 3PID binding (#997)
    • Fix "membership" property on third-party invite upgrade example (#995)
    • Fix response format and 404 example for room alias lookup (#960)
    • Fix examples of m.room.member event and room state change, and added a clarification on the membership event sent upon profile update (#950).
    • Spell out the way that state is handled by POST /createRoom (#362).
    • Clarify the fields which are applicable to different types of push rule (#365).
    • A number of clarifications to authentication (#371).
    • Correct references to user_id which should have been sender (#376).
    • Correct inconsistent specification of redacted_because fields and their values (#378).
    • Mark required fields in response objects as such (#394).
    • Make m.notice description a bit harder in its phrasing to try to dissuade the same issues that occurred with IRC (#750).
    • GET /user/{userId}/filter/{filterId} requires authentication (#1003).
    • Add some clarifying notes on the behaviour of rooms with no m.room.power_levels event (#1026).
    • Clarify the relationship between username and user_id in the /register API (#1032).
    • Clarify rate limiting and security for content repository. (#1064).
...and you can read the spec itself of course over at https://matrix.org/docs/spec.  It's worth noting that we have slightly bent the rules by including three very minor 'breaking changes' in 0.3, but all for features which to our knowledge nobody is depending on in the wild.  Technically this should mean bumping the major version prefix (i.e. moving to r1), but given how minor and nonimpacting these are we're turning a blind eye this time.

Meanwhile, Synapse 0.25 is out!

This is a medium-sized release; the main thing being to support configurable room visibility within groups (so that whenever you add a room to a group, you're not forced into sharing their existence with the general public, but can choose to just tell group members about them).  There's also a bunch of useful bug fixes and some performance improvements, including lots of contributions from the community this release (thank you!).  Full release notes are:
Changes in synapse v0.25.0 (2017-11-15)
Bug fixes:
  • Fix port script (PR #2673)
Changes in synapse v0.25.0-rc1 (2017-11-14)
Features:Changes:
  • Ignore tags when generating URL preview descriptions (PR #2576) Thanks to @maximevaillancourt!
  • Register some /unstable endpoints in /r0 as well (PR #2579) Thanks to@krombel!
  • Support /keys/upload on /r0 as well as /unstable (PR #2585)
  • Front-end proxy: pass through auth header (PR #2586)
  • Allow ASes to deactivate their own users (PR #2589)
  • Remove refresh tokens (PR #2613)
  • Automatically set default displayname on register (PR #2617)
  • Log login requests (PR #2618)
  • Always return is_public in the /groups/:group_id/rooms API (PR #2630)
  • Avoid no-op media deletes (PR #2637) Thanks to @spantaleev!
  • Fix various embarrassing typos around user_directory and add some doc. (PR#2643)
  • Return whether a user is an admin within a group (PR #2647)
  • Namespace visibility options for groups (PR #2657)
  • Downcase UserIDs on registration (PR #2662)
  • Cache failures when fetching URL previews (PR #2669)
Bug fixes:
  • Fix port script (PR #2577)
  • Fix error when running synapse with no logfile (PR #2581)
  • Fix UI auth when deleting devices (PR #2591)
  • Fix typo when checking if user is invited to group (PR #2599)
  • Fix the port script to drop NUL values in all tables (PR #2611)
  • Fix appservices being backlogged and not receiving new events due to a bug in notify_interested_services (PR #2631) Thanks to @xyzz!
  • Fix updating rooms avatar/display name when modified by admin (PR #2636) Thanks to @farialima!
  • Fix bug in state group storage (PR #2649)
  • Fix 500 on invalid utf-8 in request (PR #2663)

Finally...

If you haven't noticed already, Riot/Web 0.13 is out today, as is Riot/iOS 0.6.2 and Riot/Android 0.7.4.  These contain massive improvements across the board - particularly mainstream Communities support at last on Riot/Web; CallKit/PushKit on Riot/iOS thanks to Denis Morozov (GSoC 2017 student for Matrix) and Share Extension on iOS thanks to Aram Sargsyan (also GSoC 2017 student!); and End-to-end Key Sharing on Riot/Android and a full rewrite of the VoIP calling subsystem on Android.

Rather than going on about it here, though, there's a full write-up over on the Riot Blog.

And so there you go - new releases for eeeeeeeeveryone!  Enjoy! :)

--Matthew, Amandine & the team.

Synapse 0.22.0 released!

2017-07-06 — Tech — 

Hi Synapsefans,

Synapse 0.22.0 has just been released! This release lands a few interesting features:

  • The new User directory API which supports Matrix clients' providing a much more intuitive and effective user search capability by exposing a list of:
    • Everybody your user shares a room with, and
    • Everybody in a public room your homeserver knows about
  • New support for server admins, including a Shutdown Room API (to remove a room from a local server) and a Media Quarrantine API (to render a media item inaccessible without its actually being deleted)
As always there are lots of bug fixes and performance improvements, including increasing the default cache factor size from 0.1 to 0.5 (should improve performance for those running their own homeservers).

You can get Synapse 0.22.0 from https://github.com/matrix-org/synapse or https://github.com/matrix-org/synapse/releases/tag/v0.22.0 as normal.

Changes in synapse v0.22.0 (2017-07-06)

No changes since v0.22.0-rc2

Changes in synapse v0.22.0-rc2 (2017-07-04)

Changes:

  • Improve performance of storing user IPs (PR #2307, #2308)
  • Slightly improve performance of verifying access tokens (PR #2320)
  • Slightly improve performance of event persistence (PR #2321)
  • Increase default cache factor size from 0.1 to 0.5 (PR #2330)
Bug fixes:
  • Fix bug with storing registration sessions that caused frequent CPU churn (PR #2319)

Changes in synapse v0.22.0-rc1 (2017-06-26)

Features:
  • Add a user directory API (PR #2252, and many more)
  • Add shutdown room API to remove room from local server (PR #2291)
  • Add API to quarantine media (PR #2292)
  • Add new config option to not send event contents to push servers (PR #2301) Thanks to @cjdelisle!
Changes:Bug fixes:
  • Fix users not getting notifications when AS listened to that user_id (PR#2216) Thanks to @slipeer!
  • Fix users without push set up not getting notifications after joining rooms (PR #2236)
  • Fix preview url API to trim long descriptions (PR #2243)
  • Fix bug where we used cached but unpersisted state group as prev group, resulting in broken state of restart (PR #2263)
  • Fix removing of pushers when using workers (PR #2267)
  • Fix CORS headers to allow Authorization header (PR #2285) Thanks to @krombel!
 

Synapse 0.21.1 released!

2017-06-22 — Tech — 

Hi folks - we forgot to mention that Synapse 0.21.1 was released last week.  This contains a important fix to the report-stats option, which was otherwise failing to report any usage stats for folks who had the option turned on.

This is a good opportunity to note that the report-stats option is really really important for the ongoing health of the Matrix ecosystem: when raising funding to continue working on Matrix we have to be able to demonstrate how the ecosystem is growing and why it's a good idea to support us to work on it. In practice, the data we collect is: hostname, synapse version & uptime, total_users, total_nonbridged users, total_room_count, daily_active_users, daily_active_rooms, daily_messages and daily_sent_messages.

Folks: if you have turned off report-stats for whatever reason, please consider upgrading to 0.21.1 and turning it back on.

In return, the plan is that we'll start to publish an official Grafana of the anonymised aggregate stats, probably embedded into the frontpage of Matrix.org, and then you and everyone else can have a view of the state of the Matrix universe. And critically, it'll really help us continue to justify $ to spend on growing the project!

You can get Synapse 0.21.1 from https://github.com/matrix-org/synapse or https://github.com/matrix-org/synapse/releases/tag/v0.21.1 as normal.

Synapse 0.21.0 is released!

2017-05-18 — Tech — 

Hi all,

Synapse 0.21.0 was released a moment ago. This release lands a number of performance improvements and stability fixes, plus a couple of small features.

For those of you upgrading https://github.com/matrix-org/synapse has the details as usual.  Full changelog follows:

Changes in synapse v0.21.0 (2017-05-17)

Features:

  • Add per user rate-limiting overrides (PR #2208)
  • Add config option to limit maximum number of events requested by /sync and /messages (PR #2221) Thanks to @psaavedra!
Changes:
  • Various small performance fixes (PR #2201, #2202, #2224, #2226, #2227, #2228, #2229)
  • Update username availability checker API (PR #2209, #2213)
  • When purging, don't de-delta state groups we're about to delete (PR #2214)
  • Documentation to check synapse version (PR #2215) Thanks to @hamber-dick!
  • Add an index to event_search to speed up purge history API (PR #2218)
Bug fixes:
  • Fix API to allow clients to upload one-time-keys with new sigs (PR #2206)

Changes in synapse v0.21.0-rc2 (2017-05-08)

Changes:
  • Always mark remotes as up if we receive a signed request from them (PR #2190)
Bug fixes:
  • Fix bug where users got pushed for rooms they had muted (PR #2200)

Changes in synapse v0.21.0-rc1 (2017-05-08)

Features:
  • Add username availability checker API (PR #2183)
  • Add read marker API (PR #2120)
Changes:Bug fixes:
  • Fix nuke-room script to work with current schema (PR #1927) Thanks @zuckschwerdt!
  • Fix db port script to not assume postgres tables are in the public schema (PR #2024) Thanks @jerrykan!
  • Fix getting latest device IP for user with no devices (PR #2118)
  • Fix rejection of invites to unreachable servers (PR #2145)
  • Fix code for reporting old verify keys in synapse (PR #2156)
  • Fix invite state to always include all events (PR #2163)
  • Fix bug where synapse would always fetch state for any missing event (PR #2170)
  • Fix a leak with timed out HTTP connections (PR #2180)
  • Fix bug where we didn't time out HTTP requests to ASes (PR #2192)
Docs:
  • Clarify doc for SQLite to PostgreSQL port (PR #1961) Thanks @benhylau!
  • Fix typo in synctl help (PR #2107) Thanks @HarHarLinks!
  • web_client_location documentation fix (PR #2131) Thanks @matthewjwolff!
  • Update README.rst with FreeBSD changes (PR #2132) Thanks @feld!
  • Clarify setting up metrics (PR #2149) Thanks @encks!

Update on Matrix.org homeserver reliability

2017-05-12 — Tech — 

Hi folks,

We've had a few outages over the last week on the Matrix.org homeserver which have caused problems for folks using bridges or accounts hosted on matrix.org itself - we'd like to apologise to everyone who's been caught in the crossfire.  In the interests of giving everyone visibility on what's going on and what we're doing about it (and so folks can learn from our mistakes! :), here's a quick writeup (all times are UTC):

  • 2017-05-04 21:05: The datacenter where we host matrix.org performs an emergency unscheduled upgrade of the VM host where the main matrix.org HS & DB master lives.  This means a live-migration of the VM onto another host, which freezes the (huge) VM for 9 minutes, during which service is (obviously) down.  Monitoring fires; we start investigating and try to get in on the console, but by the point we're considering failing over to the hot-spare, the box has come back and recovers fine other than a load spike as all the traffic catches up.  The clock however is off by 9 minutes due to its world having paused.
  • 2017-05-04 22:30: We step NTP on the host to fix the clock (maximum clock skew on ISC ntpd is 500ppm, meaning it would take weeks to reconverge naturally, during which time we're issuing messages with incorrect timestamps).
  • 2017-05-05 01:25: Network connectivity breaks between the matrix.org homeserver and the DC where all of our bridges/bots are hosted.
  • 2017-05-05 01:40: Monitoring alerts fire for bridge traffic activity and are picked up.  After trying to restart the VPN between the DC a few times, it turns out that the IP routes needed for the VPN have mysteriously disappeared.
  • 2017-05-05 02:23: Routes are manually readded and VPN recovers and traffic starts flowing again.  It turns out that the routes are meant to be maintained by a post-up script in /etc/network/interfaces, which assumes that /sbin/ip is on the path.  Historically this hasn't been a problem as the DHCP lease on the host has never expired (only been renewed every 6 hours) - but the time disruption caused by the live-migration earlier means that on this renewal cycle the lease actually expires and the routes are lost and not-readded.  Basic bridging traffic checks are done (e.g. Freenode->Matrix).
  • 2017-05-05 08:30: Turns out that whilst Freenode->Matrix traffic was working, Matrix->Freenode was wedged due to a missing HTTP timeout in the AS logic on Synapse.  Synapse is restarted and the bug fixed.
  • ...the week goes by...
  • 2017-05-11 18:00: (Unconnected to the rest of this outage, an IRC DDoS on GIMPnet cause intermittent load problems and delayed messages on matrix.org; we turn off the bridge for a few hours until it subsides).
  • 2017-05-12 02:50: The postgres partition on the matrix.org DB master diskfills and postgres halts.  Monitoring alerts fire (once, phone alerts), but the three folks on call manage to sleep through their phone ringing.
  • 2017-05-12 04:45: Folks get woken up and notice the outage; clear up diskspace; restart postgres. Meanwhile, synapse appears to recover, other than mysteriously refusing to send messages from local users.  Investigation commences in the guts of the database to try to see what corruption has happened.
  • 2017-05-12 06:00: We realise that nobody has actually restarted synapse since the DB outage begun, and the failure is probably a known issue where worker replication can get fail and cause the master synapse process to fail to process writes.  Synapse is restarted; everything recovers (including bridges).
  • 2017-05-12 06:20: Investigation into the cause of the diskfill reveals it to be due to postgres replication logs (WALs) stacking up on the DB master, due to replication having broken to a DB slave during the previous networking outage.  Monitoring alerts triggered but weren't escalated due to a problem in PagerDuty.
Lessons learned:
  • Test your networking scripts and always check your box self-recovers after a restart (let alone a DHCP renewal).
  • Don't use DHCP in production datacenters unless you really have no choice; it just adds potential ways for everything to break.
  • We need better end-to-end monitoring for bridged traffic.
  • We need to ensure HS<->Bridge traffic is more reliable (improved by fixing timeout logic in synapse).
  • We need better monitoring and alerting of DB replication traffic.
  • We need to escalate PagerDuty phone alerts more aggressively (done).
  • We need better alerting for disk fill thresholds (especially "time until fill", remembering to take into account the emergency headroom reserved by the filesystem for the superuser).
  • We should probably have scripts to rapidly (or even automatedly) switch between synapse master & hot-spare, and to promote DB slaves in the event of a master failure.
Hopefully this is the last we've seen of this root cause; we'll be working through the todo list above.  Many apologies again for the instability - however please do remember that you can (and should!) run your own homeserver & bridges to stay immune to whatever operational dramas we have with the matrix.org instance!

Synapse 0.20.0 is released!

2017-04-11 — Tech — 

Hi folks,

Synapse 0.20.0 was released a few hours ago - this is a major new release with loads of stability and performance fixes and some new features too. The main headlines are:

  • Support for using phone numbers as 3rd party identifiers as well as email addresses!  This is huge: letting you discover other users on Matrix based on whether they've linked their phone number to their matrix account, and letting you log in using your phone number as your identifier if you so desire.  Users of systems like WhatsApp should find this both familiar and useful ;)
  • Fixes some very nasty failure modes where the state of a room could be reset if a homeserver received an event it couldn't verify.  Folks who have suffered rooms suddenly losing their name/icon/topic should particularly upgrade - this won't fix the rooms retrospectively (your server will need to rejoin the room), but it should fix the problem going forwards.
  • Improves the retry schedule over federation significantly - previously there were scenarios where synapse could try to retry aggressively on servers which were offline.  This fixes that.
  • Significant performance improvements to /publicRooms, /sync, and other endpoints.
  • Lots of juicy bug fixes.
We highly recommend upgrading (or installing!) asap - https://github.com/matrix-org/synapse has the details as usual.  Full changelog follows:

Changes in synapse v0.20.0 (2017-04-11)

Bug fixes:
  • Fix joining rooms over federation where not all servers in the room saw the new server had joined (PR #2094)

Changes in synapse v0.20.0-rc1 (2017-03-30)

Features:
  • Add delete_devices API (PR #1993)
  • Add phone number registration/login support (PR #1994, #2055)
Changes:
  • Use JSONSchema for validation of filters. Thanks @pik! (PR #1783)
  • Reread log config on SIGHUP (PR #1982)
  • Speed up public room list (PR #1989)
  • Add helpful texts to logger config options (PR #1990)
  • Minor /sync performance improvements. (PR #2002, #2013, #2022)
  • Add some debug to help diagnose weird federation issue (PR #2035)
  • Correctly limit retries for all federation requests (PR #2050, #2061)
  • Don't lock table when persisting new one time keys (PR #2053)
  • Reduce some CPU work on DB threads (PR #2054)
  • Cache hosts in room (PR #2060)
  • Batch sending of device list pokes (PR #2063)
  • Speed up persist event path in certain edge cases (PR #2070)
Bug fixes:
  • Fix bug where current_state_events renamed to current_state_ids (PR #1849)
  • Fix routing loop when fetching remote media (PR #1992)
  • Fix current_state_events table to not lie (PR #1996)
  • Fix CAS login to handle PartialDownloadError (PR #1997)
  • Fix assertion to stop transaction queue getting wedged (PR #2010)
  • Fix presence to fallback to last_active_ts if it beats the last sync time. Thanks @Half-Shot! (PR #2014)
  • Fix bug when federation received a PDU while a room join is in progress (PR#2016)
  • Fix resetting state on rejected events (PR #2025)
  • Fix installation issues in readme. Thanks @ricco386 (PR #2037)
  • Fix caching of remote servers' signature keys (PR #2042)
  • Fix some leaking log context (PR #2048, #2049, #2057, #2058)
  • Fix rejection of invites not reaching sync (PR #2056)

Opening up cyberspace with Matrix and WebVR!

2017-04-04 — Tech — 
TL;DR: here's the demo!Hi everyone,Today is a special day, the sort of day where you take a big step towards an ultimate dream. Starting Matrix and seeing it gaining momentum is already huge for us, a once in a lifetime opportunity. But one of the crazier things which drove us to create Matrix is the dream of creating cyberspace; the legendary promised land of the internet.Whether it's the Matrix of Neuromancer, the Metaverse of Snow Crash or the Other Plane of True Names, an immersive 3D environment where people can meet from around the world to communicate, create and share is the ultimate expression of the Internet's potential as a way to connect: the idea of an open, neutral, decentralised virtual reality within the 'net.This is essentially the software developer equivalent of lying on your back at night, looking up at the stars, and wondering if you'll ever fly among them... and Matrix is not alone in dreaming of this!  There have been many walled-garden virtual worlds over the years - Second Life, Habbo Hotel, all of the MMORPGs, Project Sansar etc.  And there have been decentralised worlds which lack the graphics but share the vision - whether it's FidoNet, Usenet, IRC servers, XMPP, the blogosphere or Matrix as it's used today.  And there are a few ambitious projects like Croquet/OpenCobalt, Open Simulator, JanusVR or High Fidelity which aim for a decentralised cyberspace, albeit without defining an open standard.But despite all this activity, where is the open cyberspace? Where is the universal fabric which could weave these worlds together?  Where is the VR equivalent of The Web itself?  Where is the neutral communication layer that could connect the galaxies of different apps and users into a coherent reality?  How do you bridge between today's traditional web apps and their VR equivalents?Aside from cultural ones, we believe there are three missing ingredients which have been technically holding back the development of an open cyberspace so far:
  1. The hardware
  2. Client software support (i.e. apps)
  3. A universal real-time data layer to store the space
Nowadays the hardware problem is effectively solved: the HTC Vive, Oculus Rift and even Google Cardboard have brought VR displays to the general public.  Meanwhile, accelerometers and head-tracking turn normal screens into displays for immersive content without even needing goggles, giving everyone a window into a virtual world.Client software is a more interesting story:  If there are many custom and proprietary VR apps that already exist out there, almost none of them can connect to other servers than the ones ran by their own vendors, or even other services and apps.   An open neutral cyberspace is just like the web: it needs the equivalent of web browsers, i.e. ubiquitous client apps which can connect to any servers and services written by any vendors and hosted by any providers, communicating together via an open common language.   And while web browsers of course exist, until very recently there was no way to link them into VR hardware.This has changed with the creation of WebVR by Mozilla - defining an API to let browsers render VR content, gracefully degrading across hardware and platforms such that you get the best possible experience all the way from a top-end gaming PC + Vive, down to tapping on a link on a simple smartphone.  WebVR is a genuine revolution: suddenly every webapp on the planet can create a virtual world! And frameworks like A-Frame, aframe-react and React VR make it incredibly easy and fun to do.So looking back at our list, the final missing piece is nothing less than a backbone: some kind of data layer to link these apps together.  Right now, all the WebVR apps out there are little islands - each its own isolated walled garden and there is no standard way to provide shared experiences.  There is no standard way for users to communicate between these worlds (or between the VR and non-VR web) - be that by messaging, VoIP, Video or even VR interactions.  There is no standard way to define an avatar, its location and movement within a world, or how it might travel between worlds.  And finally, there is no standard way to describe the world's state in general: each webapp is free to manage its scene and its content any way it likes; there is nowhere to expose the realtime scene-graph of the world such that other avatars, bots, apps, services etc. can interact with it. The same way there is no standard way to exchange messages or reuse a user profile between messaging apps today: if the cyberspace is taking shape as we speak, it is definitely not taking the path of openness. At least not yet.Predictably enough, it's this last point of the 'missing data layer for cyberspace' which we've been thinking about with Matrix: an open, neutral, decentralised meta-network powering or connecting these worlds.  To start with, we've made Matrix available as a generic communications layer for VR, taking WebVR (via A-Frame) and combining it with matrix-js-sdk, as an open, secure and decentralised way to place voip calls, video calls and instant messaging both within and between WebVR apps and the rest of the existing Matrix ecosystem (e.g. apps like Riot).In fact, the best way is to test it live: we've put together a quick demo at https://matrix.org/vrdemo to show it off, so please give it a go!

In the demo you get:
  1. a virtual lobby, providing a 1:1 WebRTC video call via Matrix through to a ‘guide' user of your choice anywhere else in Matrix (VR or not).  From the lobby you can jump into two other apps:
  2. a video conference, calling between all the participants of a given Matrix room in VR (no interop yet with other Matrix apps)
  3. a 'virtual tourism' example, featuring a 1:1 WebRTC video call with a guide, superimposed over the top of the user going skiing through 360 degree video footage.
Video calling requires a WebRTC-capable browser (Chrome or Firefox). Unfortunately no iOS browsers support it yet. If you have dedicated VR hardware (Vive or Rift), you'll have to configure your browser appropriately to use the demo - see https://webvr.rocks for the latest details.Needless to say, the demo's open sourced under the Apache License like all things Matrix - you can check out the code from https://github.com/matrix-org/matrix-vr-demo.  Huge kudos to Richard Lewis, Rob Swain and Ben Lewis for building this out - and to Aidan Taub and Tom Elliott for providing the 360 degree video footage!The demo is quite high-bandwidth and hardware intensive, so here's a video of it in action, just in case:
Now, it's important to understand that here we're using Matrix as a standard communications API for VR, but we're not using Matrix to store any VR world data (yet).  The demo uses plain A-Frame via aframe-react to render its world: we are not providing an API which exposes the world itself onto the network for folks to interact with and extend.  This is because Matrix is currently optimised for storing and synchronising two types of data structure: decentralised timelines of conversation data, and arbitrary decentralised key-value data (e.g. room names, membership, topics).However, the job of storing arbitrary world data requires storing and flexibly querying it as an object graph of some kind - e.g. as a scene graph hierarchy.  Doing this efficiently whilst supporting Matrix's decentralised eventual consistency model is tantamount to evolving Matrix into being a generic decentralised object-graph database (whilst upholding the constraints of that virtual world).  This is tractable, but it's a bunch more work than just supporting the eventually-consistent timeline & key-value store we have today.  It's something we're thinking about though. :)Also, Matrix is currently not super low-latency - on a typical busy Synapse deployment event transmission between clients has a latency of 50-200ms (ignoring network).  This is fine for instant messaging and setting up VoIP calls etc, but useless for publishing the realtime state of a virtual world: having to wait 200ms to be told that something happened in an interactive virtual world would be a terrible experience.  However, there are various fixes we can do here: Matrix itself is getting faster; Dendrite is expected to be one or two orders of magnitude faster than Synapse.  We could also use Matrix simply as a signalling layer in order to negotiate a lower latency realtime protocol to describe world data updates (much as we use Matrix as a signalling layer for setting up RTP streams for VoIP calls or MIDI sessions).Finally, you need somewhere to store the world assets themselves (textures, sounds, Collada or GLTF assets, etc).  This is no different to using attachments in Matrix today - this could be as plain HTTP, or via the Matrix decentralised content store (mxc:// URLs), or via something like IPFS.This said, it's only a matter of time before someone starts storing world data itself in Matrix.  We have more work to do before it's a tight fit, but this has always been one of the long-term goals of the project and we're going to do everything we can to support it well.So: this is the future we're thinking of.  Obviously work on today's Matrix servers, clients, spec & bridges is our focus and priority right now and we lots of work left there - but the longer term plan is critical too.  Communication in VR is pretty much a blank canvas right now, and Matrix can be the connecting fabric for it - which is unbelievably exciting.  Right now our demo is just a PoC - we'd encourage all devs reading this to have a think about how to extend it, and how we all can build together the new frontier of cyberspace!Finally, if you're interested in chatting more about VR on Matrix, come hang out over at #vr:matrix.org!- Matthew, Amandine & the Matrix team

Dendrite receives its first messages!!!

2017-03-15 — Tech — 

Hi all,

We hit a major milestone today on Dendrite, our next-generation golang homeserver: Dendrite received its first messages!!

Before you get too excited, please understand that Dendrite is still a pre-alpha work in progress - whilst we successfully created some rooms on an instance and sent a bunch of messages into them via the Client-Server API, most other functionality (e.g. receiving messages via /sync, logging in, registering, federation etc) is yet to exist. It cannot yet be used as a homeserver. However, this is still a huge step in the right direction, as it demonstrates the core DAG functionality of Matrix is intact, and the beginnings of a usable Client-Server API are hooked up.

The architecture of Dendrite is genuinely interesting - please check out the wiring diagram if you haven't already. The idea is that the server is broken down into a series of components which process streams of data stored in Kafka-style append-only logs. Each component scales horizontally (you can have as many as required to handle load), which is an enormous win over Synapse's monolithic design. Each component is also decoupled from each other by the logs, letting them run on entirely different machines as required. Please note that whilst the initial implementation is using Kafka for convenience, the actual append-only log mechanism is abstracted away - in future we expect to see configurations of Dendrite which operate entirely from within a single go executable (using go channels as the log mechanism), as well as alternatives to Kafka for distributed solutions.

The components which have taken form so far are the central roomserver service, which is responsible (as the name suggests) for maintaining the state and integrity of one or more rooms - authorizing events into the room DAG; storing them in postgres, tracking the auth chain of events where needed; etc. Much of the core matrix DAG logic of the roomserver is provided by gomatrixserverlib. The roomserver receives events sent by users via the 'client room send' component (and 'federation backfill' component, when that exists). The 'client room send' component (and in future also 'client sync') is provided by the clientapi service - which is what as of today is successfully creating rooms and events and relaying them to the roomserver!

The actual events we've been testing with are the history of the Matrix Core room: around 10k events. Right now the roomserver (and the postgres DB that backs it) are the main bottleneck in the pipeline rather than clientapi, so it's been interesting to see how rapidly the roomserver can consume its log of events. As of today's benchmark, on a generic dev workstation and an entirely unoptimised roomserver (i.e. no caching whatsoever) running on a single core, we're seeing it ingest the room history at over 350 events per second. The vast majority of this work is going into encoding/decoding JSON or waiting for postgres: with a simple event cache to avoid repeatedly hitting the DB for every auth and state event, we expect this to increase significantly. And then as we increase the number of cores, kafka partitions and roomserver instances it should scale fairly arbitrarily(!)

For context, the main synapse process for Matrix.org currently maxes out persisting events at around 15 and 20 per second (although it is also spending a bunch of time relaying events to the various worker processes, and other miscellanies). As such, an initial benchmark for Dendrite of 350 msgs/s really does look incredibly promising.

You may be wondering where this leaves Synapse? Well, a major driver for implementing Dendrite has been to support the growth of the main matrix.org server, which currently persists around 10 events/s (whilst emitting around 1500 events/s). We have exhausted most of the low-hanging fruit for optimising Synapse, and have got to point where the architectural fixes required are of similar shape and size to the work going into Dendrite. So, whilst Synapse is going to be around for a while yet, we're putting the majority of our long-term plans into Dendrite, with a distinct degree of urgency as we race against the ever-increasing traffic levels on the Matrix.org server!

Finally, you may be wondering what happened to Dendron, our original experiment in Golang servers. Well: Dendron was an attempt at a strangler pattern rewrite of Synapse, acting as an shim in front of Synapse which could gradually swap out endpoints with their golang implementations. In practice, the operational complexity it introduced, as well as the amount of room for improvement (at the time) we had in Synapse, and the relatively tight coupling to Synapse's existing architecture, storage & schema, meant that it was far from a clear win - and effectively served as an excuse to learn Go. As such, we've finally formally killed it off as of last week - Matrix.org is now running behind a normal haproxy, and Dendron is dead. Meanwhile, Dendrite (aka Dendron done Right ;) is very much alive, progressing fast, free from the shackles of Synapse.

We'll try to keep the blog updated with progress on Dendrite as it continues to grow!

How do I bridge thee? Let me count the ways...

2017-03-11 — Tech — 

Bridges come in many flavours, and we need consistent terminology within the Matrix community to ensure everyone (users, developers, core team) is on the same page. This post is primarily intended for bridge developers to refer to when building bridges.

The most recent version of this document is here (source) but we're also posting it as a blog post for visibility.

Types of rooms

Portal rooms

Bridges can register themselves as controlling chunks of room aliases namespace, letting Matrix users join remote rooms transparently if they /join #freenode_#wherever:matrix.org or similar. The resulting Matrix room is typically automatically bridged to the single target remote room. Access control for Matrix users is typically managed by the remote network's side of the room. This is called a portal room, and is useful for jumping into remote rooms without any configuration needed whatsoever - using Matrix as a ‘bouncer' for the remote network.

Plumbed rooms

Alternatively, an existing Matrix room can be can plumbed into one or more specific remote rooms by configuring a bridge (which can be run by anyone). For instance, #matrix:matrix.org is plumbed into #matrix on Freenode, matrixdotorg/#matrix on Slack, etc. Access control for Matrix users is necessarily managed by the Matrix side of the room. This is useful for using Matrix to link together different communities.

Migrating rooms between a portal & plumbed room is currently a bit of a mess, as there's not yet a way for users to remove portal rooms once they're created, so you can end up with a mix of portal & plumbed users bridged into a room, which looks weird from both the Matrix and non-Matrix viewpoints. https://github.com/matrix-org/matrix-appservice-irc/issues/387 tracks this.

Types of bridges (simplest first):

Bridgebot-based bridges

The simplest way to exchange messages with a remote network is to have the bridge log into the network using one or more predefined users called bridge bots - typically called MatrixBridge or MatrixBridge[123] etc. These relay traffic on behalf of the users on the other side, but it's a terrible experience as all the metadata about the messages and senders is lost. This is how the telematrix matrix<->telegram bridge currently works.

Bot-API (aka Virtual user) based bridges

Some remote systems support the idea of injecting messages from ‘fake' or ‘virtual' users, which can be used to represent the Matrix-side users as unique entities in the remote network. For instance, Slack's inbound webhooks lets remote bots be created on demand, letting Matrix users be shown cosmetically correctly in the timeline as virtual users. However, the resulting virtual users aren't real users on the remote system, so don't have presence/profile and can't be tab-completed or direct-messaged etc. They also have no way to receive typing notifs or other richer info which may not be available via bot APIs. This is how the current matrix-appservice-slack bridge works.

Simple puppeted bridge

This is a richer form of bridging, where the bridge logs into the remote service as if it were a real 3rd party client for that service. As a result, the Matrix user has to already have a valid account on the remote system. In exchange, the Matrix user ‘puppets' their remote user, such that other users on the remote system aren't even aware they are speaking to a user via Matrix. The full semantics of the remote system are available to the bridge to expose into Matrix. However, the bridge has to handle the authentication process to log the user into the remote bridge.

This is essentially how the current matrix-appservice-irc bridge works (if you configure it to log into the remote IRC network as your ‘real' IRC nickname). matrix-appservice-gitter is being extended to support both puppeted and bridgebot-based operation. It's how the experimental matrix-appservice-tg bridge works.

Going forwards we're aiming for all bridges to be at least simple puppeted, if not double-puppeted.

Double-puppeted bridge

A simple ‘puppeted bridge' allows the Matrix user to control their account on their remote network. However, ideally this puppeting should work in both directions, so if the user logs into (say) their native telegram client and starts conversations, sends messages etc, these should be reflected back into Matrix as if the user had done them there. This requires the bridge to be able to puppet the Matrix side of the bridge on behalf of the user.

This is the holy-grail of bridging; matrix-puppet-bridge is a community project that tries to facilitate development of double puppeted bridges, having done so for several networks. The main obstacle is working out an elegant way of having the bridge auth with Matrix as the matrix user (which requires some kind of scoped access_token delegation).

Server-to-server bridging

Some remote protocols (IRC, XMPP, SIP, SMTP, NNTP, GnuSocial etc) support federation - either open or closed. The most elegant way of bridging to these protocols would be to have the bridge participate in the federation as a server, directly bridging the entire namespace into Matrix.

We're not aware of anyone who's done this yet.

Sidecar bridge

Finally: the types of bridging described above assume that you are synchronising the conversation history of the remote system into Matrix, so it may be decentralised and exposed to multiple users within the wider Matrix network.

This can cause problems where the remote system may have arbitrarily complicated permissions (ACLs) controlling access to the history, which will then need to be correctly synchronised with Matrix's ACL model, without introducing security issues such as races. We already see some problems with this on the IRC bridge, where history visibility for +i and +k channels have to be carefully synchronised with the Matrix rooms.

You can also hit problems with other network-specific features not yet having equivalent representation in the Matrix protocol (e.g. ephemeral messages, or op-only messages - although arguably that's a type of ACL).

One solution could be to support an entirely different architecture of bridging, where the Matrix client-server API is mapped directly to the remote service, meaning that ACL decisions are delegated to the remote service, and conversations are not exposed into the wider Matrix. This is effectively using the bridge purely as a 3rd party client for the network (similar to Bitlbee). The bridge is only available to a single user, and conversations cannot be shared with other Matrix users as they aren't actually Matrix rooms. (Another solution could be to use Active Policy Servers at last as a way of centralising and delegating ACLs for a room)

This is essentially an entirely different product to the rest of Matrix, and whilst it could be a solution for some particularly painful ACL problems, we're focusing on non-sidecar bridges for now.

Synapse 0.19 is here, just in time for FOSDEM!

2017-02-04 — Tech — 

Hi all,

We're happy to announce the release of Synapse 0.19.0 (same as 0.19.0-rc4) today, just in time for anyone discovering Matrix for the first time at FOSDEM 2017!  In fact, here's Erik doing the release right now (with moral support from Luke):

release

This is a pretty big release, with a bunch of new features and lots and lots of debugging and optimisation work following on some of the dramas that we had with 0.18 over the Christmas break.  The biggest things are:

  • IPv6 Support (unless you have an IPv6 only resolver), thanks to contributions from Glyph from Twisted and Kyrias!
  • A new API for tracking the E2E devices present in a room (required for fixing many of the remaining E2E bugs...)
  • Rewrite the 'state resolution' algorithm to be orders of magnitude more performant
  • Lots of tuning to the caching logic.
If you're already running a server, please upgrade!  And if you're not, go grab yourself a brand new Synapse from Github. Debian packages will follow shortly (as soon as Erik can figure out the necessary backporting required for Twisted 16.6.0)

And here's the full changelog...

Changes in synapse v0.19.0 (2017-02-04)

No changes since RC 4.

Changes in synapse v0.19.0-rc4 (2017-02-02)

  • Bump cache sizes for common membership queries (PR #1879)

Changes in synapse v0.19.0-rc3 (2017-02-02)

  • Fix email push in pusher worker (PR #1875)
  • Make presence.get_new_events a bit faster (PR #1876)
  • Make /keys/changes a bit more performant (PR #1877)

Changes in synapse v0.19.0-rc2 (2017-02-02)

  • Include newly joined users in /keys/changes API (PR #1872)

Changes in synapse v0.19.0-rc1 (2017-02-02)

Features:
  • Add support for specifying multiple bind addresses (PR #1709, #1712, #1795, #1835). Thanks to @kyrias!
  • Add /account/3pid/delete endpoint (PR #1714)
  • Add config option to configure the Riot URL used in notification emails (PR #1811). Thanks to @aperezdc!
  • Add username and password config options for turn server (PR #1832). Thanks to @xsteadfastx!
  • Implement device lists updates over federation (PR #1857, #1861, #1864)
  • Implement /keys/changes (PR #1869, #1872)
Changes:
  • Improve IPv6 support (PR #1696). Thanks to @kyrias and @glyph!
  • Log which files we saved attachments to in the media_repository (PR #1791)
  • Linearize updates to membership via PUT /state/ to better handle multiple joins (PR #1787)
  • Limit number of entries to prefill from cache on startup (PR #1792)
  • Remove full_twisted_stacktraces option (PR #1802)
  • Measure size of some caches by sum of the size of cached values (PR #1815)
  • Measure metrics of string_cache (PR #1821)
  • Reduce logging verbosity (PR #1822, #1823, #1824)
  • Don't clobber a displayname or avatar_url if provided by an m.room.member event (PR #1852)
  • Better handle 401/404 response for federation /send/ (PR #1866, #1871)
Fixes:
  • Fix ability to change password to a non-ascii one (PR #1711)
  • Fix push getting stuck due to looking at the wrong view of state (PR #1820)
  • Fix email address comparison to be case insensitive (PR #1827)
  • Fix occasional inconsistencies of room membership (PR #1836, #1840)
Performance:
  • Don't block messages sending on bumping presence (PR #1789)
  • Change device_inbox stream index to include user (PR #1793)
  • Optimise state resolution (PR #1818)
  • Use DB cache of joined users for presence (PR #1862)
  • Add an index to make membership queries faster (PR #1867)

Matrix.org homeserver outage (25th Jan 2017)

2017-01-25 — Tech — 

Hi folks,

As many will have noticed there was a major outage on the Matrix homeserver for matrix.org last night (UK-time). This impacted anyone with an account on the matrix.org server, as well as anyone using matrix.org-hosted bots & bridges. As Matrix rooms are shared over all participants, rooms with participants on other servers were unaffected (for users on those servers). Here's a quick explanation of what went wrong (times are UTC):

  • 2017-01-24 16:00 - We notice that we're badly running out of diskspace on the matrix.org backup postgres replica. (Turns out the backup box, whilst identical hardware to the master, had been built out as RAID-10 rather than RAID-5 and so has less disk space).
  • 2017-01-24 17:00 - We decide to drop a large DB index: event_push_actions(room_id, event_id, user_id, profile_tag), which was taking up a disproportionate amount of disk space, on the basis that it didn't appear to be being used according to the postgres stats. All seems good.
  • 2017-01-24 ~23:00 - The core matrix.org team go to bed.
  • 2017-01-24 23:33 - Someone redacts an event in a very active room (probably #matrix:matrix.org) which necessitates redacting the associated push notification from the event_push_actions table. This takes out a lock within persist_event, which is then blocked on deleting the push notification. It turns out that this deletion requires the missing DB constraint, causing the query to run for hours whilst holding the transaction lock. The symptoms are that anything reading events from the DB was blocked on the transaction, causing messages not to be relayed to other clients or servers despite appearing to send correctly. Meanwhile, the fact that events are being received by the server fine (including over federation) makes the monitoring graphs look largely healthy.
  • 2017-01-24 23:35 - End-to-end monitoring detects problems, and sends alerts into pagerduty and various Matrix rooms. Unfortunately we'd failed to upgrade the pageduty trial into a paid account a few months ago, however, so the alerts are lost.
  • 2017-01-25 08:00 - Matrix team starts to wake up and spot problems, but confusion over the right escalation process (especially with Matthew on holiday) means folks assume that other members of the team must already be investigating.
  • 2017-01-25 09:00 - Server gets restarted, service starts to resume, although box suffers from load problems as traffic tries to catch up.
  • 2017-01-25 09:45 - Normal service on the homeserver itself is largely resumed (other than bridges; see below)
  • 2017-01-25 10:41 - Root cause established and the redaction path is patched on matrix.org to stop a recurrence.
  • 2017-01-25 11:15 - Bridges are seen to be lagging and taking much longer to recover than expected. Decision made to let them continue to catch up normally rather than risk further disruption (e.g. IRC join/part spam) by restarting them.
  • 2017-01-25 13:00 - All hosted bridges returned to normal.

Obviously this is rather embarrassing, and a huge pain for everyone using the matrix.org homeserver - many apologies indeed for the outage. On the plus side, all the other Matrix homeservers out there carried on without noticing any problems (which actually complicated spotting that things had broken, given many of the core team primarily use their personal homeservers).

In some ways the root cause here is that the core team has been focusing all its energy recently on improving the overall Matrix codebase rather than operational issues on matrix.org itself, and as a result our ops practices have fallen behind (especially as the health of the Matrix ecosystem as a whole is arguably more important than the health of a single homeserver deployment). However, we clearly need to improve things here given the number of people (>750K at the last count) dependent on the Matrix.org homeserver and its bridges & bots.

Lessons learnt on our side are:

  • Make sure that even though we had monitoring graphs & thresholds set up on all the right things... monitoring alerts actually have to be routed somewhere useful - i.e. phone calls to the team's phones. Pagerduty is now set up and running properly to this end.
  • Make sure that people know to wake up the right people anyway if the monitoring alerting system fails.
  • To be even more paranoid about hotfixes to production at 5pm, especially if they can wait 'til the next day (as this one could have).
  • To investigate ways to rapidly recover bridges without causing unnecessary disruption.

Apologies again to everyone who was bitten by this - we're doing everything we can to ensure it doesn't happen again.

Matthew & the team.

Synapse 0.18.7 is out - Please upgrade, especially if on 0.18.5 or 0.18.6.

2017-01-06 — Tech — 

Hi all,

TL;DR: Please upgrade to Synapse 0.18.6, especially if you are on 0.18.5 which is a bad release.

TL;DR: Please upgrade to Synapse 0.18.7 - especially if you are on 0.18.5 or 0.18.6 which both have serious federation bugs.

Synapse 0.18.5 contained a really nasty regression in the federation code which causes servers to echo transactions that they receive back out to the other servers participating in a room. This has effectively resulted in a gradual amplification of federation traffic as more people have installed 0.18.5, causing every transaction to be received N times over where N is the number of servers in the room.

We'll do a full write-up once we're happy we've tracked down all the root problems here, but the short story is that this hit critical mass around Dec 26, where typical Synapses started to fail to keep up with the traffic - especially when requests hit some of the more inefficient or buggy codepaths in Synapse.  As servers started to overload with inbound connections, this in turn started to slow down and consume resources on the connecting servers - especially due to an architectural mistake in Synapse which blocks inbound connections until the request has been fully processed (which could require the receiving server in turn to make outbound connections), rather than releasing the inbound connection asap.  This hit the point that servers were running out of file descriptors due to all the outbound and inbound connections, at which point they started to entirely tarpit inbound connections, resulting in a slow feedback loop making the whole situation even worse.

We've spent the last two weeks hunting all the individual inefficient requests which were mysteriously starting to cause more problems than they ever had before; then trying to understand the feedback misbehaviour; before finally discovering the regression in 0.18.5 as the plausible root cause of the problem.  Troubleshooting has been complicated by most of the team having unplugged for the holidays, and because this is the first (and hopefully last!) failure mode to be distributed across the whole network, making debugging something of a nightmare - especially when the overloading was triggering a plethora of different exotic failure modes.  Huge thanks to everyone who has shared their server logs with the team to help debug this.

Some of these failure modes are still happening (and we're working on fixing them), but we believe that if everyone upgrades away from the bad 0.18.5 release most of the symptoms will go away, or at least go back to being as bad as they were before.  Meanwhile, if you find your server suddenly grinding to a halt after upgrading to 0.18.6 0.18.7 please come tell us in #matrix-dev:matrix.org.

We're enormously sorry if you've been bitten by the federation instability this has caused - and many many thanks for your patience whilst we've hunted it down.  On the plus side, it's given us a lot of very useful insight into how to implement federation in future homeservers to not suffer from any of these failure modes.  It's also revealed the root cause of why Synapse's RAM usage is quite so bad - it turns out that it actually idles at around 200MB with default caching, but there's a particular codepath which causes it to spike temporarily by 1GB or so - and that RAM is then not released back to the OS.  We're working on a fix for this too, but it'll come after 0.18.7.

Unfortunately the original release of 0.18.6 still exhibits the root bug, but 0.18.7 (originally released as 0.18.7-rc2) should finally fix this.  Sorry for all the upgrades :(

So please upgrade as soon as possible to 0.18.7. Debian packages are available as normal.

thanks,

Matthew

Changes in synapse v0.18.7 (2017-01-09)

  • No changes from v0.18.7-rc2

Changes in synapse v0.18.7-rc2 (2017-01-07)

Bug fixes:

  • Fix error in rc1's discarding invalid inbound traffic logic that was incorrectly discarding missing events

Changes in synapse v0.18.7-rc1 (2017-01-06)

Bug fixes:

  • Fix error in #PR 1764 to actually fix the nightmare #1753 bug.
  • Improve deadlock logging further
  • Discard inbound federation traffic from invalid domains, to immunise against #1753

Changes in synapse v0.18.6 (2017-01-06)

Bug fixes:
  • Fix bug when checking if a guest user is allowed to join a room - thanks to Patrik Oldsberg (PR #1772)

Changes in synapse v0.18.6-rc3 (2017-01-05)

Bug fixes:
  • Fix bug where we failed to send ban events to the banned server (PR #1758)
  • Fix bug where we sent event that didn't originate on this server to other servers (PR #1764)
  • Fix bug where processing an event from a remote server took a long time because we were making long HTTP requests (PR #1765, PR #1744)
Changes:
  • Improve logging for debugging deadlocks (PR #1766, PR #1767)

Changes in synapse v0.18.6-rc2 (2016-12-30)

Bug fixes:
  • Fix memory leak in twisted by initialising logging correctly (PR #1731)
  • Fix bug where fetching missing events took an unacceptable amount of time in large rooms (PR #1734)

Changes in synapse v0.18.6-rc1 (2016-12-29)

Bug fixes:
  • Make sure that outbound connections are closed (PR #1725)

matrix-appservice-irc 0.7.0 is out!

2016-12-19 — Tech — 

Also, we've just released a major update to the IRC bridge codebase after trialling it on the matrix.org-hosted bridges for the last few days.

The big news is:

  • The bridge uses Synapse 0.18.5's new APIs for managing the public room list (improving performance a bunch)
  • Much faster startup using the new /joined_rooms and /joined_members APIs in Synapse 0.18.5
  • The bridge will now remember your NickServ password (encrypted at rest) if you want it to via the !storepass command
  • You can now set arbitrary user modes for IRC clients on connection (to mitigate PM spam etc)
  • After a split, the bridge will drop Matrix->IRC messages older than N seconds, rather than try to catch the IRC room up on everything they missed on Matrix :S
  • Operational metrics are now implemented using Prometheus rather than statsd
  • New !quit command to nuke your user from the remote IRC network
  • Membership list syncing for IRC->Matrix is enormously improved, and enabled for all matrix.org-hosted bridges apart from Freenode.  <b>At last, membership lists should be in sync betwen IRC and Matrix; please let us know if they're not</b>.
  • Better error logging
For full details, please see the changelog.

With things like NickServ-pass storing, !quit support and full bi-directional membership list syncing, it's never been a better time to run your own IRC bridge.  Please install or upgrade today from https://github.com/matrix-org/matrix-appservice-irc!

Synapse 0.18.5 released!

2016-12-19 — Tech — 

Hi folks,

We released synapse 0.18.5 on Friday.  This is mainly about fixing performance problems with the unread room counts and the public room directory; polishing the E2E endpoints based on beta feedback; and general minor bits and bobs.

Get it whilst it's (almost) hot from https://github.com/matrix-org/synapse!  Changelog follows:

Changes in synapse v0.18.5 (2016-12-16)

Bug fixes:

  • Fix federation /backfill returning events it shouldn't (PR #1700)
  • Fix crash in url preview (PR #1701)

Changes in synapse v0.18.5-rc3 (2016-12-13)

Features:
  • Add support for E2E for guests (PR #1653)
  • Add new API appservice specific public room list (PR #1676)
  • Add new room membership APIs (PR #1680)
Changes:
  • Enable guest access for private rooms by default (PR #653)
  • Limit the number of events that can be created on a given room concurrently (PR #1620)
  • Log the args that we have on UI auth completion (PR #1649)
  • Stop generating refresh_tokens (PR #1654)
  • Stop putting a time caveat on access tokens (PR #1656)
  • Remove unspecced GET endpoints for e2e keys (PR #1694)
Bug fixes:
  • Fix handling of 500 and 429's over federation (PR #1650)
  • Fix Content-Type header parsing (PR #1660)
  • Fix error when previewing sites that include unicode, thanks to @kyrias (PR #1664)
  • Fix some cases where we drop read receipts (PR #1678)
  • Fix bug where calls to /sync didn't correctly timeout (PR #1683)
  • Fix bug where E2E key query would fail if a single remote host failed (PR #1686)

Changes in synapse v0.18.5-rc2 (2016-11-24)

Bug fixes:
  • Don't send old events over federation, fixes bug in -rc1.

Changes in synapse v0.18.5-rc1 (2016-11-24)

Features:
  • Implement "event_fields" in filters (PR #1638)
Changes:
  • Use external ldap auth pacakge (PR #1628)
  • Split out federation transaction sending to a worker (PR #1635)
  • Fail with a coherent error message if /sync?filter= is invalid (PR #1636)
  • More efficient notif count queries (PR #1644)

Synapse 0.18.4

2016-11-22 — Tech — 

Uncharacteristically, we're actually remembering to announce a new release of Synapse!

Major performance fixes on federation, as well as the changes required to support E2E encrypted attachments (yay!)

Please install or upgrade from https://github.com/matrix-org/synapse :)

Changes in synapse v0.18.4 (2016-11-22)

Bug fixes:
  • Add workaround for buggy clients that the fail to register (PR #1632)

Changes in synapse v0.18.4-rc1 (2016-11-14)

Changes:
  • Various database efficiency improvements (PR #1188, #1192)
  • Update default config to blacklist more internal IPs, thanks to Euan Kemp @euank (PR #1198)
  • Allow specifying duration in minutes in config, thanks to Daniel Dent @DanielDent (PR #1625)
Bug fixes:
  • Fix media repo to set CORs headers on responses (PR #1190)
  • Fix registration to not error on non-ascii passwords (PR #1191)
  • Fix create event code to limit the number of prev_events (PR #1615)
  • Fix bug in transaction ID deduplication (PR #1624)

Matrix-IRC Bridge reaches v0.4.0

2016-08-15 — Tech — 

A new version of the IRC bridge has been released onto NPM and the matrix.org bridges!

The IRC bridge has undergone quite a number of modifications since its original inception over a year ago. Version 0.4 introduces a number of additional features and improvements, which can be found in the changelog. These include automatically linkifying large blocks of text and mirroring kicks/bans to and from Matrix.

With a plethora of protocol gotchas and non-standard features on well-known IRC networks, IRC is a challenging protocol to work with. It's inevitable that some corner cases are not handled well by the bridge. Over time, the bridge has been hardened by edge cases which we have encountered and patched. These releases signify the continual improvement in the robustness of the bridge, which we aim to continue with into the foreseeable future.

Performance wise, our busiest bridge which we host is the bridge to Freenode. We now have over 1300 active connections to it and have a steady rate of about 240 messages per minute going through to Matrix. We expect to see this number increase significantly over the next few months. Let's see what the next year will bring!

Synapse 0.14 is released!

2016-03-30 — Tech — 

We just released Synapse 0.14.0 - a major update which incorporates lots of work on making Synapse more RAM efficient. There's still a lot of room for further improvements, but the main headlines are reducing the resident memory footprint dramatically by interning strings and deduplicating events across the many different caches. It also adds a much-needed SYNAPSE_CACHE_FACTOR environment variable that can be used to globally decrease or increase the sizing of all of Synapse's various caches (with an associated slow-down or speed-up in performance). Quite how improved the new memory footprint seems to very much depend on your own use case, but it's certainly a step in the right direction.

For more details on recent Synapse performance work (and a general state of the union for the whole Matrix ecosystem), check out our Spring update.

Get all new synapse from https://github.com/matrix-org/synapse - we recommend upgrading (or installing!) asap :)

Full changelog follows:

Changes in synapse v0.14.0 (2016-03-30)

No changes from v0.14.0-rc2

Changes in synapse v0.14.0-rc2 (2016-03-23)

Features:
  • Add published room list API (PR #657)
Changes:
  • Change various caches to consume less memory (PR #656, #658, #660, #662, #663, #665)
  • Allow rooms to be published without requiring an alias (PR #664)
  • Intern common strings in caches to reduce memory footprint (#666)
Bug fixes:
  • Fix reject invites over federation (PR #646)
  • Fix bug where registration was not idempotent (PR #649)
  • Update aliases event after deleting aliases (PR #652)
  • Fix unread notification count, which was sometimes wrong (PR #661)

Changes in synapse v0.14.0-rc1 (2016-03-14)

Features:
  • Add event_id to response to state event PUT (PR #581)
  • Allow guest users access to messages in rooms they have joined (PR #587)
  • Add config for what state is included in a room invite (PR #598)
  • Send the inviter's member event in room invite state (PR #607)
  • Add error codes for malformed/bad JSON in /login (PR #608)
  • Add support for changing the actions for default rules (PR #609)
  • Add environment variable SYNAPSE_CACHE_FACTOR, default it to 0.1 (PR #612)
  • Add ability for alias creators to delete aliases (PR #614)
  • Add profile information to invites (PR #624)
Changes:
  • Enforce user_id exclusivity for AS registrations (PR #572)
  • Make adding push rules idempotent (PR #587)
  • Improve presence performance (PR #582, #586)
  • Change presence semantics for last_active_ago (PR #582, #586)
  • Don't allow m.room.create to be changed (PR #596)
  • Add 800x600 to default list of valid thumbnail sizes (PR #616)
  • Always include kicks and bans in full /sync (PR #625)
  • Send history visibility on boundary changes (PR #626)
  • Register endpoint now returns a refresh_token (PR #637)
Bug fixes:
  • Fix bug where we returned incorrect state in /sync (PR #573)
  • Always return a JSON object from push rule API (PR #606)
  • Fix bug where registering without a user id sometimes failed (PR #610)
  • Report size of ExpiringCache in cache size metrics (PR #611)
  • Fix rejection of invites to empty rooms (PR #615)
  • Fix usage of bcrypt to not use checkpw (PR #619)
  • Pin pysaml2 dependency (PR #634)
  • Fix bug in /sync where timeline order was incorrect for backfilled events (PR #635)

Synapse 0.13 released!

2016-02-10 — Tech — 

Hi all,

Synapse 0.13 was released this afternoon, bringing a new wave of features, bug fixes and performance fixes. The main headlines include: huge performance increases (big catchup /syncs that were taking 20s now take 0.3s!), support for server-side per-room unread message and notification badge counts, ability for guest accounts to upgrade into fully-fledged accounts, change default push rules back to notifying for group chats, and loads of bug fixes. This release incorporates what-was 0.12.1-rc1.

Please note that on first launch after upgrading a pre-0.13 server to 0.13 or later, synapse will add a large database index which may take several minutes to complete. Whilst the index is added the service will be unresponsive.

Please get the new release from https://github.com/matrix-org/synapse and have fun!

Matthew

Full release notes:

Changes in synapse v0.13.1 (2016-02-10) =======================================
  • Bump matrix-angular-sdk (matrix web console) dependency to 0.6.8 to pull in the fix for SYWEB-361 so that the default client can display HTML messages again(!)

Changes in synapse v0.13.0 (2016-02-10)

This version includes an upgrade of the schema, specifically adding an index to the events table. This may cause synapse to pause for several minutes the first time it is started after the upgrade.

Changes:

  • Improve general performance (PR #540, #543. #544, #54, #549, #567)
  • Change guest user ids to be incrementing integers (PR #550)
  • Improve performance of public room list API (PR #552)
  • Change profile API to omit keys rather than return null (PR #557)
  • Add /media/r0 endpoint prefix, which is equivalent to /media/v1/ (PR #595)

Bug fixes:

  • Fix bug with upgrading guest accounts where it would fail if you opened the registration email on a different device (PR #547)
  • Fix bug where unread count could be wrong (PR #568)

Changes in synapse v0.12.1-rc1 (2016-01-29)

Features:

  • Add unread notification counts in /sync (PR #456)
  • Add support for inviting 3pids in /createRoom (PR #460)
  • Add ability for guest accounts to upgrade (PR #462)
  • Add /versions API (PR #468)
  • Add event to /context API (PR #492)
  • Add specific error code for invalid user names in /register (PR #499)
  • Add support for push badge counts (PR #507)
  • Add support for non-guest users to peek in rooms using /events (PR #510)

Changes:

  • Change /sync so that guest users only get rooms they've joined (PR #469)
  • Change to require unbanning before other membership changes (PR #501)
  • Change default push rules to notify for all messages (PR #486)
  • Change default push rules to not notify on membership changes (PR #514)
  • Change default push rules in one to one rooms to only notify for events that are messages (PR #529)
  • Change /sync to reject requests with a from query param (PR #512)
  • Change server manhole to use SSH rather than telnet (PR #473)
  • Change server to require AS users to be registered before use (PR #487)
  • Change server not to start when ASes are invalidly configured (PR #494)
  • Change server to require ID and as_token to be unique for AS's (PR #496)
  • Change maximum pagination limit to 1000 (PR #497)

Bug fixes:

  • Fix bug where /sync didn't return when something under the leave key changed (PR #461)
  • Fix bug where we returned smaller rather than larger than requested thumbnails when method=crop (PR #464)
  • Fix thumbnails API to only return cropped thumbnails when asking for a cropped thumbnail (PR #475)
  • Fix bug where we occasionally still logged access tokens (PR #477)
  • Fix bug where /events would always return immediately for guest users (PR #480)
  • Fix bug where /sync unexpectedly returned old left rooms (PR #481)
  • Fix enabling and disabling push rules (PR #498)
  • Fix bug where /register returned 500 when given unicode username (PR #513)

Synapse 0.12 released!

2016-01-04 — Tech — 

Happy 2016 everyone!

To greet the new year, we bring you all new Synapse 0.12. The focus here has been a wide range of polishing, bugfixes, performance improvements and feature tweaks. The biggest news are that the 'v2' sync APIs are now production ready; the search APIs now work much better; 3rd party ID invites now work; and we now mount the whole client-server API under the /_matrix/client/r0 URI prefix, as per the r0.0.0 release of the Client Server API from a few weeks ago. The r0 release unifies what were previously the somewhat confusing mix of 'v1' and 'v2' APIs as a single set of endpoints which play nice together.

We highly recommend all homeservers upgrading to v0.12.0 as soon as possible. Get it now from https://github.com/matrix-org/synapse/ or via our shiny new Debian packages at https://matrix.org/packages/debian/.

Full changelog follows:

Changes in synapse v0.12.0 (2016-01-04)

  • Expose /login under r0 (PR #459)

Changes in synapse v0.12.0-rc3 (2015-12-23)

  • Allow guest accounts access to /sync (PR #455)
  • Allow filters to include/exclude rooms at the room level rather than just from the components of the sync for each room. (PR #454)
  • Include urls for room avatars in the response to /publicRooms (PR #453)
  • Don't set a identicon as the avatar for a user when they register (PR #450)
  • Add a display_name to third-party invites (PR #449)
  • Send more information to the identity server for third-party invites so that it can send richer messages to the invitee (PR #446)
  • Cache the responses to /initialSync for 5 minutes. If a client retries a request to /initialSync before the a response was computed to the first request then the same response is used for both requests (PR #457)
  • Fix a bug where synapse would always request the signing keys of remote servers even when the key was cached locally (PR #452)
  • Fix 500 when pagination search results (PR #447)
  • Fix a bug where synapse was leaking raw email address in third-party invites (PR #448)

Changes in synapse v0.12.0-rc2 (2015-12-14)

  • Add caches for whether rooms have been forgotten by a user (PR #434)
  • Remove instructions to use --process-dependency-link since all of the dependencies of synapse are on PyPI (PR #436)
  • Parallelise the processing of /sync requests (PR #437)
  • Fix race updating presence in /events (PR #444)
  • Fix bug back-populating search results (PR #441)
  • Fix bug calculating state in /sync requests (PR #442)

Changes in synapse v0.12.0-rc1 (2015-12-10)

  • Host the client APIs released as r0 by https://matrix.org/docs/spec/r0.0.0/client_server.html on paths prefixed by/_matrix/client/r0. (PR #430, PR #415, PR #400)
  • Updates the client APIs to match r0 of the matrix specification.
    • All APIs return events in the new event format, old APIs also include the fields needed to parse the event using the old format for compatibility. (PR #402)
    • Search results are now given as a JSON array rather than a JSON object (PR #405)
    • Miscellaneous changes to search (PR #403, PR #406, PR #412)
    • Filter JSON objects may now be passed as query parameters to /sync (PR #431)
    • Fix implementation of /admin/whois (PR #418)
    • Only include the rooms that user has left in /sync if the client requests them in the filter (PR #423)
    • Don't push for m.room.message by default (PR #411)
    • Add API for setting per account user data (PR #392)
    • Allow users to forget rooms (PR #385)
  • Performance improvements and monitoring:
    • Add per-request counters for CPU time spent on the main python thread. (PR #421, PR #420)
    • Add per-request counters for time spent in the database (PR #429)
    • Make state updates in the C+S API idempotent (PR #416)
    • Only fire user_joined_room if the user has actually joined. (PR #410)
    • Reuse a single http client, rather than creating new ones (PR #413)
  • Fixed a bug upgrading from older versions of synapse on postgresql (PR #417)

Matrix: One year in.

2015-09-07 — Tech — 

Hi all,

Just realised that the release of Synapse 0.10.0 on Sept 3rd 2015 was precisely one year from the initial launch of Matrix. As such, it feels only right to have a quick update on where we've got to so far, and where we expect things to go from here!

Back at the original launch, all we had was a very rough and ready Synapse homeserver, an early draft of the spec, and precisely one client - the Angular webclient, much of which was mainly written at the last minute on the plane to TechCrunch Disrupt SF (and has never quite recovered :S). From this initial seed it's been incredibly exciting and slightly scary to see how much things have advanced and grown - the big headlines for the past year (in roughly chronological order) include:

  • Making Federation Work:
    • Switching all of federation over to SSL, using perspectives for key management
    • Crypto-signing all the events in a room's message graph to assert integrity
    • Sorting out 'power levels' and 'auth events' to allow totally decentralised kicks/bans/etc to work in an open federated environment
  • Decentralised content repository and thumbnailing
  • Reference mobile "Matrix Console" clients for iOS and Android
  • Official client SDKs for iOS and Android - both at the API wrapping layer and the reusable UI component layer
  • Push notifications for APNS and GCM (both on server & clients)
  • Official client SDKs for JavaScript, Python and Perl
  • Typing notifications
  • The sytest integration test harness
  • Proper WebRTC support for VoIP, including TURN.
  • Application Services and Bots - actually letting Matrix defragment communications :)
    • Bridging to all of Freenode, Moznet and other IRC networks
    • Matrix<->SMS bridge from OpenMarket
    • SIP bridges via FreeSWITCH and Verto
    • Parrot Bebop Drone <-> Matrix bridge via Janus
    • ODB2 telemetry <->  Matrix bridge via Android SDK
    • Reusable bridging framework in Node
  • Many iterations and refinements to the spec, including designing v2 of the client SDK
  • Switching from Angular to React for all of our web-client development
  • Customisable skins and embedding support for the matrix-react-sdk
  • End-to-end encryption (not quite formally released yet, but it's written, specced and it works!)
  • VoIP support on mobile (landed in Android; still experimenting with different WebRTC stacks on iOS)
  • History ACLs
  • Delivery reports
  • Switching from access_tokens to macaroons for authentication (not yet released)
  • Lots and lots of performance work on Synapse, as we've tried to get the most out of Twisted.
...and last but not least, the evolution of the #matrix:matrix.org community - including loads of 3rd party clients, SDKs and application services, synapse packaging and even experimental home servers :)

Overall the last year was an exercise in actually fleshing out the whole ecosystem of Matrix and getting it to a stable usable beta acceptable to early adopters. The plan for the next 12 months is then to make the transition from stable beta to a properly production grade environment that can be used to run large scale services used by normal end-users. In practice, this means:

  • A major rearchitecture of Synapse.
    • Synapse currently has no support for horizontal scaling or clustering within a single instance, and many will have seen the performance problems we've hit with a relatively monolithic Twisted app architecture. Profiling deferreds in Twisted has been a particular nightmare.
    • During September we are starting the process of splitting Synapse apart into separate services (e.g. separating reading eventstreams from writing messages) both to allow horizontal scalability and to experiment with implementing the services in more efficient languages than Python/Twisted.
    • We will continue the normal Synapse release process in parallel with this work.
  • Ensuring Matrix can support a genuinely excellent UX for normal end-users on glossy clients, and supporting glossy client development as required.  The days of Matrix being just for powerusers are numbered... :)
  • Switching to use 3rd party IDs as the primary means of referring to users in Matrix, hiding matrix IDs as a feature for powerusers and developers.
  • Finishing the spec. You may have noticed the spec has been quietly evolving over the last few months - finally gaining a versioning system, and with larger chunks of it being automatically generated from formal API spec descriptions. We will be finishing off and filling in the remaining holes.
  • Improving the documentation (and FAQ!) on matrix.org in general by switching to a git-backed jekyll system for all the staticish content
  • Replace the Angular-based reference webapp bundled with Synapse entirely with a matrix-react-sdk based reference app, and providing better examples and documentation for using it to embed Matrix functionality into existing websites.
  • Moving to v2 of the client-server API. This fixes some significant limitations in the v1 API that everyone's been using all year, and should improve performance significantly for many use cases (especially when launching apps). The v1 API will hang around for a very very long time for backwards compatibility.
  • Writing *lots* more bridges and integrations to other protocols, now we have a nice framework for rapidly developing them.
  • General security audits and double-checking the security model.
  • New features, including:
    • Multiway VoIP and Video conferencing, most likely using FreeSWITCH's new conferencing capabilities via an Application Service bridge (should be ready very shortly!)
    • Getting E2E crypto reviewed/audited and putting it live across the board.
    • Adding VoIP to iOS
    • Implementing delivery reports in all clients
    • Improving how invites work (ability to reject them; metadata about where they came from)
    • Search API
    • Improved file management
  • ...and an awful lot of bug fixing as we work through the backlog we've accumulated on JIRA.
Hopefully this won't take up all year(!) and is just a beginning - there's a huge list of interesting ideas beyond this baseline which we'll be looking at assuming the core stuff above is on track. For instance, we need to work out how to decentralise the identity services that mapping 3rd party IDs to matrix IDs. We need to work out how to avoid spam. And it could be fascinating to start bridging more internet-of-things devices and ecosystems into Matrix, or decentralising user accounts between homeservers, or perhaps using Matrix for synchronising more sophisticated data structures than timelines and key-value state dictionaries...

Finally, we also want to save as much time as possible to help support the wider community in building out clients, services and servers on top of Matrix. Just like the web itself, Matrix is only as useful as the content and services built on top of it - and we will do everything we can to help the pioneers who are interested in colonising this brave new world :)

Huge thanks to everyone over the last year who have supported us - whether that's by creating an account and using the system, running a homeserver, hacking on top of the platform, contributing to the core project, enduring one of our presentations, or even paying for us to work on this. The coming year should prove incredibly interesting, and we hope you'll stay and bring along all your friends, family and colleagues for the ride as the adventure continues!

Matthew, Amandine & the whole Matrix.org team.

Synapse 0.10.0 is released!!

2015-09-03 — Tech — 

Hi folks,

Whilst the blog has been a bit quiet, we've actually had an incredibly busy summer refining Synapse, building the new matrix-react-sdk and example apps, building an entirely new matrix-appservice-bridge framework for rapidly creating Matrix<->other-protocol bridges, getting end-to-end encryption ready for primetime and lots more fun stuff as we keep chipping away to take Matrix out of beta. We'll write about all of these once they're ready, but right now the big news is that after 6 release candidates we have a major new update for Synapse out today - version 0.10.0. This also includes 0.9.4, which we never quite got around to releasing and ended up skipping from 0.9.4-rc1 straight to 0.10.0-rc1.

The release focuses mainly on performance, bugfixes, and infrastructure work to support forthcoming features like end-to-end encryption, read receipts, etc. Some of the more exciting new features include preset ACLs for room creation; history visibility ACLs; SAML2 single-sign-on login (courtesy of Ericsson, thanks guys!), filename support when sending files, support for specifying a canonical alias for a room, support for intermediary SSL certificates, etc.

The full changelog is below.

To upgrade, go read https://github.com/matrix-org/synapse/blob/master/UPGRADE.rst - to install for the first time, go to https://github.com/matrix-org/synapse/blob/master/README.rst.

Changes in synapse v0.10.0 (2015-09-03) =======================================

No change from release candidate.

Changes in synapse v0.10.0-rc6 (2015-09-02)

  • Remove some of the old database upgrade scripts.
  • Fix database port script to work with newly created sqlite databases.

Changes in synapse v0.10.0-rc5 (2015-08-27)

  • Fix bug that broke downloading files with ascii filenames across federation.

Changes in synapse v0.10.0-rc4 (2015-08-27)

  • Allow UTF-8 filenames for upload. (PR #259)

Changes in synapse v0.10.0-rc3 (2015-08-25)

  • Add --keys-directory config option to specify where files such as certs and signing keys should be stored in, when using --generate-config or --generate-keys. (PR #250)

  • Allow --config-path to specify a directory, causing synapse to use all \*.yaml files in the directory as config files. (PR #249)

  • Add web_client_location config option to specify static files to be hosted by synapse under /_matrix/client. (PR #245)

  • Add helper utility to synapse to read and parse the config files and extract the value of a given key. For example::

    $ python -m synapse.config read server_name -c homeserver.yaml localhost

    (PR #246)

Changes in synapse v0.10.0-rc2 (2015-08-24)

  • Fix bug where we incorrectly populated the event_forward_extremities table, resulting in problems joining large remote rooms (e.g. #matrix:matrix.org)
  • Reduce the number of times we wake up pushers by not listening for presence or typing events, reducing the CPU cost of each pusher.

Changes in synapse v0.10.0-rc1 (2015-08-21)

Also see v0.9.4-rc1 changelog, which has been amalgamated into this release.

General:

  • Upgrade to Twisted 15 (PR #173)
  • Add support for serving and fetching encryption keys over federation. (PR #208)
  • Add support for logging in with email address (PR #234)
  • Add support for new m.room.canonical_alias event. (PR #233)
  • Change synapse to treat user IDs case insensitively during registration and login. (If two users already exist with case insensitive matching user ids, synapse will continue to require them to specify their user ids exactly.)
  • Error if a user tries to register with an email already in use. (PR #211)
  • Add extra and improve existing caches (PR #212, #219, #226, #228)
  • Batch various storage request (PR #226, #228)
  • Fix bug where we didn't correctly log the entity that triggered the request if the request came in via an application service (PR #230)
  • Fix bug where we needlessly regenerated the full list of rooms an AS is interested in. (PR #232)
  • Add support for AS's to use v2_alpha registration API (PR #210)

Configuration:

  • Add --generate-keys that will generate any missing cert and key files in the configuration files. This is equivalent to running --generate-config on an existing configuration file. (PR #220)
  • --generate-config now no longer requires a --server-name parameter when used on existing configuration files. (PR #220)
  • Add --print-pidfile flag that controls the printing of the pid to stdout of the demonised process. (PR #213)

Media Repository:

  • Fix bug where we picked a lower resolution image than requested. (PR #205)
  • Add support for specifying if a the media repository should dynamically thumbnail images or not. (PR #206)

Metrics:

  • Add statistics from the reactor to the metrics API. (PR #224, #225)

Demo Homeservers:

  • Fix starting the demo homeservers without rate-limiting enabled. (PR #182)
  • Fix enabling registration on demo homeservers (PR #223)

Changes in synapse v0.9.4-rc1 (2015-07-21)

General:

  • Add basic implementation of receipts. (SPEC-99)
  • Add support for configuration presets in room creation API. (PR #203)
  • Add auth event that limits the visibility of history for new users. (SPEC-134)
  • Add SAML2 login/registration support. (PR #201. Thanks Muthu Subramanian!)
  • Add client side key management APIs for end to end encryption. (PR #198)
  • Change power level semantics so that you cannot kick, ban or change power levels of users that have equal or greater power level than you. (SYN-192)
  • Improve performance by bulk inserting events where possible. (PR #193)
  • Improve performance by bulk verifying signatures where possible. (PR #194)

Configuration:

  • Add support for including TLS certificate chains.

Media Repository:

  • Add Content-Disposition headers to content repository responses. (SYN-150)

Micropub support as an Application Service!

2015-07-29 — Tech — 

I was at IndieWebCamp Edinburgh last week and during the hack day I created a Matrix Application Service (AS) which could act as an IndieWeb Micropub client. Any Matrix message sent to the AS (@micropub:domain) would be converted to a request to a Micropub endpoint.

This required the AS to support IndieAuth - which it does by sending !indieauth http://yourdomain.dom to @micropub:domain which then returns an OAuth2 URL to login via. Currently, the AS just supports a 1:1 mapping from m.text to h:entry but in the future, it can be expanded to include categories and potentially the reverse mapping (where Micropub clients can act as Matrix users!).

Overall, it was a great weekend and I look forward to adding more support for IndieWeb protocols in the future.

Synapse 0.9.3 is out! (as of a few weeks ago)

2015-07-10 — Tech — 

Hi all,

We seem to have done that thing again where we were too busy writing new stuff (E2E crypto, all-new React-based Web SDK, long-awaited iOS Console update) to remember that we'd released a new version of Synapse - sorry!

Synapse 0.9.3 was released on July 1st (actually on June 23rd as 0.9.3-rc1, but was released without changes). It's a fairly minor release but does provide some performance improvements and bug fixes - see below for details. Get it from http://github.com/matrix-org/synapse if you haven't already.

Changes in synapse v0.9.3 (2015-07-01) ======================================

General:

  • Fix a memory leak in the notifier. (SYN-412)
  • Improve performance of room initial sync. (SYN-418)
  • General improvements to logging.
  • Remove access_token query params from INFO level logging.

Configuration:

  • Add support for specifying and configuring multiple listeners. (SYN-389)

Application services:

  • Fix bug where synapse failed to send user queries to application services.

The matrix.org IRC bridge now bridges all of Freenode!

2015-06-22 — Tech — 

We've been running an IRC gateway from Matrix to Freenode almost since day 1 - originally it was a simple perl bot written by tm604 using Net::Async::Matrix; later it was rewritten by LeoNerd to be a bit more modular, and nowadays it's a full IRC/Matrix Application Service written in Node.js by Kegan.

Up until the end of last week the bridge was limited to synchronising a fixed number of channels between Freenode and Matrix (#matrix, #matrix-dev, #openwebrtc, #vuc and #hypothes.is, to be precise), as well as any PMs. But as of Friday, with huge thanks to the admins over at Freenode, we can now bridge any channel in Freenode through to Matrix.

Doing this is trivial - you just /join #freenode#channelname:matrix.org from a typical Matrix client - e.g. /join #freenode#Node.js:matrix.org will give you access to all of #Node.js via Matrix, effectively using Matrix as a great big distributed IRC bouncer in the sky ;)

There are a few limitations in the current setup:

  • We only incrementally synchronise the membership lists when folks speak in either IRC or Matrix. This is to avoid flooding either IRC or Matrix with lurkers when the bridge initially joins a channel. We have some plans to improve this in future - see BOTS-53 for some of the sordid details.
  • We don't synchronise joins/parts currently to avoid flooding Matrix with lots of IRC join/part spam. Again, this will improve in future.
  • You can't join +k channels.
  • Kicks/bans/invites and other ACLs currently don't propagate between IRC & Matrix. (I.e. a Matrix user can be kicked from IRC, but it'll continue to sit in Matrix unless also kicked for there). +i chanmode supported however.
  • Bridged public IRC rooms are not yet advertised in the public room list on matrix.org.
  • It should be possible to change the nickname of your IRC user by messaging !nick irc.freenode.net MrFlibble to @appservice-irc:matrix.org. (This isn't actually turned on right now, but should be fixed shortly. See https://github.com/matrix-org/matrix-appservice-irc/blob/master/HOWTO.md#changing-nicks for details.

We'll keep chipping away at making the IRC<->Matrix mapping perfect, but in its current state it's still really usable. It's obviously beta still, but please give it a go and let us know in #matrix:matrix.org how you get on with it!

Android Matrix Console 0.4.0

2015-06-19 — Tech — 

Matrix Console 0.4.0 for Android has been pushed to the Google Play store and shall be available shortly.

This release contains a whole lot of bugfixes, new features, and nicer UI and UX - see the changes file for details, but in summary:

  • Lots of UI and UX improvements (inc. long click on images)
  • Added GCM support (can be enabled/disabled)
  • Added Google analytics support
  • Added badges management
  • Added "orientation" management for images
  • Now offering image resizing before upload
  • Lots of bugfixes!

Introducing all new matrix-js-sdk v0.1.0

2015-06-12 — Tech — 

Hi all,

If things seem a bit quiet here at the moment it's because most of the Matrix team is in the 'build' phase across the board on a bunch of major new projects - including the long-awaited reusable UI component library for the Web, end-to-end encryption support, stable VoIP on mobile and a few other surprises. The first wave of all this new work landed today in the form of a new major version release (0.1.0) of matrix-js-sdk: our javascript SDK library.

The history of JS client support on Matrix is that in the original rush to get a PoC webclient out the door back in September, we jumped straight to talking to the Matrix HTTP API from Angular controllers in 'syweb'. This then got split into an Angular SDK (matrix-angular-sdk) containing a bunch of services for handling the Matrix client state with the actual webapp sitting alongside as an example of the SDK. This then in turn got split into a basic matrix-js-sdk (wrapping the Matrix client-server HTTP API) and matrix-angular-sdk (handling the clientside state and exposing it as Angular services) - and this is how the current demo Angular webapp works at http://matrix.org/beta.

However, this poses a major problem if you want a richer client SDK but don't want to use Angular - e.g. if you prefer another framework (or no framework at all), or are using the SDK in an Application Service or similar. So, this has prompted a major upgrade to the matrix-js-sdk in order to extend it to act both as a simple wrapper of the Matrix Client-Server HTTP API, but also expose a full object and event model to describe the state of a Matrix client together with all the higher level functions to drive it. To quote from the README:

This SDK provides a full object model around the Matrix Client-Server API and emits events for incoming data and state changes. Aside from wrapping the HTTP API, it:
  • Handles syncing (via /initialSync and /events)
  • Handles the generation of "friendly" room and member names.
  • Handles historical RoomMember information (e.g. display names).
  • Manages room member state across multiple events (e.g. it handles typing, power levels and membership changes).
  • Exposes high-level objects like Rooms, RoomState, RoomMembers and Users which can be listened to for things like name changes, new messages, membership changes, presence changes, and more.

Later versions of the SDK will:

  • Automatically retry requests to send messages due to network errors.
  • Automatically retry requests to send messages due to rate limiting errors.
  • Mark events' sent status (e.g. 'not sent').
  • Handle "local echo" of messages sent.
  • Handle queueing of messages.
  • Handle pagination.
  • Expose a RoomSummary which would be suitable for a recents page.
  • Provide different pluggable storage layers (e.g. local storage, database-backed)

It also is designed to provide an API that abstracts both version 1 of the client-server API and the upcoming version 2 (which fixes various thinkos we've discovered along the way in v1).

This is incredibly exciting stuff as it now gives a really robust yet lightweight API for client-side and application-service development in JavaScript. Predictably enough we're using it to develop our new reusable web UI components, and I suspect we'll see new Node-based application services making use of in the near future. Meanwhile, it's more than topical as it should run straight on top of the Tessel JavaScript hardware modules we're giving away this weekend as our prize for the best hack built on Matrix at TADHack - letting the little Tessel ARM system-on-a-chip speak a very rich and native Matrix dialect!

The new matrix-js-sdk comes with comprehensive jsdoc detailing the new API which you can find at http://matrix-org.github.io/matrix-js-sdk/global.html. There are also some basic examples available, including a new simple node-powered command-line client designed for testing the SDK. Finally, if you're interested in reading the design process by which we've converged on the current API, all the details may be found in JIRA on SYJS-5.

So: if you've been itching to get your hands on a much more powerful pure-JS Matrix SDK, now's your chance! It's very early days for the new API so we expect to see a few incompatible changes on the horizon, but we encourage you to have a play and tell us what you think!.

To find out more, go check out the project at http://github.com/matrix-org/matrix-js-sdk, or if you're feeling daring just grab the library with npm install matrix-js-sdk and get hacking!

Synapse 0.9.1 released

2015-06-01 — Tech — 

In the excitement of KamailioWorld last week we completely forgot to mention that we released Synapse 0.9.1.

This is a pretty important performance and stability update of all the new stuff that landed in 0.9.0, as well as landing a major new feature in the form of 'backfill': we finally have the ability to correctly sync in historical conversation history over federation for newly federated rooms. In other words, if you join a remote room you should be able to navigate its history as intended.

We highly recommend upgrading to 0.9.1 for all the performance improvements and backfill support - get it now from http://github.com/matrix-org/synapse.

Changes in synapse v0.9.1 (2015-05-26) ======================================

General:

  • Add support for backfilling when a client paginates. This allows servers to request history for a room from remote servers when a client tries to paginate history the server does not have - SYN-36
  • Fix bug where you couldn't disable non-default pushrules - SYN-378
  • Fix register_new_user script - SYN-359
  • Improve performance of fetching events from the database, this improves both initialSync and sending of events.
  • Improve performance of event streams, allowing synapse to handle more simultaneous connected clients.

Federation:

  • Fix bug with existing backfill implementation where it returned the wrong selection of events in some circumstances.
  • Improve performance of joining remote rooms.

Configuration:

  • Add support for changing the bind host of the metrics listener via the metrics_bind_host option.

iOS Console 0.4.0 released!

2015-05-04 — Tech — 

As a follow-on to the previous post, Apple approved version 0.4.0 of the iOS Matrix Console app over the weekend - grab it now from the app store!. Whilst cosmetically it looks almost identical to 0.3.x, the architecture is completely different given the app is now built using MatrixKit - and stability has improved considerably as a result. Please install and let us know how you get on!

The All New Matrix-IRC Application Service

2015-04-22 — Tech — 
This post has now been edited into a guide - you can find the source in github, and the formatted guide on the matrix.org website!

Like a lot of open source projects, we use IRC a lot. Naturally, we also use Matrix to communicate with each other. For some time now we've had an IRC bot sitting on specific channels to "bridge" together IRC and Matrix. This bot simply sent IRC messages when it received Matrix messages and vice versa. As we started to rely on it more and more though, we realised that there were things that were impossible for simple client-side bots to do by themselves. This spurred the development of Application Services which I introduced in my previous post. In this blog post, I want to outline some of the features and techniques of the IRC application service which we've been working on over the past few weeks.

Features:

  • Specific channel-to-matrix room bridging : This is what the original IRC bot did. You can specify specific channels and specific room IDs, and messages will be bridged.
  • Dynamic channel-to-matrix room bridging : This allows Matrix users to join any channel on an IRC network, rather than being forced to use one of the specific channels configured.
  • Two-way PM support : IRC users can PM the virtual "M-" users and private Matrix rooms will be created. Likewise, Matrix users can invite the virtual "@irc_Nick:domain" user IDs to a room and a PM to the IRC nick will be made.
  • IRC nick changing support : Matrix users are no longer forced to use "M-" nicks and can change them by sending "!nick" messages directly to the bridge.
  • Ident support : This allows usernames to be authenticated for virtual IRC clients, which means IRC bans can be targeted at the Matrix user rather than the entire application service.
The use of the Application Services API means:
  • The bot can reserve user IDs. This prevents humans from registering for @irc_... user IDs which would then clash with the operation of the bot.
  • The bot can reserve room aliases. This prevents humans from register for #irc_... aliases which would then clash with the operation of the bot.
  • The bot can trivially manage hundreds of users. Events are pushed to the application service directly. If you tried to do this as a client-side bot, you would need one event stream connection per virtual user.
  • The bot can lazily create rooms on demand. This means Matrix users can join non-existent room aliases and have the application service quickly track an IRC channel and create a room with that alias, allowing the join request to succeed.
Implementation details:
  • Written in Node.js, designed to be run using forever.
  • Built on the generic matrix-appservice-node framework.
  • Supports sending metrics in statsd format.
  • Uses matrix-appservice-node to provide a standardised interface when writing application services, rather than an explicit web framework (though under the hood matrix-appservice-node is using Express).
At present, the IRC application service is in beta, and is being run on #matrix and #matrix-dev. If you want to give it a go, check it out on Github - it is not currently released on npm. N.B. it requires features from the develop branch of synapse; either run your own synapse off the develop branch or wait a few days for us to release Synapse 0.9.0.

Needless to say, we look forward to this being the first of many full network<->network bridges into Matrix - come chat on #matrix:matrix.org if you'd like to write or run your own! Next up is Lync and XMPP...

Android 0.2.3 SDK and Application released

2015-03-10 — Tech — 

Hi folks,

We released a new version (0.2.3) of the Android Matrix Console application today - this is a decent upgrade over 0.2.2. Changelog below.

If you're an Android user, please install the app from the Play Store and give it a go and give us some feedback on #android:matrix.org or Google Play.

thanks!

Changes in Matrix Android SDK in 0.2.3 (2015-03-10) ===================================================

SDK


Matrix Console

Improvements:

  • Avoid refreshing the home page when it is not displayed.
  • Display a piechart while uploading a media.
  • Refresh the display when some messages are automatically resent (after retrieving a data network connection for example).
  • Update the user rename message to be compliant with the web client.
  • Use the local media files instead of downloading them when they are acknowledged (messages sending).
  • Create a medias management class.
  • Display the offline status in the members list.
  • Avoid creating new homeActivity instance when joining a room from member details sheet.
  • The public rooms list are now saved in the bundle state : it should avoid having a spinner when rotated the device.
  • The animated GIFs are now supported.

Features:

  • Add the rate limits error management. The server could request to delay the messages sending because they were too many messages sent in a short time (to avoid spam).
  • Can take a photo to send it.
  • A chat room page is automatically paginated to fill. It used to get only the ten latest messages : it displayed half filled page on tablet.
  • Add the sending failure reason in the message details (long tap on a message, “Message details”).
  • The user is not anymore notified it the push rules are not fulfilled.
  • Add some room settings (Display all events, hide unsupported events, sort members by last seen time, display left members, display public rooms in the home page).
  • Add various accessibility tweaks.

Bug fixes:

  • The media downloads/uploads were sometimes stuck.
  • The private room creation was broken.
  • SYAND-33 : number of unread messages disappears when entering another room.
  • The RoomActivity creation used to crash when it was cancelled because the Room id param was not provided.
  • The client used to crash when the home server was invalid but started with http.
  • The account creation used to fail if the home server had a trailing slash.
  • SYAND-44 In progress text entry could be saved across crashes.
  • SYAND-38 Inline image viewer in Android app.

Synapse 0.7.0 and matrix-angular-sdk 0.6.2 released!

2015-02-12 — Tech — 

We just pushed out a major new release of Synapse 0.7.0, the python reference server for Matrix, and a minor maintenance update for matrix-angular-sdk 0.6.2, the reference web client implementation.

The emphasis here has been on federation performance and stability - with all the recent interest from FOSDEM and LWN and HackerNews we've been getting a lot of traffic on matrix.org and new servers federating up, so we've been busy profiling and fixing performance on some of the hot paths by adding in-memory caches and similar.

This release is highly recommended if you are running a federated server, as it fixes many denial-of-service failure modes in the federation implementation, as well as many performance improvements

Behind the scenes, there's also lots of work going on for v2 of the Client-Server API (which lets you filter the events your client subscribes to in a room, and combines the 'initial sync' and 'eventstream' APIs into a single simplified '/sync' API) - the alpha of this has landed in 0.7.0, but we don't recommend trying to use it yet. Meanwhile the Application Service API is effectively finished, but hasn't landed on the master branch yet. Synapse also finally has initial support for push notifications for the iOS client now via the sygnal APNS gateway, although we need to actually document how to set this up to be usable in general.

Finally, we have switched to recommending that synapse is installed in a virtualenv rather than into ~/.local, due to various problems with how setup.py's dependency management works. Instructions on setting up a virtualenv can be found in the README.

On the clientside: we've improved performance, enabled identicons for unknown users, and now fully support OpenWebRTC for VoIP calling from Safari, Bowser, and other OpenWebRTC-capable browsers!

Thanks to everyone running servers - please upgrade, tell your friends, and help us grow Matrix!

Changes in synapse v0.7.0 (2015-02-12) ====================================== * Add initial implementation of the query auth federation API, allowing servers to agree on whether an event should be allowed or rejected. * Persist events we have rejected from federation, fixing the bug where servers would keep requesting the same events. * Various federation performance improvements, including: - Add in memory caches on queries such as: * Computing the state of a room at a point in time, used for authorization on federation requests. * Fetching events from the database. * User's room membership, used for authorizing presence updates. - Upgraded JSON library to improve parsing and serialisation speeds. * Add default avatars to new user accounts using pydenticon library. * Correctly time out federation requests. * Retry federation requests against different servers. * Add support for push notifications and push rules. * Add alpha versions of proposed new CSv2 APIs, including ``/sync`` API.
Changes in Matrix Angular SDK 0.6.2 (2015-02-12) ================================================ Bug fixes: - Fixed a bug which caused OpenWebRTC to occasionally fail. - Fixed a bug which caused multiple room initial syncs to occur in rapid succession. Features: - Display a "Joining Room" dialog when joining rooms. - Display identicons for users with no avatar. - Display m.notice events with full formatting. - Add push notification rules to settings. Improvements: - Modified the red/blue notification colours to be more noticeable on a wider range of displays. - Highlight room invitations in blue. - Calculate room names for rooms of 3+ members. - Improved page load performance.

iOS: SDK and Matrix Console 0.2.2 released

2015-02-05 — Tech — 

A new release of the iOS SDK and the iOS Matrix Console app is available on GitHub: https://github.com/matrix-org/matrix-ios-sdk.

If you use CocoaPods to manage your application dependencies, note that the Matrix SDK pod has been updated too.

The changes since the last release are:

 

SDK

Improvements:
  • MXFileStore stores data on a separated thread to avoid blocking the UI thread.
  • MXRestClient: Callback blocks in all MXRestClient methods are now optional.
  • MXEvent: Cleaned up exposed properties and added a description for each of them.
Features:
  • Added API for registering for push notifications.
  • Added generic API methods to make any kind of registration or login flow.
  • Added Identity server API: lookup3pid, requestEmailValidation, validateEmail and bind3PID.
  • Management of event redaction: there is a new method in the SDK to redact an event and the SDK updates its data on redaction event.
Bug fixes:
  • SYIOS-5 - Expose registration API
  • SYIOS-44 - Credentials persist across logout
  • SYIOS-54 - Matrix Console app slightly freezes when receiving a message
  • SYIOS-59 - Infinite loop in case of back pagination on new created room
  • MXRoom: Fixed [MXRoom sendTextMessage]

Matrix Console

Improvements:
  • When long pressing on a message, the app shows the JSON string of the Matrix event.
  • On this screen, the user can redact the event - if he has enough power level.
  • Use home server media repository facilities to use lower image size for thumbnails and avatars
  • Settings screen: show build version with the app version.
  • Settings screen: added an option to hide information related to redacted event.
  • Settings screen: added an option to enable reading of local phonebook. The country is required to internationalise phone numbers.
Features:
  • Push notifications.
  • Added a contacts screen that displays Matrix users the user had interactions with and contacts from the device phonebook.
  • Contacts from the device phonebook who have an email linked to a Matrix user id are automatically recognised.
Bug fixes:
  • SYIOS-53 - multilines text input that expands as you type mutiplines would be nice
  • SYIOS-45 - Need to check the thumbnail params requested by iOS
  • SYIOS-55 - High resolution avatars create memory pressure
  • SYIOS-57 - Back pagination does not work well for self chat
  • SYIOS-56 - add cache size handling in settings
  • SYIOS-60 - In a self chat, Console takes ages to paginate back even if messages are in cache
  • SYIOS-61 - Chat room : cannot scroll to bottom when keyboard is opened whereas the growing textview contains multi-lines text.
  • SYIOS-63 - calculate room names for 3+ memebers if no room name/alias
  • SYIOS-44 - Credentials persist across logout
  • SYIOS-64 - Chat room : unexpected blank lines are added into history when user types in growing textview
  • SYIOS-65 - IOS8 : in case of search in recents, keyboard is not dismisssed when user selects a room.
  • SYIOS-16 Add option in Console to join room thanks to its alias

Synapse 0.6.1 released and other news!

2015-01-07 — Tech — 

Happy 2015 from everyone at Matrix.org!

We're excited to kick off the new year with a major performance upgrade for Synapse: having had a chance to do some profiling over the Christmas break, Synapse 0.6.1 improves performance by up to an order of magnitude thanks to optimising the way events are constructed and manipulated, DB optimisations, etc. Suddenly things are feeling much more snappy and less of a PoC and more of a real system, which hopefully bodes well for 2015! Please upgrade if you're running a homeserver, or install one if you're not - grab the code from http://github.com/matrix-org/synapse.

The iOS SDK and demo app has also been improving lots in the last few weeks - grab the latest code from http://github.com/matrix-org/matrix-ios-sdk. We have zoomable image support; full support for the new media repository in Synapse 0.6.0; file transfer status UI; offline support; snappy message synchronisation and more...

We also released a new simple Python SDK client library for Python called matrix-client, which is now being used by NEB (our general-purpose Matrix helper bot).

Meanwhile, our (constantly evolving) plan for the beginning of 2015 is:

  • Finish v2 of the client-server API based on all the lessons learnt since launch: draft in progress here
  • Finish the Application Services API to allow proper gateways and services in and out of Matrix at last:draft in progress here
  • Re-release the spec based on the above
  • Build robust gateways (IRC, XMPP, SIP and more) on top of the AS API
  • Polish and release the iOS, Android demo clients - including push notification and VoIP support!
  • Overhaul the identity server architecture and add end-to-end crypto

Given the rate at which Matrix is maturing, the next few months should be really exciting - please come join us on #matrix:matrix.org and get involved!

Changes in synapse 0.6.1 (2015-01-07) =====================================
  • Major optimizations to improve performance of initial sync and event sending in large rooms (by up to 10x)
  • Media repository now includes a Content-Length header on media downloads.
  • Improve quality of thumbnails by changing resizing algorithm.

Synapse 0.6.0a released!

2014-12-19 — Tech — 

We're proud to announce the new 0.6.0 major release of Synapse: the Python reference implementation of Matrix - as well as 0.6.0 of the AngularJS example Matrix client, as a special early Christmas present from the Matrix team :)

The main priority here has been hunting down stability bugs whilst also adding in a few more features. The most exciting new feature is the new Media/Content API: this allows every homeserver to replicate, cache and thumbnail any files associated with Matrix rooms, rather than relying on retrieving them from the origin homeserver as was previously the case. This is really cool, as it effectively makes Matrix a distributed content distribution network and gives the same replication semantics to file attachments as to the rest of JSON in rooms.

We've also landed a major new branch which changes the way events are represented internally in order to make them immutable, and fixes a whole range of minor stability issues. In other news, performance issues are still a work in progress...

To get involved, head over to https://github.com/matrix-org/synapse and install and upgrade today!

Changes in synapse 0.6.0 (2014-12-19) =====================================
  • Add new API for media upload and download that supports thumbnailing.
  • Replicate media uploads over multiple homeservers so media is always served to clients from their local homeserver. This obsoletes the --content-addr parameter and confusion over accessing content directly from remote homeservers.
  • Implement exponential backoff when retrying federation requests when sending to remote homeservers which are offline.
  • Implement typing notifications.
  • Fix bugs where we sent events with invalid signatures due to bugs where we incorrectly persisted events.
  • Improve performance of database queries involving retrieving events.
Changes in Matrix Angular SDK 0.6.0 (2014-12-19) ================================================

Breaking changes:

  • Uploading files in the web client will now hit the new content repository URL introduced in Synapse 0.6, and be incompatible with previous Matrix clients.

Bug fixes:

  • Fixed a bug which caused the event stream to become wedged when the computer is asleep.
  • Fixed a bug which caused the recents to update but not the message window.

Features:

  • Typing notifications will now be sent.
  • Typing notifications will now be displayed for other users.
  • Use the new content repository introduced in Synapse 0.6 when uploading files.

Improvements:

  • Display more error dialogs rather than silently failing.
  • Display loading spinners on signup.
  • Display feedback when joining a room.
  • CTRL + clicking on a recents entry will now open that room in a new tab.
  • Clicking on links in messages will now open them in a new tab.
  • Provide a progress dialog when uploading files.
  • Display a red bar when the event stream connection is lost for an extended period of time.

Synapse 0.5.4 released

2014-12-03 — Tech — 

We just pushed a bugfix update to Synapse 0.5, mainly to fix a memory leak where federated events could get leaked whilst retrying to send them to a remote server which is unavailable. Please upgrade, especially if you've noticed synapse hogging RAM!

Changes in synapse 0.5.4 (2014-12-03) =====================================
  • Fix presence bug where some rooms did not display presence updates for remote users.
  • Do not log SQL timing log lines when started with "-v"
  • Fix potential memory leak.

Synapse 0.5.3a released!

2014-11-29 — Tech — 

Since we exited alpha and released Synapse 0.5.0 last week there's been a flurry of bug fix releases as we ran around firefighting some of the more exciting problems which popped up thanks to merging the federation-auth and event-signing branches.

We released 0.5.3a on Nov 27, which seems to now be pretty stable - if you were holding off on upgrading your homeserver and trying to federate with the new 0.5 release branch, now would be a great time to press the button!

Most excitingly, we believe we have finally fixed the PyNaCL dependency problems which have plagued pretty much anyone setting up a homeserver. This was a problem in PyNaCL itself - huge thanks to Mark for hunting it down and sending the PyNaCL team pull requests to get it fixed. As a result, installing synapse as of 0.5.3a really should be a one-liner at last (please let us know if it isn't!):

pip install --user --process-dependency-links https://github.com/matrix-org/synapse/tarball/master

The full changelog of what's been going on since 0.5.0 is as follows:

Changes in synapse 0.5.3a (2014-11-27) ===================================== * Depend on a fixed version of PyNaCL * Fix bug that caused joining a remote room to fail if a single event was not signed correctly. * Fix bug which caused servers to continuously try and fetch events from other servers. Changes in synapse 0.5.2 (2014-11-26) ===================================== * Fix major bug that caused rooms to disappear from people's initial sync. Changes in synapse 0.5.1 (2014-11-26) ===================================== See UPGRADES.rst for specific instructions on how to upgrade. * Fix bug where we served up an Event that did not match its signatures. * Fix regression where we no longer correctly handled the case where a homeserver receives an event for a room it doesn't recognise (but is in.)

Synapse enters beta with the release of v0.5.0!

2014-11-20 — Tech — 

We're really excited to release the biggest update yet to Synapse (the python/twisted reference Matrix homeserver implementation) - version 0.5.0 is out!

This effectively concludes our alpha testing phase - Synapse's security model is now feature complete and we are now cautiously encouraging people to run their own Synapse installation on the 'net; federate up; and help us squash all and any bugs as we get beta testing.

Synapse 0.5.0 has huge database schema changes from previous versions as we have landed both the event-signing and federation-authorisation branches and so added full cryptographic signing of all matrix events and unified the client-server and server-server model of events/PDUs.  As a result, migrating history is Hard - we recommend a clean start if possible; if not, come talk to us in #matrix:matrix.org.  UPGRADE.rst has the details.

The full changelog is as per below - please upgrade your homeserver or install a new one if you're new here and join Matrix and help us create a robust open distributed messaging fabric for the web!

Changes in synapse 0.5.0 (2014-11-19) ===================================== This release includes changes to the federation protocol and client-server API that is not backwards compatible. This release also changes the internal database schemas and so requires servers to drop their current history. See UPGRADES.rst for details. Homeserver: * Add authentication and authorization to the federation protocol. Events are now signed by their originating homeservers. * Implement the new authorization model for rooms. * Split out web client into a seperate repository: matrix-angular-sdk. * Change the structure of PDUs. * Fix bug where user could not join rooms via an alias containing 4-byte UTF-8 characters. * Merge concept of PDUs and Events internally. * Improve logging by adding request ids to log lines. * Implement a very basic room initial sync API. * Implement the new invite/join federation APIs. Webclient: * The webclient has been moved to a seperate repository.

Synapse 0.3.4 released!

2014-09-25 — Tech — 

Hi all,

Synapse 0.3.4 is out! The main changes here are a huge fleet of bugfixes to the webclient, and the addition of TURN support for VoIP relaying. This makes VoIP/Video call setup much more reliable (although the UI still needs some love).

Meanwhile, we set an IRC bridge (https://github.com/tm604/Matrix-IRCBridge) running linking #matrix:matrix.org and #matrix-dev:matrix.org to their Freenode counterparts (#matrix & #matrix-dev respectively) - much fun to see innocent IRC users assimilated up into Matrix, and for us to now be using Matrix for /all/ our development chat and interaction (whether it's through a Matrix client or an IRC client).

To upgrade to the latest homeserver, just git pull && ./synctl restart. If you want to start playing with VoIP in earnest on your own homeserver, you'll need to install a TURN server - see docs/turn-howto.rst for details on how to set one up.

http://matrix.org/alpha has already been upgraded to 0.3.4, so feel free to play with all the new stuff there.

thanks,

Matthew

Changes in synapse 0.3.4 (2014-09-25)

This version adds support for using a TURN server. See docs/turn-howto.rst on how to set one up.

Homeserver:

  • Add support for redaction of messages.
  • Fix bug where inviting a user on a remote home server could take up to 20-30s.
  • Implement a get current room state API.
  • Add support specifying and retrieving turn server configuration.

Webclient:

  • Add button to send messages to users from the home page.
  • Add support for using TURN for VoIP calls.
  • Show display name change messages.
  • Fix bug where the client didn't get the state of a newly joined room until after it has been refreshed.
  • Fix bugs with tab complete.
  • Fix bug where holding down the down arrow caused chrome to chew 100% CPU.
  • Fix bug where desktop notifications occasionally used "Undefined" as the display name.
  • Fix more places where we sometimes saw rooms as IDs rather than aliases.
  • Fix bug which caused lag when entering text in the text box.

Synapse 0.3.3 released

2014-09-22 — Tech — 

Hello world,

Synapse 0.3.3 was released today, concentrating on both feature & stability improvements on the webclient. The big news is that 1:1 video calling via WebRTC is implemented (although we're still working on some stability issues, and Synapse doesn't support TURN yet. This will be coming in the next few days however!).

Play with the demo server at http://matrix.org/alpha, or update your own homeserver with a git pull && ./synctl restart or by cloning a new checkout with git clone https://github.com/matrix-org/synapse.git

--Matthew

Changes in synapse 0.3.3 (2014-09-22)

Homeserver:

  • Fix bug where you continued to get events for rooms you had left.

Webclient:

  • Add support for video calls with basic UI.
  • Fix bug where one to one chats were named after your display name rather than the other person's.
  • Fix bug which caused lag when typing in the textarea.
  • Refuse to run on browsers we know won't work.
  • Trigger pagination when joining new rooms.
  • Fix bug where we sometimes didn't display invitations in recents.
  • Automatically join room when accepting a VoIP call.
  • Disable outgoing and reject incoming calls on browsers we don't support VoIP in.
  • Don't display desktop notifications for messages in the room you are non-idle and speaking in.

Synapse 0.3.2 released

2014-09-18 — Tech — 

We just pushed out a major new release (0.3.0) of Synapse: the current reference Matrix homeserver, swiftly followed by some hotfixes (0.3.1, 0.3.2).

The big changes are:

  • Change the VoIP setup API to be more efficient (bundling together media stream candidates into a single setup object). API breaking change!
  • Change the Registration API to be more consistent. API breaking change!
  • Lots of nice usability refinements to webclient - e.g. per-message notifications; cursor-key history navigation; faster room loading
  • More bugfixes and stability fixes on the homeserver

Please reload the page if you're lurking on http://matrix.org/alpha to pull in the new webclient.

If you're running a homeserver - please upgrade with a quick git pull && ./synctl restart.

If you're not running a homeserver... why not? ;D

--Matthew

Changes in synapse 0.3.2 (2014-09-18)

Webclient:

  • Fix regression where an empty "bing words" list in old accounts didn't send notifications when it should have done.

Changes in synapse 0.3.1 (2014-09-18)

This is a release to hotfix v0.3.0 to fix two regressions.

Webclient:

  • Fix a regression where we sometimes displayed duplicate events.
  • Fix a regression where we didn't immediately remove rooms you were banned in from the recents list.

Changes in synapse 0.3.0 (2014-09-18)

See UPGRADE for information about changes to the client server API, including breaking backwards compatibility with VoIP calls and registration API.

Homeserver:

  • When a user changes their displayname or avatar the server will now update all their join states to reflect this.
  • The server now adds "age" key to events to indicate how old they are. This is clock independent, so at no point does any server or webclient have to assume their clock is in sync with everyone else.
  • Fix bug where we didn't correctly pull in missing PDUs.
  • Fix bug where prev_content key wasn't always returned.
  • Add support for password resets.

Webclient:

  • Improve page content loading.
  • Join/parts now trigger desktop notifications.
  • Always show room aliases in the UI if one is present.
  • No longer show user-count in the recents side panel.
  • Add up & down arrow support to the text box for message sending to step through your sent history.
  • Don't display notifications for our own messages.
  • Emotes are now formatted correctly in desktop notifications.
  • The recents list now differentiates between public & private rooms.
  • Fix bug where when switching between rooms the pagination flickered before the view jumped to the bottom of the screen.
  • Add support for password resets.
  • Add bing word support.
  • Update VoIP implementation to use new API

Registration API:

  • The registration API has been overhauled to function like the login API. In practice, this means registration requests must now include the following: 'type':'m.login.password'. See UPGRADE for more information on this.
  • The 'user_id' key has been renamed to 'user' to better match the login API.
  • There is an additional login type: 'm.login.email.identity'.
  • The command client and web client have been updated to reflect these changes.

Synapse v0.2.3 released

2014-09-13 — Tech — 

We're back home from TechCrunch where we had a great time arguing the pros and cons of Matrix versus XMPP with anyone and everyone - thanks to all who came to talk to us!

Meanwhile, Synapse v0.2.3 was released yesterday: the emphasis this week has been on improving stability issues on the server, and lots of cosmetic and functionality fixes on the webclient.

If you're running a homeserver, please upgrade asap - upgrading has never been easier - assuming you've checked out the master branch, simply:

git pull ./synctl restart

...and you will be up and running on 0.2.3. We keep the master branch tracking the latest release.  http://matrix.org/alpha is already on the latest release should you wish to experiment first!

Alternatively, if you're installing from scratch, try:

git clone https://github.com/matrix-org/synapse.git python setup.py develop --user ./synctl start # to generate a default config ./synctl start # to actually set it running

and then head over to http://localhost:8008 to play with your very own Matrix homeserver!

Changes in synapse 0.2.3 (2014-09-12)

Homeserver:

  • Fix bug where we stopped sending events to remote home servers if a user from that home server left, even if there were some still in the room.
  • Fix bugs in the state conflict resolution where it was incorrectly rejecting events.
  • Many stability fixes

Webclient:

  • Display room names and topics.
  • Allow setting/editing of room names and topics.
  • Display information about rooms on the main page.
  • Handle ban and kick events in real time.
  • VoIP UI and reliability improvements.
  • Add glare support for VoIP.
  • Improvements to initial startup speed.
  • Don't display duplicate join events.
  • Show message sent/sending state via opacity
  • Multiline message sending support
  • Many other bug & stability fixes