Tech

Introducing Client Server API 0.4, and the first ever stable IS, AS and Push APIs spec releases!

Hi folks,

As many know, we've been on a massive sprint to improve the spec - both fixing omissions where features have been implemented in the reference servers but were never formalised in the spec, and fixing bugs where the spec has thinkos which stop us from being able to ratify it as stable and thus fit for purpose .

In practice, our target has been to cut stable releases of all the primary Matrix APIs by the end of August - effectively declaring Matrix out of beta, at least at the specification level.  For context: historically only one API has ever been released as stable - the Client Server API, which was the result of a similar sprint back in Jan 2016. This means that the Server Server (SS) API, Identity Service (IS) API, Application Service (AS) API and Push Gateway API have never had an official stable release - which has obviously been problematic for those implementing them.

However, as of the end of Friday Aug 31, we're proud to announce the first ever stable releases of the IS, AS and Push APIs!

To the best of our knowledge, these API specs are now complete and accurately describe all the current behaviour implemented in the reference implementations (sydent, synapse and sygnal) and are fit for purpose. Any deviation from the spec in the reference implementations should probably be considered a bug in the impl. All changes take the form of filling in spec omissions and adding clarifications to the existing behaviour in order to get things to the point that an independent party can implement these APIs without having to refer to anything other than the spec.

This is the result of a lot of work which spans the whole Spec Core Team, but has been particularly driven by TravisR, who has taken the lead on this whole mission to improve the spec.  Huge thanks are due to Travis for his work here, and also massive thanks to everyone who has suffered endured reviewed his PRs and contributed to the releases.  The spec is looking unrecognisably better for it - and Matrix 1.0 is feeling closer than ever!

Alongside the work on the IS/AS/Push APIs, there has also been a massive attempt to plug all the spec omissions in the Client Server API.  Historically the CS API releases have missed some of the newer APIs (and of course always miss the ones which postdate a given release), but we've released the APIs which /have/ been specified as stable in order to declare them stable.  However, in this release we've tried to go through and fill in as many remaining gaps as possible.

The result is the release of Client Server API version 0.4. This is a huge update - increasing the size of the CS API by ~40%. The biggest new stuff includes fully formalising support for end-to-end encryption (thanks to Zil0!), versioning for rooms (so we can upgrade rooms to new versions of the protocol), synchronised read markers, user directories, server ACLs, MSISDN 3rd party ids, and .well-known server discovery (not that it's widely used yet), but for the full picture, best bet is to look at the changelog (now managed by towncrier!).  It's probably fair to say that the CS API is growing alarmingly large at this point - Chrome says that it'd be 223 A4 pages if printed. Our solution to this will be to refactor it somehow (and perhaps switch to a more compact representation of the contents).

Some things got deliberately missed from the CS 0.4 release: particularly membership Lazy Loading (because we're still testing it out and haven't released it properly in the wild yet), the various GDPR-specific APIs (because they may evolve a bit as we refine them since the original launch), finalising ID grammars in the overall spec (because this is surprisingly hard and subtle and we don't want to rush it) and finally Communities (aka Groups), as they are still somewhat in flux.

Meanwhile, on the Server to Server API, there has also been a massive amount of work.  Since the beginning of July it's tripled in size as we've filled in the gaps, over the course of >200 commits (>150 of which from Travis).  If you take a look at the current snapshot it's pretty unrecognisable from the historical draft; with the main changes being:

• Adding the new State Resolution algorithm to address flaws in the original one.  This has been where much of our time has gone - see MSC1442 for full details.  Adopting the new algorithm requires rooms to be recreated; we'll write more about this in the near future when we actually roll it out.
• Adding room versioning so we can upgrade to the new State Resolution algorithm.
• Everything is now properly expressed as Swagger (OpenAPI), just like the CS API
• Adding all the details for E2E encryption (including dependencies like to-device messaging and device-list synchronisation)
• Improvements in specifying how to authorize inbound events over federation
• Document federation APIs such as /event_auth and /query_auth and /get_missing_events
• Document 3rd party invites over federation
• Document the /user/* federation endpoints
• Document Server ACLs
• Document read receipts over federation
• Document presence over federation
• Document typing notifications over federation
• Document content repository over federation
• Document room directory over federation
• ...and many many other minor bug fixes, omission fixes, and restructuring for coherency - see https://github.com/matrix-org/matrix-doc/issues/1464 for an even longer list :)

However, we haven't finished it all: despite our best efforts we're running slightly past the original target of Aug 31.  The current state of play for the r0 release overall (in terms of pending issues) is: ...and you can see the full breakdown over at the public Github project dashboard.

The main stuff we still have remaining on the Server/Server API at this point is:

• Better specifying how we validate inbound events. See MSC1646 for details & progress.
• Switching event IDs to be hashes. See MSC1640 for details and progress.
• Various other remaining security considerations (e.g. how to handle malicious auth events in the DAG; how to better handle DoS situations).
• Merging in the changes to authoring m.room.power_levels (as per MSC1304)
• Formally specifying the remaining identifiers which lack a formal grammar - MSC1597 and particularly room aliases ( MSC1608)

The plan here is to continue speccing and implementing these at top priority (with Travis continuing to work fulltime on spec work), and we'll obviously keep you up-to-date on progress.  Some of the changes here (e.g. event IDs) are quite major and we definitely want to implement them before speccing them, so we're just going to have to keep going as fast as we can. Needless to say we want to cut an r0 of the S2S API alongside the others asap and declare Matrix out of beta (at least at the spec level :)

In terms of visualising progress on this spec mission it's interesting to look at the rate at which we've been closing PRs: this graph shows the total number of PRs which are in state ‘open' or ‘closed' on any given day:

...which clearly shows the original sprint to get the r0 of the CS API out the door at the end 2015, and then a more leisurely pace until the beginning of July 2018 since which the pace has picked up massively.  Other ways of looking at include the number of open issues...

...or indeed the number of commits per week…

...or the overall Github Project activity for August.  (It's impressive to see Zil0 sneaking in there on second place on the commit count, thanks to all his GSoC work documenting E2E encryption in the spec as part of implementing it in matrix-python-sdk!)

Anyway, enough numerology.  It's worth noting that all of the dev for r0 has generally followed the proposed Open Governance Model for Matrix, with the core spec team made up of both historical core team folk (erik, richvdh, dave & matthew), new core team folk (uhoreg & travis) and community folk (kitsune, anoa & mujx) working together to review and approve the changes - and we've been doing MSCs (albeit with an accelerated pace) for anything which we feel requires input from the wider community.  Once the Server/Server r0 release is out the door we'll be finalising the open governance model and switching to a slightly more measured (but productive!) model of spec development as outlined there.

Meanwhile, Matrix 1.0 gets ever closer.  With (almost) all this spec mission done, our plan is to focus more on improving the reference implementations - particularly performance in Synapse,  UX in matrix-{'{'}react,ios,android{'}'}-sdk as used by Riot (especially for E2E encryption), and then declare a 1.0 and get back to implementing new features (particularly Editable Messages and Reactions) at last.

We'd like to thank everyone for your patience whilst we've been playing catch up on the spec, and hope you agree it's been worth the effort :)

Matthew & the core spec team.

Following on from the previous post, we have an update from zil0 on his GSoC project, which entailed implementing E2E support in the Matrix Python SDK.

---

The goal of my project is to implement Matrix's end-to-end encryption protocol in Python, as part of matrix-python-sdk. My mentors are Richard van der Hoff (richvdh) and Hubert Chathi (uhoreg).

It was easy to get started on the project, since the simple parts came first (adding API calls), and then the whole process to follow is documented in an implementation guide, while there is also the reference implementation in JavaScript. And most importantly, the community is nice. :)

Some parts of the work consist in wrapping around the cryptographic primitives implemented in libolm (via Python bindings), in order to handle encrypted events. Others are less straightforward, such as tracking device lists of users, or finding the right way to persist keys and related data between startups.

An interesting aspect of this project is that I am working on a new part of the Python SDK, while also having to integrate with existing code, which is a cool balance between freedom and guidelines.

As the encryption documentation is a bit outdated and incomplete, one (fun) difficulty is to look for information across old issues, Gdocs and source code (and asking my mentor when in doubt). For anyone trying to implement E2E, it should be better by the end of the project, as I am currently working on documenting the missing bits.

I have had a great experience so far. Working on an open source project differs from my previous coding experiences, as people are actually going to use what I write! I have learnt to think about the best design from a usability point of view, discuss different approaches, and I had to write tests and document my code, which sadly is not something I do on personal projects. I enjoyed reviews, and the discussions they led to. And of course I have learnt quite an interesting lot about the E2E voodoo, along with some new Python tricks.

Currently, the implementation is in a working state. Some of the code is merged, and some is awaiting review. It is possible to try it here before everything is merged. The project will be finished in about one week, after some tidying up and when I release device verification and key sharing, which should be the last missing features compared to Riot.

As you may know, for the last few months anoa (Andrew) and APWhiteHat have been working on Dendrite, the next generation Matrix homeserver, written in Go. We asked for an update on their progress, and Andrew provided the blog post below. Serious progress has been made on Dendrite this summer!

---

Hey everyone, my name is Andrew Morgan and I've been working full-time over the summer on Dendrite, our next-generation Matrix homeserver. Over the last two months, I've seen the project transform from a somewhat functioning toy server to a near-production-ready homeserver that is working towards complete feature support. I've appreciated the thought put into the project since day one, and enjoy the elegance of the multi-component design. Documentation is fairly decent at the moment, but comments are plentiful throughout the codebase, while the code itself tends towards simple and maintainable rather than complex and unmanageable.

Application Service Integration

The main focus of my time here has been on the implementation of application service support for Dendrite. Application services are external programs that act as privileged extensions to a homeserver, allowing such functionality as bots in rooms and bridges to third-party networks. Supporting application services requires a few different bits and pieces to be set up. Currently all planned features have a PR for them, with the bold items already merged:

• Sending events to application services
• Support user masquerading for events
• Support editing event timestamps
• Support room alias querying
• Support user ID querying
• Support third party lookup proxying

As you can see a decent portion of the functionality is already in master! The rest will hopefully follow after some further back and forth.

Google Summer of Code

I certainly haven't been going at this all on my own. Alongside extensive help from Erik, who's been mentoring me, our resident Google Summer of Code student, APWhiteHat, has been tackling feature after feature in Dendrite wherever he can find them. Application services received a good deal of help on client-server endpoint authentication side, however, APWhiteHat has mostly been focusing on federation and some other very useful pieces. While his GSoC period still has a week or so before its conclusion, he has so far implemented:

• Idempotency to roomserver event processing to prevent duplication
• Username auto generation
• Tokens library based on macaroons
• Lots of left-over federation stuff: state API & get missing events being the major ones
• AS support to clientapi auth
• Typing server: handling of PUT /typing by clientapi
• More typing server stuff on its way

From my perspective, APWhiteHat was an excellent developer to work with. He asked good questions and was quick to answer any myself or the community had as well. His code reviews were also very comprehensive. I learned a lot from working with him and everyone else :)

OpenTracing and Prometheus Monitoring

Placing any large server into a production environment requires extensive monitoring capabilities in order to ensure operations are running smoothly. To that effect, Dendrite has been both the addition of OpenTracing and Prometheus support. Prometheus, also used heavily in Synapse, allows a homeserver operator to track a wide range of data including endpoint usage, resource management as well as user statistics over any given range of time.

In Dendrite, we are taking this one step further by introducing OpenTracing, a language and platform-agnostic framework for tracking the journey of an endpoint call from incoming request to outgoing response, with every method, hierarchy change and database call in between. It will be immensely useful in tracking down performance issues, as well as providing insight into the most critical paths throughout the codebase and where we should focus most of our optimization efforts on. It also comes with a lovely dashboard courtesy of Jaeger:

Community

We've also seen some encouraging interest and development work from the community in the past couple months. While PR review from our own side is admittedly slow due to our focus on getting the foundational work in place, that hasn't stopped both old and new developers from sending in PRs and performing code reviews. A huge thank you to everyone involved! From this we've gotten API implementations and application service fixes from @turt2live, an end-to-end encryption implementation from @fadeAce, filtering support from @CromFr, and some PRs and numerous helpful review comments from @krombel.

We've also started to see some people running Dendrite in live environments, which is incredibly exciting for us to see! While Dendrite is not considered production-ready yet (though it moves closer every day), if you are interested in giving it a go please consult the quickstart installation guide. We look forward to any feedback you may have!

Hi all,

We've just released Synapse 0.33.0!  This is a major performance upgrade which speeds up /sync (i.e. receiving messages) by a factor of almost 2x!  This has already made a massive difference to the CPU usage and snappiness of the matrix.org homeserver since we rolled it out a few days ago - you can see the drop in sync worker CPU just before midday on July 17th; previously we were regularly hitting the CPU ceiling (at which point everything grinds to a halt) - now we're back down hovering between 40% and 60% CPU (at the current load).  This is actually fixing a bug which crept in around Synapse 0.31, so please upgrade - especially if Synapse has been feeling slower than usual recently, and especially if you are still on Synapse 0.31.

Meanwhile we have a lot of new stuff coming on the horizon - a whole new algorithm for state resolution (watch this space for details); incremental state resolution (at last!) to massively speed up state resolution and mitigate extremities build up (and speed up the synapse master process, which is now the bottleneck again on the matrix.org homeserver); better admin tools for managing resource usage, and all the Python3 porting work (with associated speedups and RAM & GC improvements).  Fun times ahead!

The full changelog follows below; as always you can grab Synapse from https://github.com/matrix-org/synapse.   Thanks for flying Matrix!

Synapse 0.33.0 (2018-07-19)

Bugfixes

• Disable a noisy warning about logcontexts. (#3561)

Synapse 0.33.0rc1 (2018-07-18)

Features

• Enforce the specified API for report_event. (#3316)
• Include CPU time from database threads in request/block metrics. (#3496, #3501)
• Add CPU metrics for _fetch_event_list. (#3497)
• Optimisation to make handling incoming federation requests more efficient. (#3541)

Bugfixes

• Fix a significant performance regression in /sync. (#3505, #3521, #3530, #3544)
• Use more portable syntax in our use of the attrs package, widening the supported versions. (#3498)
• Fix queued federation requests being processed in the wrong order. (#3533)
• Ensure that erasure requests are correctly honoured for publicly accessible rooms when accessed over federation. (#3546)

Misc

• Refactoring to improve testability. (#3351, #3499)
• Use isort to sort imports. (#3463, #3464, #3540)
• Use parse and asserts from http.servlet. (#3534, #3535).

Hi all,

On Monday (2018-06-11) we had an incident where #matrix:matrix.org was hijacked by a malicious user pretending to join the room immediately after its creation in 2014 and then setting an m.room.power_levels event ‘before' the correct initial power_level for the room.

Under normal circumstances this should be impossible because the initial m.room.power_levels for a room should be set before its m.room.join_rules event, meaning users who join the room are subject to its power levels. However, back before we'd even released Synapse, the first two rooms ever created in Matrix (#test:matrix.org and #matrix:matrix.org) were manually created and set the join_rules before the power_levels event, letting users join before the room's power_levels were defined, and so were vulnerable to this attack. We've since re-created #matrix:matrix.org - please re-/join the room if you haven't already!

As a defensive measure, we are releasing a security update of Synapse (0.31.2) today which changes the rules used to authenticate power_level events, such that we fail-safe rather than fail-deadly if the existing auth mechanisms fail. In practice this means changing the default power level required to set state to be 50 rather than 0 if there is no power_levels event present, thus meaning that only the room creator can set the initial power_levels event.

We are not aware of anyone abusing this (other than the old #matrix:matrix.org room) but we'd rather be safe than sorry, so would recommend that everyone upgrade as soon as possible.

This of course constitutes a change to the spec, so full technical details and ongoing discussion around the Matrix Spec Change proposal can be followed over at MSC1304.

EDIT: if you are aware of your server participating in rooms whose first power_levels event is deliberately set by a different user to their creator, please let us know asap (and don't upgrade!)

This work is all part of a general push to finalise and harden and fully specify the Server-Server API as we push towards a long-awaited stable release of Matrix!

As always, you can get the new update from https://github.com/matrix-org/synapse/releases/tag/v0.31.2 or from any of the sources mentioned at https://github.com/matrix-org/synapse.

thanks, and apologies for the inconvenience.

Changes in synapse v0.31.2 (2018-06-14)

SECURITY UPDATE: Prevent unauthorised users from setting state events in a room when there is no m.room.power_levels event in force in the room. (PR #3397)

Discussion around the Matrix Spec change proposal for this change can be followed at https://github.com/matrix-org/matrix-doc/issues/1304.

Hi all,

We've been able to start investing more time in advancing the Matrix Specification itself over the last month or so thanks to Ben joining the core team (and should be able to accelerate even more with uhoreg joining in a few weeks!)  The first step in the new wave of work has been to provide much better infrastructure for the process of actually evolving the spec - whether that's from changes proposed by the core team or the wider Matrix Community.

So, without further ado, we'd like to introduce https://matrix.org/docs/spec/proposals - a dashboard for all the spec change proposals we've accumulated so far (ignoring most of the ones which have already been merged), as well as a clearer workflow for how everyone can help improve the Matrix spec itself.  Part of this is introducing a formal numbering system - e.g. MSC1228 stands for Matrix Spec Change 1228 (where 1228 is the ID of the Github issue on the github.com/matrix-org/matrix-doc/issues repository that tracks the proposal).

Please note that these are NOT like XEPs or RFCs - i.e. optional proposals or add-ons to the protocol; instead they are literally proposals for changes to the Matrix Spec itself.  Once merged into the spec, they are only of historical interest.

We've also created a new room: #matrix-spec:matrix.org to discuss specific spec proposal changes - please join if you want to track how proposals are evolving! (Conversation is likely to fork off into per-proposal rooms or overflow into #matrix-dev:matrix.org or #matrix-architecture:matrix.org depending on traffic levels, however).

Feedback would be much appreciated on this - so please head over to #matrix-spec:matrix.org and let us know how it feels and how it could do better.

This is also a major step towards properly formalising Matrix.org's governance model - hopefully the changes above are sufficient to improve the health of the evolution of the Spec as we work towards an initial stable release later this year, and then you should expect to see a spec proposal for formal governance once we've (at last!) exited beta :)

Huge thanks to Ben for putting this together, and thanks to everyone who's contributed so far to the spec - we're looking forward to working through the backlog of proposals and turning them all into merged spec PRs!!

Matthew

Hi all,

Many people will have noticed disruption in #matrix:matrix.org and #matrix-dev:matrix.org on Sunday, when a validation bug in Synapse was exploited which allowed a malicious event to be inserted into the room with 'depth' value that made the rooms temporarily unusable. Whilst a transient workaround was found at the time (thanks to /dev/ponies, kythyria and Po Shamil for the workaround and to Half-Shot for working on a proposed fix), we're doing an urgent release of Synapse 0.28.1 to provide a temporary solution which will mitigate the attack across all rooms in upgraded servers and un-break affected ones.  Meanwhile we have a full long-term fix on the horizon (hopefully) later this week.

This vulnerability has already been exploited in the wild; please upgrade as soon as possible.

Synapse 0.28.1 is available from https://github.com/matrix-org/synapse/releases/tag/v0.28.1 as normal.

The 'depth' parameter is used primarily as a way for servers to signal the intended cosmetic order of their events within a room (particularly when the room's message graph has gaps in it due to the server being offline, or due to users backfilling old disconnected chunks of conversation). This means that affected rooms may experience message ordering problems until a full long-term fix is provided, which we're working on currently (and tentatively involves no longer trusting 'depth' information from servers).  For full details you can see the proposal documents for the temporary fix in 0.28.1 and the options for the imminent long-term fix.

We'd like to acknowledge jzk for identifying the vulnerability, and Max Dor for providing feedback on the fixes.

As a general reminder, Synapse is still beta (as is the Matrix spec) and the federation API particularly is still being debugged and refined and is pre-r0.0.0. For the benefit of the whole community, please disclose vulnerabilities and exploits responsibly by emailing security@ or DMing someone from +matrix:matrix.org. Thanks.

Changes in synapse v0.28.1 (2018-05-01)

SECURITY UPDATE

• Clamp the allowed values of event depth received over federation to be [0, 2^63 - 1]. This mitigates an attack where malicious events injected with depth = 2^63 - 1 render rooms unusable. Depth is used to determine the cosmetic ordering of events within a room, and so the ordering of events in such a room will default to using stream_ordering rather than depth (topological_ordering). This is a temporary solution to mitigate abuse in the wild, whilst a long solution is being implemented to improve how the depth parameter is used.Full details at https://docs.google.com/document/d/1I3fi2S-XnpO45qrpCsowZv8P8dHcNZ4fsBsbOW7KABI
• Pin Twisted to <18.4 until we stop using the private _OpenSSLECCurve API.

Hi all,

Heads up that we made an emergency release of Riot/Web 0.13.5 a few hours ago to fix a XSS vulnerability found and reported by walle303 - many thanks for disclosing it responsibly.

Please upgrade to Riot/Web 0.13.5 asap. If you're using riot.im/app or riot.im/develop this simply means hitting Refresh; otherwise please upgrade your Riot deployment as soon as possible. Alpine, Debian and Fedora/RPM packages are already updated - huge thanks to the maintainers for the fast turnaround.

The issue lies in the relatively obscure external_url feature, which lets bridges specify a URL for bridged events, letting Riot/Web users link through to the 'original' event (e.g. a twitter URL on a bridged tweet).  The option is hidden in a context menu and labelled "Source URL", and is only visible on events which have the external_url field set.  Unfortunately Riot/Web didn't sanitise the URL correctly, allowing a malicious URL to be injected - and this has been the case since the feature landed in Riot 0.9.0 (Nov 2016).

If you're not able to upgrade to Riot/Web 0.13.5 for some reason, then please do not click on the 'Source URL' feature on the event context menu:

Apologies for the inconvenience,

thanks,

Matthew

Hi folks,

Synapse 0.26 is here (with no changes since RC1 which we released just before Christmas).  It's a general maintenance release, albeit with a few new features but mainly lots of bugfixes and general refinements.  Enjoy!

As always, you can get it from https://github.com/matrix-org/synapse/releases/tag/v0.26.0.

Changes in synapse v0.26.0 (2018-01-05)

No changes since v0.26.0-rc1

Changes in synapse v0.26.0-rc1 (2017-12-13)

Features:

• Add ability for ASes to publicise groups for their users (PR #2686)
• Add all local users to the user_directory and optionally search them (PR #2723)
• Add support for custom login types for validating users (PR #2729)

Changes:

• Update example Prometheus config to new format (PR #2648) Thanks to @krombel!
• Rename redact_content option to include_content in Push API (PR #2650)
• Declare support for r0.3.0 (PR #2677)
• Improve upserts (PR #2684, #2688, #2689, #2713)
• Improve documentation of workers (PR #2700)
• Improve tracebacks on exceptions (PR #2705)
• Allow guest access to group APIs for reading (PR #2715)
• Support for posting content in federation_client script (PR #2716)
• Delete devices and pushers on logouts etc (PR #2722)

Bug fixes:

• Fix database port script (PR #2673)
• Fix internal server error on login with ldap_auth_provider (PR #2678) Thanks to @jkolo!
• Fix error on sqlite 3.7 (PR #2697)
• Fix OPTIONS on preview_url (PR #2707)
• Fix error handling on dns lookup (PR #2711)
• Fix wrong avatars when inviting multiple users when creating room (PR #2717)
• Fix 500 when joining matrix-dev (PR #2719)

Hi all,

Today is a crazy release day here - not only do we have Synapse 0.25, but we've also made a formal release of the Matrix Specification (CS API) for the first time in 16 months!

Matrix CS API 0.3

Talking first about the spec update: the workflow of the Matrix spec is that new experimental features get added to an /unstable API prefix, and then whenever we release the Matrix spec, these get moved over to being part of the /r0 prefix (or whatever version we happen to be on).  We've been very constrained on manpower to work on the spec over the last ~18 months, but we've been keeping it up-to-date on a best effort basis, with a bit of help from the wider community.   As such, this latest release does not contain all the latest APIs (and certainly not experimental ones like Groups/Communities which are still evolving), but it does release all of the unstable ones which we've managed to document and which are considered stable enough to become part of the 'r0' prefix.  Going forwards, we're hoping that the wider community will help us fill in the remaining gaps (i.e. propose PRs against the matrix-org/matrix-doc repository to formalise the various spec drafts flying around the place) - and we're also hoping (if/when funding crisis is abated) to locate full-time folk to work on the spec.

The full changelog for 0.3 of the spec is:

• Breaking changes:
  • Change the rule kind of .m.rule.contains_display_name from underride to override. This works with all known clients which support push rules, but any other clients implementing the push rules API should be aware of this change. This makes it simple to mute rooms correctly in the API (#373).
  • Remove /tokenrefresh from the API (#395).
  • Remove requirement that tokens used in token-based login be macaroons (#395).
• Changes to the API which will be backwards-compatible for clients:
  • Add filename parameter to POST /_matrix/media/r0/upload (#364).
  • Document CAS-based client login and the use of m.login.token in /login (#367).
  • Make origin_server_ts a mandatory field of room events (#379).
  • Add top-level account_data key to the responses to GET /sync and GET /initialSync (#380).
  • Add is_direct flag to POST /createRoom and invite member event. Add 'Direct Messaging' module (#389).
  • Add contains_url option to RoomEventFilter (#390).
  • Add filter optional query param to /messages (#390).
  • Add 'Send-to-Device messaging' module (#386).
  • Add 'Device management' module (#402).
  • Require that User-Interactive auth fallback pages call window.postMessage to notify apps of completion (#398).
  • Add pagination and filter support to /publicRooms. Change response to omit fields rather than return null. Add estimate of total number of rooms in list. (#388).
  • Allow guest accounts to use a number of endpoints which are required for end-to-end encryption. (#751).
  • Add key distribution APIs, for use with end-to-end encryption. (#894).
  • Add m.room.pinned_events state event for rooms. (#1007).
  • Add mention of ability to send Access Token via an Authorization Header.
  • New endpoints:
    • GET /joined_rooms (#999).
    • GET /rooms/{'{'}roomId{'}'}/joined_members  (#999).
    • GET /account/whoami (#1063).
    • GET /media/{'{'}version{'}'}/preview_url  (#1064).
• Spec clarifications:
  • Add endpoints and logic for invites and third-party invites to the federation spec and update the JSON of the request sent by the identity server upon 3PID binding (#997)
  • Fix "membership" property on third-party invite upgrade example (#995)
  • Fix response format and 404 example for room alias lookup (#960)
  • Fix examples of m.room.member event and room state change, and added a clarification on the membership event sent upon profile update (#950).
  • Spell out the way that state is handled by POST /createRoom (#362).
  • Clarify the fields which are applicable to different types of push rule (#365).
  • A number of clarifications to authentication (#371).
  • Correct references to user_id which should have been sender (#376).
  • Correct inconsistent specification of redacted_because fields and their values (#378).
  • Mark required fields in response objects as such (#394).
  • Make m.notice description a bit harder in its phrasing to try to dissuade the same issues that occurred with IRC (#750).
  • GET /user/{'{'}userId{'}'}/filter/{'{'}filterId{'}'}  requires authentication (#1003).
  • Add some clarifying notes on the behaviour of rooms with no m.room.power_levels event (#1026).
  • Clarify the relationship between username and user_id in the /register API (#1032).
  • Clarify rate limiting and security for content repository. (#1064).

...and you can read the spec itself of course over at https://matrix.org/docs/spec.  It's worth noting that we have slightly bent the rules by including three very minor 'breaking changes' in 0.3, but all for features which to our knowledge nobody is depending on in the wild.  Technically this should mean bumping the major version prefix (i.e. moving to r1), but given how minor and nonimpacting these are we're turning a blind eye this time.

Meanwhile, Synapse 0.25 is out!

This is a medium-sized release; the main thing being to support configurable room visibility within groups (so that whenever you add a room to a group, you're not forced into sharing their existence with the general public, but can choose to just tell group members about them).  There's also a bunch of useful bug fixes and some performance improvements, including lots of contributions from the community this release (thank you!).  Full release notes are:

Changes in synapse v0.25.0 (2017-11-15)

Bug fixes:

• Fix port script (PR #2673)

Changes in synapse v0.25.0-rc1 (2017-11-14)

Features:

• Add is_public to groups table to allow for private groups (PR #2582)
• Add a route for determining who you are (PR #2668) Thanks to @turt2live!
• Add more features to the password providers (PR #2608, #2610, #2620, #2622, #2623, #2624, #2626, #2628, #2629)
• Add a hook for custom rest endpoints (PR #2627)
• Add API to update group room visibility (PR #2651)

Changes:

• Ignore tags when generating URL preview descriptions (PR #2576) Thanks to @maximevaillancourt!
• Register some /unstable endpoints in /r0 as well (PR #2579) Thanks to @krombel!
• Support /keys/upload on /r0 as well as /unstable (PR #2585)
• Front-end proxy: pass through auth header (PR #2586)
• Allow ASes to deactivate their own users (PR #2589)
• Remove refresh tokens (PR #2613)
• Automatically set default displayname on register (PR #2617)
• Log login requests (PR #2618)
• Always return is_public in the /groups/:group_id/rooms API (PR #2630)
• Avoid no-op media deletes (PR #2637) Thanks to @spantaleev!
• Fix various embarrassing typos around user_directory and add some doc. (PR #2643)
• Return whether a user is an admin within a group (PR #2647)
• Namespace visibility options for groups (PR #2657)
• Downcase UserIDs on registration (PR #2662)
• Cache failures when fetching URL previews (PR #2669)

Bug fixes:

• Fix port script (PR #2577)
• Fix error when running synapse with no logfile (PR #2581)
• Fix UI auth when deleting devices (PR #2591)
• Fix typo when checking if user is invited to group (PR #2599)
• Fix the port script to drop NUL values in all tables (PR #2611)
• Fix appservices being backlogged and not receiving new events due to a bug in notify_interested_services (PR #2631) Thanks to @xyzz!
• Fix updating rooms avatar/display name when modified by admin (PR #2636) Thanks to @farialima!
• Fix bug in state group storage (PR #2649)
• Fix 500 on invalid utf-8 in request (PR #2663)

Finally...

If you haven't noticed already, Riot/Web 0.13 is out today, as is Riot/iOS 0.6.2 and Riot/Android 0.7.4.  These contain massive improvements across the board - particularly mainstream Communities support at last on Riot/Web; CallKit/PushKit on Riot/iOS thanks to Denis Morozov (GSoC 2017 student for Matrix) and Share Extension on iOS thanks to Aram Sargsyan (also GSoC 2017 student!); and End-to-end Key Sharing on Riot/Android and a full rewrite of the VoIP calling subsystem on Android.

Rather than going on about it here, though, there's a full write-up over on the Riot Blog.

And so there you go - new releases for eeeeeeeeveryone!  Enjoy! :)

--Matthew, Amandine & the team.