Introducing Matrix Specification Changes

Hi all,

We’ve been able to start investing more time in advancing the Matrix Specification itself over the last month or so thanks to Ben joining the core team (and should be able to accelerate even more with uhoreg joining in a few weeks!)  The first step in the new wave of work has been to provide much better infrastructure for the process of actually evolving the spec – whether that’s from changes proposed by the core team or the wider Matrix Community.

So, without further ado, we’d like to introduce https://matrix.org/docs/spec/proposals – a dashboard for all the spec change proposals we’ve accumulated so far (ignoring most of the ones which have already been merged), as well as a clearer workflow for how everyone can help improve the Matrix spec itself.  Part of this is introducing a formal numbering system – e.g. MSC1228 stands for Matrix Spec Change 1228 (where 1228 is the ID of the Github issue on the github.com/matrix-org/matrix-doc/issues repository that tracks the proposal).

Please note that these are *NOT* like XEPs or RFCs – i.e. optional proposals or add-ons to the protocol; instead they are literally proposals for changes to the Matrix Spec itself.  Once merged into the spec, they are only of historical interest.

We’ve also created a new room: #matrix-spec:matrix.org to discuss specific spec proposal changes – please join if you want to track how proposals are evolving! (Conversation is likely to fork off into per-proposal rooms or overflow into #matrix-dev:matrix.org or #matrix-architecture:matrix.org depending on traffic levels, however).

Feedback would be much appreciated on this – so please head over to #matrix-spec:matrix.org and let us know how it feels and how it could do better.

This is also a major step towards properly formalising Matrix.org’s governance model – hopefully the changes above are sufficient to improve the health of the evolution of the Spec as we work towards an initial stable release later this year, and then you should expect to see a spec proposal for formal governance once we’ve (at last!) exited beta :)

Huge thanks to Ben for putting this together, and thanks to everyone who’s contributed so far to the spec – we’re looking forward to working through the backlog of proposals and turning them all into merged spec PRs!!

Matthew

SECURITY UPDATE: Synapse 0.28.1

Hi all,

Many people will have noticed disruption in #matrix:matrix.org and #matrix-dev:matrix.org on Sunday, when a validation bug in Synapse was exploited which allowed a malicious event to be inserted into the room with ‘depth’ value that made the rooms temporarily unusable. Whilst a transient workaround was found at the time (thanks to /dev/ponies, kythyria and Po Shamil for the workaround and to Half-Shot for working on a proposed fix), we’re doing an urgent release of Synapse 0.28.1 to provide a temporary solution which will mitigate the attack across all rooms in upgraded servers and un-break affected ones.  Meanwhile we have a full long-term fix on the horizon (hopefully) later this week.

This vulnerability has already been exploited in the wild; please upgrade as soon as possible.

Synapse 0.28.1 is available from https://github.com/matrix-org/synapse/releases/tag/v0.28.1 as normal.

The ‘depth’ parameter is used primarily as a way for servers to signal the intended cosmetic order of their events within a room (particularly when the room’s message graph has gaps in it due to the server being offline, or due to users backfilling old disconnected chunks of conversation). This means that affected rooms may experience message ordering problems until a full long-term fix is provided, which we’re working on currently (and tentatively involves no longer trusting ‘depth’ information from servers).  For full details you can see the proposal documents for the temporary fix in 0.28.1 and the options for the imminent long-term fix.

We’d like to acknowledge jzk for identifying the vulnerability, and Max Dor for providing feedback on the fixes.

As a general reminder, Synapse is still beta (as is the Matrix spec) and the federation API particularly is still being debugged and refined and is pre-r0.0.0. For the benefit of the whole community, please disclose vulnerabilities and exploits responsibly by emailing [email protected] or DMing someone from +matrix:matrix.org. Thanks.

Changes in synapse v0.28.1 (2018-05-01)

SECURITY UPDATE

  • Clamp the allowed values of event depth received over federation to be [0, 2^63 – 1]. This mitigates an attack where malicious events injected with depth = 2^63 – 1 render rooms unusable. Depth is used to determine the cosmetic ordering of events within a room, and so the ordering of events in such a room will default to using stream_ordering rather than depth (topological_ordering). This is a temporary solution to mitigate abuse in the wild, whilst a long solution is being implemented to improve how the depth parameter is used.Full details at https://docs.google.com/document/d/1I3fi2S-XnpO45qrpCsowZv8P8dHcNZ4fsBsbOW7KABI
  • Pin Twisted to <18.4 until we stop using the private _OpenSSLECCurve API.

Security update: Riot/Web 0.13.5 released – fixing XSS vulnerability

Hi all,

Heads up that we made an emergency release of Riot/Web 0.13.5 a few hours ago to fix a XSS vulnerability found and reported by walle303 – many thanks for disclosing it responsibly.

Please upgrade to Riot/Web 0.13.5 asap. If you’re using riot.im/app or riot.im/develop this simply means hitting Refresh; otherwise please upgrade your Riot deployment as soon as possible. Alpine, Debian and Fedora/RPM packages are already updated – huge thanks to the maintainers for the fast turnaround.

The issue lies in the relatively obscure external_url feature, which lets bridges specify a URL for bridged events, letting Riot/Web users link through to the ‘original’ event (e.g. a twitter URL on a bridged tweet).  The option is hidden in a context menu and labelled “Source URL”, and is only visible on events which have the external_url field set.  Unfortunately Riot/Web didn’t sanitise the URL correctly, allowing a malicious URL to be injected – and this has been the case since the feature landed in Riot 0.9.0 (Nov 2016).

If you’re not able to upgrade to Riot/Web 0.13.5 for some reason, then please do not click on the ‘Source URL’ feature on the event context menu:

Apologies for the inconvenience,

thanks,

Matthew

Synapse 0.26 released!

Hi folks,

Synapse 0.26 is here (with no changes since RC1 which we released just before Christmas).  It’s a general maintenance release, albeit with a few new features but mainly lots of bugfixes and general refinements.  Enjoy!

As always, you can get it from https://github.com/matrix-org/synapse/releases/tag/v0.26.0.

Changes in synapse v0.26.0 (2018-01-05)

No changes since v0.26.0-rc1

Changes in synapse v0.26.0-rc1 (2017-12-13)

Features:

  • Add ability for ASes to publicise groups for their users (PR #2686)
  • Add all local users to the user_directory and optionally search them (PR
    #2723)
  • Add support for custom login types for validating users (PR #2729)

Changes:

  • Update example Prometheus config to new format (PR #2648) Thanks to
    @krombel!
  • Rename redact_content option to include_content in Push API (PR #2650)
  • Declare support for r0.3.0 (PR #2677)
  • Improve upserts (PR #2684#2688#2689#2713)
  • Improve documentation of workers (PR #2700)
  • Improve tracebacks on exceptions (PR #2705)
  • Allow guest access to group APIs for reading (PR #2715)
  • Support for posting content in federation_client script (PR #2716)
  • Delete devices and pushers on logouts etc (PR #2722)

Bug fixes:

  • Fix database port script (PR #2673)
  • Fix internal server error on login with ldap_auth_provider (PR #2678) Thanks
    to @jkolo!
  • Fix error on sqlite 3.7 (PR #2697)
  • Fix OPTIONS on preview_url (PR #2707)
  • Fix error handling on dns lookup (PR #2711)
  • Fix wrong avatars when inviting multiple users when creating room (PR #2717)
  • Fix 500 when joining matrix-dev (PR #2719)

Synapse 0.25 is out… as is Matrix Specification 0.3(!!!)

Hi all,

Today is a crazy release day here – not only do we have Synapse 0.25, but we’ve also made a formal release of the Matrix Specification (CS API) for the first time in 16 months!

Matrix CS API 0.3

Talking first about the spec update: the workflow of the Matrix spec is that new experimental features get added to an /unstable API prefix, and then whenever we release the Matrix spec, these get moved over to being part of the /r0 prefix (or whatever version we happen to be on).  We’ve been very constrained on manpower to work on the spec over the last ~18 months, but we’ve been keeping it up-to-date on a best effort basis, with a bit of help from the wider community.   As such, this latest release does not contain all the latest APIs (and certainly not experimental ones like Groups/Communities which are still evolving), but it does release all of the unstable ones which we’ve managed to document and which are considered stable enough to become part of the ‘r0’ prefix.  Going forwards, we’re hoping that the wider community will help us fill in the remaining gaps (i.e. propose PRs against the matrix-org/matrix-doc repository to formalise the various spec drafts flying around the place) – and we’re also hoping (if/when funding crisis is abated) to locate full-time folk to work on the spec.

The full changelog for 0.3 of the spec is:

  • Breaking changes:
    • Change the rule kind of .m.rule.contains_display_name from underride to override. This works with all known clients which support push rules, but any other clients implementing the push rules API should be aware of this change. This makes it simple to mute rooms correctly in the API (#373).
    • Remove /tokenrefresh from the API (#395).
    • Remove requirement that tokens used in token-based login be macaroons (#395).
  • Changes to the API which will be backwards-compatible for clients:
    • Add filename parameter to POST /_matrix/media/r0/upload (#364).
    • Document CAS-based client login and the use of m.login.token in /login (#367).
    • Make origin_server_ts a mandatory field of room events (#379).
    • Add top-level account_data key to the responses to GET /sync and GET /initialSync (#380).
    • Add is_direct flag to POST /createRoom and invite member event. Add ‘Direct Messaging’ module (#389).
    • Add contains_url option to RoomEventFilter (#390).
    • Add filter optional query param to /messages (#390).
    • Add ‘Send-to-Device messaging’ module (#386).
    • Add ‘Device management’ module (#402).
    • Require that User-Interactive auth fallback pages call window.postMessage to notify apps of completion (#398).
    • Add pagination and filter support to /publicRooms. Change response to omit fields rather than return null. Add estimate of total number of rooms in list. (#388).
    • Allow guest accounts to use a number of endpoints which are required for end-to-end encryption. (#751).
    • Add key distribution APIs, for use with end-to-end encryption. (#894).
    • Add m.room.pinned_events state event for rooms. (#1007).
    • Add mention of ability to send Access Token via an Authorization Header.
    • New endpoints:
      • GET /joined_rooms (#999).
      • GET /rooms/{roomId}/joined_members (#999).
      • GET /account/whoami (#1063).
      • GET /media/{version}/preview_url (#1064).
  • Spec clarifications:
    • Add endpoints and logic for invites and third-party invites to the federation spec and update the JSON of the request sent by the identity server upon 3PID binding (#997)
    • Fix “membership” property on third-party invite upgrade example (#995)
    • Fix response format and 404 example for room alias lookup (#960)
    • Fix examples of m.room.member event and room state change, and added a clarification on the membership event sent upon profile update (#950).
    • Spell out the way that state is handled by POST /createRoom (#362).
    • Clarify the fields which are applicable to different types of push rule (#365).
    • A number of clarifications to authentication (#371).
    • Correct references to user_id which should have been sender (#376).
    • Correct inconsistent specification of redacted_because fields and their values (#378).
    • Mark required fields in response objects as such (#394).
    • Make m.notice description a bit harder in its phrasing to try to dissuade the same issues that occurred with IRC (#750).
    • GET /user/{userId}/filter/{filterId} requires authentication (#1003).
    • Add some clarifying notes on the behaviour of rooms with no m.room.power_levels event (#1026).
    • Clarify the relationship between username and user_id in the /register API (#1032).
    • Clarify rate limiting and security for content repository. (#1064).

…and you can read the spec itself of course over at https://matrix.org/docs/spec.  It’s worth noting that we have slightly bent the rules by including three very minor ‘breaking changes’ in 0.3, but all for features which to our knowledge nobody is depending on in the wild.  Technically this should mean bumping the major version prefix (i.e. moving to r1), but given how minor and nonimpacting these are we’re turning a blind eye this time.

Meanwhile, Synapse 0.25 is out!

This is a medium-sized release; the main thing being to support configurable room visibility within groups (so that whenever you add a room to a group, you’re not forced into sharing their existence with the general public, but can choose to just tell group members about them).  There’s also a bunch of useful bug fixes and some performance improvements, including lots of contributions from the community this release (thank you!).  Full release notes are:

Changes in synapse v0.25.0 (2017-11-15)

Bug fixes:

  • Fix port script (PR #2673)
Changes in synapse v0.25.0-rc1 (2017-11-14)

Features:

Changes:

  • Ignore tags when generating URL preview descriptions (PR #2576)
    Thanks to @maximevaillancourt!
  • Register some /unstable endpoints in /r0 as well (PR #2579) Thanks to
    @krombel!
  • Support /keys/upload on /r0 as well as /unstable (PR #2585)
  • Front-end proxy: pass through auth header (PR #2586)
  • Allow ASes to deactivate their own users (PR #2589)
  • Remove refresh tokens (PR #2613)
  • Automatically set default displayname on register (PR #2617)
  • Log login requests (PR #2618)
  • Always return is_public in the /groups/:group_id/rooms API (PR #2630)
  • Avoid no-op media deletes (PR #2637) Thanks to @spantaleev!
  • Fix various embarrassing typos around user_directory and add some doc. (PR
    #2643)
  • Return whether a user is an admin within a group (PR #2647)
  • Namespace visibility options for groups (PR #2657)
  • Downcase UserIDs on registration (PR #2662)
  • Cache failures when fetching URL previews (PR #2669)

Bug fixes:

  • Fix port script (PR #2577)
  • Fix error when running synapse with no logfile (PR #2581)
  • Fix UI auth when deleting devices (PR #2591)
  • Fix typo when checking if user is invited to group (PR #2599)
  • Fix the port script to drop NUL values in all tables (PR #2611)
  • Fix appservices being backlogged and not receiving new events due to a bug in
    notify_interested_services (PR #2631) Thanks to @xyzz!
  • Fix updating rooms avatar/display name when modified by admin (PR #2636)
    Thanks to @farialima!
  • Fix bug in state group storage (PR #2649)
  • Fix 500 on invalid utf-8 in request (PR #2663)

Finally…

If you haven’t noticed already, Riot/Web 0.13 is out today, as is Riot/iOS 0.6.2 and Riot/Android 0.7.4.  These contain massive improvements across the board – particularly mainstream Communities support at last on Riot/Web; CallKit/PushKit on Riot/iOS thanks to Denis Morozov (GSoC 2017 student for Matrix) and Share Extension on iOS thanks to Aram Sargsyan (also GSoC 2017 student!); and End-to-end Key Sharing on Riot/Android and a full rewrite of the VoIP calling subsystem on Android.

Rather than going on about it here, though, there’s a full write-up over on the Riot Blog.

 

And so there you go – new releases for eeeeeeeeveryone!  Enjoy! :)

–Matthew, Amandine & the team.

Synapse 0.22.0 released!

Hi Synapsefans,

Synapse 0.22.0 has just been released! This release lands a few interesting features:

  • The new User directory API which supports Matrix clients’ providing a much more intuitive and effective user search capability by exposing a list of:
    • Everybody your user shares a room with, and
    • Everybody in a public room your homeserver knows about
  • New support for server admins, including a Shutdown Room API (to remove a room from a local server) and a Media Quarrantine API (to render a media item inaccessible without its actually being deleted)

As always there are lots of bug fixes and performance improvements, including increasing the default cache factor size from 0.1 to 0.5 (should improve performance for those running their own homeservers).

You can get Synapse 0.22.0 from https://github.com/matrix-org/synapse or https://github.com/matrix-org/synapse/releases/tag/v0.22.0 as normal.

Changes in synapse v0.22.0 (2017-07-06)

No changes since v0.22.0-rc2

Changes in synapse v0.22.0-rc2 (2017-07-04)

Changes:

  • Improve performance of storing user IPs (PR #2307, #2308)
  • Slightly improve performance of verifying access tokens (PR #2320)
  • Slightly improve performance of event persistence (PR #2321)
  • Increase default cache factor size from 0.1 to 0.5 (PR #2330)

Bug fixes:

  • Fix bug with storing registration sessions that caused frequent CPU churn
    (PR #2319)

Changes in synapse v0.22.0-rc1 (2017-06-26)

Features:

  • Add a user directory API (PR #2252, and many more)
  • Add shutdown room API to remove room from local server (PR #2291)
  • Add API to quarantine media (PR #2292)
  • Add new config option to not send event contents to push servers (PR #2301)
    Thanks to @cjdelisle!

Changes:

Bug fixes:

  • Fix users not getting notifications when AS listened to that user_id (PR
    #2216) Thanks to @slipeer!
  • Fix users without push set up not getting notifications after joining rooms
    (PR #2236)
  • Fix preview url API to trim long descriptions (PR #2243)
  • Fix bug where we used cached but unpersisted state group as prev group,
    resulting in broken state of restart (PR #2263)
  • Fix removing of pushers when using workers (PR #2267)
  • Fix CORS headers to allow Authorization header (PR #2285) Thanks to @krombel!