Security update: Synapse 0.32.0

Folks, Synapse 0.32.0 is an important security update: please upgrade as soon as you can.

The release focuses on security; fixing several federation bugs and adding new features for countering abuse. Notably it includes the ability to blacklist & whitelist servers allowed to send events to a room on a per-room basis via the new m.room.server_acl state event: see MSC1383 for details.  This also closes out https://github.com/matrix-org/matrix-doc/issues/709 – one of our oldest feature requests from users who wish to be able to limit the servers allowed to participate in a given room.

It’s important to understand that server ACLs only work if all the servers participating in the room honour them.  In future this will be handled better (as part of ongoing work in making it easier to incrementally version and upgrade the federation protocol).  This means that for the ACLs to work, any servers which don’t yet implement ACLs (e.g. older Synapses) have to be ACL’d from the room for the access control to work.  Therefore please upgrade as soon as possible to avoid this problem.

This ongoing flurry of security work is in general all part of moving towards the long-awaited stable release of the Server-Server API. In parallel we’ve been working on the other main outstanding point: State Resets (i.e. scenarios where you get unexpected results when resolving conflicts between different servers’ copies of a room).  There will be a few more major changes and upgrades on the horizon as we fix these, but then we’ll finally be able to cut an r0 release of the Server-Server API and Matrix will be one massive step closer to being out of beta!

As always, you can get the new update from https://github.com/matrix-org/synapse/releases/tag/v0.32.1 or any of the sources mentioned at https://github.com/matrix-org/synapse.

Changes in synapse v0.32.0 (2018-07-06)

No changes since 0.32.0rc1

Synapse 0.32.0rc1 (2018-07-05)

 

Features

  • Add blacklist & whitelist of servers allowed to send events to a
    room via m.room.server_acl event. (merge)
  • Cache factor override system for specific caches (#3334)
  • Add metrics to track appservice transactions (#3344)
  • Try to log more helpful info when a sig verification fails
    (#3372)
  • Synapse now uses the best performing JSON encoder/decoder according
    to your runtime (simplejson on CPython, stdlib json on PyPy).
    (#3462)
  • Add optional ip_range_whitelist param to AS registration files to
    lock AS IP access (#3465)
  • Reject invalid server names in federation requests (#3480)
  • Reject invalid server names in homeserver.yaml (#3483)

Bugfixes

  • Strip access_token from outgoing requests (#3327)
  • Redact AS tokens in logs (#3349)
  • Fix federation backfill from SQLite servers (#3355)
  • Fix event-purge-by-ts admin API (#3363)
  • Fix event filtering in get_missing_events handler (#3371)
  • Synapse is now stricter regarding accepting events which it cannot
    retrieve the prev_events for. (#3456)
  • Fix bug where synapse would explode when receiving unicode in HTTP
    User-Agent header (#3470)
  • Invalidate cache on correct thread to avoid race (#3473)

Improved Documentation

Deprecations and Removals

  • Remove was_forgotten_at (#3324)

Misc

Towards open governance for Matrix.org

Hi all,

Since we created Matrix.org back in 2014, the majority of the Matrix core team has worked for the same company – originally subsidiaries of Amdocs, and then since August 2017 for New Vector; the startup we incorporated to rehire the core team and support Matrix after we parted ways from Amdocs.

Despite working for a for-profit company, our prime directive has always been the long term mission to successfully run Matrix as a non-profit initiative for the benefit of the whole internet: to create a ubiquitous secure open network which gives users control back over their communication and avoids them ever being locked into silos again.  And even though Matrix.org hasn’t been a formal non-profit foundation, we’ve treated it as such in all respects (e.g. gathering donations to support development on Matrix)

Running Matrix.org as a non-profit means prioritising to neutrally support all players in the whole ecosystem without ever giving unfair advantage to any individual participant (particularly New Vector) – where that ecosystem includes end-users, client devs, testers, spec devs, server admins, companies building products on Matrix, bridge devs, bot devs, widget devs, server devs, distro maintainers, moderators, even end-users who are using Matrix indirectly via bridges.

That being said, having the core team work for the same startup is still a somewhat unorthodox model for an open source project building an open standard, so we’d like to explain the main reasons for doing it up to this point:

  • To ensure that Matrix is fit for real-world usage and to force us to dogfood it. To ensure that it is a protocol that works well enough that you can build a commercial startup around it if you so wanted, and to motivate us to build Matrix as something more than an academic or nerdy exercise in protocol design – rather one which can be commercially viable.
  • To help ensure the core team is aligned and pulling towards the same goal, especially during the process of actually designing and “giving birth” to the initial protocol and getting it to an ‘r0’ release across all APIs.  We strongly believe that when a project is in the design phase you get faster and better design from a bunch of people who are collaborating together towards the same goal, rather than independent factions who might pursue their own agendas at the expense of the overall project.
  • Because we believe the value of Matrix lies in the size of the ecosystem, and if Matrix realises its full potential (i.e. it grows as big as the web), it only makes it more useful and valuable for *everyone*. We realise that it might be a leap of faith to believe that we don’t have any incentive to sabotage Matrix by privileging specific players (after all, there are so many companies out there in it just for the cash), but the fact is that this is where we stand, and we’re doing our best to prove it. To spell it out: it is in New Vector’s interest (and also in the interests of other Matrix-focused companies) to grow Matrix to be as big, open, unfragmented and as neutral as possible.  Matrix should be big enough for a multitude of wildly successful companies and projects to benefit from it, and everyone wins – just like the web.

However, this approach is not perfect and comes with some major problems:

  • Without clear separation of responsibilities and incentives, we have to ask the community to take it on faith that our efforts are never intended to privilege New Vector ahead of the wider ecosystem. This leaves room for doubt, especially when our reasoning is unclear or our conclusions controversial.

    A good example of a controversial decision is the lack of investment by the core team in the Server-Server API.  For the last ~2 years (since Mar 2016) we made the judgement call to prioritise user-facing features and experience.  The rationale was that to grow Matrix we need to provide a viable alternative to Slack/Discord/WhatsApp etc, and doing that means providing a Client-Server API which lets clients do just that, and server implementations capable of running at scale. This is why the CS API has had a stable release since Dec 2015 (currently at r0.3.0) and why we’ve put so much effort into server scaling/perf… but the SS API itself still has bugs and has still not yet made it to a stable release.

    This is obviously incredibly frustrating to server devs who tried to implement the SS API despite it being unstable and unreleased. In retrospect it might have been a mistake and we could probably have turned off signup on matrix.org and diverted the resources to the SS API work instead.  However, this is a case of making the judgement call to prioritising the overall ecosystem over one class of stakeholders (server devs) by focusing on providing users usable and featureful decentralised communication apps. Indeed we strongly believe that users are the main means to grow the ecosystem (others have failed without them): no one joins a network with no friends, however popular it is among devs.  Nonetheless, we are finally in a position to hire spec maintainers and get to a stable S2S as fast as we possibly can, and frankly feel relieved to be able to unblock this situation.

    Another good example is the recent 0.31.2 security update of Synapse: this was a defensive patch to the protocol that we added to ensure that even if bugs occur when authing events over federation, it should be impossible for a random user to ever hijack a room again.  We specced this out as a formal proposal and are gathering feedback, but expedited implementation in Synapse to protect the overall ecosystem. However, it turns out that the change breaks a small number of highly custom rooms, and so we find ourselves accused of privileging NV.  The reality is that we made a judgement call to protect the vast majority of today’s ecosystem (and hope to provide a longer-term fix in the relatively near future which /will/ be compatible with more custom room use cases).
  • Another problem is some companies find it a turn-off to participate in Matrix unless they have a well-defined process for influencing the direction of the protocol.  Now, sometimes this could be seen as a feature rather than a bug; the last thing the ecosystem needs is a greedy corp trying to subvert the protocol to its own competitive advantage, and we don’t want to be locked in that kind of battle either.  However, there are also lots of well-meaning and constructive companies who want to participate too, and there’s an argument that they want a well-defined process for doing so.
  • The other main problem is simply one of checks & balances.  Even though NV may be a good guardian today, what if something changed in future? e.g. if NV got bought by Microsoft, or if someone on the team had some crisis and changed priorities?  Whilst one could always fork, such things are incredibly disruptive and fragmenting and it’d be good to engineer Matrix.org’s governance to be resilient to such eventualities as much as is possible.

To address these problems, in March of this year we started work on a long term proposal to establish an open governance model for Matrix which ensures the neutrality of the protocol, lets the community contribute as widely as possible, and incorporates a dedicated neutral non-profit Matrix.org Foundation separate to New Vector.

As work progressed on the proposal, it became clear that actually transitioning to a new governance model would seriously slow down the sprint towards a stable r0 release. We therefore decided to put completing the governance model on hold until after the r0 release (scheduled for the end of August).

With the end of r0 now in sight, completing work on the governance model is back on the agenda. We obviously want to ensure that the proposed governance model is going to work for everyone, so we’d like to introduce the first draft of Matrix Spec Change 1318: a proposal for open governance of the Matrix.org Spec.  This is quite an early draft; the idea is to gather feedback over the next few months and we’ll then incorporate the Foundation and deploy the new governance model to coincide with the long-awaited stable release of all APIs of the Matrix Spec (assuming the release doesn’t slip).

The main points in the proposal are:

  • To adopt the new governance model once all APIs have had a stable r0 release.  For S2S API, this means fixing the remaining flaws in the federation protocol and closing the spec omissions such that compliant independent implementations can be written purely based on the spec.  For the AS and IS and Push API it means just closing spec omissions (if any) and doing a final review.
  • To define the mission of Matrix: to return control of communication to users by building a standards-based open secure decentralised communication network.
  • To define the mandate of the core team to act as a neutral custodian of the Matrix Spec, prioritising the long-term success and growth of the overall network over individual commercial concerns.
  • To define the guiding principles of the core team, e.g. collaboration rather than competition and contrarianism.
  • To restructure the core team to incorporate members of the community as well as the founding core team.
  • To propose succession logistics for the core team
  • To propose the role and governance structure of the Matrix.org Foundation legal entity.

Feedback would be much appreciated on the MSC1318 Google Doc – or come talk about it on #matrix-spec-process:matrix.org (which we might as well use for governance too).

It’s exciting times as we finally move towards an initial stable release of Matrix across all APIs – we are firmly on the road to a 1.0, and improving our governance model is a massive part of that process.

thanks,

Matthew, Amandine & the core team.

Synapse 0.31.1 Released!

Folks,

v0.31.1 fixes a security bug in the “get_missing_events“ federation API where event visibility rules were not applied correctly.

We are not aware of it being actively exploited but please upgrade asap.

Sorry for the inconvenience, Synapse and the Matrix spec are still in beta and we still ironing out gaps such as this one.

You can get the release here.

Changes in synapse v0.31.1 (2018-06-08)

v0.31.1 fixes a security bug in the get_missing_events federation API
where event visibility rules were not applied correctly.

We are not aware of it being actively exploited but please upgrade asap.

Bug Fixes:

  • Fix event filtering in get_missing_events handler (PR #3371)

Synapse v0.31.0 released!

Good people, it’s release time.

With the core team focusing on upcoming performance work and GDPR management tooling, v0.31.0 is most notable for improvements to system stats. Additionally, work continues on our py3 port and a host of small bug fixes and perf improvements.

Get it now from https://github.com/matrix-org/synapse/releases/tag/v0.31.0

Changes in synapse v0.31.0 (2018-06-06)

Most notable change from v0.30.0 is to switch to python prometheus library to improve system stats reporting. WARNING this changes a number of prometheus metrics in a backwards-incompatible manner. For more details, seedocs/metrics-howto.rst

Bug Fixes:

  • Fix metric documentation tables (PR #3341)
  • Fix LaterGuage error handling (694968f)
  • Fix replication metrics (b7e7fd2)

Changes in synapse v0.31.0-rc1 (2018-06-04)

Features:

  • Switch to the Python Prometheus library (PR #3256#3274)
  • Let users leave the server notice room after joining (PR #3287)

Changes:

  • daily user type phone home stats (PR #3264)
  • Use iter* methods for _filter_events_for_server (PR #3267)
  • Docs on consent bits (PR #3268)
  • Remove users from user directory on deactivate (PR #3277)
  • Avoid sending consent notice to guest users (PR #3288)
  • disable CPUMetrics if no /proc/self/stat (PR #3299)
  • Add local and loopback IPv6 addresses to url_preview_ip_range_blacklist (PR #3312) Thanks to @thegcat!
  • Consistently use six’s iteritems and wrap lazy keys/values in list() if they’re not meant to be lazy (PR #3307)
  • Add private IPv6 addresses to example config for url preview blacklist (PR #3317) Thanks to @thegcat!
  • Reduce stuck read-receipts: ignore depth when updating (PR #3318)
  • Put python’s logs into Trial when running unit tests (PR #3319)

Changes, python 3 migration:

Bugs:

  • Fix federation backfill bugs (PR #3261)
  • federation: fix LaterGauge usage (PR #3328) Thanks to @intelfx!

Matrix.org homeserver privacy policy and terms of use being enforced today

Hi all,

As mentioned in our last blog post on GDPR, to make sure that everyone has read and understood the important details about how their personal data is processed by the matrix.org homeserver, users who haven’t yet agreed to the privacy notice and terms and conditions will be blocked from sending new messages until they have.

Users will continue to be able to receive messages, so they won’t miss out on any messages sent to them before they’ve agreed to the terms.

The System Alerts room has already sent every user their unique link to review and agree, and if anyone missed that message, the latest Riot.im web and mobile will display a helpful error message guiding users who are yet to agree through the agreement process.

If you have any questions or difficulties, please let us know at [email protected].

Thanks!

Tom

 

GDPR on matrix.org

If you’ve connected to the matrix.org homeserver today, you’ll have noticed some activity in support of GDPR compliance. The most obvious of these is an invite from System Alerts (aka @server:matrix.org):

We’ve rolled out the System Alerts feature to communicate important platform information to all of a homeserver’s users. Today, we’re using it to communicate the arrival of our new (and much-improved) Privacy Notice and Terms and Conditions to users on matrix.org.

The System Alerts service takes the form of an (unrejectable) invite to a room. We took this approach to support maximum compatibility with the myriad Matrix clients (since all Matrix clients can support conversations in a room 😊).

When we first rolled out System Alerts, we didn’t allow users leave the System Alerts room. Sorry! We got a bit overexcited – we’ve fixed that now (though please do provide your agreement before you leave).

What do I need to do?

At some point today the System Alerts service will provide you with unique link, directing you to review the new terms and provide your agreement.

For us to process your personal data lawfully, it’s really important that we know you understand and agree to our Privacy Notice and Terms and Conditions. For that reason, we will shortly be blocking any users who haven’t indicated their acceptance, so please act quickly when you receive your link.

Once the block is enabled, users who haven’t accepted the terms will see an error when they try and send a message, join a room, or send an invite. This message will also include the unique link to review and accept the terms, so users who haven’t seen the message from System Alerts will know what to do.

Don’t worry if you’re reading this some time after May 25 – accepting the terms at any time will unblock message sending on your account, and you won’t have missed any messages sent to you.

If you have any thoughts or suggestions on the legal documentation, you can provide comment via github.