Synapse 0.33.4 released!

Roll up, roll up, get it while it’s hot, Synapse 0.33.4 is here.

This release brings together a whole host of bug fixes, some enhancements to resource usage management and a bunch of internal changes in readiness for room member state lazy loading and our ongoing port to Python 3 (we are hoping to ship a py3 test candidate rsn!).

As ever, you can get the new update from https://github.com/matrix-org/synapse/releases/tag/v0.33.4 or any of the sources mentioned at https://github.com/matrix-org/synapse.

Features

  • Support profile API endpoints on workers (#3659)
  • Server notices for resource limit blocking (#3680)
  • Allow guests to use /rooms/:roomId/event/:eventId (#3724)
  • Add mau_trial_days config param, so that users only get counted as MAU after N days. (#3749)
  • Require twisted 17.1 or later (fixes #3741). (#3751)

Bugfixes

  • Fix error collecting prometheus metrics when run on dedicated thread due to threading concurrency issues (#3722)
  • Fix bug where we resent “limit exceeded” server notices repeatedly (#3747)
  • Fix bug where we broke sync when using limit_usage_by_mau but hadn’t configured server notices (#3753)
  • Fix ‘federation_domain_whitelist’ such that an empty list correctly blocks all outbound federation traffic (#3754)
  • Fix tagging of server notice rooms (#3755#3756)
  • Fix ‘admin_uri’ config variable and error parameter to be ‘admin_contact’ to match the spec. (#3758)
  • Don’t return non-LL-member state in incremental sync state blocks (#3760)
  • Fix bug in sending presence over federation (#3768)
  • Fix bug where preserved threepid user comes to sign up and server is mau blocked (#3777)

Internal Changes

  • Removed the link to the unmaintained matrix-synapse-auto-deploy project from the readme. (#3378)
  • Refactor state module to support multiple room versions (#3673)
  • The synapse.storage module has been ported to Python 3. (#3725)
  • Split the state_group_cache into member and non-member state events (and so speed up LL /sync) (#3726)
  • Log failure to authenticate remote servers as warnings (without stack traces) (#3727)
  • The CONTRIBUTING guidelines have been updated to mention our use of Markdown and that .misc files have content. (#3730)
  • Reference the need for an HTTP replication port when using the federation_reader worker (#3734)
  • Fix minor spelling error in federation client documentation. (#3735)
  • Remove redundant state resolution function (#3737)
  • The test suite now passes on PostgreSQL. (#3740)
  • Fix MAU cache invalidation due to missing yield (#3746)
  • Make sure that we close db connections opened during init (#3764)
  • Unignore synctl in .dockerignore to fix docker builds (#3802)

 

Critical Security Update: Synapse 0.33.3.1

Hi All,

As referenced in yesterday’s pre-disclosure, today we are releasing Synapse 0.33.3.1 as a critical security update.

We have patched two security vulnerabilities we identified whilst working on the upcoming r0 spec release for the Server-Server API (see details below). We do not believe either have been exploited in the wild, but strongly recommend everybody running a federated Synapse upgrades immediately.

As always you can get the new update here or from any of the sources mentioned at https://github.com/matrix-org/synapse/

Many thanks for your patience and understanding; with fixes like this we are moving ever closer to Synapse reaching a 1.0 Thanks also to the package maintainers who have coordinated with us to ensure distro packages are available for a speedy upgrade!

Note, for anyone running Debian Jessie, we have prepared a 0.33.2.1 deb (as 0.33.3 dropped support for Jessie).

 

Synapse 0.33.3.1 (2018-09-06)

SECURITY FIXES

  • Fix an issue where event signatures were not always correctly validated (#3796)
  • Fix an issue where server_acls could be circumvented for incoming events (#3796)

Internal Changes

  • Unignore synctl in .dockerignore to fix docker builds (#3802)

Pre-disclosure: Upcoming critical security fix for Synapse

Hi all,

During the ongoing work to finalise a stable release of Matrix’s Server-Server federation API, we’ve been doing a full audit of Synapse’s implementation and have identified a serious vulnerability which we are going to release a security update to address (Synapse 0.33.3.1) on Thursday Sept 6th 2018 at 12:00 UTC.

We are coordinating with package maintainers to ensure that patched versions of packages will be available at that time – meanwhile, if you run your own Synapse, please be prepared to upgrade as soon as the patched versions are released.  All previous versions of Synapse are affected, so everyone will want to upgrade.

Thank you for your time, patience and understanding while we resolve the issue,

signed_predisclosure.txt

Synapse 0.33.3 Released

All the threes, Synapse 0.33.3!

This release brings together a lot of bugfixes, and also some preparation for support for Lazy Loading and Room Versioning.

We also have, as a great contribution from @vojeroen, SNI extension support! With v0.33.3, Synapse now supports sending SNI over federation for vhosted servers, which resolves this long-standing request.

As always, you can get the new update from https://github.com/matrix-org/synapse/releases/tag/v0.33.3 or any of the sources mentioned at https://github.com/matrix-org/synapse.

 

Features

  • Add support for the SNI extension to federation TLS connections. Thanks to @vojeroen! (#3439)
  • Add /_media/r0/config (#3184)
  • speed up /members API and add at and membership params as per MSC1227 (#3568)
  • implement summary block in /sync response as per MSC688 (#3574)
  • Add lazy-loading support to /messages as per MSC1227 (#3589)
  • Add ability to limit number of monthly active users on the server (#3633)
  • Support more federation endpoints on workers (#3653)
  • Basic support for room versioning (#3654)
  • Ability to disable client/server Synapse via conf toggle (#3655)
  • Ability to whitelist specific threepids against monthly active user limiting (#3662)
  • Add some metrics for the appservice and federation event sending loops (#3664)
  • Where server is disabled, block ability for locked out users to read new messages (#3670)
  • set admin uri via config, to be used in error messages where the user should contact the administrator (#3687)
  • Synapse’s presence functionality can now be disabled with the “use_presence” configuration option. (#3694)
  • For resource limit blocked users, prevent writing into rooms (#3708)

Bugfixes

  • Fix occasional glitches in the synapse_event_persisted_position metric (#3658)
  • Fix bug on deleting 3pid when using identity servers that don’t support unbind API (#3661)
  • Make the tests pass on Twisted < 18.7.0 (#3676)
  • Don’t ship recaptcha_ajax.js, use it directly from Google (#3677)
  • Fixes test_reap_monthly_active_users so it passes under postgres (#3681)
  • Fix mau blocking calulation bug on login (#3689)
  • Fix missing yield in synapse.storage.monthly_active_users.initialise_reserved_users (#3692)
  • Improve HTTP request logging to include all requests (#3700, #3723)
  • Avoid timing out requests while we are streaming back the response (#3701)
  • Support more federation endpoints on workers (#3705, #3713)
  • Fix “Starting db txn ‘get_all_updated_receipts’ from sentinel context” warning (#3710)
  • Fix bug where state_cache cache factor ignored environment variables (#3719)

Deprecations and Removals

Internal Changes

  • The test suite now can run under PostgreSQL. (#3423)
  • Refactor HTTP replication endpoints to reduce code duplication (#3632)
  • Tests now correctly execute on Python 3. (#3647)
  • Sytests can now be run inside a Docker container. (#3660)
  • Port over enough to Python 3 to allow the sytests to start. (#3668, #3732)
  • Update docker base image from alpine 3.7 to 3.8. (#3669)
  • Rename synapse.util.async to synapse.util.async_helpers to mitigate async becoming a keyword on Python 3.7. (#3678)
  • Synapse’s tests are now formatted with the black autoformatter. (#3679)
  • Implemented a new testing base class to reduce test boilerplate. (#3684)
  • Rename MAU prometheus metrics (#3690)
  • add new error type ResourceLimit (#3707)
  • Logcontexts for replication command handlers (#3709)
  • Update admin register API documentation to reference a real user ID. (#3712)

Synapse 0.33.2 is here!

Folks, it’s release time, Synapse 0.33.2 has landed.

The release focuses on performance, notable highlights include reducing CPU consumption through speeding up state delta calculations (#3592) and reducing I/O through lazily loading state on the master process (#3579#3581#3582#3584)

Separately work continues on our python 3 port and we hope to have something concrete to trial very soon – we’re really excited about this and expect step change improvements in CPU and memory use.

Finally we have some ground work for upcoming room membership lazy loading, there is nothing to see here as yet, but rest assured we will make a lot of noise as soon as it is ready. Stay tuned.

As always, you can get the new update from https://github.com/matrix-org/synapse/releases/tag/v0.33.2 or any of the sources mentioned at https://github.com/matrix-org/synapse.

 

Synapse 0.33.2 (2018-08-09)

No significant changes.

Synapse 0.33.2rc1 (2018-08-07)

Features

  • add support for the lazy_loaded_members filter as per MSC1227 (#2970)
  • add support for the include_redundant_members filter param as per MSC1227 (#3331)
  • Add metrics to track resource usage by background processes (#3553#3556#3604#3610)
  • Add code label to synapse_http_server_response_time_seconds prometheus metric (#3554)
  • Add support for client_reader to handle more APIs (#3555#3597)
  • make the /context API filter & lazy-load aware as per MSC1227 (#3567)
  • Add ability to limit number of monthly active users on the server (#3630)
  • When we fail to join a room over federation, pass the error code back to the client. (#3639)
  • Add a new /admin/register API for non-interactively creating users. (#3415)

Bugfixes

  • Make /directory/list API return 404 for room not found instead of 400 (#2952)
  • Default inviter_display_name to mxid for email invites (#3391)
  • Don’t generate TURN credentials if no TURN config options are set (#3514)
  • Correctly announce deleted devices over federation (#3520)
  • Catch failures saving metrics captured by Measure, and instead log the faulty metrics information for further analysis. (#3548)
  • Unicode passwords are now normalised before hashing, preventing the instance where two different devices or browsers might send a different UTF-8 sequence for the password. (#3569)
  • Fix potential stack overflow and deadlock under heavy load (#3570)
  • Respond with M_NOT_FOUND when profiles are not found locally or over federation. Fixes #3585 (#3585)
  • Fix failure to persist events over federation under load (#3601)
  • Fix updating of cached remote profiles (#3605)
  • Fix ‘tuple index out of range’ error (#3607)
  • Only import secrets when available (fix for py < 3.6) (#3626)

Internal Changes

  • Remove redundant checks on who_forgot_in_room (#3350)
  • Remove unnecessary event re-signing hacks (#3367)
  • Rewrite cache list decorator (#3384)
  • Move v1-only REST APIs into their own module. (#3460)
  • Replace more instances of Python 2-only iteritems and itervalues uses. (#3562)
  • Refactor EventContext to accept state during init (#3577)
  • Improve Dockerfile and docker-compose instructions (#3543)
  • Release notes are now in the Markdown format. (#3552)
  • add config for pep8 (#3559)
  • Merge Linearizer and Limiter (#3571#3572)
  • Lazily load state on master process when using workers to reduce DB consumption (#3579#3581#3582#3584)
  • Fixes and optimisations for resolve_state_groups (#3586)
  • Improve logging for exceptions when handling PDUs (#3587)
  • Add some measure blocks to persist_events (#3590)
  • Fix some random logcontext leaks. (#3591#3606)
  • Speed up calculating state deltas in persist_event loop (#3592)
  • Attempt to reduce amount of state pulled out of DB during persist_events (#3595)
  • Fix a documentation typo in on_make_leave_request (#3609)
  • Make EventStore inherit from EventFederationStore (#3612)
  • Remove some redundant joins on event_edges.room_id (#3613)
  • Stop populating events.content (#3614)
  • Update the /send_leave path registration to use event_id rather than a transaction ID. (#3616)
  • Refactor FederationHandler to move DB writes into separate functions (#3621)
  • Remove unused field “pdu_failures” from transactions. (#3628)
  • rename replication_layer to federation_client (#3634)
  • Factor out exception handling in federation_client (#3638)
  • Refactor location of docker build script. (#3644)
  • Update CONTRIBUTING to mention newsfragments. (#3645)

 

Security update: Synapse 0.33.1

Hi All,

We have patched two securities vulnerabilities (details follow), we do not believe either have been exploited in the wild, but recommend upgrading asap.

As always you can get the new update from https://github.com/matrix-org/synapse/releases/tag/v0.33.1 or from any of the sources mentioned at https://github.com/matrix-org/synapse/

Thanks

 

Changes in Synapse v0.33.1 (2018-08-2)

  • Fix a potential issue where servers could request events for rooms they have not joined. (#3641)
  • Fix a potential issue where users could see events in private rooms before they joined. (#3642)

Synapse 0.33.0 is here!!

Hi all,

We’ve just released Synapse 0.33.0!  This is a major performance upgrade which speeds up /sync (i.e. receiving messages) by a factor of almost 2x!  This has already made a massive difference to the CPU usage and snappiness of the matrix.org homeserver since we rolled it out a few days ago – you can see the drop in sync worker CPU just before midday on July 17th; previously we were regularly hitting the CPU ceiling (at which point everything grinds to a halt) – now we’re back down hovering between 40% and 60% CPU (at the current load).  This is actually fixing a bug which crept in around Synapse 0.31, so please upgrade – especially if Synapse has been feeling slower than usual recently, and especially if you are still on Synapse 0.31.

Meanwhile we have a lot of new stuff coming on the horizon – a whole new algorithm for state resolution (watch this space for details); incremental state resolution (at last!) to massively speed up state resolution and mitigate extremities build up (and speed up the synapse master process, which is now the bottleneck again on the matrix.org homeserver); better admin tools for managing resource usage, and all the Python3 porting work (with associated speedups and RAM & GC improvements).  Fun times ahead!

The full changelog follows below; as always you can grab Synapse from https://github.com/matrix-org/synapse.   Thanks for flying Matrix!

Synapse 0.33.0 (2018-07-19)

Bugfixes

  • Disable a noisy warning about logcontexts. (#3561)

Synapse 0.33.0rc1 (2018-07-18)

Features

  • Enforce the specified API for report_event. (#3316)
  • Include CPU time from database threads in request/block metrics. (#3496#3501)
  • Add CPU metrics for _fetch_event_list. (#3497)
  • Optimisation to make handling incoming federation requests more efficient. (#3541)

Bugfixes

  • Fix a significant performance regression in /sync. (#3505#3521#3530#3544)
  • Use more portable syntax in our use of the attrs package, widening the supported versions. (#3498)
  • Fix queued federation requests being processed in the wrong order. (#3533)
  • Ensure that erasure requests are correctly honoured for publicly accessible rooms when accessed over federation. (#3546)

Misc

Security update: Synapse 0.32.0

Folks, Synapse 0.32.0 is an important security update: please upgrade as soon as you can.

The release focuses on security; fixing several federation bugs and adding new features for countering abuse. Notably it includes the ability to blacklist & whitelist servers allowed to send events to a room on a per-room basis via the new m.room.server_acl state event: see MSC1383 for details.  This also closes out https://github.com/matrix-org/matrix-doc/issues/709 – one of our oldest feature requests from users who wish to be able to limit the servers allowed to participate in a given room.

It’s important to understand that server ACLs only work if all the servers participating in the room honour them.  In future this will be handled better (as part of ongoing work in making it easier to incrementally version and upgrade the federation protocol).  This means that for the ACLs to work, any servers which don’t yet implement ACLs (e.g. older Synapses) have to be ACL’d from the room for the access control to work.  Therefore please upgrade as soon as possible to avoid this problem.

This ongoing flurry of security work is in general all part of moving towards the long-awaited stable release of the Server-Server API. In parallel we’ve been working on the other main outstanding point: State Resets (i.e. scenarios where you get unexpected results when resolving conflicts between different servers’ copies of a room).  There will be a few more major changes and upgrades on the horizon as we fix these, but then we’ll finally be able to cut an r0 release of the Server-Server API and Matrix will be one massive step closer to being out of beta!

As always, you can get the new update from https://github.com/matrix-org/synapse/releases/tag/v0.32.1 or any of the sources mentioned at https://github.com/matrix-org/synapse.

Changes in synapse v0.32.0 (2018-07-06)

No changes since 0.32.0rc1

Synapse 0.32.0rc1 (2018-07-05)

 

Features

  • Add blacklist & whitelist of servers allowed to send events to a
    room via m.room.server_acl event. (merge)
  • Cache factor override system for specific caches (#3334)
  • Add metrics to track appservice transactions (#3344)
  • Try to log more helpful info when a sig verification fails
    (#3372)
  • Synapse now uses the best performing JSON encoder/decoder according
    to your runtime (simplejson on CPython, stdlib json on PyPy).
    (#3462)
  • Add optional ip_range_whitelist param to AS registration files to
    lock AS IP access (#3465)
  • Reject invalid server names in federation requests (#3480)
  • Reject invalid server names in homeserver.yaml (#3483)

Bugfixes

  • Strip access_token from outgoing requests (#3327)
  • Redact AS tokens in logs (#3349)
  • Fix federation backfill from SQLite servers (#3355)
  • Fix event-purge-by-ts admin API (#3363)
  • Fix event filtering in get_missing_events handler (#3371)
  • Synapse is now stricter regarding accepting events which it cannot
    retrieve the prev_events for. (#3456)
  • Fix bug where synapse would explode when receiving unicode in HTTP
    User-Agent header (#3470)
  • Invalidate cache on correct thread to avoid race (#3473)

Improved Documentation

Deprecations and Removals

  • Remove was_forgotten_at (#3324)

Misc

Security update: Synapse 0.31.2

Hi all,

On Monday (2018-06-11) we had an incident where #matrix:matrix.org was hijacked by a malicious user pretending to join the room immediately after its creation in 2014 and then setting an m.room.power_levels event ‘before’ the correct initial power_level for the room.

Under normal circumstances this should be impossible because the initial m.room.power_levels for a room should be set before its m.room.join_rules event, meaning users who join the room are subject to its power levels. However, back before we’d even released Synapse, the first two rooms ever created in Matrix (#test:matrix.org and #matrix:matrix.org) were manually created and set the join_rules before the power_levels event, letting users join before the room’s power_levels were defined, and so were vulnerable to this attack. We’ve since re-created #matrix:matrix.org – please re-/join the room if you haven’t already!

As a defensive measure, we are releasing a security update of Synapse (0.31.2) today which changes the rules used to authenticate power_level events, such that we fail-safe rather than fail-deadly if the existing auth mechanisms fail. In practice this means changing the default power level required to set state to be 50 rather than 0 if there is no power_levels event present, thus meaning that only the room creator can set the initial power_levels event.

We are not aware of anyone abusing this (other than the old #matrix:matrix.org room) but we’d rather be safe than sorry, so would recommend that everyone upgrade as soon as possible.

This of course constitutes a change to the spec, so full technical details and ongoing discussion around the Matrix Spec Change proposal can be followed over at MSC1304.

EDIT: if you are aware of your server participating in rooms whose first power_levels event is deliberately set by a different user to their creator, please let us know asap (and don’t upgrade!)

This work is all part of a general push to finalise and harden and fully specify the Server-Server API as we push towards a long-awaited stable release of Matrix!

As always, you can get the new update from https://github.com/matrix-org/synapse/releases/tag/v0.31.2 or from any of the sources mentioned at https://github.com/matrix-org/synapse.

thanks, and apologies for the inconvenience.

Changes in synapse v0.31.2 (2018-06-14)

SECURITY UPDATE: Prevent unauthorised users from setting state events in a room when there is no m.room.power_levels event in force in the room. (PR #3397)

Discussion around the Matrix Spec change proposal for this change can be followed at https://github.com/matrix-org/matrix-doc/issues/1304.

Synapse 0.31.1 Released!

Folks,

v0.31.1 fixes a security bug in the “get_missing_events“ federation API where event visibility rules were not applied correctly.

We are not aware of it being actively exploited but please upgrade asap.

Sorry for the inconvenience, Synapse and the Matrix spec are still in beta and we still ironing out gaps such as this one.

You can get the release here.

Changes in synapse v0.31.1 (2018-06-08)

v0.31.1 fixes a security bug in the get_missing_events federation API
where event visibility rules were not applied correctly.

We are not aware of it being actively exploited but please upgrade asap.

Bug Fixes:

  • Fix event filtering in get_missing_events handler (PR #3371)