Dan Callahan

Synapse 1.38.0 is out now!

NOTE: We released Synapse 1.38.1 on Thursday, July 22nd. It mitigates a client bug with Synapse 1.38.0's smaller sync responses which prevented new Element Android sessions from successfully participating in encrypted conversations. Server administrators are strongly encouraged to upgrade.

(Big) Integers

Synapse's database schema used integer columns in a few places where values could potentially overflow a maximum value of 2³¹. One such column is events.stream_ordering, which surpassed 2³¹ on matrix.org last week.

To prevent overflows, Synapse 1.38 will automatically convert several integer columns to bigint as a background update. While homeservers will function normally during this task, it could result in increased disk I/O for several hours or days. Note that homeservers may need several gigabytes of free space to successfully rebuild associated database indexes and complete the upgrade.

See the upgrade notes for more details.

Expiring Caches

Synapse has a new configuration option, caches.expiry_time, which can be set to enable evicting items from caches if they go too long without being accessed. This helps servers reclaim memory used by large yet infrequently used caches.

Smaller Sync Responses

The response to /sync now omits optional keys when they would otherwise be empty. This can significantly reduce the size of incremental syncs, as demonstrated in #6579. Thanks to deepbluev7 for initially submitting this in #9919, which made it into this release via #10214.

Everything Else

A few other items worth calling out:

• This release includes an experimental implementation of MSC2918: Refresh tokens, which adds initial support for complementary access / refresh tokens in line with OAuth best practices (#9450).
• Synapse now ships a script to review recently registered accounts, which can be useful in cleaning up servers in the wake of malicious, automated registrations like we witnessed during last month's spam attack.
• We've also fixed a few rough edges (#10263, #10303, #10336) in the spam mitigations from 1.37.1, and would encourage you to update.
• The Admin API for querying user information now includes information about a user's SSO identities in its response.

These are just the highlights; please see the Upgrade Notes and Release Notes for a complete list of changes in this release.

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including deepbluev7, dklimpel, fkr, and sideshowbarker

Synapse 1.36.0 is out, and it's a big one!

Room Join Memory Improvements

We did it! Synapse no longer experiences a memory spike when joining large / complex rooms.

These improvements mainly arise from processing join responses incrementally, rather than trying to load everything into memory at once. However, realizing these gains involved a fair bit of rewriting, as the entire processing pipeline had to work incrementally, and with appropriately sized batches, to avoid downstream bottlenecks. You can hear more about our original plans for this work in last month's Matrix Live: S6E23 — Dan and Erik talk about Synapse.

Presence Improvements

Running presence on a single worker process is now expected to work correctly. This feature first debuted in Synapse 1.33, but a few bugs cropped up which could lead to presence state becoming outdated. With #10149 merged, we believe the last of these issues to be resolved.

We had also noticed a recent increase in presence load on federation workers; this was ultimately tracked to two bugs, both fixed in this release: We were processing local presence via federation workers (#10163) and we were occasionally sending duplicate presence updates (#10165).

With both issues fixed, outgoing federation load has returned to normal levels:

(Thank you to David Mehren for this graph from issue #10153)

Everything Else

Synapse now has two new Admin APIs for unprotecting and removing media from quarantine, thanks to contributions by dklimpel.

Synapse now implements the stable /_matrix/client/r0/rooms/{roomId}/aliases endpoint originally introduced by MSC2432, and, thanks to contributions by govynnus, makes the reason and score fields of event reports optional per MSC2414.

These are just the highlights; please see the Release Notes for a complete list of changes in this release.

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including 14mRh4X0r, aaronraimist, bradtgmurray, crcastle, dklimpel, govynnus, and RhnSharma.

Synapse 1.35.0 is out! This release focused on improving internals as we drive toward better memory performance during room joins, but more on that below.

Update: Synapse 1.35.1 was published on Thursday, June 3rd. It resolves a bug (#10109) which mistakenly listed invite-only rooms in the Spaces summary.

We'd also like to call the attention of client developers to a deprecation: The unstable prefixes used during development of MSC2858: Multiple SSO Identity Providers will be removed from Synapse 1.38, due out in August. Please ensure your client supports the stable identifiers for this feature.

Spaces: On by Default

Following the successful release of Synapse 1.34, the experimental Spaces flag is now enabled by default. If you had manually enabled the experimental_features: { spaces_enabled: true } flag in your homeserver configuration, you may now remove it.

Bug Squashing

This release of Synapse fixes an issue which could cause federated room joins to fail when the join response exceeded a size limit which was too low (#10082). We've also improved what Synapse logs when it drops a connection in similar circumstances (#10091), which should aid diagnosis if a similar issue were to arise in the future.

GitHub user thermaq contributed a fix (#10014) for a bug which could cause user presence state to become stale.

Lastly our OpenTracing support now allows for profiling end-to-end performance on a per-user basis (#9978).

An Update on Room Joins

We've been hammering away at shrinking Synapse's memory footprint when joining large / complex rooms, and while we're not there yet, the end is in sight! In particular, this release includes many internal refactorings, including using ijson to parse the JSON response to /send_join (#9958), clearing the way for substantial improvements.

Memory usage still spikes because we're effectively doing the same work with a different library, but ijson's design allows for iterative parsing. This will pay dividends once we modify the code downstream of /send_join to take advantage of it.

Concretely, Erik Johnston has an experimental branch of Synapse which completely eliminates the memory spike:

The remaining work is centered on splitting that branch into self-contained, reviewable pull requests, like a rewrite of the Synapse Keyring class (#10035). After that's merged, we'll need to make one further change to properly batch up work, at which point we should attain the efficiency gains from Erik's experiment.

Everything Else

GitHub user savyajha contributed a security hardened systemd unit file which effectively sandboxes Synapse (#9803). While not enabled by default, we'd encourage security conscious users to review the example file and associated documentation.

Please see the Release Notes for a complete list of changes in this release.

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including dklimpel, jerinjtitus, junquera, lonyeon, savyajha, and thermaq.

Synapse 1.34.0 is now available, and it's loaded with new features and performance improvements.

Note: This release deprecates and replaces the room_invite_state_types configuration option. If you've customized that for your homeserver, please review the Upgrade Notes.

We've also marked the v1 room deletion Admin API as deprecated. Instead of sending a POST to a path ending in /delete, administrators are encourage to instead send an HTTP DELETE to /_synapse/admin/v1/rooms/<room_id>. Thanks to ThibF for implementing this (#9889).

Spaces

The highlight of this release is support for Spaces, now that MSC1772: Matrix Spaces has merged into the Matrix spec!

Synapse also has support for MSC2946: Spaces Summary and MSC3083: Restricting room membership based on space membership, but these are off by default as they're still under development. To enable these experimental MSCs, set experimental_features: { spaces_enabled: true } in your homeserver configuration. These are enabled on the matrix.org homeserver, and we encourage you to experiment with Spaces there and let us know in the Spaces Feedback Room if you encounter any issues.

Memory and Caching

Memory consumption and caching have been a major focus of the Synapse team this quarter, and we've made significant strides:

• Synapse has a new gc_min_interval configuration option with reasonable defaults to prevent Python's garbage collector from running too frequently and thrashing when a large homeserver has its collection thresholds set too low.
• Synapse will report memory allocation stats to Prometheus when using jemalloc.
• Synapse will also measure Redis cache response times and report those to Prometheus.
• For debugging, Synapse can optionally track the memory use of each LruCache.

We have a few more tricks up our sleeves; to learn more about how we're planning to improve the memory cost of joining large rooms, check out last week's Matrix Live.

Other Fixes and Improvements

We've also landed significant improvements to:

• Sending events when Redis is available (#9905, #9950, #9951)
• Joining large rooms when presence is enabled (#9910, #9916)
• Backfilling history in large rooms (#9935)

...and fixed bugs that:

• Prevented cross-account m.room_key_request messages from being delivered (#9961, #9965)
• Incorrectly applied room creation / invitation rate limits to users and app services which should have been exempt (#9968)

The health check on our Docker images now responds more quickly upon successful startup thanks to improvements by maquis196 (#9913), and for especially privacy-conscious homeservers, device names can now be shielded over federation thanks to a contribution by aaronraimist (#9945).

New Access Token Format

Inspired by GitHub's new token format, we've restructured Synapse's access token format to follow the pattern:

syt_<UnpaddedBase64MXIDLocalPart>_<20RandomLetters>_<CRC32>

So a token for @example:matrix.org might look like:

syt_ZXhhbXBsZQ_KfJetOcLWEKCvYdKnQLV_0i3W80

Existing tokens remain valid; this is just for new tokens. We hope the new format reduces network overhead while also making it easier identify misplaced tokens in logs and repositories.

Everything Else

See the Upgrading Instructions and Release Notes for further information on this release.

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including aaronraimist, maquis196, and ThibF.

Synapse 1.33.2 is now available.

This release fixes a denial of service issue (CVE-2021-29471) where evaluating specially crafted push rules could lead to excessive CPU load. Server administrators are encouraged to upgrade.

To learn more about Synapse 1.33, see last week's release announcement.

Synapse 1.33.0 is out! Three main items of note:

1. We plan to release 1.33.1 1.33.2 with a low severity security fix on Tuesday next week, and we're interested in your thoughts on decoupling routine security fixes from normal releases. Please weigh in on this discussion.

   Note: We shipped 1.33.1 with a small dependency fix when installing Synapse via pip. A security release is still planned for Tuesday, which will now be 1.33.2.

2. If you use Synapse's optional account revalidation feature (see account_validity in config.yaml), you'll want to review the upgrading instructions as we've made a few small changes to the email templates it uses.

3. Synapse now has very experimental support for moving presence off of the main process. This has not yet been extensively validated, so please proceed with caution. We expect to get this to a point where we can confidently recommend it in the coming weeks.

Otherwise, this is another release focused on internals. We're driving toward a goal of reducing excess memory consumption when joining large or complex rooms, and most of our effort (aside from the presence work) has been focused on measurement, instrumentation, and experimentation for that.

We did manage to slightly speed up room joins, improve the performance of the user directory, and refine our implementation of MSC3083. Additionally, thanks to work by ShadowJonathan, Synapse now passes all of flake8-bugbear's lints.

See the Upgrading Instructions and Release Notes for further information.

Thank You

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including rkfg, and ShadowJonathan.

Synapse 1.32.2 is out! Synapse now requires Python 3.6 (or later) and we've made a few small changes which you should be aware of before upgrading. These are documented in the upgrade notes.

Note: We scrubbed the releases of Synapse 1.32.0 and 1.32.1 as we discovered a pair of regressions including a bug with Prometheus metrics after tagging the release. These have been resolved.

On Monday, humankind flew a helicopter on Mars. And while our pursuit of Space(s) is considerably more modest, it is nevertheless progressing apace: Synapse 1.32 includes an experimental implementation of MSC3083.

This release also includes a new Synapse module for routing of presence updates, which can allow devices to share presence information without requiring that they also share a room. Please note there are some nuances to worker configuration when using this module which we hope to iron out in a future release.

The Admin API is newly able to manage rate limits, and the user listing endpoint can finally sort its results by a variety of criteria.

Otherwise, this is again a very internals-focused release: many additional type hints, improvements to structured logging, and small cleanups, especially those possible now that we've left Python 3.5 behind. We've made changes to how we check whether accounts are exempt from rate limits to avoid cases where we mistakenly applied limits to Application Services which should have been exempt, and we've fixed a bug with sharded federation senders which could occasionally pin the CPU.

See the Upgrading Instructions and Release Notes for further information.

Thank You

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including dklimpel, languitar, ShadowJonathan, and xmunoz.

We've released Synapse 1.31.0!

Mainly internal changes this time (type hints, code lints, etc.) but we've also landed some initial work on MSC2946: Spaces Summary. And speaking of MSCs, Synapse 1.31 has an experimental flag which can enable support for the draft MSC3026: "busy" presence state.

Synapse 1.31 can now restrict OpenID Connect logins based on userinfo attributes (Thanks, HubbeKing!).

This release fixes a rare infinite loop when fetching cross-signing keys or handling device list updates, and further improves the speed of federation catchup. It also makes Admin APIs around user reactivation behave correctly when account passwords are disabled.

See the Release Notes for further information.

The Final Python 3.5 Release

This is the last release of Synapse to support Python 3.5 or PostgreSQL 9.5, and the last release of official packages for Debian 9 (Stretch) and Ubuntu 16.04 LTS (Xenial).

Future releases of Synapse will follow upstream end-of-life dates for Python and Postgres.

Accordingly, we anticipate ending support for Python 3.6 and PostgreSQL 9.6 in December of this year. We will also cease producing packages for Ubuntu 18.04 LTS (Bionic) at that time.

Thank You

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including ankitdobhal, blakehawkins, dklimpel, fuzzmz, HubbeKing, languitar, sandhose, and ShadowJonathan.

Synapse 1.30.1 is now available. This release is identical to Synapse 1.30.0, with the exception of explicitly setting a minimum version of the Python Cryptography library to ensure that users of Synapse are protected from yesterday's OpenSSL security advisories, especially CVE-2021-3449.

Note that Cryptography defaults to bundling its own statically linked copy of OpenSSL, which means that you may not be protected by your operating system's security updates.

It's also worth noting that Cryptography no longer supports Python 3.5, so admins deploying to older environments like Debian 9 (Stretch) or Ubuntu 16.04 (Xenial) may not be protected against this or future vulnerabilities.

The next release of Synapse will be the last to support Python 3.5.

We've released Synapse 1.30.0!

A key theme of this release was stability and resilience around federation. We've landed changes to be less eager about entering catch up mode and to retry on HTTP 500 errors, while also rejecting transactions which arrive before we've completed processing earlier transactions from that same server. We've significantly optimized how we handle missing events when receiving incoming federation traffic, and we've found a way to re-use the chain cover index (from Synapse 1.26) when responding to the state_ids endpoint.

That last one turns out to be a pretty big deal: we've seen an order of magnitude improvement in both the CPU and DB cost of the state_ids endpoint. For example, the average CPU usage by that endpoint on matrix.org dropped from few seconds to well under 100ms:

Enjoy. 🙂

This release also includes further improves to our SSO support, including allowing spam checkers to distinguish between new registrations and first-time SSO users and fixing account reactivation when local passwords are disabled. Now that MSC2858: Multiple SSO Identity Providers has passed its Final Comment Period, we've also updated Synapse to respond to the stable versions of endpoints introduced by that MSC.

See the Release Notes for further information.

Python / Platform Deprecations

As a reminder, the next release of Synapse (1.31, scheduled for April 5th) will be the last to support Python 3.5 or PostgreSQL 9.5, both of which have reached their upstream end of life.

We will also cease building packages for Ubuntu 16.04 (Xenial) and Debian 9 (Stretch) at the same time.

Application Service Registration Changes

Note that Application Services must provide a type parameter with the value "m.login.application_service" when calling POST /_matrix/client/r0/register. Synapse currently allows registration without an explicit type, but this divergence from the spec will be resolved in a future release.

Thanks

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including dklimpel, ShadowJonathan, and tlvb.