Synapse 0.99.0

Hey hey, Synapse 0.99.0 is here!

You may have heard that we recently published the first stable release of the Server to Server Spec (r0.1). The spec makes some changes which are not compatible with the protocol of the past – particularly, self-signed certificates are no longer valid for homeservers. Synapse 1.0.0 will be compliant with r0.1 and the goal of Synapse 0.99.0 is to act as a stepping stone to Synapse 1.0. Synapse 0.99.0 supports the r0.1 release of the server to server specification, but is compatible with both the legacy Matrix federation behaviour (pre-r0.1) as well as post-r0.1 behaviour, in order to allow for a smooth upgrade across the federation.

It is critical that all admins upgrade to 0.99.0 and configure a valid TLS certificate. Admins will have 1 month to do so, after which 1.0.0 will be released and those servers without a valid certificate will no longer be able to federate with >= 1.0.0 servers.

First of all, please don’t panic :) We have taken steps to make this process as simple as possible – specifically implementing ACME support to allow servers to automatically generate free Let’s Encrypt certificates if you choose to. What’s more, it is not necessary to add the certificate right away, you have at least a month to get set up.

For more details on exactly what you need to do (and also why this change is essential), we have provided an extensive FAQ as well as the Upgrade notes for Synapse

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.

This was a huge effort! Congratulations to all involved, especially those of you in the community who contributed to spec MSCs and tested our release candidates. Thank you for bearing with us as we move the whole public Matrix Federation onto r0.1 compliant servers.

Onwards!

Changelog

Synapse v0.99.x is a precursor to the upcoming Synapse v1.0 release. It contains foundational changes to room architecture and the federation security model necessary to support the upcoming r0 release of the Server to Server API.

Features

  • Synapse’s cipher string has been updated to require ECDH key exchange. Configuring and generating dh_params is no longer required, and they will be ignored. (#4229)
  • Synapse can now automatically provision TLS certificates via ACME (the protocol used by CAs like Let’s Encrypt). (#4384#4492#4525#4572#4564#4566#4547#4557)
  • Implement MSC1708 (.well-known routing for server-server federation) (#4408#4409#4426#4427#4428#4464#4468#4487#4488#4489#4497#4511#4516#4520#4521#4539#4542#4544)
  • Search now includes results from predecessor rooms after a room upgrade. (#4415)
  • Config option to disable requesting MSISDN on registration. (#4423)
  • Add a metric for tracking event stream position of the user directory. (#4445)
  • Support exposing server capabilities in CS API (MSC1753, MSC1804) (#447281b7e7eed))
  • Add support for room version 3 (#4483#4499#4515#4523#4535)
  • Synapse will now reload TLS certificates from disk upon SIGHUP. (#4495#4524)
  • The matrixdotorg/synapse Docker images now use Python 3 by default. (#4558)

Bugfixes

  • Prevent users with access tokens predating the introduction of device IDs from creating spurious entries in the user_ips table. (#4369)
  • Fix typo in ALL_USER_TYPES definition to ensure type is a tuple (#4392)
  • Fix high CPU usage due to remote devicelist updates (#4397)
  • Fix potential bug where creating or joining a room could fail (#4404)
  • Fix bug when rejecting remote invites (#4405#4527)
  • Fix incorrect logcontexts after a Deferred was cancelled (#4407)
  • Ensure encrypted room state is persisted across room upgrades. (#4411)
  • Copy over whether a room is a direct message and any associated room tags on room upgrade. (#4412)
  • Fix None guard in calling config.server.is_threepid_reserved (#4435)
  • Don’t send IP addresses as SNI (#4452)
  • Fix UnboundLocalError in post_urlencoded_get_json (#4460)
  • Add a timeout to filtered room directory queries. (#4461)
  • Workaround for login error when using both LDAP and internal authentication. (#4486)
  • Fix a bug where setting a relative consent directory path would cause a crash. (#4512)

Deprecations and Removals

  • Synapse no longer generates self-signed TLS certificates when generating a configuration file. (#4509)

Improved Documentation

  • Update debian installation instructions (#4526)

Internal Changes

  • Synapse will now take advantage of native UPSERT functionality in PostgreSQL 9.5+ and SQLite 3.24+. (#4306#4459#4466#4471#4477#4505)
  • Update README to use the new virtualenv everywhere (#4342)
  • Add better logging for unexpected errors while sending transactions (#4368)
  • Apply a unique index to the user_ips table, preventing duplicates. (#4370#4432#4434)
  • Silence travis-ci build warnings by removing non-functional python3.6 (#4377)
  • Fix a comment in the generated config file (#4387)
  • Add ground work for implementing future federation API versions (#4390)
  • Update dependencies on msgpack and pymacaroons to use the up-to-date packages. (#4399)
  • Tweak codecov settings to make them less loud. (#4400)
  • Implement server support for MSC1794 – Federation v2 Invite API (#4402)
  • debian package: symlink to explicit python version (#4433)
  • Add infrastructure to support different event formats (#4437#4447#4448#4470#4481#4482#4493#4494#4496#4510#4514)
  • Generate the debian config during build (#4444)
  • Clarify documentation for the public_baseurl config param (#4458#4498)
  • Fix quoting for allowed_local_3pids example config (#4476)
  • Remove deprecated –process-dependency-links option from UPGRADE.rst (#4485)
  • Make it possible to set the log level for tests via an environment variable (#4506)
  • Reduce the log level of linearizer lock acquirement to DEBUG. (#4507)
  • Fix code to comply with linting in PyFlakes 3.7.1. (#4519)
  • Add some debug for membership syncing issues (#4538)
  • Docker: only copy what we need to the build image (#4562)

Porting Synapse to Python 3

Matrix’s reference homeserver, Synapse, is written in Python and uses the Twisted networking framework to power its bitslinging across the Internet. The Python version used has been strictly Python 2.7, the last supported version of Python 2, but as of this week that changes! Since Twisted and our other upstream dependencies now support the newest version of Python, Python 3, we are now able to finish the jump and port Synapse to use it by default. The port has been done in a backwards compatible way, written in a subset of Python that is usable in both Python 2 and Python 3, meaning your existing Synapse installs still work on Python 2, while preparing us for a Python 3 future.

Why port?

Porting Synapse to Python 3 prepares Synapse for a post-Python 2 world, currently scheduled for 2020. After the 1st of January in 2020, Python 2 will no longer be supported by the core Python developers and no bugfixes (even critical security ones) will be issued. As the security of software depends very much on the runtime and libraries it is running on top of, this means that by then all Python 2 software in use should have moved to Python 3 or other runtimes.

The Python 3 port has benefits other than just preparing for the End of Life of Python 2.7. Successive versions of Python 3 have improved the standard library, provided newer and clearer syntax for asynchronous code, added opt-in static typing to reduce bugs, and contained incremental performance and memory management improvements. These features, once Synapse stops supporting Python 2, can then be fully utilised to make Synapse’s codebase clearer and more performant. One bonus that we get immediately, though, is Python 3’s memory compaction of Unicode strings. Rather than storing as UCS-2/UTF-16 or UCS-4/UTF-32, it will instead store it in the smallest possible representation giving a 50%-75% memory improvement for strings only containing Latin-1 characters, such as nearly all dictionary keys, hashes, IDs, and a large proportion of messages being processed from English speaking countries. Non-English text will also see a memory improvement, as it can be commonly stored in only two bytes instead of the four in a UCS-4 “wide” Python 2 build.

Editor’s note: If you were wondering how this fits in with Dendrite (the next-gen golang homeserver): our plan is to use Synapse as the reference homeserver for all the current work going on with landing a 1.0 release of the Matrix spec: it makes no sense to try to iterate and converge on 1.0 on both Synapse and Dendrite in parallel. In order to prove that the 1.0 spec is indeed fit for purpose we then also need Synapse to exit beta and hit a 1.0 too, hence the investment to get it there. It’s worth noting that over the last year we’ve been plugging away solidly improving Synapse in general (especially given the increasing number of high-profile deployments out there), so we’re committed to getting Synapse to a formal production grade release and supporting it in the long term. Meanwhile, Dendrite development is still progressing – currently acting as a place to experiment with more radical blue-sky architectural changes, especially in low-footprint or even clientside homeservers. We expect it to catch up with Synapse once 1.0 is out the door; and meanwhile Synapse is increasingly benefiting from performance work inspired by Dendrite.

When will the port be released?

The port is has been released in a “production ready” form in Synapse 0.34.0, supporting Python 3.5, 3.6, and 3.7. This will work on installations with and without workers.

What’s it like in the real world?

Beta testers of the Python 3 port have reported lower memory usage, including lower memory “spikes” and slower memory growth. You can see this demonstrated on matrix.org:

See 10/15, ~20:00 for the Python 3 migration. This is on some of the Synchrotrons on matrix.org.

See ~11/8 for the Python 3 migration. This is on the Synapse master on matrix.org.

We have also noticed some better CPU utilisation:


See 21:30 for the migration of federation reader 1, and 21:55 for the others. The federation reader is a particular pathological case, where the replacement of lists with iterators internally on Python 3 has given us some big boosts.

See 10/15, 4:00.The CPU utilisation has gone down on synchrotron 1 after the Python 3 migration, but not as dramatically as the federation reader. Synchrotron 3 was migrated a few days later.

As some extra data-points, my personal HS consumes about 300MB now at initial start, and grows to approximately 800MB — under Python 2 the growth would be near-immediate to roughly 1.4GB.

Where to from here?

Python 2 is still a supported platform for running Synapse for the time being. We plan on ending mainstream support on 1st April 2019, where upon Python 3.5+ will be the only officially supported platform. Additionally, we will give notice ahead of time once we are ready to remove Python 2.7 compatibility from the codebase (which will be no sooner than 1st April). Although slightly inconvenient, we hope that this gives our users and integrators adequate time to migrate, whilst giving us the flexibility to use modern Python features and make Synapse a better piece of software to help power the Matrix community.

How can I try it?

The port is compatible with existing homeservers and configurations, so if you install Synapse inside a Python 3 virtualenv, you can run it from there. Of course, this differs based on your installation method, operating system, and what version of Python 3 you wish to use. Full upgrade notes live here but if you’re having problems or want to discuss specific packagings of Synapse please come ask in #synapse:matrix.org.

Thanks

Many thanks go to fellow Synapse developers Erik and Rich for code review, as well as community contributors such as notafile and krombel for laying the foundations many months ago allowing this port to happen. Without them, this wouldn’t have happened.

Happy Matrixing,

Amber Brown (hawkowl)

Pre-disclosure: Upcoming critical security fix for Synapse

Hi all,

During the ongoing work to finalise a stable release of Matrix’s Server-Server federation API, we’ve been doing a full audit of Synapse’s implementation and have identified a serious vulnerability which we are going to release a security update to address (Synapse 0.33.3.1) on Thursday Sept 6th 2018 at 12:00 UTC.

We are coordinating with package maintainers to ensure that patched versions of packages will be available at that time – meanwhile, if you run your own Synapse, please be prepared to upgrade as soon as the patched versions are released.  All previous versions of Synapse are affected, so everyone will want to upgrade.

Thank you for your time, patience and understanding while we resolve the issue,

signed_predisclosure.txt

Synapse 0.33.3 Released

All the threes, Synapse 0.33.3!

This release brings together a lot of bugfixes, and also some preparation for support for Lazy Loading and Room Versioning.

We also have, as a great contribution from @vojeroen, SNI extension support! With v0.33.3, Synapse now supports sending SNI over federation for vhosted servers, which resolves this long-standing request.

As always, you can get the new update from https://github.com/matrix-org/synapse/releases/tag/v0.33.3 or any of the sources mentioned at https://github.com/matrix-org/synapse.

 

Features

  • Add support for the SNI extension to federation TLS connections. Thanks to @vojeroen! (#3439)
  • Add /_media/r0/config (#3184)
  • speed up /members API and add at and membership params as per MSC1227 (#3568)
  • implement summary block in /sync response as per MSC688 (#3574)
  • Add lazy-loading support to /messages as per MSC1227 (#3589)
  • Add ability to limit number of monthly active users on the server (#3633)
  • Support more federation endpoints on workers (#3653)
  • Basic support for room versioning (#3654)
  • Ability to disable client/server Synapse via conf toggle (#3655)
  • Ability to whitelist specific threepids against monthly active user limiting (#3662)
  • Add some metrics for the appservice and federation event sending loops (#3664)
  • Where server is disabled, block ability for locked out users to read new messages (#3670)
  • set admin uri via config, to be used in error messages where the user should contact the administrator (#3687)
  • Synapse’s presence functionality can now be disabled with the “use_presence” configuration option. (#3694)
  • For resource limit blocked users, prevent writing into rooms (#3708)

Bugfixes

  • Fix occasional glitches in the synapse_event_persisted_position metric (#3658)
  • Fix bug on deleting 3pid when using identity servers that don’t support unbind API (#3661)
  • Make the tests pass on Twisted < 18.7.0 (#3676)
  • Don’t ship recaptcha_ajax.js, use it directly from Google (#3677)
  • Fixes test_reap_monthly_active_users so it passes under postgres (#3681)
  • Fix mau blocking calulation bug on login (#3689)
  • Fix missing yield in synapse.storage.monthly_active_users.initialise_reserved_users (#3692)
  • Improve HTTP request logging to include all requests (#3700, #3723)
  • Avoid timing out requests while we are streaming back the response (#3701)
  • Support more federation endpoints on workers (#3705, #3713)
  • Fix “Starting db txn ‘get_all_updated_receipts’ from sentinel context” warning (#3710)
  • Fix bug where state_cache cache factor ignored environment variables (#3719)

Deprecations and Removals

Internal Changes

  • The test suite now can run under PostgreSQL. (#3423)
  • Refactor HTTP replication endpoints to reduce code duplication (#3632)
  • Tests now correctly execute on Python 3. (#3647)
  • Sytests can now be run inside a Docker container. (#3660)
  • Port over enough to Python 3 to allow the sytests to start. (#3668, #3732)
  • Update docker base image from alpine 3.7 to 3.8. (#3669)
  • Rename synapse.util.async to synapse.util.async_helpers to mitigate async becoming a keyword on Python 3.7. (#3678)
  • Synapse’s tests are now formatted with the black autoformatter. (#3679)
  • Implemented a new testing base class to reduce test boilerplate. (#3684)
  • Rename MAU prometheus metrics (#3690)
  • add new error type ResourceLimit (#3707)
  • Logcontexts for replication command handlers (#3709)
  • Update admin register API documentation to reference a real user ID. (#3712)