Okay folks, this is an important one. v0.99.5.1 will be the last release before we ship Synapse v1.0. It is really important that you upgrade to v0.99.5.1 because it implements rooms version 4 - which is the room version that Synapse 1.0 will default to.

This means that Synapse 1.0 servers will create new rooms as version 4 by default and servers that have not upgraded to at least v0.99.5.1 will not be able to join those rooms.

Over the coming days we will announce a release day for Synapse v1.0, the idea is to give admins 2 weeks notice so that anyone yet to configure their federation SSL certificate has time to do so. This is important, failure to configure your certs will mean not being able to federate with v1.0 servers. If you are not sure if you certs are valid, you can test here and read here for more info on what to do.

Aside from room v4, this release also includes the ability to blacklist specific IPs from federating as well as experimental support for edits and reactions. We are not quite ready to mark the feature 'done done', but it is very close. Watch out for news as the feature lands properly.

We're really close to v1.0 now, give us a few more days and we'll announce an official release date.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our Synapse installation guide page

Synapse v0.99.5.1 Changelog (since v0.99.4)

Features

• Add ability to blacklist IP ranges for the federation client. (#5043)
• Ratelimiting configuration for clients sending messages and the federation server has been altered to match login ratelimiting. The old configuration names will continue working. Check the sample config for details of the new names. (#5181)
• Drop support for the undocumented /_matrix/client/v2_alpha API prefix. (#5190)
• Add an option to disable per-room profiles. (#5196)
• Stick an expiration date to any registered user missing one at startup if account validity is enabled. (#5204)
• Add experimental support for relations (aka reactions and edits). (#5209, #5211, #5203, #5212)
• Add a room version 4 which uses a new event ID format, as per MSC2002. (#5210, #5217)

Bugfixes

• Fix image orientation when generating thumbnails (needs pillow>=4.3.0). Contributed by Pau Rodriguez-Estivill. (#5039)
• Exclude soft-failed events from forward-extremity candidates: fixes "No forward extremities left!" error. (#5146)
• Re-order stages in registration flows such that msisdn and email verification are done last. (#5174)
• Fix 3pid guest invites. (#5177)
• Fix a bug where the register endpoint would fail with M_THREEPID_IN_USE instead of returning an account previously registered in the same session. (#5187)
• Prevent registration for user ids that are too long to fit into a state key. Contributed by Reid Anderson. (#5198)
• Fix incompatibility between ACME support and Python 3.5.2. (#5218)
• Fix error handling for rooms whose versions are unknown. (#5219)

Internal Changes

• Make /sync attempt to return device updates for both joined and invited users. Note that this doesn't currently work correctly due to other bugs. (#3484)
• Update tests to consistently be configured via the same code that is used when loading from configuration files. (#5171, #5185)
• Allow client event serialization to be async. (#5183)
• Expose DataStore._get_events as get_events_as_list. (#5184)
• Make generating SQL bounds for pagination generic. (#5191)
• Stop telling people to install the optional dependencies by default. (#5197)

Hey ho Synapse release day.

0.99.4 is a maintenance release collecting together all of the bug fixes and performance improvements over the past few weeks, additionally there is further support for the upcoming 1.0 release (more info coming soon). One thing worth calling out is how many community contributions have made their way into 0.99.4, take a look at the change log for details, but many thanks to everyone submitting PRs, keep them coming!

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our Synapse installation guide page

Synapse 0.99.4 Changelog

Features

• Add systemd-python to the optional dependencies to enable logging to the systemd journal. Install with pip install matrix-synapse[systemd]. (#4339)
• Add a default .m.rule.tombstone push rule. (#4867)
• Add ability for password provider modules to bind email addresses to users upon registration. (#4947)
• Implementation of MSC1711 including config options for requiring valid TLS certificates for federation traffic, the ability to disable TLS validation for specific domains, and the ability to specify your own list of CA certificates. (#4967)
• Remove presence list support as per MSC 1819. (#4989)
• Reduce CPU usage starting pushers during start up. (#4991)
• Add a delete group admin API. (#5002)
• Add config option to block users from looking up 3PIDs. (#5010)
• Add context to phonehome stats. (#5020)
• Configure the example systemd units to have a log identifier of matrix-synapse instead of the executable name, python. Contributed by Christoph Müller. (#5023)
• Add time-based account expiration. (#5027, #5047, #5073, #5116)
• Add support for handling /versions, /voip and /push_rules client endpoints to client_reader worker. (#5063, #5065, #5070)
• Add an configuration option to require authentication on /publicRooms and /profile endpoints. (#5083)
• Move admin APIs to /_synapse/admin/v1. (The old paths are retained for backwards-compatibility, for now). (#5119)
• Implement an admin API for sending server notices. Many thanks to @krombel who provided a foundation for this work. (#5121, #5142)

Bugfixes

• Avoid redundant URL encoding of redirect URL for SSO login in the fallback login page. Fixes a regression introduced in #4220. Contributed by Marcel Fabian Krüger ("zaugin"). (#4555)
• Fix bug where presence updates were sent to all servers in a room when a new server joined, rather than to just the new server. (#4942, #5103)
• Fix sync bug which made accepting invites unreliable in worker-mode synapses. (#4955, #4956)
• start.sh: Fix the --no-rate-limit option for messages and make it bypass rate limit on registration and login too. (#4981)
• Transfer related groups on room upgrade. (#4990)
• Prevent the ability to kick users from a room they aren't in. (#4999)
• Fix issue #4596 so synapse_port_db script works with --curses option on Python 3. Contributed by Anders Jensen-Waud [email protected]. (#5003)
• Clients timing out/disappearing while downloading from the media repository will now no longer log a spurious "Producer was not unregistered" message. (#5009)
• Fix "cannot import name execute_batch" error with postgres. (#5032)
• Fix disappearing exceptions in manhole. (#5035)
• Workaround bug in twisted where attempting too many concurrent DNS requests could cause it to hang due to running out of file descriptors. (#5037)
• Make sure we're not registering the same 3pid twice on registration. (#5071)
• Don't crash on lack of expiry templates. (#5077)
• Fix the ratelimting on third party invites. (#5104)
• Add some missing limitations to room alias creation. (#5124, #5128)
• Limit the number of EDUs in transactions to 100 as expected by synapse. Thanks to @superboum for this work! (#5138)
• Fix bogus imports in unit tests. (#5154)

Internal Changes

• Add test to verify threepid auth check added in #4435. (#4474)
• Fix/improve some docstrings in the replication code. (#4949)
• Split synapse.replication.tcp.streams into smaller files. (#4953)
• Refactor replication row generation/parsing. (#4954)
• Run black to clean up formatting on synapse/storage/roommember.py and synapse/storage/events.py. (#4959)
• Remove log line for password via the admin API. (#4965)
• Fix typo in TLS filenames in docker/README.md. Also add the '-p' commandline option to the 'docker run' example. Contributed by Jurrie Overgoor. (#4968)
• Refactor room version definitions. (#4969)
• Reduce log level of .well-known/matrix/client responses. (#4972)
• Add config.signing_key_path that can be read by synapse.config utility. (#4974)
• Track which identity server is used when binding a threepid and use that for unbinding, as per MSC1915. (#4982)
• Rewrite KeyringTestCase as a HomeserverTestCase. (#4985)
• README updates: Corrected the default POSTGRES_USER. Added port forwarding hint in TLS section. (#4987)
• Remove a number of unused tables from the database schema. (#4992, #5028, #5033)
• Run black on the remainder of synapse/storage/. (#4996)
• Fix grammar in get_current_users_in_room and give it a docstring. (#4998)
• Clean up some code in the server-key Keyring. (#5001)
• Convert SYNAPSE_NO_TLS Docker variable to boolean for user friendliness. Contributed by Gabriel Eckerson. (#5005)
• Refactor synapse.storage._base._simple_select_list_paginate. (#5007)
• Store the notary server name correctly in server_keys_json. (#5024)
• Rewrite Datastore.get_server_verify_keys to reduce the number of database transactions. (#5030)
• Remove extraneous period from copyright headers. (#5046)
• Update documentation for where to get Synapse packages. (#5067)
• Add workarounds for pep-517 install errors. (#5098)
• Improve logging when event-signature checks fail. (#5100)
• Factor out an "assert_requester_is_admin" function. (#5120)
• Remove the requirement to authenticate for /admin/server_version. (#5122)
• Prevent an exception from being raised in a IResolutionReceiver and use a more generic error message for blacklisted URL previews. (#5155)
• Run black on the tests directory. (#5170)
• Fix CI after new release of isort. (#5179)

Hey hey, a Synapse release for you today.

The big news in 0.99.3 is that the user directory has been completely re-written and should now be much more performant - this will benefit all installations, but especially those housing larger servers.

Aside from that we continue our 1.0 preparations and relatedly we've improved our docs, in particular to explain how .well-known works. On the perf side we've added rate limiting to login and register endpoints as well as now batching up read receipts to send over federation.

I've said it before, and I'll say it again:-

The most important thing that admins should know is that prior to 1.0 landing later this month, it is essential that the federation API has a valid TLS certificate - self signed certificates will no longer be accepted. For more details see our handy guide. Failure to do this will result in being unable to federate with other 1.0 servers.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.

Synapse 0.99.3 changelog

Features

• The user directory has been rewritten to make it faster, with less chance of falling behind on a large server. (#4537, #4846, #4864, #4887, #4900, #4944)
• Add configurable rate limiting to the /register endpoint. (#4735, #4804)
• Move server key queries to federation reader. (#4757)
• Add support for /account/3pid REST endpoint to client_reader worker. (#4759)
• Add an endpoint to the admin API for querying the server version. Contributed by Joseph Weston. (#4772)
• Include a default configuration file in the 'docs' directory. (#4791, #4801)
• Synapse is now permissive about trailing slashes on some of its federation endpoints, allowing zero or more to be present. (#4793)
• Add support for /keys/query and /keys/changes REST endpoints to client_reader worker. (#4796)
• Add checks to incoming events over federation for events evading auth (aka "soft fail"). (#4814)
• Add configurable rate limiting to the /login endpoint. (#4821, #4865)
• Remove trailing slashes from certain outbound federation requests. Retry if receiving a 404. Context: #3622. (#4840)
• Allow passing --daemonize flags to workers in the same way as with master. (#4853)
• Batch up outgoing read-receipts to reduce federation traffic. (#4890, #4927)
• Add option to disable searching the user directory. (#4895)
• Add option to disable searching of local and remote public room lists. (#4896)
• Add ability for password providers to login/register a user via 3PID (email, phone). (#4931)

Bugfixes

• Fix a bug where media with spaces in the name would get a corrupted name. (#2090)
• Fix attempting to paginate in rooms where server cannot see any events, to avoid unnecessarily pulling in lots of redacted events. (#4699)
• 'event_id' is now a required parameter in federated state requests, as per the matrix spec. (#4740)
• Fix tightloop over connecting to replication server. (#4749)
• Fix parsing of Content-Disposition headers on remote media requests and URL previews. (#4763)
• Fix incorrect log about not persisting duplicate state event. (#4776)
• Fix v4v6 option in HAProxy example config. Contributed by Flakebi. (#4790)
• Handle batch updates in worker replication protocol. (#4792)
• Fix bug where we didn't correctly throttle sending of USER_IP commands over replication. (#4818)
• Fix potential race in handling missing updates in device list updates. (#4829)
• Fix bug where synapse expected an un-specced prev_state field on state events. (#4837)
• Transfer a user's notification settings (push rules) on room upgrade. (#4838)
• fix test_auto_create_auto_join_where_no_consent. (#4886)
• Fix a bug where hs_disabled_message was sometimes not correctly enforced. (#4888)
• Fix bug in shutdown room admin API where it would fail if a user in the room hadn't consented to the privacy policy. (#4904)
• Fix bug where blocked world-readable rooms were still peekable. (#4908)

Internal Changes

• Add a systemd setup that supports synapse workers. Contributed by Luca Corbatto. (#4662)
• Change from TravisCI to Buildkite for CI. (#4752)
• When presence is disabled don't send over replication. (#4757)
• Minor docstring fixes for MatrixFederationAgent. (#4765)
• Optimise EDU transmission for the federation_sender worker. (#4770)
• Update test_typing to use HomeserverTestCase. (#4771)
• Update URLs for riot.im icons and logos in the default notification templates. (#4779)
• Removed unnecessary $ from some federation endpoint path regexes. (#4794)
• Remove link to deleted title in README. (#4795)
• Clean up read-receipt handling. (#4797)
• Add some debug about processing read receipts. (#4798)
• Clean up some replication code. (#4799)
• Add some docstrings. (#4815)
• Add debug logger to try and track down #4422. (#4816)
• Make shutdown API send explanation message to room after users have been forced joined. (#4817)
• Update example_log_config.yaml. (#4820)
• Document the generate option for the docker image. (#4824)
• Fix check-newsfragment for debian-only changes. (#4825)
• Add some debug logging for device list updates to help with #4828. (#4828)
• Improve federation documentation, specifically .well-known support. Many thanks to @vaab. (#4832)
• Disable captcha registration by default in unit tests. (#4839)
• Add stuff back to the .gitignore. (#4843)
• Clarify what registration_shared_secret allows for. (#4844)
• Correctly log expected errors when fetching server keys. (#4847)
• Update install docs to explicitly state a full-chain (not just the top-level) TLS certificate must be provided to Synapse. This caused some people's Synapse ports to appear correct in a browser but still (rightfully so) upset the federation tester. (#4849)
• Move client read-receipt processing to federation sender worker. (#4852)
• Refactor federation TransactionQueue. (#4855)
• Comment out most options in the generated config. (#4863)
• Fix yaml library warnings by using safe_load. (#4869)
• Update Apache setup to remove location syntax. Thanks to @cwmke! (#4870)
• Reinstate test case that runs unit tests against oldest supported dependencies. (#4879)
• Update link to federation docs. (#4881)
• fix test_auto_create_auto_join_where_no_consent. (#4886)
• Use a regular HomeServerConfig object for unit tests rater than a Mock. (#4889)
• Add some notes about tuning postgres for larger deployments. (#4895)
• Add a config option for torture-testing worker replication. (#4902)
• Log requests which are simulated by the unit tests. (#4905)
• Allow newsfragments to end with exclamation marks. Exciting! (#4912)
• Refactor some more tests to use HomeserverTestCase. (#4913)
• Refactor out the state deltas portion of the user directory store and handler. (#4917)
• Fix nginx example in ACME doc. (#4923)
• Use an explicit dbname for postgres connections in the tests. (#4928)
• Fix ClientReplicationStreamProtocol.__str__(). (#4929)

Well now, what have we here? Synapse 0.99.2 is the latest in the 0.99.x series as we step ever closer to 1.0.

0.99.2 is an incremental release including a bunch of performance improvements, enhancements to room upgrades and generally a plethora of bug fixes.

The most important thing that admins should know is that prior to 1.0 landing later this month, it is essential that the federation API has a valid TLS certificate - self signed certificates will no longer be accepted. For more details see our handy guide. Failure to do this will result in being unable to federate with other 1.0 servers.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.

Synapse 0.99.2 changelog

Features

• Added an HAProxy example in the reverse proxy documentation. Contributed by Benoît S. (“Benpro”). (#4541)
• Add basic optional sentry integration. (#4632, #4694)
• Transfer bans on room upgrade. (#4642)
• Add configurable room list publishing rules. (#4647)
• Support .well-known delegation when issuing certificates through ACME. (#4652)
• Allow registration and login to be handled by a worker instance. (#4666, #4670, #4682)
• Reduce the overhead of creating outbound federation connections over TLS by caching the TLS client options. (#4674)
• Add prometheus metrics for number of outgoing EDUs, by type. (#4695)
• Return correct error code when inviting a remote user to a room whose homeserver does not support the room version. (#4721)
• Prevent showing rooms to other servers that were set to not federate. (#4746)

Bugfixes

• Fix possible exception when paginating. (#4263)
• The dependency checker now correctly reports a version mismatch for optional dependencies, instead of reporting the dependency missing. (#4450)
• Set CORS headers on .well-known requests. (#4651)
• Fix kicking guest users on guest access revocation in worker mode. (#4667)
• Fix an issue in the database migration script where the e2e_room_keys.is_verified column wasn't considered as a boolean. (#4680)
• Fix TaskStopped exceptions in logs when outbound requests time out. (#4690)
• Fix ACME config for python 2. (#4717)
• Fix paginating over federation persisting incorrect state. (#4718)

Internal Changes

• Run black to reformat user directory code. (#4635)
• Reduce number of exceptions we log. (#4643, #4668)
• Introduce upsert batching functionality in the database layer. (#4644)
• Fix various spelling mistakes. (#4657)
• Cleanup request exception logging. (#4669, #4737, #4738)
• Improve replication performance by reducing cache invalidation traffic. (#4671, #4715, #4748)
• Test against Postgres 9.5 as well as 9.4. (#4676)
• Run unit tests against python 3.7. (#4677)
• Attempt to clarify installation instructions/config. (#4681)
• Clean up gitignores. (#4688)
• Minor tweaks to acme docs. (#4689)
• Improve the logging in the pusher process. (#4691)
• Better checks on newsfragments. (#4698, #4750)
• Avoid some redundant work when processing read receipts. (#4706)
• Run push_receipts_to_remotes as background job. (#4707)
• Add prometheus metrics for number of badge update pushes. (#4709)
• Reduce pusher logging on startup (#4716)
• Don't log exceptions when failing to fetch remote server keys. (#4722)
• Correctly proxy exception in frontend_proxy worker. (#4723)
• Add database version to phonehome stats. (#4753)

Hey, everyone, today is the day we release Synapse 0.99.1.1

This release contains improved ACME support to make it even easier to get going with TLS certs on your federation end points, plus some tweaks to make the room version upgrade path easier.

Just as a reminder that the 0.99.x series is precursor for our 1.0 release (which will land in early March, exact date to be confirmed) - it is really important that all server admins are aware that self signed certificates on the Server to Server API will no longer be accepted by >= Synapse 1.0. If you have not already done so, now is the time to configure your certificate. For more info see our FAQ and if you get stuck come and join us in #Synapse.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.

Synapse 0.99.1.1 Changelog

Bugfixes

• Fix "TypeError: '>' not supported" when starting without an existing certificate. Fix a bug where an existing certificate would be reprovisoned every day. (#4648)

Synapse 0.99.1 Changelog

Features

• Include m.room.encryption on invites by default (#3902)
• Federation OpenID listener resource can now be activated even if federation is disabled (#4420)
• Synapse's ACME support will now correctly reprovision a certificate that approaches its expiry while Synapse is running. (#4522)
• Add ability to update backup versions (#4580)
• Allow the "unavailable" presence status for /sync. This change makes Synapse compliant with r0.4.0 of the Client-Server specification. (#4592)
• There is no longer any need to specify no_tls: it is inferred from the absence of TLS listeners (#4613, #4615, #4617, #4636)
• The default configuration no longer requires TLS certificates. (#4614)

Bugfixes

• Copy over room federation ability on room upgrade. (#4530)
• Fix noisy "twisted.internet.task.TaskStopped" errors in logs (#4546)
• Synapse is now tolerant of the tls_fingerprints option being None or not specified. (#4589)
• Fix 'no unique or exclusion constraint' error (#4591)
• Transfer Server ACLs on room upgrade. (#4608)
• Fix failure to start when not TLS certificate was given even if TLS was disabled. (#4618)
• Fix self-signed cert notice from generate-config. (#4625)
• Fix performance of user_ips table deduplication background update (#4626, #4627)

Internal Changes

• Change the user directory state query to use a filtered call to the db instead of a generic one. (#4462)
• Reject federation transactions if they include more than 50 PDUs or 100 EDUs. (#4513)
• Reduce duplication of synapse.app code. (#4567)
• Fix docker upload job to push -py2 images. (#4576)
• Add port configuration information to ACME instructions. (#4578)
• Update MSC1711 FAQ to clarify .well-known usage (#4584)
• Clean up default listener configuration (#4586)
• Clarifications for reverse proxy docs (#4607)
• Move ClientTLSOptionsFactory init out of refresh_certificates (#4611)
• Fail cleanly if listener config lacks a 'port' (#4616)
• Remove redundant entries from docker config (#4619)
• README updates (#4621)

Hey hey, Synapse 0.99.0 is here!

You may have heard that we recently published the first stable release of the Server to Server Spec (r0.1). The spec makes some changes which are not compatible with the protocol of the past - particularly, self-signed certificates are no longer valid for homeservers. Synapse 1.0.0 will be compliant with r0.1 and the goal of Synapse 0.99.0 is to act as a stepping stone to Synapse 1.0. Synapse 0.99.0 supports the r0.1 release of the server to server specification, but is compatible with both the legacy Matrix federation behaviour (pre-r0.1) as well as post-r0.1 behaviour, in order to allow for a smooth upgrade across the federation.

It is critical that all admins upgrade to 0.99.0 and configure a valid TLS certificate. Admins will have 1 month to do so, after which 1.0.0 will be released and those servers without a valid certificate will no longer be able to federate with >= 1.0.0 servers.

First of all, please don't panic :) We have taken steps to make this process as simple as possible - specifically implementing ACME support to allow servers to automatically generate free Let's Encrypt certificates if you choose to. What's more, it is not necessary to add the certificate right away, you have at least a month to get set up.

For more details on exactly what you need to do (and also why this change is essential), we have provided an extensive FAQ as well as the Upgrade notes for Synapse

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.

This was a huge effort! Congratulations to all involved, especially those of you in the community who contributed to spec MSCs and tested our release candidates. Thank you for bearing with us as we move the whole public Matrix Federation onto r0.1 compliant servers.

Onwards!

Changelog

Synapse v0.99.x is a precursor to the upcoming Synapse v1.0 release. It contains foundational changes to room architecture and the federation security model necessary to support the upcoming r0 release of the Server to Server API.

Features

• Synapse's cipher string has been updated to require ECDH key exchange. Configuring and generating dh_params is no longer required, and they will be ignored. (#4229)
• Synapse can now automatically provision TLS certificates via ACME (the protocol used by CAs like Let's Encrypt). (#4384, #4492, #4525, #4572, #4564, #4566, #4547, #4557)
• Implement MSC1708 (.well-known routing for server-server federation) (#4408, #4409, #4426, #4427, #4428, #4464, #4468, #4487, #4488, #4489, #4497, #4511, #4516, #4520, #4521, #4539, #4542, #4544)
• Search now includes results from predecessor rooms after a room upgrade. (#4415)
• Config option to disable requesting MSISDN on registration. (#4423)
• Add a metric for tracking event stream position of the user directory. (#4445)
• Support exposing server capabilities in CS API (MSC1753, MSC1804) (#4472, 81b7e7eed))
• Add support for room version 3 (#4483, #4499, #4515, #4523, #4535)
• Synapse will now reload TLS certificates from disk upon SIGHUP. (#4495, #4524)
• The matrixdotorg/synapse Docker images now use Python 3 by default. (#4558)

Bugfixes

• Prevent users with access tokens predating the introduction of device IDs from creating spurious entries in the user_ips table. (#4369)
• Fix typo in ALL_USER_TYPES definition to ensure type is a tuple (#4392)
• Fix high CPU usage due to remote devicelist updates (#4397)
• Fix potential bug where creating or joining a room could fail (#4404)
• Fix bug when rejecting remote invites (#4405, #4527)
• Fix incorrect logcontexts after a Deferred was cancelled (#4407)
• Ensure encrypted room state is persisted across room upgrades. (#4411)
• Copy over whether a room is a direct message and any associated room tags on room upgrade. (#4412)
• Fix None guard in calling config.server.is_threepid_reserved (#4435)
• Don't send IP addresses as SNI (#4452)
• Fix UnboundLocalError in post_urlencoded_get_json (#4460)
• Add a timeout to filtered room directory queries. (#4461)
• Workaround for login error when using both LDAP and internal authentication. (#4486)
• Fix a bug where setting a relative consent directory path would cause a crash. (#4512)

Deprecations and Removals

• Synapse no longer generates self-signed TLS certificates when generating a configuration file. (#4509)

Improved Documentation

• Update debian installation instructions (#4526)

Internal Changes

• Synapse will now take advantage of native UPSERT functionality in PostgreSQL 9.5+ and SQLite 3.24+. (#4306, #4459, #4466, #4471, #4477, #4505)
• Update README to use the new virtualenv everywhere (#4342)
• Add better logging for unexpected errors while sending transactions (#4368)
• Apply a unique index to the user_ips table, preventing duplicates. (#4370, #4432, #4434)
• Silence travis-ci build warnings by removing non-functional python3.6 (#4377)
• Fix a comment in the generated config file (#4387)
• Add ground work for implementing future federation API versions (#4390)
• Update dependencies on msgpack and pymacaroons to use the up-to-date packages. (#4399)
• Tweak codecov settings to make them less loud. (#4400)
• Implement server support for MSC1794 - Federation v2 Invite API (#4402)
• debian package: symlink to explicit python version (#4433)
• Add infrastructure to support different event formats (#4437, #4447, #4448, #4470, #4481, #4482, #4493, #4494, #4496, #4510, #4514)
• Generate the debian config during build (#4444)
• Clarify documentation for the public_baseurl config param (#4458, #4498)
• Fix quoting for allowed_local_3pids example config (#4476)
• Remove deprecated --process-dependency-links option from UPGRADE.rst (#4485)
• Make it possible to set the log level for tests via an environment variable (#4506)
• Reduce the log level of linearizer lock acquirement to DEBUG. (#4507)
• Fix code to comply with linting in PyFlakes 3.7.1. (#4519)
• Add some debug for membership syncing issues (#4538)
• Docker: only copy what we need to the build image (#4562)

After releasing Synapse v0.34.1, we have become aware of a security vulnerability affecting all previous versions (CVE-2019-5885). v0.34.1 closed the vulnerability but, in some cases, caused users to be logged out of their clients, so we do not recommend v0.34.1 for production use.

Today we release two mitigating versions v0.34.0.1 and v0.34.1.1. Both versions close the vulnerability and will not cause users to be logged out. All installations should be upgraded to one or other immediately.

• Admins who would otherwise upgrade to v0.34.1 (or those that have already done so) should upgrade to v0.34.1.1.
• Admins on v0.34.0, who do not wish to bring in new non-security related behaviour, should upgrade to v0.34.0.1.

You can get the new updates for v0.34.0.1 and v0.34.1.1 here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. See also our Synapse installation guide page.

We will publish more details of the vulnerability once admins have had a chance to upgrade. To our knowledge the vulnerability has not been exploited in the wild.

Many thanks for your patience, we are moving ever closer to Synapse reaching v1.0, and fixes like this one edge us ever closer.

Thanks also to the package maintainers who have coordinated with us to ensure distro packages are available for a speedy upgrade!

• Synapse v0.34.0.1 Changelog
• Synapse v0.34.1.1 Changelog

Folks this is a big day for us at Matrix Towers, because today we release 0.34.0.

The big news for 0.34.0 is that we now recommend Python 3 for production use and have been running matrix.org under Python 3 for the past month.

Performance improvements have been marked, in some contexts we have seen 50% reductions in RAM and CPU usage. Here are some illustrative graphs to get you going but look out for a dedicated post delving into much more detail on the port. You can also see a Matrix Live interview with the project lead Amber (hawkowl) here.

Matrix.org federation reader workers, the big drops signify roll over to python 3

Synapse master on matrix.org, again the drop in RAM signifies the roll over to python 3

Many thanks to Amber for leading the effort, Rich and Erik for providing support as well as Notafile and Krombel from the community for pushing this effort right from the early days of the project.

If that wasn't enough, 0.34.0 also all the usual bug fixes and perf improvements. In particular the media repository now no longer fails to decode UTF-8 filenames when downloading remote media and auto joining rooms now work on servers with consent requirements enabled.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.

In particular, if you want to run Synapse 0.34.0 on Python 3 take a look at the upgrade notes.

Synapse 0.34.0 changelog

Synapse 0.34.0 is the first release to fully support Python 3. Synapse will now run on Python versions 3.5 or 3.6 (as well as 2.7). Support for Python 3.7 remains experimental.

We recommend upgrading to Python 3, but make sure to read the upgrade notes when doing so.

Features

• Add 'sandbox' to CSP for media reprository (#4284)
• Make the new landing page prettier. (#4294)
• Fix deleting E2E room keys when using old SQLite versions. (#4295)
• Add a welcome page for the client API port. Credit to @krombel! (#4289)
• Remove Matrix console from the default distribution (#4290)
• Add option to track MAU stats (but not limit people) (#3830)
• Add an option to enable recording IPs for appservice users (#3831)
• Rename login type m.login.cas to m.login.sso (#4220)
• Add an option to disable search for homeservers that may not be interested in it. (#4230)

Bugfixes

• Pushrules can now again be made with non-ASCII rule IDs. (#4165)
• The media repository now no longer fails to decode UTF-8 filenames when downloading remote media. (#4176)
• URL previews now correctly decode non-UTF-8 text if the header contains a <meta http-equiv="Content-Type" header. (#4183)
• Fix an issue where public consent URLs had two slashes. (#4192)
• Fallback auth now accepts the session parameter on Python 3. (#4197)
• Remove riot.im from the list of trusted Identity Servers in the default configuration (#4207)
• fix start up failure when mau_limit_reserved_threepids set and db is postgres (#4211)
• Fix auto join failures for servers that require user consent (#4223)
• Fix exception caused by non-ascii event IDs (#4241)
• Pushers can now be unsubscribed from on Python 3. (#4250)
• Fix UnicodeDecodeError when postgres is configured to give non-English errors (#4253)

Internal Changes

• Debian packages utilising a virtualenv with bundled dependencies can now be built. (#4212)
• Disable pager when running git-show in CI (#4291)
• A coveragerc file has been added. (#4180)
• Add a GitHub pull request template and add multiple issue templates (#4182)
• Update README to reflect the fact that #1491 is fixed (#4188)
• Run the AS senders as background processes to fix warnings (#4189)
• Add some diagnostics to the tests to detect logcontext problems (#4190)
• Add missing jpeg package prerequisite for OpenBSD in README. (#4193)
• Add a note saying you need to manually reclaim disk space after using the Purge History API (#4200)
• More logcontext checking in unittests (#4205)
• Ignore __pycache__ directories in the database schema folder (#4214)
• Add note to UPGRADE.rst about removing riot.im from list of trusted identity servers (#4224)
• Added automated coverage reporting to CI. (#4225)
• Garbage-collect after each unit test to fix logcontext leaks (#4227)
• add more detail to logging regarding "More than one row matched" error (#4234)
• Drop sent_transactions table (#4244)
• Add a basic .editorconfig (#4257)
• Update README.rst and UPGRADE.rst for Python 3. (#4260)
• Remove obsolete verbose and log_file settings from homeserver.yaml for Docker image. (#4261)

Well here you are then, the 9th episode in the Synapse 0.33.x series.

Features wise, 0.33.9 contains a change to the way that GDPR consent works under the hood. It is now plumbed in to the login flow (rather than following immediately afterwards) such that it does not inadvertently break on-boarding. This is part of a broader set of changes that span Synapse and Riot to improve initial first impressions of using matrix.

Separately we now have support for room version upgrades which is pre-requisite for rolling out the new state resolution algorithm, come and join us in #teststateresv2:jki.re if you would like to help us test.

Finally we've spent a bunch of time further improving perf especially in and around reducing device ids federation traffic.

I know I say it every time, but full python 3 support is really really close now, matrix.org is now running entirely on py3 and seeing some amazing perf improvements - the remaining blocker is getting py3 deb packages ready and then we'll ship an official python 3 release. There will also be a blog post to explain what we've been up to and what to expect perf wise.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.

Synapse 0.33.9 changelog

Features

• Include flags to optionally add m.login.terms to the registration flow when consent tracking is enabled. (#4004, #4133, #4142, #4184)
• Support for replacing rooms with new ones (#4091, #4099, #4100, #4101)

Bugfixes

• Fix exceptions when using the email mailer on Python 3. (#4095)
• Fix e2e key backup with more than 9 backup versions (#4113)
• Searches that request profile info now no longer fail with a 500. (#4122)
• fix return code of empty key backups (#4123)
• If the typing stream ID goes backwards (as on a worker when the master restarts), the worker's typing handler will no longer erroneously report rooms containing new typing events. (#4127)
• Fix table lock of device_lists_remote_cache which could freeze the application (#4132)
• Fix exception when using state res v2 algorithm (#4135)
• Generating the user consent URI no longer fails on Python 3. (#4140, #4163)
• Loading URL previews from the DB cache on Postgres will no longer cause Unicode type errors when responding to the request, and URL previews will no longer fail if the remote server returns a Content-Type header with the chartype in quotes. (#4157)
• The hash_password script now works on Python 3. (#4161)
• Fix noop checks when updating device keys, reducing spurious device list update notifications. (#4164)

Deprecations and Removals

• The disused and un-specced identicon generator has been removed. (#4106)
• The obsolete and non-functional /pull federation endpoint has been removed. (#4118)
• The deprecated v1 key exchange endpoints have been removed. (#4119)
• Synapse will no longer fetch keys using the fallback deprecated v1 key exchange method and will now always use v2. (#4120)

Internal Changes

• Fix build of Docker image with docker-compose (#3778)
• Delete unreferenced state groups during history purge (#4006)
• The "Received rdata" log messages on workers is now logged at DEBUG, not INFO. (#4108)
• Reduce replication traffic for device lists (#4109)
• Fix synapse_replication_tcp_protocol_*_commands metric label to be full command name, rather than just the first character (#4110)
• Log some bits about room creation (#4121)
• Fix tox failure on old systems (#4124)
• Add STATE_V2_TEST room version (#4128)
• Clean up event accesses and tests (#4137)
• The default logging config will now set an explicit log file encoding of UTF-8. (#4138)
• Add helpers functions for getting prev and auth events of an event (#4139)
• Add some tests for the HTTP pusher. (#4149)
• add purge_history.sh and purge_remote_media.sh scripts to contrib/ (#4155)
• HTTP tests have been refactored to contain less boilerplate. (#4156)
• Drop incoming events from federation for unknown rooms (#4165)

Wowzers - our 8th dot release for v0.33!

This time we have a bunch of bug fixes and db performance improvements as well as better support for auto-join rooms and the ability for admins to limit who can create rooms aliases.

v0.33.8 also contains more python 3 fixes: we are running most of matrix.org on python 3 as of right now and seeing some pretty impressive performance improvements. Look out for Hawkowl's write up coming soon.

For those interested in what we are working on right now, take a look at our task board.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here.

Synapse 0.33.8 changelog

Features

• Servers with auto-join rooms will now automatically create those rooms when the first user registers (#3975)
• Add config option to control alias creation (#4051)
• The register_new_matrix_user script is now ported to Python 3. (#4085)
• Configure Docker image to listen on both ipv4 and ipv6. (#4089)

Bugfixes

• Fix HTTP error response codes for federated group requests. (#3969)
• Fix issue where Python 3 users couldn't paginate /publicRooms (#4046)
• Fix URL previewing to work in Python 3.7 (#4050)
• synctl will use the right python executable to run worker processes (#4057)
• Manhole now works again on Python 3, instead of failing with a "couldn't match all kex parts" when connecting. (#4060, #4067)
• Fix some metrics being racy and causing exceptions when polled by Prometheus. (#4061)
• Fix bug which prevented email notifications from being sent unless an absolute path was given for email_templates. (#4068)
• Correctly account for cpu usage by background threads (#4074)
• Fix race condition where config defined reserved users were not being added to the monthly active user list prior to the homeserver reactor firing up (#4081)
• Fix bug which prevented backslashes being used in event field filters (#4083)

Internal Changes

• Add information about the matrix-docker-ansible-deploy playbook (#3698)
• Add initial implementation of new state resolution algorithm (#3786)
• Reduce database load when fetching state groups (#4011)
• Various cleanups in the federation client code (#4031)
• Run the CircleCI builds in docker containers (#4041)
• Only colourise synctl output when attached to tty (#4049)
• Refactor room alias creation code (#4063)
• Make the Python scripts in the top-level scripts folders meet pep8 and pass flake8. (#4068)
• The README now contains example for the Caddy web server. Contributed by steamp0rt. (#4072)
• Add psutil as an explicit dependency (#4073)
• Clean up threading and logcontexts in pushers (#4075)
• Correctly manage logcontexts during startup to fix some "Unexpected logging context" warnings (#4076)
• Give some more things logcontexts (#4077)
• Clean up some bits of code which were flagged by the linter (#4082)