Matrix 1.0 – Are We Ready Yet?

TL;DR

  • If you run a Synapse ensure that your federation certificates are valid here.
  • If they are not valid check out the FAQ.
  • Follow along with progress at https://arewereadyyet.com
  • Tell all your admin friends.

Folks, as you know we are now very close to achieving Matrix 1.0 and finally being in a position to shed our ‘beta’ tag. It has been a long time coming and speaks to the huge effort from hundreds of people over the past 5 years.

A critical step towards this goal is the release of Synapse 1.0. We want to ship Synapse 1.0 as soon as possible but can’t do so without your help!

We’d like to introduce AreWeReadyYet.com – a quick and easy way for everyone to track the progress and check if their federation is ready for Matrix 1.0!!

Are we ready yet?

Synapse 1.0 is good news for anyone running a Synapse installation – it contains critical bug fixes, security patches, a new room algorithm version and dramatically improved user and room search. However, as part of the security work it also contains a breaking change from previous Synapse versions. From 1.0 onwards it will necessary to ensure a valid TLS certificate on the federation API. Self signed certificates will no longer be accepted. Why would we do such a thing?

In anticipation for this, everyone currently running a homeserver must ensure that they have checked their federation certificate (check yours here). Failure to do so will mean being unable to federate with any Matrix 1.0 compliant server. If your server fails the check, our FAQ has all the details on what you need to do.

This post is a call to arms to try and get as many admins to upgrade their certificates as possible. We are tracking adoption at https://arewereadyyet.com – currently this sits at about 55% – we need this figure to be higher before we can pull the lever.  

So what are you waiting for? Check that your server has valid certs – then tell all your admin pals to do the same. Friends don’t let friends miss out on Synapse 1.0, send them to arewereadyyet.com (or tweet here to remind them!) We really need the community to help us here because at some point soon, we will need to pull the lever and release.

Once we make more progress on adoption, we will announce an official release date and finally get Synapse out of beta!

Synapse 0.99.2 released!

Well now, what have we here? Synapse 0.99.2 is the latest in the 0.99.x series as we step ever closer to 1.0.

0.99.2 is an incremental release including a bunch of performance improvements, enhancements to room upgrades and generally a plethora of bug fixes.

The most important thing that admins should know is that prior to 1.0 landing later this month, it is essential that the federation API has a valid TLS certificate – self signed certificates will no longer be accepted. For more details see our handy guide. Failure to do this will result in being unable to federate with other 1.0 servers.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.

 

Synapse 0.99.2 changelog

Features

  • Added an HAProxy example in the reverse proxy documentation. Contributed by Benoît S. (“Benpro”). (#4541)
  • Add basic optional sentry integration. (#4632#4694)
  • Transfer bans on room upgrade. (#4642)
  • Add configurable room list publishing rules. (#4647)
  • Support .well-known delegation when issuing certificates through ACME. (#4652)
  • Allow registration and login to be handled by a worker instance. (#4666#4670#4682)
  • Reduce the overhead of creating outbound federation connections over TLS by caching the TLS client options. (#4674)
  • Add prometheus metrics for number of outgoing EDUs, by type. (#4695)
  • Return correct error code when inviting a remote user to a room whose homeserver does not support the room version. (#4721)
  • Prevent showing rooms to other servers that were set to not federate. (#4746)

Bugfixes

  • Fix possible exception when paginating. (#4263)
  • The dependency checker now correctly reports a version mismatch for optional
    dependencies, instead of reporting the dependency missing. (#4450)
  • Set CORS headers on .well-known requests. (#4651)
  • Fix kicking guest users on guest access revocation in worker mode. (#4667)
  • Fix an issue in the database migration script where the
    e2e_room_keys.is_verified column wasn’t considered as
    a boolean. (#4680)
  • Fix TaskStopped exceptions in logs when outbound requests time out. (#4690)
  • Fix ACME config for python 2. (#4717)
  • Fix paginating over federation persisting incorrect state. (#4718)

Internal Changes

  • Run black to reformat user directory code. (#4635)
  • Reduce number of exceptions we log. (#4643#4668)
  • Introduce upsert batching functionality in the database layer. (#4644)
  • Fix various spelling mistakes. (#4657)
  • Cleanup request exception logging. (#4669#4737#4738)
  • Improve replication performance by reducing cache invalidation traffic. (#4671#4715#4748)
  • Test against Postgres 9.5 as well as 9.4. (#4676)
  • Run unit tests against python 3.7. (#4677)
  • Attempt to clarify installation instructions/config. (#4681)
  • Clean up gitignores. (#4688)
  • Minor tweaks to acme docs. (#4689)
  • Improve the logging in the pusher process. (#4691)
  • Better checks on newsfragments. (#4698#4750)
  • Avoid some redundant work when processing read receipts. (#4706)
  • Run push_receipts_to_remotes as background job. (#4707)
  • Add prometheus metrics for number of badge update pushes. (#4709)
  • Reduce pusher logging on startup (#4716)
  • Don’t log exceptions when failing to fetch remote server keys. (#4722)
  • Correctly proxy exception in frontend_proxy worker. (#4723)
  • Add database version to phonehome stats. (#4753)

Publishing the Backend Roadmap

Good people,

2019 is a big year for Matrix, in the next month we will have shipped:

  • Matrix spec 1.0 (including the first stable release of the Server to Server Spec)
  • Synapse 1.0
  • Riot 1.0

This is huge in itself, but is really only the beginning, and now we want to grow the ecosystem as quickly as possible. This means landing a mix of new features, enhancing existing ones, some big performance improvements as well as generally making life easier for our regular users, homeserver admins and community developers.

Today we are sharing the Matrix core team’s backend roadmap. The idea is that this will make it easier for anyone to understand where the project is going, what we consider to be important, and why.

To see the roadmap in its full glory, take a look here.

What is a roadmap and why is it valuable?

A roadmap is a set of high level projects that the team intend to work on and a rough sense of the relative priority. It is essential to focus on specific goals, which inevitably means consciously not working on other initiatives.

Our roadmap is not a delivery plan – there are explicitly no dates. The reason for this is that we know that other projects will emerge, developers will be needed to support other urgent initiatives, matrix.org use continues to grow exponentially and will require performance tweaking.

So simply, based on what we know now, this is the order we will work on our projects.

Why are we sharing it?

We already share our day to day todo list, and of course our commit history, but it can be difficult for a casual observer to see the bigger picture from such granular data. The purpose of sharing is that we want anyone from the community to understand where our priorities lie.

We are often asked ‘Why are you not working on X, it is really important’ where the answer is often ‘We agree that X is really important, but A, B and C are more important and must come first’.

The point of sharing the roadmap is to make that priority trade off more transparent and consumable.

How did we build it?

The core contributors to Synapse and Dendrite are 6 people, of 5 nationalities spread across 3 locations. After shipping the r0 release of the Server to Server spec last month we took some time to step back and have a think about what to do after Synapse 1.0 lands. This meant getting everyone in one place to talk it through.

We also had Ben (benpa) contribute from a community perspective and took input from speaking to so many of you at FOSDEM.

In the end we filled a wall with post-its, each post-it representing a sizeable project. The position of the post-it was significant in that the vertical axis being a sense of how valuable we thought the task would be, and the horizontal axis being a rough guess on how complex we considered it to be.

We found this sort of grid approach to be really helpful in determining relative priority.

After many hours and plenty of blood, sweat and tears we ended up with something we could live with and wrote it up in the shared board.

And this is written in blood right?

Not at all (it’s written in board marker). This is simply a way to express our plan of action and we are likely to make changes to it dynamically. However, this means that at any given moment, if someone wants to know what we are working on then the roadmap is the place to go.

But wait I want to know more!

Here is a video of myself and Matthew to talk you through the projects

Interesting, but I have questions …

Any feedback gratefully received, come and ask questions in #synapse or #dendrite or feel free to ping me direct at @neilj:matrix.org

 

Synapse 0.99.1.1 Released!

Hey, everyone, today is the day we release Synapse 0.99.1.1

This release contains improved ACME support to make it even easier to get going with TLS certs on your federation end points, plus some tweaks to make the room version upgrade path easier.

Just as a reminder that the 0.99.x series is precursor for our 1.0 release (which will land in early March, exact date to be confirmed) – it is really important that all server admins are aware that self signed certificates on the Server to Server API will no longer be accepted by >= Synapse 1.0. If you have not already done so, now is the time to configure your certificate. For more info see our FAQ and if you get stuck come and join us in #Synapse.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.

Synapse 0.99.1.1 Changelog

Bugfixes

  • Fix “TypeError: ‘>’ not supported” when starting without an existing certificate.
    Fix a bug where an existing certificate would be reprovisoned every day. (#4648)

Synapse 0.99.1 Changelog

Features

  • Include m.room.encryption on invites by default (#3902)
  • Federation OpenID listener resource can now be activated even if federation is disabled (#4420)
  • Synapse’s ACME support will now correctly reprovision a certificate that approaches its expiry while Synapse is running. (#4522)
  • Add ability to update backup versions (#4580)
  • Allow the “unavailable” presence status for /sync.
    This change makes Synapse compliant with r0.4.0 of the Client-Server specification. (#4592)
  • There is no longer any need to specify no_tls: it is inferred from the absence of TLS listeners (#4613#4615#4617#4636)
  • The default configuration no longer requires TLS certificates. (#4614)

Bugfixes

  • Copy over room federation ability on room upgrade. (#4530)
  • Fix noisy “twisted.internet.task.TaskStopped” errors in logs (#4546)
  • Synapse is now tolerant of the tls_fingerprints option being None or not specified. (#4589)
  • Fix ‘no unique or exclusion constraint’ error (#4591)
  • Transfer Server ACLs on room upgrade. (#4608)
  • Fix failure to start when not TLS certificate was given even if TLS was disabled. (#4618)
  • Fix self-signed cert notice from generate-config. (#4625)
  • Fix performance of user_ips table deduplication background update (#4626#4627)

Internal Changes

  • Change the user directory state query to use a filtered call to the db instead of a generic one. (#4462)
  • Reject federation transactions if they include more than 50 PDUs or 100 EDUs. (#4513)
  • Reduce duplication of synapse.app code. (#4567)
  • Fix docker upload job to push -py2 images. (#4576)
  • Add port configuration information to ACME instructions. (#4578)
  • Update MSC1711 FAQ to calrify .well-known usage (#4584)
  • Clean up default listener configuration (#4586)
  • Clarifications for reverse proxy docs (#4607)
  • Move ClientTLSOptionsFactory init out of refresh_certificates (#4611)
  • Fail cleanly if listener config lacks a ‘port’ (#4616)
  • Remove redundant entries from docker config (#4619)
  • README updates (#4621)

Further details on Critical Security Update in Synapse affecting all versions prior to 0.34.1 (CVE-2019-5885)

On Thursday Jan 10th we released a Critical Security Update (Synapse 0.34.0.1/0.34.1.1), which fixes a serious security bug in Synapse 0.34.0 and earlier.  Many deployments have now upgraded to 0.34.0.1 or 0.34.1.1, and we now consider it appropriate to disclose more information about the issue, to provide context and encourage the remaining affected servers to upgrade as soon as possible.

In Synapse 0.11 (Nov 2015) we added a configuration parameter called “macaroon_secret_key” which relates to our use of macaroons in authentication. Macaroons are authentication tokens which must be signed by the server which generates them, to prevent them being forged by attackers. “macaroon_secret_key” defines the key which is used for this signature, and it must therefore be kept secret to preserve the security of the server.

If the option is not set, Synapse will attempt to derive a secret key from other secrets specified in the configuration file. However, in all versions of Synapse up to and including 0.34.0, this process was faulty and a predictable value was used instead.

So if, your homeserver.yaml does not contain a macaroon_secret_key, you need to upgrade to 0.34.1.1 or 0.34.0.1 or Debian 0.34.0-3~bpo9+2 immediately to prevent the risk of account hijacking.

The vulnerability affects any Synapse installation which does not have a macaroon_secret_key setting. For example, the Debian and Ubuntu packages from Matrix.org, Debian and Ubuntu include a configuration file without an explicit macaroon_secret_key and must upgrade. Anyone who hasn’t updated their config since Nov 2015 or who grandfathered their config from the Debian/Ubuntu packages will likely also be affected.

We are not aware of this vulnerability being exploited in the wild, but if you are running an affected server it may still be wise to check your synapse’s user_ips database table for any unexpected access to your server’s accounts. You could also check your accounts’ device lists (shown under Settings in Riot) for unexpected devices, although this is not as reliable as an attacker could cover their tracks to remove unexpected devices.

We’ll publish a full post-mortem of the issue once we are confident that most affected servers have been upgraded.

We’d like to apologise for the inconvenience caused by this – especially to folks who upgraded since Friday who were in practice not affected.  Due to the nature of the issue we wanted to minimise details about the issue until people had a chance to upgrade. We also did not follow a planned disclosure procedure because Synapse 0.34.1 already unintentionally disclosed the existence of the bug by fixing it (causing the logout bug for affected users which led us to pull the original Synapse 0.34.1 release).

On the plus side, we are approaching the end of beta for Synapse, and going forwards hope to see much better stability and security across the board.

Thanks again for your patience,

The Matrix.org Team

 

Critical Security Update: Synapse 0.34.0.1/Synapse 0.34.1.1

After releasing Synapse v0.34.1, we have become aware of a security vulnerability affecting all previous versions (CVE-2019-5885). v0.34.1 closed the vulnerability but, in some cases, caused users to be logged out of their clients, so we do not recommend 0.34.1 for production use.

Today we release two mitigating versions v0.34.0.1 and v0.34.1.1. Both versions close the vulnerability and will not cause users to be logged out. All installations should be upgraded to one or other immediately.

  • Admins who would otherwise upgrade to v0.34.1 (or those that have already done so) should upgrade to v0.34.1.1.
  • Admins on v0.34.0, who do not wish to bring in new non-security related behaviour, should upgrade to v0.34.0.1.

You can get the new updates for v0.34.0.1 and v0.34.1.1 here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. See also our Synapse installation guide page.

We will publish more details of the vulnerability once admins have had a chance to upgrade. To our knowledge the vulnerability has not been exploited in the wild.

Many thanks for your patience, we are moving ever closer to Synapse reaching v1.0, and fixes like this one edge us ever closer.

Thanks also to the package maintainers who have coordinated with us to ensure distro packages are available for a speedy upgrade!