Category – General
143 posts tagged with "General" (See all categories)

Tightening up privacy in Matrix

2019-06-30 — General — Matthew Hodgson

Hi all,

A few weeks ago there was some discussion around the privacy of typical Matrix configurations, particularly how Riot's default config uses vector.im as an Identity Server (for discovering users on Matrix by their email address or phone number) and scalar.vector.im as an Integration Manager (i.e. the mechanism for adding hosted bots/bridges/widgets into rooms). This means that Riot, even if using a custom homeserver and running from a custom Riot deployment, will try to talk to *.vector.im (run by New Vector; the company formed by the core Matrix team in 2017) for some operations unless an alternative IS or IM has been specified in the config.

We haven't done as good a job at explaining this as we could have, and this blog post is a progress update on how we're fixing that and improving other privacy considerations in general.

Firstly, the reason Riot is configured like this is for the user's convenience: in general, we believe most users just want to discover other people on Matrix as easily as possible, and a logically-centralised server for looking up user matrix IDs by email/phone number (called third party IDs, or 3PIDs) is the only comprehensive way of doing so. Decentralising this data while protecting the privacy of the 3PIDs and their matrix IDs is a Hard Problem which we're unaware of anyone having solved yet. Alternatively, you could run a local identity server, but it will end up having to delegate to a centralised identity server anyway for IDs it has no other way to know about. Similarly, providing a default integration server that just works out of the box (rather than mandating the user configures their own) is a matter of trying to keep Riot's UX simple, especially when onboarding users, and especially given Riot's reputation for complexity at the best of times.

That said, the discussion highlighted some areas for improvement. Specifically:

  1. When doing work on making Matrix GDPR compliant back in May 2018, we set up a single privacy policy for the services we run, and got users to agree to it by locking them out of the matrix.org homeserver until they did. However, we missed that users not on the matrix.org homeserver might still be using our Identity Service (IS) & Integration Manager (IM) without accepting the privacy policy. Over the last few weeks we've been working on addressing this - it turns out that it's a pain to fix, given the Identity Service doesn't have the concept of users, so tracking which users have agreed to the policy policy or not means some fairly major changes. The current proposal is to change the Identity Service to use a form of OpenID to authenticate users against their homeserver in order to check they've accepted the IS's terms of use - see MSC2140 for the gory details.

Meanwhile, Riot is being updated to prompt the user to accept the IS & IM terms of use (if different to the HS's), and thus make it crystal clear to the user that they are using an IS & IM and that they have the option not to if desired - see https://github.com/vector-im/riot-web/issues/10167 and associated issues. This includes also explicitly prompting the user as to whether they want 3PIDs they provide at registration to be discoverable, as per https://github.com/vector-im/riot-web/issues/10091.

  1. Riot on iOS & Android gives the option of scanning your local addressbook to discover which of your contacts are on Matrix. The wording explaining this wasn't clear enough on Android - which we promptly fixed. Separately, the contact details sent to the server are currently not obfuscated. This is partially because we hadn't got to it, and partially because obfuscating them doesn't actually help much with privacy, given an attacker can just scan through possible obfuscated phone numbers and email addresses to deobfuscate them. However, we've been working through obfuscating the contact details anyway by hashing as per MSC2134, which has all the details. We're also adding an explicit lookup warning in Riot/Web, as per https://github.com/vector-im/riot-web/issues/10093.

  2. There was a bug where Riot/Web was querying the Integration Manager every time you opened a room, even if that room had no integrations (actually, it did it 3 times in a row). This got fixed and released in Riot/Web 1.2.2 back on June 19th.

  3. Matrix needs to authenticate whether events were actually sent by the server that claimed to send them. We do this by having servers sign their events when they create them, and publishing the public half of their signing keys for anyone to query. However, this poses problems if you receive an event which is signed by a server which isn't currently online. To solve this, we have the concept of trusted_key_servers (aka notary servers), which your server can query to see if they know about the missing server's keys. By default, matrix.org is configured as Synapse's trusted notary, but you can of course change this. If you choose an unreliable server as the notary (e.g. by not setting one at all) then there's a risk that you won't be able to look up signing keys, and a splitbrain will result where your server can't receive certain events, but other servers in the room can. This can then result in your server being unable to participate in the room entirely, if it's missing key events in the room's lifetime.

    Our plan here is to get rid of notaries entirely by changing how event signing works as per MSC1228, but this is going to take a while. Meanwhile we're going to check Synapse's code to ensure it doesn't talk to the notary server unnecessarily. (E.g. it should be caching the signing keys locally, and it should only use the notary server if the remote server is down.)

  4. When doing VoIP in Matrix, clients need to use a TURN server to discover their network conditions and perform firewall traversal. The TURN server should be specified by your homeserver (and each homeserver deployment should ideally include a TURN server). However, for users who have not configured a TURN server, Riot (on all 3 platforms) defaulted to use Google's public STUN service (stun.l.google.com). STUN is a subset of TURN which provides firewall discovery, but not traffic relaying. This slightly increased the chances of calls working for users without a proper TURN server, but not by much - and rather than fall back to Google, we've decided to simply remove it from Riot (e.g. https://github.com/matrix-org/matrix-ios-sdk/commit/24832a2b14fb72ae6f051d5aba40262d11eef65d). This means that VoIP might get less reliable for users who were relying on this fallback, but you really should be running your own TURN server anyway if you want VoIP to work reliably on your homeserver.

  5. We should make it clearer in Riot that device names are world-readable, and not just for the user's own personal reference. This is https://github.com/vector-im/riot-web/issues/10216

As you can see, much of the work on improving these issues is still in full swing, although some has already shipped. As should also be obvious, these issues are categorically not malicious: Matrix (and Riot) literally exists to give users full control and autonomy over their communication, and privacy is a key part of that. These are avoidable issues which can and will be solved. It's worth noting that we have to prioritise privacy issues alongside all the other development in Matrix however: there's no point in having excellent privacy if there are other bugs stopping the platform from being usable.

We'll do another blog post to confirm once most of the fixes here have landed - meanwhile, hopefully this post provides some useful visibility on how we're going about improving things.

Introducing Matrix 1.0 and the Matrix.org Foundation

2019-06-11 — General — Matthew Hodgson

Matrix 1.0

Hi all,

We are very excited to announce the first fully stable release of the Matrix protocol and specification across all APIs - as well as the Synapse 1.0 reference implementation which implements the full Matrix 1.0 API surface.

This means that after just over 5 years since the initial work on Matrix began, we are proud to have finally exited beta!! This is the conclusion of the work which we announced at FOSDEM 2019 when we cut the first stable release of the Server-Server API and began the Synapse 0.99 release series in anticipation of releasing a 1.0.

Now, before you get too excited, it’s critical to understand that Matrix 1.0 is all about providing a stable, self-consistent, self-contained and secure version of the standard which anyone should be able to use to independently implement production-grade Matrix clients, servers, bots and bridges etc. It does not mean that all planned or possible features in Matrix are now specified and implemented, but that the most important core of the protocol is a well-defined stable platform for everyone to build on.

On the Synapse side, our focus has been exclusively on ensuring that Synapse correctly implements Matrix 1.0, to provide a stable and secure basis for participating in Matrix without risk of room corruption or other nastinesses. However, we have deliberately not focused on performance or features in the 1.0 release - so I’m afraid that synapse’s RAM footprint will not have got significantly better, and your favourite long-awaited features (automatically defragmenting rooms with lots of forward extremities, configurable message retention, admin management web-interface etc) have not yet landed. In other words, this is the opposite of the Riot 1.0 release (where the entire app was redesigned and radically improved its performance and UX) - instead, we have adopted the mantra to make it work, make it work right, and then (finally) make it fast. You can read the full release notes here. It’s also worth looking at the full changelog through the Synapse 0.99 release series to see the massive amount of polishing that’s been going on here.

All this means that the main headline features which land in Matrix 1.0 are vitally important but relatively dry:

  • Using X.509 certificates to trust servers rather than perspective notaries, to simplify and improve server-side trust. This is a breaking change across Matrix, and we’ve given the community several months now to ensure their homeservers run a valid TLS certificate. See MSC1711 for full details, and the 2 week warning we gave. As of ~9am UTC today, the matrix.org homeserver is running Synapse 1.0 and enforcing valid TLS certificates - the transition has begun (and so far we haven’t spotted any major breakage :). Thank you to everyone who got ready in advance!
  • Using .well-known URIs to discover servers, in case you can’t get a valid TLS certificate for your server’s domain.
  • Switching to room version 4 by default for creating new rooms. This fixes the most important defects that the core room algorithm has historically encountered, particularly:
  • Specifying the ability to upgrade between room versions
  • Full specification of lazy loading room members
  • Short Authentication String (Emoji!) interactive verification of E2EE devices
  • ...and lots and lots and lots of bugfixes and spec omission fixes.

That said, there is a lot of really exciting stuff in flight right now which sadly didn’t stabilise in time for Matrix 1.0, but will be landing as fast as we can finalise it now that 1.0 is at last out the door. This includes:

  • Editable messages! (These are in Synapse 1.0 and Riot already, but still stabilising so not enabled by default)
  • Reactions! (Similarly these are in develop)
  • Threading!! (We’ve planted the seeds for this in the new ‘aggregations’ support which powers edits & reactions - but full thread support is still a bit further out).
  • Cross-signed verification for end-to-end encryption (This is on a branch, but due to land any day now). We’ve also held off merging E2E backups into the Matrix 1.0 spec until cross-signing lands, given it may change the backup behaviour a bit. Once this is done, we can seriously talk about turning on E2E by default everywhere.
  • Live-tracking of room statistics and state in Synapse! (This is in Synapse 1.0 already if you check out the new room_stats and room_state tables, but we need to provide a nice admin interface for it).
  • Support for smaller footprint homeservers by reducing memory usage and stopping them from joining overly complex rooms.

Then stuff which we haven’t yet started, but is now unlocked by the 1.0 release:

  • Fixing extremities build-up (and so massively improving performance)
  • Rewriting Communities. Groups/Communities deliberately didn’t land in Matrix 1.0 as the current implementation has issues we want to fix first. MSC1772 has the details.
  • Rewritten room directory using the new room stats/state tables to be super-speedy.
  • Super-speedy incremental state resolution
  • Removing MXIDs from events (MSC1228)

Just to give a quick taster of the shape of things to come, here’s RiotX/Android, the all-new Riot client for Android, showing off Edits & Reactions in the wild…



...and here’s a screenshot of the final test jig for cross-signing devices in end-to-end encryption, so you will never have to manually verify new devices for a trusted user ever again! We demoed a *very* early version of this at FOSDEM, but this here is the testing harness for real deal, after several iterations of the spec and implementation to nail down the model. + means the device/user's cross-signing key is trusted, T means it's TOFU:


So, there you have it - welcome to Matrix 1.0, and we look forward to our backlog of feature work now landing!

Massive massive thanks to everyone who has stuck with the project over the years and helped support and grow Matrix - little did we think back in May 2014 that it’d take us this long to exit beta, but hopefully you’ll agree that it’s been worth it :)

Talking of which, we were looking through the photos we took from the first ever session hacking on Matrix back in May 2014…

Whiteboard 1

...suffice it to say that of the architectural options, we went with #3 in the end...

Whiteboard 2

...and that nowadays we actually know how power levels work, in excruciating and (hopefully) well-specified detail :)

There has been an absolutely enormous amount of work to pull Matrix 1.0 together - both on the spec side (thanks to the Spec Core Team for corralling proposals, and everyone who's contributed proposals, and particularly to Travis for editing it all) and the implementation side (thanks to the whole Synapse team for the tedious task of cleaning up everything that was needed for 1.0). And of course, huge thanks go to everyone who has been helping test and debug the Synapse 1.0 release candidates, or just supporting the project to get to this point :)

The Matrix.org Foundation

Finally, as promised, alongside Matrix 1.0, we are very happy to announce the official launch of the finalised Matrix.org Foundation!

This has been a long-running project to ensure that Matrix’s future is governed by a neutral non-profit custodian for the benefit of everyone in the Matrix ecosystem. We started the process nearly a year ago back with the initial proposal Towards Open Governance of Matrix.org, and then legally incorporated the Foundation in October, and published the final governance proposal in January.

As of today the Foundation is finalised and operational, and all the assets for Matrix.org have been transferred from New Vector (the startup we formed in 2017 to hire the core Matrix team). In fact you may already have seen Matrix.org Foundation notices popping up all over the Matrix codebase (as all of New Vector’s work on the public Matrix codebase for the forseeable is being assigned to the Matrix.org Foundation).

Most importantly, we’re excited to introduce the Guardians of the Matrix.org Foundation. The Guardians are the legal directors of the non-profit Foundation, and are responsible for ensuring that the Foundation (and by extension the Spec Core Team) keeps on mission and neutrally protects the development of Matrix. Guardians are typically independent of the commercial Matrix ecosystem and may even not be members of today’s Matrix community, but are deeply aligned with the mission of the project. Guardians are selected to be respected and trusted by the wider community to uphold the guiding principles of the Foundation and keep the other Guardians honest.

We have started the Foundation with five Guardians - two being the original founders of the Matrix project (Matthew and Amandine) and three being entirely independent, thus ensuring the original Matrix team forms a minority which can be kept in check by the rest of the Guardians. The new Guardians are:

  • Prof. Jon Crowcroft - Marconi Professor of Communications Systems in the Computer Lab at the University of Cambridge and the Turing Institute. Jon is a pioneer in the field of decentralised communication, and a fellow of the Royal Society, the ACM, the British Computer Society, the Institution of Engineering and Technology, the Royal Academy of Engineering and the Institute of Electrical and Electronics Engineers.

    Jon is a global expert in decentralisation and data privacy, and is excellently placed to help ensure Matrix stays true to its ideals.

  • Ross Schulman - Ross is a senior counsel and senior policy technologist at New America’s Open Technology Institute, where he focuses on internet measurement, emerging technologies, surveillance, and decentralization. Prior to joining OTI, Ross worked for Google.

    Ross brings a unique perspective as a tech- and decentralisation-savvy lawyer to the Foundation, as well as being one of the first non-developers in the Matrix community to run his own homeserver. Ross has been known to walk around Mozfest clutching a battery-powered Synapse in a box, promoting decentralised communication for all.

  • Dr. Jutta Steiner - As co-founder and CEO of Parity Technologies, Jutta is dedicated to building a better internet - Web 3.0 - where users’ privacy & control come first. Parity Technologies is a leader in the blockchain space – known to many as the creator of one of the most popular Ethereum clients, it is also the creator of two ambitious new blockchain technlogies, Polkadot and Substrate, that make it easier to experiment and innovate on scalability, encryption and governance.

    Parity has been pioneering Matrix enterprise use since the moment they decided to rely on Matrix for their internal and external communication back in 2016, and now run their own high-volume deployment, with end-to-end encryption enabled by default. Jutta represents organisations who are professionally dependent on Matrix day-to-day, as well as bringing her unique experiences around decentralisation and ensuring that Web 3.0 will be a fair web for all.

We’d like to offer a very warm welcome to the new Guardians, and thank them profusely for giving up their time to join the Foundation and help ensure Matrix stays on course for the years to come.

For the full update on the Foundation, please check out the new website content at https://matrix.org/foundation which should tell you everything you could possibly want to know about the Foundation, the Guardians, the Foundation’s legal Articles of Association, and the day-to-day Rules which define the Open Governance process.

And finally…

Matrix 1.0 has been a bit of an epic to release, but puts us on a much stronger footing for the future.

However, it’s very unlikely that we’d have made it this far if most of the core dev team wasn’t able to work on Matrix as their day job. Right now we are actively looking for large-scale donations to the Matrix.org Foundation (and/or investment in New Vector) to ensure that the team can maintain as tight a focus on core Matrix work as possible, and to ensure the project realises its full potential. While Matrix is growing faster than ever, this perversely means we have more and more distractions - whether that’s keeping the Matrix.org server safe and operational, or handling support requests from the community, or helping new members of the ecosystem get up and running. If you would like Matrix to succeed, please get in touch if you’d like to sponsor work, prioritise features, get support contracts, or otherwise support the project. We’re particularly interested in sponsorship around decentralised reputation work (e.g. publishing a global room directory which users can filter based on their preferences).

Finally, huge thanks to everyone who has continued to support us through thick and thin on Patreon, Liberapay or other platforms. Every little helps here, both in terms of practically keeping the lights on, and also inspiring larger donations & financial support.

So: thank you once again for flying Matrix. We hope you enjoy 1.0, and we look forward to everything else landing on the horizon!

- Matthew, Amandine & the whole Matrix.org Team.

Synapse 1.0.0 released

2019-06-11 — General — Neil Johnson

Well here it is: Synapse 1.0.

Synapse 1.0 is the reference implementation of the Matrix 1.0 spec. The goal of the release overall has been to focus on security and stability, such that we can officially declare Synapse (and Matrix) out of beta and recommended for production use. This means changing the default room protocol version used for new rooms to be v4, which includes the new state resolution algorithm, as well as collision-resistant event IDs, which are now formatted to be URL safe.

Synapse 1.0 also ships with support for the upcoming v5 room protocol (which enforces honouring server key validity periods), but this will not be used as the default for new rooms until a sufficient number of servers support it.

Please note that Synapse 1.0 does not include significant performance work or new features - our focus has been almost exclusively on providing a reference implementation of the Matrix 1.0 protocol. But having cleared our backlog on security/stability issues we will finally be now unblocked to pursue work around reducing RAM footprint, eliminating forward-extremity build up, and shipping new features like Edits, Reactions & E2E cross-signing support.

As part of the security work, Synapse 1.0 contains a breaking change that requires a valid TLS certificate on the federation API endpoint. Servers that do not configure their certificate will no longer be able to federate post 1.0.

It is also worth noting that Synapse 1.0.0 is the last release that will support Python 2.x and Postgres 9.4. For more information see here but the TL;DR is that you should upgrade asap.

This release has been a long time coming. Many thanks indeed to everyone who helped test the release candidates and provided feedback along the way.

Synapse 1.0 is just one component of a larger Matrix 1.0 release, which you can read all about here.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our Synapse installation guide page

The changelog since 0.99.5 follows:

Synapse 1.0.0 (2019-06-11)

Bugfixes

  • Fix bug where attempting to send transactions with large number of EDUs can fail. (#5418)

Improved Documentation

  • Expand the federation guide to include relevant content from the MSC1711 FAQ (#5419)

Internal Changes

  • Move password reset links to /_matrix/client/unstable namespace. (#5424)

Synapse 1.0.0rc3 (2019-06-10)

Security: Fix authentication bug introduced in 1.0.0rc1. Please upgrade to rc3 immediately

Synapse 1.0.0rc2 (2019-06-10)

Bugfixes

  • Remove redundant warning about key server response validation. (#5392)
  • Fix bug where old keys stored in the database with a null valid until timestamp caused all verification requests for that key to fail. (#5415)
  • Fix excessive memory using with default federation_verify_certificates: true configuration. (#5417)

Synapse 1.0.0rc1 (2019-06-07)

Features

  • Synapse now more efficiently collates room statistics. (#4338, #5260, #5324)
  • Add experimental support for relations (aka reactions and edits). (#5220)
  • Ability to configure default room version. (#5223, #5249)
  • Allow configuring a range for the account validity startup job. (#5276)
  • CAS login will now hit the r0 API, not the deprecated v1 one. (#5286)
  • Validate federation server TLS certificates by default (implements MSC1711). (#5359)
  • Update /_matrix/client/versions to reference support for r0.5.0. (#5360)
  • Add a script to generate new signing-key files. (#5361)
  • Update upgrade and installation guides ahead of 1.0. (#5371)
  • Replace the perspectives configuration section with trusted_key_servers, and make validating the signatures on responses optional (since TLS will do this job for us). (#5374)
  • Add ability to perform password reset via email without trusting the identity server. (#5377)
  • Set default room version to v4. (#5379)

Bugfixes

  • Fixes client-server API not sending "m.heroes" to lazy-load /sync requests when a rooms name or its canonical alias are empty. Thanks to @dnaf for this work! (#5089)
  • Prevent federation device list updates breaking when processing multiple updates at once. (#5156)
  • Fix worker registration bug caused by ClientReaderSlavedStore being unable to see get_profileinfo. (#5200)
  • Fix race when backfilling in rooms with worker mode. (#5221)
  • Fix appservice timestamp massaging. (#5233)
  • Ensure that server_keys fetched via a notary server are correctly signed. (#5251)
  • Show the correct error when logging out and access token is missing. (#5256)
  • Fix error code when there is an invalid parameter on /_matrix/client/r0/publicRooms (#5257)
  • Fix error when downloading thumbnail with missing width/height parameter. (#5258)
  • Fix schema update for account validity. (#5268)
  • Fix bug where we leaked extremities when we soft failed events, leading to performance degradation. (#5274, #5278, #5291)
  • Fix "db txn 'update_presence' from sentinel context" log messages. (#5275)
  • Fix dropped logcontexts during high outbound traffic. (#5277)
  • Fix a bug where it is not possible to get events in the federation format with the request GET /_matrix/client/r0/rooms/{roomId}/messages. (#5293)
  • Fix performance problems with the rooms stats background update. (#5294)
  • Fix noisy 'no key for server' logs. (#5300)
  • Fix bug where a notary server would sometimes forget old keys. (#5307)
  • Prevent users from setting huge displaynames and avatar URLs. (#5309)
  • Fix handling of failures when processing incoming events where calling /event_auth on remote server fails. (#5317)
  • Ensure that we have an up-to-date copy of the signing key when validating incoming federation requests. (#5321)
  • Fix various problems which made the signing-key notary server time out for some requests. (#5333)
  • Fix bug which would make certain operations (such as room joins) block for 20 minutes while attemoting to fetch verification keys. (#5334)
  • Fix a bug where we could rapidly mark a server as unreachable even though it was only down for a few minutes. (#5335, #5340)
  • Fix a bug where account validity renewal emails could only be sent when email notifs were enabled. (#5341)
  • Fix failure when fetching batches of events during backfill, etc. (#5342)
  • Add a new room version where the timestamps on events are checked against the validity periods on signing keys. (#5348, #5354)
  • Fix room stats and presence background updates to correctly handle missing events. (#5352)
  • Include left members in room summaries' heroes. (#5355)
  • Fix federation_custom_ca_list configuration option. (#5362)
  • Fix missing logcontext warnings on shutdown. (#5369)

Improved Documentation

  • Fix docs on resetting the user directory. (#5282)
  • Fix notes about ACME in the MSC1711 faq. (#5357)

Internal Changes

  • Synapse will now serve the experimental "room complexity" API endpoint. (#5216)
  • The base classes for the v1 and v2_alpha REST APIs have been unified. (#5226, #5328)
  • Simplifications and comments in do_auth. (#5227)
  • Remove urllib3 pin as requests 2.22.0 has been released supporting urllib3 1.25.2. (#5230)
  • Preparatory work for key-validity features. (#5232, #5234, #5235, #5236, #5237, #5244, #5250, #5296, #5299, #5343, #5347, #5356)
  • Specify the type of reCAPTCHA key to use. (#5283)
  • Improve sample config for monthly active user blocking. (#5284)
  • Remove spurious debug from MatrixFederationHttpClient.get_json. (#5287)
  • Improve logging for logcontext leaks. (#5288)
  • Clarify that the admin change password API logs the user out. (#5303)
  • New installs will now use the v54 full schema, rather than the full schema v14 and applying incremental updates to v54. (#5320)
  • Improve docstrings on MatrixFederationClient. (#5332)
  • Clean up FederationClient.get_events for clarity. (#5344)
  • Various improvements to debug logging. (#5353)
  • Don't run CI build checks until sample config check has passed. (#5370)
  • Automatically retry buildkite builds (max twice) when an agent is lost. (#5380)

Synapse 0.99.5.2 released

2019-05-30 — General — Neil Johnson

0.99.5.2 contains a critical performance fix following a regression that was introduced in 0.99.5. Affected servers will have experienced increased CPU and RAM usage with a knock on effect of generally sluggish performance.

Separately, we are also looking into reports relating to further performance degradations that may have been introduced as part of 0.99.5, though consider the 0.99.5.2 fix to be a significant improvement on previous 0.99.5.x releases.

Please upgrade asap.

You can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our Synapse installation guide page.

Synapse v0.99.5.2 Changelog

Bugfixes

  • Fix bug where we leaked extremities when we soft failed events, leading to performance degradation. (#5274, #5278, #5291)

Final countdown to 1.0

2019-05-24 — General — Matthew Hodgson

Hi all,

After lots of refinements, polishing and a few distractions we’re finally at the point of announcing the final timeline for both Matrix 1.0 and Synapse 1.0! We are targeting Monday 10th June as our release date - please consider this your two week warning!

This is the end game of the process we began back in February when we released the first stable release of the Server-Server API at FOSDEM, and started the Synapse 0.99 release series to prepare for 1.0.

Matrix 1.0 refers to the upcoming set of API releases which provides a matched set of stable and secure APIs across all of Matrix - at which point the project (at last) exits beta! In practice, this will be Client-Server API 0.5 (including final membership lazy loading, E2E backups and interactive verification and lots more), SS API 0.2 (including server key validity period fixes and associated v5 room protocol) and any other spec updates. The next 2 weeks will see a flurry of spec activity as we get everything together - you can see the full list and track the progress for the CS 0.5 spec release at https://github.com/matrix-org/matrix-doc/projects/2.

Meanwhile, Synapse 1.0 will be the reference implementation of Matrix 1.0, and so makes the changes required to implement Matrix 1.0 and close all currently known security and stability issues and thus exit beta. This means changing the default room protocol version used for new rooms to be v4, which includes the new state resolution algorithm, as well as collision-resistant event IDs, which are now formatted to be URL safe. Support for v4 rooms shipped in Synapse 0.99.5.1, so please upgrade asap to 0.99.5.1 before 1.0 is released to ease the transition.. Synapse 1.0 will also ship with support for the upcoming v5 room protocol (which enforces honouring server key validity periods), but this will not used as the default for new rooms until sufficient servers are speaking Matrix 1.0.

As part of the security work, Matrix 1.0 and Synapse 1.0 also contains a breaking change that requires a valid TLS certificate on the federation API endpoint. Servers that do not configure their certificate will no longer be able to federate post 1.0

You can check that your server has been correctly configured here and see here for more info on what you need to do. If in doubt head to #synapse:matrix.org.

We've been tracking readiness for the certificate change at https://arewereadyyet.com, at the time of writing 68% of active servers on the federation have valid certificates. We obviously would want that number to be higher, however since the largest installations have upgraded the total number of users who are ready for 1.0 stands at 96%, which we consider to be high enough to release 1.0.

This is not a drill, from here until 10th June we need everyone to not only ensure that their own server is ready, but also to encourage their fellow admins to update as well. With your help we can get everyone over the line!

Thanks everyone for your help to date, especially those providing support in #synapse:matrix.org.

Onwards!

Synapse 0.99.5.1 released!

2019-05-21 — General — Neil Johnson

Okay folks, this is an important one. v0.99.5.1 will be the last release before we ship Synapse v1.0. It is really important that you upgrade to v0.99.5.1 because it implements rooms version 4 - which is the room version that Synapse 1.0 will default to.

This means that Synapse 1.0 servers will create new rooms as version 4 by default and servers that have not upgraded to at least v0.99.5.1 will not be able to join those rooms.

Over the coming days we will announce a release day for Synapse v1.0, the idea is to give admins 2 weeks notice so that anyone yet to configure their federation SSL certificate has time to do so. This is important, failure to configure your certs will mean not being able to federate with v1.0 servers. If you are not sure if you certs are valid, you can test here and read here for more info on what to do.

Aside from room v4, this release also includes the ability to blacklist specific IPs from federating as well as experimental support for edits and reactions. We are not quite ready to mark the feature 'done done', but it is very close. Watch out for news as the feature lands properly.

We're really close to v1.0 now, give us a few more days and we'll announce an official release date.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our Synapse installation guide page

Synapse v0.99.5.1 Changelog (since v0.99.4)

Features

  • Add ability to blacklist IP ranges for the federation client. (#5043)
  • Ratelimiting configuration for clients sending messages and the federation server has been altered to match login ratelimiting. The old configuration names will continue working. Check the sample config for details of the new names. (#5181)
  • Drop support for the undocumented /_matrix/client/v2_alpha API prefix. (#5190)
  • Add an option to disable per-room profiles. (#5196)
  • Stick an expiration date to any registered user missing one at startup if account validity is enabled. (#5204)
  • Add experimental support for relations (aka reactions and edits). (#5209, #5211, #5203, #5212)
  • Add a room version 4 which uses a new event ID format, as per MSC2002. (#5210, #5217)

Bugfixes

  • Fix image orientation when generating thumbnails (needs pillow>=4.3.0). Contributed by Pau Rodriguez-Estivill. (#5039)
  • Exclude soft-failed events from forward-extremity candidates: fixes "No forward extremities left!" error. (#5146)
  • Re-order stages in registration flows such that msisdn and email verification are done last. (#5174)
  • Fix 3pid guest invites. (#5177)
  • Fix a bug where the register endpoint would fail with M_THREEPID_IN_USE instead of returning an account previously registered in the same session. (#5187)
  • Prevent registration for user ids that are too long to fit into a state key. Contributed by Reid Anderson. (#5198)
  • Fix incompatibility between ACME support and Python 3.5.2. (#5218)
  • Fix error handling for rooms whose versions are unknown. (#5219)

Internal Changes

  • Make /sync attempt to return device updates for both joined and invited users. Note that this doesn't currently work correctly due to other bugs. (#3484)
  • Update tests to consistently be configured via the same code that is used when loading from configuration files. (#5171, #5185)
  • Allow client event serialization to be async. (#5183)
  • Expose DataStore._get_events as get_events_as_list. (#5184)
  • Make generating SQL bounds for pagination generic. (#5191)
  • Stop telling people to install the optional dependencies by default. (#5197)
PreviousPage 1
NextPage 3