Wheyhey as I live and breathe 1.4.0 is here!
1.4.0 should be considered ‘The Privacy Release’ and is the cumulation of a huge body of work spanning multiple projects to improve data privacy in the matrix ecosystem. You can read more about the full project here.
While we consider 1.4.0 to be a huge leap forward in terms of data privacy, it is really important to note that it contains breaking changes.
If either apply to you failure to act will break your installation. Full details can be found in the upgrade notes.
It is possible for a user to associate an email address or phone number with their account, for a number of reasons:
Before an email address or phone number can be added to a user's account, or before such an address is used to carry out a password-reset, Synapse must confirm the operation with the owner of the email address or phone number. It does this by sending an email or text giving the user a link or token to confirm receipt. This process is known as '3pid verification'. ('3pid', or 'threepid', stands for third-party identifier, and we use it to refer to external identifiers such as email addresses and phone numbers.)
Previous versions of Synapse delegated the task of 3pid verification to an identity server by default. In most cases this server is vector.im or matrix.org.
In Synapse 1.4.0, for security and privacy reasons, the homeserver will no longer delegate this task to an identity server by default. Instead, the server administrator will need to explicitly decide how they would like the verification messages to be sent.
In the medium term, the vector.im and matrix.org identity servers will disable support for delegated 3pid verification entirely. However, in order to ease the transition, they will retain the capability for a limited period. Delegated email verification will be disabled on Monday 2nd December 2019 (giving roughly 2 months notice). Disabling delegated SMS verification will follow some time after that once SMS verification support lands in Synapse.
Once delegated 3pid verification support has been disabled in the vector.im and matrix.org identity servers, all Synapse versions that depend on those instances will be unable to verify email and phone numbers through them. There are no imminent plans to remove delegated 3pid verification from Sydent generally. (Sydent is the identity server project that backs the vector.im and matrix.org instances).
Prior to 1.4.0, the identity server was providing two related-but-separate functions:
a directory for users to publish their contact details and to look up their contacts by their email addresses and phone numbers. a "trusted third party communications network delegate", providing SMS- and email-sending functionality to all homeservers for registration and password reset.
The intention behind the identity server's providing 2. was one of convenience: to save homeserver admins the burden of configuring their homeservers to send email, as well as sparing us the effort of writing generic SMS-gateway integration that would allow any homeserver admin to configure their homeserver to send SMS via their chosen SMS aggregator.
By exposing this 'trusted email/sms delegate' functionality, however, and by including references to the New Vector identity servers in the default configuration for both Synapse and Riot, we created a situation in which users on non-New Vector run homeservers (who had never seen our privacy notice) could easily end up sharing their email addresses and phone numbers with a New Vector identity server.
Using identity servers for registration and password reset introduced yet further complexity - since password reset is a sensitive operation, it was important that the homeserver only use 'trusted' identity servers for this purpose (with the trust being configured by the homeserver admin in the homeserver config). But this created ambiguity over who was ultimately responsible for the choice of identity server and when they could make that choice - the client could present a default identity server at login/registration/password reset time, which the user could choose to override before logging in, but unless the identity server ultimately selected were on the homeserver admin's 'trusted identity servers' list, any identity server operations that were proxied via the homeserver would have been blocked by the homeserver. However, not all identity server operations initiated by the client were proxied via the homeserver - some were sent directly from the client to the identity server, and would not have been filtered.
The best way to solve this problem is to have individual homeservers take ownership for their own email and SMS needs. In the case of email this means configuring details of an appropriate SMTP server or continuing to delegate through an identity server that will allow it to do so. If a homeserver admin makes the active choice to use New Vector's identity servers for delegation (up until 1st December), they should make their own users aware through their privacy notice.
This delegation is entirely separate from the user's choice of identity server for user directory services. As of right now the user is free to choose and trust whichever identity server they wish, or to choose not to use an identity server at all.
Yes, 1.4.0 now automatically garbage collects redacted messages (defaults to 7 days) and removes unused IP and user agent information stored in the user_ips table (defaults to 30 days). Finally, Synapse now warns in its logs if you are using matrix.org as a trusted key server, in case you wish to use a different server to help discover other servers’ keys.
Aside from privacy, we’ve expanded our OpenTracing support and fixed a host of bugs. However the thing that is most exciting is switching on our solution for mitigating forward extremities build up’ by default.
In some cases rooms can accumulate ‘forward extremities’, which are simply an artefact of attempting to resolve the room state over multiple servers. Forward extremities are necessary to ensure that each server can independently arrive at the same view of the room eventually, however processing these extremities can be computationally expensive and degrade server performance overall.
Originally it was an experimental config option but we now feel confident to turn it on by default for all instances - it should make a big difference for the CPU of servers in fragmented rooms.
So that’s it folks, thanks for making it this far. As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Also, check out our Synapse installation guide page
The changelog since 1.3.1 follows:
client_secretin server logs. (#6158)
devicestable, and improve its performance on Postgres. (#6135)
Note that this release includes significant changes around 3pid verification. Administrators are reminded to review the upgrade notes.
trust_identity_server_for_password_resetsconfig option with
account_threepid_delegates, and make the
id_serverparameteter optional on
*/requestTokenendpoints, as per MSC2263. (#5876, #5969, #6028)
/lookupAPI where available, with fallback to v1. (Implements MSC2134 plus
id_access_token authenticationfor v2 Identity Service APIs from MSC2140). (#5897)
/registerala MSC2140. (#5964)
/versionsas per MSC2264. (#5974)
POST /_matrix/client/unstable/account/3pid/unbindendpoint from MSC2140 for unbinding a 3PID from an identity server without removing it from the homeserver user account. (#5980, #6062)
account_threepid_delegate.msisdnfor validating threepid sessions. (#6011)
/account/3pid/bindas per MSC2290. (#6043)
bindparameter from Client Server POST
/accountendpoint as per MSC2290. (#6067)
POST /add_threepid/msisdn/submit_tokenendpoint for proxying submitToken on an
submit_urlresponse parameter to
m.require_identity_serverflag to /version's unstable_features. (#5972)
synapse_federation_known_servers: represents the total number of servers your server knows about (i.e. is in rooms with), including itself. Enable by setting
metrics_flags.known_serversto True in the configuration.(#5981)
synapse_build_info: exposes the Python version, OS version, and Synapse version of the running server. (#6005)
report_stats_endpointoption to configure where stats are reported to, if enabled. Contributed by @Sorunome. (#6012)
/register/availableif registration has been disabled. (#6082)
trusted_key_serversconfig field configured. (#6090)
power_level_content_override.usersdoes not contain
password_reset_success_template, when they are actually
public_baseurl. Thanks to @aaronraimist for the fix! (#5909)
falseand the thumbnail was dynamically generated. Fix reported by rkfg. (#5915)
sidis not provided to
federation_certificate_verification_whitelistnow will not cause
TypeErrorsto be raised (a regression in 1.3). Additionally, it now supports internationalised domain names in their non-canonical representation. (#5996)
submit_tokenendpoint until we implement
users_set_deactivated_flagbackground update. (#6092)
next_linkfor email validation. (#6097)
UID/GIDif they are already correct. (#5970)
SYNAPSE_WORKERenvvar to specify python module. (#6058)
INSTALL.mdto say that Python 2 is no longer supported. (#5953)
/_matrix/client/r0/registerendpoint. Contributed by Awesome Technologies Innovationslabor GmbH. (#5877)
users_in_public_roomsto improve the performance of directory queries. (#5894)
failure_tscolumn to the
destinationsdatabase table. (#6016, #6072)