Wheyhey as I live and breathe 1.4.0 is here!
1.4.0 should be considered ‘The Privacy Release’ and is the cumulation of a huge body of work spanning multiple projects to improve data privacy in the matrix ecosystem. You can read more about the full project here.
While we consider 1.4.0 to be a huge leap forward in terms of data privacy, it is really important to note that it contains breaking changes.
If either apply to you failure to act will break your installation. Full details can be found in the upgrade notes.
It is possible for a user to associate an email address or phone number with their account, for a number of reasons:
Before an email address or phone number can be added to a user's account, or before such an address is used to carry out a password-reset, Synapse must confirm the operation with the owner of the email address or phone number. It does this by sending an email or text giving the user a link or token to confirm receipt. This process is known as '3pid verification'. ('3pid', or 'threepid', stands for third-party identifier, and we use it to refer to external identifiers such as email addresses and phone numbers.)
Previous versions of Synapse delegated the task of 3pid verification to an identity server by default. In most cases this server is vector.im or matrix.org.
In Synapse 1.4.0, for security and privacy reasons, the homeserver will no longer delegate this task to an identity server by default. Instead, the server administrator will need to explicitly decide how they would like the verification messages to be sent.
In the medium term, the vector.im and matrix.org identity servers will disable support for delegated 3pid verification entirely. However, in order to ease the transition, they will retain the capability for a limited period. Delegated email verification will be disabled on Monday 2nd December 2019 (giving roughly 2 months notice). Disabling delegated SMS verification will follow some time after that once SMS verification support lands in Synapse.
Once delegated 3pid verification support has been disabled in the vector.im and matrix.org identity servers, all Synapse versions that depend on those instances will be unable to verify email and phone numbers through them. There are no imminent plans to remove delegated 3pid verification from Sydent generally. (Sydent is the identity server project that backs the vector.im and matrix.org instances).
Prior to 1.4.0, the identity server was providing two related-but-separate functions:
a directory for users to publish their contact details and to look up their contacts by their email addresses and phone numbers. a "trusted third party communications network delegate", providing SMS- and email-sending functionality to all homeservers for registration and password reset.
The intention behind the identity server's providing 2. was one of convenience: to save homeserver admins the burden of configuring their homeservers to send email, as well as sparing us the effort of writing generic SMS-gateway integration that would allow any homeserver admin to configure their homeserver to send SMS via their chosen SMS aggregator.
By exposing this 'trusted email/sms delegate' functionality, however, and by including references to the New Vector identity servers in the default configuration for both Synapse and Riot, we created a situation in which users on non-New Vector run homeservers (who had never seen our privacy notice) could easily end up sharing their email addresses and phone numbers with a New Vector identity server.
Using identity servers for registration and password reset introduced yet further complexity - since password reset is a sensitive operation, it was important that the homeserver only use 'trusted' identity servers for this purpose (with the trust being configured by the homeserver admin in the homeserver config). But this created ambiguity over who was ultimately responsible for the choice of identity server and when they could make that choice - the client could present a default identity server at login/registration/password reset time, which the user could choose to override before logging in, but unless the identity server ultimately selected were on the homeserver admin's 'trusted identity servers' list, any identity server operations that were proxied via the homeserver would have been blocked by the homeserver. However, not all identity server operations initiated by the client were proxied via the homeserver - some were sent directly from the client to the identity server, and would not have been filtered.
The best way to solve this problem is to have individual homeservers take ownership for their own email and SMS needs. In the case of email this means configuring details of an appropriate SMTP server or continuing to delegate through an identity server that will allow it to do so. If a homeserver admin makes the active choice to use New Vector's identity servers for delegation (up until 1st December), they should make their own users aware through their privacy notice.
This delegation is entirely separate from the user's choice of identity server for user directory services. As of right now the user is free to choose and trust whichever identity server they wish, or to choose not to use an identity server at all.
Yes, 1.4.0 now automatically garbage collects redacted messages (defaults to 7 days) and removes unused IP and user agent information stored in the user_ips table (defaults to 30 days). Finally, Synapse now warns in its logs if you are using matrix.org as a trusted key server, in case you wish to use a different server to help discover other servers’ keys.
Aside from privacy, we’ve expanded our OpenTracing support and fixed a host of bugs. However the thing that is most exciting is switching on our solution for mitigating forward extremities build up’ by default.
In some cases rooms can accumulate ‘forward extremities’, which are simply an artefact of attempting to resolve the room state over multiple servers. Forward extremities are necessary to ensure that each server can independently arrive at the same view of the room eventually, however processing these extremities can be computationally expensive and degrade server performance overall.
Originally it was an experimental config option but we now feel confident to turn it on by default for all instances - it should make a big difference for the CPU of servers in fragmented rooms.
So that’s it folks, thanks for making it this far. As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Also, check out our Synapse installation guide page
The changelog since 1.3.1 follows:
client_secret
in server logs. (#6158)devices
table, and improve its performance on Postgres. (#6135)redaction_retention_period
option. (#6117)Note that this release includes significant changes around 3pid verification. Administrators are reminded to review the upgrade notes.
trust_identity_server_for_password_resets
config option with account_threepid_delegates
, and make the id_server
parameteter optional on */requestToken
endpoints, as per MSC2263. (#5876, #5969, #6028)/lookup
API where available, with fallback to v1. (Implements MSC2134 plus id_access_token authentication
for v2 Identity Service APIs from MSC2140). (#5897)bind_email
and bind_msisdn
parameters from /register
ala MSC2140. (#5964)m.id_access_token
to unstable_features
in /versions
as per MSC2264. (#5974)POST /_matrix/client/unstable/account/3pid/unbind
endpoint from MSC2140 for unbinding a 3PID from an identity server without removing it from the homeserver user account. (#5980, #6062)account_threepid_delegate.email
and account_threepid_delegate.msisdn
for validating threepid sessions. (#6011)/account/3pid/add
and /account/3pid/bind
as per MSC2290. (#6043)bind
parameter from Client Server POST /account
endpoint as per MSC2290. (#6067)POST /add_threepid/msisdn/submit_token
endpoint for proxying submitToken on an account_threepid_handler
. (#6078)submit_url
response parameter to */msisdn/requestToken
endpoints. (#6079)m.require_identity_server
flag to /version's unstable_features. (#5972)synapse_federation_known_servers
: represents the total number of servers your server knows about (i.e. is in rooms with), including itself. Enable by setting metrics_flags.known_servers
to True in the configuration.(#5981)synapse_build_info
: exposes the Python version, OS version, and Synapse version of the running server. (#6005)report_stats_endpoint
option to configure where stats are reported to, if enabled. Contributed by @Sorunome. (#6012)/register/available
if registration has been disabled. (#6082)trusted_key_servers
config field configured. (#6090)user_ips
table. (#6098)power_level_content_override.users
does not contain creator_id
. (#5633)password_reset_failure_template
and password_reset_success_template
, when they are actually password_reset_template_failure_html
, password_reset_template_success_html
. (#5863)public_base_url
instead of public_baseurl
. Thanks to @aaronraimist for the fix! (#5909)dynamic_thumbnails
is false
and the thumbnail was dynamically generated. Fix reported by rkfg. (#5915)M_MISSING_PARAM
if sid
is not provided to /account/3pid
. (#5995)federation_certificate_verification_whitelist
now will not cause TypeErrors
to be raised (a regression in 1.3). Additionally, it now supports internationalised domain names in their non-canonical representation. (#5996)submit_token
endpoint until we implement submit_url
functionality. (#6056)users_set_deactivated_flag
background update. (#6092)sid
to next_link
for email validation. (#6097)threepid_behaviour_email
. (#6104)UID/GID
if they are already correct. (#5970)SYNAPSE_WORKER
envvar to specify python module. (#6058)INSTALL.md
to say that Python 2 is no longer supported. (#5953)/_matrix/client/r0/register
endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. (#5877)trusted_third_party_id_servers
option. (#5875)users_in_public_rooms
to improve the performance of directory queries. (#5894)jenkins/prepare_sytest.sh
file. (#5938)trace
method. (#5776) RoomMemberHandler
. (#6009)origin
argument on FederationHandler.add_display_name_to_third_party_invite
. (#6010)failure_ts
column to the destinations
database table. (#6016, #6072)get_user_id_by_threepid
. (#6099)