Today we are disclosing CVE-2020-26890 / GHSA-4mp3-385r-v63f, a denial of service vulnerability affecting Synapse versions prior to 1.20.0. We strongly encourage all Synapse admins to upgrade as soon as possible. If you have not upgraded in a while, please refer to the upgrade notes, especially the latter portion of that document which covers any backwards incompatible changes which you may need to take into consideration.
We extend our thanks to Denis Kasak for reporting this issue, earning a second entry in the Matrix Security Hall of Fame.
Today we have released Synapse 1.21.2, which fixes a couple of minor bugs that crept into the previous release. Full details are below.
Separately, we are advising any administrators who have not yet upgraded to Synapse 1.21.0 or later to do so as soon as possible. Previous versions of Synapse were vulnerable to a cross-site-scripting (XSS) attack; the bug was fixed in Synapse 1.21.0 with PR #8444.
The changelog for 1.21.2 is as follows:
Debian packages and Docker images have been rebuilt using the latest versions of dependency libraries, including authlib 0.15.1. Please see bugfixes below.
Synapse 1.19.2 is a security patch. All federating instances should upgrade immediately.
Today we are releasing Synapse 1.19.2, which is a security patch release containing a fix to encountering invalid events over federation. We are also putting out a fourth release candidate for the upcoming Synapse 1.20.0 release with the same fix.
The bug prevents affected Synapse instances from joining rooms with invalid events. Server administrators running federating instances are strongly encouraged to update as soon as possible.
Those on Synapse 1.19.1 or earlier should upgrade to Synapse 1.19.2, while those who are running a release candidate of Synapse 1.20.0 should upgrade to 1.20.0rc4.
Get the new releases from any of the usual sources mentioned at https://github.com/matrix-org/synapse/blob/master/INSTALL.md. 1.19.2 is on github here, and 1.20.0rc4 is here.
The changelog for 1.19.2 is as follows:
Due to the issue below server admins are encouraged to upgrade as soon as possible. Bugfixes
Folks, today we are releasing Synapse 1.15.2, which is a security release which contains fixes to two separate problems. We are also putting out the second release candidate for the forthcoming Synapse 1.16, including the same fixes.
Firstly, we have fixed a bug in the implementation of the room state resolution algorithm which could cause users to be unexpectedly ejected from rooms (Synapse issue #7742).
Secondly, we have improved the security of pages served as part of the Single-Sign-on login flows to prevent clickjacking attacks. Thank you to Quentin Gliech for reporting this.
We are not aware of either of these vulnerabilities being exploited in the wild, but we recommend that administrators upgrade as soon as possible. Those on Synapse 1.15.1 or earlier should upgrade to Synapse 1.15.2, while those who have already upgraded to Synapse 1.16.0rc1 should upgrade to 1.16.0rc2.
Get the new releases from any of the usual sources mentioned at https://github.com/matrix-org/synapse/blob/master/INSTALL.md. 1.15.2 is on github here, and 1.16.0rc2 is here.
Changelog for 1.15.2 follows:
Due to the two security issues highlighted below, server administrators are encouraged to update Synapse. We are not aware of these vulnerabilities being exploited in the wild.
A malicious homeserver could force Synapse to reset the state in a room to a small subset of the correct state. This affects all Synapse deployments which federate with untrusted servers. (96e9afe6)
HTML pages served via Synapse were vulnerable to clickjacking attacks. This predominantly affects homeservers with single-sign-on enabled, but all server administrators are encouraged to upgrade. (ea26e9a9)
This was reported by Quentin Gliech.
Over the course of today we've been made aware of folks port-scanning the general internet to discover private Matrix servers, looking for publicly visible room directories, and then trying to join rooms listed in them.
If you are running a Matrix server that is intended to be private, you must correctly configure your server to not expose its public room list to the general public - and also ensure that any sensitive rooms are invite-only (especially if the server is federated with the public Matrix network).
In Synapse, this means ensuring that the following options are set correctly in
# If set to 'false', requires authentication to access the server's public rooms # directory through the client API. Defaults to 'true'. # #allow_public_rooms_without_auth: false # If set to 'false', forbids any other homeserver to fetch the server's public # rooms directory via federation. Defaults to 'true'. # #allow_public_rooms_over_federation: false
For private servers, you will almost certainly want to explicitly set these to
false, meaning that the server's "public" room directory is hidden from the
general internet and wider Matrix network.
You can test whether your room directory is visible to arbitrary Matrix clients on the general internet by viewing a URL like https://sandbox.modular.im/_matrix/client/r0/publicRooms (but for your server). If it gives a "Missing access token" error, you are okay.
You can test whether your room directory is visible to arbitrary Matrix servers on the general internet by loading Riot (or similar) on another server, and entering the target server's domain name into the room directory's server selection box. If you can't see any rooms, then are okay.
Relatedly, please ensure that any sensitive rooms are set to be "invite only" and room history is not world visible - particularly if your server is federated, or if it has public registration enabled. This stops random members of the public peeking into them (let alone joining them).
Relying on security-by-obscurity is a very bad idea: all it takes is for someone to scan the whole internet for Matrix servers, and then trying to join (say) #finance on each discovered domain (either by signing up on that server or by trying to join over federation) to cause problems.
Finally, if you don't want the general public reading your room directory, please also remember to turn off public registration on your homeserver. Otherwise even with the changes above, if randoms can sign up on your server to view & join rooms then all bets are off.
We'll be rethinking the security model of room directories in future (e.g. whether to default them to being only visible to registered users on the local server, or whether to replace per-server directories with per-community directories with finer grained access control, etc) - but until this is sorted, please heed this advice.
If you have concerns about randoms having managed to discover or join rooms which should have been private, please contact [email protected].
This morning (06:11 UTC) it became apparent through mails to [email protected] that a security researcher was working through the TLS Certificate Transparency logs for
*.modular.im to identify and try to access non-public services run by New Vector (the company formed by the original Matrix team, which hosts
*.matrix.org on behalf of the Matrix.org Foundation, and develops Riot and runs the https://modular.im hosting service).
Certificate Transparency (CT) is a feature of the TLS ecosystem which lets you see which public certificates have been created and signed by given authorities - intended to help identify and mitigate against malicious certificates. This means that the DNS name of any host with a dedicated public TLS certificate (i.e. not using a wildcard certificate) is visible to the general public.
In practice, this revealed a handful of internal-facing services using dedicated public TLS certificates which were accessible to the general internet - some of which should have been locked to be acessible only from our internal network.
kibana.ap-southeast-1.k8s.i.modular.im- a Kibana deployment for a new experimental Modular cluster which is being set up in SE Asia. The Kibana is in the middle of being deployed, and was exposed without authentication during deployment due to a firewall & config error. However, it is not a production system and carries no production traffic or user data (it was just being used for experimentation for hypothetical geography-specific Modular deployments). We firewalled this off at 07:53 UTC, and are doing analysis to confirm there was no further compromise, and will then rebuild the cluster (having fixed the firewall config error from repeating).
Additionally, certain historical Modular homeservers & Riots (from before we switched to using wildcard certs, or where we’ve created a custom LetsEncrypt certificate for the server) are named in the CT logs - thus leaking the server’s name (which is typically public anyway in that server’s matrix IDs if the server is federated).
We’re working through the services whose names were exposed checking for any other issues, but other than the non-production SE Asia Kibana instance we are not aware of problems resulting from this activity.
Meanwhile, we’ll be ensuring that semi-internal services are only exposed on our internal network in future, and that Modular server names are not exposed by CT logs where possible.
TL;DR: You can list all the public non-wildcard TLS certs for a given domain by looking somewhere like https://crt.sh/?q=%25.matrix.org. This lets you find internal-sounding services to try to attack. In practice no production services were compromised, and most of our internal services are correctly firewalled from the public internet. However, we’re reviewing the IP locking for ones in the grey zone (and preventing the bug which caused an experimental Kibana to be exposed without auth).
We’d like to thank Linda Lapinlampi for notifying us about this. We’d also like to remind everyone that we operate a Security Disclosure Policy (SDP) and Hall of Fame at https://matrix.org/security-disclosure-policy/ which is designed to protect innocent users from being hurt by security issues - everyone: please consider disclosing issues responsibly to us as per the SDP.