Over the course of today we've been made aware of folks port-scanning the general internet to discover private Matrix servers, looking for publicly visible room directories, and then trying to join rooms listed in them.
If you are running a Matrix server that is intended to be private, you must correctly configure your server to not expose its public room list to the general public - and also ensure that any sensitive rooms are invite-only (especially if the server is federated with the public Matrix network).
In Synapse, this means ensuring that the following options are set correctly in
# If set to 'false', requires authentication to access the server's public rooms # directory through the client API. Defaults to 'true'. # #allow_public_rooms_without_auth: false # If set to 'false', forbids any other homeserver to fetch the server's public # rooms directory via federation. Defaults to 'true'. # #allow_public_rooms_over_federation: false
For private servers, you will almost certainly want to explicitly set these to
false, meaning that the server's "public" room directory is hidden from the
general internet and wider Matrix network.
You can test whether your room directory is visible to arbitrary Matrix clients on the general internet by viewing a URL like https://sandbox.modular.im/_matrix/client/r0/publicRooms (but for your server). If it gives a "Missing access token" error, you are okay.
You can test whether your room directory is visible to arbitrary Matrix servers on the general internet by loading Riot (or similar) on another server, and entering the target server's domain name into the room directory's server selection box. If you can't see any rooms, then are okay.
Relatedly, please ensure that any sensitive rooms are set to be "invite only" and room history is not world visible - particularly if your server is federated, or if it has public registration enabled. This stops random members of the public peeking into them (let alone joining them).
Relying on security-by-obscurity is a very bad idea: all it takes is for someone to scan the whole internet for Matrix servers, and then trying to join (say) #finance on each discovered domain (either by signing up on that server or by trying to join over federation) to cause problems.
Finally, if you don't want the general public reading your room directory, please also remember to turn off public registration on your homeserver. Otherwise even with the changes above, if randoms can sign up on your server to view & join rooms then all bets are off.
We'll be rethinking the security model of room directories in future (e.g. whether to default them to being only visible to registered users on the local server, or whether to replace per-server directories with per-community directories with finer grained access control, etc) - but until this is sorted, please heed this advice.
If you have concerns about randoms having managed to discover or join rooms which should have been private, please contact [email protected].
Back in June we wrote about our plans to tighten up data privacy in Matrix after some areas for improvement were brought to our attention. To quickly recap: the primary concern was that the default config for Riot specifies identity servers and integration managers run by New Vector (the company which the original Matrix team set up to build Riot and fund Matrix dev) - and so folks using a standalone homeserver may end up using external services without realising it. There were some other legitimate issues raised too (e.g. contact information should be obfuscated when checking if your contacts are on Matrix; Riot defaulted to using Google for STUN (firewall detection) if no TURN server had been set up on their server; Synapse defaults to using matrix.org as a key notary server).
We’ve been working away at this fairly solidly over the last few months. Some of the simpler items shipped quickly (e.g. Riot/Web had a stupid bug where it kept incorrectly loading the integration manager; Riot/Android wasn’t clear enough about when contact discovery was happening; Riot/Web wasn’t clear enough about the fact device names are publicly visible; etc) - but other bits have turned out to be incredibly time-consuming to get right.
However, we’re in the process today of releasing Synapse 1.4.0 and Riot/Web 1.4.0 (it’s coincidence the version numbers have lined up!) which resolve the majority of the remaining issues. The main changes are as follows:
Riot no longer automatically uses identity servers by default. Identity servers are only useful when inviting users by email address, or when discovering whether your contacts are on Matrix. Therefore, we now wait until the user tries to perform one of these operations before explaining that they need an identity server to do so, and we prompt them to select one if they want to proceed. This makes it abundantly clear that the user is connecting to an independent service, and why.
Synapse no longer uses identity servers for verifying registrations or verifying password reset. Originally, Synapse made use of the fact that the Identity Service contains email/msisdn verification logic to handle identity verification in general on behalf of the homeserver. However, in retrospect this was a mistake: why should the entity running your identity server have the right to verify password resets or registration details on your homeserver? So, we have moved this logic into Synapse. This means Synapse 1.4.0 requires new configuration for email/msisdn verification to work - please see the upgrade notes for full details.
Sydent now supports discovering contacts based on hashed identifiers. MSC2134 specifies entirely new IS APIs for discovering contacts using a hash of their identifier rather than directly exposing the raw identifiers being searched for. This is implemented in Riot/iOS and Riot/Android and should be in the next major release; Riot/Web 1.4.0 has it already.
Synapse now warns in its logs if you are using matrix.org as a default trusted key server, in case you wish to use a different server to help discover other servers’ keys.
Synapse now garbage collects redacted messages after N days (7 days by default). (It doesn’t yet garbage collect attachments referenced from redacted messages; we’re still working on that).
Synapse now deletes account access data (IP addresses and User Agent) after N days (28 days by default) of a device being deleted.
Riot warns before falling back to using STUN (and defaults to turn.matrix.org rather than stun.google.com) for firewall discovery (STUN) when placing VoIP calls, and makes it clear that this is an emergency fallback for misconfigured servers which are missing TURN support. (We originally deleted the fallback entirely, but this broke things for too many people, so we’ve kept it but warn instead).
All of this is implemented in Riot/Web 1.4.0 and Synapse 1.4.0. Riot/Web 1.4.0 shipped today (Fri Sept 27th) and we have a release candidate for Synapse 1.4 (1.4.0rc1) today which fully ship on Monday.
Riot/Mobile is following fast behind - most of the above has been implemented and everything should land in the next release. RiotX/Android doesn’t really have any changes to make given it hadn’t yet implemented Identity Service or Integration Manager APIs.
This has involved a surprisingly large amount of spec work; no fewer than 9 new Matrix Spec Changes (MSC) have been required as part of the project. In particular, this results in a massive update to the Identity Service API, which will be released very shortly with the new MSCs. You can see the upcoming changes on the unstable branch and compare with the previous 0.2.1 stable release, as well as checking the detailed MSCs as follows:
This said, there is still some work remaining for us to do here. The main things which haven’t made it into this release are:
We’ll continue to work on these as part of our ongoing maintenance backlog.
Finally, it’s really worth calling out the amount of effort that went into this project. Huge huge thanks to everyone involved (given it’s cut across pretty much every project & subteam we have working on the core of Matrix) who have soldiered through the backlog. We’ve been tracking progress using our feature-dashboard tool which summarises Github issues based on labels & issue lifecycle, and for better or worse it’s ended up being the biggest project board we’ve ever had. You can see the live data here (warning, it takes tens of seconds to spider Github to gather the data) - or, for posterity and ease of reference, I’ve included the current issue list below. The issues which are completed have “done” after them; the ones still in progress say “in progress”, and ones which haven’t started yet have nothing. We split the project into 3 phases - phases 1 and 2 represent the items needed to fully solve the privacy concerns, phase 3 is right now a mix of "nice to have" polish and some more speculative items. At this point we’ve effectively finished phase 1 on Synapse & Riot/Web, and Riot/Mobile is following close behind. We're continuing to work on phase 2, and we’ll work through phase 3 (where appropriate) as part of our general maintenance backlog.
I hope this gives suitable visibility on how we’re considering privacy; after all, Matrix is useless as an open communication protocol if the openness comes at the expense of user privacy. We’ll give another update once the remaining straggling issues are closed out; and meanwhile, now the bulk of the privacy work is out of the way on Riot/Web, we can finally get back to implementing the UI E2E cross-signing verification and improving first time user experience.
Thanks for your patience and understanding while we’ve sorted this stuff out; and thanks once again for flying Matrix :)
id_access_tokenin CS API when required (done)
id_access_tokenin CS API when required (done)
It was drawn to our attention this afternoon that there is a bug in our GDPR data portability tooling that resulted in the data dump including some events that should not have been included.
This tooling has recently been updated (here is the new code), and the bug only affects reports generated with the updated tool. So far we have generated one report using the updated tooling.
The bug affects events which:
As a reminder, 'shared' message visibility means anyone in the room can view the message, from the point in time at which visibility was set to 'shared' and 'world readable' means anyone can read the messages without joining the room, from the point in time at which visibility was set to 'world readable'.
Events are pulled onto a homeserver over federation when a user on that homeserver tries to access events which, for whatever reason, their homeserver does not already have a local copy. This most often happens when their homeserver is offline for any period of time, but can also happen when a user is the first user from their homeserver to join a room with active participants on other homeservers.
We're still analysing the data but so far it looks like the bug resulted in only a small number of events that were not publicly-accessible being shared (there were also publicly-accessible events mistakenly included). At this stage we have identified 19 events from 4 users across 2 rooms (the dump contained ~3.5 million events). This is not to diminish the severity of the bug - just to reassure that the scale of its impact appears to be extremely limited.
It is also worth noting that any encrypted events erroneously included in the dump will not have been decryptable (since the data subject would not have had access to the keys).
In our original analysis we stated that 19 events were shared erroneously. On closer analysis we missed 5 other timeline events - the correct figure is 24 timeline events originating from 4 users over 2 rooms. However, this figure focused on timeline data and does not take into account all state events (such as user joins, parts, topic changes etc). When considering these too, a further 56 state events were erroneously shared, referencing 64 users across these 2 rooms (mainly detailing when users had joined/left the room after the requesting user themselves had left). These membership events contained avatar & display name details which may not have been public (but in practice, the vast majority appear to be public data).
Aside from the events referenced above, the full dump contained ~20,000 events that also ought not to have been included; however these events were already publicly accessible due to being part of publicly accessible rooms (eg Matrix HQ) and so we do not consider them a breach of data.
Events that are pulled in over federation are assigned a negative 'stream ordering' ID. This is designed to avoid their being sent down the sync (where they would likely be out of sequence). In normal operation (accessing your homeserver via a Matrix client) these events would be appropriately filtered, but a bug in the data dump tooling caused them to be included.
The bug was introduced as a result of two factors:
We are working to fix the bug, and we'll update here when it is resolved. As a reminder, please do report security bugs responsibly as per the Security Disclosure Policy so we can validate the issue and mitigate abuse.
As is standard practice for any data breach, we have notified the ICO.
As a step towards implementing Terms of Service for Sydent Identity Servers (MSC2140), we're rolling out a couple of changes to the two Identity Servers run by New Vector (running at vector.im and matrix.org):
The impact of these changes is that users on homeservers not run by New Vector will no longer be discoverable by their email or telephone number via the Identity Servers running at vector.im and matrix.org. As we roll out the rest of the changes for Terms of Service for Identity Servers, this functionality will again be made available for users who make an informed choice to opt in.
In the short term, the New Vector Identity Servers will continue to support registration with email (signing up with an email address as well as a matrix username) and password reset. However, as we continue to improve Identity Server data hygiene practices, we will phase out their use in registration with email and password reset entirely. We have already made the change to Synapse to support password reset without relying on an Identity Server (though this can optionally be re-enabled).
Once Synapse can support registration with email without relying on an Identity Server we will announce a schedule for disabling registration with email and password reset in our Identity Servers entirely. After this point, homeserver administrators will have to make sure their homeservers are configured to send email to keep registration with email and password reset working. More details on this to follow - please watch this space.
A few weeks ago there was some discussion around the privacy of typical Matrix configurations, particularly how Riot's default config uses vector.im as an Identity Server (for discovering users on Matrix by their email address or phone number) and scalar.vector.im as an Integration Manager (i.e. the mechanism for adding hosted bots/bridges/widgets into rooms). This means that Riot, even if using a custom homeserver and running from a custom Riot deployment, will try to talk to *.vector.im (run by New Vector; the company formed by the core Matrix team in 2017) for some operations unless an alternative IS or IM has been specified in the config.
We haven't done as good a job at explaining this as we could have, and this blog post is a progress update on how we're fixing that and improving other privacy considerations in general.
Firstly, the reason Riot is configured like this is for the user's convenience: in general, we believe most users just want to discover other people on Matrix as easily as possible, and a logically-centralised server for looking up user matrix IDs by email/phone number (called third party IDs, or 3PIDs) is the only comprehensive way of doing so. Decentralising this data while protecting the privacy of the 3PIDs and their matrix IDs is a Hard Problem which we're unaware of anyone having solved yet. Alternatively, you could run a local identity server, but it will end up having to delegate to a centralised identity server anyway for IDs it has no other way to know about. Similarly, providing a default integration server that just works out of the box (rather than mandating the user configures their own) is a matter of trying to keep Riot's UX simple, especially when onboarding users, and especially given Riot's reputation for complexity at the best of times.
That said, the discussion highlighted some areas for improvement. Specifically:
Riot on iOS & Android gives the option of scanning your local addressbook to discover which of your contacts are on Matrix. The wording explaining this wasn't clear enough on Android - which we promptly fixed. Separately, the contact details sent to the server are currently not obfuscated. This is partially because we hadn't got to it, and partially because obfuscating them doesn't actually help much with privacy, given an attacker can just scan through possible obfuscated phone numbers and email addresses to deobfuscate them. However, we've been working through obfuscating the contact details anyway by hashing as per MSC2134, which has all the details. We're also adding an explicit lookup warning in Riot/Web, as per https://github.com/vector-im/riot-web/issues/10093.
There was a bug where Riot/Web was querying the Integration Manager every time you opened a room, even if that room had no integrations (actually, it did it 3 times in a row). This got fixed and released in Riot/Web 1.2.2 back on June 19th.
Matrix needs to authenticate whether events were actually sent by the server
that claimed to send them. We do this by having servers sign their events
when they create them, and publishing the public half of their signing keys
for anyone to query. However, this poses problems if you receive an event
which is signed by a server which isn't currently online. To solve this, we
have the concept of
trusted_key_servers (aka notary servers), which your
server can query to see if they know about the missing server's keys. By
default, matrix.org is configured as Synapse's trusted notary, but you can
of course change this. If you choose an unreliable server as the notary
(e.g. by not setting one at all) then there's a risk that you won't be able
to look up signing keys, and a splitbrain will result where your server
can't receive certain events, but other servers in the room can. This can
then result in your server being unable to participate in the room entirely,
if it's missing key events in the room's lifetime.
Our plan here is to get rid of notaries entirely by changing how event signing works as per MSC1228, but this is going to take a while. Meanwhile we're going to check Synapse's code to ensure it doesn't talk to the notary server unnecessarily. (E.g. it should be caching the signing keys locally, and it should only use the notary server if the remote server is down.)
When doing VoIP in Matrix, clients need to use a TURN server to discover
their network conditions and perform firewall traversal. The TURN server
should be specified by your homeserver (and each homeserver deployment
should ideally include a TURN server). However, for users who have not
configured a TURN server, Riot (on all 3 platforms) defaulted to use
Google's public STUN service (
stun.l.google.com). STUN is a subset of
TURN which provides firewall discovery, but not traffic relaying. This
slightly increased the chances of calls working for users without a proper
TURN server, but not by much - and rather than fall back to Google, we've
decided to simply remove it from Riot (e.g.
This means that VoIP might get less reliable for users who were relying on
this fallback, but you really should be running your own TURN server anyway
if you want VoIP to work reliably on your homeserver.
We should make it clearer in Riot that device names are world-readable, and not just for the user's own personal reference. This is https://github.com/vector-im/riot-web/issues/10216
As you can see, much of the work on improving these issues is still in full swing, although some has already shipped. As should also be obvious, these issues are categorically not malicious: Matrix (and Riot) literally exists to give users full control and autonomy over their communication, and privacy is a key part of that. These are avoidable issues which can and will be solved. It's worth noting that we have to prioritise privacy issues alongside all the other development in Matrix however: there's no point in having excellent privacy if there are other bugs stopping the platform from being usable.
We'll do another blog post to confirm once most of the fixes here have landed - meanwhile, hopefully this post provides some useful visibility on how we're going about improving things.
As mentioned in our last blog post on GDPR, to make sure that everyone has read and understood the important details about how their personal data is processed by the matrix.org homeserver, users who haven't yet agreed to the privacy notice and terms and conditions will be blocked from sending new messages until they have.
Users will continue to be able to receive messages, so they won't miss out on any messages sent to them before they've agreed to the terms.
The System Alerts room has already sent every user their unique link to review and agree, and if anyone missed that message, the latest Riot.im web and mobile will display a helpful error message guiding users who are yet to agree through the agreement process.
If you have any questions or difficulties, please let us know at [email protected].