Matrix.org - matrix.org homeserver

Post-mortem of the September 2 outage

2025-10-29T10:00:00+00:00

On 2nd September 2025 the matrix.org homeserver</a> suffered a ~24h outage.

During routine maintenance to increase disk capacity, the primary database failed, and we fell back to the secondary. In attempting to restore the original primary, we lost the secondary-turned-primary rendering matrix.org unavailable.

To recover, it was necessary to restore from S3 storage, however the restore process was lengthy due to the size of the dataset (51TB).

The matrix.org homeserver was unavailable from 2025-09-02 17:45 UTC and full service resumed at 2025-09-03 18:00 UTC. No data was lost as a result of the incident.

🔗</a>What happened</h2>

The matrix.org homeserver is made of a main Synapse instance with hundreds of workers, backed by a single logical Postgres cluster made up of two machines. The primary database is replicated to a secondary, read-only instance via streaming</a> replication.

A schema showing Synapse connected to a primary database. It also shows a secondary database pulling WALs from the primary. Finally the primary database also pushes WALs to a S3 bucket. </figcaption> </figure>

Confusingly, at the time of the incident, the primary database server is called db-02</code>, and the secondary database server is called db-01</code>. The deployment runs on bare metal servers at Mythic Beasts</a> and the Postgres database servers both use their own logical RAID 10 array with mdraid</code></a>.

Our primary database is backed up to an S3 bucket in AWS. At the time of the incident, we performed a full database backup weekly, incremental database backups daily, and we archived WALs continuously to a separate S3 bucket. If you are not familiar with WALs, you can see them as the primary database recording what it does when inserting or removing records into its tables.

Since WALs are exact records of what happened, they can be useful for two things


Archive/backups.</strong> WALs can be seen as “small incremental backups” to aid point-in-time recovery and/or bridge the gap between full backups. This is why we keep them in the S3 bucket in addition to the weekly and daily backups.</li>
Replication.</strong> The secondary database will fetch those WALs from the primary database and also replay them on itself, to have the exact same records as the primary database.</li>
</ol>
The primary database will produce WALs as it adds or removes records, and keep them until they have been both archived to a S3 bucket and</em> been fetched by the secondary database.</p>
We monitor the database size and growth, and when the database reached roughly 51TB (90% of disk capacity) we set about adding more disks in the array.</p>
🔗</a>Timeline</h3>
At 11:03 UTC on Sept 2nd 2025, Mythic Beasts’ teams added 2 NVMe drives to our primary and secondary database db-02</code> and db-01</code>, respectively the primary and secondary database servers. We then set about introducing the new drives to the respective RAID arrays.</p>
At 11:17 UTC, one existing drive disappeared from the RAID array of db-02</code>, our primary database server. Our monitoring fired, and Mythic Beasts confirmed the issue. Because we’re using RAID 10</a>, the setup was still functional but running in degraded mode. There was no data loss, but the RAID array could potentially not survive another drive failure, and performance could be degraded.</p>
We had to restore the RAID array of db-02</code>, our primary database server, to a non-degraded state. That meant failing over to our secondary database on db-01</code> and doing maintenance on db-02</code>, a decision we took at 12:57 UTC.
At 13:27 UTC the fail-over to the database on db-01</code> was complete, and db-01</code> was now the primary. Synapse happily started writing to it. At this point there has been minimal disruption. But the new primary didn’t archive WALs to S3 due to an issue in the archiving script. Because of this and because the new secondary was offline, WALs could not be discarded from db-01</code> yet.</p>
At 13:30 UTC, we restarted the postgres instance on db-02</code> in replica mode, effectively turning our former primary database into a secondary. The new secondary needed to catch up with what had been happening on the new primary running on db-01</code> by consuming its WALs.</p>
At 13:53 UTC, after the new secondary on db-02</code> caught up with the new primary on db-01</code>, we decided to restart the db-02</code> server, in the hope of restoring its RAID 10 array to a fully functional state.</p>
At 14:01 UTC, the db-02</code> server rebooted in recovery mode, because its RAID array could not be assembled as an additional drive was now missing. Recovery mode means no network, no ssh, no postgres instance was running. At this point, our secondary database was offline, and our new primary still didn’t archive WALs to S3. WALs kept accumulating on db-01</code>.</p>
At 15:44 UTC, we reached the conclusion that</p>

The RAID array on our db-02</code> server was not recoverable as the RAID headers were missing on both drives that were missing from the RAID array.</li>
We needed to recreate a fresh RAID array.</li>
We would need to restore the database on db-02</code>, ideally by making it a replica of the new primary running on db-01</code>.</li>
</ol>
At 16:11 UTC, the db-02</code> server went back online with a fresh RAID 10 array, and by 16:50 UTC we unblocked the WALs archival from the primary on db-01</code> to S3. WALs could start being discarded on the primary on db-01; it was time to restore the secondary on db-02.</p>
At 17:20 UTC, we upgraded the Postgres on the brand new and empty secondary on db-02</code> to the latest patch version. That meant not having to do another set of failovers to upgrade the databases after getting back to a healthy state. At this point, we still had a fully functional primary database.</p>
At 17:25 UTC we attempted to start restoring the data on db-02</code>.</p>
First we ran a command on the machine to list all of the backups and identify the correct backup ID:</p>
sudo /opt/wal-g/wal-g \
</span>  --walg-s3-prefix=s3://<backup-bucket> \
</span>  --aws-shared-credentials-file=/home/postgres/.aws/credentials \
</span>  --aws-region=eu-west-2 backup-list
</span></code></pre>
We were able to identify the most recent backup and target it with a restore command that we have documented as part of our restore procedures:</p>
sudo time /opt/wal-g/wal-g \
</span>  --walg-s3-prefix=s3://<backup-bucket> \
</span>  --aws-shared-credentials-file=/home/postgres/.aws/credentials \
</span>  --aws-region=eu-west-2  \
</span>  --walg-download-concurrency=32 \
</span>  backup-fetch /mnt/data/postgresql-14/ <backup_id> \
</span>  2>&1 | tee restore.log
</span></code></pre>
This command was entered while the current directory was the Postgres database directory, which caused the tee</code> command to fail and abort the restore process, which had enough time to create some directories in the data path but nothing else. We switched to the home path and re-ran the command, which successfully wrote to the log file, but failed due to the data directory being non-empty after the previous aborted restore.</p>
The necessary course of action at this point was to clear the remains of the failed restore attempt from the data directory and start again. Since db-02</code> had already been cleared and needed to be restored, this didn’t register as a particularly high risk manoeuvre.</p>
Unfortunately, in attempting to do so, we erroneously deleted the data directory of the primary on db-01</code>.</p>
After realising our mistake, we decided to keep our Postgres up on db-01</code> in case deleted files were still open in Postgres processes, with the hopes that the open file handles would forestall the actual deletion of the data on disk.</p>
With both db-01</code> and db-02</code> out of action we had no other option but to restore at least one database from offsite backup. Since db-02</code> was in a pristine state, with an expanded RAID array, we decided to restore the database on this server.</p>
As detailed earlier, our backup strategy at the time was: full database backups weekly, incremental database backups daily, and WALs archival continuously. To perform a complete backup without any data loss on db-02</code>, we needed to</p>

Restore the latest weekly full database backup from S3.</li>
Restore all the daily incremental backups from S3 since the last daily backup.</li>
Replay the WALs since the last daily incremental backup.</li>
</ol>
So at 17:30 UTC, we started restoring the database on db-02</code> by using wal-g</a> -  a well known tool that pulls the backups from S3 to restore databases. That was going to be costly and slow, but we didn’t have a choice and that’s what backups are for.</p>
In the meantime, the backend team was paged to manage the impact to Synapse, an incident was opened, and an emergency was declared. Our primary database on db-01</code> was partially wiped and throwing errors, but not corrupt enough to crash Synapse. We decided to shut down both Synapse and the primary database to avoid unknown database states. At this point, the matrix.org homeserver was down.</p>
At 18:06 UTC we decided to re-mount the data partition of db-01</code> as read-only. We were now in emergency mode, and wanted to ensure we couldn’t damage the database further, in case we could salvage it later.</p>
At 18:40 UTC, after taking the time to consider our options, we realised the following</p>

extundelete</a> and ext4magic</a> were both unmaintained for a decade, and are unable to work on an unmounted filesystem. ext4magic even explicitly documents it “can no longer successfully process current ext4 file systems”</li>
We also tried R-Linux</a>, but weren’t confident in the integrity of the recovered files - especially with our recent experiences with slow-burning postgres corruption</a>.</li>
So we decided against trying to recover the lost data by carving or undeletion, in favour of a guaranteed reliable restore from offsite backup.</li>
</ul>
At 20:30 UTC, db-02</code> was still restoring from the S3 backup. After restoring the database on db-02</code> from its full and incremental backups, we would need to replay the WALs produced by db-01</code> to fill the gap between the last backup taken from db-02</code> and the moment we lost db-01</code>.</p>
When we promoted db-01</code> as the primary, the script that archives WALs to S3 started erroring out. As a result, there were WALs on db-01</code> that were not in S3. We were going to need those to bring db-02</code> up to date with the point of the outage. We started copying these WALs from db-01</code> to db-02</code> to have them ready to replay once the restore from S3 backup would complete. Restoring 51 TB from S3 takes time</em> so we didn’t have much more to do than wait for the restore to complete.</p>
At 07:21 UTC the next morning, the data extraction from the full weekly backup was complete. However as soon as wal-g attempted to start restoring the next daily increment backup it immediately errored out due to an issue</a> with wal-g that had already received a fix</a>. Now, we regularly run backup recovery tests during which we spin up a short lived EC2 instance, called our Disaster Recovery Server, perform a full database restore on it and run a few tests before tearing it down. During one of those recovery tests, we had already run into the wal-g problem and fixed it in the backup recovery test ansible playbook… but unfortunately this got missed on the actual database servers.</p>
This meant that our production version of wal-g was outdated and hadn’t received this fix. At this point, we had pulled all the full base backup data from S3, but wal-g had failed to restore any incremental backups on top of it because of this bug. We needed to update wal-g to the latest release of the same major version to benefit from the fix. After doing so, we tried to relaunch the restore, and it failed because the data directory already contained a partial restore.
So, we decided to patch wal-g to recover from a partial failed restore, and after fighting with the dependencies we figured out how to make it accept a non-empty data directory that contained a pristine full base backup, so we didn’t have to pull everything from S3 again. We patched it, built it, and used it against db-02</code> at 09:23 UTC.</p>
At 09:35 UTC the first incremental backup was restored, then the second at 09:44 UTC, the third at 09:54 UTC, and the final backup was restored at 10:03 UTC.</p>
At 10:45 UTC we attempted to start the new instance in standby mode to check its consistency. But the standby mode of Postgres is meant to be for replicas, and replicas need either a primary to grab WALs from, or a remote_command</code> set to fetch WALs. Since the new Postgres on db-02</code> couldn’t reach any primary and it didn’t have any restore_command</code> set, it refused to start in standby mode.</p>
So we configured a restore_command</code> with a wrapper script that could fetch WALs from both S3 (our “continuous backups”) or from the filesystem (db WALs carried over from db-01</code>) and started Postgres in standby mode successfully. It started catching up on WALs from S3 at 11:00 UTC.</p>
Frustratingly, the playback rate was slower than expected - to replay the ~18 hours of WALs ended up taking 5.5 hours (we had been hoping it would take around 10 minutes for every 1 hour of WALs). It took until 16:27 UTC to replay all the WALs. And at this point we could log into the Postgres database on db-02</code>.</p>
At long last, we had a working database instance, with no data loss. We promoted it to a primary database at 16:45 UTC, and started a Synapse test worker at 16:51 UTC. We could see new WALs start to appear in S3, which meant WAL shipping worked. It was time to restart Synapse and bring matrix.org back online. We started Synapse at 16:54 UTC, and after various thundering-herd overloads as everyone reconnected, all the workers were online and stable by 18:00 UTC.</p>
At this point, the server was back online, matrix.org was catching up with everything that had happened on the rest of the federation while it was offline, albeit with a single database node (although WALs were being archived to S3 for safety).</p>
At this point, if our database had caught fire we could have been able to restore it without losing data, but at the cost of bringing matrix.org offline again. We had just been through it, we didn’t want to do it again. We needed our secondary back.</p>
But we also needed the team to get some rest. Given how slow it was to replay WALs, we reconfigured our backups to happen against the primary database rather than against the (missing) replica. We let the European team go to bed, while our American SRE kept tabs on everything. At 03:26 UTC a new incremental backup completed.</p>
At 09:21 UTC we added the two NVMe disks to the RAID array and to the LVM volumes group of db-01</code>. We rebooted to ensure the disks were properly detected and mounted - but the server didn’t come back. We opened the lights-out console Mythic Beasts provides us, and saw that the RAID array was not in the functional state. We had rebooted db-01</code> at a critical moment of the array reshaping.
After fixing up the array to bring it in a bootable state, db-01</code> finally restarted, and we copied over the basebackup from db-02</code> and set it to replicating.</p>
🔗</a>Lessons learned</h2>
🔗</a>We have a massive database</h3>
A lot of the pain we experienced during this outage came from how massive our database is.</p>

Now that we have extra storage, it’s the right time to run pg_repack</code> and reclaim free space.</li>
We have already increased the frequency of incremental backups, since they’re much faster to restore than it is to replay WALs.</li>
We also know Synapse could do much better in terms of data storage and there are plans to drastically reduce storage requirements in future</a>, also see Matthew’s “how hard could it be” hack from the week before the incident: https://youtu.be/D5zAgVYBuGk?t=1852</a>.</li>
</ul>
🔗</a>Our safeguards can be improved</h3>
Running a destructive command on the incorrect server was a key moment in the incident. While it can be attributed to human error, it is incorrect to focus on the individual, and instead consider how to improve the tooling and processes surrounding them to minimise the chances of a repeat in the future.</p>
On making the sensitive changes, the on-call group effectively paired as a trio, however, in the heat of the moment, this was insufficient to catch the error.</p>
We realised that the database servers names were a source of confusion. db-01</code> reads like “Primary DB” and db-02</code> reads like “Secondary DB”. Not only is this false in our case, a primary database server can become a secondary database server, and the other way around. Names with intrinsic meanings are a source of confusion.</p>
We’re considering changing the background colour of the terminal dynamically depending on the role the database is playing in the cluster. An idea we floated is to monitor the presence of the standby.signal</code> file in the database data directory to know whether it is a primary or a secondary database, and update the terminal’s background colour accordingly. This is not a silver bullet since the background colour would only change after a command has been sent, but that would already be an improvement.</p>
We also discussed wrapper scripts around sensitive commands (such as an alias for rm</code>) or automating some operations, such as starting a base backup from primary to secondary as a means to minimise risk.</p>
We could restore the service after 24h offline, without any data loss</em> despite losing both our primary and secondary databases. This accounts for a great Recovery Point Objective and is testament to our PITR processes that we test regularly. We should take pride in the recovery, but we need to work on a shorter Recovery Time Objective, we’re currently talking to service providers to get free infrastructure that would make it easier and faster to recover.</p>
🔗</a>We can have better tools</h3>
We upgraded wal-g on all servers, not just the Disaster Recovery Server, and have done a round of Disaster Recovery testing with it. We didn’t explore yet how we can ensure the Disaster Recovery Server and the production servers can stay aligned.</p>
At the next hardware refresh, we will explore using ZFS so we can make local snapshots and recover much more quickly from not so happy accidents such as accidentally wiping the wrong database.</p>
🔗</a>We have a great community and providers</h3>
We received a lot of support on social media where we communicated actively around the incident. This was welcomed positively by the broad community, despite our status page not receiving the attention it deserved. We’re adding steps to our incident response playbook to update status.matrix.org</a> as the canonical source of truth during an incident, and liaise with the advocacy team to keep social media updated as well.</p>
The SRE team would like to thank our hosting provider Mythic Beasts. They reached out quickly and proactively when adding new disks, reporting the errors they were seeing. They have been much more than just a pair of remote hands. They also reached out with an offer of support during the incident.</p>
Finally, we’d like to sincerely apologise again to everyone impacted by the outage, we hope you found the post-mortem informative and if you would like to investigate running your own homeserver, there are plenty of distributions</a> to choose from.</p>

Matrix.org (Official Account) and Terms updates 2025-07-31T00:00:00+00:00 Users of the Matrix.org homeserver have recently received – or will shortly receive as the notifications are rolled out progressively – an invite from a user called Matrix.org (Official Account). Those checking the room will have noticed that it announces upcoming changes to our the Matrix.org Homeserver Terms and Conditions. Some of you have asked us questions about these two events so we would like to offer some clarification and (hopefully) some reassurance. 🔗</a>Matrix.org (Official Account)</h2> Firstly, the Matrix.org (Official Account): given not all the users having an account on the Matrix.org homeserver have an email or another way to reach out to them linked to their Matrix accounts we decided to use a room to send official communications to them. This user is currently sending out messages about the upcoming changes to the Terms and Conditions updates (see next section), but we anticipate sending out other account related messages in future. For long time users of the homeserver if you scroll back you may also see some messages from several years ago. You can verify that this user is legitimate by checking the page on our website</a> we have dedicated to it. In short, the user’s matrix identifier is @server:matrix.org and the room should look like the screenshot below from various clients. Any communications coming from a different identifier are not coming from us. 🔗</a>Terms and Conditions updates</h2> It is the first time we have introduced significant changes to our Terms and Conditions (also known as “Homeserver Terms</a>”) since their creation in 2018. These changes have been material enough to warrant notifications going out to all matrix.org accounts. Some of these changes are associated with the incoming premium accounts</a> and others are directly related to our compliance with the Online Safety Act (OSA). But all in all, if you’re a typical matrix.org user there shouldn’t be any impacts to how you use Matrix on a day-to-day basis. 🔗</a>Premium accounts and fair usage limits</h3> When it comes to the premium accounts, the main changes we’ve introduced are directly related to payment and clarification of fair usage limits. As announced a couple of weeks ago</a>, we will be progressively introducing premium accounts on the Matrix.org homeserver, first to new users, then slowly migrating existing accounts. This is due to the need for the Foundation to help both cover the costs of running the server and limit the abuse we are seeing from a few users. The introduction of payment is why you might have noticed the terms becoming a bit heavier on the “legalese”: there are certain things we need to make sure we cover when payments are involved which can be quite dense. These are mainly clauses 5 and 6, around liability and payment, respectively. We will work to make these friendlier over time. In terms of fair usage: so far there was no mention of fair usage in the terms, which of course led to some users pushing the limits of their usage, by storing a large amount of data on the server. We have now made it clear that this won’t be tolerated, but these fair usage limits should not impact normal users. We are introducing both a per 24h and per 4 weeks rolling cap, the values of which have been based on looking at the stats of usage and we may iterate on them if needed. It is worth noting that whilst the terms will be effective on the 14th of August for existing users, they will be effective immediately for new users. With regards to the roll out of premium accounts, here are a few important notes: Pricing details and usage limits are published here</a> and may still evolve</li> Premium accounts will start being rolled out to some of the new users in a few days</li> Existing users will only be migrated from their legacy account to the new Free accounts in several weeks (we will communicate when we start the migration)</li> Existing users will be notified with enough advance warning before the migration of their account happens</li> For any question, please refer to the announcement blog post</a> or join https://matrix.to/#/#foundation-office:matrix.org</a>.</li> </ul> 🔗</a>Online Safety Act and Digital Services Act related changes</h3> Outside of these, the most common questions we are seeing are associated with clause 3.2 “user content”. This explains the measures we take to comply with our OSA and DSA obligations around proactive monitoring for illegal content. To reiterate and be as clear as possible: Your encrypted messages are not being looked at - we have never introduced backdoors and never will;</li> Your private discussions remain private.</li> </ul> What we are doing is using readily available tooling like Cloudflare’s scanning tool</a> to identify unencrypted media which might have a match with content identified in reputable Child Sexual Abuse Materials (CSAM) databases such as NCMEC</a>. Like all proactive tooling described in this clause, this is all done in public and unencrypted rooms. We have no way of looking into encrypted rooms nor do we have plans to do so. If you’ve been around for a while you will have seen that we have started raising the alarm</a> about the dangers and potential risks of the OSA back in 2021. Whilst we have certain legal obligations as a UK based organisation, we will not falter and compromise on our principles and values. We are planning to do a longer blog post on our regulatory compliance in the near future. If there is anything in particular you would like to see answered by that please feel free to drop us an email at legal@matrix.org</a>. How we discovered, and recovered from, Postgres corruption on the matrix.org homeserver 2025-07-23T00:00:00+00:00 Greetings from Element's backend/SRE team, who run the matrix.org</code> homeserver</a> on behalf of the Matrix.org Foundation. Recently users of the matrix.org</code> homeserver began seeing problems where rooms would simply stop working</a>. Operations such as sending a new message, or joining the room as a new member, would fail for mysterious reasons. Where an error message was shown at all, it tended to be something cryptic like "No create event in auth events". After a couple of weeks of hard work by a team of Element staff including backend developers and systems engineers, we were able to repair almost all of the affected rooms. Although we're still investigating exactly what went wrong and checking that everything is now working as it should, we'd like to share some details about what we know and the work we've done to date. We'll be diving into some quite technical details. Hopefully you'll find it interesting learning a bit about how Synapse works, how Postgres works, and the work we sometimes find ourselves doing to keep the matrix.org</code> homeserver running. 🔗</a>TL;DR</h2> Let's start with a high-level summary. The matrix.org</code> homeserver is backed by a large PostgreSQL database instance. Parts of an index on one of tables in this database had become corrupted. We are unsure exactly what caused this corruption, but believe it happened at least a year ago, and likely significantly longer. The nature of this corruption was such that it had little or no effect at first. However, a background maintenance task which removes old, unreferenced data from this table recently started working on the corrupted region. Due to the corrupt index, the maintenance task incorrectly removed active data from the table, in effect corrupting rooms. Having identified the problem, we rebuilt the corrupted index, and then restored the data that had been incorrectly removed, from database backups. 🔗</a>Initial investigations, or "what exactly is a state group?"</h2> We were first alerted to the problem via a bug report</a> from a user, and similar reports in public Matrix rooms and other social media. As more anecdotal reports came in, we started to investigate what was going on. To understand what we found, you'll need to understand what we mean by a "state group". As most readers probably know, Matrix allows applications to associate "state" with a room. In contrast to "message" events which are normal messages that fit at one particular point in the timeline, state sticks around, visible to all, until it is replaced. One example of state is a user's room membership — whether or not they are currently a member of that room. Another example is m.room.name</code>, which, as the name implies, holds the room's name. Yet another type of state is the "create event": this is the very first event that happened in a room. The create event is somewhat special in that it can never be changed, but we still always expect it to be part of the room state. Obviously, the state of a room changes over time. What may be less obvious is that a homeserver often needs to know what the state of a room was at some point in the past, to answer questions such as "should this user be allowed to see this event" or "should I accept this event that has been sent to me over federation from another homeserver". Whilst in theory we could figure out what the state was at any given point in history by replaying each event that happened in the room before that point, that would be extremely computationally intensive. So in practice, homeservers end up storing what amounts to a snapshot of the room state at each historical event. Of course, regular events don't change the state of the room, so there is no point actually storing the state at each of those events. So, at last we can understand what a "state group" is: Synapse groups together a set of events in a given room, where the state in that room remained unchanged. In other words, a run of m.room.message</code> events (normal room messages) will likely all share the same "state group". Once somebody changes the room state (for example, by joining the room), we'll start a new state group, and subsequent events will be part of that new state group. The diagram below illustrates this. Blue creates a new room, and Yellow joins. The first few events each change the state of the room, meaning that each new event goes into a new state group. But events F</code> and G</code> are regular messages, meaning they don't change the state of the room. The room state after each of events E</code>, F</code> and G</code> is the same, so they can all be in state group 5. Things get a bit more complicated at H</code> and I</code>: both Yellow and Blue try to change the name at the same time, so the state after H</code> includes H</code> and the state after I</code> includes I</code>. The state resolution algorithm determines that I</code> ends up "winning", so the state after J</code> includes I</code> and not H</code>, meaning that J</code> (and K</code>) can share state group 7 with I</code>. Now, when we started investigating the rooms where people had reported problems, we found clear signs of corrupted state groups. For example, the state in some of the state groups in affected rooms was completely empty. As I said earlier, the room's create event is always part of the state of a room, and it can never change, so finding state groups whose state does not at least include a create event was a big red flag. This also gives a clue to the meaning of that error I mentioned earlier: when we decide whether to accept an event into the room, we check the state of the room. One of the things we check for is the presence of a create event: "No create event in auth events" means Synapse rejected the new event because there was no create event in the room state. There's one more wrinkle we'll need to understand about state groups. As you can see in the diagram above, most state groups only differ very slightly (typically by a single piece of state) from the previous state group in the same room. Storing a complete snapshot of the state every time the state in a room changes would be very expensive in terms of storage. So instead, Synapse normally just stores the difference from an earlier state group; then, to stop lookups becoming too expensive, we store a complete snapshot every 100 state groups or so. Again, you can see that "compression" technique at play in the diagram above. Most state groups have a grey arrow representing the link to the previous state group, meaning that each state group only needs to store the delta from the previous state group (shown in bold whilst those states implied by the "previous" link are greyed out). State groups 1 and 8 are stored as complete snapshots. Synapse stores all this data in its database: the event_to_state_groups</code> table tells us which state group each event is in, state_groups_state</code> stores the actual state snapshot or delta for that state group, and state_group_edges</code> gives us the previous state group for delta-stored state groups. 🔗</a>The hunt for suspects</h2> Thanks to the way Matrix works, once Synapse has created a state group, we very rarely ever have to change it. (If more events arrive, they may be assigned to an existing state group, but the state group itself, and the room state for that state group, remain unchanged). The only exceptions are: the state compressor</a>, which rewrites state groups so that they can be stored more efficiently.</li> purge operations, where all or part of a room's history is removed from the database, making the corresponding state groups redundant.</li> a cleanup job which removes state groups which were created but never used.</li> </ul> ... and of course, the creation of the state group in the first place. At least that gave us a place to start looking, but since we hadn't made any changes to those areas of the code recently, we were still at a bit of a loss. The state compressor was easy to rule out, at least, since it runs as a separate process and we were certain it wasn't running on matrix.org</code>. As a precaution, we temporarily disabled the cleanup job that removes redundant state groups. We couldn't figure out how it could cause the problem, but better safe than sorry, and disabling it would just mean we used a bit more disk space for a while. 🔗</a>More evidence comes to light</h2> Our next step was to try and figure out when the problem started. Searching the logs for one Synapse process gave some clear, and worrying, results: 2025-06-24: 0 results for “No create event”</li> 2025-06-25: 0 results for “No create event”</li> 2025-06-26: 0 results for “No create event”</li> 2025-06-27: 48 results for “No create event”</li> 2025-06-28: 1100 results for “No create event”</li> 2025-06-29: 3610 results for “No create event”</li> 2025-06-30: 6902 results for “No create event”</li> </ul> So, we double-checked for changes that had been made around 27th June, and still didn't find anything. We considered rolling back Synapse to an older version, but since we couldn't figure out what had changed, we didn't know how far we would have to roll back. What's more, we found state groups that must have been fine initially (say, on 2025-06-29) were now corrupt: in other words, this confirmed that the problem wasn't that we were creating new, invalid state groups, but there was a process somewhere in the system that was corrupting existing state groups. The diagram below illustrates the problem. The state in state group 4 has been corrupted, meaning that that state group (and state groups 5, 6, and 7 which all reference it) are now missing an important part of the room state, and events will not be authorised. 🔗</a>Some remedial steps</h2> Now that we knew we were dealing with data loss, it seemed likely that we would need to restore data from backup, so started the process of restoring the database backup from 26th June into a new Postgres instance hosted in Amazon EC2. The restore process takes several hours, so we wanted to get it started. On the other hand, it would leave the Matrix Foundation an EC2 bill of hundreds of USD per day for an EC2 instance large enough to host the database! We also set up a guard against further corruption: we added</a> a Postgres "constraint" which would reject any SQL queries which attempted to delete the state from a state group while that state group was still in use. 🔗</a>A culprit emerges</h2> By this point, it was the morning of 3rd July. The cleanup job had been disabled for 24 hours, and we hadn't seen any further corruption. Now that we had the protective constraint in place, we decided to re-enable the cleanup job, and see what happened. Almost immediately, we could see that the cleanup job was hitting the constraint. From the Postgres logs: 2025-07-03 12:30:38.250 UTC [matrix background_worker1] ERROR: Deleting state_groups_state row when it still exists in state_groups_edges: prev_state_group = 963361509 </code></pre> ... meaning it was trying to delete the state for state group 963361509</code> while that state group was still in use. The Synapse logs, meanwhile, suggested it was actually trying to delete completely different state groups. Was it a bug in Synapse? Or the Python Postgres driver</a>? We spent a while narrowing down the problem, even resorting to tcpdump</a> to see what was happening between Synapse and the database. With tcpdump</code>, we could see DELETE</code> queries being made, but none which would affect state group 963361509</code>. Maybe this was actually a bug in PgCat</a>, which we use to pool Postgres connections? Or even in Postgres itself? We tried replaying the query that tcpdump</code> had captured. Here's a screenshot from our ops room: Oh wow indeed. That shouldn't happen. We narrowed the problem down to one particular state group: 483128098</code>. What happens if we just try and read that state group from the database? matrix=> SELECT state_group, room_id FROM state_groups_state WHERE state_group = 483128098; state_group | room_id ------------+---------------------------------------- 483128098 | !XtFbidoIcAVPuQtXcG:matrix.org 963361875 | !IvVovpFpWhKsKMCGCO:irc.snt.utwente.nl 483128098 | !XtFbidoIcAVPuQtXcG:matrix.org (3 rows) </code></pre> Oh dear. Once your database starts returning nonsense results, you're going to be in for a bad time. What it meant here was that, although the cleanup job was (correctly) trying to clean up state group 483128098</code>, Postgres would also delete the data for state group 963361875</code>. Suddenly things started to make sense: rooms were getting corrupted by cleanup jobs for completely unrelated rooms. We've encountered Postgres index corruption before</a>, and this matched the symptoms perfectly. In short: the index entries for state group 483128098</code> point to the wrong place in the main table data (the "heap"). So, if we did a query that Postgres could answer by just looking at the index, we'd get plausible-looking results: matrix=> SELECT state_group, type FROM state_groups_state WHERE state_group = 483128098; state_group | type ------------+-------------- 483128098 | m.room.member 483128098 | m.room.member 483128098 | m.room.member (3 rows) </code></pre> ... but as soon as Postgres had to look at the heap, it would return nonsense, as above. 🔗</a>Give it to me straight, doc</h2> The good news, such as it was, was that we could now be reasonably certain that other homeservers would not be affected: this was data corruption on the matrix.org</code> Postgres instance. On the other hand, we had no idea how extensive the corruption was, when it had happened, or if it was still happening. We did several things to try to assess the damage. The first thing to check was whether both Postgres instances had the same problem. (We replicate all our data to a warm standby server using streaming replication</a> so that we can fail over rapidly in the event of a hardware failure.) As far as we could tell, both servers had identical corruption. Secondly, we wrote a script which sampled the state_groups_state</code> table to look for corruption. It told us that the problem was worryingly large: millions of state groups were affected. But for some reason, it only seemed to affect state groups in the range 147M - 541M. (State group 541M was created in January 2021. As of July 2025, we're now up to 1040M.) We also ran pg_amcheck</a> on the affected index. This is a tool that forms part of the Postgres distribution, and it checks for inconsistencies in all or part of a database. It took a while, but didn't return any problems. This mostly told us that amcheck</code> couldn't detect this sort of corruption, but one thing it checks is that all rows in the table also appear in the index; so now we knew that we weren't missing any index rows — we just had extra ones. Meanwhile, we tried reaching out</a> to the helpful folks on the pgsql-general</code> mailing list. We figured if anyone knew what could have caused this, they would. The final thing we did at this point was to take a look at the actual index data with pageinspect</a>, to see if there were any clues there. It didn't really tell us anything we didn't already know (i.e., that the index rows were pointing at the wrong place in the heap), but it was interesting to check out the structure of the index. 🔗</a>A deeper dive into Postgres indexes</h2> On the morning of 4th July, our backup from 26th June at last finished restoring. That meant two things: first, we could check if it had the same index corruption as our primary and secondary servers (it did), and secondly, we could start to think about how to repair the damage. We noticed something else interesting, though. On the production servers, some index entries pointed to state groups which didn't yet exist on 26th June: -- On the production database matrix=> SELECT state_group, type, ctid FROM state_groups_state WHERE state_group = 353864583; state_group | type | ctid -------------+---------------------------+---------------- 353864583 | m.room.member | (39060361,12) 1034753774 | m.room.member | (264925234,54) 1034753810 | im.vector.modular.widgets | (264925240,54) 1034753803 | m.room.member | (264925252,54) 1034753803 | m.room.member | (264925252,55) (5 rows) </code></pre> (ctid</code>, or "current tuple ID" is Postgres's internal identifier for a row in a table: the format is a page</a> number, followed by an offset within that page. We'll return to ctid</code>s in a minute.) Those state groups (1034753774</code> etc.) were only created on 3rd July, so clearly the backup will look different. Indeed: -- On the restored backup matrix=# SELECT state_group, type, ctid FROM state_groups_state WHERE state_group = 353864583; state_group | type | ctid -------------+---------------+--------------- 353864583 | m.room.member | (39060361,12) (1 row) </code></pre> Did that mean that the corruption was ongoing? Time for another look with pageinspect</code>. As with most Postgres indexes, this one is a B-Tree</a>. To find a specific entry, you start at the "root" of the tree (a single page which covers the whole table, but with very coarse index entries: there might be one sub-page for all the A's, for example, and another for all the B's), and work down the tree until you get to the right "leaf" page. On our restored backup, we manually walked the tree to find the leaf index pages for state group 353864583</code>. Turned out, there were several pages of entries: it seems like, at some point in the past, this state group had lots of state associated with it. Anyway, the interesting page was this: -- On the restored backup matrix=# select ctid, left(data, 77) as data from bt_page_items('state_groups_state_type_idx', 192904826); ctid | data ----------------+------------------------------------------------------------------------------- (264925236,41) | 87 8b 17 15 00 00 00 00 1d 6d 2e 72 6f 6f 6d 2e 6d 65 6d 62 65 72 35 40 66 72 (264925234,54) | 87 8b 17 15 00 00 00 00 1d 6d 2e 72 6f 6f 6d 2e 6d 65 6d 62 65 72 4b 40 66 72 (264925235,54) | 87 8b 17 15 00 00 00 00 1d 6d 2e 72 6f 6f 6d 2e 6d 65 6d 62 65 72 47 40 66 72 (3 rows) </code></pre> Being a leaf index page, the ctid</code> points to the actual row in the heap. This is an index on (state_group, type, state_key)</code>, so the data</code> here is: a little-endian 64-bit representation of 353864583</code></li> a length/flags byte (1d</code> => 13 bytes of uncompressed text)</li> the event type (m.room.member</code>)</li> another length/flags byte</li> the state_key</code>: a user ID, which I've truncated in the above for brevity and privacy.</li> </ul> The point is, even in the backup, we have index rows pointing to heap tuple (264925234,54)</code>. And what is at that heap tuple? matrix=# SELECT * FROM heap_page_items(get_raw_page('state_groups_state', 264925234)); lp | lp_off | lp_flags | lp_len | t_xmin | t_xmax | t_field3 | t_ctid | t_infomask2 | t_infomask | t_hoff | t_bits | t_oid | t_data ----+--------+----------+--------+--------+--------+----------+--------+-------------+------------+--------+--------+-------+-------- 1 | 0 | 0 | 0 | | | | | | | | | | (1 row) </code></pre> Nothing at all. That tuple doesn't exist. It's just empty space in the table data. Finally, we can understand a bit about what's happened here. The corruption is not ongoing. Rather, the index was already corrupt at the time the backup was taken, but the index rows point into empty space -- and apparently Postgres ignores such index rows. Then, on 3rd July, that empty space got used for state group 1034753774</code>, meaning that the index entry for state group 353864583</code> now points to the data for state group 1034753774</code>. This tells us something else interesting: this corruption could have been there for months or years, without anyone noticing. It was only once Postgres started populating that bit of table space that any problem would have been observable. So why was the index entry pointing at empty space? That's a great question, and something we spent a long time discussing. Presumably, at some point in the past, we used to have lots of entries in state_groups_state</code> for state group 353864583</code>. Then, most of these entries were removed (likely by the state compressor</a>), causing a bunch of free space to be created in the table data -- but for some reason, the index entries for those rows didn't get correctly cleaned up, leaving them dangling. 🔗</a>Repairing the damage</h2> We now had enough information to start to get things working again. The first priority was to get Postgres back to a consistent state. That meant rebuilding the index, which in itself wasn't trivial, given the index takes up over 4 TB — but we had just enough spare disk, so we set the reindex going overnight. Next, we needed to repair any state groups which were incorrectly modified by the cleanup job due to the corrupt index. To do this, we considered the range of state groups that the cleanup job had been working on, and wrote a script which queried each of those state groups on our restored backup, noting down the targets of any bogus data: this was the list of potential victims of incorrect cleanup. We then cross-referenced that list of potential victims against the production database, checking for state_groups_state</code> entries which had been removed but where the state group was still in use: this gave us the actual victim list. Each of those victims had to be re-inserted into the production database. We started those scripts running on 5th July, but due to the amount of data involved, it took nearly a week before we were able to announce on 11th July that the majority of rooms were repaired. 🔗</a>Assessing the root cause</h2> So, what went wrong to cause those index pages to get corrupted? The short answer is, we don't know. First, some timeframes. We know for certain that corruption happened after January 2021 (or at least, that corruption was still ongoing at that point), since it affected state groups created at that time. And we know that it happened before July 2025, since corruption was present in the backup from the end of June. It's hard to be any more certain than that. The one thing we can be sure it's not is a bug in Synapse or PgCat: there is no way that an application should be able to cause internal corruption within a Postgres database. One possibility is a Postgres bug, but Postgres is an extremely robust piece of software, and the Postgres team treats corruption bugs extremely seriously. We were using Postgres 10.12 in January 2021, and we've looked through the Postgres release notes for every version since then, and not found any bug fixes that would fit this pattern. It's worth noting that Postgres relies heavily on its underlying filesystem, as well as the device drivers and hardware, to behave correctly: in particular, if the filesystem claims that data has been persisted, it really has been persisted. Problems in this area are far from unknown — back in 2018, the Postgres team discovered that their 20-year-old assumptions about how fsync</code> worked were incorrect (wiki page</a>, FOSDEM presentation</a>). But the fixes to that were backported to Postgres 10.7 so that problem can't explain this corruption. So that really leaves kernel or disk firmware bugs, and hardware failures. Our filesystem is nothing fancy, just ext4</code>, and we're using stock Debian kernels. Some sort of hardware problem seems like the most plausible cause. We're somewhat surprised that hardware failure would cause extensive damage to a single index, whilst apparently leaving all other data intact, but it's at least possible. For the curious: our current generation of database servers run Linux kernel 6.1, and each server uses eight 15TB Intel NVME SSDs in a RAID10 configuration to give us 64TB of storage. The previous generation (retired in November 2023) used 8TB SSDs with LVM and no RAID, on Linux 4.19. Of course, we have checked fsck</code>, smartctl</code> and mdadm</code> for any errors on the current disks: none have shown up. There was a disk failure on the primary database server in October 2021, which caused us to fail over to the secondary, so it's conceivable that the dying disk lost some writes, though it would have to have been doing so for a while for the corruption to have made it onto the secondary. We're not entirely satisfied with this explanation. If you've got any ideas, let us know</a>! 🔗</a>Conclusions</h2> Incidents like this happen from time to time when running software services, particularly relatively large scale ones like the matrix.org</code> homeserver. They are impossible to plan for and often, as in this case, take significant time and effort from people who would otherwise be developing features or fixing bugs. We know that there are plenty of users out there who will have been affected by the problem, and found themselves unable to communicate as a result. We very much share your frustration, and we'd like to apologise for the disruption to service. With that said, we're glad that we were able to get to the bottom of most of the problem, and get the lost data restored within a relatively short time. If nothing else, hopefully this blog post will be of use to future generations faced with Postgres index corruption! Introducing premium accounts to fund the matrix.org homeserver 2025-06-13T14:00:00+00:00 🔗</a>TL;DR</h2> As we need to take more concrete steps to improve the financial situation of the Foundation, we will be rolling out a freemium offer for the matrix.org homeserver users. The alternative is to turn off the server, which we want to avoid doing. The goal is for the most active users to support the cost of the service. Free users will have limits on how they can use the service (mostly around media). The change can be supported by any client with limited to no development. Premium plans will be rolled out over the summer, and we will be iterating on the exact scope in the first few weeks. The Homeserver Terms and Privacy Policy will be updated accordingly and deployed in the coming weeks. 🔗</a>The full story</h2> We have been communicating on the lack of funds in the Foundation for a while now, the latest being here</a>. And whilst we’ve been working hard to gather new members and are happy to see the number of logos increasing</a> (thank you all for seeing the need for Matrix to stay independent and safe, and the value in supporting it!), none of the big players in the ecosystem have actually committed to one of the higher membership tiers, so we need to find other ways towards sustainability. The Foundation’s mission</a> can basically be summarised by 4 main goals: Ensure the specification of the protocol stays canonical and unencumbered, to avoid fragmentation and being overridden by a single player.</li> Ensure that all players in the ecosystem are at a level playing field, helping them succeed by giving them visibility and listening to their needs.</li> Promote the Matrix standard, as the value of Matrix is directly proportional to the size of the public network and how much it is used and commercialised.</li> Ensure the public network is safe by building moderation tools that can be used by the server admins, for the sake of our users and making sure the network is attractive.</li> </ol> In practice, it means that we are currently spending money on: A small team of developers and moderators, to develop Trust & Safety tooling, moderate the matrix.org server, and redirect people who do not understand the decentralised nature of Matrix reporting abuse to us towards the appropriate server admins.</li> The infrastructure of the matrix.org homeserver, including the SRE team, who are on call to keep it running, and the support team.</li> Organise and sponsor events to promote and evangelise the protocol.</li> A tiny team to run the Foundation itself, including the support of external contractors for the administrative side (finance, legal, tax). The staff works on governance (organising the governing board elections, running the meetings, liaising between the different teams), raises money and brings members in, manages social media and liaises with the community, keeps the website up and up to date, publishes TWIMs and blogs, organises the events, etc. This team whose day job is to keep the Foundation running is also supported by a lot of volunteer (and sometimes sponsored by employer) time from the Governing Board and its Working Groups, the Spec Core team, the Guardians, and other external staff.</li> </ul> We haven’t gotten to the point of publishing the public financial report (although it should be almost finalised now), because we are frantically trying to focus on closing the financial gap, but here is an overview of the split of expenditures in the last year: As you can see, 20% of the Foundation’s expenditure goes towards hosting the matrix.org free and public homeserver. If we add in the cost of the moderation work done by the Trust and Safety team, the total share of the costs attributable to the matrix.org homeserver account for almost 50% of all expenditure. Meanwhile, today, only 50% of the spending of the Foundation is covered by its revenues (donations and memberships), and we are working hard towards reducing this gap. We’ve kept the matrix.org homeserver around so far, despite its costs, as we consider it essential to seed the network in support of the nurturing part of the Foundation’s mission: despite Matrix being decentralised by design, users need a trusted place to create a free Matrix account to try it out in the first place. However, we can’t continue to bear the cost of the server as is, and before we get to the extreme position of being forced to turn it off leaving its 370k monthly active users in the awkward position of finding a new home for their account, we’ve decided to try to alleviate some of these costs by setting-up a freemium offering and proposing premium plans in addition to the free ones. The goal is to get the server to an at least financially break-even position. If, by any chance, it was ending up profitable, the profit would directly be invested in Trust and Safety</a>, or other new programmes which can support the ecosystem. As a reminder, the Foundation is a Community Interest Company, i.e. a limited company which operates to provide a benefit to the community it serves rather than private profit. 🔗</a>What will the freemium offer look like?</h3> The idea is to set some limits for users on the free plans, which would be lifted for users on the premium plans in exchange for an affordable membership. We are still iterating (and will do for a while) on how it looks, but users can expect limits around media sizes and/or volumes. The goal is to ensure that the most active users participate in covering the costs of the service, in return for the access to a fully encrypted and decentralised open network. We are limited in scope and design by the fact we need to ship a minimum viable product as soon as possible (we need to reduce costs now) and by not wanting to impose too much development (if any) to Matrix client developers. Obviously we would have preferred to keep everything free of charge. We will never sell our users’ data or cripple our services with ads, so we need to find ethical sources of revenue. 🔗</a>When will the new plans take effect?</h3> The roll-out will happen progressively, starting in the coming weeks and hopefully completing in the summer of 2025. We will start by opening up premium plans to new users only, before progressively migrating all existing accounts to a free plan which will give them the option to upgrade to a premium plan. Users will of course be notified ahead of their account being migrated. 🔗</a>Will this work with whatever Matrix client I use?</h3> Yes. The plan management will be handled via the My Account</a> screens provided by the Matrix Authentication Service (MAS), and notifications to users will be sent in a dedicated room using the Server Notices</a> feature built into the Matrix protocol – already used by the homeserver to send automatic messages to the user – so should be seamless for every client. 🔗</a>I am a Matrix client developer, do I need to do anything?</h3> There are two considerations from a Matrix client point of view: support for the Server Notices</a> feature</li> if the client is distributed via the Apple App Store, then support for MSC4286</a></li> </ul> If the client doesn't show server notices at all then, whilst the client will remain usable with the matrix.org homeserver, your users will have a degraded UX as they won't receive notifications when encountering usage limits. Apple places restrictions</a> on how payments are implemented by iOS (et al) apps that are distributed via the App Store. We expect that most, if not all, apps that fall within scope would be classified as what Apple calls “Free Stand-alone Apps</a>”. Such apps do not need to use in-app purchases so long as “there is no purchasing inside the app, or calls to action for purchase outside of the app”. In order to meet these requirements we have proposed MSC4286</a> which provides a way for a homeserver (such as the matrix.org homeserver) to flag parts of messages as containing a call to action and for affected clients to be able to hide that content. Example implementations are linked in the MSC. 🔗</a>I am already supporting the Foundation by paying an individual membership, will I have to pay for a premium plan too?</h3> No, individual members</a> of the Foundation will get access to the premium features at no extra cost. This benefit will be implemented as part of the process of migrating existing accounts to the free plan. 🔗</a>What else will be changing?</h3> In order to support these changes we will be releasing updates to the Homeserver Terms and the Privacy Policy in the coming weeks. Users of the matrix.org homeserver will be notified and will need to accept the new terms. The scope of change will be clearly highlighted in the release note, but essentially you can expect new terms around payment and additional information on the types of information we will collect about your account, as well as the processors we will use to enable payments. We realise this is quite a big change, but our position is that a slightly limited service is better than no service at all, so we chose to ask for financial contribution rather than turn off the server. Paying a subscription for the matrix.org homeserver is basically a way to support Matrix, ensuring the Foundation can continue to play its role of neutral custodian, enabler and safeguardian of the protocol and the network. We will be publishing more details and a proper FAQ as the roll-out happens, so watch this space for more details. Matrix.org is now running MAS! 2025-04-08T15:30:00+00:00 We're thrilled to announce that the migration of matrix.org to the Matrix Authentication Service (MAS) is complete and went according to plan - having been running for over 24h in our brave new world, we’re declaring the migration a success! As of Monday April 7th 07:30 UTC, matrix.org is running on Matrix’s next-generation auth system</a> based on OAuth 2.0/OpenID Connect. This is no mean feat - the migration shifted all 45M access tokens and 110M users from Synapse to MAS in under 30 minutes (thanks in part to MAS’s cheeky use of the x86-64-v2 architecture; who knew that database migrations can be SIMD-accelerated?) - and represents the culmination of over 4 years of work to move Matrix to a modern authentication standard. Many thanks go to Element for funding, Hugh, Olivier and many other contributors who helped me make Next Gen Auth happen! 🔗</a>What does this mean for you?</h2> Check back to our previous announcement</a> for the full details of migration, but your existing sessions remain active - no logging out and back in required. The move to MAS provides enormous improvements to security and usability: Access tokens rotate regularly, a leaked token has a limited lifespan</li> A single home for your account credentials which password managers can manage for you</li> Consistent auth and account management experience across apps</li> All Matrix.org users can finally fully enjoy next generation clients like Element X</li> A solid basis for all our upcoming authentication features, which we’ll enable on matrix.org as they get approved for merge in the Matrix spec: Login via QR code, complete with E2EE identity!</li> Support for 2FA, MFA, Passkeys etc</li> OAuth 2.0 scopes let users control what features an app can access.</li> </ul> </li> </ul> 🔗</a>Your account has a new home</h2> account.matrix.org</a> is now the dedicated home for managing your matrix.org account, which you can access through your browser or supported clients. Here you can: View and manage your connected devices</li> Update your email address and contact information</li> Change your password</li> Manage account settings and security options</li> </ul> (Eagle-eyed users may notice that client URLs for web logins aren’t shown in account.matrix.org - this only affects migrated devices; new logins will show up correctly. One workaround is to use the native device manager in Element Web to see the URLs of your old sessions). 🔗</a>See it in action</h2> If you’re wondering what the new world of Next Gen Auth looks like, but don’t want to mess around logging in to a new client - fear not, for we have videos! Here’s an example of native Next Gen Auth in Element X iOS logging into the shiny new matrix.org system: