privacy; IS and IM permissions and management. Hashed lookup API. Metadata cleanup (GCing redactions; deleting media; history retention); TURN management
aggregations paused
FTUE next up
Cross-signing
Immutable DMs
...then communities
RiotX
0.3 came out last week - handles DMs; lots of polishing; reduces the APK down to ~10MB!
Bridges
moving stuff into proper managed infrastructure
bifrost should be back soon?
gitter kept breaking but we're actually working with their team to fix that
working out how to automate deploying them on modular hosts
Progress for Dendrite this week has been a bit starved due to ongoing privacy work from the backend team. However it continues to move forwards with the excellent help of cnly reviewing and merging PRs.
We had 3 authors have pushed 5 commits to master and 6 commits to all branches. On master, 14 files have changed and there have been 298 additions and 121 deletions.
Also this week was some work from cnly on refactoring gomatrixserverlib and exposing some more of its internal constants to deduplicate code between itself and Dendrite.
Finally, we had some more work on an unmerged PR from cnly, which adds some internal query APIs for which servers are currently joined to a room, for use by other components later down the line.
No new Sytests are passing this week. We’re still currently at 172 tests passing out of 761 tests in total.
Matrixmon updated
Matrixmon, the perl C2S testing utility has been updated to v0.2.0 with support for SSL connections.
Synapse
Synapse 1.3.0 was released - check out the blog post.
contains performance improvements to reduce disk I/O and reduce RAM usage. We’ve been running it on matrix.org for a week or so and are really pleased with the results.
mautrix-telegram got some fixes to bugs that appeared during the move to mautrix-python. It also got a manhole similar to the one in synapse, which gives admins access to an interactive Python shell inside the bridge while it's running.
The main part of the manhole (server and repl) is in mautrix-python, so you can expect the feature to appear in mautrix-facebook, mautrix-hangouts and possibly maubot in the future.
mautrix-whatsapp and mautrix-hangouts can now automatically add rooms to a user-specific community to help with filtering. mautrix-facebook is also getting that feature in the near future.
mautrix-facebook now has a search command, which means you can now actually initiate conversation with facebook users.
Dept of Clients 📱
Riot Web
From the team:
Privacy work continues
STUN fallback server for assisting with voice & video call negotiation (only used when your homeserver is not configured with it’s own TURN server) changed from Google to Matrix.org server and a prompt has been added to request permission before using the fallback server
Many small steps towards supporting user choice of identity servers and integration managers, including no identity server at all
Users can now download artifacts from my gitlab account since I got an SDK container from CoDerus running, cross compiling to Sailfish-RPMs in the Gitlab-Ci.
But my programming progress on master looks a bit silent because I swap to matrix-nio.
Rewrite the list view partially to handle lists with several types of items more efficiently. The message list has text messages, images, membership updates, etc. Each type needs to be rendered differently, but items of the same type could reuse GUI components when they scroll into and out of view.
Since the coming 1.3.0 release is bringing along a whole bunch of perf improvements, (and I want to test them out on my IO starved setup) I'm building K8s optimized images of the RCs. Only the debian version of the image though.
They expanded:
It's basically the regular docker image, only with some changes to make it fit nicer in Kubernetes
And there's no Helm chart at the moment, was poking at that for a bit but took a break from building one as I - back then - had trouble with the amount of data that had to be generated
You can only have one Synapse pod at any given time, as it's not scalable. Though you can scale it with workers (as I do myself)
Database-wise, I'm personally running three node HA clusters using Stolon
We run a public instance on Permaweb.io for everyone to join. We’re a large community of developers focusing on the decentralized web, with lots of channels dedicated to IPFS. In fact, we even host Riot on IPFS! We make full use of Matrix’s bridging capabilities, to Discord, IRC, Slack, Gitter and Telegram too! Come join us at #general:permaweb.io or register at https://riot.permaweb.io/ (or any other compatible client) and say hi!
Dept of Bots 🤖
Notification/Remind-Me-Bot using matrix-bot-sdk
Anton (@antonivan:matrix.org) is working at Matrix Towers this summer for work experience. He's been learning about Matrix and matrix-bot-sdk:
This week I made a Notification/Remind-Me-Bot using Travis's bot sdk, it took me 1 and half days, and is able to - set reminder, check reminder, cancel reminder, set interval reminder, cancel interval reminder. It is also able to store multiple reminders and cancel them using the numbers 1 - ∞ . Here is a link to my Github repo - https://github.com/joakimvonanka/Matrix-Remind-me-bot
Dept of Event Videos 📹
Matrix talk @ FrOSCon
Oleg's talk at FrOSCon was apparently really well received:
Video recording from the Matrix talk @ FrOSCon.
Video is in German slides in English.
He's also thinking of writing a guide to creating presentations:
If someone is interested on an efficient way of creating presentation using only open source tools I'm planing to write an article on that.
Small spoiler: matrix is one of them 😉
Final Thoughts 💭
This week, I have been mostly thinking about... merch. We blew the cache on stickers, so if you are waiting for stickers specifically and have not heard from me, that is probably why. Please contact me with any questions!
That's all I know 🏁
See you next week, and be sure to stop by #twim:matrix.org with your updates!
The main thing to know about 1.3.0 is that is contains performance improvements to reduce disk I/O and reduce RAM usage. We’ve been running it on matrix.org for a week or so and are really pleased with the results.
Check out our message send heat map.
Other than that there are a bunch of bug fixes and tweaks to generally make things run more smoothly.
Fix 500 Internal Server Error on publicRooms when the public room list was
cached. (#5851)
Synapse 1.3.0rc1 (2019-08-13)
Features
Use M_USER_DEACTIVATED instead of M_UNKNOWN for errcode when a deactivated user attempts to login. (#5686)
Add sd_notify hooks to ease systemd integration and allows usage of Type=Notify. (#5732)
Synapse will no longer serve any media repo admin endpoints when enable_media_repo is set to False in the configuration. If a media repo worker is used, the admin APIs relating to the media repo will be served from it instead. (#5754, #5848)
Synapse can now be configured to not join remote rooms of a given "complexity" (currently, state events) over federation. This option can be used to prevent adverse performance on resource-constrained homeservers. (#5783)
Allow defining HTML templates to serve the user on account renewal attempt when using the account validity feature. (#5807)
Bugfixes
Fix UISIs during homeserver outage. (#5693, #5789)
Fix stack overflow in server key lookup code. (#5724)
start.sh no longer uses deprecated cli option. (#5725)
Log when we receive an event receipt from an unexpected origin. (#5743)
Fix debian packaging scripts to correctly build sid packages. (#5775)
Correctly handle redactions of redactions. (#5788)
Return 404 instead of 403 when accessing /rooms/{roomId}/event/{eventId} for an event without the appropriate permissions. (#5798)
Fix check that tombstone is a state event in push rules. (#5804)
Fix error when trying to login as a deactivated user when using a worker to handle login. (#5806)
Fix bug where user /sync stream could get wedged in rare circumstances. (#5825)
The purge_remote_media.sh script was fixed. (#5839)
Deprecations and Removals
Synapse now no longer accepts the -v/--verbose, -f/--log-file, or --log-config command line flags, and removes the deprecated verbose and log_file configuration file options. Users of these options should migrate their options into the dedicated log configuration. (#5678, #5729)
Here at Matrix we’re frequently asked for official merch. While those who choose to sponsor Matrix development on Patreon have always been blessed with branded t-shirts, it was otherwise very difficult to obtain official merch.
Now available from shop.matrix.org, we’re offering packs of Stickers, T-Shirts, and Hoodies!
Stickers are provided in three types: hexagonal Matrix and Riot stickers, and rectangular transparent Matrix stickers.
T-Shirts are available in all sizes and in Unisex and Women's editions. We’re using Gildan SoftStyle t-shirts, which are popular amongst printed shirts for being both soft and durable.
Matrix-branded hoodies are something very special. Superbly embroidered, they are comfortable and stylish. On a recent trip to San Francisco, a passerby stopped a Matrix guardian in the street and asked where they could buy a Matrix hoodie without knowing what Matrix was.
You really need one of these.
All merchandise proceeds go to the Matrix.org Foundation, which helps fund Matrix development!
The Official Matrix Merch Store is open now: shop.matrix.org!
Yes, it's true: Official Matrix Merch is available now. There are stickers, t-shirts and hoodies all available now, go and claim yours from https://shop.matrix.org.
I will be doing a talk on Matrix this weekend (tomorrow) at the annual Free Open Source Conference (FrOSCon) in Sankt Augustin (Germany).
If you are around - come by and get some awesome [matrix] stickers! =)
This week’s stats for Dendrite are in! 4 authors have pushed 11 commits to master and 11 commits to all branches. On master, 48 files have changed and there have been 676 additions and 146 deletions.
Very old PRs from CromFR have been updated and finally merged! These lay the groundwork for providing filters across various API endpoints (used in search, /sync, etc)
We also have a number of PRs that were moved forward but not merged. Add auth fallback endpoint (our oldest PR at the moment!) which we mentioned last week is so very close, but not quite there in time for this TWIM.
In terms of passing Sytests, Dendrite now passes 172 tests, up from 167 last week. This is out of 761 tests in total. So just a small bump (0.6%), but note that things that are crucial for Dendrite development at the moment, such as federating, only cover a small subset of tests, so each passing one can bring us much closer to getting Dendrite usable day-to-day.
Synapse
From Neil:
The big news this week we’ve landed some massive DB improvements (https://github.com/matrix-org/synapse/pull/5706) which should make message sending on matrix.org (and any other server) feel noticeably snappier. :) It'll be in the next release.
Work continues on our installer to make it easier to configure Synapse and we’ll be looking for some community feedback rsn. The room directory revamp is now very close and we hope to have something live on matrix.org in about a week.
Finally we’re working on improving the efficiency of smaller instances sending messages into large rooms, we’re still at the design stage, but it will make a huge difference for anyone self hosting.
matrix-room-directory-server ( #matrix-room-directory:t2bot.io ) is less experimental than the key server but is still very early days. Currently it only offers the ability to manipulate the federated public room directory for your server, but in future it is planned to be its own standalone directory server (room aliases without having to run a whole homeserver). Check it out by searching the t2bot.io room directory from your client.
Dept of Clients 📱
Continuum progressing!
I love getting updates from yuforia - they've been consistently working on Continuum for some time, and by increments are making a great client.
Reuse the ListView of messages across different rooms to reduce memory usage
(Experimental) Remember and refocus the last read message, making it easier to go through all unread messages while switching chat rooms freely.
Renaming of QMatrixClient to Quotient has been finally merged to the master branch; Quaternion master uses it from now, too. Packagers are welcome to make test builds and report bugs in #quotient:matrix.org. libQuotient 0.6 beta is coming close now!
Riot iOS
Test on iOS 13 Beta. Beta 6 fixed most bugs discovered on Beta 5
Released 0.9.2 (Waiting for Apple review at the time of writing)
mautrix-facebook, mautrix-telegram and the tulirverse
tulir has been making big updates to two of his bridges:
mautrix-facebook can now bridge formatting, mentions, replies and reactions in both directions.
There is also a logout command now.
mautrix-telegram's switch to mautrix-python is nearly finished (i.e. it didn't cause any errors for the past few days when testing in production). The main reason for the switch is using one Matrix library for all my python bridges. It also means the bridging code like double puppeting and command handling I shared between mautrix-facebook and mautrix-hangouts is now also used in mautrix-telegram.
Visible changes directly caused by using mautrix-python:
Logs are now colorful.
Python 3.5 is no longer supported.
The bridge will refuse to start without access to the base config file.
Other changes that happened during the switch:
Telegram "Saved Messages" can now be bridged even when using double puppeting.
Mentions on Telegram are marked as read when using double puppeting (messages were already being marked as read, but mentions weren't).
Also, this actually happened last week and the week before that, but anyway: I've moved the CI and docker registry of my active projects to a self-hosted GitLab at mau.dev.
Specifically, the CI/docker registry for all four mautrix bridges and maubot and automatic builds for gomuks have been moved. My maubot plugins also have automatic .mbp builds in the CI. For the docker registry, prepending dock.mau.dev/ to the existing image names will work.
The old places (docker hub, dl.maunium.net) won't get new builds anymore. The repos on GitHub are still the "canonical" repos, but they're mirrored more or less instantly with maumirror.
He adds:
mautrix-telegram will probably get some sort of history filling in the near-ish future
Also,
I'm going to add some way to put bridged rooms into personal communities for filtering purposes. Not yet sure if it'll be fully built into the bridges or some kind of an external script
The configuration process (users, rooms, mappings) is quite manual, but it's simple and has worked well for me for the past year and a half.
I've only just polished it up a bit, released and integrated with the Ansible playbook, so others could benefit from it too.
I started a "matrix-xmpp-filter" project. It's like matrix-ircd, but with xmpp instead of irc. It can also filter messages by weekday, time and regular expressions. Target audience is Sailfish phone users, but it could be used with other xmpp clients too (no xmpp MUC support required). https://k2c42.dy.fi/matrix-xmpp-filter.git/#matrix-xmpp-filter:ellipsis.fi.
Currently working on a Puppet module to manage Matrix Synapse installs (both directly and through docker), not quite ready for prime-time yet but watch this spot.
I develop matrix Bot, which converting voice messages to text. It use Yandex Speech API and Yandex API cloud (some as aws) for temporary store voice-data before converting. https://github.com/progserega/voice2textMatrix
Bot have such logic:
user add bot to room (for example room with whatsapp bridge users - now bridge support adding bot to such rooms)
bot listen room, and when get voice-message - send it to Yandex-cloud for translate (now support only Russian language).
When translating is success - bot get result text and show it in room as notice, such: "Username said: text"
Bot also receive some command, which allow disable it in this room, or disable translating for user, which send command..
Seeing this, I wondered, "y tho?", luckily progserega was able to explain with a graphic! They say a picture paints a thousand words, and it's clear from the conversation below why he'd want to have speech-to-text capabilities.
This week we’ve been working on implementing identifier hashing in sydent, a brand new installer for Synapse, we’ve trialled some new db perf improvements which look very promising (more on this soon) and finally very close to an all new implementation of room search.
Next week we’ll push out the all new Sygnal, continue with the installer and room search and pick up some tasks under our privacy umbrella to ensure that Synapse is not holding onto any data that it does not strictly need to.
A PR for redactions opened by our GSoC student cnly
A PR for authentication fallback (for when clients don’t have a web browser built in and want to use recaptcha from trion129
This has been in the works for a while, and just has a couple small changes left to go!
TwoPRs for filtering database functionality from CromFr
A PR for fixing the scope of transaction IDs in Dendrite’s transaction cache from cnly
Cnly’s GSoC period ends on August 26th. He has been a massive boon to the project’s cadence so far and we hope he will continue even after GSoC ends when he has time :)
Provide automatic deduplication and rate-limiting when downloading media resources. This improves performance of GUI applications, where avatars of many users may appear on screen at once, and some of them may be identical.
libQuotient's master branch now supports sending and receiving reactions, and receiving message edits. We also have another PR in the works from aa13q that would add support for events decryption, as a part of his GSoC endeavour.
Nightlies of Fractal are currently stuck on a 2 weeks old build because of one of our dependencies… but the long awaited 4.2 stable release is out! You can get it out fresh from flathub as usual.
Changelog:
New features:
Adaptive window, mobile friendly
Window size and position are remembered
Redesigned login
Spellcheck
Network proxy support
Typing notifications
Badges are shown for operators and moderators
Keyboard shortcuts for easier navigation across rooms
Better uploads:
Audio and video files are now tagged correctly
Image files have a thumbnail
Various tweaks to the file chooser
Bugfixes:
Logs actually output something now
A few issues with invites and direct chats have been resolved
More reliable scrolling
Some crashes fixed
Under the hood:
Code refactor continues
We’re now using Rust 2018
Many improvements to the build system and CI
A lot of the internals have been refactored, so they should be more maintainable and reliable going forward. Also some future work planned on refactoring the state handling using Redux, which should make it much faster to fix issues and add new features in the future. There are even some ideas already for a future plugin API!
Display has been improved in a lot of places. Error messages are now more clearly formatted, images in reply-quotes are now shown as thumbnails rather than just a filename, and the reply-to popup now shows the full event that you're replying to properly. The chat window now correctly sticks to the bottom when you're scrolled to the end and receive new messages, even when an image or video loads - though per-room scroll position restoration isn't done yet.
There's now an experimental compact mode! It uses a more IRC-client-like layout for messages, and generally just fits more content onto the screen. It can be enabled with an experimental flag. A screenshot of the compact mode can be found here.
Hey folks, first an update from the bridges integrations side. We've brought back snoonet and oftc on the integrations menu so you can once again connect matrix rooms to these networks. Other networks should work as standard.
On the matrix-appservice-irc side we've made a few fixes to the handling of IRC modes (things that handle how users should behave) when bridged to IRC. This should hopefully make opping and voicing users more reliable. There are a few other fixes in the pipeline too, so a release isn't too far off.
I had an adventure this afternoon into the world of bridging again, and have made a Github to Matrix bridge. At the moment you can join aliases and saturate your homeserver with the entire history of a issue or PR. You can also chat to folks on issues in realtime.
Just pushed a debian-based K8s-optimized image as well, it's 150MB larger than the alpine one, but on the other hand it comes with jemalloc support so it's nicer on the RAM instead.
CLI federation tester
kai is someone with no fear of rapid iteration! Versions 1.2, 1.3, 1.4 and 1.5 of his script were made available this week:
jcgruenhage has been working on a bot in relation to the startup he's working with:
I made a bot for issuing Json Web Tokens to matrix users based on what homeserver they are on. It's implemented as a maubot plugin (best way to write bots right now) and licensed under the AGPLv3. You can find it over at https://gitlab.com/famedly/bots/jwt, and talk about it in #jwt-bot:famedly.de
As for the usecase, we want to give people from customer homeservers access to an API component, but don't want to maintain a separate account database. This way, they can request a token from the bot and then use that to interact with the API component
Today we release Synapse 1.2.1 as a critical security update. It contains patches relating to redactions and event federation. The patches address long standing bugs, and are not regressions specific to the previous version (1.2). All admins, regardless of current version, should upgrade asap.
This release includes four security fixes:
Prevent an attack where a federated server could send redactions for arbitrary events in v1 and v2 rooms. (#5767)
Prevent a denial-of-service attack where cycles of redaction events would make Synapse spin infinitely. Thanks to @lrizika:matrix.org for identifying and responsibly disclosing this issue. (0f2ecb961)
Prevent an attack where users could be joined or parted from public rooms without their consent. Thanks to Dylanger for identifying and responsibly disclosing this issue. (#5744)
Fix a vulnerability where a federated server could spoof read-receipts from
users on other servers. Thanks to Dylanger for identifying this issue too. (#5743)
Additionally, the following fix was in Synapse 1.2.0, but was not correctly
identified during the original release:
It was possible for a room moderator to send a redaction for an m.room.create event, which would downgrade the room to version 1. Thanks to @/dev/ponies:ponies.im for identifying and responsibly disclosing this issue! (#5701)
We had a great demo session this afternoon, unfortunately the recording quality for some of the items was not good enough to share - and it was interesting stuff too! I cut everything from the RiotX and Riot iOS sections - but we'll make it up to you by getting more news from those projects soon.
Dept of Status of Matrix 🌡
Opportunities available for paid Matrix-related work
Matthew informed the community:
We're getting more and more folks reaching out for paid help setting up smaller self-hosted Matrix deployments. The core team has its hands full currently with helping out with larger deployments; so if anyone reading this wants to offer paid support to those getting up and running on Matrix then please make us aware so we can potentially route inquiries to you.
This is a great step for Matrix, and a great opportunity for you the reader! There are already several people prepared to start work on Matrix-related endeavours, but the fact that there is more work coming in than we can currently route is amazing. If you'd like to be on an informal shortlist, contact support@, or come chat to me directly.
We’ve also been having some success in improving database performance and hope to ship that rsn so everyone can benefit.
Next week
We’ll continue on db perf, improving logging verbosity (through recategorising some log lines) and generally looking to improve the experience for those admins running on smaller instances.
The GSoc project “Matrix Visualisations” has continued its progression during the second period:
A “multi-view” has been implemented for the frontend application. It means that it is now possible to independently view multiple DAGs at the same time in the same canvas. It can be useful for observing the same DAG but on different HS’s, at the same time. The only current limitation is that you have to use the same backend for every view (with CS API or with the backend talking to a Synapse PostgreSQL database). This change took a long time as it required to make a lot of changes in the code.
The implementation of the support of the Federation API within the backend is in progress. It is slowly progressing as it needed some discussion before starting the work on it and its implementation requires me to become familiar with a lot of new things, regarding the federation process and the authentication of requests between HS’s.
Hey, I'd like to put out a call to everyone out there. If there's any script people desire for interacting with matrix homeservers, the #matrix-shell-suite:matrix.org project is taking feature requests.
I had the code laying around for quite a while, but I just completed the Olm SAS interface for my fork of RubyOlm. It also includes interfaces for generating the emoji! 😀
Dept of Clients 📱
Pattle 0.10.0
I've been playing with Pattle 0.10 and have to say, it feels silky smooth to scroll with. It's really making me look again at Flutter development.
A new version has been pushed to F-droid and TestFlight!
Note that for TestFlight the new version will be available in a few days.
Improve scrolling through chat messages!
Messages are now paginated under the hood, before the whole message list was rebuild when loading more messages, causing a jittery scrolling experience.
Support typing notifications!
Typing notifications are shown and send while typing.
Add chat settings screen
Currently you can see the chat name, avatar, description and members. More will be added in the future.
You can go to the chat settings screen by clicking on the title of the chat. This'll probably change in the future (at least to be easier).
Reverse swipe direction of images (thanks Nathan!)
Fix ripple not showing on send button
Make UI refreshing more efficient
The App Store (iOS) name is now 'Pattle' instead of 'Pattle IM'
When a chat is open, the UI will only be rebuild if that chat has updates.
Lot's of code clean ups and improvements, mostly in the SDK
What's coming up?
A release on Google Play!
A release on the official F-droid repo!
Remove Cupertino specific styling on iOS
Will now use Material Design, but altered for iOS. Having the discrepancy between Cupertino and Material was also the reason why some things didn't show up in iOS or why some things would be blue instead of the Pattle red.
Pantalaimon got a new release. This release brings experimental support for search in E2E encrypted rooms, performance improvements in the database department, better support for true headless mode for all the bot lovers, and of course the usual bug fixes and small improvements.
SolarDrew and I have been working on re-writing our Picard Bot for a conference next week. Picard helps you bridge between a slack team and a matrix community.
This second version of Picard adds support for reacting to creation of new rooms on slack in real time as well as implementing a set of commands for creating new bridged rooms from either matrix or slack. In addition to this, Picard has matrix commands for inviting you to all the bridged rooms in the community and automatically inviting you to new rooms as they are created. Finally, we have added support for sending welcome messages to all new members of your slack team or matrix community in a DM with the bot, which we are using to explain the chat setup to users and giving them a private place to issue commands.
This is uses matrix-appservice-slack for actually sending events in bridged rooms, and PIcard itself is a skill for the opsdroid bot framework #opsdroid-general:matrix.org and uses the matrix-database plugin for opsdroid which uses matrix room state to back the opsdroid memory.
mx-puppet-bridge, new bridge: mx-puppet-instagram
The work on mx-puppet-bridge things continue! The core library and the implementation received some bugfixes, however, a new bridge is there: mx-puppet-instagram. Currently it supports
It's all coming together now. While all pieces of the puzzle were roughly there, now they are put into the big picture.
The bridge SDK PR has been merged to develop, now providing bridges the benefit of error reporting. The modifications to the SDK are non-breaking, but activation of the feature requires small changes to the bridge using it. But be warned, everything is still under an unstable prefix and rightly so – the implementation is still subject to change.
>
The MSC didn't get much love in public, but the concepts required to evolve it are taking shape. It was contemplated to piggyback on MSC 1410: Rich Bridging and to add a retry mechanism. So heads up for more to come on this front.
The display of bridge errors in Riot Web is now a PR and in the state of getting its last few kinks removed. It will start to be available under a lab flag soon.
With everything getting to play together nicely, there will be the first rooms enabling bridge errors to test the code in the wild. Stabilizing the code for everyday use and getting the MSC into a respectable form are the next goals for the project. Those will help in fostering broader adoption among bridges and clients.
Feel free to ask me questions about ma1sd in #ma1sd:ru-matrix.org
Final thoughts 💭
kitsune: "the latest versions of libQMatrixClient (0.5.2 - still under the old name) and Quaternion (0.0.9.4) are now available in Debian unstable, thanks to uhoreg"
That's all I know 🏁
See you next week, and be sure to stop by #twim:matrix.org with your updates!
Hey hey, Synapse 1.2.0 is here. It contains aggregations support, better error handling for deactivated accounts and some important bug fixes for redacting messages. Special thanks to community members skalarproduktraum and Lrizika for submissions to improve our documentation.
Remove support for the invite_3pid_guest configuration setting. (#5625)
Internal Changes
Move logging code out of synapse.util and into synapse.logging. (#5606, #5617)
Add a blacklist file to the repo to blacklist certain sytests from failing CI. (#5611)
Make runtime errors surrounding password reset emails much clearer. (#5616)
Remove dead code for persiting outgoing federation transactions. (#5622)
Add lint.sh to the scripts-dev folder which will run all linting steps required by CI. (#5627)
Move RegistrationHandler.get_or_create_user to test code. (#5628)
Add some more common python virtual-environment paths to the black exclusion list. (#5630)
Some counter metrics exposed over Prometheus have been renamed, with the old names preserved for backwards compatibility and deprecated. See docs/metrics-howto.rst for details. (#5636)
It was drawn to our attention this afternoon that there is a bug in our GDPR data portability tooling that resulted in the data dump including some events that should not have been included.
This tooling has recently been updated (here is the new code), and the bug only affects reports generated with the updated tool. So far we have generated one report using the updated tooling.
The bug affects events which:
were sent in rooms in which, at the point at which the message was sent, the message visibility was set to 'shared' or 'world readable', and
were pulled in over federation from another server after the data subject left the room
As a reminder, 'shared' message visibility means anyone in the room can view the message, from the point in time at which visibility was set to 'shared' and 'world readable' means anyone can read the messages without joining the room, from the point in time at which visibility was set to 'world readable'.
Events are pulled onto a homeserver over federation when a user on that homeserver tries to access events which, for whatever reason, their homeserver does not already have a local copy. This most often happens when their homeserver is offline for any period of time, but can also happen when a user is the first user from their homeserver to join a room with active participants on other homeservers.
We're still analysing the data but so far it looks like the bug resulted in only a small number of events that were not publicly-accessible being shared (there were also publicly-accessible events mistakenly included). At this stage we have identified 19 events from 4 users across 2 rooms (the dump contained ~3.5 million events). This is not to diminish the severity of the bug - just to reassure that the scale of its impact appears to be extremely limited.
It is also worth noting that any encrypted events erroneously included in the dump will not have been decryptable (since the data subject would not have had access to the keys).
Update (2019-08-06)
In our original analysis we stated that 19 events were shared erroneously. On closer analysis we missed 5 other timeline events - the correct figure is 24 timeline events originating from 4 users over 2 rooms. However, this figure focused on timeline data and does not take into account all state events (such as user joins, parts, topic changes etc). When considering these too, a further 56 state events were erroneously shared, referencing 64 users across these 2 rooms (mainly detailing when users had joined/left the room after the requesting user themselves had left). These membership events contained avatar & display name details which may not have been public (but in practice, the vast majority appear to be public data).
Aside from the events referenced above, the full dump contained ~20,000 events that also ought not to have been included; however these events were already publicly accessible due to being part of publicly accessible rooms (eg Matrix HQ) and so we do not consider them a breach of data.
What caused the bug?
Events that are pulled in over federation are assigned a negative 'stream ordering' ID. This is designed to avoid their being sent down the sync (where they would likely be out of sequence). In normal operation (accessing your homeserver via a Matrix client) these events would be appropriately filtered, but a bug in the data dump tooling caused them to be included.
The bug was introduced as a result of two factors:
The event filtering code assumes that the user is currently in the room - this was not intuitive, and was not called out in the documentation
When we fetched the events from the database, we tried to limit to events sent before the user left the room. On reflection, we used the wrong ordering mechanism (stream ordering instead of topological ordering), resulting in the inclusion of events that were fetched from a remote server after the data subject had left
We are working to fix the bug, and we'll update here when it is resolved. As a reminder, please do report security bugs responsibly as per the Security Disclosure Policy so we can validate the issue and mitigate abuse.
As is standard practice for any data breach, we have notified the ICO.
