That's right folks Synapse 1.5.0 is here and ready to make your life just a little bit better.

First things first, this release includes a security fix (#6262, below). Administrators are encouraged to upgrade as soon as possible.

Aside from that, the main thing you'll notice in 1.5.0 is a massive performance improvement to the room directory, which means that servers with large directories to scan will return much more quickly. This is especially true for matrix.org but all servers will benefit.

Another key win is finally fixing some bugs in the sqlite -> postgres migrator script. Sqlite mode is there strictly for testing purposes and should never be used in a production setting let alone a federating homeserver. So if you are currently using Sqlite now is the time to migrate. What's more the script is now in CI so it can't easily break in the future (with apologies to anyone bitten by the old script...).

A final final point, we have some preparatory work for e2ee cross signing, the overall feature is not ready for release just yet but we are getting really close. Watch this space.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Also, check out our Synapse installation guide page

The changelog since 1.4.1 follows:

🔗Synapse 1.5.0 (2019-10-29)

🔗Security updates

This release includes a security fix (#6262, below). Administrators are encouraged to upgrade as soon as possible.

🔗Bugfixes

• Fix bug where room directory search was case sensitive. (#6268)

🔗Synapse 1.5.0rc2 (2019-10-28)

🔗Bugfixes

• Update list of boolean columns in synapse_port_db. (#6247)
• Fix /keys/query API on workers. (#6256)
• Improve signature checking on some federation APIs. (#6262)

🔗Internal Changes

• Move schema delta files to the correct data store. (#6248)
• Small performance improvement by removing repeated config lookups in room stats calculation. (#6255)

🔗Synapse 1.5.0rc1 (2019-10-24)

🔗Features

• Improve quality of thumbnails for 1-bit/8-bit color palette images. (#2142)
• Add ability to upload cross-signing signatures. (#5726)
• Allow uploading of cross-signing keys. (#5769)
• CAS login now provides a default display name for users if a displayname_attribute is set in the configuration file. (#6114)
• Reject all pending invites for a user during deactivation. (#6125)
• Add config option to suppress client side resource limit alerting. (#6173)

🔗Bugfixes

• Return an HTTP 404 instead of 400 when requesting a filter by ID that is unknown to the server. Thanks to @krombel for contributing this! (#2380)
• Fix a bug where users could be invited twice to the same group. (#3436)
• Fix /createRoom failing with badly-formatted MXIDs in the invitee list. Thanks to @wener291! (#4088)
• Make the synapse_port_db script create the right indexes on a new PostgreSQL database. (#6102, #6178, #6243)
• Fix bug when uploading a large file: Synapse responds with M_UNKNOWN while it should be M_TOO_LARGE according to spec. Contributed by Anshul Angaria. (#6109)
• Fix user push rules being deleted from a room when it is upgraded. (#6144)
• Don't 500 when trying to exchange a revoked 3PID invite. (#6147)
• Fix transferring notifications and tags when joining an upgraded room that is new to your server. (#6155)
• Fix bug where guest account registration can wedge after restart. (#6161)
• Fix monthly active user reaping when reserved users are specified. (#6168)
• Fix /federation/v1/state endpoint not supporting newer room versions. (#6170)
• Fix bug where we were updating censored events as bytes rather than text, occasionally causing invalid JSON being inserted breaking APIs that attempted to fetch such events. (#6186)
• Fix occasional missed updates in the room and user directories. (#6187)
• Fix tracing of non-JSON APIs, /media, /key etc. (#6195)
• Fix bug where presence would not get timed out correctly if a synchrotron worker is used and restarted. (#6212)
• synapse_port_db: Add 2 additional BOOLEAN_COLUMNS to be able to convert from database schema v56. (#6216)
• Fix a bug where the Synapse demo script blacklisted ::1 (ipv6 localhost) from receiving federation traffic. (#6229)

🔗Updates to the Docker image

• Fix logging getting lost for the docker image. (#6197)

🔗Internal Changes

• Update user_filters table to have a unique index, and non-null columns. Thanks to @pik for contributing this. (#1172, #6175, #6184)
• Allow devices to be marked as hidden, for use by features such as cross-signing. This adds a new field with a default value to the devices field in the database, and so the database upgrade may take a long time depending on how many devices are in the database. (#5759)
• Move lookup-related functions from RoomMemberHandler to IdentityHandler. (#5978)
• Improve performance of the public room list directory. (#6019, #6152, #6153, #6154)
• Edit header dicts docstrings in SimpleHttpClient to note that str or bytes can be passed as header keys. (#6077)
• Add snapcraft packaging information. Contributed by @devec0. (#6084, #6191)
• Kill off half-implemented password-reset via sms. (#6101)
• Remove get_user_by_req opentracing span and add some tags. (#6108)
• Drop some unused database tables. (#6115)
• Add env var to turn on tracking of log context changes. (#6127)
• Refactor configuration loading to allow better typechecking. (#6137)
• Log responder when responding to media request. (#6139)
• Improve performance of find_next_generated_user_id DB query. (#6148)
• Expand type-checking on modules imported by synapse.config. (#6150)
• Use Postgres ANY for selecting many values. (#6156)
• Add more caching to _get_joined_users_from_context DB query. (#6159)
• Add some metrics on the federation sender. (#6160)
• Add some logging to the rooms stats updates, to try to track down a flaky test. (#6167)
• Remove unused timeout parameter from _get_public_room_list. (#6179)
• Reject (accidental) attempts to insert bytes into postgres tables. (#6186)
• Make version optional in body of PUT /room_keys/version/{version}, since it's redundant. (#6189)
• Make storage layer responsible for adding device names to key, rather than the handler. (#6193)
• Port synapse.rest.admin module to use async/await. (#6196)
• Enforce that all boolean configuration values are lowercase in CI. (#6203)
• Remove some unused event-auth code. (#6214)
• Remove Auth.check method. (#6217)
• Remove format_tap.py script in favour of a perl reimplementation in Sytest's repo. (#6219)
• Refactor storage layer in preparation to support having multiple databases. (#6231)
• Remove some extra quotation marks across the codebase. (#6236)

🔗Dept of Status of Matrix 🌡

🔗Matrix.org

Public service notice from vdh:

It won't have escaped many peoples' notice that the matrix.org homeserver has been struggling over the last few weeks. The main problem is poor disk I/O performance on our database server: for various reasons, it seems that the server can no longer keep up with the demands we're putting on it.

The good news is that we have a plan to sort it out, and we're working on setting up alternative hosting which will be able to handle our traffic for the foreseeable future! Please bear with us over the next couple of weeks while we get new servers set up.

🔗Matrix Notepad

@kb1rd:kb1rd.net told us:

Just released the first version of the Matrix Notepad, a small webapp designed to allow people to collaborate on text files over Matrix. There are a bunch of bugs that I'm still working on fixing, but if you'd like to check it out it's here! I will be adding incremental fixes for various issues, but feel free to add to the issues. :) If you'd like to chat, I'll be checking #matrix-collaboration:kb1rd.net.

🔗Dept of Servers 🏢

🔗Synapse

@richvdh:sw1v.org told us:

This week much of the team's focus has been on performance, both in terms of dealing with matrix.org's immediate woes and some more strategic work on helping Synapse to scale. On the latter front, Erik has been making great progress in supporting multiple Postgres databases.

We've also put out a release candidate for Synapse 1.5.0 with lots of bugfixes and changes under the hood. As ever, help with testing the RC is much appreciated!

🔗Dept of Bridges 🌉

🔗matrix-appservice-discord v0.5.2 released

Half-Shot reported:

matrix-appservice-discord has had a new release (v0.5.2) to fix some out of date packages.

🔗matrix-appservice-bridge v0.11.0

Half-Shot announced:

matrix-appservice-bridge has had a new release (v0.11.0) which allows developers to disable usage of the stores entirely. This should solve problems facing some bridges which no longer use them.

🔗mautrix-hangouts image bridging

Tulir offered:

Cadair's pull request to add Hangouts->Matrix image bridging to mautrix-hangouts was merged.

🔗Dept of Clients 📱

🔗miitrix

From sorunome, this crazy 3DS project!

Miitrix is a client for the Nintendo 3DS, using the matrix-3ds-sdk! Here is an early demonstration video. The current featureset includes:

• Sorting room list by last messages
• Room name fallbacks (canonical alias, joined members)
• sending messages
• receiving messages
• receiving edits & redactions
• receiving join, leave, ban events
• lazy-loading extra information in the background

Support room: #miitrix:sorunome.de Donate: https://liberapay.com/Sorunome

🔗Riot web

Tulir announced:

Two of the Riot pull requests that I twimmed last week, the reaction emoji picker and edit html parser improvements, have been merged into develop. The remaining one (reply rendering improvements) still needs some code and design work.

🔗Ruby Matrix SDK version 1.5.0 released

Ananace said:

The Ruby SDK is out with another new version, 1.5.0 this time. Adding an event and a check to let applications track and handle errors that occur in the background listener thread of the client abstraction. This version also exposes a setter for the open_timeout value on the lower-level API client (and the client abstraction through it), for users that are plagued by slow-to-open TCP connections to their relevant HSes.

As always, feel free to drop by #ruby-matrix-sdk:kittenface.studio if you have questions, comments, corrections, or just want to discuss the project - or use cases - in general.

🔗Fractal

Alexandre Franke reported:

Since the 4.2.1 release a month ago, there’s been a slow but steady stream of commits including code cleanups, translation updates and minor bugfixes. The highlights are issues solved around settings not being reloaded after verification dialog was closed, account settings not loading, connections to http only servers failing, and a regression that prevented users from sending messages to newly created rooms.

🔗Riot iOS

Manu reported:

We are still working on 2 fronts:

• fixing most important issues in our stabilisation sprint
• privacy project: Adding an email or phone number is now protected by a password (if your homeserver requests this authentication)

🔗Riot Android

Manu offered:

• privacy project: Adding an email or phone number is now protected by a password (if your homeserver requires it)

🔗RiotX Android

Manu offered:

• We released 0.7.0 this week. You now have read markers and can share content to Riot. You should also find the app even faster.

🔗Dept of SDKs and Frameworks 🧰

🔗matrix-3ds-sdk

sorunome offered:

A lot has happened on the side of matrix-3ds-sdk. So much so, that it is possible now to write clients out of it! The changelog includes:

• fetching member info
• fetching room info
• proper sync loop
• aggressive filters to reduce data (initial sync would crash on large accs otherwise)
• room leave and invite callbacks
• room info callback

Support room: #matrix-3ds-sdk:sorunome.de

🔗Dept of Ops 🛠

🔗Helm charts for mautrix-telegram, mautrix-twilio and maubot

Tulir offered:

mautrix-telegram and mautrix-twilio now have experimental Helm charts: https://github.com/tulir/mautrix-telegram/tree/helm/helm/mautrix-telegram. There's no replication/horizontal scaling stuff, but they should be useful if you're already using Helm for other stuff.

To make managing maubot plugins possible with Helm, I added a simple standalone plugin executor to maubot. It's basically just a simple script that sets up all the things plugins want (matrix client, config, database, webserver) and passes them to the plugin.

The standalone executor can be used to run any maubot plugin without any changes to the plugin itself, but building good docker images needs to be done separately for each plugin. The supportportal plugin has a standalone docker image and a helm chart using that image.

🔗Dept of Ping 🏓

🔗That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

🔗Matrix Live 🎙

🔗Dept of Status of Matrix 🌡

Last week Matrix had a presence at UbuCon Europe and PyCon Ireland. We gave workshops on using Matrix to create bots, and also a session on installing Synapse (see also: Brendan's entry below!)

Hey #PyConIE! We hope you enjoyed the Matrix workshop, find the slides and code samples at https://t.co/aT1dDuEcm1 and make sure to ask any more questions! pic.twitter.com/C2qwV5R3gl

— Matrix (@matrixdotorg) October 12, 2019

Tomorrow I'm off to sunny Manchester with Michael from the Ops team. We'll visit OggCamp, where we'll show off Matrix with a fun demo I previewed on Matrix Live a few weeks ago.

Matrix will ALSO be at:

• Re­decentra­lize Conference 2019
• MozFest 2019

🔗Dept of Spec 📜

anoa:

Last week we set MSC1219 (key backups), MSC2241 (verification over DMs), and MSC2313 (ban lists) for the spec core team to focus on. Those 3 are rolling on into this week as we didn't get a lot of work done last week :)

In other Spec news, Matthew uncovered a stone tablet describing what would in future be known as "MSC2324":

🔗Dept of Servers 🏢

🔗install-party, nice result from UbuCon Europe

Brendan said:

For a workshop anoa and I did at Ubucon Europe last Sunday on how to install Synapse, I worked on a side project that creates a server, attaches a domain name to it and installs Riot and Caddy on it. Attendees can then SSH into it, follow the instructions to install a Matrix homeserver, and use Riot to register an account on it, log in, and join a federated room with all of the other attendees' homeservers in it. We tried it out last week-end for the actual workshop and it worked quite well 🙂 The project is called Install Party, and lives at https://github.com/babolivier/install-party, and if you want to chat about it I've just created #install-party:abolivier.bzh 😉

🔗Synapse

Neil reported:

This week we've been thinking about the future and brainstorming on ideas to improve perf for small instances and sparing some cycles for MSC1228. Next week we'll return to improving IO usage on matrix.org.

Synapse 1.4.1 was released, which fixes a small regression in 1.4.0.

🔗Dept of Bridges 🌉

🔗Matrix-Slack Bridge on NixOS

@christian:kampka.net said:

Just FYI, the matrix slack bridge is now available via NixOS: https://github.com/NixOS/nixpkgs/pull/70753

🔗Dept of Clients 📱

🔗riots.im available via IPFS

Remember https://riots.im from last week? Now available on IPFS!

@swedneck:permaweb.io said:

toml's riots.im is also available via IPFS: ipfs get riots.im or visit localhost:8080/ipns/riots.im with ipfs installed and running.

🔗Riot-iOS 0.10.0

Manu offered:

On Riot-iOS land, Riot-iOS 0.10.0 has been released on the App Store. We have started a stabilisation sprint. In parallel, we are still polishing the privacy work

🔗RiotX release next week

benoit announced:

RiotX: we are merging waiting PRs and have postponed the release to ensure a maximum of stability and polishing. The release will be done at the beginning of next week and will contains: read marker, camera picker and improved file picker, share to RiotX capability and many bugfixes.

🔗Dept of SDKs and Frameworks 🧰

🔗matrix-3ds-sdk

sorunome said:

Heya, matrix-3ds-sdk is a new matrix sdk - for the Nintendo 3DS! It is still deep in development, so expect bugs, some things not working properly etc. Current features include:

• logging in
• sending text messages to rooms
• redacting events from rooms
• resolving aliases to room ids
• basic framework to get a /sync loop working

If you are into 3DS homebrew development or are interested in helping out making a full client based on this, please contact soru (@sorunome:sorunome.de)! 3ds homebrew dev is pretty new to her, so there are plenty of open questions / debugging help would be great!

Support room: #matrix-3ds-sdk:sorunome.de Donate: https://liberapay.com/Sorunome

This is really impressive, I'm looking forward to playing with it. (I'm sure someone at the office must own a 3DS!)

🔗koma

yuforia offered:

koma, Kotlin library for building clients: Added API for retrieving notifications.

🔗Dept of Ops 🛠

🔗mvgorcum/docker-matrix

Mathijs announced:

I uploaded the synapse 1.4.1 image to mvgorcum/docker-matrix:v1.4.1 within about half an hour after the release

🔗docker image of synapse-develop

Mathijs announced:

I've made a docker image of the develop branch of synapse, and automated it to build daily. It's on my docker hub as mvgorcum/docker-matrix:develop, based on the avhost dockerfile. Note that there are no checks, the image is simply built from the develop branch of the synapse git repo every night.

🔗Matrix Synapse for Kubernetes

Ananace offered:

just pushed Version 1.4.1 of the Matrix Synapse for Kubernetes packaging. A bit fiddly to do on a phone 🙂

🔗Synapse 1.4.1 multi-arch docker image

Black Hat told us:

I'm building the Synapse 1.4.1 multi-arch docker image. It will be pushed to Docker Hub in a couple of hours.

🔗Dept of Tulir 🇫🇮

A special section this week from the guy with one editor open for everything, tulir:

I haven't done anything on my own projects this week, but I did contribute to a bunch of other projects:

• Updated SmsMatrix to the latest matrix-android-sdk to fix outgoing sms duplication bug (https://github.com/tijder/SmsMatrix/pull/60)
• Fixed Riot web sending reply fallbacks in edited message content (https://github.com/matrix-org/matrix-react-sdk/pull/3551)
• Fixed some things in the Riot web edit html -> markdown parsing (https://github.com/matrix-org/matrix-react-sdk/pull/3552)
• Made Riot web reply rendering much nicer and more compact (https://github.com/matrix-org/matrix-react-sdk/pull/3553)
• Added full emoji picker for reactions to Riot web (https://github.com/matrix-org/matrix-react-sdk/pull/3554)

The first two are already merged (and SmsMatrix even got a new release on f-droid), the html parsing is waiting for code review and the emoji picker and reply rendering are waiting for design review.

Also, I made a read-only status.matrix.org rss feed room since some people wanted one: #matrix.org-status:maunium.net. I don't remember if I TWIMed these before, but #xkcd:maunium.net and #commitstrip:maunium.net are similar read-only rooms, new xkcds and commitstrips are posted there whenever they come out.

🔗Dept of Ping 🏓

🔗That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Hi all,

We've released Synapse 1.4.1 as a small but important bugfix to 1.4.0.

This fixes a regression which crept in with our newly implemented "erase redacted data after N days" feature where some APIs would fail when hitting erased redactions - anyone on Synapse 1.4.0 will want to upgrade asap.

As always, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Also, check out our Synapse installation guide page

The changelog since 1.4.0 follows:

🔗Synapse 1.4.1 (2019-10-18)

No changes since 1.4.1rc1.

🔗Synapse 1.4.1rc1 (2019-10-17)

🔗Bugfixes

• Fix bug where redacted events were sometimes incorrectly censored in the database, breaking APIs that attempted to fetch such events. (#6185, 5b0e9948)

🔗Dept of Status of Matrix 🌡

As you may have heard, New Vector, a major entity in the Matrix world, recently completed a series A funding round.

Further reading:

• $8.5M to accelerate Matrix, polish Riot and expand Modular! (from New Vector)
• Why I invested in New Vector (from Notion Capital)
• Communications of the future will be open and secure: why we are partnering with New Vector (from Dawn Capital)
• Why we invested in New Vector — the team building an open and secure communication platform (Sam Endacott from firstminute capital)

🔗Mozilla IM trial ended

Mozilla have been trialing different IM solutions to replace IRC, including Matrix. This trial ended this week, and we hope to hear the results this month. (C'mon, Matrix!)

🔗Dept of Servers 🏢

🔗Synapse

Neil told us:

Matrix.org hit some IO problems earlier this week, while largely a problem with our hosting provider, we’re spending a bit of time to make Synapse more resilient if the same thing were to happen again. This will mean the ability to shard the DB (by table) and spread the load so we are not so dependent on high performance from a given db box. Outside of that we’ve been working on the final polishing of the room directory and getting the sqlite -> Postgres port script into better shape.

🔗multi-arch docker image of Synapse

Black Hat said:

My multi-arch docker image of Synapse has been updated to v1.4.0.

🔗Dendrite

anoa offered:

Not much for Dendrite this week as anoa is off at Ubucon 2019. But we had a few valuable bugs reported by the community, and a pressing reminder to get Dendrite's Monolith mode in as part of its CI.

🔗Dept of Bridges 🌉

🔗slack bridge 1.0.1

Half-Shot told us:

Hey folks, The slack bridge 1.0.1 release is out https://github.com/matrix-org/matrix-appservice-slack/releases/tag/1.0.1 containing a few bug fixes found since the original release.

🔗IRC bridge

Half-Shot offered:

I've not got much for this week, but the IRC bridge has been undergoing some serious refactors and changes for a larger release. Should be quite a big one when it lands :)

🔗Dept of Encryption 🔐

🔗Search inside E2EE rooms

Update from poljar:

A PR for riot-web has emerged that adds support for search in E2E encrypted rooms. The PR is utilizing Seshat to perform event indexing and search on riot-desktop. While the PR is missing any sort of UI, it is in a usable state.

🔗Dept of Clients 📱

🔗Riot iOS

Manu told us:

phase:1 of privacy work is done. Riot -iOS 0.10.0 will be available soon

🔗every version of Riot Web released on GitHub

toml offered:

Take a trip down memory lane with the Riots of yesteryear at https://riots.im (note the 's'). Hosting every version of Riot Web released on GitHub 😁

Comes with free Wikipedia hole.

🔗Riot Android

benoit announced:

Valere has done a release and is doing some maintenance. He has started to work on integration manager

🔗RiotX

benoit offered:

RiotX: We have fixed quite a lot of issues during the stabilization sprint. We are now working on Sprint 4: read marker, report content, mark all room read, etc. François is changing the media/file picker and we will also be able to share elements from other apps to RiotX. We will schedule a release soon (tm) (should have happen this week, but has been delayed due to stabilization)

🔗Dept of SDKs and Frameworks 🧰

🔗Elixir projects from uhoreg, Polyjuice and Igor

uhoreg told us:

Polyjuice Client, a Matrix client library for Elixir, has a new release. There is now a short tutorial that will teach you how to make a simple echo bot with it.

Then:

Igor, a bot framework for Elixir, has had its first release.

🔗Dept of Ping 🏓

🔗That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Hi all,

Massive news for the Matrix ecosystem today: New Vector (the startup which the Matrix core team formed to fund development in 2017) has raised an additional $8.5M of funding in order to speed up Riot/Matrix development and expand Matrix hosting via Modular.im!

The new funding comes in the form of a Series-A equity investment in New Vector from three of the top venture capital funds in London. The round is led by Notion - a fund set up by the founders of MessageLabs, who many will know as one of the leaders in secure hosted email services. Notion's long history with email means they immediately clocked the potential of Matrix's mission to build a new open global communication network - after all, Matrix aims to provide a worthy replacement to email (and the phone network, for that matter!). Joining Notion in the round is First Minute - a fund set up by the founders of Lastminute.com (arguably the UK's most famous original dotcom), and Dawn - one of the largest SaaS tech specialist funds in Europe (famous for backing iZettle, Mimecast, Neo4J and many more).

The last funding round in Jan 2017 from Status was instrumental in stabilising the big 1.0 release of Matrix and exiting beta in June; creating the Matrix.org Foundation as a neutral custodian for the standard; stabilising and optimising Synapse; redesigning Riot’s user interface; bringing in a full-time professional UI/UX designer to the team; supporting the huge amount of encryption work required to turn on E2EE by default (cross-signing, key backups, device verification, e2e search, the pantalaimon e2e daemon etc); creating RiotX/Android; and launching the Modular.im hosting platform.

With today’s new funding, the priorities for Matrix will be:

• Turning on end-to-end encryption by default for DMs

• Much better support for grouping rooms into Communities

• More anti-abuse/anti-spam mechanisms

• Shrinking Synapse (and/or finishing Dendrite)

• Canonical DMs (having one DM per user, and have them feel clearly distinct from ‘rooms’)

• Extensible Profiles

• Decentralised accounts

• Threading

• ...and furthering development on P2P Matrix, so users can have full control of their communications without having to run or trust a server.

On the New Vector side, this funding will support:

• A whole new wave of UX improvements to Riot (particularly around onboarding and first time user experience).

• Making Modular hosting as polished and powerful as possible.

• Creating a whole new set of next-generation Modular integrations.

While New Vector’s contributions to the Matrix ecosystem can’t be ignored, it’s important to remember that the Matrix protocol and specification itself is governed and controlled by the independent and neutral Matrix.org Foundation and its extensive governance processes. We set up the Foundation very deliberately to enforce the protocol's neutrality, formalise the project's mission, goals and values and hold true to them no matter what - specifically to protect the project from conflicts of interest with commercial Matrix endeavours, including New Vector.

That said, New Vector would not be taking money from any investors if they did not believe their goals are aligned with Matrix's. To clarify:

• Matrix exists to create an open secure decentralised communication network and protocol for the benefit of all.
• New Vector exists to help grow Matrix and be one of many successful companies in the Matrix ecosystem.
• Tech VCs exist to invest their money in growing companies in order to get a return when the company IPOs or gets bought.

It turns out that these goals are not incompatible if one understands that the potential of the Matrix ecosystem is directly linked to its openness and size (hint: funding sources who didn’t understand this self-selected out ;). By funding Matrix development and helping the open ecosystem and public network grow, New Vector can go provide more Matrix hosting via Modular.im and more Government & Enterprise deployments via Vector.im. Critically, other companies can and do build on top of Matrix too - and frankly the more players there are, the more valuable the network, and the more value to be shared for everyone (including New Vector). This model worked relatively well for the Web, and we believe it'll work for Matrix too.

Update: the best way to gauge the investment in New Vector is to hear it first hand from the investors. Jos from Notion is leading the round, and has a fascinating blog post (written with zero input from Matrix or New Vector folks) to explain where the investors are coming from.

Update 2: More excellend first-hand analysis from Dan at Dawn Capital, who does a really deep dive into how they see Matrix and New Vector. Another must read.

In this case, all of New Vector's new investors have a background as respected tech entrepreneurs, and everyone involved categorically understands that Matrix itself is a neutral open source project, and the mission is to help build up the whole network to be as successful as possible rather than sabotage it by constraining it in any way.

All in all, it’s great news for the ecosystem: Matrix is 5 years old now, and while the project is growing faster than ever (over 300% more active users in the last year!) - it's fair to say that we haven't moved as fast as our mainstream competition - for instance, Slack is only a year older, and Discord is a year younger(!) Obviously much of this is due to Matrix being a completely different proposition: we've been creating an open spec; multiple client codebases; multiple server codebases; the bridges; a fault tolerant decentralised network - not to mention the complexities of decentralised E2E encryption. Based on comparing with our endeavours prior to Matrix, we estimate building this stuff in an open and decentralised manner takes roughly 6 times longer.

But the project is now in a position where the foundations are solid: the protocol is out of beta, reference servers and clients are production ready, and it’s more than time to make all of this mainstream. We have to redouble our focus on user experience and ensure that we compare favourably to today’s established alternatives while staying true to Matrix’s principles. Making sure there are Matrix apps out there which provide a credible alternative to with the likes of Slack and WhatsApp (until they eventually join Matrix, of course) is what will make the difference between Matrix being a cliquey FOSS curiosity versus really being the natural successor to today’s instant messaging, email and phone networks.

In the end; Matrix needs full-time contributors in order to continue to grow, and keeping New Vector funded is a very good way to achieve that (New Vector is hiring!). (That said, if any philanthropic billionaires are reading this, the Matrix.org Foundation is actively soliciting donations to improve Matrix independently of New Vector's efforts - particularly around the areas of countering online abuse and disinformation).

In the meantime, huge thanks to Jos at Notion for believing in Matrix and leading this funding round in New Vector - and huge thanks to the other investors who saw the potential! And most of all, thank you to all those supporting Matrix, whether by donating to the Foundation, promoting and using the protocol, or contributing code to the ecosystem. You are the ones keeping the dream alive :)

You can read things from the NV angle over at https://blog.vector.im/8-5m-to-accelerate-matrix/. We hope you’re as excited as we are to open a whole new chapter as Matrix picks up yet more momentum :D

-- Matthew, Amandine, and the whole Matrix team

🔗Matrix Live 🎙

• Ubucon https://sintra2019.ubucon.org/
• PyCon Ireland https://python.ie/pycon-2019/
• OggCamp https://oggcamp.org/
• MozFest https://www.mozillafestival.org/en/
• and more, watch the vid

🔗Dept of Spec 📜

Two MSCs have reached final comment period this week:

• MSC 2176: Update the redaction rules
• MSC 2290: Separate Endpoints for Threepid Binding

Last week the spec core team said they'd start focusing on 3 MSCs per week. Those were MSC2290, MSC2176 and MSC1219. The first two have entered FCP, and MSC1219 will roll over into this week.

Thus, the 3 MSCs the spec core team will be focusing on this week will be MSC2199, MSC1219 and an up and coming security-related MSC. Join us in #matrix-spec:matrix.org for related spec discussions :)

There is also a new MSC, MSC2312 describing the proposed Matrix URI scheme. This is a remake of a much older proposal. The general idea is to make a standard for Matrix URIs: matrix:.

🔗Dept of Servers 🏢

🔗Synapse 1.4.0 released

Neil offered:

Synapse This week we shipped our privacy release 1.4.0 which is a huge deal in improving user data privacy. Additionally we also included a significant perf improvement which will help anyone suffering from a build up in forward extremities.

Coming up, improved room directory perf, ironing out wrinkles in the room upgrade UX as well as a major reliability boost in the sqlite -> Postgres db porting script.

🔗Dendrite dev recommences

anoa said:

Dendrite's latest hiatus has come to a close after the privacy work had taken so much of my time. Thankfully although PR review is blocked on the dendrite team, the community have continued to submit PRs and even review each other's PRs (thanks cnly !).

Fixes this week have mostly focused around the CI. We finally got Dendrite's CI unborked (t'was borked in a half-way transition between CircleCI and Buildkite), but it's now working and faster than ever. We're also working to add the Sytest test failure results to the top of the CI window such that people can see which tests failed and the associated Dendrite logs.

Additionally we had some new and merged PRs this week! A federation fix from cnly, a change to allow Dendrite to better work in kubernetes setups by aditsachde, and a codebase fix from @manasseh:matrix.org, who's also working on another fix involving updating gomatrixserverlib to support more recent spec versions.

Some things people were asking about:

• Progress of dendrite is tracked in Dendrite's milestones. We're currently aiming for #1 (Client-Side) Bot Hosting.
• CI not running for community PRs - Buildkite hosts currently run multiple PRs from many different projects, so we can't trust arbitrary code to run on them quite yet. There's a project in progress to run the code in a sandboxed environment (think VM or container) to lift this restriction :)

🔗Dept of Bridges 🌉

🔗matrix-appservice-slack 1.0 released

Half-Shot offered:

Aaaaaaaaaaand shipped 🥳 https://github.com/matrix-org/matrix-appservice-slack (1.0 is out!)

Check out the announcement post, big congrats to Half-Shot for getting this out.

🔗Bridging animated stickers to mautrix-telegram

tulir said:

mautrix-telegram will soon be able to bridge animated stickers to matrix as images or videos, thanks to a pull request by Eramde.

🔗Dept of Clients 📱

🔗Riot Web 1.4.0

Big release for riot-web, check out their release: https://medium.com/@RiotChat/new-privacy-controls-for-riot-dc3661888563

🔗Riot Android and RiotX

From Benoit:

Riot-Android: 2 main things have landed on develop, to be released early next week:

• Catching up on riot-web new Privacy Controls (choose identity server, stun server, securely compare contacts)
• [fdroid only] A new background sync mode for notifications. You can now choose between 'optimized for battery' and 'optimize for realtime'
  • build available on buildkite

RiotX:

• Read Markers have landed on develop (jump to last read) Focus on stabilization and bug fixes

🔗Continuum

yuforia offered:

Continuum, desktop client written in Kotlin, version 0.9.26:

• Width of columns can be adjusted by dragging, this is a screenshot showing the mouse cursor placed between the first two columns.

• Fix occasional out-of-bounds array access errors while calculating bounds during layout passes.

🔗FluffyWeb: Matrix client with Flutter for Web

@krille:ubports.chat offered:

As a proof of concept, I have created a little Matrix client with Flutter for Web, named FluffyWeb. This client has only basic features but it shows the possibilities of Flutter for Web and it seems to work fine so far. The client has a responsive design and should work on mobile fine too. Check it out at: https://christianpauly.gitlab.io/fluffyweb/ (Not working in Internet Explorer - I recommend the AOL Messenger in this case)

🔗Dept of SDKs and Frameworks 🧰

🔗Ruby SDK 1.4.0

Ananace said:

Just released version 1.4.0 of the Ruby SDK, the main change in this release is the addition of a naïve set of methods to replace the logger implementation. This should allow the gem to slot more easily into projects where existing logging configuration is already in place.

🔗Dept of Ops 🛠

🔗1.4.0.Mania

With the release of Synapse 1.4.0, there was a rush to get packages and containers updated, the community are always fast, but we should acknowledge Anance for having his k8s images updated within a few minutes!

🔗matrix-docker-ansible-deploy

Slavi announced:

matrix-docker-ansible-deploy has also been updated to support Synapse v1.4.0 and riot-web v1.4.0.

As always, referring to the project's changelog before upgrading is a good idea.

KUDOS to this this project! I love that I can git pull, ansible-playbook -i inventory/hosts setup.yml --tags=setup-all without really checking and my toy homeserver is automagically updated.

🔗Kubernetes

Ananace offered:

Just pushed the Kubernetes-optimized images for Synapse version 1.4.0

🔗mathijs's docker images

@mathijs:matrix.vgorcum.com reported:

I set up my own docker-hub account to push images of RC's of synapse at mvgorcum/docker-matrix that aren't built for the avhost/docker-matrix repo

🔗Dept of Bots 🤖

🔗msc name linker bot

anoa said:

I made a bot that gives the link for msc names. Code is here: https://github.com/anoadragon453/msc-chatbot

🔗Dept of Welcomes 👐

Welcome to the JRuby team who announced they will be moving their official chat to Matrix.

🔗Dept of Ping 🏓

Let's reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

🔗Final thoughts 💭

Mastodon released v3.0.0 - check it out.

🔗That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Hello Matrix enthusiasts! Yesterday we released matrix-appservice-slack 1.0. This marks a major milestone in bridge development for the Matrix.org team, being our first bridge to ever reach 1.0. The decision to release this version came after we decided that the bridge had gained enough features and reached a point of stability where it could be deployed in the wild with minimal risk.

For those not in the know, the Slack bridge is Node.JS based, and bridges slack channels & users into Matrix seamlessly. And for those wondering, yes it works with Mattermost too (since their API is compatible with Slack)! In previous versions only a limited subset of features were supported, making heavy usage of Slack’s webhook API. As of 1.0, the bridge now makes use of the newer Slack Events/RTM API which gives us all we need for a richer bridging experience. Everything from edits and reactions to typing notifications is supported in the 1.0 release.

Finally for those who are self hosting, we are pleased to offer the ability to "puppet" your Slack account using the bridge. Puppeting is the process in which the bridge will send messages as if you were sending them from the Slack client directly, when you talk using your Matrix account. This opens the door to seamless bridging and direct messaging support.

For those wishing to bridge their whole workspace across, picard exists as a tool to manage large scale Slack bridge deployments. This tool is provided by Cadair and SolarDrew

Threading & Reactions!

The bridge has undergone some pretty serious code surgery as well. The whole codebase has been rewritten in TypeScript to take advantage of type checking and generic types. The bridge is currently based upon the matrix-appservice-bridge library. The datastore interface now supports PostgreSQL, which allows for administrators to inspect and edit the database while the bridge is running, as well as offering helpful performance boost over the NeDB datastore format that was used previously. Finally, the codebase has proper Unit and Integration Tests to ensure new changes will not cause any regressions in behaviour. In short, now is an excellent time to get involved and hack on the bridge. There is already a crafted list of easy issues for new and experienced bridge devs.

Memory usage of the bridge comparison

CPU usage of the bridge comparison

In terms of how many users matrix.org is currently serving at the moment, we present to you some figures:

• 2562 bridged rooms
• 764 teams connected to the bridge
• 103711 events have passed through the bridge since the launch of 0.3.2

Of course, our work doesn’t stop at 1.0. The plan for the immediate future of the bridge is to continue adding support for other event types coming from Matrix and Slack to create an ever richer experience. Obvious features are things like topic changes, and syncing membership across the bridge. In the long term future we would also like to add community support to the bridge, so whole Slack workspaces can be bridged across with a single click.

That’s all from me, and I would like to say a massive thank you to Cadair and Ben for both their code and review work on the project and as always, thank you to the community for using the bridge and reporting issues. 🙂

Useful links:

• Matrix Room
• Project Home
• Docs

Wheyhey as I live and breathe 1.4.0 is here!

1.4.0 should be considered ‘The Privacy Release’ and is the cumulation of a huge body of work spanning multiple projects to improve data privacy in the matrix ecosystem. You can read more about the full project here.

While we consider 1.4.0 to be a huge leap forward in terms of data privacy, it is really important to note that it contains breaking changes.

• If you currently rely on email or SMS delivery via an identity server you must modify your Synapse configuration.
• If you have configured custom templates, then eight new templates must be added to your templates directory.

If either apply to you failure to act will break your installation. Full details can be found in the upgrade notes.

🔗So what has changed?

It is possible for a user to associate an email address or phone number with their account, for a number of reasons:

• For use when logging in, as an alternative to the user id.
• In the case of email, as an alternative contact to help with account recovery.
• In the case of email, to receive notifications of missed messages.

Before an email address or phone number can be added to a user's account, or before such an address is used to carry out a password-reset, Synapse must confirm the operation with the owner of the email address or phone number. It does this by sending an email or text giving the user a link or token to confirm receipt. This process is known as '3pid verification'. ('3pid', or 'threepid', stands for third-party identifier, and we use it to refer to external identifiers such as email addresses and phone numbers.)

Previous versions of Synapse delegated the task of 3pid verification to an identity server by default. In most cases this server is vector.im or matrix.org.

In Synapse 1.4.0, for security and privacy reasons, the homeserver will no longer delegate this task to an identity server by default. Instead, the server administrator will need to explicitly decide how they would like the verification messages to be sent.

In the medium term, the vector.im and matrix.org identity servers will disable support for delegated 3pid verification entirely. However, in order to ease the transition, they will retain the capability for a limited period. Delegated email verification will be disabled on Monday 2nd December 2019 (giving roughly 2 months notice). Disabling delegated SMS verification will follow some time after that once SMS verification support lands in Synapse.

Once delegated 3pid verification support has been disabled in the vector.im and matrix.org identity servers, all Synapse versions that depend on those instances will be unable to verify email and phone numbers through them. There are no imminent plans to remove delegated 3pid verification from Sydent generally. (Sydent is the identity server project that backs the vector.im and matrix.org instances).

🔗Why is this necessary?

Prior to 1.4.0, the identity server was providing two related-but-separate functions:

a directory for users to publish their contact details and to look up their contacts by their email addresses and phone numbers. a "trusted third party communications network delegate", providing SMS- and email-sending functionality to all homeservers for registration and password reset.

The intention behind the identity server's providing 2. was one of convenience: to save homeserver admins the burden of configuring their homeservers to send email, as well as sparing us the effort of writing generic SMS-gateway integration that would allow any homeserver admin to configure their homeserver to send SMS via their chosen SMS aggregator.

By exposing this 'trusted email/sms delegate' functionality, however, and by including references to the New Vector identity servers in the default configuration for both Synapse and Riot, we created a situation in which users on non-New Vector run homeservers (who had never seen our privacy notice) could easily end up sharing their email addresses and phone numbers with a New Vector identity server.

Using identity servers for registration and password reset introduced yet further complexity - since password reset is a sensitive operation, it was important that the homeserver only use 'trusted' identity servers for this purpose (with the trust being configured by the homeserver admin in the homeserver config). But this created ambiguity over who was ultimately responsible for the choice of identity server and when they could make that choice - the client could present a default identity server at login/registration/password reset time, which the user could choose to override before logging in, but unless the identity server ultimately selected were on the homeserver admin's 'trusted identity servers' list, any identity server operations that were proxied via the homeserver would have been blocked by the homeserver. However, not all identity server operations initiated by the client were proxied via the homeserver - some were sent directly from the client to the identity server, and would not have been filtered.

The best way to solve this problem is to have individual homeservers take ownership for their own email and SMS needs. In the case of email this means configuring details of an appropriate SMTP server or continuing to delegate through an identity server that will allow it to do so. If a homeserver admin makes the active choice to use New Vector's identity servers for delegation (up until 1st December), they should make their own users aware through their privacy notice.

This delegation is entirely separate from the user's choice of identity server for user directory services. As of right now the user is free to choose and trust whichever identity server they wish, or to choose not to use an identity server at all.

🔗Are there any other data privacy features?

Yes, 1.4.0 now automatically garbage collects redacted messages (defaults to 7 days) and removes unused IP and user agent information stored in the user_ips table (defaults to 30 days). Finally, Synapse now warns in its logs if you are using matrix.org as a trusted key server, in case you wish to use a different server to help discover other servers’ keys.

🔗Anything else?

Aside from privacy, we’ve expanded our OpenTracing support and fixed a host of bugs. However the thing that is most exciting is switching on our solution for mitigating forward extremities build up’ by default.

In some cases rooms can accumulate ‘forward extremities’, which are simply an artefact of attempting to resolve the room state over multiple servers. Forward extremities are necessary to ensure that each server can independently arrive at the same view of the room eventually, however processing these extremities can be computationally expensive and degrade server performance overall.

Originally it was an experimental config option but we now feel confident to turn it on by default for all instances - it should make a big difference for the CPU of servers in fragmented rooms.

So that’s it folks, thanks for making it this far. As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Also, check out our Synapse installation guide page

The changelog since 1.3.1 follows:

🔗Synapse 1.4.0 (2019-10-03)

🔗Bugfixes

• Redact client_secret in server logs. (#6158)

🔗Synapse 1.4.0rc2 (2019-10-02)

🔗Bugfixes

• Fix bug in background update that adds last seen information to the devices table, and improve its performance on Postgres. (#6135)
• Fix bad performance of censoring redactions background task. (#6141)
• Fix fetching censored redactions from DB, which caused APIs like initial sync to fail if it tried to include the censored redaction. (#6145)
• Fix exceptions when storing large retry intervals for down remote servers. (#6146)

🔗Internal Changes

• Fix up sample config entry for redaction_retention_period option. (#6117)

🔗Synapse 1.4.0rc1 (2019-09-26)

Note that this release includes significant changes around 3pid verification. Administrators are reminded to review the upgrade notes.

🔗Features

• Changes to 3pid verification:
  • Add the ability to send registration emails from the homeserver rather than delegating to an identity server. (#5835, #5940, #5993, #5994, #5868)
  • Replace trust_identity_server_for_password_resets config option with account_threepid_delegates, and make the id_server parameteter optional on */requestToken endpoints, as per MSC2263. (#5876, #5969, #6028)
  • Switch to using the v2 Identity Service /lookup API where available, with fallback to v1. (Implements MSC2134 plus id_access_token authentication for v2 Identity Service APIs from MSC2140). (#5897)
  • Remove bind_email and bind_msisdn parameters from /register ala MSC2140. (#5964)
  • Add m.id_access_token to unstable_features in /versions as per MSC2264. (#5974)
  • Use the v2 Identity Service API for 3PID invites. (#5979)
  • Add POST /_matrix/client/unstable/account/3pid/unbind endpoint from MSC2140 for unbinding a 3PID from an identity server without removing it from the homeserver user account. (#5980, #6062)
  • Use account_threepid_delegate.email and account_threepid_delegate.msisdn for validating threepid sessions. (#6011)
  • Allow homeserver to handle or delegate email validation when adding an email to a user's account. (#6042)
  • Implement new Client Server API endpoints /account/3pid/add and /account/3pid/bind as per MSC2290. (#6043)
  • Add an unstable feature flag for separate add/bind 3pid APIs. (#6044)
  • Remove bind parameter from Client Server POST /account endpoint as per MSC2290. (#6067)
  • Add POST /add_threepid/msisdn/submit_token endpoint for proxying submitToken on an account_threepid_handler. (#6078)
  • Add submit_url response parameter to */msisdn/requestToken endpoints. (#6079)
  • Add m.require_identity_server flag to /version's unstable_features. (#5972)
• Enhancements to OpenTracing support:
  • Make OpenTracing work in worker mode. (#5771)
  • Pass OpenTracing contexts between servers when transmitting EDUs. (#5852)
  • OpenTracing for device list updates. (#5853)
  • Add a tag recording a request's authenticated entity and corresponding servlet in OpenTracing. (#5856)
  • Add minimum OpenTracing for client servlets. (#5983)
  • Check at setup that OpenTracing is installed if it's enabled in the config. (#5985)
  • Trace replication send times. (#5986)
  • Include missing OpenTracing contexts in outbout replication requests. (#5982)
  • Fix sending of EDUs when OpenTracing is enabled with an empty whitelist. (#5984)
  • Fix invalid references to None while OpenTracing if the log context slips. (#5988, #5991)
  • OpenTracing for room and e2e keys. (#5855)
  • Add OpenTracing span over HTTP push processing. (#6003)
• Add an admin API to purge old rooms from the database. (#5845)
• Retry well-known lookups if we have recently seen a valid well-known record for the server. (#5850)
• Add support for filtered room-directory search requests over federation (MSC2197, in order to allow upcoming room directory query performance improvements. (#5859)
• Correctly retry all hosts returned from SRV when we fail to connect. (#5864)
• Add admin API endpoint for setting whether or not a user is a server administrator. (#5878)
• Enable cleaning up extremities with dummy events by default to prevent undue build up of forward extremities. (#5884)
• Add config option to sign remote key query responses with a separate key. (#5895)
• Add support for config templating. (#5900)
• Users with the type of "support" or "bot" are no longer required to consent. (#5902)
• Let synctl accept a directory of config files. (#5904)
• Increase max display name size to 256. (#5906)
• Add admin API endpoint for getting whether or not a user is a server administrator. (#5914)
• Redact events in the database that have been redacted for a week. (#5934)
• New prometheus metrics:
  • synapse_federation_known_servers: represents the total number of servers your server knows about (i.e. is in rooms with), including itself. Enable by setting metrics_flags.known_servers to True in the configuration.(#5981)
  • synapse_build_info: exposes the Python version, OS version, and Synapse version of the running server. (#6005)
• Give appropriate exit codes when synctl fails. (#5992)
• Apply the federation blacklist to requests to identity servers. (#6000)
• Add report_stats_endpoint option to configure where stats are reported to, if enabled. Contributed by @Sorunome. (#6012)
• Add config option to increase ratelimits for room admins redacting messages. (#6015)
• Stop sending federation transactions to servers which have been down for a long time. (#6026)
• Make the process for mapping SAML2 users to matrix IDs more flexible. (#6037)
• Return a clearer error message when a timeout occurs when attempting to contact an identity server. (#6073)
• Prevent password reset's submit_token endpoint from accepting trailing slashes. (#6074)
• Return 403 on /register/available if registration has been disabled. (#6082)
• Explicitly log when a homeserver does not have the trusted_key_servers config field configured. (#6090)
• Add support for pruning old rows in user_ips table. (#6098)

🔗Bugfixes

• Don't create broken room when power_level_content_override.users does not contain creator_id. (#5633)
• Fix database index so that different backup versions can have the same sessions. (#5857)
• Fix Synapse looking for config options password_reset_failure_template and password_reset_success_template, when they are actually password_reset_template_failure_html, password_reset_template_success_html. (#5863)
• Fix stack overflow when recovering an appservice which had an outage. (#5885)
• Fix error message which referred to public_base_url instead of public_baseurl. Thanks to @aaronraimist for the fix! (#5909)
• Fix 404 for thumbnail download when dynamic_thumbnails is false and the thumbnail was dynamically generated. Fix reported by rkfg. (#5915)
• Fix a cache-invalidation bug for worker-based deployments. (#5920)
• Fix admin API for listing media in a room not being available with an external media repo. (#5966)
• Fix list media admin API always returning an error. (#5967)
• Fix room and user stats tracking. (#5971, #5998, #6029)
• Return a M_MISSING_PARAM if sid is not provided to /account/3pid. (#5995)
• federation_certificate_verification_whitelist now will not cause TypeErrors to be raised (a regression in 1.3). Additionally, it now supports internationalised domain names in their non-canonical representation. (#5996)
• Only count real users when checking for auto-creation of auto-join room. (#6004)
• Ensure support users can be registered even if MAU limit is reached. (#6020)
• Fix bug where login error was shown incorrectly on SSO fallback login. (#6024)
• Fix bug in calculating the federation retry backoff period. (#6025)
• Prevent exceptions being logged when extremity-cleanup events fail due to lack of user consent to the terms of service. (#6053)
• Remove POST method from password-reset submit_token endpoint until we implement submit_url functionality. (#6056)
• Fix logcontext spam on non-Linux platforms. (#6059)
• Ensure query parameters in email validation links are URL-encoded. (#6063)
• Fix a bug which caused SAML attribute maps to be overridden by defaults. (#6069)
• Fix the logged number of updated items for the users_set_deactivated_flag background update. (#6092)
• Add sid to next_link for email validation. (#6097)
• Threepid validity checks on msisdns should not be dependent on threepid_behaviour_email. (#6104)
• Ensure that servers which are not configured to support email address verification do not offer it in the registration flows. (#6107)

🔗Updates to the Docker image

• Avoid changing UID/GID if they are already correct. (#5970)
• Provide SYNAPSE_WORKER envvar to specify python module. (#6058)

🔗Improved Documentation

• Convert documentation to markdown (from rst) (#5849)
• Update INSTALL.md to say that Python 2 is no longer supported. (#5953)
• Add developer documentation for using SAML2. (#6032)
• Add some notes on rolling back to v1.3.1. (#6049)
• Update the upgrade notes. (#6050)

🔗Deprecations and Removals

• Remove shared-secret registration from /_matrix/client/r0/register endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. (#5877)
• Deprecate the trusted_third_party_id_servers option. (#5875)

🔗Internal Changes

• Lay the groundwork for structured logging output. (#5680)
• Retry well-known lookup before the cache expires, giving a grace period where the remote well-known can be down but we still use the old result. (#5844)
• Remove log line for debugging issue #5407. (#5860)
• Refactor the Appservice scheduler code. (#5886)
• Compatibility with v2 Identity Service APIs other than /lookup. (#5892, #6013)
• Stop populating some unused tables. (#5893, #6047)
• Add missing index on users_in_public_rooms to improve the performance of directory queries. (#5894)
• Improve the logging when we have an error when fetching signing keys. (#5896)
• Add support for database engine-specific schema deltas, based on file extension. (#5911)
• Update Buildkite pipeline to use plugins instead of buildkite-agent commands. (#5922)
• Add link in sample config to the logging config schema. (#5926)
• Remove unnecessary parentheses in return statements. (#5931)
• Remove unused jenkins/prepare_sytest.sh file. (#5938)
• Move Buildkite pipeline config to the pipelines repo. (#5943)
• Remove unnecessary return statements in the codebase which were the result of a regex run. (#5962)
• Remove left-over methods from v1 registration API. (#5963)
• Cleanup event auth type initialisation. (#5975)
• Clean up dependency checking at setup. (#5989)
• Update OpenTracing docs to use the unified trace method. (#5776)
• Small refactor of function arguments and docstrings in RoomMemberHandler. (#6009)
• Remove unused origin argument on FederationHandler.add_display_name_to_third_party_invite. (#6010)
• Add a failure_ts column to the destinations database table. (#6016, #6072)
• Clean up some code in the retry logic. (#6017)
• Fix the structured logging tests stomping on the global log configuration for subsequent tests. (#6023)
• Clean up the sample config for SAML authentication. (#6064)
• Change mailer logging to reflect Synapse doesn't just do chat notifications by email now. (#6075)
• Move last-seen info into devices table. (#6089)
• Remove unused parameter to get_user_id_by_threepid. (#6099)
• Refactor the user-interactive auth handling. (#6105)
• Refactor code for calculating registration flows. (#6106)

Hi all,

Back in June we wrote about our plans to tighten up data privacy in Matrix after some areas for improvement were brought to our attention. To quickly recap: the primary concern was that the default config for Riot specifies identity servers and integration managers run by New Vector (the company which the original Matrix team set up to build Riot and fund Matrix dev) - and so folks using a standalone homeserver may end up using external services without realising it. There were some other legitimate issues raised too (e.g. contact information should be obfuscated when checking if your contacts are on Matrix; Riot defaulted to using Google for STUN (firewall detection) if no TURN server had been set up on their server; Synapse defaults to using matrix.org as a key notary server).

We’ve been working away at this fairly solidly over the last few months. Some of the simpler items shipped quickly (e.g. Riot/Web had a stupid bug where it kept incorrectly loading the integration manager; Riot/Android wasn’t clear enough about when contact discovery was happening; Riot/Web wasn’t clear enough about the fact device names are publicly visible; etc) - but other bits have turned out to be incredibly time-consuming to get right.

However, we’re in the process today of releasing Synapse 1.4.0 and Riot/Web 1.4.0 (it’s coincidence the version numbers have lined up!) which resolve the majority of the remaining issues. The main changes are as follows:

1. Riot no longer automatically uses identity servers by default. Identity servers are only useful when inviting users by email address, or when discovering whether your contacts are on Matrix. Therefore, we now wait until the user tries to perform one of these operations before explaining that they need an identity server to do so, and we prompt them to select one if they want to proceed. This makes it abundantly clear that the user is connecting to an independent service, and why.

2. Integration Managers and identity servers now have the ability to force users to accept terms of use before using them. This means they can explicitly spell out the data privacy & usage policy of the server as required by GDPR, and it should now be impossible for a user to use these services without realising it. This was particularly fun in the case of identity servers, which previously had no concept of users and so couldn’t track whether users had agreed to their terms & conditions or not… and because homeservers sometimes talk to the identity server on behalf of users rather than the user talking direct, the privacy policy flow gets even hairier. But it’s solved now, and a nice side-effect of this is that users can now explicitly select their Integration Manager in Riot, in case they want to use Dimension or similar rather than the default provided by Modular.

3. Synapse no longer uses identity servers for verifying registrations or verifying password reset. Originally, Synapse made use of the fact that the Identity Service contains email/msisdn verification logic to handle identity verification in general on behalf of the homeserver. However, in retrospect this was a mistake: why should the entity running your identity server have the right to verify password resets or registration details on your homeserver? So, we have moved this logic into Synapse. This means Synapse 1.4.0 requires new configuration for email/msisdn verification to work - please see the upgrade notes for full details.

4. Sydent now supports discovering contacts based on hashed identifiers. MSC2134 specifies entirely new IS APIs for discovering contacts using a hash of their identifier rather than directly exposing the raw identifiers being searched for. This is implemented in Riot/iOS and Riot/Android and should be in the next major release; Riot/Web 1.4.0 has it already.

5. Synapse now warns in its logs if you are using matrix.org as a default trusted key server, in case you wish to use a different server to help discover other servers’ keys.

6. Synapse now garbage collects redacted messages after N days (7 days by default). (It doesn’t yet garbage collect attachments referenced from redacted messages; we’re still working on that).

7. Synapse now deletes account access data (IP addresses and User Agent) after N days (28 days by default) of a device being deleted.

8. Riot warns before falling back to using STUN (and defaults to turn.matrix.org rather than stun.google.com) for firewall discovery (STUN) when placing VoIP calls, and makes it clear that this is an emergency fallback for misconfigured servers which are missing TURN support. (We originally deleted the fallback entirely, but this broke things for too many people, so we’ve kept it but warn instead).

All of this is implemented in Riot/Web 1.4.0 and Synapse 1.4.0. Riot/Web 1.4.0 shipped today (Fri Sept 27th) and we have a release candidate for Synapse 1.4 (1.4.0rc1) today which fully ship on Monday.

For full details please go check out the Riot 1.4.0 and Synapse 1.4.0 blog posts.

Riot/Mobile is following fast behind - most of the above has been implemented and everything should land in the next release. RiotX/Android doesn’t really have any changes to make given it hadn’t yet implemented Identity Service or Integration Manager APIs.

This has involved a surprisingly large amount of spec work; no fewer than 9 new Matrix Spec Changes (MSC) have been required as part of the project. In particular, this results in a massive update to the Identity Service API, which will be released very shortly with the new MSCs. You can see the upcoming changes on the unstable branch and compare with the previous 0.2.1 stable release, as well as checking the detailed MSCs as follows:

• MSC1961: Integration manager authentication APIs
• MSC2078: Sending Third-Party Request Tokens via the Homeserver
• MSC2134: Identity Hash Lookups
• MSC2140: Terms of Service for ISes and IMs
• MSC2229: Allowing 3PID Owners to Rebind
• MSC2230: Store Identity Server in Account Data
• MSC2263: Give homeservers the ability to handle their own 3PID registrations/password resets
• MSC2264: Add an unstable feature flag to MSC2140 for clients to detect support
• MSC2290: Separate Endpoints for Threepid Binding

This said, there is still some work remaining for us to do here. The main things which haven’t made it into this release are:

• Preferring to get server keys from the source server rather than the notary server by default (https://github.com/matrix-org/synapse/pull/6110). This almost made it in, but we need to test it more first - until then, your specified notary server will see roughly what servers your servers are trying to talk to. In future this will be mitigated properly by MSC1228 (removing mxids from events).
• Configurable data retention periods for rooms. We are tantalisingly close with this - https://github.com/matrix-org/synapse/pull/5815 is an implementation that the French Govt deployment is using; we need to port it into mainline Synapse.
• Authenticating access to the media repository - for now, we still rely on media IDs being almost impossible to guess to protect the data rather than authenticating the user.
• Deleting items from the media repository - we still need to hook up deletion APIs.
• Garbage collecting forgotten rooms. If everyone leaves & forgets a room, we should delete it from the DB.
• Communicating erasure requests over federation

We’ll continue to work on these as part of our ongoing maintenance backlog.

Separately to the data privacy concerns, we’ve had a separate wave of feedback regarding how we handle GDPR Data Subject Access Requests (DSARs). Particularly: whether DSAR responses should contain solely the info your have directly keyed by the requesting Matrix ID - or if we should provide all the data “visible” to that ID (i.e. the history of the conversations they’ve been part of). We went and got professional legal advice on this one, and the conclusion is that we should keep our responses to DSARs as tightly scoped as possible. We updated Matrix.org’s privacy policy and DSAR tools to reflect the new legal input.

Finally, it’s really worth calling out the amount of effort that went into this project. Huge huge thanks to everyone involved (given it’s cut across pretty much every project & subteam we have working on the core of Matrix) who have soldiered through the backlog. We’ve been tracking progress using our feature-dashboard tool which summarises Github issues based on labels & issue lifecycle, and for better or worse it’s ended up being the biggest project board we’ve ever had. You can see the live data here (warning, it takes tens of seconds to spider Github to gather the data) - or, for posterity and ease of reference, I’ve included the current issue list below. The issues which are completed have “done” after them; the ones still in progress say “in progress”, and ones which haven’t started yet have nothing. We split the project into 3 phases - phases 1 and 2 represent the items needed to fully solve the privacy concerns, phase 3 is right now a mix of "nice to have" polish and some more speculative items. At this point we’ve effectively finished phase 1 on Synapse & Riot/Web, and Riot/Mobile is following close behind. We're continuing to work on phase 2, and we’ll work through phase 3 (where appropriate) as part of our general maintenance backlog.

I hope this gives suitable visibility on how we’re considering privacy; after all, Matrix is useless as an open communication protocol if the openness comes at the expense of user privacy. We’ll give another update once the remaining straggling issues are closed out; and meanwhile, now the bulk of the privacy work is out of the way on Riot/Web, we can finally get back to implementing the UI E2E cross-signing verification and improving first time user experience.

Thanks for your patience and understanding while we’ve sorted this stuff out; and thanks once again for flying Matrix :)

In the absence of comments on the current blog, please feel free to discuss over at HN, or alternatively come ask stuff in our AMA over at /r/privacy (starting ~5pm GMT+1 (UK) on Friday Sept 27th).

🔗The Privacy Project Dashboard Of Doom

• phase:1
  • vector-im/riot-web
    • 5846 Riot duplicates calls to Scalar (done)
    • 5921 Ability to disable all identity server functionality via the config file (done)
    • 6802 riot shouldn't try to load integrations from an integration manager unless the user has accepted that manager's privacy policies (done)
    • 10088 Prompt to accept integration manager polices on use (done)
    • 10090 Make sure there are no ugly edge cases running Riot without an integrations manager (done)
    • 10091 Decouple setting an email for password reset from publishing your threepid to the identity server and support choice of identity server (on registration) (done)
    • 10092 Prompt to accept Identity Server policies on registration (done)
    • 10093 Prompt to accept identity server policies before inviting them to a room (done)
    • 10094 Store identity server in Account Data and support choosing identity server integration in User Settings (done)
    • 10159 Enable toggling a threepid's association with the current IS in User Settings (done)
    • 10173 VoIP: Stop falling back to Google for STUN (done)
    • 10216 We should make it clear in the UI that device names are publicly readable (done)
    • 10386 Terms modal prompt should appear without a flash of the integration manager portal first (done)
    • 10408 Identity server v2 API authentication (done)
    • 10423 Deploy a develop environment for t's and c's flows for IM and IS (done)
    • 10424 Remove the bind true flag from 3PID calls on registration (done)
    • 10425 Ability to disconnect from the identity server by pressing buttons in user settings. (done)
    • 10452 Test IS v2 access tokens for validity (done)
    • 10497 Integration manager terms hides terms that need signing (done)
    • 10510 Remove the bind true flag from 3PID adds in settings (done)
    • 10519 Update discovery and account 3PID sections when either changes (done)
    • 10522 Handle terms agreement in the Discovery section of settings (done)
    • 10525 IS interactions proxied through the HS also need terms agreement (done)
    • 10528 don't try & show threepids in discovery section if we don't have an ID server (done)
    • 10539 Inline terms agreement on change IS and IM (done)
    • 10540 When using Riot without an IS, identity requests are sent to server hosting Riot (done)
    • 10546 Prompt for fallback VOIP should only happen when the user tries to make a call (done)
    • 10550 warn when disconnecting ID server if there are any current bindings (done)
    • 10552 Allow email registration, password reset & adding 3pids when no IS (done)
    • 10553 Remove the ability to set an IS at login/registration (done)
    • 10556 Use the hashed v2 lookup API for 3PIDs (done)
    • 10561 Scalar should serve a .well-known file (done)
    • 10566 Use nicer APIs for toggling a 3PID's shared status (done)
    • 10571 Allow email registration when no IS (done)
    • 10572 Allow password reset when no IS (done)
    • 10573 Allow adding 3pids when no IS (done)
    • 10574 Placeholder issue for MSCs that should land prior to the privacy release (done)
    • 10593 Placeholder issue for privacy migration path (done)
    • 10597 Ensure IS in account data is read / written as specced (done)
    • 10599 Send IS for adding MSISDNs only when required by HS (done)
    • 10603 Getting a terms prompt when inviting an email address clears the address picker (done)
    • 10619 Handle the case of no IS in features that require IS to lookup (done)
    • 10620 Change auth flows so that you default to no IS (done)
    • 10634 Changing to vector.im IS in Settings fails because /terms 404s (done)
    • 10635 Inline terms agreement in Discovery should still show change and do not use (done)
    • 10636 The UX is a little confusing if you haven't accepted the terms of the default identity server. (done)
    • 10637 Unable to add email address to HS account without IS set (done)
    • 10660 Sydent implementation: Unauthed v1, clone all endpoints, auth v2 (done)
    • 10666 In login/register, riot freezes if you go to Advanced, remove the IS url and click next (done)
    • 10669 Checking email invite account is unclear when no IS set (done)
    • 10674 Email help text on registration should be updated without binding (done)
    • 10677 Change text about other users being able to invite you by email on registration page (done)
    • 10744 Only set terms URLs in the account when they change (done)
    • 10749 Changing from one IS to another does not trigger bound 3PID warning (done)
    • 10750 Bound 3PIDs warning should better explain what's being left behind (done)
    • 10754 Lowercase emails during IS lookup calls (done)
    • 10755 Terms account data is meant to be additive, but currently sets only new URLs (done)
    • 10756 After changing IS to termstest.matrix.org the text field isn't cleared and the button remains active (done)
    • 10757 After changing IS back to vector.im the text field isn't cleared and the button remains active (done)
    • 10758 Revoke may be too vague for unbinding a 3PID (done)
    • 10779 When publishing a threepid to an IS, if you click 'continue' before clicking the link in your email, we delete your threepid from the HS. (done)
    • 10800 Community invite should not talk about ISes at all. (done)
    • 10823 IS with unagreed terms will block you from adding even unshared 3PIDs to your HS (done)
    • 10839 Use the new backend APIs for adding-3pid-to-homeserver and binding-3pid-on-identity-server when they exist. (done)
    • 10878 UX regression when trying to invite by email if no IS is configured (done)
    • 10883 Riot should check for r0.6.0 server support alongside feature flags (done)
    • 10923 Send validation tokens to submit_url from HS (done)
    • 10933 Do not include ID server or access token when HS supports MSC2290 (done)
    • 10939 Registration with MSISDN needs to use submit_url (done)
    • 10941 threepid_creds for registration and password reset still include id_server (done)
    • 10959 Remove id_server from email threepid_creds (done)
    • 10604 QA: Re-test ALL of the identity server interactions (in progress)
    • 10958 Terms of Service nitpicks (in progress)
    • 10704 We treat an empty string identity server as not having one
  • vector-im/riot-ios
    • 2518 Make sure there are no ugly edge cases running Riot without an integrations manager (done)
    • 2521 Privacy: Improve wording on contact upload UI (done)
    • 2532 VoIP: Stop falling back to Google for STUN (done)
    • 2600 Prompt to accept integration manager polices on use (done)
    • 2601 Decouple setting an email for password reset from publishing your threepid to the identity server and support choice of identity server (on registration) (done)
    • 2602 Prompt to accept identity server policies before inviting them to a room (done)
    • 2603 Identity server v2 API authentication (done)
    • 2606 Settings: Add a Discovery section (done)
    • 2608 Registration: Update consents flow (done)
    • 2643 Ability to disable all identity server functionality via the config file (done)
    • 2646 VoIP: Fallback to matrix.org STUN server with a confirmation dialog (done)
    • 2647 MXRestClient: Move IS API to its own (done)
    • 2648 Remove the bind true flag from 3PID calls on registration (done)
    • 2650 Remove the bind true flag from 3PID adds in settings (done)
    • 2651 Store identity server in Account Data and support choosing identity server integration in User Settings (done)
    • 2652 Use the hashed v2 lookup API for 3PIDs (done)
    • 2653 Allow email registration, password reset & adding 3pids when no IS (done)
    • 2654 Use nicer APIs for toggling a 3PID's shared status (done)
    • 2657 Allow email registration when no IS (done)
    • 2658 Allow password reset when no IS (done)
    • 2659 Allow adding 3pids when no IS (done)
    • 2660 Handle terms agreement in the Discovery section of settings (done)
    • 2661 Remove the ability to set an IS at login/registration (done)
    • 2662 We should make it clear in the UI that device names are publicly readable (done)
    • 2664 Placeholder issue for privacy migration path (done)
    • 2665 Ensure IS in account data is read / written as specced (done)
    • 2672 Handle the case of no IS in features that require IS to lookup (done)
    • 2675 Email help text on registration should be updated without binding (done)
    • 2682 Send IS for adding MSISDNs only when required by HS (done)
    • 2686 Use wellknown to discover the IS of a custom HS (done)
    • 2696 Lowercase emails during IS lookup calls (done)
    • 2704 Use id_access_token in CS API when required (done)
    • 2604 Settings: Ability to change the identity server (in progress)
    • 2649 Ability to disconnect from the identity server by pressing buttons in user settings (in progress)
    • 2713 Use the new backend APIs for adding-3pid-to-homeserver and binding-3pid-on-identity-server when they exist. (in progress)
    • 2718 Riot should check for r0.6.0 server support alongside feature flags (in progress)
    • 2683 Checking email invite account is unclear when no IS set
    • 2731 Send validation tokens to submit_url from HS
    • 2732 Do not include ID server or access token when HS supports MSC2290
    • 2736 IS terms cannot be accepted anymore
  • vector-im/riot-android
    • 3223 VoIP: Stop falling back to Google for STUN (done)
    • 3224 Make sure there are no ugly edge cases running Riot without an integrations manager (done)
    • 3225 Prompt to accept integration manager polices on use (done)
    • 3226 Decouple setting an email for password reset from publishing your threepid to the identity server and support choice of identity server (on registration) (done)
    • 3227 Prompt to accept identity server policies before inviting them to a room (done)
    • 3228 Identity server v2 API authentication (done)
    • 3229 Settings: Ability to change the identity server (done)
    • 3230 Settings: Add a Discovery section (done)
    • 3231 Registration: Update consents flow (done)
    • 3252 Remove the bind true flag from 3PID calls on registration (done)
    • 3253 Ability to disconnect from the identity server by pressing buttons in user settings (done)
    • 3254 Remove the bind true flag from 3PID adds in settings (done)
    • 3256 Store identity server in Account Data and support choosing identity server integration in User Settings (done)
    • 3257 Use the hashed v2 lookup API for 3PIDs (done)
    • 3258 Allow email registration, password reset & adding 3pids when no IS (done)
    • 3260 Allow email registration when no IS (done)
    • 3261 Allow password reset when no IS (done)
    • 3262 Allow adding 3pids when no IS (done)
    • 3263 Handle terms agreement in the Discovery section of settings (done)
    • 3264 Remove the ability to set an IS at login/registration (done)
    • 3265 We should make it clear in the UI that device names are publicly readable (done)
    • 3268 Placeholder issue for privacy migration path (done)
    • 3269 Ensure IS in account data is read / written as specced (done)
    • 3277 Handle the case of no IS in features that require IS to lookup (done)
    • 3278 Email help text on registration should be updated without binding (done)
    • 3281 Send IS for adding MSISDNs only when required by HS (done)
    • 3283 Use wellknown to discover the IS of a custom HS (done)
    • 3289 Lowercase emails during IS lookup calls (done)
    • 3295 Use id_access_token in CS API when required (done)
    • 3300 Use the new backend APIs for adding-3pid-to-homeserver and binding-3pid-on-identity-server when they exist. (done)
    • 3312 Riot should check for r0.6.0 server support alongside feature flags (done)
    • 3316 Implement MSC2290 (done)
    • 3324 id_server param sent when registering an email on server that does not requires one (done)
    • 3326 Discovery Settings / infinite loading if terms not signed on existing IS (done)
    • 3327 vector.im is seen as having no terms, but it does (done)
    • 3251 VoIP: Fallback to matrix.org STUN server with a confirmation dialog (in progress)
    • 3248 Ability to disable all identity server functionality via the config file
    • 3318 Send validation tokens to submit_url from HS
    • 3322 Do not include ID server or access token when HS supports MSC2290
  • vector-im/riotX-android
    • 490 Allow email registration, password reset & adding 3pids when no IS (done)
    • 432 Prompt to accept integration manager polices on use (in progress)
    • 431 Make sure there are no ugly edge cases running Riot without an integrations manager
    • 433 Decouple setting an email for password reset from publishing your threepid to the identity server and support choice of identity server (on registration)
    • 434 Prompt to accept identity server policies before inviting them to a room
    • 435 Identity server v2 API authentication
    • 436 Settings: Ability to change the identity server
    • 438 Settings: Add a Discovery section
    • 439 Registration: Update consents flow
    • 489 Use the hashed v2 lookup API for 3PIDs
  • matrix-org/matrix-doc
    • 1961 MSC1961: Integration manager authentication APIs (done)
    • 2078 MSC2078: Sending Third-Party Request Tokens via the Homeserver (done)
    • 2130 Identity service should do lookups based on hashed 3PIDs, not plaintext ones. (done)
    • 2134 MSC2134: Identity Hash Lookups (done)
    • 2139 Method for ISes and Integration managers to provide terms of service (done)
    • 2140 MSC2140: Terms of Service for ISes and IMs (done)
    • 2229 MSC2229: Allowing 3PID Owners to Rebind (done)
    • 2230 MSC2230: Store Identity Server in Account Data (done)
    • 2263 MSC2263: Give homeservers the ability to handle their own 3PID registrations/password resets (done)
    • 2264 MSC2264: Add an unstable feature flag to MSC2140 for clients to detect support (done)
    • 2290 MSC2290: Separate Endpoints for Threepid Binding (in progress)
  • matrix-org/sydent
    • 178 Support for MSC2140 (done)
    • 179 Support testing with matrix-is-tester (done)
    • 180 Support for MSC2140 (done)
    • 181 Deny bindings to anything other than the mxid you're authed as (done)
    • 184 Hash 3PID lookups (done)
    • 186 Don't fail the unbind request if the binding doesn't exist (done)
    • 187 Add more tweaks to terms branch (done)
    • 189 Redact looked-up threepids from the application logs. (done)
    • 191 Add OpenID auth to 3PID hash lookup (done)
    • 197 Have 'mappings' contain medium as well as address (done)
    • 198 Throw exception on bad args & catch in wrapper (done)
    • 199 Return algorithm, lookup_pepper in M_INVALID_PEPPER response (done)
    • 200 Support MSC2140 register/terms endpoints (done)
    • 201 Port v1 endpoints to v2 (done)
    • 202 Deny bindings to anything other than the mxid you're authed as (done)
    • 204 Fix the signing servlet (done)
    • 205 Correct API v1 path check in get_args (done)
    • 206 Fix terms POST (done)
    • 207 Deploy policy file (done)
    • 208 Remove PII logging from log-level INFO (done)
    • 213 Delete threepid assocs on unbind and remove existing with migration (done)
    • 217 Allow non-matrix.org MXIDs via v2 only once terms are enabled (done)
    • 218 Unbind fails to remove the binding on matrix.org IS (done)
    • 220 Unbind fails to work for bindings where it was attempted with #213 (done)
    • 192 Sydent doesn't delete 3PIDs from DB on unbind (in progress)
  • matrix-org/synapse
    • 1287 We need to actually censor redactions from the DB (SYN-284) (done)
    • 5751 Make homeservers able to handle registration-with-email without depending on an Identity Server (done)
    • 5786 Support IS v2 API with authentication for requests proxied to the IS (done)
    • 5827 Add 3PID unbind API for MSC2140 (done)
    • 5835 [1/2] Allow homeservers to send registration emails | Sending the email (done)
    • 5861 Use the v2 lookup API for 3PID invites (done)
    • 5862 Allow account owners to rebind 3PIDs as in MSC2229 (done)
    • 5868 Add flag in /versions for whether clients should send id_server params (done)
    • 5874 Placeholder issue for privacy migration path (done)
    • 5875 Remove trusted_third_party_id_servers functionality (done)
    • 5876 Replace trust_identity_server_for_password_resets with account_threepid_delegate (done)
    • 5892 Switch to using v2 Identity Service APIs other than lookup (MSC 2140) (done)
    • 5897 Use the v2 lookup API for 3PID invites (done)
    • 5927 Add a m.id_access_token flag to unstable_features (done)
    • 5928 Split account_threepid_handler into a msisdn and email versions (done)
    • 5929 Use account_threepid_delegate for sending SMS (done)
    • 5930 Add m.id_access_token flag (done)
    • 5934 Censor redactions in DB after a month (done)
    • 5935 Use federation blacklist for requests to identity servers (done)
    • 5937 Revert "Use the v2 lookup API for 3PID invites" (done)
    • 5940 [2/2] Allow homeservers to send registration emails | Accepting the verification (done)
    • 5945 Revert "Add m.id_access_token flag" (done)
    • 5948 Support v2 Identity Server APIs (done)
    • 5964 Remove bind_email and bind_msisdn (done)
    • 5968 Set m.require_identity_server to always be False (done)
    • 5969 Change account_threepid_delegate to a dictionary (done)
    • 5972 Add m.require_identity_server to /versions unstable_flags (done)
    • 5974 Add m.id_access_token to /versions unstable_features (MSC2264) (done)
    • 5976 Use the v2 Identity Service API for lookups (MSC2134 + MSC2140) (done)
    • 5979 v2 3PID Invites (part of MSC2140) (done)
    • 5980 Add POST /_matrix/client/r0/account/3pid/unbind (MSC2140) (done)
    • 5987 Allow Synapse to send registration emails + choose Synapse or an external server to handle 3pid validation (done)
    • 5996 Allow users to rebind 3pids they own (MSC2229) (done)
    • 6000 Use the federation blacklist for requests to untrusted Identity Servers (done)
    • 6011 Use account_threepid_delegate for 3pid validation (done)
    • 6013 Fix existing v2 identity server calls (MSC2140) (done)
    • 6042 Allow HS to send emails when adding an email to the HS (done)
    • 6043 Implement MSC2290 (done)
    • 6044 Add an unstable feature flag for separate add/bind 3pid APIs (done)
    • 6062 Use unstable prefix for 3PID unbind API (done)
    • 6067 Drop support for bind param on POST /account/3pid (MSC2290) (done)
    • 6076 Synapse doesn't give clients a submit_url on requestToken breaking msisdn adding (done)
    • 6078 Add POST submit_token endpoint for MSISDN (done)
    • 6079 Add submit_url response parameter to msisdn /requestToken (done)
    • 6087 Remove hardcoded defaults of matrix.org and vector.im in configuration (done)
    • 6090 Explicitly log when a homeserver does not have a trusted key server configured (done)
    • 6096 Riot Web expects the email validation next_link to have the sid appended (done)
    • 6100 Remove email from registration flows if it’s unsupported (done)
    • 6103 _check_threepid in auth.py incorrect for MSISDN (done)
    • 6045 Support Client-Server API r0.6.0 (in progress)
    • 1263 When we redact events, any mxc content they refer to should be redacted too (SYN-216)
    • 4565 Metadata resistance.
    • 6086 Fetch signing-keys directly from servers before falling back to the trusted_key_servers
• phase:2
  • vector-im/riot-web
    • 4913 Option for homeservers to set a default integration manager (done)
    • 10161 Store Integration Manager preferences in account data and allow user to change them somewhere sensible (done)
    • 10967 Disconnect from identity server fails (done)
    • 10443 Remove v1 IS API fallbacks once servers are updated
    • 10557 Prompt users each time before sending data to an Identity Server that doesn't have a terms of service (unless you have actively set that IS in your account data).
    • 10696 Allow users to disconnect from an integration manager entirely in the same way that we support doing this for identity servers
    • 10909 Unable to disconnect from IS if the server is unavailable
    • 10936 Refine submit_url handling to only activate with separate add and bind
  • vector-im/riot-ios
    • 2711 Provide a better UX for users considering sharing their contact list with an IS to discover people they know already using Matrix
  • vector-im/riot-android
    • 3282 Checking email invite account is unclear when no IS set
    • 3323 Post MSC2290 bug / Clicking on the validation mail link fails registration
  • matrix-org/matrix-doc
    • 1957 MSC1957: Integration manager discovery (done)
  • matrix-org/sydent
    • 196 Remove the lookup_hash field from local_threepid_associations (in progress)
  • matrix-org/synapse
    • 5881 Remove trust_identity_server_for_password_resets and account_threepid_delegate options
• phase:3
  • vector-im/riot-web
    • 10563 Add better domain for turn.matrix.org use as STUN server (done)
    • 10087 Prompt to accept integration manager policies on registration
    • 10167 Present an aggregated terms of service dialogue at registration if possible
    • 10168 If a user has disabled the identity server integration on their account, we should make invite user handle this nicely
    • 10401 Invite new users to publish their threepids to the identity server
    • 10422 Use more contextual prompt for integration manager polices on use
    • 10453 Log out from IM on Riot log out and IM removal
    • 10455 Log out from IS on Riot log out and IS removal
    • 10487 Store the date of users having read and accepted the IM policies in the IM db
    • 10488 Store the date of users having read and accepted the IS policies in the IS db
    • 10498 Terms test scalar requires the legacy ?v query param on the new account route
    • 10575 We should show the 'must configure TURN' warning when a call fails, even after using the fallback turn.matrix.org
    • 10615 Change all IS access token APIs to use getIdentityAccessToken only
    • 10671 riot submits sign-ed25519 requests as POST requests with params in query string and empty body
    • 10746 /invite doesn't force terms with older homeservers
    • 10830 Show IS policy links in user settings somewhere.
    • 10950 Unhelpful 400 error when adding MSISDN and server doesn’t delegate
  • vector-im/riot-ios
    • 2710 Show IS policy links in user settings somewhere.
  • matrix-org/matrix-doc
    • 447 We need a way to be able to expire data from a HS. (SPEC-141)
  • matrix-org/synapse
    • 5830 pushers table contains user device names, which may include user real names