Synapse 1.9.0 released

23.01.2020 00:00 โ€” Releases โ€” Neil Johnson

Synapse 1.9.0 contains a bunch of new admin APIs as well as bug fixes to existing ones. In particular we have a new end point that allows admins to query which rooms their server participates in.

Aside from that it is worth noting that we have officially dropped support for SQLite versions < 3.11. Though more generally we would recommend that all admins migrate to Postgres for anything other than test and evaluation purposes.

Get the new release from github or any of the sources mentioned at https://github.com/matrix-org/synapse/blob/master/INSTALL.md.

Changelog since Synapse 1.8.0

๐Ÿ”—Synapse 1.9.0 (2020-01-23)

WARNING: As of this release, Synapse no longer supports versions of SQLite before 3.11, and will refuse to start when configured to use an older version. Administrators are recommended to migrate their database to Postgres (see instructions here).

If your Synapse deployment uses workers, note that the reverse-proxy configurations for the synapse.app.media_repository, synapse.app.federation_reader and synapse.app.event_creator workers have changed, with the addition of a few paths (see the updated configurations here). Existing configurations will continue to work.

๐Ÿ”—Improved Documentation

  • Fix endpoint documentation for the List Rooms admin API. (#6770)

๐Ÿ”—Synapse 1.9.0rc1 (2020-01-22)

๐Ÿ”—Features

  • Allow admin to create or modify a user. Contributed by Awesome Technologies Innovationslabor GmbH. (#5742)
  • Add new quarantine media admin APIs to quarantine by media ID or by user who uploaded the media. (#6681, #6756)
  • Add org.matrix.e2e_cross_signing to unstable_features in /versions as per MSC1756. (#6712)
  • Add a new admin API to list and filter rooms on the server. (#6720)

๐Ÿ”—Bugfixes

  • Correctly proxy HTTP errors due to API calls to remote group servers. (#6654)
  • Fix media repo admin APIs when using a media worker. (#6664)
  • Fix "CRITICAL" errors being logged when a request is received for a uri containing non-ascii characters. (#6682)
  • Fix a bug where we would assign a numeric user ID if somebody tried registering with an empty username. (#6690)
  • Fix purge_room admin API. (#6711)
  • Fix a bug causing Synapse to not always purge quiet rooms with a low max_lifetime in their message retention policies when running the automated purge jobs. (#6714)
  • Fix the synapse_port_db not correctly running background updates. Thanks @tadzik for reporting. (#6718)
  • Fix changing password via user admin API. (#6730)
  • Fix /events/:event_id deprecated API. (#6731)
  • Fix monthly active user limiting support for worker mode, fixes #4639. (#6742)
  • Fix bug when setting account_validity to an empty block in the config. Thanks to @Sorunome for reporting. (#6747)
  • Fix AttributeError: 'NoneType' object has no attribute 'get' in hash_password when configuration has an empty password_config. Contributed by @ivilata. (#6753)
  • Fix the docker-compose.yaml overriding the entire /etc folder of the container. Contributed by Fabian Meyer. (#6656)

๐Ÿ”—Improved Documentation

  • Fix a typo in the configuration example for purge jobs in the sample configuration file. (#6621)
  • Add complete documentation of the message retention policies support. (#6624, #6665)
  • Add some helpful tips about changelog entries to the GitHub pull request template. (#6663)
  • Clarify the account_validity and email sections of the sample configuration. (#6685)
  • Add more endpoints to the documentation for Synapse workers. (#6698)

๐Ÿ”—Deprecations and Removals

  • Synapse no longer supports versions of SQLite before 3.11, and will refuse to start when configured to use an older version. Administrators are recommended to migrate their database to Postgres (see instructions here). (#6675)

๐Ÿ”—Internal Changes

  • Add local_current_membership table for tracking local user membership state in rooms. (#6655, #6728)
  • Port synapse.replication.tcp to async/await. (#6666)
  • Fixup synapse.replication to pass mypy checks. (#6667)
  • Allow additional_resources to implement IResource directly. (#6686)
  • Allow REST endpoint implementations to raise a RedirectException, which will redirect the user's browser to a given location. (#6687)
  • Updates and extensions to the module API. (#6688)
  • Updates to the SAML mapping provider API. (#6689, #6723)
  • Remove redundant RegistrationError class. (#6691)
  • Don't block processing of incoming EDUs behind processing PDUs in the same transaction. (#6697)
  • Remove duplicate check for the session query parameter on the /auth/xxx/fallback/web Client-Server endpoint. (#6702)
  • Attempt to retry sending a transaction when we detect a remote server has come back online, rather than waiting for a transaction to be triggered by new data. (#6706)
  • Add StateMap type alias to simplify types. (#6715)
  • Add a DeltaState to track changes to be made to current state during event persistence. (#6716)
  • Add more logging around message retention policies support. (#6717)
  • When processing a SAML response, log the assertions for easier configuration. (#6724)
  • Fixup synapse.rest to pass mypy. (#6732, #6764)
  • Fixup synapse.api to pass mypy. (#6733)
  • Allow streaming cache 'invalidate all' to workers. (#6749)
  • Remove unused CI docker compose files. (#6754)

This Week in Matrix 2020-01-17

17.01.2020 21:37 โ€” This Week in Matrix โ€” Ben Parsons
Last update: 17.01.2020 19:46

๐Ÿ”—Matrix Live ๐ŸŽ™

๐Ÿ”—Dept of Status of Matrix ๐ŸŒก

๐Ÿ”—Google Summer of Code 2020

Sometimes it seems like existence is a constant loop. Cycles within cycles are replayed, with only minor differences to help us discern the current and the foregoing.
How can we endure it?

It's not all bad though! Google have announced the return of the Summer of Code programme, and have updated their website with the current year (2020) and latest information.

Matrix will be applying to be a Mentoring Organisation, as we have for several years now. This year, we'd like to open the umbrella a little. If you think your (Matrix-related) project could help mentor a student, and that you could offer a meaningful project to that student, please contact me directly. For reference, there were four students under the Matrix heading last year, of which one was assigned to work with Kitsune on encryption support in libQuotient.

๐Ÿ”—FOSDEM 2020

Will you be at FOSDEM this year? Matrix will be very well represented, so please come and see us at our stand, plus attend all of the three talks we'll be conducting. Many more details to follow!

๐Ÿ”—Dept of Services ๐Ÿš€

New from bit.nl, a public Matrix Homeserver focused on the Dutch market. Read the announcement and sign up here.

๐Ÿ”—Dept of Spec ๐Ÿ“œ

anoa offered:

Here are this week's spec ๐Ÿ•๏ธ updates!

Merged MSCs:

No MSCs were merged this week.

MSCs in Final Comment Period:

No MSCs are in Final Comment Period.

New MSCs:

The Spec Core Team is still working on implementation, and likely will continue to be until after FOSDEM.

๐Ÿ”—Dept of Servers ๐Ÿข

๐Ÿ”—Dendrite / P2P

Neil Alexander reported:

I've spent some time this week experimenting with integrating libp2p into dendrite and gomatrixserverlib to understand if we can get link-local homeserver discovery working along with direct or indirect federation. It's still early at the moment but I have had success with early federation requests working over libp2pโ€”hopefully I'll have something interesting to show soon!

dendrite in riot

๐Ÿ”—Synapse

Neil reported:

This week weโ€™ve been continuing to work on migrating sydent to py3, improving admin tooling and working on fixing e2ee bugs. Weโ€™re also looking to make changes to allow admins the power to remove room aliases (MSCs 2260 2261).

๐Ÿ”—Dept of Built on Matrix ๐Ÿ—

๐Ÿ”—Matrix Presents ๐Ÿ“ฝ๏ธ

Half-Shot told us:

What is it? Slideshows in Matrix, basically using the matrix room / event primitives to construct presentable federated slide shows.

What's new this week you ask? Well, it breaks less. It also does other neat things.

  • Supports guests for viewing presentations, including the ability to specify a guest HS via the URL.
  • Supports code highlighting in the presentation.
  • Lots of CSS fixes to make slides line up properly.
  • A fullscreen mode, which is actually unobstructive and quite handy.
  • Presenter mode, which allows you to switch between following the presenters view of the room, and being able to view at your own leisure.
  • Room joins now actually work, so you can be linked to a slide and expect to be able to view it.

Come check us out at #presents:half-shot.uk

๐Ÿ”—Dept of Bridges ๐ŸŒ‰

๐Ÿ”—matrix-imposter-bot, interesting relay idea

@mr_johnson22:matrix.org said:

matrix-imposter-bot - A bot that uses your account to repeat other people's messages. This gives relay-bot capabilities to puppet-only bridges. For more details, see https://github.com/mrjohnson22/matrix-imposter-bot

๐Ÿ”—mx-puppet-bridge and other soru adventures

sorunome told us:

Will the development ever stop? Who knows! Here's the latest mx-puppet-bridge updates:

  • Fix unbridging of direct chats
  • Allow room-specific ghost avatar and name overrides
  • Enhanced logging config
  • Ability to configure name patterns for users/rooms/groups

๐Ÿ”—mx-puppet-discord

Of course mx-puppet-discord received some updates, too!

  • Receive notification on friends request
  • Add optional profile syncing
  • Implement guild-specific nicknames
  • Implement variables for name patterns

๐Ÿ”—mx-puppet-slack

What, mx-puppet-slack updates? Yay!

  • Set external_url field
  • Implement variables for name patterns

As always, if you have any questions, please drop by the support chat and consider to donate! ๐ŸฆŠ๐ŸฆŠ๐ŸฆŠ

๐Ÿ”—Dept of Clients ๐Ÿ“ฑ

๐Ÿ”—Nheko replies rendering

@deepbluev7:neko.dev said:

I added pretty rendering of replies to nheko. It uses the same rendering as for normal messages, so you will see an image being replied to and you can even play a replied to video. You can also click on the reply to navigate to that part of the history.

๐Ÿ”—RiotX: Release v0.13.0

benoit said:

RiotX: Release v0.13.0 has been uploaded today to the PlayStore and should follow in F-Droid store. It contains typing notification feature, and new screens for Room profile and room members list, and new screen for Room Member profile. Those new screens are still missing a lots of features, that will be implemented in the next weeks. For F-Droid version, it contains a fix on the broken background sync. We are also making big progress on cross-signing implementation, and we have started to write integration tests on the SDK.

@bubu:bubu1.eu told us:

RiotX is now available via the f-droid.org repository here: https://f-droid.org/packages/im.vector.riotx/

๐Ÿ”—Riot iOS

Manu announced:

We are still working hard on the implementation of cross-signing.

๐Ÿ”—Riot Web Cross-signing update

Thanks to Ryan from the team:

All of this is part of cross-signing:

  • Room shield decoration added
  • New post-login device verification flow
  • Toasts added to verify a new session
  • Will begin working on post-login / registration bootstrapping flows

๐Ÿ”—Fractal

Alexandre Franke reported:

In the past month, a few things happened:

  • When redacting messages, they get removed from history view.
  • Thumbnail for pasted images have been fixed.
  • A handful of languages got updated.
  • Moar refactoring.

๐Ÿ”—koma + Continuum: kotlin lib and client

yuforia told us:

koma, Kotlin library for building clients:

  • Update network library ktor to latest release 1.3.0
  • Add option to use authentication with GET /_matrix/client/r0/publicRooms to fix HTTP Unauthorized errors with newer Synapse versions which may have allow_public_rooms_without_auth set to False.

Continuum desktop client based on Koma:

  • Fix empty room directory list by always using the authenticated API POST /_matrix/client/r0/publicRooms

๐Ÿ”—Dept of Ops ๐Ÿ› 

๐Ÿ”—matrix-docker-ansible-deploy updates

Slavi told us:

matrix-docker-ansible-deploy can now easily configure automatic Double Puppeting for all of its Mautrix bridges (Facebook, Hangouts, Whatsapp, Telegram). This is possible due to those bridges' integration with matrix-synapse-shared-secret-auth - yet another component that the playbook can install. Our documentation page for each bridge provides relevant information on how to get it enabled.

Also:

matrix-docker-ansible-deploy can now install and configure matrix-appservice-webhooks, thanks to a contribution from Bjรถrn Marten from netresearch. Our documentation page about Appservice Webhooks tells you how to get started.

๐Ÿ”—auto-update Riot-Web script

@murz:ru-matrix.org said:

I have created simple Bash script for auto-update Riot-Web to latest version. Because https://github.com/vector-im/riot-web now have no public Wiki, I publish it as Gist here: https://gist.github.com/MurzNN/ee64f98ab2e71b886c41d55594e5dd9e

๐Ÿ”—ruby-grafana-matrix ingress updates

Ananace said:

My Grafana notification ingress gem received the missing piece of functionality for configuring it for posting both m.text as well as the original m.notice messages (as well as defaulting to m.text now as the more common expectation) - for those that want the messages to also provide Matrix notifications instead of just acting as an added history/flow. This can also be configured per-rule, so that you are able to have multiple notification ingresses configured with different message types.

It's always so lovely when people - other than myself - use my projects, especially when providing feedback on them. So many simple-to-implement features that get forgotten just because they weren't part of the original MVP.

๐Ÿ”—Dept of Bots ๐Ÿค–

๐Ÿ”—TWIM bot updated for opsdroid 0.17

@cadair:cadair.com offered:

TWIM bot has been updates to use all the new features of opsdroid 0.17 it now supports Edits, both in updating it's database and by editing messages which get posted to #twim_updates:cadair.com . Also it supports 'TWIMing' a post by the user who posted it reacting with โญ•๏ธ, this is mainly designed as an easy way to mark images as TWIM posts. ๐Ÿš€

๐Ÿ”—Dept of Ping ๐Ÿ“

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1bau-ha.us213.5
2foodscience.rocks254
3getflexedon.me298
4kif.rocks375
5maunium.net385
6gottliebtfreitag.de405
7netzgemeinde.eu408
8deadgoldfish.party423
9matrix.envs.net430
10lyseo.edu.ouka.fi456

๐Ÿ”—That's all I know ๐Ÿ

See you next week, and be sure to stop by #twim:matrix.org with your updates!

This Week in Matrix 2020-01-10

10.01.2020 00:00 โ€” This Week in Matrix โ€” Ben Parsons

๐Ÿ”—Matrix Live ๐ŸŽ™

๐Ÿ”—Dept of Spec ๐Ÿ“œ

anoa informed us:

We've had a slight lull from people crawling out from winter holiday hibernation caves, but there's likely more to come as everyone gets back into the swing of things.

Merged MSCs

No MSCs were merged this week.

MSCs in Final Comment Period

No MSCs are currently in FCP.

New MSCs

The Spec Core Team is continuing to work on implementation of existing MSCs.

๐Ÿ”—Dept of Servers ๐Ÿข

๐Ÿ”—Synapse 1.8.0 is out now

Get the latest news here!

Synapse 1.8.0 has arrived, it contains a whole host of bug fixes and tweaks, most notably fixing some long standing problems with search.
More generally we are spending a lot of time improving the e2ee experience ahead of switching on e2ee by default, so watch this space.

๐Ÿ”—Deploying Synapse

Several packaging projects have been updated to deploy the new version:

๐Ÿ”—matrix-media-repo v1.0.0 - repeat, v1.0.0!!

TravisR offered:

matrix-media-repo v1.0.0 has been released! If it is suitable for your environment, please give it a go.

๐Ÿ”—Dendrite federation work!

Neil Alexander announced:

Federation fixes have been pushed to gomatrixserverlib and Dendrite's master branches, that include the following tweaks:

  • Using the v2 endpoints for /send_join and /send_leave
  • Fixing the resolution of room aliases to room IDs in the Dendrite federation API
  • A rewrite of the auth chain functions which should be a bit smoother
  • A couple of other minor tweaks to some of the types

In addition, I've spent the last few days working on Dendrite's storage backends, adding somewhat-hacky support for SQLite and investigating ORM modelling for some of the simpler components, as a part of getting Dendrite to run as a "true monolith" for the P2P work.

๐Ÿ”—cortex (synapse worker in Rust) supports e2ee rooms

Black Hat offered:

cortex's federation sender finally supports e2ee rooms! I'm testing its performance on an Intel Atom z8350 and it looks great.

๐Ÿ”—Dept of Bridges ๐ŸŒ‰

๐Ÿ”—Matrix App for Zapier

@coppero1237 appeared to us, then announced:

The MVP is now available: https://zapier.com/developer/public-invite/77712/033209ffe96c0c0cdd618c8071355c01/

Use the Zapier App to integrate your Matrix room with any of Zapier's 1500+ apps, including:

  • Github
  • Trello
  • Pagerduty
  • Google calendar
  • Jira
  • Salesforce

Currently the MVP supports sending messages to a room. Reading messages from a room is future work.

If you're interested providing feedback, requesting a feature, future development, or just understanding how it works, please join the matrix room, #zapier:matrix.org

Source code: https://github.com/tyleradams/Zapier-Matrix

๐Ÿ”—matrix-zammad bridge

Half-Shot offered:

matrix-zammad now supports reacting to tickets to close them, provided you've set up your puppeted token in the config. It's useful if you get a lot of spam tickets :p

๐Ÿ”—mx-puppet-bridge

sorunome reported:

Lots of changes in mx-puppet-bridge!

  • Allow sending status messages into bridged rooms
  • add a bridgeChannel function for protocol implementations
  • add config options to set displayname and avatar url of the AS bot
  • leave the bridge bot of a bridged room, whenever possible
  • auto-leave a puppeted ghost after an hour inactivity (to prevent DMs having three users in them)
  • automatically dedupe media when uploading
  • matrix group to remote protocol group mapping

๐Ÿ”—mx-puppet-discord

Along with mx-puppet-bridge, things got implemented in mx-puppet-discord!

  • allow bridging of single channels in a guild
  • display an error if sending a message to discord fails
  • add joinentireguild command
  • handle webhooks properly
  • map discord guilds to matrix groups

If you enjoy this software, please consider to donate, thank you! ๐ŸฆŠ

And another mx-puppet-discord update!

  • [User Tokens] Proper User-Agent spoofing - friends management now seems mostly safe! Leaving the warning thing in just-in-case, though

๐Ÿ”—Dept of Clients ๐Ÿ“ฑ

๐Ÿ”—๐Ÿ“ฝ๏ธ Matrix Presents!

I'm adding this to the client section because it behaves like a client, but it's not what we'd normally think of...

Half-Shot reported:

Work has resumed on matrix-presents, newly rewritten in Vue.js! The project was first demoed back in Oggcamp 2018 where I gave a meta presentation on the virtues Matrix for other mediums. This time around, it's being brought back with:

  • A user interface for managing, joining and creating slideshows.
  • Control over how slides are advanced (pinned to the presenters view, or unlocked)
  • Finer control over how slides are laid out, using a fragments system to build slides out of submessages.

This is currently in heavy development and will debut at Fosdem 2020!

A regularly updated version of the app is hosted at https://presents.half-shot.uk/.

(And for those of you expecting a form of table tennis, there is an easter egg in progress ๐Ÿ˜ƒ)

๐Ÿ”—Continuum, desktop client in Kotlin

yuforia offered:

Continuum, a desktop client in Kotlin:

  • Update README to include information on building from source
  • Fix: database not updated after leaving a room
  • Move user's access token and list of joined rooms from database to more lightweight key-value storage

๐Ÿ”—RiotX v0.12.0

benoit reported:

RiotX v0.12.0 has been released with some performance improvement, especially on initial sync and on timeline loading time. The release also contains bugfixes and a cleanup in the application settings. A new "developer mode" has been added to show advanced features only to power users. We are now working on the room profile screen, and we are making progress on the cross-signing implementation. Besides that, we are working to make RiotX available on the F-Droid store.

I must say RiotX is getting a lot more stable and reliable recently!

๐Ÿ”—Riot iOS

Manu told us:

We are still working on the implementation of cross-signing and verification by DM.

๐Ÿ”—Dept of Ops ๐Ÿ› 

๐Ÿ”—K8s

In addition to mentioning Synapse 1.8.0 support, Ananace said:

Synapse 1.8.0 Kubernetes-optimized images are pushed, I've also updated the example manifests as part of some work on making it easier to deploy - expect a MVP of a Helm Chart some time Soonโ„ข

Also, to add a bit of context/information to this;

Helm is the de-facto standard package manager for Kubernetes clusters, where a Chart is a package for an application that can be installed and configured. The Helm Chart I'm working on won't be a one-click install to begin with, for the Minimum Viable Product it will require manually generating and storing the Synapse signing key - though I have thoughts on how to later delegate that to a small one-time job that Helm can run if necessary

๐Ÿ”—Opsdroid 0.17

Cadair offered:

Opsdroid 0.17 was released in December, it comes with many changes but the main matrix improvements are support for extra event types like Replies, Edits, Reactions, Room Name changes, Room Topic changes, Alias changes, Room avatar changes, Power levels, and support for generic matrix state events. All these events are supported for both sending and implementing skills based on receiving them.

๐Ÿ”—Dept of Ping ๐Ÿ“

Let's reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1getflexedon.me180.5
2maclemon.at278
3im.leptonics.com299
4maunium.net327
5lkas.cc339.5
6services.pyrahex.com351
7tx0.co361
8matrix.vgorcum.com381
9nerdsin.space406
10neko.dev442.5

๐Ÿ”—That's all I know ๐Ÿ

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Synapse 1.8.0 released

09.01.2020 00:00 โ€” Releases โ€” Neil Johnson

Synapse 1.8.0 has arrived, it contains a whole host of bug fixes and tweaks, most notably fixing some long standing problems with search.

More generally we are spending a lot of time improving the e2ee experience ahead of switching on e2ee by default, so watch this space.

Get the new release from github or any of the sources mentioned at https://github.com/matrix-org/synapse/blob/master/INSTALL.md.

๐Ÿ”—Synapse 1.8.0 (2020-01-09)

๐Ÿ”—Bugfixes

  • Fix GET request on /_synapse/admin/v2/users endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. (#6563)
  • Fix incorrect signing of responses from the key server implementation. (#6657)

๐Ÿ”—Synapse 1.8.0rc1 (2020-01-07)

๐Ÿ”—Features

  • Add v2 APIs for the send_join and send_leave federation endpoints (as described in MSC1802). (#6349)
  • Add a develop script to generate full SQL schemas. (#6394)
  • Add custom SAML username mapping functionality through an external provider plugin. (#6411)
  • Automatically delete empty groups/communities. (#6453)
  • Add option limit_profile_requests_to_users_who_share_rooms to prevent requirement of a local user sharing a room with another user to query their profile information. (#6523)
  • Add an export_signing_key script to extract the public part of signing keys when rotating them. (#6546)
  • Add experimental config option to specify multiple databases. (#6580)
  • Raise an error if someone tries to use the log_file config option. (#6626)

๐Ÿ”—Bugfixes

  • Prevent redacted events from being returned during message search. (#6377, #6522)
  • Prevent error on trying to search a upgraded room when the server is not in the predecessor room. (#6385)
  • Improve performance of looking up cross-signing keys. (#6486)
  • Fix race which occasionally caused deleted devices to reappear. (#6514)
  • Fix missing row in device_max_stream_id that could cause unable to decrypt errors after server restart. (#6555)
  • Fix a bug which meant that we did not send systemd notifications on startup if acme was enabled. (#6571)
  • Fix exception when fetching the matrix.org:ed25519:auto key. (#6625)
  • Fix bug where a moderator upgraded a room and became an admin in the new room. (#6633)
  • Fix an error which was thrown by the PresenceHandler _on_shutdown handler. (#6640)
  • Fix exceptions in the synchrotron worker log when events are rejected. (#6645)
  • Ensure that upgraded rooms are removed from the directory. (#6648)
  • Fix a bug causing Synapse not to fetch missing events when it believes it has every event in the room. (#6652)

๐Ÿ”—Improved Documentation

๐Ÿ”—Deprecations and Removals

  • Remove redundant code from event authorisation implementation. (#6502)
  • Remove unused, undocumented /_matrix/content API. (#6628)

๐Ÿ”—Internal Changes

  • Add experimental support for multiple physical databases and split out state storage to separate data store. (#6245, #6510, #6511, #6513, #6564, #6565)
  • Port sections of code base to async/await. (#6496, #6504, #6505, #6517, #6559, #6647, #6653)
  • Remove SnapshotCache in favour of ResponseCache. (#6506)
  • Silence mypy errors for files outside those specified. (#6512)
  • Clean up some logging when handling incoming events over federation. (#6515)
  • Test more folders against mypy. (#6534)
  • Update mypy to new version. (#6537)
  • Adjust the sytest blacklist for worker mode. (#6538)
  • Remove unused get_pagination_rows methods from EventSource classes. (#6557)
  • Clean up logs from the push notifier at startup. (#6558)
  • Improve diagnostics on database upgrade failure. (#6570)
  • Reduce the reconnect time when worker replication fails, to make it easier to catch up. (#6617)
  • Simplify http handling by removing redundant SynapseRequestFactory. (#6619)
  • Add a workaround for synapse raising exceptions when fetching the notary's own key from the notary. (#6620)
  • Automate generation of the sample log config. (#6627)
  • Simplify event creation code by removing redundant queries on the event_reference_hashes table. (#6629)
  • Fix errors when frozen_dicts are enabled. (#6642)

Get the new release from github or any of the sources mentioned at https://github.com/matrix-org/synapse/blob/master/INSTALL.md.

This Week in Matrix 2020-01-03

03.01.2020 00:00 โ€” This Week in Matrix โ€” Ben Parsons

๐Ÿ”—Matrix Live ๐ŸŽ™

๐Ÿ”—Dept of Status of Matrix ๐ŸŒก

๐Ÿ”—Responses to The Ecosystem is Moving

Matthew wrote a response to https://signal.org/blog/the-ecosystem-is-moving/ : https://matrix.org/blog/2020/01/02/on-privacy-versus-freedom/.

This is in response to a talk Moxie Marlinspike gave at 36c3. There is another response here from the XMPP community.

๐Ÿ”—Dept of Servers ๐Ÿข

๐Ÿ”—Dendrite and gomatrixserverlib

Neil Alexander announced:

My first TWIM update as a member of NV but I'm happy to announce that with some experimental changes to Dendrite and gomatrixserverlib, I've been able to get Dendrite-to-Dendrite federation working, which is a key component for the P2P work that we are planning!

๐Ÿ”—matrix-media-repo v1.0.0-rc.2

TravisR reported:

matrix-media-repo has received its first ever release candidate (finally): v1.0.0-rc.1. It's complicated to set up, but please do give it a go if it suites your environment and use case, and report any bugs along the way.

v1.0.0-rc.2 was released later in the week to fix a small bug with exports in v1.0.0-rc.1.

๐Ÿ”—Synapse 1.7.3 released

Synapse 1.7.3 includes an important bugfix, hosts are encouraged to upgrade.

Ananace reported that the K8s optimized Synapse 1.7.3 images have been updated, but you should find that all distributors have the latest version now.

๐Ÿ”—Dept of Bridges ๐ŸŒ‰

๐Ÿ”—mx-puppet-bridge

sorunome said:

mx-puppet-bridge received some updates!

  • Automatic Puppeting, thanks to tulir PR!
  • Allow protocol implementations to specify an external_url as per spec
  • Double Puppeting: Option to specify a custom homeserver -> URL map for local setups where .well-known resolution isn't possible
  • Fix unbridging of rooms (remove alias correctly)
  • Set filename of uploaded avatars to circumvent a synapse bug
  • Update matrix-bot-sdk dependency to 0.4.0

๐Ÿ”—mx-puppet-discord and mx-puppet-slack

sorunome again:

Both mx-puppet-discord and mx-puppet-slack updated to the newest library version and implemented the setting of an external URL.

If you like these projects, please check out the support chat and consider to donate!

๐Ÿ”—mautrix-telegram

Tulir said:

mautrix-telegram v0.7.0 was released.

Other than bugfixes, there were a few main changes since the first release candidate a month ago:

  • Option for automatic custom puppeting using shared secret login
  • version command to get the exact bridge version (also coming to my other bridges)
  • Config checks to make sure important fields like permissions aren't left unconfigured

Full changelog available on GitHub: https://github.com/tulir/mautrix-telegram/releases/tag/v0.7.0

๐Ÿ”—mautrix-whatsapp

Tulir again:

Sticker bridging works slightly better now and added automatic double puppeting here too.

๐Ÿ”—mautrix-cookiemonster

Tulir again:

To make the mautrix-facebook and mautrix-hangouts login flow simpler, I'm making a browser extension to eat cookies automatically rather than having the user go into the devtools. It's currently in development and should be ready by next week.

๐Ÿ”—maubot

Tulir, busiest fellow in the North, again:

I finally got around to actually making the maubot github plugin, which is now spamming commits and other github things in all my project rooms.

Next up is per-room config options for webhooks and more matrix -> github actions (currently it's mostly just the github -> matrix webhooks). I might also end up making a more advanced plugin configuration system that supports multiple files, since jinja2 templates in a yaml file isn't that nice.

๐Ÿ”—Dept of Clients ๐Ÿ“ฑ

๐Ÿ”—FluffyChat for Android and iOS in Flutter

@krille:ubports.chat said:

MTRNord and me are working on FluffyChat for Android and iOS based on Flutter. You can already check it out if you like. :-) Install using F-Droid: https://mtrnord.gitlab.io/fluffychat-flutter-fdroid/fdroid/repo/ More info here: https://www.ko-fi.com/post/FluffyChat-for-Android-and-iOS-S6S71BMEY

๐Ÿ”—Continuum

yuforia said:

Continuum, desktop client based in Kotlin, version 0.9.34:

  • Perform database operations in an async way on IO dispatcher
  • Added loading indicator when loading messages from server

continuum

๐Ÿ”—Dept of Ping ๐Ÿ“

Let's reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

Tulir:

The ping room was upgraded to v5 last weekend. People who joined early got some nice and low pings before everyone rejoined :D

RankHostnameMedian MS
1kif.rocks240
2maunium.net260
3eisfunke.com298
4pixie.town311
5hackerspaces.be336
6synod.im376.5
7matrix.vgorcum.com383
8flobob.ovh394
9aryasenna.net407
10lyseo.edu.ouka.fi418

Also, @lub:imninja.net told us:

I created a #ping:maunium.net compatible bot in PowerShell https://gitea.lubiland.de/lub/pingposh/src/branch/master

Very nice - we don't see a lot of PowerShell in this ecosystem.

๐Ÿ”—Final Thoughts ๐Ÿ’ญ

Work is still happening on Cross Signing, patience will be rewarded sooner than you know.

Half-Shot is working on a fun and genuinely useful non-chat Matrix application, more on this soon.

36c3 was exciting, exhausing and educational. Thanks to everyone who made the Matrix Assembly one of the liveliest places around. :D

๐Ÿ”—That's all I know ๐Ÿ

See you next week, and be sure to stop by #twim:matrix.org with your updates!

On Privacy versus Freedom

02.01.2020 00:00 โ€” Thoughts โ€” Matthew Hodgson

A few years ago, back when Matrix was originally implementing end-to-end encryption, we asked Moxie (the project lead for Signal) whether heโ€™d ever consider connecting Signal (then TextSecure) to Matrix. After all, one of Matrixโ€™s goals is to be an interoperability layer between other communication silos, and one of the reasons for us using Signalโ€™s Double Ratchet Algorithm for Matrixโ€™s encryption was to increase our chances of one day connecting with other apps using the same algorithm (Signal, WhatsApp, Google Allo, Skype, etc). Moxie politely declined, and then a few months later wrote โ€œThe ecosystem is movingโ€ to elaborate his thoughts on why he feels he โ€œno longer believes that it is possible to build a competitive federated messenger at all.โ€

At the time we didnโ€™t respond via a blog post; instead we ended up talking it through a few times in person to see how misaligned we really were. The conclusion was that we agreed to disagree and Moxie said heโ€™d be happy to be proved wrong, and wished us good luck. However, the subject has come up again thanks to Moxieโ€™s talk on the same subject at 36C3 last week, and we keep getting asked to write a formal response on the Matrix side. So, hereโ€™s an attempt to do so. (Moxie didnโ€™t want the 36C3 talk recorded, and I havenโ€™t watched it, so this is responding to the original blog post).

From my perspective, the main points proposed in โ€˜The ecosystem is movingโ€™ boil down to:

  • Decentralised systems are harder to design and build than centralised ones, as coordination is harder if you donโ€™t have a single authority to trust.

  • Decentralised systems are harder and slower to evolve than centralised ones, as you canโ€™t force participants to rapidly roll out (or even agree on) new features.

  • Users in federated systems tend to coalesce around the best/biggest server that the bulk of people use - which means that server typically gets to see a disproportionate amount of communication metadata (whoโ€™s talking to who, and when), and has disproportionate power over the network, which could bully others away from running their own deployments.

  • If users donโ€™t trust their app provider, they can always go switch apps, which gives them freedom.

  • Open systems are less secure because you have no control over the quality of the implementations - if anyone can bring their own client or server to the table, all it takes is one bad implementation to compromise everyone in the vicinity.

Now, all of these points are valid to some extent.

Itโ€™s absolutely true that decentralised systems are harder than centralised ones. Prior to Matrix we built centralised comms systems - we literally can do a side-by-side comparison for the same team to see how easily and fast we built our centralised comms system relative to Matrix. Empirically It took us around 6 times longer to get to the same feature-set with Matrix.

Itโ€™s also true that decentralised systems are harder to evolve than centralised ones - you canโ€™t just push out a given feature with a single app update, but you have to agree and publish a public spec, support incremental migration, and build governance processes and community dynamics which encourage everyone to implement and upgrade. This is hard, but not impossible: weโ€™ve spent loads of time and money on Matrixโ€™s governance model and spec process to get it right. Itโ€™s still not perfect, but we havenโ€™t seen much fragmentation so far, and when weโ€™re pushing out a feature empirically we can and do go just as fast as the centralised alternatives. (E2E by default is a bit of a special case because weโ€™ve had to go and reimplement many features users take for granted today in an E2E-capable manner, but weโ€™re sprinting to get it done in the coming weeks). A bigger problem is that there are hundreds of spec change proposals which folks would like to see in the protocol, and finding a way to manage expectations and parallelise spec progress is hard - something weโ€™re looking to improve in 2020 (although still figuring out how!)

Itโ€™s also fair that in a multi-server federated model, users naturally tend to sign up on the most prominent server(s) (e.g. the matrix.org homeserver in the case of Matrix). In practice, the matrix.org homeserver currently makes up about 35% of the visible Matrix network by active users. Itโ€™s also true that Matrix servers currently store metadata about whoโ€™s talking to who, and when, as a side-effect of storing and relaying messages on behalf of their users. And without an adequate protocol governance system in place, a large server could start pushing around smaller ones in terms of protocol behaviour. In practice, weโ€™re looking into solving metadata protection in Matrix by experimenting with hybrid P2P / Client Server models - letting users store their metadata purely clientside if they so desire, and potentially obfuscating whoโ€™s talking to who via mixnets of blinded store & forward servers (more about this coming up at FOSDEM). Combined with nomadic accounts, this would let us eventually turn off the matrix.org server entirely and eliminate the pseudo-centralisation effect - the default โ€˜serverโ€™ would be the one running on your client.

Itโ€™s true that if a user doesnโ€™t trust (say) Telegram, they are free to go switch to Signal or WhatsApp or whatever insteadโ€ฆ at the massive expense of having to persuade all their friends to install yet another app, and fragmenting their conversation history across multiple apps.

Finally, itโ€™s also true that because anyone can develop a Matrix client or server and connect to the global network, thereโ€™s a risk of bad quality implementations in the wild. There are many forks of Riot on the app stores - we simply canโ€™t vouch for whether they are secure. Similarly there are Matrix clients whose E2E encryption is partial, missing, or unreviewed. And there are a wide range of different Matrix servers run by different people with different agendas in different locations, which may be more or less trustworthy.

HOWEVER: all of this completely ignores one critical thing - the value of freedom. Freedom to select which server to use. Freedom to run your own server (perhaps invisibly in your app, in a P2P world). Freedom to pick which country your server runs in. Freedom to select how much metadata and history to keep. Freedom to choose which apps to use - while still having the freedom to talk to anyone you like (without them necessarily installing yet another app). Freedom to connect your own functionality - bots, bridges, integrations etc. Freedom to select which identifiers (if any) to use to register your account. Freedom to extend the protocol. Freedom to write your own client, or build whole new as-yet-unimagined systems on top.

Itโ€™s true that if youโ€™re writing a messaging app optimised for privacy at any cost, Moxieโ€™s approach is one way to do it. However, this ends up being a perversely closed world - a closed network, where unofficial clients are banned, with no platform to build on, no open standards, and you end up thoroughly putting all your eggs in one basket, trusting past, present & future Signal to retain its values, stay up and somehow dodge compromise & censorshipโ€ฆ despite probably being the single highest value attack target on the โ€˜net.

Quite simply, that isnโ€™t a world I want to live in.

We owe the entire success of the Internet (let alone the Web) to openness, interoperability and decentralisation. To declare that openness, interoperability and decentralisation is โ€˜too hardโ€™ and not worth the effort when building a messaging solution is to throw away all the potential of the vibrancy, creativity and innovation that comes from an open network. Sure, you may end up with a super-private messaging app - but one that starts to smell alarmingly like a walled garden like Facebookโ€™s Internet.org initiative, or an AOL keyword, or Googleโ€™s AMP.

So, we continue to gladly take up Moxieโ€™s challenge to prove him wrong - to show that itโ€™s both possible and imperative to create an open decentralised messaging platform which (if you use reputable apps and servers) can be as secure and metadata-protecting as Signalโ€ฆ and indeed more so, given you can run your server off the grid, and donโ€™t need to register with a phone number, and in future may not even need a server at all.

--Matthew

(Comments over at HN)

Synapse 1.7.3 released

31.12.2019 00:00 โ€” Releases โ€” Matthew Hodgson

Hi all,

We've just released Synapse 1.7.3 - an important bug fix to address a class of failures due to malformed events. We've seen this in the wild over the last few days, so we'd recommend updating as soon as possible, especially if you are having problems federating.

Get the new release from github or any of the sources mentioned at https://github.com/matrix-org/synapse/blob/master/INSTALL.md.

The changelog since 1.7.2 is:

๐Ÿ”—Synapse 1.7.3 (2019-12-31)

This release fixes a long-standing bug in the state resolution algorithm.

๐Ÿ”—Bugfixes

  • Fix exceptions caused by state resolution choking on malformed events. (#6608)

This Week in Matrix 2019-12-27

27.12.2019 23:59 โ€” This Week in Matrix โ€” Ben Parsons
Last update: 27.12.2019 23:52

๐Ÿ”—Dept of Status of Matrix ๐ŸŒก

๐Ÿ”—36c3: Matrix Assembly is the place to be

If you're at 36c3 this weekend, come and find us! Use c3nav app to find our assembly, or just join #chaosevents:matrix.org to come chat

๐Ÿ”—Bundeswehr considering Matrix

Oleg said:

The German Army is considering using Matrix as "secure WhatsApp" for soldiers. (In German) https://www.heise.de/newsticker/meldung/Open-Source-Bundeswehr-baut-eigene-verschluesselte-Messenger-App-4623404.html

๐Ÿ”—Dept of Servers ๐Ÿข

๐Ÿ”—matrix-oauth

TravisR reported:

For those who want to integrate Matrix into their application with OAuth, there's now matrix-oauth ( #oauth:t2bot.io ). Ideally useful for "Login with Matrix" buttons, this is a relatively easy OAuth 2.0 provider to set up in front of your homeserver. In future it'll support more granular scopes to avoid having to give a real access token to the third party application.

A demo of matrix-oauth in action is available at https://demo.oauth.t2host.io/

๐Ÿ”—Dept of Bridges ๐ŸŒ‰

๐Ÿ”—Amazon Alexa skill

TravisR offered:

Yelling at your Amazon Alexa to send a message to your Matrix contacts is now possible with matrix-alexa-skill ( #alexa:t2bot.io ). A hosted version using matrix-oauth is coming soon, assuming I can get the privacy policy and such over to Amazon to review in a timely manner, though you're more than able to self-host in a matter of minutes. Check out the README for more info - some experience with setting up complicated things is required.

๐Ÿ”—mx-puppet-bridge

sorunome offered:

mx-puppet-bridge got a new feature: protocol implementations can now specify custom commands that are invoked via the provisioning room!

๐Ÿ”—mx-puppet-discord

mx-puppet-discord received quite a few bug fixes and new features!

  • Fix echo back of edits
  • [User Tokens] being friends is enough now to DM each other!
  • fix multi-edits
  • [User Tokens] support group DMs
  • Implement ability to bridge guilds!
  • [User Tokens] add friends management

Description on how to use these features are found in the readme!

If you enjoy these projects, please consider to donate. Thank you!

๐Ÿ”—Dept of Clients ๐Ÿ“ฑ

๐Ÿ”—Spectral gains public room directory

Black Hat reported:

Public room directory and user directory support in Spectral is finally there!

spectral room directory

๐Ÿ”—Continuum, plus koma library

yuforia offered:

koma, a Kotlin library. Dominic Fischer (github: Dominaezzz) started working on the project last week and so far:

  • In preparation for multiplatform support, converted JVM code to agnostic Kotlin, using the library atomicfu

  • Added Github Actions configuration to run builds automatically

Continuum, desktop client based on Koma:

  • Generate room name from members when neither name nor aliases are configured

๐Ÿ”—Dept of Ops ๐Ÿ› 

๐Ÿ”—ma1sd 2.2.2 released

ma1uta announced:

ma1sd (fork of the mxisd) 2.2.2 released: https://github.com/ma1uta/ma1sd/releases/tag/2.2.2 Changes:

  • bugfix
  • added hash lookup for the ldap provider.

๐Ÿ”—Dept of Services ๐Ÿš€

๐Ÿ”—kapsi.fi has set up a Matrix homeserver

Cos reported:

Finnish non-profit hosting service kapsi.fi has set up a Matrix homeserver for their members. Kapsi has around 5000 members and 20 volunteer administrators. Instructions for use (in FInnish) at https://www.kapsi.fi/palvelut/matrix.html

๐Ÿ”—Dept of Bots ๐Ÿค–

๐Ÿ”—MatrixVideo2oggBot

@progserega:rsprim.ru reported:

Matrix bot for converting youtube video to voice.

Bot https://github.com/progserega/MatrixVideo2oggBot get youtube URL, download video, convert to ogg-vorbis audio and send it to user. Some times my friends give me youtube video-urls, but I do not have time for see it. But I have time when I go home in car. But on road network is not always good and at end of day battery is low and phone may be hot (when I connect to charger and play video) and freeze... Simple way for me - is convert youtube video to small size voice and download it to phone and play it as music in player playlist. Bot help to this. May be it help anybody also. ๐Ÿ™‚

๐Ÿ”—Matrix in the News ๐Ÿ“ฐ

Andres offered:

Matrix gets a mention alongside other four decentralized protocols in one of the biggest argentinian newspapers (regarding Twitter's iniciative of decentralization). https://www.lanacion.com.ar/tecnologia/cinco-iniciativas-descentralizar-redes-sociales-dejar-depender-nid2317548

๐Ÿ”—Dept of Ping ๐Ÿ“

RankHostnameMedian MS
1getflexedon.me211.5
2thinker.eu.org306
3maunium.net432
4dodsorf.as438
5lyseo.edu.ouka.fi455
6matrix.vgorcum.com562
7uraziel.de626
8tout.im640.5
9kapsi.fi650
10encom.eu.org862

๐Ÿ”—That's all I know ๐Ÿ

See you next week, and be sure to stop by #twim:matrix.org with your updates!

The 2019 Matrix Holiday Update!

24.12.2019 00:00 โ€” General โ€” Matthew Hodgson

Hi all,

Every year we do an annual wrap-up and retrospective of all the things happening in the Matrix core team - if youโ€™re feeling particularly curious or bored you can check out the 2015, 2016, 2017 and 2018 editions for context. The idea is to look at the bigger picture trends in Matrix outside of the weekly TWIM posts to get an idea of the stuff which we made progress on, and the stuff which still remains.

That said, itโ€™s hard to know where to start - Matrix accelerated more than ever before in 2019, and thereโ€™s been progress on pretty much all battlefronts. So as a different format, letโ€™s take the stuff we said we had planned for 2019 from the end of last yearโ€™s update and see what we actually achieved...

๐Ÿ”—2019: the immediate priorities

So, our immediate priorities for 2019 were:

  • r0 spec releases across the board (aka Matrix 1.0)
  • Implementing them in Synapse

โœ… Well, unless youโ€™ve been floating in a sensory deprivation tank for the last year, hopefully you spotted that Matrix (as a protocol) finally exited beta - starting off with the announcement at FOSDEM in February of the first stable release of the Server-Server API, alongside the Synapse 0.99.x series as we began the process of migrating to the 1.0 APIs.

Specifically this meant killing off self-signed certificates, adding .well-known server discovery and implementing room version semantics so we could upgrade the underlying room version algorithm to fix the residual flaws. This culminated in June with the official release of Matrix 1.0 - now including the remaining APIs and a stable release of Synapse 1.0. The emphasis was on addressing all the main pre-1.0 design flaws rather than adding features or performance, but with 1.0 out the door at last weโ€™ve been able to make progress there too.

  • Landing the Riot redesign

โœ… The full redesign of Riotโ€™s UI on Web/Desktop landed shortly after FOSDEM in Feb with The Big 1.0. Cosmetically we got most of the new look & feel in place, and have had very positive feedback overall - although some of the UX thinkos of the old app remain and coming up on the radar for fixing.

  • Finalising the Matrix.org Foundation

โœ… This happened too, coincident with releasing Matrix 1.0 in June - read all about it at https://matrix.org/foundation. So far the governance and legal infrastructure the Foundation provides has helped the project significantly, and while it was a mammoth task to organise, weโ€™re very glad itโ€™s here! Huge thanks go out to Jon, Ross and Jutta for agreeing to join the foundation as Guardians - they have been excellent in patiently listening to the various dramas of the year and ensuring Matrixโ€™s neutrality and that we keep an even keel.

  • Landing all the new E2E encryption UX and features

The good news on E2E encryption is that weโ€™ve been making solid progress throughout the year - the bad news is that we are still yet to turn it on by default. Progress updates for the various pieces of the puzzle are as follows:

  • โœ… The final UX is pretty much locked down (after several iterations as we try to tread the balance between trustworthiness and security) - hereโ€™s a sneak preview of what weโ€™re aiming at.

  • ๐Ÿ— Cross-signing is the single biggest remaining piece of work in progress - letting users attest to the trustworthiness of their own devices, so you only ever have to trust a given user once rather than trusting all their devices individually. We gave a very early demo of an experimental implementation back at FOSDEM in Feb, inspired by some of the initial spec proposal at MSC1680 (MSC = Matrix Spec Change, our process for evolving Matrix).

    However, having played with it a bit, MSC1680 turned out to be too generic and complicated (it worked by the user signing a device with any other of their devices, building a twisted maze of which device vouched for which) - and we replaced it with MSC1756, which shifts the model to be the simpler โ€œthe user has a key, which they use to sign their devicesโ€. However, this in turn requires more infrastructure - you need somewhere secure to store your signing key, which prompted MSC1946 - Secure Secret Storage & Sharing (SSSS): the ability to sync your signing key between devices by storing it (encrypted, of course) on the server.

    Meanwhile, it also became obvious that the primitives for key verification needed to be improved too: introducing verification by emoji comparison (MSC1267) and QR codes (MSC1543), and switching key verification to be performed in the context of a DM (MSC2241) so that you can see your verification history, find verifications, and easily dip in and out of verifying users as needed.

    Whilst everyone else was panicking about Matrix 1.0 and associated baggage, Uhoreg was off in the wilderness plugging his way through all of this - iterating on the design, speccing it and implementing it in synapse and matrix-js-sdk, complete with a test jig to demonstrate it all working (part 1 and part 2). Over the last few months the rest of the team has joined him though, and weโ€™ve been frantically working away implementing it all on both Riot/Web, iOS & RiotX/Android. For instance - hereโ€™s verification happening in DM between Riot/Web & RiotX a few weeks ago, and hereโ€™s a very early (unskinned) cut of verification happening in Riot/Webโ€™s RightPanel a few days ago.

    We were hoping to get cross-signing ready for the end of 2019, but in practice weโ€™re now sprinting to get it done by FOSDEM 2020 in Feb - not least because we have a main-stage talk proposed to tell everyone how we landed it and turned on E2E by default... ;)

  • โœ… Support for non-E2E clients. The last thing we want is to make it impossible to write a simple Matrix client, or to suddenly excommunicate (hah) all the existing Matrix bots & bridges which havenโ€™t implemented E2E. To this end, poljar created pantalaimon - our very own Matrix daemon, which can sit in the background and offload all your E2EE from your Matrix client by acting as a transparent Matrix proxy which magically encrypts everything. Built on matrix-nio and asyncio python3, We use it in production today for running various bots and it works excellently.

  • โœ… Support for search in E2E rooms. Hot off the heels of pantalaimonโ€™s success, poljar also created seshat - a native library for clientside indexing encrypted Matrix events written in Rust, powered by the tantivy full-text search engine. (pantalaimon also has support for indexing via tantivy, which involved contributing python bindings for tantivy, but we ended up going with Rust so we could embed it natively in as many Matrix clients as possible). Seshat is particularly cool in that the indexes themselves are encrypted in on disk - and in future could even be synced between clients using SSSS so you donโ€™t have to reindex your messages every time you log in on a new device. Seshat is implemented behind a labs flag on Riot/Desktop and it will ship as soon Riot/Desktopโ€™s build pipeline is fully updated to support native modules (which will also unlock other goodies, such as using faster/safer native E2E primitives, safer key storage, and Discord-style keyboard-shortcuts for VoIP).

  • ๐Ÿ— Fixing โ€œunable to decryptโ€ errors. Weโ€™ve done big sprint over the last month or so to track down the final straggling causes of unable to decrypt errors. Some of these are legitimate bugs (e.g. https://github.com/matrix-org/synapse/issues/6399) - but many are artefacts of the current architecture: for instance, if the sender has no way to know your device was in the room when it encrypted a message, you wonโ€™t be able to decrypt. Weโ€™re addressing this by improving better error messages and feedback so the user isnโ€™t surprised by whatโ€™s going on (aiming for Jan) - and in future weโ€™ll have to revisit E2Eโ€™s fundamentals to ensure that itโ€™s impossible to receive a message without also receiving the key to decrypt it.

  • โœ… Support for push notifications in E2E rooms. This is kinda solved right now by having all clients get (silently) pushed whenever they receive a message in an E2E room with push enabled, and relying on the client to be woken up by the push in order to decrypt the message in order to display the push notification. However, this is battery intensive, and we could probably do better - but this isnโ€™t a blocker for going live.

  • ๐Ÿ— Support for FilePanel and NotifPanel in E2E rooms. Seshat should fix this by indexing all your messages (and so tracking whether they contain pushes or files, and populating up your local view of your file & notif panels respectively) - just need to ensure itโ€™s hooked up.

...and thatโ€™s where things stand right now on E2E by default. Weโ€™ll start turning it on by default for private rooms as soon as the UX has landed (probably starting first with new DMs and private rooms, prompting the user in case they want to opt out - and then migrating existing ones). Itโ€™s worth noting that we have poured a lot of work into E2E encryption now - often to the detriment of the rest of Matrix; our rich featureset and decentralisation has combined to make this a tough nut to crack, but the end is in sight. Thanks to all for your patience and support while weโ€™ve been working through this.

That takes us to the end of the stuff we planned to prioritise in 2019 - but what about the more speculative medium-term stuff which was on the menu this time last year?

๐Ÿ”—2019: the medium-term priorities

  • Reworking and improving Communities/Groups.

We have some really promising UX work and a fairly early spec proposal (MSC1772), but work in earnest hasnโ€™t kicked off yet. Itโ€™s going to be one of the next big projects though.

  • Reactions.

โœ… Riot now has Reactions! ๐ŸŽ‰๐ŸŽ‰๐ŸŽ‰ The only remaining work is to finish the remaining rough edges of the spec proposal (MSC1849) and actually land them in the Matrix spec proper.

  • E2E-encrypted Search

โœ… Seshat exists! (see above)

  • Filtering. (empowering users to filter out rooms & content they're not interested in).

โœ… Weโ€™ve ended up thinking lots in 2019 about empowering users to filter content. The main impetus has been to ensure that users and communities can filter out abuse (on their own terms), and also to start building infrastructure which can be used for folks to share their own filters. Over the last few months, this has started to take concrete form - with the arrival of MSC2313 โ€œModeration policies as roomsโ€, and Mjolnir - a bot you can run to enforce moderation policies on your rooms. Itโ€™s all quite early, but we expect a lot more work in this space over the coming year (and itโ€™s wryly amusing that Twitter has also woken up to it being an interesting problem needing to be solved.)

  • Extensible events

Sorry folks; no progress here since a flurry of spec work (MSC1767) back in Jan 2019. The good news is that the spec proposal seems to be relatively well received. The bad news is that we havenโ€™t had bandwidth to finish reviewing it, implementing it and migrating it anywhere. It blocks a bunch of really useful stuff in Matrix, and there are users willing to pay for it (via New Vector) - weโ€™ll get to it as soon as we can.

  • Editable messages.

โœ… These landed too and are a thing of joy! Just need to merge MSC1849.

  • Extensible Profiles (we've actually been experimenting with this already).

Similar to Extensible Events, there was a flurry of spec work (MSC1769) back in Jan, but little progress since. This will also unlock a lot of really useful features - e.g. custom status, custom profile data, social timeline rooms etc. Weโ€™ll likely get to it shortly after communities work.

  • Threading.

๐Ÿ— So we actually landed label-based threading (MSC2326) in Synapse 1.6, but itโ€™s not exposed in Riot yet (or elsewhere). It doesnโ€™t have quite the same semantics as Slack-style threading; the idea is to filter down your room based on which messages are tagged as part of a given topic. However, itโ€™s very powerful, and itโ€™ll be fun to add it to Riot at some point in 2020. Meanwhile, better-than-label-based-threading is also on the cards, although slightly lower priority than some of the other stuff in this section.

  • Landing the Riot/Android rewrite

๐Ÿ— As you probably know, RiotX is a full rewrite of Riot/Android in Kotlin using modern AndroidX and Jetpack idioms - and it entered beta back in June. Since then weโ€™ve been frantically working away on both playing catch-up with the old appโ€ฆ as well as implementing all the new stuff (reactions, edits, new E2E verification, cross-signing etc) which makes no sense to waste time adding in Riot/Android, but also pushes out the timeline on RiotX itself.

Weโ€™re currently sprinting to try to get RiotX ready for FOSDEM in February - hopefully users will have felt the app starting to really stabilise over the last few months (it even supports breadcrumbs now!)

  • Considering whether to do a similar overhaul of Riot/iOS

๐Ÿ— Itโ€™s cheating a bit, but Manu (the lead developer on Riot/iOS and delivery manager of Riot/Mobile in general) has been hacking on an entirely new client called Messagerie in his spare time, using SwiftUI. The idea of throwing away the whole UI layer and replacing it with the latest best practices sounds suspiciously like RiotX - itโ€™ll be interesting to see how RiotX/iOS takes shape next year!

  • Scaling synapse via sharding the master process

We ended up bottlenecked on IO rather than CPU in 2019, and as a result we worked on splitting synapseโ€™s database across multiple database instances on a per-table granularity. However, the master process itself doesnโ€™t shard yet; so weโ€™re now bottlenecked on CPU and need to get on and do this asap to unlock further Synapse scalability for mega-monolithic-deployments like the Matrix.org homeserver.

  • Bridge UI for discovery of users/rooms and bridge status

๐Ÿ— Thereโ€™s been a bit of movement in the last few weeks on this, but nothing concrete yet.

  • Bandwidth-efficient transports

โœ… We finished the 100bps CoAP transport proof-of-concept for Matrix, demoed it at FOSDEM and shipped it in March. However, we havenโ€™t progressed it much further; it really needs a corporate sponsor who wants to fund work to finish it off and bake it properly into Matrix. If youโ€™re interested, please get in touch.

  • Bandwidth-efficient routing

๐Ÿ— We also did a bunch of related work on bandwidth-efficient routing, which sadly hasnโ€™t been released yet. However, itโ€™s interesting to note that the Decentralized Systems and Network Services Research Group at Karlsruhe Institute of Technologyโ€™s Institute of Telematics has been looking into this space too - c.f. their A Glimpse of the Matrix paper, which ponders very similar problems.

  • Getting Dendrite to production.

๐Ÿ— Dendrite work has been bubbling away in the background thanks to Anoa, Brendan, cnly (our GSoC dendrite contributor) and others. Inevitably most of our bandwidth has gone into getting Synapse to 1.0 and making sure itโ€™s fit for purpose, but we want and need to keep Dendrite alive for next-generation purposes - and in fact New Vector is hiring new people to work on it in 2020.

  • Inline widgets (polls etc)

๐Ÿ— We have an MSC (MSC2192), but not an implementation.

  • Improving VoIP over Matrix.

Very little progress here, frustratingly. Jitsi has been upgraded and conference calls should kick ass these days (let us know if they donโ€™t), but 1:1 needs a lot of love. Hopefully weโ€™ll get to it in 2020.

  • Adding more bridges, and improving the current ones.

Lots of bridging progress in 2020 - all new puppeting Slack support; huge fixes to the IRC bridge (including shifting to Postgres at last); Bifrost (the XMPP bridge) progressed too, and thereโ€™s been loads of community bridging work around WhatsApp, Discord and others.

  • Account portability
  • Replacing MXIDs with public keys

Weโ€™ve just started looking at implementing these seriously via MSC1228 (as of last week) - expect progress in 2020.

So that sums up progress on the medium term menu - as you can see, a bunch actually happened; a bunch made progress; a few didnโ€™t happen at all.

๐Ÿ”—2019: the longer-term priorities

Finally, on the longer term radar:

  • Shared-code cross-platform client SDKs (e.g. sharing a native core library between matrix-{js,ios,android}-sdk)

No progress here. Instead, all three main platforms have continued to write and maintain their own platform-specific SDKs for now. Seshat however will be the first piece of native rust code shared across all 3 platforms - letโ€™s see how that goes first...

  • Matrix daemons (e.g. running an always-on client as a background process in your OS which apps can connect to via a lightweight CS API)

โœ… Pantalaimon lives!

  • Push notifications via Matrix (using a daemon-style architecture)

No progress here, unless you count the CoAP low-bandwidth work. However, Bubu (also Riot/Android Fdroid maintainer) has been working on a project called OpenPush which looks to help in this space (albeit not built on Matrix, but could be used by Matrix). There are a few other related projects. If someone wants to build this on top of Matrix + CoAP please get in touch asap!

  • Clientside homeservers (i.e. p2p matrix) - e.g. compiling Dendrite to WASM and running it in a service worker.

๐Ÿ— Work is actually happening on this currently. Dendrite has successfully compiled to WASM and runs, and weโ€™ve had it (almost) talking HTTP tunnelled over libp2p as part of P2P Matrix experiments. In 2020 weโ€™re going to be investing a lot in P2P Matrix - to give users full control of their communication without even having to run a server, and also to simplify onboarding and account portability enormously. We have a talk about this accepted for FOSDEM 2020 (The Path to P2P Matrix) and weโ€™re actively (frantically) hacking on Dendrite to make it happen - keep an eye out for how things develop!

  • Experimenting with MLS for E2E Encryption

๐Ÿ— Now that E2E-by-default has entered the โ€œit works! letโ€™s land it in Riot asapโ€ phase, Uhoreg has had some time to start thinking about the longer term future of encryption in Matrix. MLS (Messaging Layer Security) is the IETFโ€™s initiative to define a standard mechanism for end-to-end-encrypted group chats, which has some major algorithmic improvements over Olm/Megolm and the Double Ratchet Algorithm as used by Signal. The catch is that it doesnโ€™t work at all well with decentralisation - however, weโ€™ve been working with them to try to ensure MLS can work in a decentralised world. More recently, uhoreg has had a chance to think a lot more about this and weโ€™re working on a proposal for Decentralised MLS which builds on plain MLS while also giving the semantics needed for Matrix. Itโ€™s all very experimental at this point (and the proof-of-concept implementation is written in Julia!) - but looks promising. Weโ€™ll share more asap, and will certainly be investing more time in this in 2020..

  • Storing and querying more generic data structures in Matrix (e.g. object trees; scene graphs)

Sadly no progress here :(

  • Alternate use cases for VR, IoT, etc.

...and none here either.

So, of all the myriad things on our radar for 2019 (as of Dec 2018), hopefully this gives some idea of where we hit the mark.

๐Ÿ”—2019: the unpredictable bits

However, thereโ€™s also a tonne of other stuff which happened which wasnโ€™t explicitly on the radar. On the synapse side, we finished fully migrating from Python 2 to Python 3, and started using asyncio and all the latest Python 3 goodies! We finally implemented configurable history retention for servers and rooms! We even implemented self-destructing messages in Synapse (not that Riot exposes them yet). And there has been loads of optimisation and performance work since 1.0 landed in June.

On the ops side, we overhauled all our ops processes and security after the Matrix.org datacenter breach in April, throwing away our legacy infrastructure and rebuilding it properly - and subsequently have been expanding our ops team from one dedicated ops person to four. We also found ourselves having to do another emergency datacenter migration back in November when the old one was unable to reliably service IO for our database cluster.

We also spent a bunch time after shipping Matrix 1.0 working on tightening up Matrixโ€™s privacy model - particularly around third party identity servers, integration managers, and making sure that folks self-hosting Matrix donโ€™t accidentally depend on use 3rd party services without realising it. If you missed out on the fun at the time, you can read all about it here and here. This ended up being way more work than we expected, but weโ€™re very glad to have sorted it out now.

Meanwhile, mainstream uptake of Matrix has properly taken off, with the French Government launching Tchap (their fork of Riot), now with hundreds of thousands of daily active users. The German Government revealed today that they are also formally trialling Matrix, starting with the Bundeswehr (Ministry of Defense); weโ€™ve been helping them out with the deployment too. It is not an exaggeration to suggest that we could end up with an official cross-government Matrix network, publicly federated with the wider Internet, for self-hosted encrypted decentralised instant messaging. In fact Ulrich Kelber, the Bundesdatenschutzbeauftragte (Federal Data Protection Commissioner) for Germany pointed out: โ€œYou could even set up a privacy-friendly messenger service in cooperation with France, which in the medium term could represent a real alternative to existing products on the market as a pan-European solutionโ€.

Alongside all this, Mozilla announced they are replacing the Moznet IRC network with Matrix; KDE joined Matrix in Feb, Wikimedia is getting set up on their server, and more and more massive players (including the largest in the world) keep getting in touch to find out how they can best get onboard Matrix - itโ€™s incredibly exciting. It also means that we were able to raise capital to keep folks employed to work on Matrix fulltime via New Vector and scale up Modular.im as a paid hosting platform - which massively helps support core Matrix development.

๐Ÿ”—2020

All that remains now is to make some predictions for 2020. Our main priorities are:

  • Get E2E enabled for private rooms by default (see above).

  • Riot First-time User Experience (FTUE). While we redesigned Riotโ€™s UI in 2019, there are still far too many weird gotchas which trip over new users. Starting in October we began a shift to completely change how Riot development works - transitioning the project to being led by the UX design team rather than the dev team, and ensuring that the design team considers the app holistically across all 3 platforms. Above all else, our priority is to make it kick ass for normal non-technical mainstream users - not just for opensourcey wizards. This is a tough ask, but we believe itโ€™s literally make-or-break for the project in the long term if Matrix is ever to become as prevalent as Slack or WhatsApp, and we are throwing everything we have at it. The second that E2E is on by default, the entirety of the Riot teams will be focusing on the mission to clear our FTUE backlog.

  • RiotX. Weโ€™re shipping RiotX on Android as fast as we can - currently users on Riot/Android are left high and dry and we need to do everything we can to finish RiotX and get them upgraded as rapidly as possible.

  • Communities. Off the back of FTUE comes the importance of grouping rooms & users together into communities in a much better way than we have today. This will be up next.

  • Synapse: shard the master by user/room to avoid being it being bottlenecked on CPU. We also need to apply smarter queue management on federation traffic to better reduce the memory footprint (and so eliminate complexity limits on small-footprint hosted servers!) - and we also desperately need to speed up joins.

  • Dendrite & P2P Matrix: the plan currently is to use Dendrite as the basis for our P2P Matrix experiments. In practice this means making it federate using MSC1228-semantics (no point in wasting time implementing the โ€˜legacyโ€™ key management), and then experiment with hooking it up to various P2P transports (e.g. our low-bandwidth CoAP transport) and discovery systems (e.g. mDNS; libp2p; etc). How we go about actually getting it into production depends entirely on how well the experiment goes; we could evolve Matrix to be hybrid CS/P2P; we could treat it as a new protocol and bridge to it; who knows. Watch this space...

  • MLS: figure out our plan for next-generation E2E - for better scaling, and better reliability, and what (if anything) we should do with MLS.

  • Bridges: loads of work on the horizon to put a better UX on Bridging. Bridge stability has improved enormously over the last year (thanks Half-Shot!) but we need to transition from being robust but ugly to being robust and polished...

  • Spec: we need to work out how to go faster on reviewing MSCs (both our own and from the wider community). While the governance process in general feels healthier than itโ€™s ever been, empirically weโ€™re not exactly burning through the MSC backlog - and this is in part that MSC work is squeezed in alongside the other dayjob stuff everyoneโ€™s working on. Finding the right balance between sculpting spec and sculpting code is tough, but weโ€™re going to try to improve it in 2020.

  • Abuse / Reputation: we want to empower users to make their own minds up about what content they want to see and not see on Matrix (or what they want to host or not host on their servers / communities / rooms). Mjolnir is a good start, but weโ€™ll be continuing to work on this throughout the year.

Meanwhile, all the things listed above that we didnโ€™t get to in 2019 are of course still options on the menu too.

So there you have it. Iโ€™ve not even tried to talk about the amazing stuff that the wider Matrix community has been up to - whether thatโ€™s amazing new clients like Ditto (React Native!) or Nio! (SwiftUI), or new bridges like mautrix-facebook and mautrix-hangouts, or even poljarโ€™s secret rewrite of weechat-matrix in Rust; your best bet there is to skim through TWIM. Huge undying thanks go out though to everyone who builds on Matrix and keeps the ecosystem maturing and growing (especially while weโ€™re scurrying around shoring up the foundations) - thereโ€™s simply no point in Matrix as a protocol without the vibrant community building on top.

All told, itโ€™s been a bit of an epic year (both in terms of wins and fails), and all that remains is to thank everyone who continues to use Matrix (particularly our Patreon supporters) for their ongoing support and for helping the project accelerate forwards. More than ever before, the world needs free and open communication open to all; the age of proprietary communication silos may be coming to an end - consigned to live alongside AOL CDs and Compuserve IDs in the history books. With your support, Matrix can provide a decent mainstream yet decentralised alternative - and weโ€™ll do everything we can to make that happen in 2020.

Happy holidays!

Matthew, Amandine & the whole Matrix.org team.

This Week in Matrix 2019-12-20

21.12.2019 00:20 โ€” This Week in Matrix โ€” Ben Parsons
Last update: 20.12.2019 18:29

๐Ÿ”—Dept of Status of Matrix ๐ŸŒก

๐Ÿ”—Matrix selected for the public Mozilla community

You may well have read about it by now, but Mozilla (purveyor of popular web browsers and champion of the open web) selected Matrix to replace IRC for their comms! You can read their own announcement here. Please note that this doesn't have to mean the death of Moznet on IRC - if someone wants to pick up matrix-ircd and finish it off, we can keep exposing an IRC listener too! Huge thanks to everyone who participated in the Mozilla trial and placed their trust in Matrix :)

๐Ÿ”—A Glimpse of the Matrix

Florian reported:

Florian presented his poster A Glimpse of the Matrix:Scalability issues of a new message-oriented data synchronizationmiddleware at the 20th International Middleware Conference at UC Davis, California on 2019-12-11. The poster abstract describes measurements of the public Matrix federation and discusses scalability issues of the current message routing mechanism. Additional details can be found in the Extended Tech Report.

Those scientific publications were based on the data gathered by the DSN Traveller in 2018 which was part of Florian's master's thesis. The anonymized raw data was published in conjunction.

All related resources

pic.twitter.com/NYxbYllQ9F

— Middleware2019 (@middleware2019) December 12, 2019

๐Ÿ”—Accessibility in Riot/Matrix

Very thorough article on accessibility in Riot/Matrix, written partly in light of the Mozilla announcement. https://marcozehe.de/2019/12/20/how-to-get-around-matrix-and-riot-with-a-screen-reader/

๐Ÿ”—Dept of Servers ๐Ÿข

๐Ÿ”—Synapse v1.7.2

Neil told us:

We shipped 1.7.2 (and 1.7.1) - all admins are encouraged to upgrade asap, note 1.7.1 is a security release, and 1.7.2 fixes a back pagination bug introduced in 1.7.1. Aside from that we are looking at implementing MSC2260: Update the auth rules for m.room.aliases events and adding a per media quarantine API.

๐Ÿ”—Deploying Synapse

Several packaging projects have been updated to deploy the new version:

๐Ÿ”—Ruma

jplatte reported:

another blog post has appeared on the ruma website: https://ruma.dev/news/these-weeks-in-ruma-2019-12-14/

๐Ÿ”—cortex workers performance

Black Hat has been using his Rust cortex Synapse workers project. He reported:

I flexed on other homeservers by making getflexedon.me the fastest homeserver in the ping room, made possible with cortex.

Black Hat does point out that this is still in a testing phase, but it's great to see workers being created.

๐Ÿ”—Dept of Bridges ๐ŸŒ‰

๐Ÿ”—famedly-email-bridge

sorunome said:

Some more work has been done on famedly-email-bridge! Now you can define email routes (e.g. [email protected] -> @bob:example.org) and optionally have conversations create a new thread room, instead of dumping them into the email room.

๐Ÿ”—zammad tickets bot

It might seem like Half-Shot hasn't made a new bridge in a while, but here he is:

I've started another bot project: https://github.com/half-shot/matrix-zammad. This currently splurts zammad tickets into Matrix rooms, and will eventually do a lot more.

๐Ÿ”—Dept of Clients ๐Ÿ“ฑ

๐Ÿ”—Continuum

yuforia said:

Continuum, client for the desktop:

  • Start using experimental asynchronous Flow as observable value for UI. Making use of Kotlin's coroutine features, it makes it possible to update values while avoiding switching to the main UI thread. It's also easier to cancel on-going HTTP requests when their values are no longer needed.

๐Ÿ”—Riot-iOS

Manu told us:

This week, we have been still working hard on verification by DM. We have started the implementation of cross-signing.

๐Ÿ”—RiotX v0.11.0 released

benoit said:

RiotX: We've released RiotX v0.11.0 on Thursday. It includes support to open (some of) matrix.to links, soft (and hard) logout, and lots of small UI/UX/crash fixes. For the first release of 2020, we will change the way we handle the initial sync, which can be a long task, by running it in a foreground service. Also the room profile screen should finally arrive.

๐Ÿ”—riot-web

Bruno reported:

this week I've been working on the new verification flow in the right panel. it's nearly there, but likely won't get merged today anymore.

๐Ÿ”—Dept of Ops ๐Ÿ› 

๐Ÿ”—Matrix Message github action

Nice and simple project for using Matrix messages in Github actions. See the code, or the marketplace page.

๐Ÿ”—Dept of Ping ๐Ÿ“

RankHostnameMedian MS
1getflexedon.me312
2thinker.eu.org346
3tedomum.net384
4aime.lesmatric.es440
5dodsorf.as463
6bubu1.eu534.5
7lyseo.edu.ouka.fi558.5
8maunium.net563
9matrix.vgorcum.com654
10testmatrix.vgorcum.com751

๐Ÿ”—Final Thoughts ๐Ÿ’ญ

It being the time of year that it is, some of us will be at 36c3 in a week or so, come chat in #chaosevents:matrix.org if you'd like to say "hi". (You can also say "Guten Tag", which is more fun!)

๐Ÿ”—That's all I know ๐Ÿ

See you next week, and be sure to stop by #twim:matrix.org with your updates!