Matrix Live 🎙 Demos Week

Demos week is fun! Reminds me of walk-around-the-office-interrupting-people week, which we don't have any longer. :(

Dept of Status of Matrix 🌡

Open Tech Will Save Us #8 will take place next Wednesday, join us! Calendar event coming soon.

• Muhammad Radwan, on behalf of Tactical Tech, will present their work on The Glass Room Misinformation Edition. What happens when we increasingly rely on social media and the web for nearly all our information? What information do we see, and what do we miss? How do we know if a picture or a tweet is genuine or truthful? And what can we do if we can’t be sure?
• Popi from CHATONS will present the work done by CHATONS ("Collective of Alternative, Transparent, Open, Neutral and United Hosters") CHATONS has been initiated by Framasoft, in the wake of the success of its campaign "De-google-ify Internet". A CHATONS member pledges to propose fully FLOSS-based online services, in an ethical and united spirit.
• Hubert|Uhoreg Chathi will present on the state of encryption in Matrix, and more broadly Why the PGP model is insufficient, what properties we want from encryption (e.g. why we want perfect forward secrecy), how the newer protocols (OTR, double-ratchet, megolm) give us some of the properties we want, and then what's coming in the future (MLS, and what that gives us)

Dept of Spec 📜

anoa said:

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals

MSC Status

Merged MSCs:

• No MSCs were merged this week.

MSCs in Final Comment Period:

• MSC2765: Widget avatars (merge)

New MSCs:

• MSC2848: Globally unique event IDs

• MSC2846: Decentralizing media through CIDs

• MSC2845: Phone number lookups using third party API

• [WIP] MSC2844: Global version number for the whole spec

Spec Core Team

In terms of Spec Core Team MSC focus for this week, we're switching out MSC2765 (widget avatars) for MSC1544 (QR code verification), and keeping

MSC2774 (widget URL template param), and MSC2790 (modal widgets).

Dept of Servers 🏢

Synapse

callahad offered:

Happy November from the Synapse team! As mentioned last week, we pushed a small v1.22.1 release last Friday which fixed two regressions:

• Fix a bug where an appservice may not be forwarded events for a room it was recently invited to. Broke in v1.22.0. (#8676)

• Fix Object of type frozendict is not JSON serializable exceptions when using third-party event rules. Broke in v1.22.0. (#8678)

If you haven't upgraded your Synapse in a while, please do.

A major focus of Synapse is being able to meet the performance and reliability needs of massive homeservers like matrix.org. If you're curious about how Synapse's architecture has evolved over the years to meet these scaling challenges, check our our blog post from Tuesday: How we fixed Synapse's scalability!

Lastly, we anticipate releasing 1.23.0 in the next fortnight; keep your eyes peeled for release candidates and let us know if you have any feedback. For a preview of what's coming, check out GitHub for the new commits that have landed on the develop branch since our last release.

PLUS Matthew said:

Synapse now horizontally scales across multiple python processes, as of 1.22: you can configure it so that events are no longer sent through the main proc, eliminating the single biggest bottleneck for large scale Synapse deployments. Read all about it at https://matrix.org/blog/2020/11/03/how-we-fixed-synapses-scalability

Dendrite / gomatrixserverlib

Dendrite is a next-generation homeserver written in Go

Neil Alexander told us:

Things have been quiet for Dendrite over the last week as I have been working on Pinecone/P2P and Kegan has been working on threading.

That said, a couple of minor changes have been merged:

• Forgetting rooms is now supported (thanks S7evinK!)

• The gjson dependency has been updated for correct integer safe ranges

Spec compliance is the same as last week:

• Client-server APIs: 57%

• Server-server APIs: 81%

As always, feel free to join us in #dendrite:matrix.org for general Dendrite chat or #dendrite-dev:matrix.org for development discussion.

Synapse Deployment 📥️

YunoHost

Pierre announced:

YunoHost is an operating system aiming for the simplest administration of a server, and therefore democratize self-hosting.

Synapse integration had been updated to 1.21.2 (1.22.1 available in branch testing)

Element Web integration had been updated to 1.7.9 (1.7.12 available in branch testing)

Dept of Bridges 🌉

Gitter

Eric Eastwood told us:

Exciting visual progress this week with actual bridging between Gitter and Matrix utilizing the virtualUser feature,

we've been iterating on the past couple weeks. Check out the image with all of the user avatars and display name goodness to make both chats on Element and Gitter feel one in the same!

You can also check out the live demo in Matrix Live!

Dept of Clients 📱

Hydrogen

Bruno reported:

As mentioned in the sync on Monday, I was mostly distracted from Hydrogen this week. (sorry Bruno -BP) I did release the picture lightbox on Monday, and yesterday managed to close 4 bugs. There's also a community PR for better usability and accessibility in the login screen (keep those coming!) and after some work to make encryption more robust, I hope to do a release with all those goodies tonight.

Check out the demos vid for more Hydrogen!

Fractal

Alexandre Franke said:

Since a couple of weeks ago, we have merged a couple of branches that do a couple of things:

• Clean up use of deprecated gtk-rs API

• Minimize use of child properties, which are gone in GTK4

• Remove use of widgets what are gone in GTK4, such as GtkAlignment

• Ensure we unlock the default secrets collection for login

And here’s another nudge, calling for reviewers for that mega merge request for us to switch to matrix-rust-sdk.

Element for Android 1.0.10

benoit said:

Element for Android 1.0.10 has been released to the beta channel of the PlayStore. We will push it to prod if there is no major problem with it. Full release notes: https://github.com/vector-im/element-android/releases/tag/v1.0.10.

#1921 being fixed! ❤️

Element-iOS

Manu offered:

This week, we came back to the background sync work to quickly display a notified message in the app. In parallel, we created a profiling tool at the SDK level to track performance like this one.

Element Web/Desktop

Neil enunciated:

We are working through some low hanging fruit around post registration, blank screen interaction prompts and toast tweaks. We are also experimenting with SSO for matrix.org. Meaning that Element will give the user the option of either username/password or SSO. Finally, we are continuing our VoIP efforts and nailing down the designs, checkout Matrix Live for all the details. Next week we'll carry on with post registration UX, VoIP improvements.

Dept of SDKs and Frameworks 🧰

Matrix SDK for Elixir

cognitive_tea reported:

Hi all! I think this is the right place to share this 🤞. I've been working on a Matrix SDK for Elixir over the last few months as a side project, it's very early days and it's currently just a bare-bones wrapper for the Client-Server API. I've also written the Elixir/Erlang bindings for Olm (currently missing group sessions) which should be added to the SDK soon. The repo can be found here: https://github.com/niklaslong/matrix-elixir-sdk and the Elixir bindings for Olm are linked in the readme.

It is the right place! Thanks cognitive_tea :D

Asked if there were big plans for use of the project

Not as yet, though a few people have reached out to me already and are building on top of it. I started it as a way to get going with Matrix dev and as a fun side project. That being said, I think providing the tools to Matrix-enable Elixir apps might lead to some interesting things. If anyone has any precise ideas on how they would want to integrate their Elixir apps with Matrix, I'd be super happy to have a chat 👍️ Less precise ideas are also welcome, of course 🙂

Dept of Bots 🤖

Hemppa

Cos announced:

Hemppa the bot is a generic bot for writing modules as easily as possible in Python. Thanks to issues with Freenode IRC bridge Hemppa got a new module for basic relaybot bridging of any Matrix rooms. Relaybots are stupid, but sometimes there's no working alternative. https://github.com/vranki/hemppa#relay-bridge

Dept of Events and Talks 🗣️

Arch Conf 2020

Brendan Abolivier reported:

I did a talk at Arch Conf 2020 last month, on a generic introduction of Matrix and how to install a Matrix homeserver on Arch Linux. The recording has just been uploaded; it can be found on CCC's media site as well as YouTube 🙂

YES BRENDAN!

Matrix in the News 📰

Oleg's talk, the article

emorrp1 told us:

New Matrix coverage in LWN via an Open Source Summit Europe talk https://lwn.net/SubscriberLink/835880/bd73956d4ceb6cf5/

See last week for the talk!

Final Thoughts 💭

TeeCee reported:

I stumbled upon this: https://www.reddit.com/r/linux/comments/jozg0v/how_i_got_my_group_chat_to_move_to_matrix/

I love that the comments, even on Reddit, are mostly positive. A really nice report.

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

Rank	Hostname	Median MS
1	envs.net	556
2	privacytools.io	579.5
3	maescool.be	621
4	casavant.org	750
5	apetre.sc	750
6	matrix.thedisco.zone	1113
7	matrix.org	1159
8	zemos.net	1256.5
9	matrix.sp-codes.de	1263
10	halogen.city	2616

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Hi all,

We had a major break-through in Synapse 1.22 which we want to talk about in more detail: Synapse now scales horizontally across multiple python processes.

Horizontal scaling means that you can support more users and traffic by adding in more python processes (spread over more machines, if necessary) without there being a single bottleneck which all the traffic is passing through - as opposed to vertical scaling where you make things go faster overall by making the bottleneck go faster.

After many years of having to vertically scale Synapse (by trying to make the main process go faster) we’re now finally at the point where you can configure Synapse so that messages no longer flow through the main process - eliminating the bottleneck entirely. What’s more, the Matrix.org homeserver has now been successfully running in this config and enjoying the massive scalability improvements for the last 2 weeks! Huge kudos goes to Erik and the wider Synapse team for pulling this off.

Some readers might wonder how this ties in with Dendrite entering beta, given one of Dendrite’s design goals is full horizontal scalability. The answer is that we’re very much using Dendrite for experimentation and next-gen stuff at the moment (currently focused more on scaling downwards for P2P rather than scaling upwards for megaservers) - while Synapse is the stable and long-term supported option.

So, that’s the context - now over to Erik with more than you could possibly ever want to know about how we actually did it...

Background

Synapse started life off back in 2014 as a single monolithic python process, and for quite a while we made it scale by adding more and more in-memory caches to speed things up by avoiding hitting the database (at the expense of RAM). It looked like this:

Eventually the caches stopped helping and we needed more than one thread of execution in order to spread CPU across multiple cores. Python’s Global Interpreter Lock (GIL) means that Python can mainly only use one CPU core at a time, so starting more threads doesn’t help with scalability - you have to run multiple processes.

Now, the vast majority of the work that Synapse does is related to “streams”. These are append only sequences of rows, such as the events stream, typing stream, receipts stream, etc. When a new event arrives (for example) we write it to the events stream, and then notify anything waiting that there has been an update. The /sync endpoint, for instance, will wait for updates to streams and send them down to long-polling Matrix clients.

Streams support being added to concurrently, so have a concept of the “persisted up-to position”. This is the point where all rows before that point have finished persisting. Readers only read up to the current “persisted up-to position”, so that they don’t skip updates that haven’t finished persisting at that point. (E.g. if two events A and B get assigned positions 5 and 6, but B finishes persisting first, then the persisted up to position will remain at 4 until A finishes persisting and then it jumps to 6).

To split any meaningful amount of work into separate processes, we need to add a mechanism where processes can be told that updates to streams have happened (otherwise they’d have to repeatedly poll the DB, which would be deeply inefficient). The architecture ended up being one where we had the “main” process that streams updates via a custom replication protocol (initially long-polling HTTP; later custom TCP) to any number of “worker” processes. This meant that we could move sync stream handling (and other read apis) off the main process and onto workers, but also that all database writes had to go through the single main process (as it was a star topology, the main process could talk to all workers but workers could only talk to the main process and not each other).

As an aside: cache invalidations also had to be streamed down the replication connections, which has the side effect that we could only cache things that would only be invalidated on the main process.

We continued to move more and more read APIs out onto separate workers. We also added workers in front of the main process that would e.g. handle the creation of the new events, authenticating, etc, and then call out to the main process with the event for it to persist the event.

Moving writes off the main process

Eventually we ran out of stuff to move out of the main process that didn’t involve writing to the DB. To write stuff from other processes we needed a way for the workers to stream updates to each other. The easiest and most obvious way was to just use Redis and its pub/sub support.

This almost allowed us to move writing of a particular stream to a different worker, except writing to streams generally also meant invalidating caches which in itself requires writing to a stream. We needed a way of writing to the cache invalidation stream from multiple workers at once.

Sharding the cache invalidation thankfully turned out to be easy, as workers would simply call the cache invalidation function whenever they get an invalidation notice over replication. In particular, the ordering of invalidations from different workers doesn’t matter and so there isn’t a need to calculate a single “persisted up-to position”. Sharding then just becomes a case of adding the name of the worker that is writing the update to the replication stream, and then workers reading from it can basically treat the cache stream the same as if they were multiple streams, one per worker.

This then unlocks the ability to move writing of streams off the main process and onto different workers - and so we added the “event persister” worker for offloading the main event stream off the main process:

Sharding the events stream

Eventually the worker responsible for doing nothing but persisting events started maxing out CPU. This meant that we had to look at sharding the events stream, i.e. writing to it from multiple workers.

This is more complicated than sharding the cache invalidation stream as the ordering of the events does matter; we send them down sync streams, in order, with a token that indicates where the sync stream is up to in the events stream. This means that workers need to be able to calculate a “persisted up-to position” when getting updates from different workers.

The easiest way of doing that is to simply set the persisted up-to position as the minimum position received over federation from all active writers. This works, except events would only be processed after all other writers have subsequently written events (to advance the persisted position past the point at which the event was written), which can add a lot of latency depending on how often events are written.

A refinement is to note that if you have a persisted up-to position of 10, then receive updates at sequential positions 11, 12, 13 and 14, you know that everything between 10 and 14 has finished persisting (as you received updates about them), and so can set the persisted up-to position to 14. Annoyingly, it’s not required that positions are sequential without gaps (due to various technical considerations), and so in the worst case this still has the same problems as the naïve solution.

To avoid these problems we change the persisted up-to position to be a vector clock of positions; tracking a vector of positions - one per writer. This still allows answering the query of “get all events after token X” (as events are written with the position and the name of the writer). The persisted up-to position is then calculated by just tracking the last position seen to arrive over replication from each writer.

This allows writing events from multiple workers, while ensuring that other workers can correctly keep track of a “persisted up-to position”. Then it's just a matter of inspecting the code to ensure that it does not assume that it is the only writer to the stream. In the case of writing to the events stream, we note that the function persisting events assumes it's the only writer for a given room, so when sharding we have to ensure that there are no concurrent writes to the same room. This is most easily done by sharding based on room ID, and ensuring that the mapping of room ID to worker does not change (without coordination).

The only thing left is to then encode the vector clock position into the sync tokens. We want to ensure that these tokens are not too long, as they get included as query string parameters (e.g. the since= parameter of /sync). By assigning persistent unique integer IDs to workers the vector clock can be persisted as a sequence of pairs of integers, which is relatively few bytes so long as we don’t have too many workers writing to the events stream. We can further reduce the size of the tokens by calculating an integer “persisted up-to position” as we did before, encoding that and only including positions for workers that are larger than the integer persisted upto position. (The idea here is that most of the time only a small number of workers will be ahead of the calculated persisted up-to position, and so we only need to encode those).

And this is what we have today:

The major limitation of the current situation is that you can’t dynamically add/remove workers which persist events, as the sharding by room ID is calculated at startup, and so changing it requires restarting the whole system. This could be replaced by any system that allowed coordination over which persister is allowed to write to a room at any given point. However this is likely tricky to get right in practice, but would allow dynamic auto scaling of deployments, or automatically recovering from a worker that gets wedged/dies.

Finally, it’s worth noting that sharding event persisters isn’t the only performance work that’s been going on - switching everything over to python 3 and async twisted has helped, along with lots of smaller optimisations on the hot paths, and further rebalancing workers (e.g. moving background jobs off the master process to dedicated workers). We’ve also benefited a lot from the maintainability of rolling out mypy typing throughout the codebase. And next up, we’ll be going back to speeding up the codebase as a whole - starting with algorithmic state resolution improvements! 🎉

Performance

So, how does it stack up?

Here’s the send time heatmap on Matrix.org showing the step change on Oct 16th when we rolled out the second event persister (full disclosure: this also coincides with moving background processes off the main Synapse process to a background worker). As you can see, we go from messages being spread over a huge range of durations (up to several seconds) to the sweet spot being 50ms or less - a spectacular improvement!

Meanwhile, here’s the actual CPU utilisation as we split the traffic from a single event persister (yellow) to two persisters (one yellow, one blue), showing the sharding beautifully horizontally balancing CPU between the two active/active worker processes:

We’ve yet to loadtest to see just how fast we can go now (before we start hitting bottlenecks on the postgres cluster), but it sure feels good to have all our CPU headroom back on Matrix.org again, ready for the next wave of users to arrive.

Conclusion

So there you have it: folks running massive homeservers (50K+ concurrent users) like Matrix.org (and cough various high profile public sector deployments) are no longer held hostage by the bottleneck of the main synapse process and should feel free to experiment with setting up event persister workers to handle high traffic loads. Otherwise, if you can spread your users over smaller servers, that’s also a good bet (assuming they don’t have massively overlapping room membership, like we see on Matrix.org.)

The current worker documentation is up-to-date, although does assume you are already very familiar with how to administer Synapse. It’s also very much subject to change, as we keep adding new workers and improving the architecture. However, now is a pretty good time to get involved if you’re interested in large-scale Matrix deployments.

-- The Synapse Team

Matrix Live 🎙

sometimes you'll come across us at FOSDEM and we'll say "oh it's the future", and we're trying to make this an actual thing

- Half-Shot on getting from sci-fi to reality

Dept of Spec 📜

Spec

anoa told us:

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.

MSC Status

Closed MSCs:

• [WIPish] MSC1777: peeking over federation (via server pseudousers)
  • Obsoleted in favour of MSC2444

Merged MSCs:

• MSC2758: Proposal for a common identifier grammar

MSCs in Final Comment Period:

• No MSCs are in FCP.

New MSCs:

• MSC2839: Dynamic User-Interactive Authentication

• MSC2836: Threading

• MSC2835: Add UIA to the /login endpoint

• MSC2834: Media IDs as hashes

• MSC2832: HS -> AS authorization header

Also heads up, the nomenclature for Communities v2 (groups-as-rooms) is now Matrix Spaces! Check out MSC1772 for the details!

Spec Core Team

In terms of Spec Core Team MSC focus for this week, we're continuing with the widget theme: MSC2774 (widget URL template param), MSC2765 (widget avatars), and MSC2790 (modal widgets).

New spec platform

wbamberg told us:

Updates on the new spec platform: we can render HTTP APIs (https://adoring-einstein-5ea514.netlify.app/spec/client-server/#login) and events (https://adoring-einstein-5ea514.netlify.app/spec/client-server/#room-events).

Dept of Built on Matrix 🏗️

Chupacabra Social

patrick told us:

From the creators of Noteworthy, introducing Chupacabra, a Matrix powered content sharing and discussion layer.

Video demo: https://youtu.be/hAouGTL7XAQ

Github: https://github.com/decentralabs/chupacabra

Join us in #chupacabra:chupa.social to learn more.

Dept of Servers 🏢

Conduit

Conduit is a Matrix homeserver written in Rust https://conduit.rs

Timo announced:

Hello everyone, I have some amazing news to share with you! While Conduit is getting better at federating, Famedly (https://famedly.com) has agreed to support me working on Conduit financially. With this news come some organizational changes:

Conduit development now happens at https://gitlab.com/famedly/conduit, please submit new issues and pull requests over there. I will update all links in the coming days.

Note: Famedly does not own the project and Conduit will stay free and open source forever!

matrix-media-repo

matrix-media-repo is a highly customizable multi-domain media repository for Matrix

TravisR announced:

v1.2.1 of matrix-media-repo, a third-party media repo for large homeservers, is out now. It's primarily a maintenance update though also has support for audio files if for some reason you need that.

Synapse

callahad said:

Synapse 1.22.0 is out! We announced its first release candidate last week, and after a small rc2 the final release was published last Tuesday. We anticipate a small 1.22.1 release later today with fixes for messages not always being sent to app services (#8673) and serialization errors with third-party event rules (#8678).

We continue to see improved client join Apdex scores for matrix.org, indicating that our work in 1.22.0 to split background tasks into separate workers and allow for sharded event persisters successfully improved the user-visible performance of large homeservers.

In other news, we pushed a temporary hotfix to the matrix.org homeserver earlier this week, instructing it to drop all cross-user m.key_share_request messages. This was necessary to mitigate a bug in a third-party library which caused some clients to flood the server with requests. We'll re-enable these messages once we resolve issue #8677. In the meantime, we strongly encourage FluffyChat users to upgrade to version 0.21.1.

We're hard at work on the next release of Synapse, and the development branch already includes many bugfixes, several new admin APIs, and support for structured logging—stay tuned!

Dendrite

Dendrite is a next-generation homeserver written in Go

kegan said:

There is no release this week, be sure to have v0.2.1 installed for a more stable experience! A few documentation changes have been made this week:

• Docker sample configs are now correct.

• The MaxMessageBytes for Kafka messages is now configurable - thanks @S7evinK!

• A reverse-proxy sample now exists for Hiawatha - thanks @ErgoPoe!

Spec compliance remains unchanged:

• Client-server APIs: 57%

• Server-server APIs: 81%

Things have been quiet this week because Neil has been working on new P2P routing schemes and I have been working

on a Threading proposal which will be tried out in Dendrite in the coming days.

Synapse Deployment 📥️

Kubernetes

Ananace announced:

Just pushed the Synapse 1.22.0 versions for my K8s-optimized image and Helm chart.

... 🕛 time 🕗 went 🕟 by 🕥 ...

Updated my Synapse chart and K8s-optimized image to 1.22.1 as well, and got the element-web chart updated to 1.7.12

YunoHost

Pierre reported:

YunoHost is an operating system aiming for the simplest administration of a server, and therefore democratize self-hosting.

Synapse integration had been updated to 1.20.1 (1.21.2 available in branch testing)

Element Web integration had been updated to 1.7.9 (1.7.10 available in branch testing)

Dept of Bridges 🌉

🌈🌉 Bifrost reaches 0.2.0

Half-Shot reported:

Hey folks, today I bring you a gift wrapped rainbow coated present, which could only mean Bifrost 0.2.0 is out!.

We've been making major progress trying to align bifrost with the many XMPP clients out there like Gajim and Swift, by improving it's compatibility with the various XEPs. I've also noticed a few users have started using it to bridge their Matrix and XMPP communities together which is super cool :)

Please read the latest changelogs over at https://github.com/matrix-org/matrix-bifrost/releases/tag/0.2.0 and upgrade away!

Gitter

Eric reported:

The merge request for the native Gitter bridge has just got underway and we're making progress towards sharing all Gitter messages in public rooms across to Matrix.

We'll continue to iterate on the Gitter virtualUser support as we go along.

mautrix-telegram

Tulir said:

v0.9.0-rc1 was released last weekend. Changes since v0.8.x include:

• Prometheus metric names are now prefixed with bridge_

• Support for Telegram QR code login

• Support for double puppeting for users on other servers

• Options for automatic backfilling of missed messages and old messages when creating portals

• Switched end-to-bridge encryption to use mautrix-python instead of the previous hacky matrix-nio solution

This week I fixed some bugs, so I'll probably make a rc2 in the near future.

Dept of Clients 📱

Fluffychat 0.21.1 is released!

FluffyChat is a cute cross-platform matrix client. It is available for Android, iOS, Web and Desktop.

sorunome announced:

It is already in fdroid, google play and ios should follow shortly. We highly encourage people to update, as it contains an important bugfix of sending out way too many key requests, which can cause bad server performance

Features

• New user viewer

• Add code syntax highlighting in messages

• Updated translations: Thanks to all helpers

Changes

• Stories feature removed

Fixes

• Fixes sentry

• Fixes Android download

• Minor fixes

quotient

kitsune reported:

Hot on the heels of 0.0.9.5 beta, Quaternion 0.0.9.5 beta2 is released, fixing a couple of blunders, notably inability to build with external libQuotient. Keep testing, keep translating!

Hydrogen

Bruno announced:

Hydrogen can now show images in encrypted rooms! I hope to also release a lightbox feature this afternoon to show a zoomed version of an image.

Element-iOS

Manu announced:

This week, we have almost finished the authentication for widgets and jisti in particular. The project is now fully compatible with Xcode 12.

Element Android

benoit offered:

We are making progress on the performance side. Now sending an event is much faster than before. We also are optimizing the crypto code. All those improvements will be available in the next release (v1.0.10), maybe next week?

Besides that, we are implementing the remaining features, we are trying to have the same level of functionality (= parity) than Element Web. We know that we have a great number of bugs to fix on the existing feature, we are also trying to fight them.

As a reminder, the new Android Matrix SDK is available at https://github.com/matrix-org/matrix-android-sdk2 and a nice sample application has been developed and is available at https://github.com/matrix-org/matrix-android-sdk2-sample.

You had me at "progress on the performance side"! I am looking forward to the new Element Android :D

Element Web

Neil reported:

This week we shipped Element 1.7.12 which contains some high priority fixes, specifically:

• Fixes secret storage / cross-signing reset to avoid asking for the previous key you no longer have
• Fixes widget pinning and Jitsi calls when custom themes are used

Aside from that we continue to work on the voice and video calling experience as well as improving the initial onboarding experience of the app.

Nheko

Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE (with the notable exception being device verification for now) and intends to be full featured and nice to look at

Nico (@deepbluev7:neko.dev) offered:

This was an exciting week again. Trilene did the usual and just opened a PR, that implements the video part of call support. In my testing so far this seems to work amazingly well (ignore, that my webcam is crappy in the video, I only have so many devices...)! It's hard to overstate my satisfaction, if I am allowed to quote songs without getting a DMCA! If you want to try it out, you will need the qmlgl plugin at runtime (I had to patch some ebuilds on Gentoo to do so) and build Nheko from source. Support in our AppImage and Flatpak and for our Windows and MacOS builds will come at a later date. A big shoutout to trilene, who just works on VOIP in Nheko, without saying a word, and then drops a ready PR.

Another exciting new feature by lorendb, you can now specify --profile <profilename>, when you start Nheko, to create a separate profile. This allows you to open multiple instances and use multiple accounts at the same time (but it still uses separate instances of Nheko). This is pretty useful, if you have multiple accounts on different homeservers or are testing stuff for example. He also added a shortcut to delete the current content of the message area (Ctrl-U).

We also fixed a long standing bug, that crashed Nheko when pasting an image on mac OS, prevented copying text in some cases and build times should be about halfed again.

That's all I got today. I guess we should do a new release at some point?

I asked about trilene, who is a reliable Nheko contributor, Nico replied:

trilene seems to be a bit camera shy and prefer to work on code than take credit and talk about upcoming features. I'm surprised everytime, when a new PR is opened or trilene asks a weird question, that can only end up in an amazing contribution :3

\o/

Dept of SDKs and Frameworks 🧰

matrix-bot-sdk 0.5.8 out now

TravisR told us:

matrix-bot-sdk v0.5.8 is out now with experimental support for EDUs being sent to appservices (per MSC2409).

To enable it you'll need Synapse 1.22.0 (released this week) and v0.5.8 of the bot-sdk. Then, add "de.sorunome.msc2409.ephemeral": true to your appservice registration file (at the root level) and turn on the de.sorunome.msc2409.ephemeral flag in your IAppserviceRegistration supplied to the bot-sdk. If all goes according to plan, you'll be able to use appservice.on("ephemeral.event", (ev) => {}) to start processing EDUs.

Dept of Ops 🛠

Icinga End-to-End Check

Nik said:

I hacked together a maubot-based roundtrip test that leverages the echo bot's ping command reply and reports rtt to Icinga as a passive check result. Its practical use is scientifically questionnable, but it gives a hint on end-user experience. Find it here: https://edugit.org/nik/maubot-pingcheck

Dept of Services 🚀

Enabling encryption for bots on t2bot.io

TravisR offered:

Starting November 28th and 29th of this year, many bots on t2bot.io will be supporting end-to-end encryption. Though not all bots will be supporting it, this is an important milestone towards getting end-to-bridge encryption enabled on t2bot.io as a proof of being able to scale to the higher demand of encrypted rooms.

The eventual goal is to support encryption on all of t2bot.io’s bots and bridges, however we need to take small steps to get there 🙂. Note that in order to function, bots will decrypt all messages they see, but only respond to the ones they care about - this can still be uncomfortable for some rooms though, so feel free to kick them out.

For more detail on which bots are getting support and what all this entails/means, please see the dedicated blog post.

We teased this a little on Matrix Live last week (I think?), but so awesome to see this publicly announced.

Keymaker (Serverlist Project)

MTRNord reported:

Keymaker is a new WIP Project of some people (over at #serverlist:nordgedanken.dev ) that aim to provide a mastodon alike Server List and we would love to get some more input from the Community for this project on whats wanted, whats needed and whats maybe not that good to base on the mastodon counterpart.

This means we are building:

• A list of Servers where Owners can add their servers

• We try to do Quality controls (No fully self add. Servers get reviewed.) using a Code of Conduct Ruleset

• Verified Listings using well-known files on serverside (Also allowing Admins to modify server data themself)

• Server Details like:

  • Ping and Availability Stats (thanks to tulir for providing a API)

  • Public Room Lists fetched from the Server

  • NSFW Ratings (A NSFW tag was too generic for us)

  • A Section to list Rules

  • Admin addresses for easy ways of reaching the Server admins

• Allowing to select registration state ("Open", "Invite Only", "Closed")

The Code is fully written in Rust and using Postgres as a backend. Have a look at: https://github.com/daydream-mx/keymaker

Join us at: #serverlist:nordgedanken.dev

In a further post we plan to announce the launch of this Project as a Website. Server owners might get a ping before that to allow for setting up servers for this. This Project is not yet deployed to be used.

This is really cool. I suggested it might start to kickstart people hosting their own small, publicly open servers.

MTRNord replied:

As we also allow non public servers (registration -> closed) it may also be a nice way to find communities that federate and have a look if they have a interesting room to join in the public rooms list. :)

Dept of Bots 🤖

Bubo

kapina-jaywink reported:

Bubo, the community helping bot-in-progress, gets releases and a new command: breakout. The command can be used to create a breakout room from the current room. Bubo will create the room, invite and make the requester an admin, and confirm in the original room. Anyone who reacts with an emoji to the confirmation will get an invite to the breakout room. Currently breakout rooms are non-public and non-encrypted by default.

Find Bubo v0.2.0 here.

Dept of Events and Talks 🗣️

Neil on being an eng manager

Neil announced:

Hey all, I do engineering managery stuff at Element, if you ever wondered what on earth that actually means, here is a video of me going on and on about it. https://www.youtube.com/watch?v=2NflccKdGqU

Sell it Neil! This is an insightful chat - if you're interested in the dilemmas and thoughts of an eng manager, be sure to check this out!

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

Rank	Hostname	Median MS
1	maescool.be	653
2	aragon.space	707
3	envs.net	800
4	matrix.vgorcum.com	828
5	elcyb.org	1044
6	neko.dev	1104
7	fab.network	1112
8	mailstation.de	1128
9	aragon.sh	1364
10	dodsorf.as	1605

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Synapse 1.22.0 now available!

This release focused on improving Synapse's horizontal scalability, including:

• Support for running background tasks in separate worker processes.
• Fixes to sharded event persisters, which were experimentally introduced in 1.21.0.
• Fixing a message duplication bug with worker-based deployments. (#8476)

Synapse 1.22.0 also has a few other notable changes:

• Defaulting to version 6 rooms, per MSC2788.
• Initial support for three new experimental MSCs:
  • MSC2732: Supporting olm fallback keys
  • MSC2697: Supporting device dehydration
  • MSC2409: Allowing appservices to receive ephemeral events like read receipts, presence, and typing indicators.
• Multi-arch Docker images, covering arm64 and arm/v7 in addition to amd64.

Installation instructions are available on GitHub, as is the v1.22.0 release tag.

Lastly, Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including @Akkowicz, @BBBSnowball, @maquis196, and @samuel-p.

The full changelog for 1.22.0 is as follows:

Synapse 1.22.0 (2020-10-27)

No significant changes.

Synapse 1.22.0rc2 (2020-10-26)

Bugfixes

• Fix bugs where ephemeral events were not sent to appservices. Broke in v1.22.0rc1. (#8648, #8656)
• Fix user_daily_visits table to not have duplicate rows per user/device due to multiple user agents. Broke in v1.22.0rc1. (#8654)

Synapse 1.22.0rc1 (2020-10-22)

Features

• Add a configuration option for always using the "userinfo endpoint" for OpenID Connect. This fixes support for some identity providers, e.g. GitLab. Contributed by Benjamin Koch. (#7658)
• Add ability for ThirdPartyEventRules modules to query and manipulate whether a room is in the public rooms directory. (#8292, #8467)
• Add support for olm fallback keys (MSC2732). (#8312, #8501)
• Add support for running background tasks in a separate worker process. (#8369, #8458, #8489, #8513, #8544, #8599)
• Add support for device dehydration (MSC2697). (#8380)
• Add support for MSC2409, which allows sending typing, read receipts, and presence events to appservices. (#8437, #8590)
• Change default room version to "6", per MSC2788. (#8461)
• Add the ability to send non-membership events into a room via the ModuleApi. (#8479)
• Increase default upload size limit from 10M to 50M. Contributed by @Akkowicz. (#8502)
• Add support for modifying event content in ThirdPartyRules modules. (#8535, #8564)

Bugfixes

• Fix a longstanding bug where invalid ignored users in account data could break clients. (#8454)
• Fix a bug where backfilling a room with an event that was missing the redacts field would break. (#8457)
• Don't attempt to respond to some requests if the client has already disconnected. (#8465)
• Fix message duplication if something goes wrong after persisting the event. (#8476)
• Fix incremental sync returning an incorrect prev_batch token in timeline section, which when used to paginate returned events that were included in the incremental sync. Broken since v0.16.0. (#8486)
• Expose the uk.half-shot.msc2778.login.application_service to clients from the login API. This feature was added in v1.21.0, but was not exposed as a potential login flow. (#8504)
• Fix error code for /profile/{userId}/displayname to be M_BAD_JSON. (#8517)
• Fix a bug introduced in v1.7.0 that could cause Synapse to insert values from non-state m.room.retention events into the room_retention database table. (#8527)
• Fix not sending events over federation when using sharded event writers. (#8536)
• Fix a long standing bug where email notifications for encrypted messages were blank. (#8545)
• Fix increase in the number of There was no active span... errors logged when using OpenTracing. (#8567)
• Fix a bug that prevented errors encountered during execution of the synapse_port_db from being correctly printed. (#8585)
• Fix appservice transactions to only include a maximum of 100 persistent and 100 ephemeral events. (#8606)

Updates to the Docker image

• Added multi-arch support (arm64,arm/v7) for the docker images. Contributed by @maquis196. (#7921)
• Add support for passing commandline args to the synapse process. Contributed by @samuel-p. (#8390)

Improved Documentation

• Update the directions for using the manhole with coroutines. (#8462)
• Improve readme by adding new shield.io badges. (#8493)
• Added note about docker in manhole.md regarding which ip address to bind to. Contributed by @Maquis196. (#8526)
• Document the new behaviour of the allowed_lifetime_min and allowed_lifetime_max settings in the room retention configuration. (#8529)

Deprecations and Removals

• Drop unused device_max_stream_id table. (#8589)

Internal Changes

• Check for unreachable code with mypy. (#8432)
• Add unit test for event persister sharding. (#8433)
• Allow events to be sent to clients sooner when using sharded event persisters. (#8439, #8488, #8496, #8499)
• Configure public_baseurl when using demo scripts. (#8443)
• Add SQL logging on queries that happen during startup. (#8448)
• Speed up unit tests when using PostgreSQL. (#8450)
• Remove redundant database loads of stream_ordering for events we already have. (#8452)
• Reduce inconsistencies between codepaths for membership and non-membership events. (#8463)
• Combine SpamCheckerApi with the more generic ModuleApi. (#8464)
• Additional testing for ThirdPartyEventRules. (#8468)
• Add -d option to ./scripts-dev/lint.sh to lint files that have changed since the last git commit. (#8472)
• Unblacklist some sytests. (#8474)
• Include the log level in the phone home stats. (#8477)
• Remove outdated sphinx documentation, scripts and configuration. (#8480)
• Clarify error message when plugin config parsers raise an error. (#8492)
• Remove the deprecated Handlers object. (#8494)
• Fix a threadsafety bug in unit tests. (#8497)
• Add user agent to user_daily_visits table. (#8503)
• Add type hints to various parts of the code base. (#8407, #8505, #8507, #8547, #8562, #8609)
• Remove unused code from the test framework. (#8514)
• Apply some internal fixes to the HomeServer class to make its code more idiomatic and statically-verifiable. (#8515)
• Factor out common code between RoomMemberHandler._locally_reject_invite and EventCreationHandler.create_event. (#8537)
• Improve database performance by executing more queries without starting transactions. (#8542)
• Rename Cache to DeferredCache, to better reflect its purpose. (#8548)
• Move metric registration code down into LruCache. (#8561, #8591)
• Replace DeferredCache with the lighter-weight LruCache where possible. (#8563)
• Add virtualenv-generated folders to .gitignore. (#8566)
• Add get_immediate method to DeferredCache. (#8568)
• Fix mypy not properly checking across the codebase, additionally, fix a typing assertion error in handlers/auth.py. (#8569)
• Fix synmark benchmark runner. (#8571)
• Modify DeferredCache.get() to return Deferreds instead of ObservableDeferreds. (#8572)
• Adjust a protocol-type definition to fit sqlite3 assertions. (#8577)
• Support macOS on the synmark benchmark runner. (#8578)
• Update mypy static type checker to 0.790. (#8583, #8600)
• Re-organize the structured logging code to separate the TCP transport handling from the JSON formatting. (#8587)
• Remove extraneous unittest logging decorators from unit tests. (#8592)
• Minor optimisations in caching code. (#8593, #8594)

Matrix Live 🎙

Dept of Spec 📜

TravisR offered:

Hello everyone, normally anoa would be doing this update but today you get me (TravisR) instead. Luckily anoa has left me a script to run, so here's hoping I haven't completely messed up this week's update 😅

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.

MSC Status

Merged MSCs:

• Nothing to report 😢

MSCs in Final Comment Period:

• MSC2758: Proposal for a common identifier grammar (merge)

New MSCs:

• MSC2828: Proposal to restrict allowed user IDs over federation

Closed MSCs:

• MSC2773: Room kinds
• MSC1722: Support for displaying math(s) in messages

Spec Core Team

In terms of Spec Core Team MSC focus for this week, we're still focusing on the same MSCs from previous weeks to get widgets over the line. This includes MSC2774 to have widgets aware of their widget ID, MSC2765 so widgets can be pretty in clients like Element, and MSC2790 to support a form of user input with widgets.

Normally there would be a graph here of our MSC progress, however my machine refuses to accept that line graphs are a real thing today. As a replacement, here's a snowfall accumulation graph for Banff as reported by Environment Canada.

Dept of Servers 🏢

mauliasproxy

Tulir told us:

I made a simple room alias proxy: https://mau.dev/tulir/mauliasproxy. It can be used to create room aliases on custom domains without having to host an actual Matrix homeserver there. The proxy basically just responds to federation room alias queries using data from another homeserver (that data is fetched with the C-S API).

Synapse

callahad reported: >

A new release candidate appears! Synapse 1.22.0rc1 was published yesterday and includes support for running background tasks in separate worker processes, as well as fixes to sharded event persisters which were first introduced in 1.21.0.

These changes significantly improved our client join Apdex score for matrix.org by making join performance both faster and less variable.

Synapse 1.22.0rc1 also includes support for a few experimental MSCs:

• MSC2732: Supporting olm fallback keys

• MSC2697: Supporting device dehydration

• MSC2409: Allowing appservices to receive ephemeral events like read receipts, presence, and typing indicators.

Lastly, the default room version is now version 6, per MSC2788.

In the coming weeks we'll focusing on improving Synapse's resilience in smaller to medium-sized deployments, primarily through improvements to state resolution. Stay tuned!

Thanks Dan!

Today's update comes courtesy of Dan Callahan (@callahad:matrix.org), who joined Element on Monday as an engineering manager supporting the Synapse team. Dan previously worked in Developer Relations at Mozilla, and he's excited to help Matrix realize its ambitious vision for private, secure, and standards-based communication for all.

Dendrite / gomatrixserverlib

Dendrite is a next-generation homeserver written in Go

Neil Alexander offered:

This week we released v0.2.0 on Tuesday, a reasonably big update containing various improvements over the initial beta release, and then followed it up with a bug-fix v0.2.1 release on Thursday.

Thanks to everyone who has been running Dendrite and reporting their findings, and also to contributors who have been submitting pull requests!

Changes this week include:

• Dendrite no longer builds separate binaries for each polylith component, but instead has one multi-personality binary

• Our Docker images have been simplified into two images: dendrite-monolith and dendrite-polylith

• Internal HTTP API calls are now made using H2C (HTTP/2) in polylith mode, which resolves some head-of-line issues with the connection pool

• Forward extremities have been refactored, which should fix some cases where room state can end up corrupted

• A couple of bugs when handling state rewrites have been fixed

• The sync API no longer sends old state events to clients as if they were new

• SQLite locking bugs around the latest events updater have been resolved

• Notification levels are now parsed correctly in power level events (thanks to Pestdoktor)

• Invalid UTF-8 is now correctly rejected when making federation requests (thanks to Pestdoktor)

Spec compliance for v0.2.1:

• Client-server APIs: 57%, up from 56% last week

• Server-server APIs: 81%, up from 80% last week

As always, please feel free to join us in #dendrite:matrix.org for general Dendrite chat, and #dendrite-dev:matrix.org if you are interested in contributing!

Conduit

Timo stepped in to tell us:

This was another productive week:

• Improved thumbnailing algorithm (higher quality, less stored data, correct)
• Allow unjoined users to read state of world readable rooms (this makes shields.io work with conduit)
• Docs for cross compiling conduit
• Fixed stuck / double-join over federation
• Fixed random timeline reload bug
• Welcome message in admin room
• More frequent flushing

Some WIP things:

• Provide Conduit binaries for most platforms to make setting up or updating a Conduit instance even easier
• More reliable sending over federation
• Bring all features of our Ruma fork upstream

Thanks to everyone who supports me on Liberapay or Bitcoin!

Homeserver Deployment 📥️

Dendrite docker images

Dendrite is a next-generation homeserver written in Go

TR_SLimey offered:

I built some unofficial Dendrite docker images for Linux/ARM64, for those trying to run Dendrite on a Raspberry Pi, RockPro64 or others. They can be found here: https://hub.docker.com/r/trslimey/dendrite-monolith & https://hub.docker.com/r/trslimey/dendrite-polylith.

balaa reported:

Cool, we run Synapse on Pi0, Pi2 and Pi4 -- works reasonably well on each of them, i'm excited to try Dendrite

Synapse running on a Pi zero..?

Dept of Bridges 🌉

Gitter

Eric Eastwood announced:

The initial iteration of virtualUsers is in shape to merge(check out the flair 🔥) and will probably deploy in a release next week. We've split the rest of the virtualUser work into follow-up issues we can iterate on. We're working on adding room ban and spam detection support for virtualUsers to stop any bad actors. Then want to start on the actual Application Service bridge (Gitter <-> Matrix).

If you're curious about more of the details, you can track the greater GitLab epic.

🌈🌉 Bifrost* 0.2.0 is (nearly) out

Half-Shot offered:

I couldn't really wait to talk about before we actually hit 0.2.0 so here is a sneak peek at what's happening. We've spent a ton of time working on ironing out the bugs and making the bridge more XMPP complaint. The major headlines are:

• Support Matrix -> XMPP edits

• Set XMPP user displayname in the room based on their nickname (thanks uhoreg for mucking in there)

• Improve performance of Matrix -> XMPP gateway messages and joining

• Improve support for multiple devices for XMPP users connected to the gateway

• maaaany bugfixes

You can read about (and run!) the latest release over at https://github.com/matrix-org/matrix-bifrost/releases/tag/0.2.0-rc1.

Incidentally, if you've not yet, then try joining some rooms such as #twim#[email protected] from XMPP and see it live!

*rainbum, as Mathijs prefers

Dept of Clients 📱

Hydrogen

Bruno said:

Hydrogen gained a settings panel this week with a better session backup UX and your end-to-end device information, which should make the manual verification easier. Messages with multiple lines are also rendered as such now, which makes a big difference in usability. The app also works offline again after session backup broke that. Apart from that, several smaller fixes also landed.

Also, image decryption is well on it's way with a prototype working. 🎉

Fractal

Alexandre Franke reported:

The massive MR to switch to matrix-rust-sdk is still being reviewed and help is still welcome. We have been working on other stuff as well. Actually, since our last news piece for the release of 4.4, quite a lot happened (around 60 commits) that we haven’t reported here yet:

• Users can now go to the room settings to toggle notifications for each room individually.

• Rounded corners around everything to match the latest upstream design tweaks (in Adwaita, the official GNOME theme).

• Many maintenance changes: several dependencies have been updated, cleanups in various places, tightened flatpak permissions for better sandboxing…

And that’s not all! Good progress has been made towards rendering formatted_body. Hopefully that should be merged soon.

Element Web

Ryan offered:

We released Element 1.7.10 on Tuesday with some high priority fixes:

• Several bugs fixed for both all widgets as well as a few specific to Jitsi call widgets

• Widgets are now working again in Safari 13.1 (regressed by 1.7.9)

Quite soon after that on Wednesday, Element Web 1.7.11-rc.1 made it's way to staging:

• Improved state management for voice / video calls

• Revamped pinned widget UI to support resizing and more flexibility

Fluffychat

sorunome told us:

Fluffychat 0.20.0 is out! It should be available in fdroid, google play and on iOS soon!

Features

• Added translations: Arabic

• Add ability to enable / disable emotes globally

• Add ability to manage emote packs with different state keys

• Add swipe to reply - Thanks @inexcode

• Initial support for compiling to desktop

• Initial snap metadata - Thanks @RAOF_47

• Add latex parsing as per MSC2191 - $tex$ for inline and $$ for blocks

Changes

• Re-scale images in a separate isolate to prevent the UI from freezing

• URLs without https:// now linkify

• Parse all URIs, not just URLs

• emails will linkify now

• Make sure login to dendrite is working properly

Fixes

• Fix amoled / theme settings not always saving properly

• Show device name in account information correctly

• Fix tapping on aliases / room pills not always working

• Link clicking in web not always working

• Return message input field to previous state after editing message - Thanks @inexcode

MTRNord added:

perfect for university start. Finally I can write the thesis in the University matrix ;P

Element Android

benoit offered:

Element Android v1.0.9 is now available on the PlayStore. For the next release (1.0.10), we will optimize the performance (again!). We already have made some progress when sending a message to a room. We are now working on the crypto module and also we will probably upgrade the Realm database library, which seems more stable now. Besides that, we are still implementing the remaining features with the objective of getting a good feature parity with the other Element Matrix clients.

Element-iOS

Manu reported:

This week, we have been working to upgrade libs and tools to be compatible with Xcode 12.

We are making good progress to revive a kind of background sync so that a message appears quicker in the timeline when you tap on its notification. Authentication for widgets is still in progress.

Nheko

Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE (with the notable exception being device verification for now) and intends to be full featured and nice to look at

Nico (@deepbluev7:neko.dev) said:

• Nheko now shows the filename of an image on hover. (Contributed by kamath_manu)

• Nheko now shows the fontname in the font selection rendered using that font. This makes it easier to know, how the font will look, once you select it. (Contributed by lorendb)

• Fixed a crash when closing Nheko. While this didn't really cause issues, since you were closing it anyway, it's just bad form to crash instead of exit properly.

• Removed the membercache. Nheko used to load all members on startup and store them in memory. This made startup slower. Removing it sped up the start by a nice chunk and freed about 30MB of RAM on my system. One step closer to using reasonable amounts of RAM!

• Fixed excessive clipping when rendering the timeline. This prevented all batching of text messages. Now it only clips the replied to message, which makes scrolling much smoother again!

Finally, some controversial change, which is currently in master but may be reverted at some point: Nheko now automatically forwards keys to verified devices, when they request the keys, without prompting you. While that can be toggled off in the settings, it currently defaults to on. This weakens backward secrecy of e2ee a bit, but it makes it possible to recover from e2ee issues much more easily. Currently I'd argue, that it is an acceptable tradeoff. It is very hard to verify room membership of users at any point in time but the current one and room membership is not verified end to end in any way, so you need to trust the server to provide you with the correct memberlist or you just send keys to verified users. While Nheko still sends keys to all members of a room, when creating the session, it only forwards your own keys to trusted users without prompting you. Currently I think this is an acceptable tradeoff, as opening a popup with "user x wants to have session y shared in room z" is unlikely to be understood by anyone properly either. I'd be glad to hear your opinions though!

That's it for this week, but next week will be interesting too. Lorendb has been hacking on profile support, allowing you to run multiple login sessions of Nheko in separate windows and some other UI features.

Dept of SDKs and Frameworks 🧰

Ruma

iinuwa reported:

In the past couple of weeks we implemented the last endpoint for the Federation

API. We are working on smoothing out some rough edges in the ruma-federation-api crate, like a few that @Timo addressed this past week, so it will be a little while

before it's completely finished.

We've also created a milestone to track implementation of Identity Service API,

the last Matrix API we have to complete.

Finally, we've created a new Matrix room focused on Ruma development,

#ruma-dev:matrix.org, focusing the original room #ruma:matrix.org on Ruma usage.

quotient

kitsune told us:

Quotient 0.6.2 has been released, with a couple of minor bugfixes. This release is used as a foundation of Quaternion 0.0.9.5 beta that's also getting out today - with support of (proper Matrix subset of) HTML, rich text user links (like pills, only lighter), initial Markdown support (if you build with fresh enough Qt), reactions (thanks to Karel Kosek @krkk), navigation to earlier events (thanks to Roland Pallai @rpallai) and quite a few other improvements. To make this release Quaternion had to gain its own basic HTML parser and Matrix-to-Qt-to-Matrix converter, which is likely to end up being a separate micro-library, in the hope that it will be useful for other Matrix projects building on Qt (even non-Quotient ones). A separate call to translators - quite a few strings got updated, so please head to Quaternion project at Lokalise and push the numbers at least to 80%!

Dept of Events and Talks 🗣️

Three talks this week!

Matrix talk at Open Source Summit EU (virtual)

Oleg announced:

If you are visiting the OSS EU next week - come to the Matrix talk. 😉

Or join us at #welcome:osseu2020.fiksel.info !

dette er på utenlandsk

dandellion reported:

In august I held an introduction to and demo of matrix talk during a conference hosted by my local makerspace.

This week the talk was uploaded! (Norwegian) https://www.youtube.com/watch?v=s9Xd0Wg_XqA

AstriCon Plan (9) on building Omnichannel contact centers with Matrix (and other tech)

Matthew said:

Jose Franco gave a great talk at AstriCon Plan (9) on building Omnichannel contact centers with Matrix, Asterisk, Kamailio and friends. You can see the video at https://youtu.be/7S6GZz8f91o?t=18558 and the talk details at https://astricon2020.sched.com/event/e0GA/blending-open-source-rtc-tools-to-build-an-omnichannel-contact-center

Dept of Interesting Projects 🛰️

Noteworthy (matrix powered distributed overlay networks via WireGuard)

balaa told us:

hey TWIM peeps! we’ve updated the README for our project Noteworthy (matrix powered distributed overlay networks via WireGuard) https://github.com/decentralabs/noteworthy - join us over in #noteworthy:tincan.community with questions / comments!

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

Rank	Hostname	Median MS
1	maescool.be	528
2	chatcloud.net	533.5
3	matrix.thedisco.zone	654.5
4	pc.koesters.xyz:59003	779
5	shortestpath.dev	817.5
6	helderferreira.io	851.5
7	aragon.sh	898
8	pleasecuminside.me	1176
9	dodsorf.as	1253
10	envs.net	1304.5

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Hi all,

It's been over a week since our next-generation homeserver Dendrite entered beta, and it's been a wild rollercoaster ride as the team has been frantically zapping all the initial teething issues that came up - mostly around room federation getting 'stuck' due to needing to fix bugs in how room state is managed. Huge huge thanks to everyone who has spun up a Dendrite to experiment and report bugs!

We're now in an impressively better place, and it's feeling way more stable now (but please don't trust it with your data yet). So we've skipped 0.1.x and jumped straight to 0.2.0.

Now would be a great time for more intrepid explorers to try spinning up a server from https://github.com/matrix-org/dendrite and see how it feels - the more feedback the better. And if you got scared off by weird bugs in 0.1.0, now's the right time to try it again!

Full changelog follows:

Dendrite 0.2.0 (2020-10-20)

Important

• This release makes breaking changes for polylith deployments, since they now use the multi-personality binary rather than separate binary files
  • Users of polylith deployments should revise their setups to use the new binary - see the Features section below
• This release also makes breaking changes for Docker deployments, as are now publishing images to Docker Hub in separate repositories for monolith and polylith
  • New repositories are as follows: matrixdotorg/dendrite-monolith and matrixdotorg/dendrite-polylith
  • The new latest tag will be updated with the latest release, and new versioned tags, e.g. v0.2.0, will preserve specific release versions
  • Sample Compose configs have been updated - if you are running a Docker deployment, please review the changes
  • Images for the client API proxy and federation API proxy are no longer provided as they are unsupported - please use nginx (or another reverse proxy) instead

Features

• Dendrite polylith deployments now use a special multi-personality binary, rather than separate binaries
  • This is cleaner, builds faster and simplifies deployment
  • The first command line argument states the component to run, e.g. ./dendrite-polylith-multi roomserver
• Database migrations are now run at startup
• Invalid UTF-8 in requests is now rejected (contributed by Pestdoktor)
• Fully read markers are now implemented in the client API (contributed by Lesterpig)
• Missing auth events are now retrieved from other servers in the room, rather than just the event origin
• m.room.create events are now validated properly when processing a /send_join response
• The roomserver now implements KindOld for handling historic events without them becoming forward extremity candidates, i.e. for backfilled or missing events

Fixes

• State resolution v2 performance has been improved dramatically when dealing with large state sets
• The roomserver no longer processes outlier events if they are already known
• A SQLite locking issue in the previous events updater has been fixed
• The client API /state endpoint now correctly returns state after the leave event, if the user has left the room
• The client API /createRoom endpoint now sends cumulative state to the roomserver for the initial room events
• The federation API /send endpoint now correctly requests the entire room state from the roomserver when needed
• Some internal HTTP API paths have been fixed in the user API (contributed by S7evinK)
• A race condition in the rate limiting code resulting in concurrent map writes has been fixed
• Each component now correctly starts a consumer/producer connection in monolith mode (when using Kafka)
• State resolution is no longer run for single trusted state snapshots that have been verified before
• A crash when rolling back the transaction in the latest events updater has been fixed
• Typing events are now ignored when the sender domain does not match the origin server
• Duplicate redaction entries no longer result in database errors
• Recursion has been removed from the code path for retrieving missing events
• QueryMissingAuthPrevEvents now returns events that have no associated state as if they are missing
• Signing key fetchers no longer ignore keys for the local domain, if retrieving a key that is not known in the local config
• Federation timeouts have been adjusted so we don't give up on remote requests so quickly
• create-account no longer relies on the device database (contributed by ThatNerdyPikachu)

Known issues

• Old events can incorrectly appear in /sync as if they are new when retrieving missing events from federated servers, causing them to appear at the bottom of the timeline in clients
• Memory can explode when catching up after a federation outage.

UPDATE: Nov 9th 2020

Not only are UK/US/AU/NZ/CA/IN/JP considering mandating backdoors, but it turns out that the Council of the European Union is working on it too, having created an advanced Draft Council Resolution on Encryption as of Nov 6th, which could be approved by the Council as early as Nov 25th if it passes approval. This doesn't directly translate into EU legislation, but would set the direction for subsequent EU policy.

Even though the Draft Council Resolution does not explicitly call for backdoors, the language used...

Competent authorities must be able to access data in a lawful and targeted manner

...makes it quite clear that they are seeking the ability to break encryption on demand: i.e. a backdoor.

Please help us spread the word that backdoors are fundamentally flawed - read on for the rationale, and an alternative approach to combatting online abuse.

---

Hi all,

Last Sunday (Oct 11th 2020), the UK Government published an international statement on end-to-end encryption and public safety, co-signed by representatives from the US, Australia, New Zealand, Canada, India and Japan. The statement is well written and well worth a read in full, but the central point is this:

We call on technology companies to [...] enable law enforcement access to content in a readable and usable format where an authorisation is lawfully issued, is necessary and proportionate, and is subject to strong safeguards and oversight.

In other words, this is an explicit request from seven of the biggest governments in the world to mandate a backdoor in end-to-end encrypted (E2EE) communication services: a backdoor to which the authorities have a secret key, letting them view communication on demand. This is big news, and is of direct relevance to Matrix as an end-to-end encrypted communication protocol whose core team is currently centred in the UK.

Now, we sympathise with the authorities’ predicament here: we utterly abhor child abuse, terrorism, fascism and similar - and we did not build Matrix to enable it. However, trying to mitigate abuse with backdoors is, unfortunately, fundamentally flawed.

• Backdoors necessarily introduce a fatal weak point into encryption for everyone, which then becomes the ultimate high value target for attackers. Anyone who can determine the secret needed to break the encryption will gain full access, and you can be absolutely sure the backdoor key will leak - whether that’s via intrusion, social engineering, brute-force attacks, or accident. And even if you unilaterally trust your current government to be responsible with the keys to the backdoor, is it wise to unilaterally trust their successors? Computer security is only ever a matter of degree, and the only safe way to keep a secret like this safe is for it not to exist in the first place.

• End-to-end encryption is nowadays a completely ubiquitous technology; an attempt to legislate against it is like trying to turn back the tide or declare a branch of mathematics illegal. Even if Matrix did compromise its encryption, users could easily use any number of other approaches to additionally secure their conversations - from PGP, to OTR, to using one-time pads, to sharing content in password-protected ZIP files. Or they could just switch to a E2EE chat system operating from a jurisdiction without backdoors.

• Governments protect their own data using end-to-end encryption, precisely because they do not want other governments being able to snoop on them. So not only is it hypocritical for governments to argue for backdoors,** it immediately puts their own governmental data at risk of being compromised**. Moreover, creating infrastructure for backdoors sets an incredibly bad precedent to the rest of the world - where less salubrious governments will inevitably use the same technology to the massive detriment of their citizens’ human rights.

• Finally, in Matrix’s specific case: Matrix is an encrypted decentralised open network powered by open source software, where anyone can run a server. Even if the Matrix core team were obligated to add a backdoor, this would be visible to the wider world - and there would be no way to make the wider network adopt it. It would just damage the credibility of the core team, push encryption development to other countries, and the wider network would move on irrespectively.

In short, we need to keep E2EE as it is so that it benefits the 99.9% of people who are good actors. If we enforce backdoors and undermine it, then the bad 0.1% percent simply will switch to non-backdoored systems while the 99.9% are left vulnerable.

We’re not alone in thinking this either: the GDPR (the world-leading regulation towards data protection and privacy) explicitly calls out robust encryption as a necessary information security measure. In fact, the risk of US governmental backdoors explicitly caused the European Court of Justice to invalidate the Privacy Shield for EU->US data. The position of the seven governments here (alongside recent communications by the EU commissioner on the ‘problem’ of encryption) is a significant step back on the protection of the fundamental right of privacy.

So, how do we solve this predicament for Matrix?

Thankfully: there is another way.

This statement from the seven governments aims to protect the general public from bad actors, but it clearly undermines the good ones. What we really need is something that empowers users and administrators to identify and protect themselves from bad actors, without undermining privacy.

What if we had a standard way to let users themselves build up and share their own views of whether other users, messages, rooms, servers etc. are obnoxious or not? What if you could visualise and choose which filters to apply to your view of Matrix?

Just like the Web, Email or the Internet as a whole, there is literally no way to unilaterally censor or block content in Matrix. But what we can do is provide first-class infrastructure to let users (and room/community moderators and server admins) make up their own mind about who to trust, and what content to allow. This would also provide a means for authorities to publish reputation data about illegal content, providing a privacy-respecting mechanism that admins/mods/users can use to keep illegal content away from their servers/clients.

The model we currently have in mind is:

• Anyone can gather reputation data about Matrix rooms / users / servers / communities / content, and publish it to as wide or narrow an audience as they like - providing their subjective score on whether something in Matrix is positive or negative in a given context.
• This reputation data is published in a privacy preserving fashion - i.e. you can look up reputation data if you know the ID being queried, but the data is stored pseudonymised (e.g. indexed by a hashed ID).
• Anyone can subscribe to reputation feeds and blend them together in order to inform how they filter their content. The feeds might be their own data, or from their friends, or from trusted sources (e.g. a fact-checking company). Their blended feed can be republished as their own.
• To prevent users getting trapped in a factional filter bubble of their own devising, we’ll provide UI to visualise and warn about the extent of their filtering - and make it easy and fun to shift their viewpoint as needed.
• Admins running servers in particular jurisdictions then have the option to enforce whatever rules they need on their servers (e.g. they might want to subscribe to reputation feeds from a trusted source such as the IWF, identifying child sexual abuse content, and use it to block it from their server).
• This isn’t just about combating abuse - but the same system can also be used to empower users to filter out spam, propaganda, unwanted NSFW content, etc on their own terms.

This forms a relative reputation system. As uncomfortable as it may be, one man’s terrorist is another man’s freedom fighter, and different jurisdictions have different laws - and it’s not up to the Matrix.org Foundation to play God and adjudicate. Each user/moderator/admin should be free to make up their own mind and decide which reputation feeds to align themselves with. That is not to say that this system would help users locate extreme content - the privacy-preserving nature of the reputation data means that it’s only useful to filter out material which would otherwise already be visible to you - not to locate new content.

In terms of how this interacts with end-to-end-encryption and mitigating abuse: the reality is that the vast majority of abuse in public networks like Matrix, the Web or Email is visible from the public unencrypted domain. Abusive communities generally want to attract/recruit/groom users - and that means providing a public front door, which would be flagged by a reputation system such as the one proposed above. Meanwhile, communities which are entirely private and entirely encrypted typically still have touch-points with the rest of the world - and even then, the chances are extremely high that they will avoid any hypothetical backdoored servers. In short, investigating such communities requires traditional infiltration and surveillance by the authorities rather than an ineffective backdoor.

Now, this approach may sound completely sci-fi and implausibly overambitious (our speciality!) - but we’ve actually started successfully building this already, having been refining the idea over the last few years. MSC2313 is a first cut at the idea of publishing and subscribing to reputation data - starting off with simple binary ban rules. It’s been implemented and in production for over a year now, and is used to maintain shared banlists used by both matrix.org and mozilla.org communities. The next step is to expand this to support a blendable continuum of reputation data (rather than just binary banlists), make it privacy preserving, and get working on the client UX for configuring and visualising them.

Finally: we are continuing to hire a dedicated Reputation Team to work full time on building this (kindly funded by Element). This is a major investment in the future of Matrix, and frankly is spending money that we don’t really have - but it’s critical to the long-term success of the project, and perhaps the health of the Internet as a whole. There’s nothing about a good relative reputation system which is particularly specific to Matrix, after all, and many other folks (decentralised and otherwise) are clearly in desperate need of one too. We are actively looking for funding to support this work, so if you’re feeling rich and philanthropic (or a government wanting to support a more enlightened approach) we would love to hear from you at [email protected]!

Here’s to a world where users have excellent tools to protect themselves online - and a world where their safety is not compromised by encryption backdoors.

-- The Matrix.org Core Team

*Comments at HN, lobste.rs, and r/linux, LWN

Matrix Live 🎙

In which I chat with Kitsune about the work done to get a Matrix URI schema agreed, and the state of the work.

See also, Open Tech Will Save Us #7 took place this week! Go watch.

Dept of Status of Matrix 🌡️

Meta-counting

As a crude measure of growth, Matthew commented about #twim:matrix.org:

I love that this room has ~700 people in it, spread over ~350 servers :D

That is something to love. Come join us in the room to share your news and see what's new from others.

Elokapina (Extinction Rebellion Finland) migrates to Matrix

kapina-jaywink told us:

In recent weeks the XR Finland community has been moving over from Wire to our own Matrix homeserver for encrypted secure chat. This was something that had been planned for a while but kicked off in recent weeks due to Wire suffering from serious encryption key delivery issues, causing messages for many to be unreadable in large groups. Currently we've migrated almost 300 rebels with more to come. Feedback has mostly been very positive, people generally like the Element clients 🤩 One of the interesting changes has been a huge uptick in the amount of discussion, which can be taken as a good sign. The plan is to next start bridging to some of the international XR chapters, for example those on Mattermost, Telegram and Slack. And maybe get them over to Matrix too eventually ;)

To aid in community management, we've started creating a bot called Bubo. Right now it mostly helps with maintaining rooms and allowing mass invites, but more features to help the community cooperate are coming. We were planning to utilize (actual) communities so it has some functionality for those, but decided then to wait for the communities rewrite. It doesn't yet have any releases, will update in coming weeks as features are added and releases made.

kapina-jaywink also

wants to clarify that XR is a decentralized movement and this does not mean other chapters will adapt Matrix - but we can hope and for sure here in Finland we'll be spreading the good experiences to other chapters ;)

Dept of Spec 📜

New Spec Website!

wbamberg told us:

We're working on a new platform for the specification docs, aimed initially at improving navigation and general usability.

The initial demo site is at https://adoring-einstein-5ea514.netlify.app/spec/ and the main tracking bug is at https://github.com/matrix-org/matrix-doc/issues/2822

Spec

anoa reported:

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.

MSC Status

Merged MSCs:

• No MSCs were merged this week.

MSCs in Final Comment Period:

• No MSCs are in FCP.

New MSCs:

• MSC2821: Testing Pushers

• MSC2815: Proposal to allow room moderators to view redacted event content

• MSC2813: Widget API error handling and validation

Spec Core Team

In terms of Spec Core Team MSC focus for this week, widgets are still the main focus: MSC2774 (widget URL template param), MSC2765 (widget avatars), and MSC2790 (modal widgets).

Dept of Servers 🏢

Dendrite / gomatrixserverlib

Dendrite is a next-generation homeserver written in Go

Neil Alexander reported:

We released our first beta last week and we've been busy taking feedback from the community and fixing problems as they have been reported to us. We will be cutting a v0.2.0 release candidate later today, which contains a significant number of important fixes and performance improvements.

We didn't publish a full changelog last week due to the beta announcement, therefore the following list includes changes from the last two weeks:

• Database migrations are now ran automatically at startup

• State resolution v2 performance has been improved dramatically on large state sets, seen to be running up to 20x faster in some rooms

• Dendrite no longer runs state resolution over single trusted state snapshots

• Monolith deployments with Kafka now work again (each component sets up connections independently to avoid duplicate consumers)

• Dendrite now correctly rejects invalid UTF-8 - thanks to Lesterpig

• Fully read markers are now handled - thanks to Lesterpig

• The completed field is now returned properly for user-interactive auth - thanks to Lesterpig

• The devices table now tracks last seen timestamps, IPs and user agents - thanks to S7evinK

• A bug has been fixed in the reverse topological ordering algorithm which resulted in us giving up on inbound references after the first prev/auth event

• A bug with concurrent map writes in the rate limiting in the client API has been fixed

• Forward extremities and their previous events are now checked fully against the database

• Typing events are now ignored if the sender domain doesn't match the origin

• Duplicate redaction entries no longer result in database errors

• Some bugs in /send have been fixed where the full room state wasn't requested properly before sending a new snapshot to the roomserver

• The membership updaters now use database writers properly, which fixes some SQLite locking issues

• The sync API no longer burns CPU processing unnecessary device key change notifications

• QueryStateAfterEvents now resolves state from multiple snapshots properly

• Cumulative room state is now sent to the roomserver when creating rooms locally

• Missing auth events will now be retrieved from multiple servers in the room if necessary

• Federation requests now have variable timeouts, allowing us to wait much longer for a remote server to process certain tasks

• The /send endpoint now returns 500 errors far less often, reducing the frequency that other servers back off from sending to Dendrite

• Backfill no longer uses the request context for persisting events, which was resulting in us failing to store those events sometimes

• Invite stripped state from the sync API now includes the stripped invite itself, so that Element Web can display who sent the invite properly

• The signing key fetch mechanism no longer gives up if it is unable to fetch specific keys

• Handling of invalid display name or avatar URLs in membership events has been improved

Spec compliance has improved a little:

• Client-server APIs: 56%, same as last week

• Server-server APIs: 80%, up from 79% last week

As always, please feel free to join us in #dendrite:matrix.org for general Dendrite chat, and #dendrite-dev:matrix.org if you are interested in contributing!

Good grief that's a big update! For a video discussion of the status and future of Dendrite, check out Open Tech Will Save Us #7.

Matthew added:

my personal dendrite is now in roughly the same set of rooms as my personal synapse. Dendrite is idling at 180MB of RAM, Synapse is idling at 1.8GB of RAM :)

Conduit

Timo:

Hello everyone, this week we merged the federation branch into master. It's not ready to be used properly yet, but we're merging it as it seems stable enough for now. We also improved performance of the federation branch a lot by turning off debug logs.

Other news:

• I opened two issues on element-ios which currently break register and login support on Conduit, making it completely unusable. Hopefully they can be resolved soon (https://github.com/vector-im/element-ios/issues/3736, https://github.com/vector-im/element-ios/issues/3737)
• I'm working on an MSC for threading. It's still WIP, but you can take a look here: https://demo.codimd.org/s/SykbuAUwP

Thanks to everyone who supports me on Liberapay or Bitcoin!

Synapse

Neil offered:

Stonking week for Synapse as we landed sharded event persisters and deployed to matrix.org. This is the last significant component other than the main process to go through the sharding process and a major hurdle in horizontal scalability of Synapse.

Initial results look good with event persistence apdex improving, however we think there are still some significant performance improvements available through configuration and will continue to experiment.

We also moved off background processes from the main process. This is significant because it means that while the main process is not shardable it really doesn’t do anything anymore other than orchestration.

Again the initial impact looks very promising and we will continue to tune. Having moved the background processes away it also makes profiling the main process that much easier.

Aside from all of that we continue to progress room knocking put out a 1.21.2 - a bug fix release though please please ensure you are running at least Synapse >= 1.21.0 since 1.21.0 contains a XSS security fix.

Next week we will carrying on tuning matrix.org and start to look at improving state resolution performance.

Synapse Deployment 📥️

Kubernetes

Ananace told us:

Just pushed the updated image tags and chart version for my K8s-optimized Synapse image for version 1.21.0, as well as a chart update for element-web 1.7.9

then later

And to expand on my previous update, got Synapse 1.21.2 up on the chart and K8s-optimized image.

YunoHost

Pierre said:

YunoHost is an operating system aiming for the simplest administration of a server, and therefore democratize self-hosting.

Synapse integration had been updated to 1.20.1 (1.21.0 available in branch testing)

Element Web integration had been updated to 1.7.8 (1.7.9 available in branch testing)

Dept of Bridges 🌉

Bridges increase their presence

Half-Shot told us:

Small bit of news I wanted to talk about from Bridge Island. My implementation for MSC2409 has been merged which means appservices / bridges can now listen in for incoming presence, typing and read receipt events! This means that the Slack bridge can now reliably send your typing status to Slack, and Bifrost can reliably bridge your everything to XMPP. The MSC is still in flux and could change, but for now this could really improve the native feeling of bridges :)

(Oh and I should mention anyone using matrix-appservice-bridge v2.2.0+ can use this behaviour for free)

mautrix-python implements Half-Shot's new features

Hot on his heels, Tulir announced:

I've been adding support for the MSCs Half-Shot implemented in Synapse to my bridges:

• Enabling end-to-bridge encryption now uses appservice login (MSC2778), which means setting up the shared secret login module is no longer required for e2be.

• mautrix-python has support for receiving ephemeral events via MSC2409 in a branch, which will be merged once Synapse v1.22 is released. After it's merged, /syncing with double puppets will no longer be necessary to bridge ephemeral events.

Both of these will also be implemented in mautrix-go/whatsapp soon.

Now I just need Half-Shot to make synapse send to-device events to appservices, after which bridges won't need any hacky /syncing at all.

Dept of Clients 📱

Hydrogen

Bruno offered:

Several releases this week (0.1.11 to 0.1.15) with lots of changes:

• url-based navigation has landed! All navigation in the app is now done through urls, meaning you can also bookmark any UI state (e.g. grid configurations).

• fixed 2 memory leaks (exposed now because you can unload your session without refreshing the page)

• fixed an issue with libolm running out of memory if you send a message to more than 44 devices (see issue #150).

• some logical additions now we have url navigation: restoring the last url when opening the app with the default route, and a button to close your session and go back to the picker.

• the app now blocks concurrent access to the same session from different tabs (it just closes the session in the non-active tab). This will prevent multiple syncs tripping over each other writing to indexeddb (e.g. ConstraintErrors and friends).

• updates are announced in the app (for now through a confirm dialog, but will use an in app notification once we have it)

• fixes updates not installing on iOS, by having an update prompt. To get this update on iOS though, you'll need to unpin the app, and pinning it again. You'll need to login again after this. All future updates should be installable through the update prompt once you have 0.1.15 though, you won't have to do this again normally.

• uses the hydrogen icon when pinning on iOS

I really recommend hitting https://hydrogen.element.io/ - what great progress!

Element Android

benoit reported:

We are currently preparing the release of the version 1.0.9 of Element Android, which contain searching messages in clear rooms and a lots of other improvements and new features.

The SDK 1.0.9 will also be released, with an updated readme, and a brand new sample app, written by Ganfra. It will help developers to start using the new SDK and can be found here: https://github.com/matrix-org/matrix-android-sdk2-sample. This sample app is able to let the user connects to an existing account on any homeserver with password login, display the room list, display a room timeline and send message to a room. a brand new sample app

YES! This is the best documentation

Element-iOS

Manu offered:

This week, we released 1.0.16 on the App Store (and TestFlight).

Konheko (Sailfish client)

Nico (@deepbluev7:neko.dev) offered:

I published the first preview of my Sailfish client called Konheko. While you can run Android applications on Sailfish, they usually are a subpar experience, since they really don't fit the platforms design and style and also usually don't properly send notifications.

So about a year ago I started working on a Matrix client for SailfishOS, but I never really made much progress. Well, last weekend I did, and so it can now send plain text messages as well as various forms of media messages, I made a basic application icon and I've been using it this week already (for unencrypted rooms).

It is still missing a lot of features, but if you want you can install it from OpenRepos. Sources are available here. Just be aware, that it currently stores all messages in RAM, so every restart will take forever to load your rooms and it may run out of RAM at some point. Storing messages in some database will come at some point. Also, a lot of menus may lead nowhere, since those are just placeholders for me atm.

SchildiChat Web/Desktop

SpiritCroc announced:

Recently, I tweaked Element-web to feature a few changes similar to SchildiChat for Android.

For now, it's probably best seen rather as a proof-of-concept than a finished product, as there are still some layout bugs, and no settings available for the added features (I know some people prefer separate lists for direct and group chats). I consider it usable though.

Particular changes compared to upstream Element-web include:

• A common section for groups and direct chats in the overview

• Message bubbles

• Bigger items in the room overview

• A different dark theme, similar to SchildiChat for Android

I don't know how much I will work on this in the future, but I figured it might be interesting to share either way. Maybe even someone with more web-development skills than me might want to help improving it :)

For further discussion, I have created #schildichat-web:matrix.org .

The current version of SchildiChat-Desktop is available for Desktop here, and I host the web variant here. If you want to build it yourself, check out this repo.

Element Web

Neil offered:

Element 1.7.9 has landed, highlights include

• Many small fixes with edits and replies

• Fixed a race during cross-signing key upload at registration time

• Clarified when you have unsaved changes in profile settings

Aside from that has been pushing on with the widgets project. A picture says a thousand words, so here you go.

As you can see T3chguy was really pleased to have me interrupt him to take this picture. Expect the new design to merge next week.

Finally we will most likely ship a new release next week to fix some Jitsi bugs.

Dept of Ops 🛠

Wake-on-LAN bot

JCG announced:

I wrote a Wake-on-LAN bot to wake up hosts by sending a matrix message. It is configurable with multiple hosts and has a list of users per host who are allowed to wake it up. It's using the matrix-rust-sdk, source is available over at https://git.jcg.re/jcgruenhage/matrix-wol, and if anyone has questions, feel free to join #matrix-wol:jcg.re.

When I asked what this is used for:

I have stuff on my workstation that I need access to most of the time, but keeping it running uses too much power (but I did it anyway so far), this is so that I can suspend it when I leave but can still power it on when I need something from there on the got

Dept of Events and Talks 🗣️

Talking about Bridges and Bots in Matrix (German)

Oleg announced:

I was invited to the German Podcast MacMittwoch (no, it's not only about Macs) to talk about Bridges and Bots in Matrix. It was a very interesting and funny round.

Here is the recording:

• https://macmittwoch.de/podcast/matrix_mit_bridges_und_bots

Dept of Interesting Projects 🛰️

matrix-emoji-upload

mewmew offered:

This is a script I've created for use with MSC2545. It allows easy uploading of emoji packs to Matrix rooms. Feel free to check it out on Gitea, or join the project room #matrix-emoji-upload:blob.cat if you have any questions/comments/issues.

n8n.io support

Matthew announced:

n8n.io (FOSS extendable workflow automation) just added Matrix support! https://n8n.io/integrations/n8n-nodes-base.matrix and https://github.com/n8n-io/n8n/pull/1046

Exciting! Yet the saga that followed only adds to the excitement!

First Oleg noticed a problem:

Could it be that only matrix.org as HS currently is supported?

I'm getting Matrix credentials are not valid! and I see the request to matrix.org on a traffic capture.

jaywink discovered the alarming truth:

the original PR mentions mantrixorg indeed: https://github.com/n8n-io/n8n/pull/1024 .. sounds like PR time for someone :)

Tulir, like a coiled spring, provided a PR:

well now it's a pull request https://github.com/n8n-io/n8n/pull/1065

...

oh nice it got merged already

Faelar noticed:

For those not following on github, n8n released a new version including the fix for homeserver

Oleg, who brings us back to the start said:

Just tested it.

It works! 🥳

Open Source!

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

Rank	Hostname	Median MS
1	imninja.net	466.5
2	midov.pl	531
3	mdpnd.ch	540
4	chatcloud.net	569
5	elcyb.org	801
6	fab.network	967
7	conduit.rs	1168
8	envs.net	1863.5
9	blob.cat	2660
10	aragon.sh	2743.5

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Hi folks,

Today we have released Synapse 1.21.2, which fixes a couple of minor bugs that crept into the previous release. Full details are below.

Separately, we are advising any administrators who have not yet upgraded to Synapse 1.21.0 or later to do so as soon as possible. Previous versions of Synapse were vulnerable to a cross-site-scripting (XSS) attack; the bug was fixed in Synapse 1.21.0 with PR #8444.

The changelog for 1.21.2 is as follows:

Synapse 1.21.2 (2020-10-15)

Debian packages and Docker images have been rebuilt using the latest versions of dependency libraries, including authlib 0.15.1. Please see bugfixes below.

Bugfixes

• Fix rare bug where sending an event would fail due to a racey assertion. (#8530)
• An updated version of the authlib dependency is included in the Docker and Debian images to fix an issue using OpenID Connect. See #8534 for details.

Synapse 1.21.1 has landed!

Highlights of 1.21.1 include:-

• Add experimental support for sharding event persister. (#8294, #8387, #8396, #8419)

• Add experimental prometheus metric to track numbers of "large" rooms for state resolutiom. (#8425)

• Add prometheus metrics to track federation delays. (#8430)

• Fix messages not being sent over federation until an event is sent into the same room. (#8230, #8247, #8258, #8272, #8322)

We've also made some improvements to SSO and added new admin APIs.

Get the new releases from any of the usual sources mentioned at https://github.com/matrix-org/synapse/blob/master/INSTALL.md. 1.21.1 is on github here

The changelog for 1.21.1 is as follows:

Synapse 1.21.1 (2020-10-13)

This release fixes a regression in v1.21.0 that prevented debian packages from being built.

It is otherwise identical to v1.21.0.

Synapse 1.21.0 (2020-10-12)

No significant changes since v1.21.0rc3.

As noted in v1.20.0, a future release will drop support for accessing Synapse's Admin API under the /_matrix/client/* endpoint prefixes. At that point, the Admin API will only be accessible under /_synapse/admin.

Synapse 1.21.0rc3 (2020-10-08)

Bugfixes

• Fix duplication of events on high traffic servers, caused by PostgreSQL could not serialize access due to concurrent update errors. (#8456)

Internal Changes

• Add Groovy Gorilla to the list of distributions we build .debs for. (#8475)

Synapse 1.21.0rc2 (2020-10-02)

Features

• Convert additional templates from inline HTML to Jinja2 templates. (#8444)

Bugfixes

• Fix a regression in v1.21.0rc1 which broke thumbnails of remote media. (#8438)
• Do not expose the experimental uk.half-shot.msc2778.login.application_service flow in the login API, which caused a compatibility problem with Element iOS. (#8440)
• Fix malformed log line in new federation "catch up" logic. (#8442)
• Fix DB query on startup for negative streams which caused long start up times. Introduced in #8374. (#8447)

Synapse 1.21.0rc1 (2020-10-01)

Features

• Require the user to confirm that their password should be reset after clicking the email confirmation link. (#8004)
• Add an admin API GET /_synapse/admin/v1/event_reports to read entries of table event_reports. Contributed by @dklimpel. (#8217)
• Consolidate the SSO error template across all configuration. (#8248, #8405)
• Add a configuration option to specify a whitelist of domains that a user can be redirected to after validating their email or phone number. (#8275, #8417)
• Add experimental support for sharding event persister. (#8294, #8387, #8396, #8419)
• Add the room topic and avatar to the room details admin API. (#8305)
• Add an admin API for querying rooms where a user is a member. Contributed by @dklimpel. (#8306)
• Add uk.half-shot.msc2778.login.application_service login type to allow appservices to login. (#8320)
• Add a configuration option that allows existing users to log in with OpenID Connect. Contributed by @BBBSnowball and @OmmyZhang. (#8345)
• Add prometheus metrics for replication requests. (#8406)
• Support passing additional single sign-on parameters to the client. (#8413)
• Add experimental reporting of metrics on expensive rooms for state-resolution. (#8420)
• Add experimental prometheus metric to track numbers of "large" rooms for state resolutiom. (#8425)
• Add prometheus metrics to track federation delays. (#8430)

Bugfixes

• Fix a bug in the media repository where remote thumbnails with the same size but different crop methods would overwrite each other. Contributed by @deepbluev7. (#7124)
• Fix inconsistent handling of non-existent push rules, and stop tracking the enabled state of removed push rules. (#7796)
• Fix a longstanding bug when storing a media file with an empty upload_name. (#7905)
• Fix messages not being sent over federation until an event is sent into the same room. (#8230, #8247, #8258, #8272, #8322)
• Fix a longstanding bug where files that could not be thumbnailed would result in an Internal Server Error. (#8236, #8435)
• Upgrade minimum version of canonicaljson to version 1.4.0, to fix an unicode encoding issue. (#8262)
• Fix longstanding bug which could lead to incomplete database upgrades on SQLite. (#8265)
• Fix stack overflow when stderr is redirected to the logging system, and the logging system encounters an error. (#8268)
• Fix a bug which cause the logging system to report errors, if DEBUG was enabled and no context filter was applied. (#8278)
• Fix edge case where push could get delayed for a user until a later event was pushed. (#8287)
• Fix fetching malformed events from remote servers. (#8324)
• Fix UnboundLocalError from occurring when appservices send a malformed register request. (#8329)
• Don't send push notifications to expired user accounts. (#8353)
• Fix a regression in v1.19.0 with reactivating users through the admin API. (#8362)
• Fix a bug where during device registration the length of the device name wasn't limited. (#8364)
• Include guest_access in the fields that are checked for null bytes when updating room_stats_state. Broke in v1.7.2. (#8373)
• Fix theoretical race condition where events are not sent down /sync if the synchrotron worker is restarted without restarting other workers. (#8374)
• Fix a bug which could cause errors in rooms with malformed membership events, on servers using sqlite. (#8385)
• Fix "Re-starting finished log context" warning when receiving an event we already had over federation. (#8398)
• Fix incorrect handling of timeouts on outgoing HTTP requests. (#8400)
• Fix a regression in v1.20.0 in the synapse_port_db script regarding the ui_auth_sessions_ips table. (#8410)
• Remove unnecessary 3PID registration check when resetting password via an email address. Bug introduced in v0.34.0rc2. (#8414)

Improved Documentation

• Add /_synapse/client to the reverse proxy documentation. (#8227)
• Add note to the reverse proxy settings documentation about disabling Apache's mod_security2. Contributed by Julian Fietkau (@jfietkau). (#8375)
• Improve description of server_name config option in homserver.yaml. (#8415)

Deprecations and Removals

• Drop support for prometheus_client older than 0.4.0. (#8426)

Internal Changes

• Fix tests on distros which disable TLSv1.0. Contributed by @danc86. (#8208)
• Simplify the distributor code to avoid unnecessary work. (#8216)
• Remove the populate_stats_process_rooms_2 background job and restore functionality to populate_stats_process_rooms. (#8243)
• Clean up type hints for PaginationConfig. (#8250, #8282)
• Track the latest event for every destination and room for catch-up after federation outage. (#8256)
• Fix non-user visible bug in implementation of MultiWriterIdGenerator.get_current_token_for_writer. (#8257)
• Switch to the JSON implementation from the standard library. (#8259)
• Add type hints to synapse.util.async_helpers. (#8260)
• Simplify tests that mock asynchronous functions. (#8261)
• Add type hints to StreamToken and RoomStreamToken classes. (#8279)
• Change StreamToken.room_key to be a RoomStreamToken instance. (#8281)
• Refactor notifier code to correctly use the max event stream position. (#8288)
• Use slotted classes where possible. (#8296)
• Support testing the local Synapse checkout against the Complement homeserver test suite. (#8317)
• Update outdated usages of metaclass to python 3 syntax. (#8326)
• Move lint-related dependencies to package-extra field, update CONTRIBUTING.md to utilise this. (#8330, #8377)
• Use the admin_patterns helper in additional locations. (#8331)
• Fix test logging to allow braces in log output. (#8335)
• Remove __future__ imports related to Python 2 compatibility. (#8337)
• Simplify super() calls to Python 3 syntax. (#8344)
• Fix bad merge from release-v1.20.0 branch to develop. (#8354)
• Factor out a _send_dummy_event_for_room method. (#8370)
• Improve logging of state resolution. (#8371)
• Add type annotations to SimpleHttpClient. (#8372)
• Refactor ID generators to use async with syntax. (#8383)
• Add EventStreamPosition type. (#8388)
• Create a mechanism for marking tests "logcontext clean". (#8399)
• A pair of tiny cleanups in the federation request code. (#8401)
• Add checks on startup that PostgreSQL sequences are consistent with their associated tables. (#8402)
• Do not include appservice users when calculating the total MAU for a server. (#8404)
• Typing fixes for synapse.handlers.federation. (#8422)
• Various refactors to simplify stream token handling. (#8423)
• Make stream token serializing/deserializing async. (#8427)