Matrix Live 🎙

Dept of Status of Matrix 🌡️

Matrix in use at IETF and TeamSpeak

Matthew said:

IETF is trialling Matrix as a replacement to Slack and a complement to XMPP: https://mailarchive.ietf.org/arch/msg/tools-discuss/cUd9P35cj-nGsaioZ9HmMLmf1G4/

ALSO

Matthew commented:

looks like teamspeak is using Matrix (a Matrix endpoint was previously noticed at https://tschat-1.teamspeak.com/)

Nico (@deepbluev7:neko.dev) offered:

More info here: https://community.teamspeak.com/t/teamspeak-development-status-update/11419/36

Dept of Spec 📜

anoa told us:

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals

MSC Status

Merged MSCs:

• MSC2788: Room version 6 as the default room version
• MSC2414: Make reason and score parameters optional for reporting content

MSCs in Final Comment Period:

• No MSCs are in FCP.

New MSCs:

• [WIP] MSC2812: Role-based power/permissions

• MSC2810: Consistent globs specification

• [WIP] MSC2802: Flip The Table for Full Room Abstraction

Spec Core Team

In terms of Spec Core Team MSC focus for this week, we're going to be keeping the same three MSCs as last week, as their review is still a work in progress. As a reminder, those are MSC2774 (widget URL template parameters), MSC2765 (widget avatars), and MSC2790 (modal widgets).

Dept of Servers 🏢

Dendrite goes beta

Dendrite is a next-generation homeserver written in Go

kegan told us:

Dendrite has officially entered beta!

It aims to be an efficient, reliable and scalable alternative to Synapse.

We encourage early adopters to try it out starting with v0.1.0 in Monolith/Postgres mode.

Dendrite supports a large amount of the Matrix spec including:

• All room versions

• E2E encryption (but not cross-signing)

• Federation

• and many others!

We'll be hard at work in the coming weeks ironing out any issues that crop up as Dendrite enters the global

federation of Matrix servers. At present, Dendrite has not been optimised for memory/CPU usage, nor does it fully support sharding, so there is still much scope to improve the already impressive resource usage whilst

heading towards 100% spec compliance.

Spec compliance:

• Client-Server APIs: 56%, same as last week

• Server-Server APIs: 77%, same as last week

stefan announced:

I created an AUR package for dendrite. So it is easy to get it running on Arch Linux.

Conduit

Conduit is a Matrix homeserver written in Rust https://conduit.rs

timo told us:

This week I worked on a new feature: the Admin Room. This is a room that is automatically created for the first user of a new homeserver. This room will be used to send warnings to server admins in the future and can be used to control Conduit (like querying information, running cleanup, banning servers, hiding public rooms or restarting Conduit). Power levels can be used to control who in the room has access to which commands. The admin room will work over federation, so it's easy to help someone fix his server.

Other news:

• Federation is now disabled by default and can be enabled in the config

• Bug with typing notifications is fixed

Thanks to everyone who supports me on Liberapay or Bitcoin!

Synapse

Neil told us:

This week we put out v1.21.0rc3, which fixes a duplicate messages bug. We expect the full release to go out early next week.

After a long and sometimes painful journey we plan to run sharded event persisters on matrix.org from Monday. All being well we will start running background processes on their own worker to free up the main process. These two projects combined should noticeably improve performance of matrix.org and any large scale deployments.

Aside from that we are continuing to work on room knocking as a feature and have been considering ways to reduce load attributable to state resolution.

We will also dust off the notifications project.

Dept of Bridges 🌉

Gitter

Eric Eastwood told us:

The native Gitter <-> Matrix bridge is underway and we're working on virtual users which is the first piece to making the bridge feel native. You can see what this will look like in this ominous and exciting tweet: https://twitter.com/gitchat/status/1314252991705292800

There is also a GitLab epic you can track which breaks down some of the tasks and holds more details of what we're thinking of for the bridge.

🌈 Once more into the Bifrost

Half-Shot announced:

Asgardians rejoice, for Bifrost development resumed this week! We've worked extremely hard this week to improve general gateway performance, edge case bugs and basically plowing on with the project. A 0.2.0 release will shortly be due with the new changes, and matrix.org is already running the latest and greatest. You can checkout the project over at https://github.com/matrix-org/matrix-bifrost

Dept of Clients 📱

Fractal

Alexandre Franke reported:

Former Fractal GSoC intern Alejandro Dominguez 😎️ got his Integrate matrix-sdk (I) merge request in “Ready for review” state. That’s a +3910/-5439, 38 commits diff‼️ poljar, who is the main contributor of matrix-rust-sdk, already had a look and provided some insights, but given 😱️ the sheer size of the changeset, the moar 👀️ we can get on it the better.

fluffychat

krille offered:

FluffyChat now runs on native Linux desktop (x86).

You can download Linux tarballs from the CI and run them on any system which has sqlite3 installed (which all distros may have). If you have libolm3 installed, you can also use e2ee. Oh and by the way. Its NOT electron or any html5 stuff! Its a native Linux app. We are working on packaging solutions like snap and Flatpak. 😊

Vtx

CraftedCart offered:

I've started to make a Matrix client over the past few weeks, just as a hobby project for now, using Qt and a 'lil C++20! :D

There's not a whole lot to show right now - currently we've got password login, and a simple timeline that can show plain text or m.image thumbnails.

Source code's up at https://gitlab.com/CraftedCart/vtx

Nheko

Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE (with the notable exception being device verification for now) and intends to be full featured and nice to look at

Nico (@deepbluev7:neko.dev) reported:

This week we finally cleaned up the remaining parts of Chethans GSoC work and merged it. This means Nheko now has partial support for cross-signing. You can either start the verification from another client or when viewing the profile of a user. There are a few things still missing though:

• Nheko does not use the verification status at all at the moment. So while you can verify your master key and let others verify your master key, from Nhekos perspective there is not much benefit yet. You will however show up with a green checkmark in other clients.

• While nheko supports verifying your own master key and after that will also let others verify your master key, it does not yet download the self and user signing keys. This means you won't be able to upload cross-signing signatures for your own devices and other users master keys yet. I'll see if I can implement in the next weeks or so (since literally only the private key download or sharing is missing).

• Nheko can't bootstrap cross-signing yet. This means you need to bootstrap it from Element or another client once. That will probably take a bit until this is complete.

• The UX is still WIP. Currently it shows a lot of unnecessary buttons, that are a bit confusing as well as not enough green checkmarks, so that needs some work.

Other things that happened in the mean time or are work in progress:

• Decryption errors should be reduced by a bit now. This is still work in progress though. With device trust landed now, we should be able to do automatic key requests, sharing and key backup soon.

• Lurkki is currently cleaning up the design of the built in video player.

• manu_kamath is improving the design when showing images at the same time.

• LorenDB has been updating our screenshots and working on an Esperanto translation.

• trilene has been ever busy improving edge cases around the Voip support in Nheko.

So far Hacktoberfest seems to be going well with quite a few people trying to clean up some of the rougher edges in Nheko. If you are using Nheko and think some things could be better, why not try to fix it yourself? October is the perfect time for that!

Featurework of this cycle is also slowly nearing its end. Once cross-signing, voip and the new event store are stable, we plan to improve the UX a bit more as well as some other annoyances and maybe we can do a new release then! A nice and spooky Hacktober everyone!

David Mehren told us:

I just built nheko-git 0.7.2.r2021.517a126a-1 from the AUR and cross-signing works

Element Web

Neil offered:

This week we put out 1.7.9-rc.1 which is available at https://staging.element.io

Highlights include

• Many small fixes with edits and replies

• Fixed a race during cross-signing key upload at registration time

• Clarified when you have unsaved changes in profile settings

Finally, the improved widget support is looking very exciting. Here is a design grab to give you an idea of what to expect. Note this is not a real Element screen shot, the base functionality is there, but the look and feel of the final version is subject to change.

Michael (t3chguy) added:

Element Web is currently transitioning the codebase over to TypeScript - track our progress at http://arewetsyet.bit.ovh/

Hydrogen

Bruno told us:

This week, Hydrogen gained better caching (and a little over-eager on Safari), room list filtering, a grid layout to show up to 6 rooms simultaneously (see screenshot) and work is well on its way for url-based navigation within the app.

also adding the screenshot on iOS that I didn't get in last week, showing the full range of display formats supported 🙂

The app also received some visual polish, thanks to Nad, our designer.

Element-iOS

Manu offered:

We blocked the last week release (1.0.14) because of 2 issues: Unable to decrypt errors and timeline not updating when coming from a notification.

1.0.15 fixes those issues and more. The release should be available in TF soon. Hopefully, we will be able to publish it on the App Store on Monday

Dept of VoIP 🤙

Matrix VoIP Tester

reivilibre said:

https://github.com/matrix-org/voip-tester/pull/18 (updated demo: https://test.voip.librepush.net)

This week, I have improved the scoring rubric, so that the verdict given to you by the tester more accurately reflects the likelihood of connection establishment.

I have also added more hints on what the verdicts mean.

Finally, I have added workarounds to support Chromium browsers that decided to break their WebRTC API so that they could pedantically rename ip to address. In any case, it now seems to be on par with Firefox.

The VoIP tester is now roughly 'usable' but please note that IPv6 support is totally untested (I don't have IPv6) — it is also suspected that some browsers won't try IPv6 anyway.

There are also some more features on the wish list, such as screenshot safety and extra help when things go wrong.

For discussion, feel free to join #voip-tester:librepush.net .

Dept of Encryption 🔐

Olm 🦎

uhoreg reported:

Versions 3.2.0 and 3.2.1 of libolm have been released. 3.2.0 supports fallback keys (MSC2732) in the C library and the JavaScript binding, and includes several JavaScript/TypeScript fixes and improvements. 3.2.1 was just a fix to the TypeScript definition file.

Dept of SDKs and Frameworks 🧰

Ruma

Ruma is a Rust project to create a comprehensive set of APIs for Matrix. Previously there was a Ruma homeserver project.

jplatte said:

This week, q-b implemented moderation policy events, bringing us to full compatibility with r0.6.1 of the client-server API. iinuwa added the only endpoint we were missing from the appservice API. Additionally, I improved Ruma's JSON canonicalization, making it both simpler and more efficient.

Due to two issues regarding future-compatibility, we're still a while away from our next set of releases, but we're working on it!

New library: ObjMatrix (Objective-C)

js introduced it with:

ObjMatrix (GitHub) is a new Matrix client library written in modern Objective-C for ObjFW. Because it uses ObjFW, it is not limited to Apple platforms, and instead is extremely portable and should run on basically everything.

This is currently in its very early stages and is intended to be suitable to develop bots and clients with very soon. Being suitable to develop bridges with it is another goal for later.

Since ObjFW also works on platforms that are usually hard to support or port software to (e.g. MorphOS), this will hopefully also bring clients for these less mainstream operating systems. I'm especially hopeful about MorphOS here, since ObjFW already runs quite well there (parts of it are included with the OS even!), and hardware on which MorphOS runs on is usually not powerful enough to run Element Web. And eventually, I will want to have a Matrix client on my Amiga, too 😉 (which this should enable).

This currently does not have a decicated Matrix room yet, so all discussions about it currently happen in #objfw:nil.im. So please feel free to join there and follow along development. And contributions are of course very welcome!

Dept of Ops 🛠

famedly.matrix ansible collection

JCG told us:

the famedly.matrix ansible collection has seen a release v0.2.0. Highlights since the last TWIM mention:

• synapse_register: a new module for registering users using synapse's admin API.

• matrix_member: another new module, for inviting/kicking/banning people to/from rooms.

• updates to synapse and element have been merged into their roles

• support deploying release candidates of synapse and element

Dept of Guides 🧭

Noteworthy Hub Guide

balaa told us:

We’ve put together a write up on how to deploy a Noteworthy Hub (transparent TLS proxy) in order to make it easy to run a federation capable matrix home server behind a firewall or NAT: https://noteworthy.tech/hub

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

Rank	Hostname	Median MS
1	helderferreira.io	487
2	heitkoetter.net	627
3	asra.gr	802
4	conduit.rs	1178
5	matrix.thedisco.zone	1275.5
6	shortestpath.dev	1803
7	lelux.net	2419
8	elcyb.org	2685
9	test.zemos.net	2800
10	blob.cat	2882

That's all I know 🏁

Thanks anoa for TCB* the last two weeks! Was fun to be the one to read TWIM for myself. :D

See you next week, and be sure to stop by #twim:matrix.org with your updates!

* Taking Care of Business

Hi all,

We’re very excited to announce that Dendrite, the next-generation Matrix homeserver from the core Matrix team, is at last exiting alpha development and entering beta testing!

The path we’ve taken to get here has been quite a curious one, and it’s worth recapping to give context on why it’s taken reality a little while to catch up with the dream. :)

The Dendrite project has its roots in 2016 as Dendron: an attempt to write a next-generation homeserver in Golang rather than Python, in order to benefit from Go’s stronger typing, ease of profiling (no twisted stack-shredding via deferredInlineCallbacks), multithreading and faster GC performance. The idea for Dendron was to do a strangler pattern rewrite of Synapse - where we’d insert Dendron in front of Synapse as a load balancer, and incrementally replace Synapse’s API endpoints with ones implemented by Dendron.

However, as the project started to progress, it became clear that this was going to end up with many of Synapse’s architectural choices being baked into the project - particularly the DB schema and data flow architecture, such that the new endpoints could interoperate with the existing Python ones. We got as far as putting Dendron live on matrix.org and moving some of the login/registration APIs over to it… but then work fizzled out due to Synapse demanding more urgent attention as traffic grew on Matrix.org, combined with concerns about whether Dendron was the right approach in general.

So, towards the end of 2016 (after the rush to launch Vector Riot Element that summer), we went back to the drawing board to devise Dendrite—“Dendron done right!”—as opposed to Dendron, which in retrospect was Dendrite done wrong. ;) The new vision was:

• Build a massively horizontally scalable architecture, such that large Matrix deployments like matrix.org and big government deployments could run smoothly without the constant scalability headaches we were seeing at the time with Synapse
• Do so by splitting the server into well-defined microservice components, each of which could independently horizontally scale, each with its own DB (if desired)
• Connect the components together with a set of append-only logs via Kafka or similar, easily letting components shard and maintain their databases from the logs, allowing rolling upgrades, possibly schema upgrades, and all sorts of other niceties. The logs effectively become a primary source of truth rather than putting all the onus on a massive monolithic ever-growing database

Rather than Dendron’s top-down approach, instead Dendrite started bottom-up with the very hardest bit: gomatrixserverlib, a standalone Go library implementing the state resolution algorithms and performing federation requests (such that it might also someday be used as a general purpose way to add Matrix federation support to an existing Go codebase).

Then we started building out the various components to implement the various services, starting with the roomserver (the service which models the history and state of one or more rooms in the server), then the syncserver (the service which implements the /sync API to let clients receive messages), etc. We even implemented a simplified in-memory version of Kafka named naffka—useful for glueing together the microservice components when running them all within a single binary.

Things were looking pretty positive by the summer of 2017: we had the server sending/receiving messages, federating with Synapse, and looking tantalisingly close:

We just sent the first ever synapse->dendrite federated traffic, including full dendrite media API (thumbnailing, fed, etc)!!! :D :D :D pic.twitter.com/sBcM2jMAr6

— Matrix (@matrixdotorg) June 8, 2017

However, we then hit three fairly major obstacles:

• Matrix lost its funding
• In the ensuing uncertainty, the two lead developers (Mjark & Kegan) went to work elsewhere
• Meanwhile, Matrix uptake was starting to explode and Synapse was failing to scale to handle the traffic on matrix.org (and elsewhere)

At first, having formed what would become New Vector (now Element) to keep the rest of the core team hired, we pushed to see if we could get Dendrite finished fast enough to replace Synapse, with Erik & richvdh jumping over from Synapse to pick up the remaining work. However, it became clear that we urgently needed a quicker solution to address all the overloaded Synapses out there, and so they swung back to focus on improving Synapse (taking inspiration from some of the design of Dendrite - e.g. offloading endpoints onto worker processes connected via replication streams, and using OpenTracing to debug traffic as it flows over the various services).

At this point, Dendrite maintenance was in effect valiantly taken over by the community, with Brendan and later Anoa keeping the ball going in 2017, joined by APWhitehat in GSoC 2018 and cnly in GSoC 2019. The fact that Dendrite is now here today is thanks in no small part to their work to keep the project alive in its “wilderness years” between Sept 2017 and Dec 2019.

Meanwhile, it became clear that we were overdue getting Matrix itself out of beta - and the last thing we wanted to do was to split and dilute the implementation work of Matrix 1.0 over both Synapse and Dendrite - so we consciously made the decision to focus all our effort on Synapse for solving the remaining bugs and challenges.

Then, in July 2019, Matrix and Synapse exited beta, and we finally started to see light at the end of the tunnel. In October we started dusting off Dendrite again - looking to use it as a relatively simple and flexible codebase for experimenting with Peer-to-Peer Matrix, not least because being Go it can compile to WebAssembly and run clientside, and because even though Dendrite was originally built with massive deployments in mind, it turns out the elastic scaling means it can also scale down pretty small too—as a part of the iOS P2P demo, we’ve even ran full Dendrite homeservers on iPhones embedded into Element iOS! :)

In Dec 2019, we finally got to the point where Element could fund full-time dedicated development on Dendrite once again, with Neil Alexander joining the project and focusing fulltime on getting Dendrite out of alpha and getting it working for P2P and embedded usage (adding libp2p as a federation transport, and adding SQLite support) - and in Jan 2020 we got Dendrite successfully running clientside in a WASM service worker (just in time for FOSDEM!). Then, in Feb 2020, Kegan returned to the project to work fulltime on Dendrite - and the race began in earnest to get Dendrite ready for beta!

Here’s a pretty picture courtesy of GitHub to visualise the progress:

Throughout 2020 there’s been a huge amount of stabilisation work and polish:

• Refactoring much of Dendrite’s foundation to make the codebase more maintainable
• Created all-new user server, key server, signing key server microservices
• Moving some work from existing microservices (ultimately superseding the former currentstateserver, publicroomsapi and typingserver microservices altogether)
• Developing new testing infrastructure:
  • Complement - our brand new Golang Matrix integration test harness
  • Are We Synapse Yet - an aggregator which parses sytest/complement output to compare how close Dendrite is to passing
• All the Matrix 1.0 work - particularly state res v2 & room version support
• Making it work with more P2P transports for all the exciting P2P experiments
• Supporting backfill and fetching missing events
• Fixing up SQLite support to make it work as a first class citizen (with shared storage code where we can!)
• Supporting both sending and rejecting invites (even over federation)
• E2E encryption support (one-time keys, device lists, send-to-device support)
• Improved federation sender logic (resend retries, backoffs, blacklisting, metrics, resetting backoffs when receiving transactions)
• Handling both inbound and outbound redactions
• User interactive authentication (and implemented on various ‘sudo’ endpoints e.g. deleting devices and changing passwords)
• Respecting server ACLs
• Rejecting / soft-failing events properly
• Support for database schema upgrades

... which brings us at last to the present day (Oct 2020), as we declare Dendrite sufficiently stable that we consider it ready for beta testing!

In practice, this means **Dendrite is now ready for experimentation by adventurous Matrix sysadmins. It is NOT ready for production usage yet, but we need folks to test it and help us iron out the remaining bugs! **Please do not trust it with sensitive data yet, and we don’t recommend trying to run it at scale yet as we haven’t done any serious optimisation work yet.

That said, we do provide the following guarantees:

• We’re providing versioned releases from here on in, beginning with 0.1.0
• We don’t expect any major breaking changes to the config or architecture before 1.0
• Ready for early adopters to try running Dendrite without experiencing ~daily breaking churn
• The database schema is now stable and will upgrade itself going forwards - your database should now be here to stay! (assuming we don’t hit any nasty data loss bugs during beta)

In terms of comparison with Synapse, the main things you should get excited about are:

• Dendrite aims to provide an efficient, reliable and scalable alternative to Synapse:
  • Efficient: A small memory footprint with better baseline performance than an out-of-the-box Synapse
  • Reliable: Implements the Matrix specification as written, using the same test suite as Synapse as well as a brand new Go test suite
  • Scalable: can run on multiple machines and eventually scale to massive homeserver deployments
• This means significantly less memory usage than Synapse (depends on joined rooms, often between 50MB - 400MB resident memory) - although we haven’t tuned this at all yet!
• All-new database model, where every microservice instance has its own database tables, letting them scale arbitrarily wide
• The ability to efficiently use all your available CPU cores without needing to split into separate processes, thanks to Go and our extensive use of goroutines. No more Python global interpreter lock! :)
• Future experimental MSCs are likely to land in Dendrite before Synapse (e.g MSC2753 Peeking via /sync and MSC2444 Peeking over Federation are already being prototyped (#1370 and #1391) in Dendrite rather than Synapse!)

The provisos you should know about however are:

• We’re not feature complete yet: sytest reports 56% CS API coverage and 77% Federation coverage. NB: these are always going to be underestimates of how much Dendrite actually performs due to how the tests are spread out, in actuality it’s likely more 70% CS, 95% Fed.
• No read receipts, membership lazy-loading, presence, push notifications, search, event context, key backups, cross-signing. See changelog for full limitations.
• Not battle-tested in the wild by many people (there are probably only ~10 dendrites on the open network today!) - so there’s likely to be a broad spectrum of bugs at first.
• Clients that require more exotic features, like lazy loading, may not behave properly yet
• Please use Postgres rather than SQLite wherever possible—it’s faster and has fewer issues regarding concurrency (some requests on SQLite Dendrites may 500 with ‘database is locked’ - though we’ve worked hard to eliminate most of these)
• Dendrite can run in either “monolith” or “polylith” mode. In monolith, all the microservices are linked into a single binary - and we recommend running in this configuration wherever possible for now. Monolith mode is extremely capable as it is and has fewer moving parts for things to go wrong and will be the right choice for the majority of beta deployments!
• Whilst Dendrite is nearly 100% federation compatible, there may still be situations where it will split-brain and disagree with the current room state that Synapse has calculated. We expect these issues to resolve as we get more user feedback.

Architecture-wise, this is what Dendrite looks like under the hood today:

To get up and running, please install Go and head on over to the Get Started guide at https://github.com/matrix-org/dendrite#get-started to join the fun :)

In terms of where we’re going next:

• Read receipts. It’s a major missing feature and impacts UX significantly.
• 100% Federation coverage (according to sytest). It’s crucial that Dendrite instances play nicely with other servers. This will be the best metric we have for asserting that we are just as capable as Synapse at the fed level.
• Optimisation—Dendrite has not been optimised yet for speed or resource utilisation!
  • We plan to add benchmarks which will stress test different microservices in the presence of many different scaling factors (number of users, number of rooms, size of room, number of devices per user, number of sync requests, etc). This will hopefully allow us to identify early on bottlenecks and slow algorithms
  • Good old fashioned pprof with known slow scenarios to see what’s consuming CPU/memory and fixing issues ad-hoc (which we’ve already done a bit of pre-beta). This may involve adding additional in-memory caches, with a healthy respect for the complexities it may introduce (which Synapse has been bitten by)
• We plan to add first class feature flag support for experimental MSCs—experimentation is one thing which makes Dendrite notably different from Synapse, and supporting it more thoroughly going forwards will be important. This may mean adding additional hooks; potentially a dedicated microservice to cleanly separate experiments, we don’t know yet
• P2P work will continue with vigour now we have a working, featureful, and relatively stable HS to embed and play with

Longer term, it’s pretty hard to say right now when we expect to exit beta (it took Synapse 5 years to exit beta, after all ;) - but obviously we’ll need Dendrite to have parity with Synapse and have no known serious bugs.

Finally: you’re probably wondering what this means for Synapse. Synapse is here to stay - with tens of thousands of deployments around the world serving tens of millions of users. The majority of the core team is still focused on improving and optimising Synapse, and we’ll be keeping improving it for the foreseeable.

However, we’ll certainly be experimenting with new stuff on Dendrite first - whether that’s P2P, portable accounts, new-style communities, peeking etc. We expect Synapse to be the stable long-term-supported solution, while Dendrite (particularly while in beta) will be the more unstable and experimental platform. In the longer term we’ll provide ways of migrating from Synapse to Dendrite however (probably via portable accounts), and perhaps in future new deployments may choose to use Dendrite - a bit like you might choose to use nginx rather than Apache for a new web server these days. But this will be a long transition—meanwhile we expect to see more and more next-generation homeservers like Conduit, Mascarene or Construct coming of age too.

So, there you have it. If you’re an intrepid sysadmin please spin up a Dendrite and start filing bugs! :)

— Matthew, Neil Alexander, Kegan and the whole Matrix team.

Here’s the official changelog:

Client-Server API Features

Account registration and management

• Registration: By password only.
• Login: By password only. No fallback.
• Logout: Yes.
• Change password: Yes.
• Link email/msisdn to account: No.
• Deactivate account: Yes.
• Check if username is available: Yes.
• Account data: Yes.
• OpenID: No.

Rooms

• Room creation: Yes, including presets.
• Joining rooms: Yes, including by alias or ?server_name=.
• Event sending: Yes, including transaction IDs.
• Aliases: Yes.
• Published room directory: Yes.
• Kicking users: Yes.
• Banning users: Yes.
• Inviting users: Yes, but not third-party invites.
• Forgetting rooms: No.
• Room versions: All (v1 - v6)
• Tagging: Yes.

User management

• User directory: Basic support.
• Ignoring users: No.
• Groups/Communities: No.

Device management

• Creating devices: Yes.
• Deleting devices: Yes.
• Send-to-device messaging: Yes.

Sync

• Filters: Timeline limit only. Rest unimplemented.
• Deprecated /events and /initialSync: No.

Room events

• Typing: Yes.
• Receipts: No.
• Read Markers: No.
• Presence: No.
• Content repository (attachments): Yes.
• History visibility: No, defaults to joined.
• Push notifications: No.
• Event context: No.
• Reporting content: No.

End-to-End Encryption

• Uploading device keys: Yes.
• Downloading device keys: Yes.
• Claiming one-time keys: Yes.
• Querying key changes: Yes.
• Cross-Signing: No.

Misc

• Server-side search: No.
• Guest access: Partial.
• Room previews: No, partial support for Peeking via MSC2753.
• Third-Party networks: No.
• Server notices: No.
• Policy lists: No.

Federation Features

• Querying keys (incl. notary): Yes.
• Server ACLs: Yes.
• Sending transactions: Yes.
• Joining rooms: Yes.
• Inviting to rooms: Yes, but not third-party invites.
• Leaving rooms: Yes.
• Content repository: Yes.
• Backfilling / get_missing_events: Yes.
• Retrieving state of the room (/state and /state_ids): Yes.
• Public rooms: Yes.
• Querying profile data: Yes.
• Device management: Yes.
• Send-to-Device messaging: Yes.
• Querying/Claiming E2E Keys: Yes.
• Typing: Yes.
• Presence: No.
• Receipts: No.
• OpenID: No.

Matrix Live 🎙

Don't forget that Matrix Live is also available in podcast form. Search for "Matrix Live" wherever you get your podcasts.

Dept of Status of Matrix 🌡️

Gitter Enters the Matrix

Half-Shot reported:

Gitter is joining the Matrix ecosystem!

It's true! Element has acquired Gitter from GitLab and will be implementing Gitter's current chat features in Matrix, before rebuilding Gitter as a Matrix client! Before all that however, the first step is to build a proper, modern bridge between the two networks, replacing the old one that's been around for years.

The full spectrum of news is:

• The Matrix.org blog post
• The Element blog post
• The Gitter blog post
• Techcrunch article
• ChangeLog Podcast episode w/ Matthew Hodgson from Matrix and Sytse "Sid" Sijbrandij from GitLab

And if you'd like to chat with your fellow Matrix and Gitter users, there's a bridged room already set up at #gitter:matrix.org!

Welcome to the community, Gitter! 🎉

Dept of Spec 📜

anoa reported:

Spec

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.

MSC Status

Merged MSCs:

• No MSCs were merged this week.

MSCs in Final Comment Period:

• MSC2788: Room version 6 as the default room version (merge)
• MSC2414: Make reason and score parameters optional for reporting content (merge)

New MSCs:

• MSC2801: Make it explicit that event bodies are untrusted data

Spec Core Team

In terms of Spec Core Team MSC focus for this week, after dropping off MSC2414 in the FCP bucket, we're heading back for another big swing at widgets. MSC2774 (widget ID URL parameter), MSC2765 (widget avatars) and MSC2790 (modal widgets) are the focus for this week 🙂

Matrix URIs

kitsune told us:

being totally shook up about the Gitter announcement for a good part of the week, I completed the work on MSC2312 (it's about Matrix URIs, ICYMI) and it hopefully won't be long before it becomes the actual standard to share Matrix coordinates in popular web browsers. Ok, who am I kidding - before Matrix clients get on with adoption.

This will allow people to post links to Matrix rooms/messages/users and when clicked will open right in your favourite Matrix client. Super convenient and great for adoption!

Dept of P2P 👥

iOS P2P Demo

Dendrite is a next-generation homeserver written in Go. It is currently serving as the basis for peer-to-peer Matrix experiments

Neil Alexander announced:

Build 38 of the iOS P2P Demo, as built using Element iOS and Dendrite, has been submitted to TestFlight and will hopefully be available for testers shortly (pending Apple approval)! It features lots of updates in the Dendrite backend which should hopefully make it more reliable.

If you have an iPhone or iPad and enjoy things that sometimes work, join the TestFlight here!

Dept of Servers 🏢

Synapse

Neil reported:

This week we put out a new release candidate -1.21.0rc2

Highlights include

• Add experimental support for sharding event persister. (#8294, #8387, #8396, #8419)

• Add experimental prometheus metric to track numbers of "large" rooms for state resolutiom. (#8425)

• Add prometheus metrics to track federation delays. (#8430)

• Fix messages not being sent over federation until an event is sent into the same room. (#8230, #8247, #8258, #8272, #8322)

• Fix a regression in v1.21.0rc1 which broke thumbnails of remote media. #8438

Aside from that we are working on moving background processes away from the main process, actually getting the event persister sharding onto matrix.org and trying to improve Synapse stability generally.

Dendrite / gomatrixserverlib

Dendrite is a next-generation homeserver written in Go

Neil Alexander said:

Dendrite is nearing beta! Today we will be cutting a candidate 0.1.0rc1 version after a week of hunting and fixing bugs. We are on track to release version 0.1.0 next week, at which point we will be inviting people to try installing and using Dendrite!

Changes this week include:

• Initial sync is fixed after a bug caused us to fall back on incremental sync

• The Content-Type HTTP header is now handled properly when MIME-formatted

• Internal API calls over HTTP in polylith mode now use their own HTTP client with higher timeouts

• Dendrite now tries harder to find missing auth events, using fetcher workers

• Dendrite no longer falls back on /state unnecessarily, which was the cause of a major memory leak and high CPU usage

• Federation HTTP calls now include the User-Agent header

• The event depth field is now ignored in the federation API

• We now report the password change capability properly (thanks bn4t!)

• Registration flows now include the completed field after a failure (thanks Lesterpig!)

• A bug in the previous event updater in the roomserver has been fixed

• A bug where we didn't check our own old verify keys when verifying event signatures is fixed

• A bug where we didn't handle event ID domains being different to the event origin in earlier room versions has been fixed

• TLS fingerprints have been removed

Spec compliance is unchanged:

• Client-Server APIs: 56%, same as last week

• Server-Server APIs: 77%, same as last week

Recently, we updated some of our documentation and created a whole new set of easy first issues and medium to hard issues for contributors.

If contributing to Dendrite sounds like something you would be interested in, please take a look at these issues and join us in #dendrite-dev:matrix.org! There's also #dendrite:matrix.org for general Dendrite chat and updates and #dendrite-alerts:matrix.org for release notifications and important alerts.

Conduit

Conduit is a Matrix homeserver written in Rust https://conduit.rs

timo reported:

• Fix bug leading to many requests at the same time due to error in waiting code

• Implement /query/profile over federation

• Start work on /invite over federation

Thanks to everyone who supports me on Liberapay or Bitcoin!

The Construct

Jason reported:

This week in Matrix Construct supports aarch64 (ARMv8) architectures. The experience has been phenomenal, with performance exceeding all expectations. Last week I wrote about all new vector code inside Construct using SSE through AVX-512 hardware acceleration in servers; that same code is now accelerated by SVE (Scalable Vector Extensions) on ARM architectures when you compile with clang.

This result is important: ARM virtual machines are offered at a significantly lower price compared to x86 from the same vendor. The systems are cheaper, require less power, but generally perform worse. The trick is to optimize the software for the weaker hardware. The benefit allows, for example, Matrix server hosts running Construct to make th eir profit margin from the lower TCO, getting more out of every computing cycle.

Construct is available at https://github.com/matrix-construct/construct and don't forget to idle and idle #construct:zemos.net.

Synapse Deployment 📥️

YunoHost

Pierre announced:

YunoHost is an operating system aiming for the simplest administration of a server, and therefore democratize self-hosting.

Synapse integration had been updated to 1.19.3 (1.20.1 available in branch testing)

Element Web integration had been updated to 1.7.7 (1.7.8 available in branch testing)

Dept of Bridges 🌉

mautrix-signal

Tulir said:

I made a Signal bridge using signald.

Currently it supports bridging messages, reactions and signal read receipts. End-to-bridge encryption also exists (mautrix-python does all the work there). The bridge can be linked as a secondary device and possibly even registered as the main device, but I didn't actually test registering yet. Setup instructions are currently somewhat non-existent, but it's mostly the same as my other bridges plus signald as a separate daemon.

The repo is https://github.com/tulir/mautrix-signal and the room is #signal:maunium.net

He then popped up again a few days later to say:

After the initial announcement earlier this week, I implemented a few of the missing features like media bridging, and even added setup instructions.

The speed at which Tulir writes bridges scares me sometimes.

mx-puppet-discord

mx-puppet-discord is a (double)puppeting and relay bridge for discord, based on mx-puppet-bridge

sorunome said:

mx-puppet-discord got updated to the newest discord.js version, meaning you have to update if you want to continue to operate it, due to discord having changed their gateway url!

mx-puppet-discord now also supports the intent stuff, so be sure to update by 7th oct

If you run mx-puppet-discord (like I do), make sure to update by October 7th or it will stop working!

Bridges do Hackertoberfest

Half-Shot reported:

Hey folks, I wanted to give another shoutout to say that we are still accepting PRs as part of hacktoberfest. Contributing 4 PRs will get you a T-Shirt (sadly not a Matrix one). Obviously, please ensure your PRs are meaningful (no copyright adjustments, typo fixes).

You can work on any issue, but we've highlighted some issues that would be perfect for newcomers over at https://github.com/matrix-org/matrix-appservice-bridge/issues?q=is%3Aopen+is%3Aissue+label%3Ahacktoberfest and https://github.com/matrix-org/matrix-appservice-slack/issues?q=is%3Aopen+is%3Aissue+label%3Ahacktoberfest

Cadair reported:

In the hope of expanding the number of people contributing to the matrix-appservice-slack repo I have spent a chunk of my morning improving the issue descriptions and labelling up issues. If you are interested in fixing a little annoyance with the slack bridge or just fancy writing some typescript see the good first issue label on the repo.

matrix-appservice-slack 1.6.0 rocks the block

Half-Shot reported:

The matrix.org team are delighted to bring you the latest in Slack bridging technology. Do not let the

minor version bump fool you, this release is packed with the good stuff. The headline feature is that our phase 1 encryption feature has landed and is free for users to experiment with. Head over to the

docs to see how to set this up.

There have been other notable changes, such as:

• New configuration options to allow or deny some channels from being bridged.

• Support removing reactions from Slack and Matrix messages.

• Add onboarding message for new users when puppeting is enabled, to encourage them to puppet.

• Improved feature documentation

(and many many bugfixes)

You can read the release information over at https://github.com/matrix-org/matrix-appservice-slack/releases

Looking forward to seeing all the new Hacktoberfest contributors! 🎃

Dept of Clients 📱

fluffychat

sorunome offered:

Fluffychat 0.19.1 has been released!

Features

• Implemented ignore list

• Jump to events in timeline: When tapping on a reply and when tapping a matrix.to link

• Display messages with up to 10 emotes or emoji bigger

• New design for the chat list and message bubbles

• Implement reactions

• Implement password change

• Implement deactivate user account

Fixes

• Timeline randomly resorting while more history is being fetched

• Automatically request history if the "load more" button is on the screen

Hydrogen

Bruno said:

More browser compatibility work this week, making Hydrogen run IE11 on Windows 7, and on Safari on macOS and iOS (still with some caveats). Also fixed several bugs:

• fix for unable to open session after a synapse bug manifested itself

• prevent the app locking up when you start the app with previously unsent messages

• fix sync errors being reported as "null" in the banner

• handle timeout during initial sync (important for large accounts) (although I have seen 1 report that this still isn't fixed, please report if you can't login with a large account)

SchildiChat for Android

SpiritCroc said:

SchildiChat's codebase has been updated to Element 1.0.8!

Furthermore, there have been a few design updates:

• Media items (pictures, videos, stickers) are no longer displayed in message bubbles

• Bigger stickers

• Avatars are now hidden in direct chats (when using dual-side message bubbles)

Finally, Schildi doesn't crash anymore if somebody sends an empty message.

SpiritCroc also mentioned some relevant links!

• website: https://spiritcroc.de/schildichat-android

• repo: https://github.com/SpiritCroc/SchildiChat-android

• room: #schildichat-android:matrix.org

Element-iOS

Manu announced:

1.0.14 is in on the release path. It has:

• Room: Differentiate wordings for DMs

• Room: New room details screen

• Add Estonian support

• Polishment in several areas and many bug fixes

Full changelog:

https://github.com/vector-im/element-ios/releases/tag/v1.0.14 https://github.com/vector-im/element-ios/releases/tag/v1.0.13

Element Web

Neil announced:

This week released v1.7.8

Highlights include

• Secure Backup has been moved out of the registration flow to a toast when you first encounter an E2EE room, which simplifies the new user experience

• Added options to hide various UI features when hosting a custom Element

Aside from that we continue to improve on widget support for resizable widgets, modal widgets and generally making widgets better.

We are also continuing to work on instrumenting the app, improving mobile support for matrix.to and making jitsi calling more reliable.

Element Android

benoit told us:

Element Android: Version 1.0.8 is now available on the stores, it fixes issues with verification and PIN code among other issues (see https://github.com/vector-im/element-android/releases/tag/v1.0.8 for more details). Now we are working on improving performance when sending messages to rooms, and also improving global UX, especially of the home (rooms list). Search messages (in clear rooms for the moment) is coming soon, and it will be also possible to filter the room members list.

We will also spend some time on the new Android SDK, https://github.com/matrix-org/matrix-android-sdk2, which is for the moment a quick extract of what we have in Element Android. We have to take care of it as a real product now: document it properly, set up CI, export Javadoc, develop a sample app, etc.

Element for Nextcloud

Gary Kim reported:

Element for Nextcloud v0.6.11 has been released this week. The new version comes with various bug fixes, dependency upgrades, and an upgrade to Element Web v1.7.8. The version is also compatible with Nextcloud 20 which is being released soon.

Dept of SDKs and Frameworks 🧰

Ruma

iinuwa told us:

Over the past couple of weeks, we've received PRs for all of the remaining federation endpoints, and all but one have already been merged!

Now that the end is in sight, we're turning our focus elsewhere. We're working on cleaning up and fixing a few bugs in our event signing code and soon will create tracking issues for filling out the Identity Service API.

Dept of Services 🚀

t2bot.io

TravisR reported:

t2bot.io has crossed 1M monthly active users

All of these users are Telegram/Discord users that have been brought into Matrix over the last 30 days. This doesn't appear to be a temporary spike either: over the last 8 weeks t2bot.io has been hovering at 900-950 thousand monthly active users, up from 600-700 thousand. Record-setting traffic levels have also been achieved, with matrix.org being able to keep up for the first time in a long while.

Overall it's a good sign to see so many communities making the jump to Matrix and sticking around ❤️

Dept of Bots 🤖

zabbix-matrix

progserega told us:

I added zabbix-bot to my zabbix-matrix repo, which can get information about current problems from zabbix-server and send it to matrix user. https://github.com/progserega/matrix_zabbix

In our company we use it for get current situation and https://github.com/progserega/im_sender_service for sending events from zabbix.

I've found that having your systems reporting in a room while you chat around it can be really productive. Props to supporting yet another monitoring platform!

Dept of Interesting Projects 🛰️

Jitsi E2EE Calls using Olm

While 1-1 calls benefit from end-to-end encryption due to WebRTC, Jitsi group calls have always only benefitted from transport encryption.

A while ago Jitsi announced that they were adding E2EE to Jitsi. But did you know that it's using Matrix's Olm encryption under the hood? It's currently available as an experimental feature on https://meet.jit.si!

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

Rank	Hostname	Median MS
1	fairydust.space	368
2	blob.cat	543
3	nuclearlimes.co.uk	580
4	nuclearlemons.uk	889
5	conduit.rs	912.5
6	mailstation.de	1075
7	matrix.org	1092
8	test.zemos.net	1123
9	shortestpath.dev	1251.5
10	chatcloud.net	1338.5

We also now have a room where the ping bots are only hosted on non-Synapse servers! See the scoreboard below.

Rank	Hostname	Median MS
1	conduit.rs	104
2	settgast.org	154
3	construct.grin.hu	182
4	blob.cat	209
5	grin.hu	324.5
6	dendrite.neilalexander.dev	367
7	test.zemos.net	379
8	maunium.net	448.5
9	conduit.nordgedanken.dev	476
10	inferiorlattice.com	592

Final Thoughts 💭

XKCD 2365

Alexandre Franke told us:

XKCD made another strip about messaging systems and I’m outraged to see that Matrix has been left out of it.

Not to worry though, we've fixed it up!

TWIM authorship

I hope you enjoyed this week and last's TWIM editions! Benpa will return next week for more of your regularly scheduled programming. Ciao!

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Hi all,

We are ridiculously excited to announce that Gitter is joining the Matrix ecosystem and will become the first major existing chat platform to switch to natively speaking Matrix!

If you’re reading this from the Gitter community and have no idea what Matrix is: we’re an open source project that provides an open protocol for secure, decentralised communication - effectively the missing real-time communication layer of the open Web. The open Matrix network has more than 20M users on it and is growing fast (adding another 1.7M or so with the arrival of Gitter!)

Gitter is easily one of the best developer community chat systems out there, used by the communities of some massive projects (Node, TypeScript, Angular, Scala etc) and is a custodian of huge archives of knowledge via their chat logs. Gitter is unique in specifically focusing on developers: their tagline is literally “Where developers come to talk” (unlike Slack, which has barely any community features - or Discord, with its ban on unofficial clients, where developers are a bit of an afterthought relative to the gamers). With Gitter natively joining Matrix, we’re super excited to see the global developer community converging on the open Matrix network - and Gitter’s community rooms should see a huge new lease of life as they’re properly made natively available to the wider network as first class citizens :)

We’ve always had a bit of a crush on Gitter ever since we ended up opposite each other in the exhibition hall at TechCrunch Disrupt Europe 2014 - particularly when they demoed us not only their sexy webapp but also their official IRC server bridge at irc.gitter.im :D Over the years we’ve been gently nudging them to consider fully embracing Matrix, but perhaps understandably they’ve been busy focusing on their own stuff. However, earlier this year, our friends at GitLab (who acquired Gitter in 2017) reached out to explore the opportunity of Gitter becoming a core part of Matrix rather than a non-core project at GitLab… and we’ve jumped on that opportunity to bring Gitter fully into Matrix.

In practice, the way this is happening is that Element (the company founded by the Matrix core team to fund Matrix development) is acquiring Gitter from GitLab, with a combined Gitter and Element dev team focusing on giving Gitter a new life in Matrix! You can read about it from the Element angle over on the Element blog.

Practically speaking, we have a pretty interesting plan here, which we’d like to be very transparent about given it’s a little unusual:

At first, Gitter will keep running as it always has - needless to say, we will be doing everything we can to delight the Gitter community and keep the service in good shape.

Then we’re going to build out native Matrix connectivity - running a dedicated Matrix homeserver on gitter.im with a new bridge direct into the heart of Gitter; letting all Gitter rooms be available to Matrix directly as (say) #angular_angular:gitter.im, and bridging all the historical conversations into Matrix via MSC2716 or similar. We will of course do this entirely as open source, just as Gitter itself is open source thanks to GitLab releasing it under the MIT license in 2017. The plan is to comprehensively document our progress as the flagship worked example case study of “how do you make an existing chat system talk Matrix.”

This will of course replace the old and creaky matrix-appservice-gitter bridge we’ve been running since 2016. Gitter users will also be able to talk to other users elsewhere in the open Matrix network - e.g. DMing them, and (possibly) joining arbitrary Matrix rooms. Effectively, Gitter will have become a Matrix client.

Now we come to the interesting bit. Gitter has some really nice features which are sorely lacking in Element today:

• Instant live room peeking (less than a second to load the webapp into a live-view of a massive room with 20K users!!)
• Seamless onboarding thanks to using GitLab & GitHub for accounts
• Curated hierarchical room directory
• Magical creation of rooms on demand for every GitLab and GitHub project ever
• GitLab/GitHub activity as a first-class citizen in a room’s side-panel
• Excellent search-engine-friendly static content and archives
• KaTeX support for Maths communities
• Threads!

...and we promise to do everything in our power to preserve and honour these features at all costs and continue to give the Gitter community the experience they’ve come to know and love.

However: in the medium/long term, it’s simply not going to be efficient for the combined Element/Gitter team to split our efforts maintaining two high-profile Matrix clients. Our plan is instead to merge Gitter’s features into Element (or next generations of Element) itself and then - if and only if Element has achieved parity with Gitter based on the above list - we expect to upgrade the deployment on gitter.im to a Gitter-customised version of Element. The inevitable side-effect is that we’ll be adding new features to Element rather than Gitter going forwards.

In practice, the main outcome in the end should be Element having benefited massively from levelling up with Gitter - and Gitter benefiting massively from all the goodies which Element and Matrix brings, including:

• E2E Encryption
• Reactions
• Constantly improving native iOS & Android clients (which should be a welcome alternative to Gitter’s natives ones, which are already being deprecated)
• VoIP and conferencing
• All the alternative clients, bots, bridges and servers in Matrix
• The full open standard Matrix API
• Widgets (embedding webapps into rooms!)
• ...and of course participation in the wider decentralised Matrix network.

So, there you have it. It’s a new era for Gitter - and we look forward to reinvigorating Gitter’s communities over the coming months. We hope Gitter users will be blown away by the features arriving from Matrix… and we hope that Element users will be ecstatic with the performance and polish work that Gitter-parity will drive us towards. Imagine having guest access in Element that can launch and load a massive room in less than a second!

Finally, we would like to explicitly reassure the Gitter community again that we love and understand Gitter (it was one of the very first ever bridges we wrote for Matrix, for instance) - and we will be doing everything we can to not screw up our responsibility in looking after it. Please, please let us know if you have any concerns or if we ever fall short on this.

Any questions, come talk to us on #gitter:matrix.org - which is bridged with https://gitter.im/matrix-org/gitter. Exciting times ahead!

- Matthew, Amandine, and the whole Matrix, Element and Gitter teams.

Bonus update - The Changelog Interview!

Sid Sijbrandij (CEO at GitLab) and Matthew had a chance to sit down with The Changelog to talk about Gitter's Big Adventure - so tune in to hear the story first hand! Warning: contains non-ironic use of the word "synergy" :D

Hello all, and welcome to this week's addition of This Week in Matrix! Matrix is an open network protocol for secure, decentralized communication on the web.

My name is Andrew (aka anoa), and I'm a Synapse developer at Element. Thanks to Ben for letting me take over the reins of TWIM for this week! I'll actually be doing the same for next week as well, so please adjust your clocks to Anoa Nonstandard Time accordingly.

With that out of the way, let's jump right in!

Matrix Live 🎙

It's demos week again. This time around we've got the following lineup:

• Michael shows off all the new widget goodies coming to Element Web!
• Bruno shows off Hydrogen's new encrypted session backup support!
• Ismail details the new room creation flow on Element iOS!
• Jorik presents his work on revamping the UX of https://matrix.to that he completed for his second summer internship at Element!
• Hubert fixes another class of failure-to-decrypt messages edge case with Device Dehydration!
• Half-Shot closes off with the tale of Encrypted Bridges!

Don't forget that Matrix Live is also available in podcast form, if you're into that sort of thing. Search for "Matrix Live" wherever you get your podcasts.

Dept of Spec 📜

anoa (hey that's me!) told us:

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.

MSC Status

Merged MSCs:

• MSC1960: OpenID information exchange with widgets

MSCs in Final Comment Period:

• No MSCs are in FCP.

New MSCs:

• MSC2790: Modal widgets (acquiring user input from a widget)
• MSC2788: Room version 6 as the default room version
• MSC2787: Portable Identities
• MSC2785: Event notification attributes and actions
• [WIP] MSC2783: Homeserver Migration Data Format
• MSC2782: Pushers with the full event content
• MSC2781: Deprecate the (reply) fallbacks in the Matrix specification

Spec Core Team

In terms of Spec Core Team MSC focus for this week, we've been rather busy with implementation, so we'll be continuing on with the same focus as last week. As a reminder, that's MSC2414 (making reason and score optional on reports).

Dept of Servers 🏢

Dendrite / gomatrixserverlib

Dendrite is a next-generation homeserver written in Go.

Neil Alexander announced:

This week has mostly been spent trying to improve stability and to fix bugs ahead of the beta release, which is so far still planned to go ahead in the next two weeks.

Changes this week include:

• Room version 6 is now the default for newly created rooms

• Soft-fail of events that aren't allowed by the current room state is now implemented

• Support for configuring old_verify_keys has been added to the Dendrite config

• Correct formatting for signing key IDs is now enforced in the configuration

• Initial support for peeking over federation (MSC2444) is in progress and it "even works!" (thanks Matthew!)

• Federated joins will now continue being processed even if the client gives up on the join due to a HTTP timeout

• Backoff code has been refactored a bit more, and now correctly affects device list syncing

• /make_join now errors correctly if a federated user tries to join a room which all members have left

• A bug where a single user could start multiple simultaneous federated joins to the same room has been fixed

• Some initial (but unfinished) support for the /key/v2/query notary endpoint has been added

• Signature verification has been updated to not fail if the event origin field is missing (although it still requires a signature from the domain of the sender field)

• A number of places where we use SQL transactions have been updated with safe wrappers (thanks samcday!)

• A couple of error codes on invite endpoints for room version 6 JSON violations have been fixed

Spec compliance has improved slightly for federation:

• Client-Server APIs: 56%, same as last week

• Server-Server APIs: 77%, up from 74% last week

As always, if contributing to Dendrite sounds like something you would be interested in, please feel free to join us in #dendrite-dev:matrix.org ! There's also #dendrite:matrix.org

Synapse

Neil offered:

This week we released 1.20.0 (and 1.20.1) highlights include shadow banning support and including unread message counts in the sync response. This will help client developers and is a precursor to improving notification support.

We’ve also been looking at adding monitoring to get a better sense of which rooms are most expensive from a state resolution perspective, we also want a better way to track federation lag.

Up next we’ll move all background tasks away from the main process and try it out on matrix.org, we are hoping for a 10-15% saving in CPU. Event persistence sharding, specifically the new stream token format is on hold slightly while we work on a nasty race condition on start up of the event persister. We hope to get back to the main sharding project next week.

Conduit

Conduit is a Matrix homeserver written in Rust https://conduit.rs

timo announced:

• Respect SRV record when sending requests over federation
• Don't send new requests to servers if we are already waiting
• Implement get_missing_events
• Bug fixes and code cleanup

We also started work on a system that retries failed or blocked requests after some time.

Thanks to everyone who supports me on "Liberapay" (https://liberapay.com/timokoesters) or Bitcoin!

The Construct

Jason reported:

This week in Matrix concludes a summer of Construct where several transformative rounds of optimization took place. I'd like to talk about these achievements and why they are important for future directions.

Over the summer, Construct introduced new vector extensions to the project. As server software, our primary target is the datacenter which (for now) is dominated by x86 hardware. The latest features of x86 chips include AVX-2, AVX-512, and on-die GPU. These advancements are important because they help mitigate current limitations of hardware, such as the cubic relationship between a processor's frequency and its power consumption. These limitations mean that CPU cores aren't completing more cycles-per-second every iteration of their development -- they're not getting much faster.

The problem with threads is that they can introduce a lot of complications to a project. Construct is able to forego multi-threading as a network server because it is primarily IO-bound. The benefits to a single-thread design in both performance and simplicity cannot be overstated. This is why Construct stands to benefit the most from features which allow more work to be accomplished at every individual computing cycle.

Construct undertook several endeavors over the summer which directly leverage platforms featuring SSE, AVX, AVX-512, etc. As an added bonus, our approach is designed to effortlessly port to ARM's Scalable Vector Extensions (SVE) as it becomes available (inside datacenters too!). Our focus has been JSON, Unicode, and finally Base64. Detailed explanations for each of these will need to be discussed in their own posts, but in summary:

• Matrix specifies a canonical form of JSON which is not necessarily the same as the JSON the server receives. Therefor it is imperative that Construct is liberal with what JSON it accepts and correct with its transformation into canonical JSON. Over the summer, a portion of Construct's canonical transform received some optimization to go beyond the default method of character-by-character read-modify-write. As seen in [1], a streaming hardware-accelerated transform now processes at least 16 characters at a time (with more possible) without forward branching except for the parts where transformation needs to occur. One of these transformations is UTF-16 to UTF-8 surrogate conversion, which leads into the second endeavor.

• Construct introduced a completely branch-free Unicode toolset in [2]. Among the functions offered is a custom transform of UTF-16 surrogate pairs to UTF-8 sequences. Using the new vector registers, Construct can transform two UTF-16 surrogates in parallel to output two UTF-8 sequences, or two UTF-16 surrogates in a pair to output one UTF-8 sequence. Unfortunately, these specific functions were written at fixed vector widths, so more work needs to be done to really take advantage of the widest hardware. Each surrogate is 6 bytes, and a surrogate pair is 12 bytes; therefor we cannot make use of the last 4 bytes of a 16 byte vector. However, with a little more work this approach can be extended to a 64 byte vector, capable of decoding 5 surrogate pairs and 10 individual surrogates in parallel!

• Professor Daniel Lemire recently published a paper about fast Base64 from hardware acceleration in [3]. This approach is extremely elegant on the 64-byte-wide AVX-512 system. Prior to Construct's implementation of this in [4], the Boost library base64 encoding and decoding took roughly 20 and 25 cycles per character respectively. Our implementation on the same system, an old system with only SSE2 (not even AVX-512!) yields 5 and 6 cycles per character!

All of this helps lay a foundation for Construct to introduce Federated Media Rooms sometime in the future. Currently, Construct stores media in a separate database. Recently there's been work on a separate branch at [5] which stores actual file block inside events using Base64. It is for this reason sub-cycle and branchless JSON parsing and Base64 encoding is essential for maximum performance. The result is worthwhile, as the latency for querying the media database is slower than parsing and decoding the event content already in-hand.

That's all for today. Construct is available at https://github.com/matrix-construct/construct and don't forget to idle and perform #construct:zemos.net / #construct:maunium.net

Thanks!

1. https://github.com/matrix-construct/construct/blob/563f833ab325f27ff9e71af61af427fb02812f90/ircd/json.cc#L3483

2. https://github.com/matrix-construct/construct/blob/563f833ab325f27ff9e71af61af427fb02812f90/ircd/utf.cc

3. https://arxiv.org/abs/1910.05109

4. https://github.com/matrix-construct/construct/blob/563f833ab325f27ff9e71af61af427fb02812f90/ircd/b64.cc

5. https://github.com/jevolk/charybdis/tree/federated_media_rooms

It's really exciting to see Homeserver development ramping up from all angles, and nice that the protocol warts are slowly getting ironed out in the process.

Synapse Deployment 📥️

Kubernetes

Ananace told us:

Before I forget (more) about it, I pushed the 1.20.0 tag for my K8s-optimized container image as well as my Helm chart.

YunoHost

Pierre reported:

YunoHost is an operating system aiming for the simplest administration of a server, and therefore democratize self-hosting.

Synapse integration had been updated to 1.19.3 (1.20.0 available in branch testing)

Element Web integration had been updated to 1.7.5 (1.7.7 available in branch testing)

dacruz21/matrix-chart

Typo Kign announced:

Thanks to Arkaniad, v2.7.0 of my Matrix Helm chart for Kubernetes is released with support for exporting Prometheus metrics. It pairs well with the Synapse Grafana dashboard.

More Kubernetes

Ananace said:

And just pushed the 1.20.1 tag too for my K8s-optimized container image as well as my Helm chart.

Third-Party PowerPC and ARM64 support for Synapse

andreas announced:

The synapse docker image from AVENTER (https://www.aventer.biz), does support PowerPC (ppc64le) and ARM64 architecture now. But at the moment only under the docker tag "ppc". https://hub.docker.com/r/avhost/docker-matrix/tags?page=1&name=ppc We will be happy to get feedback.

As a Synapse developer, it's great to see the community making personal and enterprise Matrix deployments more accessible!

Dept of Clients 📱

FluffyChat

sorunome told us:

Fluffychat 0.19.1 has been released!

Features

• Implemented ignore list

• Jump to events in timeline: When tapping on a reply and when tapping a matrix.to link

• Display messages with up to 10 emotes or emoji bigger

• New design for the chat list and message bubbles

• Implement reactions

• Implement password change

• Implement deactivate user account

Fixes

• Timeline randomly resorting while more history is being fetched

• Automatically request history if the "load more" button is on the screen

Sorunome briefly mentioned afterwards that there is no 0.19.0 due to some accidental messed up tagging, and that it was easiest to just call the new version 0.19.1.

Element Web

Neil offered:

This week we put out a new release candidate 1.7.8-rc.1 highlights include:-

• Secure Backup has been moved out of the registration flow to a toast when you first encounter an E2EE room, which simplifies the new user experience

• Added options to hide various UI features when hosting a custom Element ...along with various smaller fixes.

Aside from that the work to improve the widget experience continues with modal widget next on the agenda.

Next week we will continue to improve widgets and add some more instrumentation into the app.

Hydrogen

Bruno said:

Released 0.1.0 (and 0.1.1) this week with read-only session backup enabled 🎉 Also doing more work to make Hydrogen work on IE11 on Windows 7 (it does already for Windows 10), Safari and other browsers where you get TransactionInactiveError during login, hope to release that soon.

I need to give Hydrogren a shot myself, the quick speeds and low RAM usage are really attractive.

Element-iOS

Manu offered:

This week, room settings have been updated with a new intermediate screen. The codebase saw the introduction of an AppCoordinator (in swift) which will help us to have a better control on navigation within the app AND to use swift from end to end.

Dept of SDKs and Frameworks 🧰

Polyjuice Client 🧙

uhoreg told us:

Polyjuice Client v0.3.0 has been released. This release includes many breaking changes. ("Breaking" in the sense of API changes, rather than the sync process suddenly failing to work, which was already featured in a previous release.) This release also includes many changes, such as the client managing more bookkeeping, detecting if it got logged out, and supporting more Matrix features. See the release notes for more information.

Igor

uhoreg reported:

Igor is a bot framework for Elixir. Igor v0.2.0 has been released. The main change is updating it to use Polyjuice Client 0.3.0.

Dept of Bots 🤖

Matrix-Architect

erdnaxeli announced:

Hi!

I present you a new project I've been working on for some time. It's a bot that allows you to use the admin API through matrix, by typing commands to the bot. The inspiration comes from the OperServ-like bots that allow IRC operators to administrate an IRC server.

The bot exposes some of the available admin APIs and aims to provide some more high level commands by combining different APIs. There is currently one "high level command", !room garbage-collect, which allow you to purge from your HS all rooms without local users.

The project is written in crystal and is my first project in this language. It should be stable in normal conditions, but don't hesitate to fill issues. A Docker image is provided for more convenience.

I hope you will find this project useful 🙂

https://github.com/erdnaxeli/matrix-architect

People administering their Synapse deployment through Matrix itself! How deep does the rabbit hole go?

Songwhip bot

Tulir reported:

benpa wanted a bot that replies with a songwhip.com link whenever someone sends a music link (youtube, spotify, apple music, etc), so I wrote a small maubot plugin to do that: https://github.com/maubot/songwhip

It's available at @songwhip:maunium.net

I'm also desperately in need of this for the 10 open.spotify.com links that get thrown at me every week. Thanks Tulir!

Cyberbot

jj reported:

There's a new bot! Cyberbot.

It supports E2EE and can be easily extended with Python plugins. Can be used e.g. for GitLab notifications, automated user invites, room creation, and of course can be programmed to react to any message posted in a room.

Dept of Interesting Projects 🛰️

matrix-gotify

sorunome said:

Soru made a new thing, matrix-gotify. It is a gotify plugin to receive matrix notifications. This means, you can now receive matrix push notifications on your android phone self-hosted without the need of google services! That is why putting the full event in the push notification is actually not a privacy leak in this case.

This plugin could also be used to receive push notifications on any other kind of device that gotify supports.

Please note that this is independent of any matrix client - you don't even need to run one to be able to use this!

I asked whether an OpenPush-like solution be built on top of this , and Sorunome responded that someone had already started working on just that! https://github.com/gotify/android/pull/115

This would allow other matrix clients on your phone to get their push notifications through your gotify client, instead of needing to run a process each. Great for battery life without proprietary Google blobs!

Homeservers on-the-go

js got Synapse running in a car on a German highway:

the setup is a cigarette lighter to 2x 230V converter, one being used to power the RockPro64, the other being used to power my notebook. My notebook is connected to the hotspot from my phone, while also being connected to a USB ethernet, the other end of which is plugged into the RockPro64.

The notebook then acts as a gateway, as well as SSHing to a tiny VPS with -R, to get a public port, and forwarding that traffic to the RockPro64.

Maybe one day we'll all have an embedded Synapse homeserver in our cars. P2P E2EE car comms anyone?

Matrix VoIP Tester

reivilibre announced:

https://github.com/matrix-org/voip-tester

I have been through the perils of setting up a TURN server and having VoIP continue to fail, whilst spending hours scratching my head and staring at Wireshark without getting anywhere. When I was given free reign over project choice last year during my internship, I chose to start a tool that tests your homeserver's VoIP configuration, hoping it would be useful to both me and the community at large.

It saw some progress but I never managed to iron out some of the 'last few things' or put a pretty front on it — until about 2 weeks ago, when I got some more time to do it.

There are known issues — such as the scores being very arbitrary, wrong and misleading; it crashes Chromium every time (on my machine) and it completely misbehaves in Brave Browser (on my machine). It may also not be the prettiest (but hopefully it is at least somewhat inoffensive). I hope to work on these annoying issues soon.

I have deployed a test instance at https://test.voip.librepush.net — it can be tried in a WebRTC-supporting browser (probably Firefox if you want half a chance). Please don't take it to heart if you get a Fail or Poor score — it's probably not your fault. :)

Oliver was interning with us at Element this Summer. Fixing up and releasing this VoIP tester was part of it, but he's still working on it in his spare time even though his internship has finished :)

Through this I found out that I forgot to turn on my turnserver on my homeserver again. Thanks Oliver!

Dept of Guides 🧭

German-Language Guide for Getting Start with Matrix

Samuel told us:

I started an article series about Matrix on my german blog. The goal is to make it easier for new users to get started with Matrix. https://blog.sp-codes.de/werde-teil-der-matrix-matrix-teil-1/

Guides like these are the essential entrypoints to a project for a lot of users. The more, in different languages, the merrier!

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

Rank	Hostname	Median MS
1	thomcat.rocks	896
2	matrix.vgorcum.com	965
3	conduit.rs	1226.5
4	neko.dev	1232
5	kif.rocks	1396
6	chatserver.ca	1533.5
7	jauriarts.org	1949.5
8	settgast.org	2328
9	vkane.cz	2419
10	conduit.nordgedanken.dev	2519

Ping Graphs by Timo™️

timo told us:

Here we look at how fast ping bots respond.

I changed the formatting of the plot a bit to make it more readable. Note that it is now using a log scale which allows us to see more data at the same time.

Non-Synapse Ping Room

Tulir reported:

Now that we have multiple somewhat federating Matrix homeserver implementations, I decided to make a ping room where all the echo bots are hosted on second-gen homeservers: #ping-no-synapse:maunium.net. Synapse users are still allowed to join and !ping, but all the pongs will come from non-Synapse servers.

Based on observations in that room so far, the new server implementations don't like eachother very much.

Rank	Hostname	Median MS
1	conduit.rs	110
2	pc.koesters.xyz:59003	162
3	cd.mau.dev	162.5
4	maunium.net	181
5	c.mau.dev	238
6	dendrite.neilalexander.dev	370
7	conduit.nordgedanken.dev	427
8	inferiorlattice.com	592
9	matrix.org	1294
10	grin.hu	3560.5

Final Thoughts 💭

• Magnus Svensson was left speechless by the amazing projects that we got to see this week.

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Synapse 1.20.0 is here!

Highlights of 1.20.0 include:-

• Shadow ban support.
• Unread message counts in the sync response to help our client developers, this is a precursor to improving notification support.
• No less than 28 async/await PRs, so we can finally share all the hard work.

Also take note that in a future release, we will be dropping support for accessing Synapse's Admin API using the /_matrix/client/* prefixes. More details follow in the changelog.

Get the new releases from any of the usual sources mentioned at https://github.com/matrix-org/synapse/blob/master/INSTALL.md. 1.20.0 is on github here, and 1.20.0rc4 is here.

The changelog for 1.20.0 is as follows:

Synapse 1.20.0 (2020-09-22)

No significant changes since v1.20.0rc5.

Removal warning

Historically, the Synapse Admin API has been accessible under the /_matrix/client/api/v1/admin, /_matrix/client/unstable/admin, /_matrix/client/r0/admin and /_synapse/admin prefixes. In a future release, we will be dropping support for accessing Synapse's Admin API using the /_matrix/client/* prefixes. This makes it easier for homeserver admins to lock down external access to the Admin API endpoints.

Synapse 1.20.0rc5 (2020-09-18)

In addition to the below, Synapse 1.20.0rc5 also includes the bug fix that was included in 1.19.3.

Features

• Add flags to the /versions endpoint for whether new rooms default to using E2EE. (#8343)

Bugfixes

• Fix rate limiting of federation /send requests. (#8342)
• Fix a longstanding bug where back pagination over federation could get stuck if it failed to handle a received event. (#8349)

Internal Changes

• Blacklist MSC2753 SyTests until it is implemented. (#8285)

Synapse 1.20.0rc4 (2020-09-16)

Synapse 1.20.0rc4 is identical to 1.20.0rc3, with the addition of the security fix that was included in 1.19.2.

Synapse 1.20.0rc3 (2020-09-11)

Bugfixes

• Fix a bug introduced in v1.20.0rc1 where the wrong exception was raised when invalid JSON data is encountered. (#8291)

Synapse 1.20.0rc2 (2020-09-09)

Bugfixes

• Fix a bug introduced in v1.20.0rc1 causing some features related to notifications to misbehave following the implementation of unread counts. (#8280)

Synapse 1.20.0rc1 (2020-09-08)

Removal warning

Some older clients used a disallowed character (:) in the client_secret parameter of various endpoints. The incorrect behaviour was allowed for backwards compatibility, but is now being removed from Synapse as most users have updated their client. Further context can be found at #6766.

Features

• Add an endpoint to query your shared rooms with another user as an implementation of MSC2666. (#7785)
• Iteratively encode JSON to avoid blocking the reactor. (#8013, #8116)
• Add support for shadow-banning users (ignoring any message send requests). (#8034, #8092, #8095, #8142, #8152, #8157, #8158, #8176)
• Use the default template file when its equivalent is not found in a custom template directory. (#8037, #8107, #8252)
• Add unread messages count to sync responses, as specified in MSC2654. (#8059, #8254, #8270, #8274)
• Optimise /federation/v1/user/devices/ API by only returning devices with encryption keys. (#8198)

Bugfixes

• Fix a memory leak by limiting the length of time that messages will be queued for a remote server that has been unreachable. (#7864)
• Fix Re-starting finished log context PUT-nnnn warning when event persistence failed. (#8081)
• Synapse now correctly enforces the valid characters in the client_secret parameter used in various endpoints. (#8101)
• Fix a bug introduced in v1.7.2 impacting message retention policies that would allow federated homeservers to dictate a retention period that's lower than the configured minimum allowed duration in the configuration file. (#8104)
• Fix a long-standing bug where invalid JSON would be accepted by Synapse. (#8106)
• Fix a bug introduced in Synapse v1.12.0 which could cause /sync requests to fail with a 404 if you had a very old outstanding room invite. (#8110)
• Return a proper error code when the rooms of an invalid group are requested. (#8129)
• Fix a bug which could cause a leaked postgres connection if synapse was set to daemonize. (#8131)
• Clarify the error code if a user tries to register with a numeric ID. This bug was introduced in v1.15.0. (#8135)
• Fix a bug where appservices with ratelimiting disabled would still be ratelimited when joining rooms. This bug was introduced in v1.19.0. (#8139)
• Fix logging in via OpenID Connect with a provider that uses integer user IDs. (#8190)
• Fix a longstanding bug where user directory updates could break when unexpected profile data was included in events. (#8223)
• Fix a longstanding bug where stats updates could break when unexpected profile data was included in events. (#8226)
• Fix slow start times for large servers by removing a table scan of the users table from startup code. (#8271)

Updates to the Docker image

• Fix builds of the Docker image on non-x86 platforms. (#8144)
• Added curl for healthcheck support and readme updates for the change. Contributed by @maquis196. (#8147)

Improved Documentation

• Link to matrix-synapse-rest-password-provider in the password provider documentation. (#8111)
• Updated documentation to note that Synapse does not follow HTTP 308 redirects due to an upstream library not supporting them. Contributed by Ryan Cole. (#8120)
• Explain better what GDPR-erased means when deactivating a user. (#8189)

Internal Changes

• Add filter name to the /users admin API, which filters by user ID or displayname. Contributed by Awesome Technologies Innovationslabor GmbH. (#7377, #8163)
• Reduce run times of some unit tests by advancing the reactor a fewer number of times. (#7757)
• Don't fail /submit_token requests on incorrect session ID if request_token_inhibit_3pid_errors is turned on. (#7991)
• Convert various parts of the codebase to async/await. (#8071, #8072, #8074, #8075, #8076, #8087, #8100, #8119, #8121, #8133, #8156, #8162, #8166, #8168, #8173, #8191, #8192, #8193, #8194, #8195, #8197, #8199, #8200, #8201, #8202, #8207, #8213, #8214)
• Remove some unused database functions. (#8085)
• Add type hints to various parts of the codebase. (#8090, #8127, #8187, #8241, #8140, #8183, #8232, #8235, #8237, #8244)
• Return the previous stream token if a non-member event is a duplicate. (#8093, #8112)
• Separate get_current_token into two since there are two different use cases for it. (#8113)
• Remove ChainedIdGenerator. (#8123)
• Reduce the amount of whitespace in JSON stored and sent in responses. (#8124)
• Update the test federation client to handle streaming responses. (#8130)
• Micro-optimisations to get_auth_chain_ids. (#8132)
• Refactor StreamIdGenerator and MultiWriterIdGenerator to have the same interface. (#8161)
• Add functions to MultiWriterIdGen used by events stream. (#8164, #8179)
• Fix tests that were broken due to the merge of 1.19.1. (#8167)
• Make SlavedIdTracker.advance have the same interface as MultiWriterIDGenerator. (#8171)
• Remove unused is_guest parameter from, and add safeguard to, MessageHandler.get_room_data. (#8174, #8181)
• Standardize the mypy configuration. (#8175)
• Refactor some of LoginRestServlet's helper methods, and move them to AuthHandler for easier reuse. (#8182)
• Fix wait_for_stream_position to allow multiple waiters on same stream ID. (#8196)
• Make MultiWriterIDGenerator work for streams that use negative values. (#8203)
• Refactor queries for device keys and cross-signatures. (#8204, #8205, #8222, #8224, #8225, #8231, #8233, #8234)
• Fix type hints for functions decorated with @cached. (#8240)
• Remove obsolete order field from federation send queues. (#8245)
• Stop sub-classing from object. (#8249)
• Add more logging to debug slow startup. (#8264)
• Do not attempt to upgrade database schema on worker processes. (#8266, #8276)

This is part of a series of reports on the six projects assigned to Matrix for Google Summer of Code 2020.

View project: Moving matrix-ircd to async/await and futures 0.3

---

The goal of this project was to port matrix-ircd from the outdated combinators style of futures 0.1 to the new-and-improved style of futures 0.3. As initially proposed, this greatly improves code readability and removes the convoluted compiler errors that were pervasive in futures 0.1

matrix-ircd

The matrix-ircd project functions as a bridge between two chat platforms: Internet relay chat (irc) and Matrix. matrix-ircd lets you use any standard IRC Client to communicate with Matrix chatrooms and direct messages.

Mentors

This project is mentored by Philipp Mandler (@phlmn) and Jonas Platte (@jplatte). I would like to thank them for the helpful advice and code reviews they have given me over the course of GSoC 2020.

Project Results

Since much of the code was written in late 2016, there were many portions of the code that were unidiomatic and produced compiler errors. The first step I made was to remove all compiler errors from cargo and clippy in #64.

Additional tests were then written in various parts of the codebase to ensure that the upgrade would produce the same results. These changes were made in #64 and #66.

Since the project master branch currently used a custom http implementation, I moved it to use hyper, a fast and correct http implementation that already supported async/await. Additionally, I moved the module that most utilized http, matrix, to async-await and the new hyper code. Not only did these changes shrink the code by ~700 lines, they also removed a lot of unnecessary complexity. These changes were made in #67.

The last large module remaining, irc, was then ported to irc, along with its dependencies in stream_fold.rs and its upstream user in the bridge module. The most exciting part of these changes was the removal of the futures 0.1 dependency. All code was now running in futures 0.3! These changes also moved away from tasked-futures which further improved code readability. This PR was in #71.

Since the bulk of the changes were now complete, I moved onto bug fixes. As @jplatte mentioned in #71, the current single threaded approach to the application was not ideal. In #72 I updated all code to be multithreaded-compatible.

Lastly, In #77 I included more tests to functions with heavy changes, added additional logging, removed unnecessary complexity that was introduced to keep code as "1:1" as possible, and added TLS support to hyper so that https would work. This patch also fixed a rather difficult bug regarding the irc TCP streams and the buffer they were reading into.

Remaining Work

Based on my personal testing there is no additional work to be done in the realm of updating to async/await. All tests pass and the IRC server and matrix bridge function as expected. @jplatte kindly announced in the This Week In Matrix blog that we will be conducting public testing of the async_await branch on github. Barring any issues the async/await code should be merged into the master branch in the next few weeks.

Pull request list

• https://github.com/matrix-org/matrix-ircd/pull/64
• https://github.com/matrix-org/matrix-ircd/pull/65
• https://github.com/matrix-org/matrix-ircd/pull/66
• https://github.com/matrix-org/matrix-ircd/pull/67
• https://github.com/matrix-org/matrix-ircd/pull/71
• https://github.com/matrix-org/matrix-ircd/pull/72
• https://github.com/matrix-org/matrix-ircd/pull/77

Matrix Live 🎙

Dept of Status of Matrix 🌡️

Element review by TechRadar

TechRadar have published a fairly thorough Element secure messenger review. This covers the Web, Android and iOS clients, and appears to use the matrix.org server for signup.

The review is very positive, awarding 4.5/5, and concludes:

The Element messenger platform scores highly for its approach to security and its commitment to decentralization, and it's definitely going to be of interest to businesses wanting control over their own chats – as well as plenty of individual users as well.

Dept of Spec 📜

Spec

anoa told us:

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.

MSC Status

Merged MSCs:

• No MSCs were merged this week.

MSCs in Final Comment Period:

• MSC1960: OpenID information exchange with widgets (merge)

New MSCs:

• MSC2778: Providing authentication method for appservice users

• MSC2775: Lazy loading over federation

• MSC2774: Expose the widget ID to the widget

• [WIP] MSC2773: Room kinds

Spec Core Team

In terms of Spec Core Team MSC focus for this week, MSC1960 has made it into FCP, so this week our focus is MSC2414.

Dept of Servers 🏢

Synapse

Neil announced:

This week we put out two point releases to fix critical bugs. At the very least ensure that you have upgraded to 1.19.2 which is a security release but you may as well go the whole hog and upgrade to the hot off the press 1.19.3. We will release 1.20.0 early next week.

Aside from that we continue to work on performance analysis and the sharding of the event persister continues - it’s proving to be a really tough job but we are getting there. We’ve also been making progress on the room knocking implementation

This week we say goodbye to Oliver (reivilibre) as a leaving present he fixed a long standing bug that prevented servers catching up after a federating outage and as we speak is furiously trying to finish a service to test TURN configuration. Thanks Oliver!

Dendrite / gomatrixserverlib

Dendrite is a next-generation homeserver written in Go

Neil Alexander said:

Not much has happened this week, as the Dendrite team have been taking some time off. However, we still have a couple of changes to report:

• Support for rejected events has been merged

• Work on soft-fail has begun (although not complete yet)

• A new mechanism for avoiding SQLite parameter limits was contributed (thanks HenrikSolver!)

Spec compliance is on the up:

• Client-Server APIs: 56%, same as last week

• Server-Server APIs: 74%, up from 71% last week

Conduit

Conduit is a Matrix homeserver written in Rust https://conduit.rs

timo announced:

Welcome back! This week we made some groundbreaking progress - this is not a joke, we found a bug in Synapse while breaking multiple Matrix rooms, causing the Matrix team a lot of unnecessary stress. Let me explain: The good news is that Conduit is starting to federate now. This means that you should be able to join all public rooms of the Matrix network and exchange messages. Note that Conduit does not do all the checks it should be doing yet making it stop sending messages from time to time as well as that advanced features like loading the history, syncing temporary data like read receipts or accepting invites are not implemented yet.

The bad news is that, while Synapse is happy to accept Conduit's messages when it is already part of the room, joining into one of the rooms Conduit servers are part of didn't work because of an event validation bug. The Matrix team did an excellent job at fixing this bug and releasing a Synapse patch the same day, but the damage has been done making a few rooms inaccessible to old Synapse servers. Thanks to everyone who supports me on "Liberapay" (https://liberapay.com/timokoesters) or Bitcoin!

Synapse Deployment 📥️

Synapse on NetBSD

js reported:

I ported Synapse + its dependencies to NetBSD and it is now in pkgsrc as chat/matrix-synapse. This means that Synapse is now easily installable on any operating system that supports pkgsrc (which is many, e.g. NetBSD, Linux, macOS, Solaris, AIX, Haiku, …). And while at it, I also ported mautrix-hangouts + dependencies (in pkgsrc as chat/mautrix-hangouts).

YunoHost

Pierre told us:

YunoHost is an operating system aiming for the simplest administration of a server, and therefore democratize self-hosting.

Synapse integration had been updated to 1.19.1 (1.19.2 available in branch testing)

Element Web integration had been updated to 1.7.5 (1.7.7 available in branch testing)

Kubernetes

Ananace offered:

Both image tags and a new chart version are hereby pushed for 1.19.2 for my Synapse image and Helm chart

then later

Just pushed the updates for the K8s-optimized Synapse image and chart for 1.19.3

Dept of Bridges 🌉

matrix-appservice-bridge hits 2.0.0ʳᶜ¹

Half-Shot said:

Hey bridge developer enthusiasts! Myself and ChristianP (bridge crew of matrix.org) have been working hard on freshening up the matrix-appservice-bridge library. It's no secret that it was using "classic" Javascript rather than ES6 syntax, was poorly documented in places and there were no types at all to make use of.

But that's all changed now, the 2.0.0-rc1 release brings fresh types, new convenience methods, a total swichover from Bluebird Promises to native Promises and much much more. If you are looking to quickly get set up writing bridges, it's never been easier.

Check out the slack-starter project to get your first taste of bridge excellence and see what you think.

As this is an RC, feedback is valuable and logging bugs helps us all. Please do so if you encounter any issues 😄

Bridge Encryption Support

Half-Shot reported:

Hi encryption fans. This week the bridge team has been churning away at adding support for encrypted rooms in the matrix-appservice-bridge library, to enable support in matrix-appservice-irc, matrix-appservice-slack, and anyone else using the library. This is probably the most exciting feature the library has seen in a long time!

The feature will be a simple config toggle, so existing bridges will have to do very little work to support it.

The first phase plans to use Pantalaimon with bridge users syncing like real users, but eventually the hope is that this will be all built into the library. Come check out this issue to keep track of progress.

Dept of Clients 📱

Element Web

Neil announced:

• Released 1.7.6 and 1.7.7 highlights include

  • Redesigned right panel where top right actions are now revealed via the i icon

  • Widgets can now be opened in the right panel

  • Widgets won't be visible in the Apps Drawer (top of timeline) by default (except Jitsi) - you need to pin them from the right panel

  • Widgets can now be resized

• A new version of matrix.to is now live you can check it out here, bug reports welcome >

• A new feature to defer e2ee set up until the user actually wishes to use an e2ee channel.

• A series of config flags to remove various features from the UI thereby simplifying the experience.

Mirage

miruka said:

0.6.3 and 0.6.4 were released this week, and I'm now working on desktop notifications support and push rules control.

Added

• Add a system tray icon.

  A left click will bring up the Mirage window, middle will quit the application and right will show a menu with these

  options.

• Add a closeMimizesToTray setting to the config file, defaults to false.

  Controls whether closing the Mirage window will leave it running in the system tray, or fully quit the application.

• Add a discrete read marker indicator to messages, shows how many people

  have this event as their last seen one in the room.
  A way to see who read the message and when will be added in the future.

• Themes: add chat.message.localEcho and chat.message.readCounter color properties

• Add a zoom setting, defaults to 1.0

• Add a lexicalRoomSorting setting, to sort rooms by their name instead of

  recent activity.
  A restart is needed to apply changes to this setting.

Changed

• Restrict Mirage to a single instance per config folder, trying to launch a

  new window will instead focus the existing one.
  The MIRAGE_CONFIG_DIR and MIRAGE_DATA_DIR environment variables can be

  set to run different "profiles" in parallel.

• Reduce the visible lag when opening a chat page, switching rooms should be

  a lot smoother

• When using the focusPreviousMessage and focusNextMessage keybinds, if no

  message is focused and the timeline has been scrolled up, focus the message in the center of the view instead of returning to the

  bottom of the timeline and focusing the last one.

• Don't re-center the room list on clicks by default.

  This prevents the list from jumping around every time a room is selected.
  The previous behavior can be restored with the new centerRoomListOnClick

  setting.

• Show a better terminal error message than "Component is not ready" when the

  window creation fails, giving details on what went wrong in the code

• If an account's access token is invalid (e.g. our session was signed out

  externally), say so with a popup and cleanly remove it from the UI, instead of spamming the user with errors.

• Rename message context menu option "Debug this event" to just "Debug"

• Unify up/down and (shift+)Tab navigation for the account Sessions page

• Changes to the UI scale/zoom via keybinds are now persisted across restarts

• Themes: uiScale is now bound to window.settings.zoom.

  This change is necessary to keep the zoom keybinds working.

Fixed

• A bunch of stuff, see the full changelog for 0.6.3 and 0.6.4

Element Android 1.0.7

benoit offered:

Element Android 1.0.7 has been released (still stuck in Google pipes writing those lines), and will be quickly followed by 1.0.8 to fix a problem with cross signing verification.

We are working to implement search of messages in a room and we keep on fighting bugs and improving performance of the application.

We are also iterating on the home screen, to improve user experience, and try to satisfy users with a few rooms as well as users with several hundreds of rooms.

Element-iOS

Manu said:

This week, we released 1.0.12 with several bug fixes including the one for background crashes due to PushKit

Hydrogen

Bruno told us:

Released 0.037 on Monday with some polish for E2EE:

• remove outdated devices when querying

• update room summary with decrypted events, also when retrying decryption

• various crash protections

• keep megolm session with earliest index when receiving a room key we already know about

• add new icon

• fix issue where olm would be loaded twice when login failed

• show decryption errors in timeline

• show encryption enabled tile in timeline

The rest of the week was implementing session backup, which I hoped would be ready to release today but it needs some more polish unfortunately. In the meantime, here's a GIF teaser:

FluffyChat

Thank you to Krille, who told us:

FluffyChat version 0.18.0 is out now in F-Droid, Testflight and soon the PlayStore:

Features

• Added translations: Armenian, Turkish, Chinese (Simplified), Estonian
• Url-ify matrix identifiers
• Use server-side generated thumbnails in cleartext rooms
• Add option to send images in their original resolution
• Add additional confirmation for sending files & share intents
• Add option to opt-in to report issues / crashes to sentry
• Write keys to online key backup, fully implementing online key backup

Changes

• Tapping links, pills, etc. now does stuff
• Better handling of sending messages in bad network
• Better recovery of "keys not cached"
• E2EE is enabled again

Fixes:

• Various html rendering and url-ifying fixes
• Added support for blurhashes
• Image viewer now eventually displays the original image, not only the thumbnail

Dept of SDKs and Frameworks 🧰

Hemppa

Hemppa the bot is a multi-purpose Matrix bot for writing new functionality easily with Python.

Cos said:

Hemppa gained a small but big feature for making fediverse better: per-room Mastodon tooting support. This means that you can set up a Mastodon account in your room and anyone with moderator rights can toot to that account. This way you and your roommates can super easily send toots to the world directly from Matrix. Mastodon (unlike the big alternative) supports RSS feeds out of the box so it's very easy to subscribe to users and hashtags with stock RSS bot. Toot #hemppathebot if you like it! Another new contribution worth mentioning is welcome which can send welcome messages to users registering on the server or joining a room. https://github.com/vranki/hemppa

Ruby

Ananace said:

Another week, another Ruby SDK release. This time adding a little fix to avoid duplicate state events being passed twice to the application if identical ones arrive in both state and timeline, also adds a global state event handler and moves room event handling to not depend on room instances themselves.

Asked how much the SDK is used in production, Ananace said:

a little, we've hooked Matrix into our server orchestration / configuration management system TheForeman - which is a Rails application. Also got a colleague who's doing a bot to linkify internal ticket IDs.

Ruma

iinuwa told us:

Thanks to @q-b and @jtescher we have implemented more of the federation endpoints (bringing us up to about 85% completion) and every (one) endpoint of the Push Gateway API!

We fixed a bug where federation endpoints expected access tokens instead of server signatures for authentication. In the future, we may add an API that requires signatures on requests.

Finally, we've also continued progress on the non-exhaustive type updates for backwards compatibility; we expect to finish this within the coming weeks.

New Contributors

• @jtescher, not new but forgot to mention earlier!

• @williewillus

Dept of Bots 🤖

Matrix Reminder Bot

anoa announced:

v0.2.0 of matrix-reminder-bot has been released! This releases includes lots of bugfixes, updates and polishing. Find the list below:

Features

• Better support for command prefixes other than the default !.

• Just writing !silence now silences the currently active alarm.

• The bot will now print the correct syntax of a command if the user fails to follow it.

• The bot will reply to events if it cannot decrypt them, as well as offer helpful tips that both the user and bot operator can try to fix things.

Bugfixes

• Timezones. They should finally work correctly! ...no 100% guarantees though.

• Alarms were a bit broken. They're fixed now.

• Fix commands with formatting and newlines not being picked up by the bot.

• Fix non-latin characters preventing reminders from being deleted.

Internal changes

• Add a dev-optimised Dockerfile for quicker iteration during development.

• Better wording revolving alarms. They're just reminders that alarm repeatedly when they go off.

• Log why the bot is unable to start.

• Don't print "Unknown help topic" in case the user is trying to ask another bot for help.

• The config dict is now a singleton.

• Type hints everywhere!

Additionally, the minimum Python version is now 3.6. matrix-reminder-bot is made with nio-template.

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

Rank	Hostname	Median MS
1	fairydust.space	412
2	matrix-dev.kapsi.fi	1308
3	utzutzutz.net	1519
4	blob.cat	1521
5	r23s.eu	1915.5
6	conduit.rs	2149
7	maescool.be	2759
8	halogen.city	2774.5
9	uraziel.de	3253
10	kapsi.fi	3713

Dept of Ping Extra:

timo announced:

Normally the ping shows how long other servers take to receive the sending-server's message. Some servers receive it quickly, some take hours. All this flows into the statistic of the sending server. I wrote a small program that's the other way around. Instead of measuring the sending-server's ping, it associates each measurement with an echo-bot (instead of the sending server), so this plot shows which maubots are the fastest to respond (it's probably not accurate at all, because most of the pings were started on conduit servers this week, but it's interesting nevertheless):

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

This is part of a series of reports on the six projects assigned to Matrix for Google Summer of Code 2020.

View project: Ruma procedural macro refactoring and more

---

Wow the summer has flown by, it feels like just yesterday I was learning how to rebase and what exactly it is Ruma does. I exaggerate slightly, but it is a big library with lots of public API surface. I have learned more in the last few months than in two years of school. I have been able to observe and participate in a project with a community growing around it, been a part of discussions about design and best practices, given and received numerous code reviews as well as learned the process of addressing the feedback, and working from a specification. In short, this has been an amazing opportunity to gain experience in all the things that are hard to obtain in a classroom.

My project goal was to improve the existing macros in ruma-events-macros and ruma-api-macros. It became clear early on that this would include some major API changes and that improving the macros as they were was pointless without also moving to a new public API. While improving the durability and readability of the macro code I also rewrote entire sections to accommodate the new design.

A quick overview of the Matrix protocol for reference: a client sends content that is interpreted by the server as events. The server distributes those events out to other clients and other servers (the server case is known as federation). Ruma groups these events by kind Message, State, Ephemeral, ToDevice and Basic which are represented as generic structs (StateEvent<C>). Each event kind needs to be able to hold many different content types, for state events there is room creation, room name, and membership events to name a few. Using the macros, enums are generated to represent all state event possibilities, so a variant for membership, room name, etc. These types exist to support the core API request and response types for each endpoint that is defined by the Matrix specification.

GSoC Starts and I start with ruma-events

• remove old ruma-events macros
• start work for event content derive, move to generic Event structs
• collect all content types into enum
• add more types, making sure the macro works
• work on Event custom macro derive
• remove raw modules and FromRaw trait
• add each events enum
• add trybuild tests for ruma-events

More work on ruma-events, now a part of the new mono-repo

• deserialization optimization for Any*Event, and benchmarking to verify
• reverting the removal of enum_enum macro, learning a lot of git along the way
• it took me awhile but I got the hang of it, only one review issue!
• add accessor methods to the enums, making them much nicer to use

Work on ruma-client-api

• refactoring the Api::to_tokens code helped me get my head around this crate
• notice my commit messages are starting to looking better
• not everything works out...
• but you get it eventually
• deep dive into rust features, I learned a lot about them

Back to ruma-events

• a ruma events and client-api rename
• add code to generate redact method in event_enum!
• adding to existing code without breaking everything is hard
• converting to and from the rusty way
• I love writing proc_macros, this was a fun one! Adding lifetimes in the req/resp macro

Continuing maintenance

• quality of life stuff, the day to day work of a maintainer
• allow the internal macros to be used outside of the ruma crate more easily
• adding small needed features to the macros
• good ol' fashioned closing issues!

One of my personal goals was to become more familiar with git. With the help of my mentor I now feel more confident using this tool that is so essential to developers. I became fairly adept at merging, rebasing, and navigating all the headaches that come with that. I learned plenty of new commands. A few highlights: cherry-pick and specific uses of reset to avoid copy-pasting fixes and adding more commits. I used the reset command to craft good commits, splitting work into appropriate chunks. I am glad that I had the opportunity to hone my git skills. I feel like I have accomplished my goal and then some!

I am proud of the work that I have done: Being part of moving ruma-events much closer to the 0.22 release and creating macros to generate types specific to the Matrix specification. Working with the community that has grown around Ruma has been rewarding and I plan on sticking around.

Synapse 1.19.2 is a security patch. All federating instances should upgrade immediately.

Today we are releasing Synapse 1.19.2, which is a security patch release containing a fix to encountering invalid events over federation. We are also putting out a fourth release candidate for the upcoming Synapse 1.20.0 release with the same fix.

The bug prevents affected Synapse instances from joining rooms with invalid events. Server administrators running federating instances are strongly encouraged to update as soon as possible.

Those on Synapse 1.19.1 or earlier should upgrade to Synapse 1.19.2, while those who are running a release candidate of Synapse 1.20.0 should upgrade to 1.20.0rc4.

Get the new releases from any of the usual sources mentioned at https://github.com/matrix-org/synapse/blob/master/INSTALL.md. 1.19.2 is on github here, and 1.20.0rc4 is here.

The changelog for 1.19.2 is as follows:

Synapse 1.19.2 (2020-09-16)

Due to the issue below server admins are encouraged to upgrade as soon as possible. Bugfixes

• Fix joining rooms over federation that include malformed events. (#8324)