How do you implement interoperability in a DMA world?

29.03.2022 17:57 — General Matthew Hodgson
Last update: 29.03.2022 09:23

With last week’s revelation that the EU Digital Markets Act will require tech gatekeepers (companies valued at over $75B or with over $7.5B of turnover) to open their communication APIs for the purposes of interoperability, there’s been a lot of speculation on what this could mean in practice, To try to ground the conversation a bit, we’ve had a go at outlining some concrete proposals for how it could work.

However, before we jump in, we should review how the DMA has come to pass.

What’s driven the DMA?

Today’s gatekeepers all began with a great product, which got more and more popular until it grew to such a size that one of the biggest reasons to use the service is not necessarily the product any more, but the benefits of being able to talk to a large network of users. This rapidly becomes anti-competitive, however: the user becomes locked into the network and can’t switch even if they want to. Even when people have a really good reason to move provider (e.g. WhatsApp’s terms of use changing to share user data with Facebook, Apple doing a 180 on end-to-end encrypting iCloud backups, or Telegram not actually being end-to-end encrypted), in practice hardly anything changes - because the users are socially obligated to keep using the service in order to talk to the rest of the users on it.

As a result, it’s literally harmful to the users. Even if a new service launches with a shiny new feature, there is enormous inertia that must be overcome for users to switch, thanks to the pull of their existing network - and even if they do, they’ll just end up with their conversations haphazardly fragmented over multiple apps. This isn’t accepted for email; it isn’t accepted for the phone network; it isn’t accepted on the Internet itself - and it shouldn’t be accepted for messaging apps either.

Similarly: the closed networks of today’s gatekeepers put a completely arbitrary limit on how users can extend and enrich their own conversations. On email, if you want to use a fancy new client like Superhuman - you can. If you want to hook up a digital assistant or translation service to help you wrangle your email - you can. If you want to hook up your emails to a CRM to help you run your business - you can. But with today’s gatekeepers, you have literally no control: you’re completely at the mercy of the service provider - and for something like WhatsApp or iMessage the options are limited at best.

Finally - all the users’ conversation metadata for that service (who talks to who, and when) ends up centralised in the gatekeepers’ databases, which then become an incredibly valuable and sensitive treasure trove, at risk of abuse. And if the service provider identifies users by phone number, the user is forced to disclose their phone number (a deeply sensitive personal identifier) to participate, whether they want to or not. Meanwhile the user is massively incentivised not to move away: effectively they are held hostage by the pull of the service’s network of users.

So, the DMA exists as a strategy to improve the situation for users and service providers alike by building a healthier dynamic ecosystem for communication apps; encouraging products to win by producing the best quality product rather than the biggest network. To quote Cédric O (Secretary of State for the Digital Sector of France), the strategy of the legislation came from Washington advice to address the anticompetitive behaviour of the gatekeepers “not by breaking them up… but by breaking them open.” By requiring the gatekeepers to open their APIs, the door has at last been opened to give users the option to pick whatever service they prefer to use, to choose who they trust with their data and control their conversations as they wish - without losing the ability to talk to their wider audience.

However, something as groundbreaking as this is never going to be completely straightforward. Of course while some basic use cases (i.e. non-E2EE chat) are easy to implement, they initially may not have a UX as smooth as a closed network which has ingested all your address book; and other use cases(eg E2EE support) may require some compromises at first. It’s up to the industry to figure out how to make the most of that challenge, and how to do it in a way which minimises the impact on privacy - especially for end-to-end encrypted services.

What problems need to be solved?

We’ve already written about this from a Matrix perspective, but to recap - the main challenge is the trade-off between interoperability and privacy for gatekeepers who provide end-to-end encryption, which at a rough estimate means: WhatsApp, iMessage, secret chats in Facebook Messenger, and Google Messages. The problem is that even with open APIs which correctly expose the end-to-end encrypted semantics (as DMA requires), the point where you interoperate with a different system inevitably means that you’ll have to re-encrypt the messages for that system, unless they speak precisely the same protocol - and by definition you end up trusting the different system to keep the messages safe. Therefore this increases the attack surface on the conversations, putting the end-to-end encryption at risk.

Alex Stamos (ex-CISO at Facebook) said that “WhatsApp rolling out mandatory end-to-end encryption was the largest improvement in communications privacy in human history” – and we agree. Guaranteed end-to-end encrypted conversations on WhatsApp is amazing, and should be protected at all costs. If users are talking to other users on WhatsApp (or any set of users communicating within the same E2EE messenger), E2EE should and must be maintained - and there is nothing in the DMA which says otherwise.

But what if the user consciously wants to prioritise interoperability over encryption? What if the user wants to hook their WhatsApp messages into a CRM, or run them through a machine translation service, or try to start a migration to an alternative service because they don’t trust Meta? Should privacy really come so spectacularly at the expense of user freedom?

We also have the problem of figuring out how to reference users on other platforms. Say that I want to talk to a user with a given phone number, but they’re not on my platform - how do I locate them? What if my platform only knows about phone numbers, but you’re trying to talk to a user on a platform which uses a different format for identifiers?

Finally, we have the problem of mitigating abuse: opening up APIs makes it easier for bad actors to try to spam or phish or otherwise abuse users within the gatekeepers. There are going to have to be changes in anti-abuse services/software, and some signals that the gatekeeper platforms currently use are going to go away or be less useful, but that doesn't mean the whole thing is intractable. It will require changes and innovative thinking, but we’ve been making steady progress (e.g. the work done by Element’s trust and safety team). Meanwhile, the gatekeepers already have massive anti-abuse systems in place to handle the billions of users within their walled gardens, and unofficial APIs are already widespread: adding official APIs does not change the landscape significantly (assuming interoperability is implemented in such a way that the existing anti-abuse mechanisms still apply).

In the past, gatekeepers dismissed the effort of interop as not being worthwhile - after all, the default course of action is to build a walled garden, and having built one, the temptation is to try to trap as many users as possible. It was also not always clear that there were services worth interoperating with (thanks to the chilling effects of the gatekeepers themselves, in terms of stifling innovation for communication startups). Nowadays this situation has fundamentally changed however: there is a vibrant ecosystem of open communication startups out there, and a huge appetite to build a vibrant open ecosystem for interoperable communication, but like the open web itself.

What are the requirements?

Before going further in considering solutions, we need to review the actual requirements of the DMA. Our best understanding at this point is that the DMA will mandate that:

  • Gatekeepers will have to provide open and documented APIs to their services, on request, in order to facilitate interoperability (i.e. so that other services can communicate with their users).
  • These APIs must preserve the same level of end-to-end encryption (if any) to remote users as is available to local users.
  • This applies to 1:1 messaging and file transfer in the short term, and group messaging, file-transfer, 1:1 VoIP and group VoIP in the longer term.

So, what could this actually look like?

The DMA legislation deliberately doesn’t focus on implementation, instead letting the industry figure out how this could actually work in practice. There are many different possible approaches, and so from our point of view as Matrix we’ve tried to sketch out some options to make the discussion more concrete. Please note these are preliminary thoughts, and are far from perfect - but hopefully useful as a starting point for discussion.

Finding Bob

Imagine that you have a user Alice on an existing gatekeeper, which we’ll call AliceChat, who runs an E2EE messaging service which identifies users using phone numbers. Say that they want to start a 1-to-1 conversation with Bob, who doesn’t use AliceChat, but Alice knows he is a keen user of BobChat. Today, you’d have no choice but to send them an SMS and nag them to join AliceChat (sucks to be them if they don’t want to use that service, or if they’re unable to for whatever reason - e.g. their platform isn’t supported, or their government has blocked access, etc), or join BobChat yourself.


However, imagine if instead the gatekeeper app had a user experience where the app prompted you to talk to the user via a different platform instead. It’d be no different to your operating system prompting you to pick which app to use to open a given file extension (rather than the OS vendor hardcoding it to one of their own apps - another win for user rights led by the EU!).


Now, the simplest approach in the short term would be for each gatekeeper to pre-provision a set of options of possible alternative networks. (The DMA says that, on request, other service providers can ask to have access to the gatekeeper’s APIs for the purposes of interoperability, so the gatekeeper knows who the alternative networks may be). “Bob is not on AliceChat - do you want to try to reach him instead on BobChat, CharlieChat, DaveChat (etc)”.

Much like users can configure their preferred applications for file extensions in an operating system today, users would also be able to add their own preferred service providers - simply specifying their domain name.

Connecting to Bob

Now, AliceChat itself needs to figure out how to query the remote service provider to see if Bob actually exists there. Given the DMA requires that gatekeepers provide open APIs with the same level of security to remote users as their local ones using today’s private APIs - and very deliberately doesn’t mandate specific protocols for interoperability - they will need to locate a bridge which can connect to the other system.

In this thought experiment, the bridge used would be up to the destination provider. For instance, bobchat.com could announce that AliceChat users should connect to it via alicechat-bridge.bobchat.com using the AliceChat protocol(or matrix-bridge.bobchat.com via Matrix or xmpp-bridge.bobchat.com via XMPP) by a simple HTTP API or even a .well-known URL. Users might also be able to override the bridge used to connect to them (e.g. to point instead at a client-side bridge), and could sign the advertisement to prove that it hadn’t been tampered with.

AliceChat would then connect to the discovered bridge using AliceChat’s vendor-specific newly opened API, and would then effectively treat Bob as if they were a real AliceChat user and client to all intents and purposes. In other words, Bob would effectively be a “ghost user” on AliceChat, and subject to all their existing anti-abuse mechanisms.

Meanwhile, the other side of the bridge converts through to whatever the target system is - be that XMPP, Matrix, a different proprietary API, etc. For Matrix, it’d be chatting away to a homeserver via the Application Service API (using End-to-Bridge Encryption via MSC3202). It’s also worth noting that the target might not even be a bridge - it could be a system which already natively speaks AliceChat’s end-to-end encrypted API, thus preserving end-to-end encryption without any need to re-encrypt. It’s also worth noting that while historically bridges have had a bad reputation as being a second class (often a second class afterthought), Matrix has shown that by considering them as a first class citizen and really focusing on mapping the highest common denominator between services rather than lowest common denominator, it’s possible for them to work transparently in practice. Beeper is a great example of Matrix bridging being used for real in the wild (rather amusingly they just shipped emoji reactions for WhatsApp on iOS via their WhatsApp<->Matrix bridge before WhatsApp themselves did…)

Architecturally, it could look like this:

Or, more likely (given a dedicated bridge between two proprietary services would be a bit of a special case, and you’d have to solve the dilemma of who hosts the bridge), both services could run a bridge to a common open standard protocol like Matrix or XMPP instead (thus immediately enabling interoperability with everyone else connected to that network):

Please note that while these examples show server-side bridges, in practice it would be infinitely preferable to use client-side bridges when connecting to E2EE services - meaning that decrypted message data would only ever be exposed on the client (which obviously has access to the decrypted data already). Client-side bridges are currently complicated by OS limits on background tasks and push notification semantics (on mobile, at least), but one could envisage a scenario where you install a little stub AliceChat client on your phone which auths you with AliceChat and then sits in the background receiving messages and bridging them through to Matrix or XMPP, like this:

Another possible architecture could be for the E2EE gatekeeper to expose their open APIs on the clients, rather than the server. DMA allows this, to the best of our knowledge - and would allow other apps on the device to access the message data locally (with appropriate authorisation, of course) - effectively doing a form of realtime data liberation from the closed service to an open system, looking something like this:

Finally, it's worth noting that when peer-to-peer decentralised protocols like P2P Matrix enter production, clientside bridges could bridge directly into a local communication server running on the handset - thus avoiding metadata being exposed on Matrix or XMPP servers providing a common language between the service providers.

Locating users

Now, the above describes the simplest and most naive directory lookup system imaginable - the problem of deciding which provider to use to connect to each user is shouldered by the users. This isn’t that unreasonable - after all, users may have strong feelings about what providers to use to talk to a given user. Alice might be quite happy to talk to Bob via BobChat, but might be very deliberately avoiding talking to him on DaveChat, for whatever ominous reasons.

However, it’s likely in future we will also see other directory services appear in order to map phone numbers (or other identities) to providers - whether these piggyback on top of existing identity providers (gatekeepers, DNS, telcos, SSO providers, governments) or are decentralised through some other mechanism. For instance, Bob could send AliceChat a blinded proof that he authorises them to automatically route traffic to him over at BobChat, with BobChat maintaining a matching proof that Bob is who he claims to be (having gone through BobChat’s auth process) - and the proofs could be linked using a temporary key such that Bob doesn’t even need to maintain a long-term one. (Thanks to James Monaghan for suggesting this one!)

Another alternative to having the user decide where to find each other could be to use a decentralised Keybase-style identity system to track verified mappings of identities (phone numbers, email addresses etc) through to service providers - perhaps something like IDX might fit the bill? While this decentralised identity lookups have historically been a hard problem, there is a lot of promising work happening in this space currently and the future looks promising.

Talking to Bob

Meanwhile, Alice still needs to talk to Bob. As already discussed, unless everyone speaks the same end-to-end encrypted protocol (be it Matrix, WhatsApp or anything else), we inevitably have a trade-off here between interoperability and privacy if Bob is not on the same system as Alice (assuming AliceChat is end-to-end encrypted) - and we will need to clearly warn Alice that the conversation is no longer end-to-end encrypted:


To be clear: right now, today, if Bob were on AliceChat, he could be copy-pasting all your messages into (say) Google Translate in a frantic effort to workaround the fact that his closed E2EE chat platform has no way to do machine translation. However, in a DMA world, Bob could legitimately loop a translation bot into the conversation… and Alice would be warned that the conversation was no longer secure (given the data is now being bridged over to Google).

This is a clear improvement in user experience and transparency. Likewise, if I’m talking to a bridged user today on one of these platforms, I have no way of telling that they have chosen to prioritise interop over E2EE - which is frankly terrifying. If I’m talking to someone on WhatsApp today I blindly assume that they are E2EE as they are on the same platform - and if they’re using an unofficial app or bridge, I have no way to tell. Whereas in a DMA world, you would expect the gatekeeper to transparently expose it.

If anything, this is good news for the gatekeeper in that it consciously advertises a big selling point for them: that for full E2EE, users need to talk to other users in the same walled garden (unless of course the platform speaks the same protocol). No more need for bus shelter adverts to remind everywhere that WhatsApp is E2EE - instead they can remind the user every time they talk to someone outside the walled garden!

Just to spell it out: the DMA does not require or encourage any reduction in end-to-end encryption for WhatsApp or similar: full end-to-end encryption will still be there for users in the same platform, including through to users on custom clients (assuming the gatekeeper doesn’t flex and turn it off for other reasons).

Obviously, this flow only considers the simple case of Alice inviting Bob. The flow is of course symmetrical for Bob inviting Alice; AliceChat will need to advertise bridges which can be used to connect to its users. As Bob pops up from BobChat, the bridge would use AliceChat’s newly open APIs to provision a user for him, authing him as per any other user (thus ensuring that AliceChat doesn’t need to have trusted BobChat to have authenticated the user). The bridge then sends/receives messages on Bob’s behalf within AliceChat.

Group communication

This is all very well for 1:1 chats - which are the initial scope of the DMA. However, over the coming years, we expect group chats to also be in scope. The good news is that the same general architecture works for group chats too. We need a better source of identity though: AliceChat can’t possibly independently authenticate all the new users which might be joining via group conversations on other servers (especially if they join indirectly via another server). This means adopting one of the decentralised identity lookup approaches outlined earlier to determine whether Charlie on CharlieChat is the real Charlie or an imposter.

Another problem which emerges with group chats which span multiple service providers is that of indirect routing, especially if the links between the providers use different protocols. What if AliceChat has a direct bridge to BobChat (a bit like AIM and ICQ both spoke OSCAR), BobChat and CharlieChat are connected by Matrix bridges, and AliceChat and CharlieChat are connected via XMPP bridges? We need a way for the bridges to decide who forwards traffic for each network, and who bridges the users for which network. If they were all on Matrix or XMPP this would happen automatically, but with mixed protocols we’d probably have to extend the lookup protocol to establish a spanning tree for each conversation to prevent forwarding loops.

Here’s a deliberately twisty example to illustrate the above thought experiment:

There is also a risk of bridge proliferation here - in the worst case, every service would have to source bridges to directly connect to every other service who came along, creating a nightmarish n-by-m problem. But in practice, we expect direct proprietary-to-proprietary bridges to be rare: instead, we already have open standard communication protocols like Matrix and XMPP which provide a common language between bridges - so in practice, you could just end up in a world where each service has to find a them-to-Matrix or them-to-XMPP bridge (which could be run by them, or whatever trusted party they delegate to).

Conclusion

A mesh of bridges which connect together the open APIs of proprietary vendors by converting them into open standards may seem unwieldy at first - but it’s precisely the sort of ductwork which links both phone networks and the Internet together in practice. As long as the bridging provides for highest common denominator fidelity at the best impedance ratio, then it’s conceptually no different to converting circuit switched phone calls to VoIP, or wired to wireless Ethernet, or any of the other bridges which we take entirely for granted in our lives thanks to their transparency.

Meanwhile, while this means a bit more user interface in the communication apps in order to select networks and warn about trustedness, the benefits to users are enormous as they put the user squarely back in control of their conversations. And the UX will improve as the tech evolves.

The bottom line is, we should not be scared of interoperability, just because we’ve grown used to a broken world where nothing can interconnect. There are tractable ways to solve it in a way that empowers and informs the user - and the DMA has now given the industry the opportunity to demonstrate that it can work.

Interoperability without sacrificing privacy: Matrix and the DMA

25.03.2022 22:39 — General Matthew Hodgson
Last update: 25.03.2022 18:01

Yesterday the EU Parliament & Council agreed on the contents of the Digital Markets Act - new legislation from the EU intended to limit anticompetitive behaviour from tech “gatekeepers”, i.e. big tech companies (those with market share larger than €75B or with more than €7.5B a year of revenue).

This is absolutely landmark legislation, where the EU has decided not to break the gatekeepers up in order to create a more competitive marketplace - but instead to “break them open”. This is unbelievably good news for the open Internet, as it is obligating the gatekeepers to provide open APIs for their communication services. In other words: no longer will the tech giants be able to arbitrarily lock their users inside their walled gardens - there will be a legal requirement for them to expose APIs to other services.

While the formal outcomes of yesterday’s agreement haven’t been published yet (beyond this press release), our understanding is that the DMA will mandate:

  • Gatekeepers will have to provide open and documented APIs to their services, on request, in order to facilitate interoperability (i.e. so that other services can communicate with their users).
  • These APIs must preserve the same level of end-to-end encryption (if any) to remote users as is available to local users.
  • This applies to 1:1 messaging and file transfer in the short term, and group messaging, file-transfer, 1:1 VoIP and group VoIP in the longer term.

This is the best possible outcome imaginable for the open internet. Never again will a big tech company be able to hold their users hostage in a walled garden, or arbitrarily close down or sabotage their APIs.

So, what’s the catch?

Since the DMA announcement on Thursday, there’s been quite a lot of yelling from some very experienced voices that mandating interoperability via open APIs is going to irrevocably undermine end-to-end encrypted messengers like WhatsApp. This seems to mainly be born out of a concern that the DMA is somehow trying to subvert end-to-end encryption, despite the fact that the DMA explicitly mandates that the APIs must expose the same level of security, including end-to-end encryption, that local users are using. (N.B. Signal doesn’t qualify as a gatekeeper, so none of this is relevant to Signal).

So, for WhatsApp, it means that the API would expose both the message-passing interface as well as the key management APIs required to interoperate with WhatsApp using your own end-to-end-encrypted WhatsApp client - E2EE would be preserved.

However, this does mean that if you were to actively interoperate between providers (e.g. if Matrix turned up and asked WhatsApp, post DMA, to expose an API we could use to write bridges against), then that bridge would need to convert between WhatsApp’s E2EE’d payloads and Matrix’s E2EE’d payloads. (Even though both WhatsApp and Matrix use the Double Ratchet, the actual payloads within the encryption are completely different and would need to be converted). Therefore such a bridge has to re-encrypt the traffic - which means that the plaintext is exposed on the bridge, putting it at risk and breaking the end-to-end encryption guarantee.

There are solutions to this, however:

  • We could run the bridge somewhere relatively safe - e.g. the user’s client. There’s a bunch of work going on already in Matrix to run clientside bridges, so that your laptop or phone effectively maintains a connection over to iMessage or WhatsApp or whatever as if it were logged in… but then relays the messages into Matrix once re-encrypted. By decentralising the bridges and spreading them around the internet, you avoid them becoming a single honeypot that bad actors might look to attack: instead it becomes more a question of endpoint compromise (which is already a risk today).

  • The gatekeeper could switch to a decentralised end-to-end encrypted protocol like Matrix to preserve end-to-end encryption throughout. This is obviously significant work on the gatekeeper’s side, but we shouldn’t rule it out. For instance, making the transition for a non-encrypted service is impressively little work, as we proved with Gitter. (We’d ideally need to figure out decentralised/federated identity-lookup first though, to avoid switching from one centralised identity database to another).

  • Worst case, we could flag to the user that their conversation is insecure (the chat equivalent of a scary TLS certificate warning). Honestly, this is something communication apps (including Matrix-based ones!) should be doing anyway: as a user you should be able to tell what 3rd parties (bots, integrations etc) have been added to a given conversation. Adding this sort of semantic actually opens up a much richer set of communication interactions, by giving the user the flexibility over who to trust with their data, even if it breaks the platonic ideal of pure E2E encryption.

On balance, we think that the benefits of mandating open APIs outweigh the risks that someone is going to run a vulnerable large-scale bridge and undermine everyone’s E2EE. It’s better to have the option to be able to get at your data in the first place than be held hostage in a walled garden.

Other considerations

One other complaint which has come up a bunch is around speed of innovation: the idea that WhatsApp or similar would be seriously slowed down by having to (effectively) maintain stable documented federation APIs, and figure out how to do backwards compatibility for new features. It’s true that this will take a bit more effort (similar to how adding GDPR compliance takes some effort), but the ends make it more than worth it. Plus, if the rag-tag Matrix ecosystem can do it, it doesn’t seem unreasonable to think that a $600B company like Meta can figure it out too...

Another consideration is that it might make it too easy to build malicious 3rd party clients - e.g. building your own "special" version of Signal which connects to the official service, but deliberately or otherwise has security flaws. The fact is that we're already in this position though: there are illicit alternative clients flying around all over the place, and the onus is on the app stores to protect their users from installing malware. This isn't reason to throw the baby of interoperability out with the bathwater of bootleg clients.

The final complaint is about moderation and abuse: while open APIs are good news for consumer choice, they can also be used by spammers, phishers and other miscreants to cause problems for the users within the gatekeeper. Much like a mediaeval citadel; opening up your walled garden means that both good and bad people can turn up. And much like real life, this is a solvable problem, even if it’s unfortunate: the benefits of free trade massively outweigh the downsides of having to police strangers more effectively. Frankly, moderation and anti-abuse approaches on the Internet today are infamously broken, with centralised moderation by gatekeepers producing increasingly erratic results. By opening the walled gardens, we are forcing a much-needed opportunity to review how to empower users and admins to filter unwanted content on their own terms. There’s a recent write-up of the proposed approach for Matrix at https://element.io/blog/moderation-needs-a-radical-change/, which outlines one strategy - but there are many others. Honestly, having to improve moderation tooling is a worthwhile price to pay for the benefits of open APIs.

So, there you have it. Hopefully you’ll agree that the benefits here outweigh the risks: without open APIs we wouldn't even have the option to talk about interoperability. We should be celebrating a new dawn for open access, rather than fearing that the sky is falling and this is nefarious attempt to undermine end-to-end encryption.

This Week in Matrix 2022-03-25

25.03.2022 00:00 — This Week in Matrix Thib

Matrix Live 🎙

Dept of Status of Matrix 🌡️

Thib announces

You might have been following our past efforts on making sure the DMA was meaningful. Those efforts are being rewarded, and Amandine covers in a blog post what this means for Element and other businesses in Europe. Matthew covers what this means for Matrix and end to end ecrypted services.

Dept of Spec 📜

Andrew Morgan (anoa) announces

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.

MSC Status

New MSCs:

MSCs in Proposed Final Comment Period:

MSCs in Final Comment Period:

Merged MSCs:

  • No MSCs were merged this week.

Spec Updates

Mostly continued review of MSCs, while implementation work continues on in the background.

Random MSC of the Week

The random MSC of the week is... MSC2499: Fixes for Well-known URIs!

This MSC proposes some much-desired practical and usability fixes to the "Well-Known" discovery method for Matrix homeservers. This is part of the system that enables homeserver's to "delegate" their homeserver address (your Matrix ID is @alice:example.com, but your homeserver is listening for requests on homeserver.example.com).

Well-known for homeserver delegation was introduced a number of years ago, and since then some friction has arisen from various implementations. This MSC aims to address a collection of them!

Dept of Servers 🏢

Synapse (website)

Synapse is a Matrix homeserver implementation developed by the matrix.org core team

Brendan Abolivier says

Hey everyone! This week we've released Synapse 1.55, which includes a bunch of new features, performance improvements and bugfix. If you're using Mjolnir to moderate your rooms, and/or synctl to manage your homeserver, this release also introduces a few changes you definitely want to be aware of, head over to the announcement on the matrix.org blog for more!

Yesterday, we noticed a compatibility issue with a newly released version of Jinja, which is the tool we use in Synapse to render e-mail and web templates. We've quickly put out Synapse 1.55.2 to address this - so if you were unable to start Synapse because of it just update and you should be fine 🙂 Note that this doesn't apply to deployments of Synapse using the matrixdotorg/synapse Docker image or Debian packages from packages.matrix.org since they are already using a compatible version of Jinja.

Apart from that, work continues towards integrating Poetry with Synapse, which should prevent this kind of issues from happening in the future, and on improving room join times.

Dendrite (website)

Second generation Matrix homeserver

neilalexander announces

We've just released Dendrite 0.7.0, which has quite a lengthy list of changes:

  • The roomserver input API will now queue all events into NATS, which provides better crash resilience
  • The roomserver input API now configures per-room consumers, which should use less memory
  • Canonical aliases can now be added and removed
  • MSC2946 Spaces Summary now works correctly, both locally and over federation
  • Healthcheck endpoints are now available at:
    • /_dendrite/monitor/up, which will return 200 when Dendrite is ready to accept requests
    • /_dendrite/monitor/health, which will return 200 if healthy and 503 if degraded for some reason
  • The X-Matrix federation authorisation header now includes a destination field, as per MSC3383
  • The /sync endpoint now uses less memory by only ranging state for rooms that the user has participated in
  • The /messages endpoint now accepts stream positions in both the from and to parameters
  • Dendrite will now log a warning at startup if the file descriptor limit is set too low
  • The federation client will now attempt to use HTTP/2 if available
  • The federation client will now attempt to resume TLS sessions if possible, to reduce handshake overheads
  • The built-in NATS Server has been updated to version 2.7.4
  • NATS streams that don't match the desired configuration will now be recreated automatically
  • When performing a graceful shutdown, Dendrite will now wait for NATS Server to shutdown completely, which should avoid some corruption of data on-disk
  • The create-account tool has seen a number of improvements, will now ask for passwords automatically
  • The /sync endpoint will no longer lose state events when truncating the timeline for history visibility
  • The /context endpoint now works correctly with lazy_load_members
  • The /directory/list/room/{roomID} endpoint now correctly reports whether a room is published in the server room directory or not
  • Some bugs around appservice username validation have been fixed
  • Roomserver output messages are no longer unnecessarily inflated by state events, which should reduce the number of NATS message size errors
  • Stream IDs for device list updates are now always 64-bit, which should fix some problems when running Dendrite on a 32-bit system
  • Purging room state in the sync API has been fixed after a faulty database query was corrected
  • The federation client will now release host records for remote destinations after 5 minutes instead of holding them in memory forever
  • Remote media requests will now correctly return an error if the file cannot be found or downloaded
  • A panic in the media API that could happen when the remote file doesn't exist has been fixed
  • Various bugs around membership state and invites have been fixed
  • The memberships table will now be correctly updated when rejecting a federated invite
  • The client API and appservice API will now access the user database using the user API rather than accessing the database directly

In addition, there are some minor changes for Docker users:

  • Docker images are now published to GitHub Container Register as well as Docker Hub — they can be found here
  • Docker images are now being automatically generated for the main branch by CI and will be available on the :main tag

As always, feel free to join us in #dendrite:matrix.org for Dendrite-related discussion!

Dept of Bridges 🌉

matrix-appservice-kakaotalk (website)

A Matrix-KakaoTalk puppeting bridge.

Fair reports

This week brings yet more bugfixes & stability to DMs: now, there should be no trouble sending messages into a KakaoTalk DM that hasn't been active for a few days.

The other major improvements are:

  • A command for listing your KakaoTalk friends list, appropriately named list-friends
  • Receiving images from KakaoTalk

I had hoped to finish support for sending images from Matrix by today, but I couldn't quite crack it in time--either KakaoTalk changed their API for sending media messages, or I'm doing something wrong 🙃

Nonetheless, the bridge is rapidly approaching a state of being usable!


Discussion: #matrix-appservice-kakaotalk:miscworks.net Issue page: https://src.miscworks.net/fair/matrix-appservice-kakaotalk/issues

Dept of Clients 📱

Syphon (website)

Chat with your privacy and freedom intact

0x1a8510f2 announces

Hi all 👋.

A little over a year has passed since our last update, but we're not dead! On the contrary, Syphon has seen some significant improvements over the last year both in terms of features and project structure; the project is more alive than ever before, and we're really excited to share some updates!

Core Team

Syphon now has a core team, consisting of @ereio:matrix.org (creator of Syphon), @ed:geraghty.london, @dnisbetjones:mozilla.org, and @0x1a8510f2:0x1a8510f2.space (me)! With some of the workload now being shared, we're hoping Syphon will grow even faster with less time needed to review and merge PRs, triage and fix bugs, evaluate and implement features and answer support requests.

Cross-platform Availability

Syphon has been steadily expanding its list of supported platforms over the past year:

  • @ed:geraghty.london worked hard to bring Syphon to Windows with full feature parity, including (currently unofficial) Windows builds of Syphon.
  • @btdmaster:asra.gr has created an AUR package of Syphon for Arch Linux users.
  • I've been busy creating a flatpak for Syphon which is available on Flathub.
  • Regular and official ARM64 builds are also planned soon, with the first manual one already released. Once the CI infrastructure is properly set up, these will be included for every release and the flatpak will receive ARM64 support too!

Releases

Between 0.1.7 (21 Mar 2021) and 0.2.12 (23 Mar 2022), 21 releases were published. Of course, too much changed to list everything, but here are some highlights:

  • Multi-account support 🎉
  • Media messages 🎉
  • Markdown support 🎉
  • Message editing 🎉
  • Key import and export 🎉
  • App lock with encryption 🎉
  • HTTP proxy support 🎉
  • Theme tweaks
  • 3pid auth support
  • EXIF data removal from sent images
  • System brightness setting
  • Various customisation options
  • (Unencrypted) chat message search
  • Ability to deactivate an account
  • Countless new translations
  • Massive performance improvements
  • Numerous bug fixes (including encryption and SSO)
  • Lots of refactoring and cleanup

Many of these would not have been possible without our awesome contributors.

Translations

We have also received many contributions in the form of translations and Syphon now supports, at least partially, 26 languages. Of these, 6 languages are fully or almost fully translated. To deal with all the incoming translations, there is now also an official Weblate page for Syphon.

All of these translations are submitted by volunteers and we're really grateful for their contributions in making Syphon more accessible to an international audience.

Syphon Space

We also now have an official Syphon space which we use for support, development discussion or just friendly offtopic chat. Come join us if you like at #syphon-space:matrix.org!

SchildiChat (website)

SchildiChat is a fork of Element that focuses on UI changes such as message bubbles and a unified chat list for both direct messages and groups, which is a more familiar approach to users of other popular instant messengers.

SpiritCroc reports

SchildiChat-Android now has more styles of message bubbles! So if you prefer the Schildi-bubbles over the Element ones (which you can use too in SchildiChat if you want), you can now also select between bubbles with or without tail, and select some more round bubbles than what we had previously, which was a highly requested feature!

If you wonder what makes SchildiChat special except for the different design now that Element has bubbles as well: it's hard to describe in a couple of short sentences, as it's mostly in the details. So if you are interested in a list of things that we did on Android, I wrote up some aspects here. Note however that this list is usually not really kept up-to-date, and there might already be some things that are now outdated since I added them. However, it might give you some general ideas of what SchildiChat has to offer :)

Nheko (website)

Desktop client for Matrix using Qt and C++17.

Nico reports

Beep, boop, v0.9.3 is out. It's mostly bugfixes and small improvements, changelog below:

Highlights

  • New upload UX
    • Queue multiple uploads by pasting or dragging multiple files.
    • Videos will now properly have a thumbnail as well as images.
    • Duration, width and height is now also properly included so that clients can resize appropriately.
    • Thumbnails are excluded if they are bigger than the original image. (tastytea)
  • Improvements for mobile devices (Malte E)
    • You should now be able to scroll by touching anywhere with no random dead zones.
    • Preedit text can now be used in a completer and is properly sent
    • If an input method is active, pressing Enter will not send the current message.

Features

  • Optionally always open videos and images in an external program. (math)

Improvements

  • Build macOS releases against Qt 5.15.3 to resolve missing spaces after some punctuation.
  • Send the shortcode as the body for stickers without a body.
  • Elide long usernames in the timeline. (Malte E)
  • Cleanup the reply popup. (Malte E)
  • Use standard buttons where possible. (tastytea)
  • Various improvements to the bubble layout. (Malte E)
  • Enable online key backup by default.
  • Update the bundled gstreamer in our Flatpaks.

Translations

  • Indonesian (Linerly)
  • Estonian (Priit)
  • Finnish (Priit)
  • Esperanto (Tirifto)

Bugfixes

  • Fix hovering the action menu.
  • Try to avoid using unknown UIA flows.
  • Don't Components actively in use.
  • Fix screensharing.
  • Fix device id when doing SSO logins.

Downloads should be live now, flatpak might take a while to publish still :3

Fractal (website)

Matrix messaging app for GNOME written in Rust.

Julian Sparber reports

Hello folks, quick update on what major things happened in Fractal-next the last two month. The most exciting addition is definitely the SSO support we merged this week and therefore we could close a 2 years old issue.

Timeline

  • You can now send files via drag-n-drop and via the file send button to a room. It also includes a nice preview for Images.
  • The timeline now shows audio messages with a small inline player.
  • Fractal-next lets you now remove messages you sent

Session verification

  • During first login, Fractal checks if the user hasn't started session verification from another client before offering to start a new one
  • The QrCode scanning is now spec compliant, and asks for user's confirmation after scanning.
  • We dropped screenshot support for QrCode scanning, since it makes the UX worse without adding any real benefit.

Room details

  • The room details show now the members of the room including the power level

Login

  • Fractal-Next now supports SSO 🎉️
  • We implemented auto-discovery of the homeserver via .well-known

FluffyChat (website)

Krille Fear announces

Sorry that I was a little bit silent in the last weeks. So much stuff to do...

If you haven't heard it yet (because I have not made it that much public yet) there are experimental video calls in FluffyChat now. You need to enable them first under Settings > Chat and then you can try them out. They should be fully compatible with the Element video calls. But be aware that they are very unstable at the moment and may let your app crash.

FluffyChat 1.3.1 has been released

This release contains a lot of updated translations and bugfixes. I'm very excited about the new keyboard shortcuts from TheOneWithTheBrain. Also in your stories you can now pick the background color and the size of the picture. This is still a little bit experimental but I will share of course update the stories MSC asap. Thanks to all contributors and translators.

Changelog:

  • Allow app to be moved to external storage (Marcel)
  • Translated using Weblate (Arabic) (Mads Louis)
  • Translated using Weblate (Basque) (Sorunome)
  • Translated using Weblate (Basque) (—X—)
  • Translated using Weblate (Chinese (Simplified)) (Eric)
  • Translated using Weblate (Czech) (Sorunome)
  • Translated using Weblate (Dutch) (Jelv)
  • Translated using Weblate (English) (Raatty)
  • Translated using Weblate (French) (Anne Onyme 017)
  • Translated using Weblate (Galician) (Xosé M)
  • Translated using Weblate (German) (Maciej Krüger)
  • Translated using Weblate (Indonesian) (Linerly)
  • Translated using Weblate (Irish) (Graeme Power)
  • Translated using Weblate (Persian) (Anastázius Darián)
  • Translated using Weblate (Russian) (Nikita Epifanov)
  • Translated using Weblate (Swedish) (Joaquim Homrighausen)
  • Translated using Weblate (Turkish) (Oğuz Ersen)
  • Translated using Weblate (Ukrainian) (Ihor Hordiichuk)
  • Update proguard rules to a more modern setup (MTRNord)
  • chore: Minor story viewer fixes (Krille Fear)
  • chore: Remove story line count and make answering to stories online (Krille Fear)
  • chore: Update dependencies (Dependency Update Bot)
  • design: Make pinned events use less vertical space (Krille Fear)
  • feat: Extended stories (Krille Fear)
  • feat: Restrict map zoom to tile server capabilities (Marcel)
  • feat: implement keyboard shortcuts (TheOneWithTheBraid)
  • fix: Build on macOS (Krille Fear)
  • fix: Emojipicker issues (Krille Fear)
  • fix: Hide redacted stories (Krille Fear)
  • fix: Mark story as read (Krille Fear)
  • fix: Open room from notification click produces errors (Krille Fear)
  • fix: SSO on Android 12 (Krille Fear)
  • fix: Send read receipts on all taps (Krille Fear)
  • fix: make fluffy usable at 720 px wide (Raatty)
  • fix: Add forgotten sendOnEnter (Krille Fear)
  • refactor: Switch to just audio for playing sounds (Krille Fear)

Ement.el (website)

Matrix client for emacs

alphapapa announces

[[https://github.com/alphapapa/ement.el][Ement.el]], a Matrix client for Emacs, has learned to create new rooms and invite users to rooms. Feedback is appreciated; see our chat room, [[https://matrix.to/#/#ement.el:matrix.org][#ement.el:matrix.org]], or the issue tracker on the repo.

Element Web/Desktop (website)

Secure and independent communication, connected via Matrix. Come talk with us in #element-web:matrix.org!

kittykat announces

  • Threads will be released behind a labs flag in the next release and enabled by default in the Release Candidate (RC) from 5th April

  • If you’re using an older version of Synapse (<v1.55.0) you might experience compatibility problems with stable prefixes. After upgrading Element to v1.18.0 unstable threads will be moved to the main room timeline

  • Groups have been deleted on develop. It has also landed alongside other fairly major changes so please definitely report issues if you see them

  • Continuing work to remove skinning from the application. This is a fairly major change to how everything works under the hood, so when it lands please report any issues with the app as most will be subtle and therefore might be missed.

    • Currently expected to land on or about April 5th
    • Component replacement will still be possible (and this will be documented)
  • In labs (you can enable labs in settings on develop.element.io or on Nightly):

    • Thread list is now ordered by last reply

    • Fixes for the room list counter

    • Last stretch of threads acceptance testing before releasing to beta

    • Iteration to the new search dialog to integrate people & public rooms search into the new experience.

Element iOS (website)

Secure and independent communication for iOS, connected via Matrix. Come talk with us in #element-ios:matrix.org!

Doug says

  • Element 1.8.7 was released on the App Store after delays with the review: we’ve made ignoring users easier and now suggest a Matrix ID when using “Sign in With Apple” to resolve the issue.
  • Space creation and management will be available in the next release, due next week.
  • We had some issues with publishing releases to TestFlight so the latest release candidates haven’t been updated publicly - we have been testing them internally so releases will not be delayed

Element Android (website)

Secure and independent communication for Android, connected via Matrix. Come talk with us in #element-android:matrix.org!

adam reports

Our current in progress release 1.4.6+ is running a little behind schedule, however when it does land it'll contain:

  • Fixes in the timeline for missing messages, inconsistent read marker, decryption crashes and the room icon looking like donut
  • Improved Threads via MSC3440
  • The ability to share a user specified location

For the forks, a heads up that we've updated our development branch Kotlin version to 1.6.0 which involved overhauling a lot of our when statements

Dept of Ops 🛠

Synatainer (website)

Synapse Maintenance Container – Docker container with tools for synapse & postgres database maintenance

saces reports

Synatainer v0.3.1 has been released

New since v0.2.0

  • add a db-shell and scripts to reset the state compressor
  • add options for passing purge/keep room lists

Start it without command and let it do its magic :)

What it does by default:

  • daily:
    • purge all rooms without local members
    • run the state autocompressor (small setting 500/100)
  • weekly:
    • delete old remote media (>90 days)
    • delete old message history from public joinable rooms (>180 days)
  • monthly:
    • run the state autocompressor (big setting 1500/300)
    • vacuum the database

Source: https://gitlab.com/mb-saces/synatainer

Room: #synatainer:c-base.org

Compose example: https://gitlab.com/mb-saces/synatainer/-/snippets/2264828


If you are looking for a docker container with just the auto compressor for linux amd64/arm64 in it, her you go: https://gitlab.com/mb-saces/rust-synapse-compress-state

Dept of Bots 🤖

Mjölnir (website)

The moderation bot for Matrix

Gnuxie announces

We have released Mjölnir v1.4.1 (via v1.4.0)

Which in addition to fixes, includes a few new tools that you might find useful:

  • a protection for detecting & measuring federation lag in your rooms
  • bugfixes for the synapse antispam module (in particular squashing a bug that was removing users from the profile directory when blocking was disabled)
  • a new message length limiter for the antispam module
  • a new command to grab admin permissions from rooms being administrated by users on your homeserver
  • a command to show when users in a given room have joined
  • another command to filter through recent room joiners by the time they joined.
  • Improvements that make writing protections easier.

Don't forget you can join our development room #mjolnir:matrix.org if you have any have any questions for setting up and running Mjölnir!

Dept of Events and Talks 🗣️

HarHarLinks announces

don't miss #otwsu:matrix.org on March 30th!

Thib adds

The theme of the episode is "analytics and privacy". We will have guests from the awesome non-profit Exodus Privacy to shed some light on analytics: what can your apps know about you and how you can get better informed.

Nad from Element will give us the perspective from the other side: as a vendor, does it make sense to use analytics? Are there better alternatives? Is there a way to do it right?

Join us in #otwsu and on matrix.org/open-tech-will-save-us

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server.

#ping:maunium.net

Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1neko.dev359
2envs.net482.5
3aria-net.org522
4jeroenhd.nl753.5
5g3la.de761
6kohlernet.de761
7tardis.omaelse.de1074.5
8tchncs.de1274.5
9matrix.ring-0.net1602
10asra.gr1629.5

#ping-no-synapse:maunium.net

Join #ping-no-synapse:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1matrix.sum7.eu318.5
2conduit.cyberdi.sk463
3cry.is514
4rustybever.be580.5
5beckmeyer.us798
6matrix.awesomesheep48.me1096
7dendrite.neilalexander.dev1164
8dendrite.beckmeyer.us3950

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Synapse 1.55 released

22.03.2022 00:00 — Releases Brendan Abolivier

Hi all, Synapse 1.55 is out! Let's see the main talking points of this new release.

Update: After the initial release of Synapse 1.55, the developers of a third-party tool (Jinja, which is the tool Synapse uses to render email and web templates) released a new version of their project which proved incompatible with Synapse. To address this issue, we have released Synapse 1.55.2.

Deployments of Synapse using the matrixdotorg/synapse Docker image or Debian packages from packages.matrix.org are not affected as they are already using a compatible version of the tool.

It's worth noting that the work we are doing to integrate Poetry into Synapse (more on that a bit further in this post) will, once completed, prevent this kind of issues from happening.

Removing support for Mjolnir 1.3.1 and older

Administrators of large homeservers and communities will already be familiar with Mjolnir, which is a tool designed to help them moderate a large number of rooms in an efficient way. It includes a bot as well as a Synapse module to better interface with the homeserver.

Due to a change in how we manage some of Synapse's internal utilities, this release of Synapse breaks compatibility with Mjolnir versions 1.3.1 and older. Homeserver administrators using Mjolnir and upgrading to Synapse 1.55 should make sure they're running Mjolnir version 1.3.2 or later.

See the upgrade notes for more information.

Moving synctl

synctl is a tool provided as part of Synapse to run and manage your instance and its workers (if any).

As part of our work to integrate Poetry in Synapse (which in turn will enable a lot of cool things, such as reproducible builds), we have recently moved the way we package this tool. As of this release, homeserver administrators should stop invoking it using its direct path (e.g. /path/to/synctl start), but should instead call it directly, e.g. synctl start.

This means homeserver administrators using synctl must make sure that the tool is in their PATH. This is automatically done when installing and upgrading Synapse using the matrixdotorg/synapse Docker image or Debian packages from packages.matrix.org. When installing from a wheel, sdist, or PyPI, an executable is created in your Python installation's bin directory.

See the upgrade notes for more information.

Everything else

This release also introduces new module callbacks, allowing homeserver administrators to more efficiently review which users are able to deactivate users and shutdown rooms. More information on that in Synapse's documentation.

We have also started experimenting with using the native Python asyncio event loop in Synapse which, if successful, would make it easier to use building blocks from the Python ecosystem when adding new features to Synapse.

See the full changelog for a complete list of changes in this release. Also please have a look at the upgrade notes for this version.

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including (in no particular order) Dirk Klimpel, Beeper, and ~creme.

This Week in Matrix 2022-03-18

18.03.2022 18:56 — This Week in Matrix Thib
Last update: 18.03.2022 18:52

Matrix Live 🎙

This week my guest is Adam who created the minimalistic Android client your family wants… by accident. And Adam has more ideas for the future.

Dept of Spec 📜

TravisR says

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.

MSC Status

Merged MSCs:

  • No MSCs were merged this week.

MSCs in Final Comment Period:

  • No MSCs are in FCP.

New MSCs:

Spec Core Team

In terms of Spec Core Team MSC focus for this week, we've mostly been working on our existing projects. Unfortunately this means little to report this week, but if there's something that needs out attention then please mention it in #sct-office:matrix.org 🙂

Dept of P2P 👥

Pinecone (website)

Peer-to-peer overlay routing for the Matrix ecosystem

neilalexander reports

Things have been relatively quiet on the P2P front recently as we have been working on fixing bugs in Dendrite, but we will be releasing build 94 of the P2P demos for iOS and for Android soon, which feature new fair queuing in Pinecone. This should help to reduce congestion from single nodes. It also includes all of the latest Dendrite updates.

  • Android: https://drive.google.com/drive/folders/1uK_BcHGiAYHkN6OAA7P73obAquwShRg5
  • iOS: https://testflight.apple.com/join/Tgh2MEk6

The demos are still very rough around the edges — join us in #p2p:matrix.org for discussion!

Dept of Servers 🏢

Synapse (website)

Synapse is a Matrix homeserver implementation developed by the matrix.org core team

dmr says

Hello everyone. It's been a quieter week for the Synapse team---many of us are away this week.

We released Synapse 1.55.0rc1 yesterday, a little later than usual. A full release is still planned for the coming Tuesday (22nd March). Please note that this release breaks compatibility with Mjolnir 1.3.1 and earlier. Administrators using Mjolnir should ensure it is upgraded before upgrading Synapse to 1.55.0rc1.

Otherwise, the release candidate includes:

Please feel free to try it out and let us know how you get on: either in the #synapse:matrix.org or on GitHub if you encounter any unexplained problems. As ever, many thanks to all contributors involved.

Behind the scenes we've been working on a few longer-term projects:

  • Continued support for threading (MSC3440) and the mechanism which supports it, relations (MSC2674).
  • Managing Synapse's dependencies using poetry to formally lock requirements across all our debian and docker releases.
  • Performance investigations: we're looking at getting better profiling information; and seeing if we can reclaim CPU cycles by cancelling processing when a requester terminates their request early.

Hopefully we'll see these plans come to fruition over the coming weeks.

Dept of Bridges 🌉

matrix-appservice-kakaotalk (website)

A Matrix-KakaoTalk puppeting bridge.

Fair says

This week brings some bugfixes to DMs, and general improvements to the way channels are synced on startup.

I've also written some setup instructions to assist anyone wanting to try this bridge out for themselves!

As for bugs, I just stumbled upon a big one just now: if a KakaoTalk channel hasn't had activity for a few days, sending a message in its Matrix portal may crash. This happens because of the KakaoTalk API's apparent eagerness in forgetting about inactive channels (IIRC content that's a few weeks old is deleted from their servers, for privacy reasons), and the likely fix is for the bridge to use the right API calls to trigger those channels back into an active state before trying to interact with them.

Once that's fixed, the next thing I'll get out of the way (for the sake of completeness) is the web-based login interface, which all of the other mautrix-python bridges support (and is quite nice for privacy-conscious users that would rather send their KakaoTalk passwords directly to the bridge, instead of through a Matrix homeserver).

After that, I'll work on contact searching & inviting contacts to group chats or DMs.


Discussion: #matrix-appservice-kakaotalk:miscworks.net Issue page: https://src.miscworks.net/fair/matrix-appservice-kakaotalk/issues

Dept of Clients 📱

Nheko (website)

Desktop client for Matrix using Qt and C++17.

Nico announces

After our release last week, of course all the bugreports came out of the woodworks. As such we fixed issues with device ids being set incorrectly after SSO, screensharing, overlapping in the reply popup and UIA flows without fallback support like https://github.com/devture/matrix-synapse-shared-secret-auth showing up as UIA stages. Key backup is also now enabled by default (if it has the correct signatures and everything), various layouting fixes and more.

We'll probably make a bugfix release in the next few days.

Element Web/Desktop (website)

Secure and independent communication, connected via Matrix. Come talk with us in #element-web:matrix.org!

Kegan announces

I have been working on integrating MSC3575: Sliding Sync into Element-Web. This is a pretty large endeavour but there has been a lot of progress on it already!

There now exists a JS-SDK PR to add the core bits of Sliding Sync (along with E2EE/to-device extensions), and a React-SDK branch which adds the Labs flag, config.json section for the proxy URL, and sets up room subscriptions when you select a room. The net result?

This is still early days: you cannot scroll the room list yet and there's still many outstanding issues to fix before it can land on mainline, but it's at the stage where it's almost ready for people to try out. Watch this space...

Thib reports

  • Continuing with the removal of the skinning layer, instead recommending module replacement - please talk to us in #element-dev:matrix.org if this is a surprise.

  • Continued removal of Groups/Communities - we’re expecting it to land this week (for release on about April 11th).

  • Continuing development of live location sharing.

  • We’ve reduced tsc errors by 70% in react-sdk unit tests!

  • In labs (you can enable labs in settings on develop.element.io or on Nightly)

    • We're working on a prototype for voice rooms, which are persistent voice chats similar to Discord's voice channel feature. Expect this to receive more attention and polish over the coming weeks
    • Threads are now using the stable prefix m.thread and are currently undergoing acceptance testing. We’re hoping to launch it out of labs in the next few weeks.

Element iOS (website)

Secure and independent communication for iOS, connected via Matrix. Come talk with us in #element-ios:matrix.org!

Manu says

  • The release on the App Store is still blocked on the review. We continue to work hard to solve this issue.
  • The threads feature received a lot of polish and bug fixes. We are not far from making them out of the LABS setting.
  • The timeline got several bug fixes.
  • We fixed memory leaks in the share extension.
  • The name and avatar customization after account creation is almost done.

Thib says

  • We’re currently at the last mile of testing for space creation & space management on iOS. Thanks to everyone who helped with community testing last week!
  • We’re currently on track to merge and ship these changes at the end of this month.
  • Once released, Element on all platforms will have parity across the core spaces feature set!

Element Android (website)

Secure and independent communication for Android, connected via Matrix. Come talk with us in #element-android:matrix.org!

adam announces

  • 1.4.4 has dropped across all channels, we've had some promising early feedback that the storage/RAM usage is starting to decrease, as always thanks for the reports and keep them coming! Full release notes available in the usual place

  • Threads are levelling up and becoming driven by MSC3440, increasing their reliability and cross device consistency. If you're already using threads we'll automatically migrate them for you in the next release 1.4.6

  • Live location sharing work has started! Soon you'll be able to share and receive location updates in real time

Cinny (website)

Cinny is a Matrix client focused on simplicity, elegance and security

Lozenge reports

v1.8.1 & v1.8.2

Bug fixes:

  • Fix new messages don't appear
  • Fix pressing a key with an active highlight jumps in history
  • Fix not all emoji-only messages being detected as Jumbo-emoji
  • Fix muted room show unread indicator
  • Fix view source shows original event for an edited message
  • Fix wrong notification count and NaN notification
  • Fix ⌘ CMD + k hotkey not working MacOS
  • Fix sending message not mark room as read

Release: https://github.com/ajbura/cinny/releases/tag/v1.8.2

Find more about Cinny at https://cinny.in Join our channel at: #cinny:matrix.org Github: https://github.com/ajbura/cinny Twitter: https://twitter.com/@cinnyapp

Dept of Non Chat Clients 🎛️

Thirdroom (website)

A browser-based open metaverse client

Robert Long says

Our metaverse-on-matrix project is progressing quickly!

  • Robert and Nate have been working on updating the Third Room demo to use the latest bitECS and threecs work as well as moving rendering and game update loops off the main thread!
  • Bruno has been porting Native Group VoIP to Hydrogen SDK which Third Room will use for both spatial voice chat and DataChannels
  • Ajay has a working scaffold of the early UI concepts paired with Hydrogen SDK to incorporate the room list and chat views.

More design work is happening as well, here's our latest from Rian

Matrix Wrench (website)

Matrix Wrench is a web client to tweak Matrix rooms.

ChristianP says

v0.6.0 & v0.6.1

Want to purge a space recursively? Matrix Wrench now has a button for that. It will try to fetch a list of all sub spaces and rooms and offer you to delete them via the Synapse Admin API.

Also, you can now knock on rooms and the room lists allow to navigate to a room page by clicking on a room ID.

Next up will likely be searchable lists for rooms and users as well as other usability and layout improvements. The goal is to have a nice-looking, stable v1.0 release for its anniversary on 13th June. I'm looking for help with design and documentation.

Dept of SDKs and Frameworks 🧰

Ruma (website)

A set of Rust library crates for working with the Matrix protocol. Ruma’s approach to Matrix emphasizes correctness, security, stability and performance.

Jonas Platte says

Nothing user-facing changed last week, but this week:

matrix-rust-sdk (website)

Matrix Client-Server SDK for Rust

Jonas Platte reports

In the last few weeks,

Additionally, these things have been in progress for a while and deserve a mention as well even though they're not finished yet:

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server.

#ping:maunium.net

Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1aria-net.org512.5
2pavot.ca784
3matrix.optoutpod.com836
4alemann.dev1532
5nevarro.space1560
6matrix.cirk2.de1729.5
7gottliebtfreitag.de2029
8mtrx.fail2116
9matrix.infra.reflect-media.de2408.5
10roeckx.be2577.5

#ping-no-synapse:maunium.net

Join #ping-no-synapse:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1nognu.de375.5
2rustybever.be386.5
3cry.is396.5
4matrix.sum7.eu504
5dendrite.matrix.org587.5
6beckmeyer.us741
7sspaeth.de1024
8conduit.cyberdi.sk1078
9dyoxide.online1339.5

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

This Week in Matrix 2022-03-11

11.03.2022 00:00 — This Week in Matrix Thib

Matrix Live 🎙

Dept of Spec 📜

Andrew Morgan (anoa) reports

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.

MSC Status

New MSCs:

MSCs with proposed Final Comment Period:

  • No MSCs had FCP proposed this week.

MSCs in Final Comment Period:

  • No MSCs entered FCP this week.

Merged MSCs:

Spec Updates

Keeping in step with our quarterly spec releases, the release of Matrix v1.3 is expected to land sometime early Q2 2022. This release will include Threads (as defined by MSC3440), as well as a number of other MSCs which land in the meantime.

Speaking of other MSCs, client developers should be aware of upcoming work to support live and static location sharing in the ecosystem. A notice on the subject has been posted today in the Matrix Client Developers room.

A reminder that the matrix-doc GitHub repository has been split into two new repositories: matrix-spec and matrix-spec-proposals. Read more about this in the last edition of TWIM.

Random MSC of the Week

The random MSC of the week is... MSC2398: proposal to allow mxc:// in the "a" tag within messages!

This MSC makes the case for allowing linking to MXC content by using of an <a> tag in the HTML-formatted formatted_body field of a m.room.message. As MXC urls could contain HTML files, it could be rather useful to link directly to them, as you would any other webpage. Your client would then translate this to an http/s URL which can be used to view/navigate to the file.

Dept of Servers 🏢

Synapse (website)

Synapse is the reference homeserver for Matrix

Brendan Abolivier reports

We've released Synapse 1.54! It includes quite a lot of nice improvements, including around URL previewing, and new callbacks for third-party modules. Check out the release announcement for more info on all this goodness.

We've also landed more of the ground work to improve the time it takes for new servers to join big rooms, which Rich was talking about last week. Aside from this, we're continuing the effort towards switching Synapse to use Poetry, which will enable reproducible builds for Synapse. These are some very exciting projects which will be maturing in the coming weeks, so watch this space!

Homeserver Deployment 📥️

Helm Chart (website)

Matrix Kubernetes applications packaged into helm charts

Ananace says

Missed last weeks update, but my charts rolled along regardless. Currently serving; element-web 1.10.6 and matrix-synapse 1.54.0 (and matrix-media-repo 1.2.10)

Dept of Bridges 🌉

matrix-appservice-kakaotalk (website)

A Matrix-KakaoTalk puppeting bridge.

Fair announces

The basics of the bridge are now ready! Backfilling, message sending, and message receiving are now implemented...albeit quite buggy and untested.

For anyone curious to try it out, installation follows the same general steps as tulir's Python-based mautrix bridges, plus a backend Node module that must be launched before the Python component of the bridge. I'll write a README with proper setup instructions soon!

Just note that for the time being, updates to the bridge may invalidate its database!!. I won't guarantee stable DB upgrades until the bridge's early growing pains have settled down.

Reminder that my KakaoTalk ID is fair-mw if anyone wants to help me test things out 🙂


Discussion: #matrix-appservice-kakaotalk:miscworks.net Issue page: https://src.miscworks.net/fair/matrix-appservice-kakaotalk/issues

Dept of Clients 📱

SmallTalk (website)

Minimal Android messenger powered by Matrix

adam reports

👋 I'm happy to finally reveal the side project I've been working on - SmallTalk, a tiny native android matrix client focused on chatting with family and friends!

I've been wondering, how small can you make a functional native android matrix client? So far 1.7mb! (*when served from an app bundle)

Features
  • E2E encryption (including importing Element E2E keys)
  • Merged room and DM list
  • Message bubbles
  • Push notifications

Currently in closed beta whilst ironing bugs, improving error handling and adding basic features like accepting invites, however apks are available on github for eager testers!

Nheko (website)

Desktop client for Matrix using Qt and C++17.

Nico says

I am happy to present you a new release with the cute name 0.9.2!

Apart from losts of bugfixes, expect the following:

Highlights

  • Message bubbles (Malte E) 💬
    • Give a colorful and space saving background to messages.
    • Optionally shrink the usernames to save even more space.
    • Your messages are on the opposite side of messages sent by other users.
  • Basic widgets 🗔
    • Widgets in a room are shown below the topic.
    • Open them in your browser to view them.

Features

  • Autocompleter for custom emotes using ~. Note that this currently inserts raw html into the message input.
  • Support running Nheko without a secrets service using a hidden setting.
  • Add zooming and panning to the image overlay.
  • Add a manpage. (tastytea)
  • Offline indicator. (LorenDB)
  • Proper previews for unjoined rooms in spaces (on supported servers).
  • /reset-state /command to reset the state of a single room.
  • Allow hiding some events from the timeline. (tastytea)
  • Hidden read receipts. (Symphorien)
  • Open room members dialog when clicking the encryption indicator.
  • Click to copy room id. (Malte E)
  • Allow specifying a reason for message removal, bans and kicks. (tastytea)

Improvements

  • Speed up blurhash and jdenticon rendering.
  • Use fewer threads for image decoding reducing memory use.
  • Document secret service installation on Arch. (Marshall Lochbaum)
  • Make edits replace previous notifications for the same message on Linux.
  • Add alternatives for Alt-A as a shortcut on systems where that is already used.
  • Apply clang-tidy suggestions. (MTRNord)
  • Make custom emotes twice as high as the text to improve legibility. (tastytea)
  • Ensure high quality scaling is used for custom emotes. (tastytea)
  • Reduce allocations for the timeline by around a factor of 2.
  • Render messages half as often, when displaying them for the first time.
  • Increase maximum number of items in completers to 30.
  • Run the gstreamer event loop also on macOS and Windows.
  • Make presence update dynamically.
  • Cleanup the raw message dialog.
  • Make settings responsive.
  • Improve Login and Registration pages.
  • Add custom stickers & emotes to Q&A.
  • Improve scrolling on touch screens. (Malte E)
  • Reduce size of state events.
  • Update OpenSUSE install instructions. (LorenDB)
  • Use newer flatpak runtime.
  • Fallback to using the shortcode in custom emotes, when there is no title set. (Ivan Pavluk)
  • Improve a lot of hovering behaviours.
  • Make spinboxes in scrollable pages unscrollable. (Malte E)
  • Fix deprecation warnings in gstreamer code. (Scow)
  • Make room directory fit mobile screens. (Malte E)
  • Make room search accessible on mobile. (Malte E)
  • Fix calls on mobile.
  • Add arch binary repo. (digital-mystik)
  • Improve long topics in the room settings. (Malte E)
  • Fix typos. (ISSOtm)
  • Improve the message input on mobile devices. (Malte E)

I hope this one works for you and we didn't break too much stuff and thanks a lot to everyone who contributed!

Element (website)

Everything related to Element but not strictly bound to a client

Danielle announces

Welcome back to another week at Element!

Message Threads

  • Our MSC 3440 completed its final comment period this week which is a super exciting move forwards for our team! We are now using a stable prefix and server-side support for Threads so the whole experience should be much better.
  • The team is hard at work fixing bugs so that we can launch with as much confidence in our solution as possible.
  • If you’ve been using Threads whilst in Labs, please keep giving feedback and raising any issues you may find.

Community testing

Element Web/Desktop (website)

Secure and independent communication, connected via Matrix. Come talk with us in #element-web:matrix.org!

Danielle says

  • On Web we’re focussed on improving the app and fighting bugs;
    • This week we are seeing fewer reports of decryption errors!
    • On Develop, you should no longer find that editing messages messes with their formatting.

In labs (you can enable labs in settings on develop.element.io or on Nightly)

  • Pinned messages are getting some attention in Labs. Keep providing feedback for us!

Element iOS (website)

Secure and independent communication for iOS, connected via Matrix. Come talk with us in #element-ios:matrix.org!

Manu announces

  • The next app store release is delayed due to some flows that do not match Apple guidelines - we’ll be updating the Apple SSO default values and introducing “Decline & Block” to invitations.
  • This week we checked-in on our language status and updated some things; We now support Ukrainian.
  • We’re working hard at updating the activity indicators in the app. While we work on the performance and load times in general we want to enable users to go about their tasks without being blocked by the activity indicator. The new indicator sits towards the top of the screen and should be out of the way for most tasks, whilst still informative. Let us know what you think!
  • The iOS team are also improving the onboarding experience for new users; aiming to make Matrix account creation as simple as possible.

Element Android (website)

Secure and independent communication for Android, connected via Matrix. Come talk with us in #element-android:matrix.org!

Danielle announces

  • We’re continuing on our path to increase the quality and usability of the app; the next release is v1.4.4 (already available for beta testers!) includes things like moving the typing indicator from the top bar to the bottom of the timeline.
  • We are also improving our PR process. In particular we want to reduce the time the submitters have to wait to get a review.
  • Our onboarding flow is being updated. We want new users to have a simple and straight-forward experience when they’re creating an account.

Cinny (website)

Cinny is a Matrix client focused on simplicity, elegance and security

Lozenge reports

Namaste 🙏,

We are happy to announce that Cinny now support all the features of Spacesa way to group rooms in Matrix. This means that you can now create new spaces from client, add rooms or subspaces inside them or manage the added rooms. We have also made our unique feature of Pinning spaces to sidebar more easy and organized. With this option all the spaces and subspaces will be visible at one place and you can select the one you want to pin or remove. One more unique feature we added with update is the Categorized subspaces, what this do is make all the subspaces inside a space appear as categories. This is helpful when there are deep nested spaces inside a space and breadcrumb is pain to navigate back and fourth. Keep in mind that this ignore the nesting of subspaces and all of them will appear under mainspace, but nice thing is that you can select which subspace you want to categorized and which not. Another exciting update related to spaces is the Drag-and-Drop. Earlier there was no option to reorder pinned spaces but now with DnD you can reorder them as you like.

Apart from spaces this update also include option to drag and drop files, ||sending spoiler||, desktop notification and viewing event source. (Thanks to @ginnyTheCat for these amazing additions.)

Full changelog is available at: https://github.com/ajbura/cinny/releases/tag/v1.8.0

Find more about Cinny at https://cinny.in Join our channel at: https://matrix.to/#/#cinny:matrix.org Github: https://github.com/ajbura/cinny Twitter: https://twitter.com/@cinnyapp

Dept of Non Chat Clients 🎛️

Populus Viewer (website)

A Social Annotation Tool Powered by Matrix

gleachkr announces

In the last two weeks we've made some UX improvements:

  1. The PDF viewer now has nicer tooltips
  2. The width of sidepanels is now adjustable
  3. Searchbars now have hints to help find the keyboard shortcut
  4. Error messages are now displayed more uniformly
  5. Populus-viewer now has an icon
  6. Failed messages are now indicated, with the option to resend

Fixed some bugs:

  1. room avatars should now update more quickly
  2. highlights should always display immediately after joining a new discussion

And added a service worker, for a better offline experience.

Finally, we've opened a new draft MSC (just a stub at the moment) to create a format for annotations on text documents. So... matrix annotations as a vim plugin? The future is bright 😎

As always, if you're interested in this work, or in the future of annotation and scholarship on matrix, come join us at #opentower:matrix.org.

matrix-streamchat (website)

Matrix powered stream overlay for OBS, to integrate live chat in your favorite (selfhosted) streaming setups.

f0x announces

Matrix Streamchat got it's first alpha release, and was successfully set up + used by someone else this week Recorded a little demo video on the setup with OBS: https://youtu.be/HmJ3XwJXB7I, and implemented the GUI for configuring the chat's settings and looks, to then copy the link into an iframe (embedded chat) or OBS (overlay).

Gatho (website)

Invite friends to your event with a one-click RSVP link - no matter which chat/social app they use!

Jake C reports

Gatho.party is a replacement for Facebook Events for small social gatherings. I've used it for a number of parties and received great feedback, and I'm seeing others are starting to use it too 🎉

To receive confirmations of attendance you can either:

  • Use the included Matrix bot to automatically sync RSVP emoji reactions in a linked Matrix room (great for your friends on Matrix or in a bridged Signal chat)
  • Send a unique one-click RSVP links to you friends without Matrix (eg. Instagram DMs, SMS)
  • And as of this week, guests can "self serve" and add their own RSVP via the unique event link, so you can just drop the event link in a Facebook Messenger chat or iMessage group (without email or phone verification at the moment, so hosts will need to moderate if their guests go rogue and add multiple fake names!)

Other recent improvements are:

  • Ability to delete guests
  • Improved call to action and copy text for when a guest with a "magic" link is prompted to RSVP
  • Confetti animations when your guests RSVP via the Gatho website as "Going"

I'm currently testing an experimental chat embed using Matrix Live so your guests not on Matrix can optionally see your linked Matrix room - I've found some of my friends join Matrix after seeing this so they can join the action!

I'm keen to eventually use Cactus Comments once I figure out a bit of Elm and have more spare time so that non-Matrix guests can post too.

The Gatho website and bot is open source (AGPL-3.0) on Github, PRs and Github issues are very welcome! It's built using Next.js in Typescript.

I'd love to hear your feedback! Join the Matrix room at https://matrix.to/#/#gatho-events:matrix.gatho.party

Dept of SDKs and Frameworks 🧰

Polyjuice (website)

Elixir libraries related to the Matrix communications protocol.

uhoreg says

Polyjuice Client Test is a tool for testing Matrix clients by creating a predefined environment for each test. Since the last time it was mentioned in TWIM,

  • more tests have been added: a test for basic end-to-end encryption message sending and receiving, symmetric key backup (MSC3270), and key withheld codes (MSC2399). More tests (and improvements to existing tests) to come.
  • the UI has improved a bit. Sorry, it's still ugly, but slightly less so. Also, now you can search through the tests.
  • I am now running a publicly-accessible instance of it at https://test.uhoreg.ca/ so you can test your clients without having to set it up yourself. This instance is stable-ish, in that it should always be available, but may be restarted randomly, in which case you may need to re-start your tests.

Dept of Ops 🛠

Synatainer (website)

Synapse Maintenance Container – Docker container with tools for synapse & postgres database maintenance

saces announces

Synapse Maintenance Container – Docker container with tools for synapse & postgres database maintenance

New since v0.1.0:

  • added arm64 support
  • improved output of purge_history.sh

Start it without command and let it do its magic :)

What it does by default:

  • daily:
    • purge all rooms without local members
    • run the state autocompressor (small setting 500/100)
  • weekly:
    • delete old remote media (>90 days)
    • delete old message history from public joinable rooms (>180 days)
  • monthly:
    • run the state autocompressor (big setting 1500/300)
    • vacuum the database

Source: https://gitlab.com/mb-saces/synatainer

Room: #synatainer:c-base.org

Compose example: https://gitlab.com/mb-saces/synatainer/-/snippets/2264828


If you are looking for a docker container with just the auto compressor for linux amd64/arm64 in it, her you go: https://gitlab.com/mb-saces/rust-synapse-compress-state

Dept of Interesting Projects 🛰️

libli

unonde says

hello! wanted to show a small matrix web client made with unminified js/html/css & web components (wip++) https://libli.org/ https://gitlab.com/sctlib/libli + https://gitlab.com/sctlib/matrix-room-element looking forward feedback comments or anything, cheers & viva la matrix!

for now it only uses default matrix messages, and it is read only, so users have to login/register/CRUD through element (or any other client) the idea would be to give back users control on their data, allow them to "collect" (individually or as group), any type of content, gathered into matrix room

For now, the client is making HTTP requests to matrix API, to fetch the data of a (public) room, all message & state events. it also give a "clean URL" libli.org/room:server.tld so users can access their "profile" (chat room) easily (just like a twitter, instagram, medium) page

Dept of Guides 🧭

Self-Host a Matrix Map Server

JulianF reports

Following Element's recent announcement of location sharing with support for self-hosted map tile server, I went ahead and set up one, and have written a blog post about it, explaining how to go a little further than the currently published guide, and with Ansible scripts included. *> https://wrily.foad.me.uk/self-host-a-matrix-map-server

Extensible Events

andybalaam reports

Andy Balaam wrote a blog post explaining extensible events - it shows simple examples of what message events look like now, and how they change with extensible events. It also covers the abbreviated forms of m.message: m.text and m.html, which can make it somewhat more confusing to understand.

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server.

#ping:maunium.net

Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1neko.dev308
2envs.net380
3maescool.be406
4tedomum.net450
5matrix.uni-marburg.de548
6catgirl.cloud610
7synod.im754
8settgast.org841.5
9jeroenhd.nl1052
10matrix.infra.reflect-media.de1160

#ping-no-synapse:maunium.net

Join #ping-no-synapse:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1rcp.tf225
2nognu.de301
3rustybever.be306
4cry.is460.5
5beckmeyer.us636
6sspaeth.de665.5
7xethos.net817.5
8grin.hu1003.5
9kwolek.io1107
10dendrite.beckmeyer.us1554.5

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Synapse 1.54 released

08.03.2022 00:00 — Releases Brendan Abolivier

Synapse 1.54 is out! Let's see what goodness is coming your way with this new release.

Improving URL previews

As part of the Matrix specification, Matrix clients have the option to rely on the homeserver to generate URL previews. This means the homeserver (in this case Synapse) needs to have a look at the content at that URL and extract data to send back to the client.

In the past, Synapse has had issues with generating complete URL previews, and some metadata (e.g. image, description) would be missing from the data that Synapse sends to clients. Synapse 1.54 includes improvements to the generation of URL previews, and while testing it we've been able to observe improvements to previews generated for Twitter and Reddit.

New module callbacks

Synapse modules allow third-party developers to write extra features for Synapse, that wouldn't necessarily be generic enough to fit within the Matrix specification. This includes custom behaviours such as smarter event filters, bespoke media storage providers, or spam checkers. Since we rewrote Synapse's module system in Synapse 1.37, we have been improving it by adding new callback functions that module can implement, allowing them to interface better with Synapse.

Synapse 1.54 includes three new module callbacks. One of them, get_displayname_for_registration, allows modules to define the display name for newly registered users. Together with the existing get_username_for_registration introduced in Synapse 1.52, they allow modules a better control over the user registration process.

The two other module callbacks introduced in Synapse 1.54, on_profile_update and on_user_deactivation_status_changed, allow modules to react to profile changes, as well as the deactivation (or reactivation) of users.

Everything else

This release of Synapse also includes more work to make joining large Matrix rooms faster (I was telling you about that in the Synapse 1.53 release announcement). While it's still very experimental and not yet ready for show time, it's still very exciting to see this work happening!

Synapse 1.54 also includes a change to the client-side versions endpoint to advertise Matrix 1.1 and 1.2. See the Matrix 1.2 release announcement to read all about the changes this latest version of the specification brings to the ecosystem.

See the full changelog for a complete list of changes in this release. Also please have a look at the upgrade notes for this version.

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including (in no particular order) Dirk Klimpel, Andrew Ryan, and lukasdenk.

This Week in Matrix 2022-03-04

04.03.2022 00:00 — This Week in Matrix Thib

Matrix Live 🎙

Dept of Status of Matrix 🌡️

TravisR says

t2bot.io - a public integrations network for Matrix - has passed 1 Million known rooms and 8.3 Million bridged users. 10,000 of these rooms are attributed to Voyager (a bot which actively goes out to find rooms to map Matrix, with the map currently down for maintenance), leaving the remaining ones either bridged, previously bridged, or using a different integration offered by t2bot.io for free.

The 8.3 Million users are mainly Discord and Telegram users which have been brought over to Matrix through bridges. The stats say "excluding Twitter-bridged" because there's 424,832 old accounts from back in the day when t2bot.io had a free Twitter bridge available. To further break this down, about 6.8M are Telegram users (12% of Telegram) and 1.3M are Discord (<1% of Discord).

For perspective, t2bot.io has about 569 Million events stored in its database and sees approximately 30 thousand people bridged daily from the wider world into Matrix through its bridges.

This post is just a milestone update, but it also serves as a reminder that running your own server/bridges is also possible. In fact, it's even recommended to have better control over your own data and avoid latency issues that large providers, like t2bot.io, can unintentionally introduce. Synapse is relatively easy to set up with minimal sysadmin knowledge (guide), and there's always paid offerings like Element Matrix Services (for home and also for fun) and Beeper for a richer bridging experience than t2bot.io can feasibly provide.

TravisR adds

in the eternal battle of "t2bot.io should be successful" and "please oh god self-host", the above^

Thib reports

It is to be noted that eQualitie has set-up public Matrix and Element instances for Ukrainian people who are struggling right now. If you are in Ukraine or are in touch with Ukrainian people who need secure communications, you can show them https://kyiv.dcomm.net.ua, https://odessa.dcomm.net.ua, or https://kharkiv.dcomm.net.ua depending on their location so they get instructions on how to create an account on Matrix and stay in touch

Dept of Spec 📜

Kegan reports

MSC3575 Sliding Sync work is progressing, and a number of new features have been added to the MSC with implementations in the built-in web client in the proxy. A list of changes include:

  • spec/proxy: the ability to filter the room list by room name.
  • spec/proxy: Aninitial flag on the room to distinguish between updates and the initial sync for a given room. This can be used as a hint to clients to determine if they should update/replace their local data for that room.
  • proxy: more efficient algorithm for determining overlapping sets of ranges, resulting in fewer bytes over the wire when scrolling.
  • proxy: the client implementation now supports sending basic text messages as well as issuing/join,/invite,/leave commands.
  • proxy: the client implementation now has a developer HUD which tracks the Tx/Rx bytes as well as a visual representation of the sliding window.

The net result is a basic but incredibly low bandwidth syncing client: 60.79KB to download an entire initial sync on matrix.org. Further improvements will be done on the client to make sure it doesn't scale with the number of rooms on the user's account (it currently does because it naively adds a placeholder for each room in the list) to ensure it remains extremely snappy and a vision for what Sliding Sync can do in practice, right now. Please continue giving feedback on MSC3575 or in #sliding-sync:matrix.org as the API is still in development and will change depending on what clients require.

Andrew Morgan (Element) announces

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.

MSC Status

Created MSCs:

MSCs with newly proposed Final Comment Period:

  • No FCPs were proposed this week.

MSCs in Final Comment Period:

Closed MSCs:

Merged MSCs:

  • No MSCs were merged this week.

Spec Updates

Heads up! You may have noticed some changes to the matrix-org/matrix-doc github repository this week. The first major one being that it no longer exists!

We have separated it out into two repositories:

  • matrix-org/matrix-spec - the source text of the spec. Issues and pull requests filed here should relate to the source of the spec.
  • matrix-org/matrix-spec-proposals - all Matrix Spec Change proposals. New MSCs should be filed against this repository as pull requests. Issues should not be filed against this repository (and no, the issue tracker cannot be disabled, as it would cause transferred issue link redirects to 404).

The primary motivation for this was to separate concerns of interacting with matrix-org/matrix-doc. Those interacting with the spec's source would have to wade through long-lived spec proposals, while those only interested in spec proposals would need to filter out spec source files and other issues and PRs related to them. This also helps writing tooling against the repo either as it no longer needs to filter MSCs via issue labels.

Existing links to matrix-doc should be automatically redirected by github to the right place. MSC authors and developers should update their git remote settings to point to the new repos.

What happened in detail to achieve this was:

  • matrix-doc was renamed tomatrix-spec-proposals
  • matrix-spec was created
  • All files other than the "proposals" folder were transferred frommatrix-spec-proposals tomatrix-spec
  • All open issues frommatrix-spec-proposals were transferred tomatrix-spec

Further details are available here: https://github.com/matrix-org/matrix-spec/issues/927

Random MSC of the Week

The random MSC of the week is... MSC3383: Include destination in X-Matrix Auth Header!

This MSC asks for the simple change of adding a "destination" field to the Authorization header of federation requests. This would have the benefit of allowing reverse proxies - potentially handling traffic for multiple homeservers - to know where requests should be sent. Or for forward proxies, which could validate outgoing requests from a homeserver before they're sent.

There is currently some discussion on the fundamental idea of the MSCs, which will need to be resolved first before it can move forwards.

Dept of Servers 🏢

Synapse (website)

Synapse is the reference homeserver for Matrix

richvdh announces

Quite a varied bag this week! Firstly we cut a release candidate for Synapse 1.54.0, which we hope to release next week. As ever, testing and feedback is much appreciated.

Our switch to Poetry for dependency management has come a few steps closer, by removing tox from our Continuous Integration, and converting some of our ad-hoc scripts to proper Python entry points.

Meanwhile we've been continuing to knock out blockers to improving room join speeds, including better exception handling and logging, and some exciting work to reduce wasted work when a client disconnects while we're still processing a request.

We've also been taking a wider look at performance problems, such as slow /sync and /login times.

Dendrite (website)

Second generation Matrix homeserver

Kegan reports

I have been asked by numerous people over months if there are any kind of graphs comparing Synapse and Dendrite. Up until now, my answer has been "as far as I know, no". However, this got me thinking whether I can use Complement to engineer some kind of performance testing infrastructure. After hacking on it for a day or two, I can now share some of the work I've done in this area. There now exists two new binaries in the Complement repo: perftest and perfgraph. The former will run some tests and output a .json file. The latter will consume these files to generate .svg graphs. The tests performed include:

  • Creating a bunch of rooms with different users. (currently 10 users and 50 rooms)
  • Joining a bunch of rooms with different users.
  • Sending messages into rooms with different users. (currently 100 messages)
  • Doing an initial sync for all users.
  • Updating the display name of all users.
  • Doing an incremental sync for all users.

The metrics are collected using docker stats. This means the entire container is measured, not just the homeserver binary (e.g any daemons or assistive processes are included in the metrics) and includes:

  • CPU time consumed
  • Memory consumed
  • Bytes transmitted/received
  • Disk read/writes
  • Time taken

Here are some results:

This was a surprising find and probably indicates that Dendrite has a memory leak somewhere when creating events. However, the fun doesn't stop there. We can also use this tool to compare different versions of the same homeserver:

This allows us to quantify any performance improvements we've been working on, and you can clearly see how we're being more efficient with CPU in newer versions. But the fun doesn't stop there either. We can also compare different flavours of the same version of a homeserver:

You can quantify the cost of running sqlite or postgres for your particular installation.

Care must be taken when interpreting results. Lower doesn't always mean better. For example, a low readout on "create_users" may be a cause for concern as it might indicate that the server is not hashing passwords correctly (e.g using bcrypt/scrypt which are designed to consume CPU/memory). Servers which don't implement all of the specification will also be abnormally low on CPU/memory (e.g not having to calculate unread counts or handle push notifications naturally means less work to do so less resources consumed). Furthermore, these tests do not yet test federation, so expensive remote joins are not measured (remote joins require the joining server to check signatures, hashes, etc of a lot of events which doesn't happen for local servers). That being said, I have hopefully illustrated how useful this tool can be for both server developers trying to improve their software and server admins who want to use the right server for their hardware. The graphs are still a work-in-progress and there's a lot more that can be done in this area beyond a few days work, but it's a start.

neilalexander says

Today we've released Dendrite 0.6.5 which contains early push notification support as well as a number of fixes and improvements. This release includes the following changes:

  • Early support for push notifications has been added, with support for push rules, pushers, HTTP push gateways and the /notifications endpoint (contributions by danpe, PiotrKozimor and tommie)
  • Spaces Summary (MSC2946) is now correctly supported (when msc2946 is enabled in the config)
  • All media API endpoints are now available under the /v3 namespace
  • Profile updates (display name and avatar) are now sent asynchronously so they shouldn't block the client for a very long time
  • State resolution v2 has been optimised further to considerably reduce the number of memory allocations
  • State resolution v2 will no longer duplicate events unnecessarily when calculating the auth difference
  • The create-account tool now has a -reset-password option for resetting the passwords of existing accounts
  • The /sync endpoint now calculates device list changes much more quickly with less RAM used
  • The /messages endpoint now lazy-loads members correctly
  • Read receipts now work correctly by correcting bugs in the stream positions and receipt coalescing
  • Topological sorting of state and join responses has been corrected, which should help to reduce the number of auth problems when joining new federated rooms
  • Media thumbnails should now work properly after having unnecessarily strict rate limiting removed
  • The roomserver no longer holds transactions for as long when processing input events
  • Uploading device keys and cross-signing keys will now correctly no-op if there were no changes
  • Parameters are now remembered correctly during registration
  • Devices can now only be deleted within the appropriate UIA flow
  • The /context endpoint now returns 404 instead of 500 if the event was not found
  • SQLite mode will no longer leak memory as a result of not closing prepared statements

Spec compliance, as measured by Sytest, currently sits at:

  • Client-server APIs: 76%, up from 65% last time
  • Server-server APIs: 95%, up from 94% last time

As always, you can join us in #dendrite:matrix.org for Dendrite discussion and announcements.

Dept of Bridges 🌉

matrix-hookshot (website)

A multi purpose multi platform bridge, formerly known as matrix-github

Half-Shot reports

matrix-hookshot reaches 1.2.0

Hey folks, I'm back from holidays and I'm proud to say that hookshot is kicking out release after release. This release is especially large and contains a bounty of new features and fixes. Of note are:

  • JIRA Datacenter (On Premise) is now supported.
  • You can now configure fine grained permissions for users on the bridge by userId, homeserver domain or room membership (think spaces).
  • Generic webhooks has sprouted a versioned API, with v2 allowing for finer control over the output.
  • GitHub connections now include the closing comment when issues and PRs are closed.
  • GitHub connections will also notify a room when an existing issue has been relabled to one filtered by that room.
  • Figma now uses MSC3440 for comment threads.

But seriously go check out the release, there is way more there than I can include in this TWIM post. Happy webhookin, matrix gang!

In case you did not know, Hookshot can be installed via spantaleev's Ansible playbook. In addition to updates for this new release, the role has recently gotten some improvements and fixes by the community, so in case you had issues before, now is the time to try again!

P.S oh and I almost forgot, we rehomed to https://github.com/matrix-org/matrix-hookshot. This doesn't change much, other than some redirects in your browsers cache.

Half-Shot adds

permissions:
* actor: '*'
  services:
  - level: commands
    service: github
* actor: '!the-internal-element-room:matrix.org'
  services:
  - level: manageConnections
    service: '*'
* actor: '@random-on-the-side-user:homeserver.org'
  services:
  - level: manageConnections
    service: '*'

Allows anyone to run github commands in rooms. User part of the room listed can actually go and create connections to anywhere (we use it internally at $work), and then there is a final random user who gets those permissions because they refuse to join that room :p

matrix-appservice-irc (website)

Node.js IRC bridge for Matrix

Half-Shot reports

matrix-appservice-irc 0.33.0

Two bridge updates from matrix.org in one week? You would be forgiven for thinking it was FOSDEM Christmas.

The highlights are:

  • Support splitting users from different homeservers into different IPv6 blocks. We used this at FOSDEM to split attendeeds into a different ILINE.
  • Added a new metric clientpool_by_homeserver which lists the states of IRC clients, by the top 25 homeservers. Useful to know who is using your bridge.
  • Add support for subscribing to moderation policies. This allows a bridge admin to subscribe to Matrix "ban lists" and filter who can use their bridge.

This release also now requires Node 14.x as Node 12.x will be out of support very soon.

See the changelog for yourself over at https://github.com/matrix-org/matrix-appservice-irc/releases/tag/0.33.0

Dept of Clients 📱

Neochat (website)

A client for matrix, the decentralized communication protocol

Tobias Fella says

This week, we've fixed a few annoying issues in NeoChat:

  • On platforms with a system tray, NeoChat no quits correctly when the system tray integration is disabled
  • Similar state events are now aggregated into a single message (so no more scrolling through "Moderation has changed the access control lists" 🙂)
  • Sending messages starting with a ':' works correctly again

We've also improved NeoChat's startup time from cache by a factor of several hundred, depending on the situation

Element Web/Desktop (website)

Secure and independent communication, connected via Matrix. Come talk with us in #element-web:matrix.org!

kittykat says

  • Released v1.10.5 and also v1.10.6 as a hotfix for a crash
  • Surveyed users and collated feedback from search result ordering for improvements to the new search experience (enable it in Settings -> Labs in production)
  • In labs (you can enable labs in settings on develop.element.io or on Nightly):
    • Pin drop location sharing has been added
    • Fixed message ordering issues in threads
    • Fixed threads discovery when scrolling back in a room
    • Added a couple of new ways to access pinned messages, in the room header and info panel
    • Improved the reliability of pinned messages with edits

Element iOS (website)

Secure and independent communication for iOS, connected via Matrix. Come talk with us in #element-ios:matrix.org!

Manu announces

  • New release, v1.8.3, is out
  • Still improving the activity indicator
  • Working on missing messages bugs
  • Making progress on the registration flow by setting name and avatar
  • In development:
    • We will be trying out Spaces on iOS at 16:00 UTC / 17:00 CET on Tuesday, 8th of March. Head over to #element-community-testing:matrix.org to hear the latest on all testing sessions!
    • Live location sharing has started
  • In labs:
    • An option to use the last member avatar and name in the timeline

Element Android (website)

Secure and independent communication for Android, connected via Matrix. Come talk with us in #element-android:matrix.org!

adam reports

  • Our newest release, v1.4.2, is out!
  • Progress is being made on fixing “unable to decrypt” errors (UISI) - re-architected legacy code.
  • Our language capabilities have expanded with improved Japanese translations (special thanks to the contributor for this!)
  • The team is working on process improvements to reduce incoming PR backlog and getting to those faster.
  • We’re actively looking into the reported increased memory usage.
    • As always, thank you for feedback and reports, it helps a lot.
  • For those using the SDK, we’ve deprecated the main Matrix entrypoints, more information here

Dept of Non Chat Clients 🎛️

matrix-streamchat (website)

Matrix powered stream overlay for OBS, to integrate live chat in your favorite (selfhosted) streaming setups.

f0x says

a very simple invite-accepting bot as part of matrix-streamchat to more easily facilitate peeking into rooms over federation. You run one on your guest-access enabled homeserver, invite it to your remote room, and now all guest accounts from your server know they are in fact allowed to publicly read the room over federation without joining themselves. https://git.pixie.town/f0x/matrix-streamchat/src/branch/main/autojoin-bot

Dept of Widgets 🧩

Mjolnir Widget (website)

A widget for moderating with mjolnir. Highly WIP. Use at your own risk!

MTRNord (they/them) reports

Mjölnir Moderation Widget

There have been a few changes to the widget:

  • Ability to kick people was added
  • Inter font was added to improve readability
  • Work on having a setup page for the bot was added which allows to easily add the bot to your room (sets up permissions and invites the bot as required)
  • Added support for a state event which tells the widget about "who is the bot?" and "which lists does this bot monitor?" Which also allows you to use the widget without being in the banlist room yourself. (See https://github.com/MTRNord/matrix-moderation-widget#use-a-state-event-to-allow-showing-relevant-lists-only-in-the-dropdown for more information)

Dept of VoIP 🤙

Element Call (website)

Native Decentralised End-to-end Encrypted Group Calls in Matrix, as a standalone web app

Matthew reports

Element Call has entered beta! Head over to https://call.element.io (formerly matrixvoip.dev) to play with in-browser native Matrix voice/video calling powered by MSC3401, supporting all webrtc-capable mobile & desktop browsers. See all the details over at https://element.io/blog/introducing-native-matrix-voip-with-element-call/

  • Offers high quality video calls with around 6 or 7 participants as a small & simple, standalone app that allows anyone to drop into a video conference easily.
  • This is beta - please file bugs at https://github.com/vector-im/element-call/issues and be aware that currently all participants need good bandwidth
  • This is the first implementation of MSC3401: a spec for secure, federated voice and video conferences.
  • Integrations into other clients are coming soon, Hydrogen is working on it right now
  • We’ll also be developing a server to mix media streams so calls can scale up to much larger number of participants. The server will only ever see your encrypted media, so calls will stay secure and confidential.

Dept of SDKs and Frameworks 🧰

Ruma (website)

A set of Rust library crates for working with the Matrix protocol. Ruma’s approach to Matrix emphasizes correctness, security, stability and performance.

Jonas Platte reports

We're back (to TWIM)! Here's some highlights from the past few months:

All of these changes are available in version 0.5.0 which was released about two weeks ago.

Additionally, @zecakeh recently started implementing extensible events and doing some pretty heavy internal refactoring that will make our release process faster and hopefully make contributing to Ruma easier as well.

With all these changes we're very close to supporting all of Matrix v1.1 (and Matrix v1.2 too!), with the only major omission being Secure Secret Storage and Sharing (SSSS). Some work on that was done as far back as last year's GSoC but it was blocked on a medium-sized refactoring of how we handle account data. I'm planning to get that over the finish line over the next weeks.

Dept of Ops 🛠

Synatainer (website)

Synapse Maintenance Container – Docker container with tools for synapse & postgres database maintenance

saces announces

Start it without command and let it do its magic :)

What it does by default:

  • daily:
    • purge all rooms without local members
    • run the state autocompressor (small setting 500/100)
  • weekly:
    • delete old remote media (>90 days)
    • delete old message history from public joinable rooms (>180 days)
  • monthly:
    • run the state autocompressor (big setting 1500/300)
    • vacuum the database

Source: https://gitlab.com/mb-saces/synatainer

Room: #synatainer:c-base.org

Compose example: https://gitlab.com/mb-saces/synatainer/-/snippets/2264828

Dept of Bots 🤖

Matrix Scribe

JulianF announces

I'm working on a little hobby project, The Matrix Scribe.

The Matrix Scribe helps us re-post or transcribe messages into matrix that we received from somewhere else, posing as different ghost users to represent the original authors.

Scribe will post this:

Ann I'm Ann.

Bob Hello! I'm Bob.

when I write this to @scribe-bot:

@julian /scribe as Ann

@julian I'm Ann.

@julian /scribe as Bob

@julian Hello! I'm Bob.

That is an illustration of the goal. The output part works already; the bot part isn't implemented yet.

Why?

  • I want to encourage people to experiment and play with extending Matrix in creative ways.
  • As a building block for myself towards building more usual matrix bridges. Scribe is the input half of the matrix half of a bridge, so literally one quarter of a typical matrix bridge.
  • I hope that I myself and maybe some other people will find Scribe fun or useful in itself.

If you find this interesting...

Scribe is not intended to mislead users. It should be deployed in such a way that its use is clear, and require appropriate permissions.

maubot_azuracast

boris reports

a simple bot https://github.com/borisrunakov/maubot_azuracast built with maubot, that you can use to request data from your azuracast self hosted instance. It's not much but ok...

Dept of Events and Talks 🗣️

J. Ryan Stinnett says

I'll be speaking at the new HYTRADBOI conference (29 Apr) about building collaborative, open software on Matrix. All of the talks sound super interesting, highly recommended! 😄 The talks will be public after the conference as well.

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server.

#ping:maunium.net

Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1converser.eu347
2envs.net377
3aria-net.org466
4maescool.be573
5pavot.ca742
6rollyourown.xyz1294.5
7canchat.org1473
8targodan.de1658
92808.nl1949
10mtrx.fail2093.5

#ping-no-synapse:maunium.net

Join #ping-no-synapse:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1cry.is288
2nognu.de319
3rustybever.be383.5
4conduit.grich.sk465
5bogus.ee568
6matrix.sum7.eu650.5
7matrix.awesomesheep48.me653
8dendrite.matrix.org1544
9xethos.net1756

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

This Week in Matrix 2022-02-25

25.02.2022 00:00 — This Week in Matrix Thib

OTWSU 🎙

Dept of Spec 📜

Andrew Morgan (Element) reports

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.

MSC Status

Merged MSCs:

MSCs in Final Comment Period:

  • No MSCs are in FCP.

New MSCs:

Spec Updates

MSC3440 continues to be the focus on much review as the proposal inches towards entering final comment period.

Otherwise both MSC3589 (room version 9 as the default room version) and MSC3582 (remove m.room.message.feedback) merged this week. The former brings with it updated versions for rooms as they continue to be created across the federation, whereas the latter is simply a nice clean up to the existing spec :)

Random MSC of the Week

The random MSC of the week is... MSC3593: Safety Controls through a generic Administration API!

This MSC aims to introduce a set of generic administrative APIs for Matrix homeservers, starting with those that could potentially be useful for user moderation.

There has long been talk of adding administration APIs akin to Synapse's Admin API to the spec, in part to reduce the number of tools that are specifically created for it, and thus can only be used with Synapse.

Feedback welcome!

Dept of Servers 🏢

Synapse (website)

Synapse is the reference homeserver for Matrix

Shay announces

This week we released Synapse 1.53.0! This release adds support for sending to-device messages to application services, adds a background database update to purge account data for deactivated users, and adds more features to improve performance and stability, as well as bugfixes and improved documentation. Check out the release notes here.

In addition to the release, work continues on improving the performance of room joins-progress is being made! Finally, we began the process of switching over Synapse to use Poetry for dependency management-keep your eyes peeled for more information on that project as it develops.

Dendrite (website)

Second generation Matrix homeserver

neilalexander says

This week we released Dendrite v0.6.4 which contains a significant number of improvements and fixes. It includes the following:

  • All Client-Server API endpoints are now available under the /v3 namespace
  • The /whoami response format now matches the latest Matrix spec version
  • Support added for the /context endpoint, which should help clients to render quote-replies correctly
  • Accounts now have an optional account type field, allowing admin accounts to be created
  • Server notices are now supported
  • Refactored the user API storage to deduplicate a significant amount of code, as well as merging both user API databases into a single database
  • Guest registration can now be separately disabled with the new client_api.guests_disabled configuration option
  • Outbound connections now obey proxy settings from the environment, deprecating the federation_api.proxy_outbound configuration options
  • The roomserver input API will now strictly consume only one database transaction per room, which should prevent situations where the roomserver can deadlock waiting for database connections to become available
  • Room joins will now fall back to federation if the local room state is insufficient to create a membership event
  • Create events are now correctly filtered from federation /send transactions
  • Excessive logging when federation is disabled should now be fixed
  • Dendrite will no longer panic if trying to retire an invite event that has not been seen yet
  • The device list updater will now wait for longer after a connection issue, rather than flooding the logs with errors
  • The device list updater will no longer produce unnecessary output events for federated key updates with no changes, which should help to reduce CPU usage
  • Local device name changes will now generate key change events correctly
  • The sync API will now try to share device list update notifications even if all state key NIDs cannot be fetched
  • An off-by-one error in the sync stream token handling which could result in a crash has been fixed
  • State events will no longer be re-sent unnecessary by the roomserver to other components if they have already been sent, which should help to reduce the NATS message sizes on the roomserver output topic in some cases
  • The roomserver input API now uses the process context and should handle graceful shutdowns better
  • Guest registration is now correctly disabled when the client_api.registration_disabled configuration option is set
  • One-time encryption keys are now cleaned up correctly when a device is logged out or removed
  • Invalid state snapshots in the state storage refactoring migration are now reset rather than causing a panic at startup

Sytest compliance is up slightly:

  • Client-server APIs: 67%, up from 65%
  • Server-server APIs: 95%, same as before

As always, please feel free to join us in #dendrite:matrix.org for Dendrite-related chat!

Homeserver Deployment 📥️

Helm Chart (website)

Matrix Kubernetes applications packaged into helm charts

Ananace reports

And my Helm Chart updates continue as they do, with matrix-synapse being updated to 1.53.0

Dept of Bridges 🌉

matrix-appservice-kakaotalk (website)

A Matrix-KakaoTalk puppeting bridge.

Fair reports

Here are the first steps for a bridge to KakaoTalk! The bridge is based on mautrix-python (having used mautrix-facebook as a starting point--there's still plenty of Facebook-specific code in there), with the backend handled by node-kakao (connected via RPC, as there seems to be no Python API for KakaoTalk!).

The bridge doesn't do much yet; all it can do is log in & sync your list of chats (if that). But it's under rapid development & decent momentum, so hopefully it will be usable soon!

For anyone brave enough to try it out, its setup steps are very similar to that of any of the Python-based mautrix bridges (though Docker is currently unsupported).

Discussion: #matrix-appservice-kakaotalk:miscworks.net Issue page: https://src.miscworks.net/fair/matrix-puppeteer-line/issues

Dept of Clients 📱

Nheko (website)

Desktop client for Matrix using Qt and C++17.

Nico announces

We made a small release that just is compiled against the new mtxclient version to fix an issue with servers announcing support for Matrix v1.1 or higher. We strongly recommend you update before the next Synapse stable release is out.

Apart from that Nheko now has support for hidden read receipts (thanks to symphorien, see MSC2285). ZenWalker updated our usage of deprecated gstreamer APIs. Malte has been spending a lot of effort on improving the scrolling experience on the PinePhone as well as allowing to search on mobile. Forwarding should now work properly again as well as calling on mobile and we fixed a small memory leak when opening some dialogs.

Element (website)

Everything related to Element but not strictly bound to a client

Danielle announces

“Twosday” wasn’t the only exciting thing happening this week. Take a peek at everything else we had going on…

Coming to a Poll near you…

  • From next week’s releases, you’ll discover two new updates on polls! First off, you’ll be able to edit a poll as long as no one has yet voted on it - which is great if you create a poll and realise you’ve made a small mistake. Even better, there’s now a new type of poll: ‘closed polls’ don’t show any results until the poll has ended, to keep the surprise.

Location Sharing

  • Location Sharing is now available by default for users on all platforms, except desktop (where you can receive but not send locations). Check it out!
  • The next stage is live location sharing and ‘pin dropping’, expect more soon.

Threaded Messaging

  • Designed to make catching up on rooms easier, and to keep the main timeline as clutter free as possible, Threads are nearly here.
  • You can try Threads out on all platforms - you’ll find them in Labs. This feature is experimental; let us know your feedback, and report any bugs as we continue to improve.

Community testing

  • We will be looking at search result ordering on Web as part of the new search experience at 17:30 UTC / 18:30 CET on Tuesday, 1st of March
  • We’re also hoping to test Threads on mobile devices towards the end of the week, join the testing room to get involved!
  • Head over to #element-community-testing:matrix.org to hear the latest on all testing sessions!

Element Web/Desktop (website)

Secure and independent communication, connected via Matrix. Come talk with us in #element-web:matrix.org!

Danielle announces

  • The EleWeb team has been working on Spaces, Threads, and defects this week.
  • We are starting to look at batched updates, which could be bringing performance improvements to us.
  • On the process improvement side, we are looking at test coverage and process improvements around PR submission. Don’t be surprised if our developers start a conversation around tests when you submit your next PR 🙂
  • V1.10.5 release candidate is available and release is expected to go out on Monday, 28th February.
  • And don’t forget; the new and improved search experience is available. It’s in Beta so turn it on, try it out, and send us your feedback!
    • We will be talking to the community about planned improvements in the next Community Testing Session on Tuesday over in#element-community-testing:matrix.org

In labs (you can enable labs in settings on develop.element.io or on Nightly):

  • Improvements to Threads reliability are happening everyday. We’re also making some tweaks to the user experience details, like dragging and dropping files into the Thread panel.

Element iOS (website)

Secure and independent communication for iOS, connected via Matrix. Come talk with us in #element-ios:matrix.org!

Manu reports

  • Next week’s release (1.8.3) includes changes and improvements we’ve made to our overall app experience by closing some pesky UI defects.
  • This week we’ve also been working on improving the reliability of our Labs features. If you’ve turned on Threads or Bubbles in Labs you may have experienced app slow downs or crashes. In the next version, these will be minimised.
  • Spaces on iOS are also getting some attention at the moment and we’re hoping to improve the user experience of Spaces on Mobile.

Element Android (website)

Secure and independent communication for Android, connected via Matrix. Come talk with us in #element-android:matrix.org!

Danielle announces

  • The next release of Android (1.4.2) includes support for “@ room” and other usability defects you might have seen before… Fixes include steadying the notification badge in the room list, adding the correct interactions on bottom sheets, and opening a DM from the Space member list.
  • We’ve also been working on upgrading the voice message experience, adding improvements like scrubbing! Keep sending voice notes and let us know what other improvements we should make.
  • Our onboarding flow is also getting a new lick of paint. We want new users to our platform to have a simple and straight-forward experience when they’re creating an account.

Dept of Non Chat Clients 🎛️

Populus Viewer (website)

A Social Annotation Tool Powered by Matrix

gleachkr says

Populus-Viewer

Over at Populus-Viewer, we're continuing to refine the UX, for maximum focus, efficiency and enjoyment. Since last time we've:

  1. Reworked the mobile view controls into a sidebar design.
  2. Improved the generation of highlight rectangles.
  3. Made sure that LaTeX and code listings are always displayed nicely
  4. Made it possible to modify text selections within a PDF using the keyboard
  5. Added "one-click" links for onboarding new users into a particular server, SSO flow and PDF collection.

We've also had some bug fixes related to federation, and had some of our first ever (maybe the first on matrix - first in the history of the universe?) federated social annotation sessions.

Populus-Philarchive

Populus-Philarchive, our proof-of-concept discussion overlay for preprint archives, now incorporates an OAI-PMH harvester, so it can aggregate OAI bibliographic metadata, and use that data for room creation and discovery. The implementation is pretty general, so it should be easy to tweak for any archive that supports OAI-PMH.

MSC3574

MSC3574 - marking up resources got some love this week, as we added a proposal for serializing annotations on matrix that ought to be compatible with the w3c web annotation data model. This paves the way for interoperability between the matrix annotation ecosystem and services like hypothes.is, and hopefully will make matrix a compelling option even for institutions where compliance with existing web standards is a must.

As always, if you'd like to chat about any of these developments, come visit us at #opentower:matrix.org !

Circles (website)

E2E encrypted social networking built on Matrix. Safe, private sharing for your friends, family, and community.

cvwright reports

The Circles beta on iOS continues inching toward a public release later this Spring.

  • This week I added support for infinite scrolling on timelines. (Previously, scrolling the timelines was very clunky -- the user had to manually tap a button to "Load More" every 10-15 posts.)
  • Also added a confirmation dialog when the user attempts to leave a group.

On Android, the prototype is coming along nicely, thanks to the efforts of our new developer Taras:

  • The login screen works
  • Currently working on implementing the timeline of social posts for groups

Dept of Widgets 🧩

Mjolnir Widget (website)

A widget for moderating with mjolnir. Highly WIP. Use at your own risk!

MTRNord (they/them) reports

In the last 2 weeks, I increasingly had to learn how to moderate rooms properly, which brought up a lack of nice Mjölnir gui for me. Due to that, I just started to write one. It is at the time of writing still fairly young.

The current features are:

  • An overview of the ban list data the user is in (Not for the specific Mjölnir currently. Also requires a user to have joined the list room)
  • A quick form to ban a person
  • A form to redact someone or a message

Planned features are:

  • Support for showing MSC1929 information if available
  • Writing a patch for Mjölnir, so the widget can know which banlist the bot watches, so only relevant lists show up.
  • Editing the banlist (aka unbanning)
  • Adding support for more advanced features like deactivation of users and removal of media on the matrix-media-repo.
  • Covering most of Mjölnir's commands
  • Redact on ban and similar utilities you might want while banning.

Small getting started (it is simple :D)

To use it, you simply can add it to your Mjölnir Admin room by putting /addwidget https://moderation_widget.nordgedanken.dev?room_id=$matrix_room_id (the variable will get replaced automatically) in the message bar and pressing enter. The widget runs entirely client side, so this is not sending any events to my server. If you still are concerned due to the big amount of permissions asked, you can just build it yourself and host it.

Code and Room

Code is at https://github.com/MTRNord/matrix-moderation-widget Room is at: #mjolnir-widget:nordgedanken.dev

Dept of SDKs and Frameworks 🧰

mtxclient (website)

Client API library for Matrix, built on top of libcurl

Nico reports

You know what would be embarrassing? If changing the version number of something broke Nheko... Well, completely unrelated, mtxclient 0.6.2 is out now which fixes an issue where it would aggressively validate that version numbers started with an 'r'. Otherwise that release is API and ABI compatible, so if packagers could pick that up as a bugfix release into stable releases, that would be great!

Matrix-Rust-SDK-Bot-Framework-Helper (website)

A toolkit for writing commandbots more efficient in rust for matrix.

MTRNord (they/them) says

0.3.1

A small update was released, merging 2 month old PRs.

The changes are mainly features being now deactivated because we did not actually use them and fixing the example in one case. No updates to dependencies.

0.4.0

Dependencies have been updated to the newest versions.

Dept of Ops 🛠

andrewsh reports

The distribution-provided Debian packages for Synapse will only be provided for Bookworm (in testing/unstable) and Bullseye (in bullseye-backports). If you’re still using Buster (through buster-backports-sloppy), consider switching to Bullseye or, alternatively, to packages provided by the Synapse upstream. 1.52.0 is the last version to be provided for Buster through the backports repository.

saces reports

I did not find a docker container to ease/automate synapse and postgres maintenance, so I started one: https://gitlab.com/mb-saces/synatainer

Dept of Bots 🤖

reMarkable

Paul announces

I saw an interesting (to me) reMarkable telegram bot somewhere. But I prefer matrix and node.js was more difficult to deploy on embedded. So I wrote a reMarkable matrix bot in Go. https://gitlab.com/ptman/remarkable-matrix

matrix-ukraine-donation-bot

krazykirby99999 says

A Matrix chat bot to send donation links to aid Ukraine in the 2022 Russian invasion of Ukraine.

Setup

Install Python 3.8 or higher

Install python-poetry

python -m pip install poetry

Clone Repository

git clone https://github.com/KrazyKirby99999/matrix-ukraine-donation-bot.git

Install Dependencies

cd matrix-ukraine-donation-bot
python -m poetry install

Usage:

Set environment variables

HOMESERVER=https://matrix.org
USERNAME=matrix-ukraine-donation-bot
PASSWORD=password # or ACCESS_TOKEN=syt_...

Run main.py

python -m poetry run python main.py

Example:

Matrix Community Manager (website)

Looking for a bot to manage events and feedback from your community?

Sleuth reports

Looking for a bot to manage events and feedback from your community?

MCM an information bot. It manages the flow of information between community leaders and their community.

It aggregates messages from community members in several ways.

  • A @ mention. You can mention the bot with a message.
  • A Direct Message. Members of your community can message this bot privately.
  • hash tags. Using hash tags members of your community can send messages tagged to go to a specific back room.

You as an administrator of the bot can send timed announcements to any room using the built in matrix administration interface. You can also manage tags, add and remove admin of the bot, add automatic replies and more. All from the comforts of your Matrix client.

If you have any questions or want help please join #Matrix-Community-Manager:matrix.org Docs and installation guide can be found here

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server.

#ping:maunium.net

Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1maescool.be346
22808.nl387.5
3envs.net394
4converser.eu405
5carnot.cc518
6reactos.org519.5
7plocki.ddns.net556
8mozilla.org590
9matrix.fomin.site660.5
10adminctrl.com699

#ping-no-synapse:maunium.net

Join #ping-no-synapse:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1conduit.cyberdi.sk197
2rcp.tf224.5
3conduit.grich.sk234
4nognu.de261
5cry.is319.5
6sspaeth.de466
7matrix.sum7.eu485
8dendrite.neilalexander.dev565
9rustybever.be614
10matrix.awesomesheep48.me720.5

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Synapse 1.53 released

22.02.2022 14:53 — Releases Brendan Abolivier

Rejoice everyone, Synapse 1.53 is out! Let's have a look at what's new with this release.

Stabilisation of registration tokens

Registration tokens is a feature introduced in Synapse 1.42.0, which allows homeserver administrators to force their users to use specific tokens when registering. This is similar to Synapse's registration shared secret support, but with added features, such as the possibility to limit how users can be registered with the same token, or to make a token expire. See the admin API documentation for more information on how to manage registration tokens on your homeserver.

Registration tokens were initially proposed to the Matrix specification in MSC3231 by Callum Brown during their Google Summer of Code internship last summer. The MSC has since been accepted, and released in the stable Matrix specification as of Matrix v1.2. As a result, its Synapse implementation has been updated to remove support for unstable identifiers. Administrators of homeservers on which the reverse proxy rules explicitly allow the unstable route for this feature need to update their configuration. Same goes for developers of Matrix clients that support this feature. See the upgrade notes for more information.

Time-based cache expiry now enabled by default

To avoid being overly intensive on resources by making too many queries to the database, Synapse maintains several in-memory caches to store data it needs to use frequently. However, this comes with the inconvenience that, if Synapse needs to store too much data, these caches can become fairly big and occupy too much space in the host's memory.

Historically, Synapse has dealt with this issue by having set sizes for each cache, either hardcoded or set in the configuration, and evicting the oldest items when exceeding this size. Synapse 1.38 introduced the possibility for homeserver administrators to configure Synapse to evict cache entries based on the time they were last accessed on. This mechanism acts on top of the aforementioned eviction policy, and allows automatically evicting entries that haven't been accessed for some time, leaving more room in the caches to store data that needs to be accessed more often.

Synapse 1.53 enables this behaviour by default. Without specific configuration, Synapse will automatically evict cache entries that haven't been accessed for more than 30 minutes. Server administrators that were already using this feature might need to update their configuration, as this change deprecates the expiry_time configuration setting, which will be removed in a future version of Synapse. See the upgrade notes for more information.

Everything else

You might have heard that we're working on improving the time it takes to join big Matrix rooms with Synapse. If not, then you definitely want to have a look at the demos Matrix live that was published earlier this month and includes more details and a demo of the work we've been doing in this area.

This release of Synapse includes an implementation of MSC3706, which is part of this work. It's still very experimental and definitely not production-ready, but it's a huge stepping stone towards making room joins snappier than ever.

We've also been continuing our work towards enabling end-to-end encryption for application services (see the Synapse 1.50 release blogpost for more context on that). Synapse 1.53 includes support for sending to-device messages to application services. This is also still very experimental, watch this space for future updates.

See the full changelog for a complete list of changes in this release. Also please have a look at the upgrade notes for this version.

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including (in no particular order) Dirk Klimpel, Brad Jones, and Alexander Mnich.