The Matrix Space Beta!

17.05.2021 17:50 — Tech, GeneralMatthew Hodgson
Last update: 17.05.2021 17:35

Hi all,

As many know, over the years we've experimented with how to let users locate and curate sets of users and rooms in Matrix. Back in Nov 2017 we added 'groups' (aka 'communities') as a custom mechanism for this - introducing identifiers beginning with a + symbol to represent sets of rooms and users, like +matrix:matrix.org.

However, it rapidly became obvious that Communities had some major shortcomings. They ended up being an extensive and entirely new API surface (designed around letting you dynamically bridge the membership of a group through to a single source of truth like LDAP) - while in practice groups have enormous overlap with rooms: managing membership, inviting by email, access control, power levels, names, topics, avatars, etc. Meanwhile the custom groups API re-invented the wheel for things like pushing updates to the client (causing a whole suite of problems). So clients and servers alike ended up reimplementing large chunks of similar functionality for both rooms and groups.

And so almost before Communities were born, we started thinking about whether it would make more sense to model them as a special type of room, rather than being their own custom primitive. MSC1215 had the first thoughts on this in 2017, and then a formal proposal emerged at MSC1772 in Jan 2019. We started working on this in earnest at the end of 2020, and christened the new way of handling groups of rooms and users as... Spaces!

Spaces work as follows:

  • You can designate specific rooms as 'spaces', which contain other rooms.
  • You can have a nested hierarchy of spaces.
  • You can rapidly navigate around that hierarchy using the new 'space summary' (aka space-nav) API - MSC2946.
  • Spaces can be shared with other people publicly, or invite-only, or private for your own curation purposes.
  • Rooms can appear in multiple places in the hierarchy.
  • You can have 'secret' spaces where you group your own personal rooms and spaces into an existing hierarchy.

Today, we're ridiculously excited to be launching Space support as a beta in matrix-react-sdk and matrix-android-sdk2 (and thus Element Web/Desktop and Element Android) and Synapse 1.34.0 - so head over to your nearest Element, make sure it's connected to the latest Synapse (and that Synapse has Spaces enabled in its config) and find some Space to explore! #community:matrix.org might be a good start :)

The beta today gives us the bare essentials: and we haven't yet finished space-based access controls such as setting powerlevels in rooms based on space membership (MSC2962) or limiting who can join a room based on their space membership (MSC3083) - but these will be coming asap. We also need to figure out how to implement Flair on top of Spaces rather than Communities.

This is also a bit of a turning point in Matrix's architecture: we are now using rooms more and more as a generic way of modelling new features in Matrix. For instance, rooms could be used as a structured way of storing files (MSC3089); Reputation data (MSC2313) is stored in rooms; Threads can be stored in rooms (MSC2836); Extensible Profiles are proposed as rooms too (MSC1769). As such, this pushes us towards ensuring rooms are as lightweight as possible in Matrix - and that things like sync and changing profile scale independently of the number of rooms you're in. Spaces effectively gives us a way of creating a global decentralised filesystem hierarchy on top of Matrix - grouping the existing rooms of all flavours into an epic multiplayer tree of realtime data. It's like USENET had a baby with the Web!

For lots more info from the Element perspective, head over to the Element blog. Finally, the point of the beta is to gather feedback and fix bugs - so please go wild in Element reporting your first impressions and help us make Spaces as awesome as they deserve to be!

Thanks for flying Matrix into Space;

Matthew & the whole Spaces (and Matrix) team.

Synapse 1.34.0 released

17.05.2021 16:45 — ReleasesDan Callahan
Last update: 17.05.2021 15:13

Synapse 1.34.0 is now available, and it's loaded with new features and performance improvements.

Note: This release deprecates and replaces the room_invite_state_types configuration option. If you've customized that for your homeserver, please review the Upgrade Notes.

We've also marked the v1 room deletion Admin API as deprecated. Instead of sending a POST to a path ending in /delete, administrators are encourage to instead send an HTTP DELETE to /_synapse/admin/v1/rooms/<room_id>. Thanks to ThibF for implementing this (#9889).

Spaces

The highlight of this release is support for Spaces, now that MSC1772: Matrix Spaces has merged into the Matrix spec!

Synapse also has support for MSC2946: Spaces Summary and MSC3083: Restricting room membership based on space membership, but these are off by default as they're still under development. To enable these experimental MSCs, set experimental_features: { spaces_enabled: true } in your homeserver configuration. These are enabled on the matrix.org homeserver, and we encourage you to experiment with Spaces there and let us know in the Spaces Feedback Room if you encounter any issues.

Memory and Caching

Memory consumption and caching have been a major focus of the Synapse team this quarter, and we've made significant strides:

  • Synapse has a new gc_min_interval configuration option with reasonable defaults to prevent Python's garbage collector from running too frequently and thrashing when a large homeserver has its collection thresholds set too low.
  • Synapse will report memory allocation stats to Prometheus when using jemalloc.
  • Synapse will also measure Redis cache response times and report those to Prometheus.
  • For debugging, Synapse can optionally track the memory use of each LruCache.

We have a few more tricks up our sleeves; to learn more about how we're planning to improve the memory cost of joining large rooms, check out last week's Matrix Live.

Other Fixes and Improvements

We've also landed significant improvements to:

  • Sending events when Redis is available (#9905, #9950, #9951)
  • Joining large rooms when presence is enabled (#9910, #9916)
  • Backfilling history in large rooms (#9935)

...and fixed bugs that:

  • Prevented cross-account m.room_key_request messages from being delivered (#9961, #9965)
  • Incorrectly applied room creation / invitation rate limits to users and app services which should have been exempt (#9968)

The health check on our Docker images now responds more quickly upon successful startup thanks to improvements by maquis196 (#9913), and for especially privacy-conscious homeservers, device names can now be shielded over federation thanks to a contribution by aaronraimist (#9945).

New Access Token Format

Inspired by GitHub's new token format, we've restructured Synapse's access token format to follow the pattern:

syt_<UnpaddedBase64MXIDLocalPart>_<20RandomLetters>_<CRC32>

So a token for @example:matrix.org might look like:

syt_ZXhhbXBsZQ_KfJetOcLWEKCvYdKnQLV_0i3W80

Existing tokens remain valid; this is just for new tokens. We hope the new format reduces network overhead while also making it easier identify misplaced tokens in logs and repositories.

Everything Else

See the Upgrading Instructions and Release Notes for further information on this release.

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including aaronraimist, maquis196, and ThibF.

This Week in Matrix 2021-05-14

14.05.2021 00:00 — This Week in MatrixBen Parsons

Matrix Live 🎙

This week Dan talks to Erik about the state of Synapse generally, and in particular the work Erik has been doing for performance.

Plus! Bonus content, demos! Half-Shot on the MS Teams bridge work, and Hubert previews Matrix over MLS!

Dept of Spec 📜

Matthew reported:

Some combination of me, Kegan, Bruno and neilalexander have been working on v3 of the CS /sync API. (Today's /sync API in matrix is v2; v1 was the old /events API). We're not yet at the point of publishing a draft or MSC, but it's coming soon. It's really exciting work which flips Matrix around so that sync scales independently of the number of rooms you're in - and it's at last possible to write rich clients which only ever sync the bare minimum data needed to work: i.e. lazy loading eeeeeeeeverything. Watch this space :)

anoa told us:

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://spec.matrix.org/unstable/proposals.

MSC Status

New MSCs:

MSCs with proposed Final Comment Period:

  • No MSCs entered proposed FCP state this week.

MSCs in Final Comment Period:

  • No MSCs are in FCP.

Merged MSCs:

Merged Spec PRs

Here are all the actual MSC-related changes made to the raw Matrix spec this week!

Spec Updates

There's been activity from the Spec Core Team on a number of different MSCs, such as MSC3189 (per-room/space profile data). Additionally a spec PR for Matrix URI schemes has been getting feedback and is moving forward at a quick pace! There was also some feedback from the team on MSC2448 (blurhashes) which I'll get around to answering shortly 🙂.

Otherwise I think this week was a bit implementation-heavy for the team (the Spec Core Team is a task in addition to our full-time jobs). Hopefully next week will grant us more of a breather.

2021-05-14-wdmML-stacked_area_chart.png

Dept of Servers 🏢

Synapse

callahad announced:

Hello TWiMmers! The bulk of our update is in Matrix Live today, so go check out the video above ☝️ to hear about how we're reducing the amount of memory it takes to join large rooms, and why joins take so much memory in the first place.

Otherwise, we're mainly getting ready for the public debut of Spaces as a beta feature, but more on that next week... 😉

Oh, and before we go: please make sure your Synapse is up to date! We released 1.33.2 on Tuesday, which contains a low severity security fix.

Homeserver Deployment 📥️

Kubernetes

Ananace offered:

Another installation of the regularly scheduled Kubernetes Helm Chart updates (and another bump of the deprecated Synapse image). Now up to Synapse 1.33.2 and Element Web 1.7.27.

Dept of Bridges 🌉

matrix-puppeteer-line

Fair told us:

matrix-puppeteer-line: A bridge for LINE Messenger based on running LINE's Chrome extension in Puppeteer.

This week was spent on stability improvements & bug fixes.

Calling for testers! The bridge is at a point where it's mostly usable, but it still has quite a few blindspots. If anyone is willing to try it out & report issues, it would be a great help!

Discussion: Matrix-LINE Puppeteer bridge Issue page: https://src.miscworks.net/fair/matrix-puppeteer-line/issues

Shout-outs to @lecris:lecris.me for being an early tester & having given a lot of useful feedback 🙂

Gitter

Eric Eastwood reported:

Last time, we updated you on starting DM conversations with Gitter users from Matrix. Now we have the other side of this complete! From Gitter, you can now start a one to one conversation with someone you see from Matrix 😀

Just hover over their avatar to bring up the user popover, then press the "Chat privately" button. 🗣

2021-05-14-VYQjI-chrome_2021-05-12_18-38-44.png

matrix-appservice-irc 0.26.0

Half-Shot reported:

Goooooooood afternoon folks and happy Friday! This week we're announcing the 0.26.0 release of the matrix-appservice-irc bridge which contains precious goodies:

  • You can now disable the kick behaviour of the bridge on Matrix users if you are running a personal bridge, so losing your IRC connection no longer results in a kick.

  • You can now remove bridges from rooms by using the admin room, so no need to use the provisioning API or modify the DB.

  • We've added a new feature to allow you to specify bridge options on a per room basis using room state. At the moment you can modify the limits of the automatic pastebin system but more features like reply formats are to come!

As always please come tell us about it in #irc:matrix.org and make sure to check out the new docs if you get a bit stuck.

Dept of Clients 📱

Hydrogen

A minimal Matrix chat client, focused on performance, offline functionality, and broad browser support. https://github.com/vector-im/hydrogen-web/

Bruno said:

Hydrogen can now leave rooms and forget archived rooms. URLs are now also clickable in the timeline. Get the full details in the release notes!

2021-05-14-iIgrd-image.png

Element Clients

Updates from the teams

Web

  • Element Web 1.7.28 is up on staging, targeting Monday for release.
    • New spaces Beta (new way of grouping rooms and people)
    • Added support for slash commands working in edits
  • On develop:
    • Voice messages are nearing completion - enable the labs flag and give it a go :)
    • Performance improvements to app startup time. Let us know if you run into any issues!

iOS

  • 1.3.7 is available on TestFlight. It should be on the App Store on Monday. Spaces are not yet available on Element-iOS but the app offers minimal support. The release contains a fix for background crashed due to PushKit
  • At the platform level, we are still improving stability and performance:
    • Decryption operations to be moved outside the main thread
    • More robust on initial sync
    • etc

Android

Delight

  • “Spaces are coming” (I had heard something about that - BP)

Dept of SDKs and Frameworks 🧰

matrix-spring-boot-sdk 0.5.0

Benedict announced:

matrix-spring-boot-sdk 0.5.0 is released and uses Trixnity now.

Dept of Services 🚀

GoMatrixHosting v0.4.6 released!

Michael offered:

https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/1046/files

Just had a new release, GoMatrixHosting v0.4.6


* Add database purge section.

* Add markdown copy of User manual.
* Tweak compress state find rooms timeout.

* Update compile AWX instructions for custom branding and radius fix.

Visit #general:gomatrixhosting.com to join in the discussion.

GoMatrixHosting offer a great way to get your own homeserver without needing to run it yourself. Check them out!

Matrix in the News 📰

farribeiro shared:

Fedora promoting the matrix https://fedoramagazine.org/access-freenode-using-matrix-clients/

Final Thoughts 💭

From Dept of Humour in #twim:matrix.org 😅:

  • R1 Airport: Maybe I want to see if I can build some type of bridge between Matrix and CUPS.
  • hifi: the sad part is that would probably be more reliable than just printing directly

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server.

#ping:maunium.net

Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1neko.dev422
2xirion.net471
3matrix.kalleeen.fi507
4t2bot.io516
5alra.uk556
6thomcat.rocks767
7matrix.vgorcum.com800.5
8matrix.2kgwf.fi817
9nordgedanken.dev828.5
10trolla.us878

#ping-no-synapse:maunium.net

Join #ping-no-synapse:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1dendrite01.fiksel.info810.5
2dendrite.s3cr3t.me816
3lankolol.de931

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

This Week in Matrix 2021-05-07

07.05.2021 00:00 — This Week in MatrixBen Parsons

Matrix Live 🎙

Dept of Status of Matrix 🌡️

Matrix Server_Stats Bot

MTRNord announced:

The first iteration of visually representing the data gathered by Server Stats Discoverer (traveler bot) is now publicly available at https://serverstats.nordgedanken.dev/

It for now is only a graph of room relations but in the future is supposed to be extended for a server based graph as well as a Table to search your room within.

Be aware that the page is best viewed and used on desktop. Clicking a Room Node will open a new tab with the matrix.to link. If this fails this might be because of no canonical alias being available.

For the developers the data can be taken from https://serverstats.nordgedanken.dev/relations The format currently is only available in the d3js format but in the future that API also will be extended for different usecases.

For any feedback like accessibility issues or other issues please reach out to me either via DM or in #server_stats:nordgedanken.dev

2021-05-07-jh7X9-image.png

Dept of Spec 📜

Spec

anoa said:

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://spec.matrix.org/unstable/proposals.

MSC Status

New MSCs:

MSCs with proposed Final Comment Period:

  • No MSCs entered proposed FCP state this week.

MSCs in Final Comment Period:

Merged MSCs:

Closed MSCs:

Merged Spec PRs

Here are all the actual MSC-related changes to the raw Matrix spec that landed this week!

Spec Updates

A reminder that #sct-office:matrix.org is available to communicate directly with the Spec Core Team. A clarification from the last edition of TWIM is that this room is intended to be a low-traffic room solely for asking about the status of a/your MSC, rather than the Spec process or anything else. There is however #matrix-spec-process:matrix.org for discussion of the Spec process, and #matrix-spec:matrix.org for discussion of the Matrix spec and MSCs in general.

Otherwise the Spec Core Team has been doing a little bit of house-keeping. For those that have been living under a rock, Spaces is an upcoming feature intended to replace the old Groups/Community stuff with a much-improved implementation. And one that will actually make it into the spec! We've closed all old groups-related MSCs as they are now obsolete.

Additionally we've been giving some feedback on MSC2946 (Spaces Summary) which is another part of the Spaces puzzle (and is still a blocker for the release of the feature), as well as MSC3079 (Low Bandwidth CS API) which allows Matrix to operate on resource constrained devices and networks. Yours truly has also been making some PRs (one, two) to help clarify the Spec process.

2021-05-07-tEcUz-stacked_area_chart.png

Dept of P2P 👥

Pinecone

Neil Alexander reported:

Yesterday we announced the source code release of Pinecone, our new peer-to-peer overlay network, which we are developing under the P2P Matrix banner.

If you missed it, we wrote a blog post that explains some of the juicy details. The source code is available on GitHub.

Join us in #p2p:matrix.org for the latest P2P developments!

Dept of Servers 🏢

Synapse

Synapse is a popular homeserver written in Python.

callahad offered:

It's a release! Synapse 1.33 is out, and we plan to release a security patch for it on Tuesday, May 11th. This follows our previous discussion where we committed to trying to decouple routine security updates from our regular feature releases.

Read the release notes for details, but the big news is that we finally have experimental support for moving presence off of the main process. We're still testing it, but we hope it will allow instances that need presence to more easily scale out.

In last week's TWiM we shared a graph of Synapse's memory use when joining Matrix HQ for the first time. In particular, we saw a spike to 1.4 GB before settling at 800 MB.

In the week since producing that graph, we've managed to nearly eliminate the spike, halving it to 760 MB. After backfilling history, the room settles at around 650 MB:

2021-05-07-8eyfi-Screenshotfrom2021-05-0512-10-50.png

These changes are still a work in progress, but we hope to get them merged into Synapse in time for the 1.35 release on June 1st.

Additionally, Nico (@deepbluev7:neko.dev) has a request:

I wrote a patch for synapse, that reduces the size of almost empty incremental syncs by 50% (30% if you include http headers). If you are a client developer, you may want to test your client against a synapse with that patch applied, since it broke quite a few clients, that relied on synapse sending empty fields. While synapse sends empty fields, other server implementations, like conduit, don't, so fixing any issues here will help with portability across different server implementations too. With a bit of hope this patch can actually be applied in a few weeks to the official synapse, but it was backed out from the next RC because of the breakage. So if you can, please test your client, which you are developing, against the following PR and fix any issues you experience from it: https://github.com/matrix-org/synapse/pull/9919

Thank you!

Homeserver Deployment 📥️

Kubernetes

Ananace said:

The regular updates for my Helm charts (and still deprecated Synapse image) have been pushed, for Synapse 1.33.0/1 and the Matrix Media Repo 1.2.8. (technically last week, but it was after friday, so I'm throwing it in again)

Dept of Bridges 🌉

mautrix-imessage on iOS

Tulir announced:

The initial alpha version of Brooklyn is now public, which means you can now (try to) use mautrix-imessage on a jailbroken iOS device for iMessage bridging. Setup instructions are on docs.mau.fi: https://docs.mau.fi/bridges/go/imessage/ios/setup.html

Brooklyn was developed by ethanrdoesmc. It's an app/tweak that handles communicating with iMessage and runs mautrix-imessage as a subprocess for the Matrix side. The initial alpha supports basic text and media message bridging. Sending and receiving tapbacks, replies, read receipts and typing notifications will also be supported in the future.

Feedback is welcome in the mautrix-imessage room: #imessage:maunium.net

Heisenbridge

hifi told us:

Heisenbridge the bouncer style Matrix IRC bridge has seen numerous updates in the past week:

  • Identd implementation to get verified usernames on IRC

  • TLS support for IRC connections

  • IRC excess flood prevention with a buffer

  • Proper long message splitting from Matrix to IRC

  • Retry support for Matrix requests to work around homeserver downtime/restarts

Minor fixes to ghosting issues and some other stuff. This will be the last big update for a while as it has mostly stabilized enough for daily use.

More testers are still welcome to get the remaining issues ironed out so if you need to connect to unbridged and unplumbed networks and run your own homeserver it would be an excellent time to try it out.

Thanks!

matrix-pstn-bridge

KB1RD reported:

Experimental support for call bridging has landed. Now you can call phone numbers right from Matrix, with partial support for MSC2746. A few things to keep in mind:

  • The bridge is still in alpha -- I wouldn't trust it for anything secure at the moment. I would love to hear feedback from tests, though!

  • If you tried any of the earlier versions, you will need to re-run the link command since I made some breaking changes. Sorry.

Just a reminder, the GH repo is here and the Matrix room is #matrix-pstn-bridge:kb1rd.net . There's also a general VoIP bridging room at #matrix-voip-bridging:kb1rd.net.

matrix-appservice-irc has a new release candidate

Christian reported:

The IRC bridge matrix-appservice-irc has a new release candidate. The upcoming version 0.26.0 will include many features and bug fixes. Here are three highlights:

  • Allow third-party bridged users to change their nickname with the self-serve command !irc nick anothername (thanks vranki)

  • Allow room moderators and bridge admins to unlink rooms using the !unlink command

  • Add support for specifying the paste bin limit in room state with the org.matrix.appservice-irc.config event type.

Please test it and flag any issues you have upgrading:

https://github.com/matrix-org/matrix-appservice-irc/releases/tag/v0.26.0-rc1

Dept of Clients 📱

FluffyChat

FluffyChat is the cutest cross-platform matrix client. It is available for Android, iOS, Web and Desktop.

krille announced:

FluffyChat 0.30.0 has been released

In this release we have mostly focused on bugfixing and stability. We have switched to the new Flutter 2 framework and have done a lot of refactoring under the hood. The annoying freezing bug should now be fixed. Voice messages now have a new backend which should improve the sound quality and stability. There is now a more professional UI for editing aliases of a room. Users can now see a list of all aliases, add new aliases, delete them and mark one alias as the canonical (or main) alias. Some minor design changes and design fixes should improve the overall UX of the app exspecially on tablets.

Version 0.30.0 will be the first version with arm64 support. You can download binaries from the CI and we will try to publish it on Flathub. Together with the new Linux Desktop Notifications feature, this might be interesting for the Librem 5 or the PinePhone. Sadly I don't own one of these very interesting devices. If you have one, I would very like to see some screenshots of it! :-)

Additionally, Shine told us:

Fluffychat update: Native Fluffychat Linux build now works well on aarch64 devices!

If you want to try the binary from CI, keep in mind that GDK_GL=gles needs to be set to force it to use the OpenGL ES. Flatpak build on Flathub already has it set up and works out of the box.

Some distributions can have issues with input fields and virtual keyboard, making it look like the input is set up as right-to-left. To my understanding, it's an issue of certain older GTK versions with Flutter.

2021-05-07-El6cc-fluffychat_pinetab.jpg

Nice sticker

Nheko

Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE and intends to be full featured and nice to look at

Nico (@deepbluev7:neko.dev) told us:

Nheko now should stop showing you actions, that you can't do anyway, because that is confusing and useless. This includes the following and more:

  • Invite members, if you have no invite permissions.

  • Delete messages, if you don't have redaction permissions.

  • Send a message, reply to one, edit one and more, if you can't even send the message!

Tell us, if we missed something or we removed an action, that you actually have permissions to do!

Apart from that we also started working on some of the features for the next major release. That release will mostly focus on bringing End-to-End-Encryption out of beta. As a first step, Nheko now shows if a message was sent from a verified device or not. This has 3 different trust levels:

  • green: The message is from a device you verified. Either by device verification or cross signing.

  • greyish: The message is from an unverified device, but that user has never rotated their master signing key and has cross-signed that device. As such we can probably assume, that this is a trusted device. For extra safety, you should of course verify that user, but if it is just an internet personality you will never meet, you trust you are speaking to the right party anyway and can't really verify them anyway, since you don't know how they look either! We don't want to prompt users with red warning signs in such cases, which will lead to them not doing verification properly, because they just want the red marks to go away. Many people are also not interested in the MITM aspects of E2EE. We think this is an okay tradeoff, but any feedback is welcome of course! This is basically Trust On First Use (TOFU), if you heard that term before.

  • red: The device is not verified. This can have a few reasons. Either we verified the user, but they didn't verify that device. Or we didn't verify that user and they have changed their master key at some point. Or the signatures for that device are wrong, etc. If you see such a device, you should probably investigate, why that is the case.

These are some biggish changes, so if you experience issues, tell us in #nheko:nheko.im!

Fractal

Alexandre Franke reported:

Hot on the heels of our previous announcement and building on the fresh foundations of our rewrite, Julian got busy on room history. It now appears when a room is selected in the sidebar! To make that selection easier, room filtering in the sidebar has been implemented by new contributor Veli Tasali. Once a room is selected and displayed, messages can even be sent to them! Sounds like we’re done and the client is ready… SHIP IT! Not really though, as it’s still very basic, but at least the bare minimum to make it actually usable is now here.

As a nice bonus, we do lazy loading, and rooms are now added to the sidebar in batch on startup to ensure good performances. The sidebar also got some refinements from Kévin (!732 and !736).

Finally Veli also helped us get the docs for fractal-next published, lowering the barrier to entry for other new contributors. They are available at https://gnome.pages.gitlab.gnome.org/fractal/fractal/.

For further details, Julian published a technical overview of the internals of the new codebase.

Element Clients

Updates provided by the teams

Delight

  • We’ve been shepherding MSC1772 into the spec which has now exited final comment period and merged!
  • Alongside, we’ve also been iterating on the Spaces implementations all round in preparation for wider testing soon, which has included
    • Iterating on filtering on Web to filter all Spaces
    • Iterating on logic for showing notification badges to avoid single DMs spawning multiple badges
    • Iterating on ‘Home’ to instead behave more like ‘All’
    • Iterating on implementations across the web, iOS, Android & Synapse to use stable prefixes
    • & lots of other small tweaks

Web

  • Element Web 1.7.27-rc.1 on staging
    • Added localisation support to the desktop layer (for menu items etc.)
    • Fixed encrypted search indexing on Windows
    • Hardware media keys are now ignored, so they'll go to other apps as intended
  • On develop
    • Calling architecture reworked to support multiple streams, please report any issues
  • 1.7.27 release planned for Monday

iOS

  • 1.3.6 is in review for the App Store. We have polished and fixed several issues on 1:1 and group calls. The release contains fixes for several bugs and crashes.

Very excited about the Spaces progress! Looks like everything that I found in recent testing is fixed!

Dept of SDKs and Frameworks 🧰

Trixnity released

Benedict announced:

Trixnity finally got released this week. It's a Kotlin multiplatform Matrix SDK for high level access to Client-Server API and Appservice API. It has all (and some more) features from matrix-spring-boot-sdk which is based on Trixnity now (an update for that will be released soon). Trixnity can currently be used on JVM and JS as platform (Native is not working yet). It is also very customizable by adding custom room and state event types.

Dept of Interesting Projects 🛰️

Cactus Comments 🌵

Asbjørn offered:

Cactus Comments is a federated comment system for the open web built on Matrix.

We released the Cactus Comments Web Client v0.9.0!

This version introduces a display name field for guest users, so guest users finally have sensible names.

It also brings the option to use HTML data attributes for configuration.

  • Comment sections can now be initialized using data-* attributes on the script tag (Thanks to @NicolaiSoeborg for !5).

  • Allow using strings for any config parameter, including booleans and numbers.

  • Users can now set a displayname when commenting as a guest.

  • Some styling improvements for text inputs.

/ipns/latest.cactus.chat is updated to point to the latest release, so sites linking there should already be using the new version.

Come play with the demo: https://cactus.chat/demo

Join our Matrix room: #cactus:cactus.chat

Phonehome stats viz

Matthew reported:

I did a video a few months back trying to show the physical layout of Matrix over the years by looking at the phonehome stats and the number of active users per server (showing full-mesh edges between the top 100 servers, heatmapped by how busy the servers were at either end of the edge). There was a bug in the phonehome stats during 2017, but it's still fairly cool...

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server.

#ping:maunium.net

Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1kapsi.fi611
2queersin.space623.5
3maescool.be678.5
4chatcloud.net702.5
5woefdram.nl743
6matrix.sp-codes.de769.5
7dodsorf.as788
8wallace.sh803
9kittenface.studio1180
10dendrite.foxomy.com1320

#ping-no-synapse:maunium.net

Join #ping-no-synapse:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1dendrite.thomcat.rocks493
2dendrite.s3cr3t.me625.5
3dendrite01.fiksel.info1086
4dendrite.foxomy.com1347

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Introducing the Pinecone overlay network

06.05.2021 00:00 — TechNeil Alexander

Since the end of 2019, we have spent quite a bit of time thinking about and exploring different technologies whilst building various demos for P2P Matrix. Our mission for P2P Matrix is to evolve Matrix into a hybrid between today's server-oriented network and a pure P2P network - empowering users to have total autonomy and privacy over their data if they want (by storing it in P2P Matrix, by embedding their server into their Matrix client), while also letting users store their data in serverside nodes if they so desire.

The goal is to protect metadata much better (as users no longer have to depend on a server run by someone else to communicate), as well as drive new features such as account portability, multi-homed accounts, low-bandwidth Matrix and smarter federation transports - and provide support for internet-less mesh communication via Matrix which can also interoperate with the wider network. You can read more about it in our Introducing P2P Matrix blog post from last summer, or watch our FOSDEM 2021 talk where we previewed Pinecone. It's important to note that this has been a small but important long-term project for Matrix, and has been progressing entirely outside our business-as-usual work of improving the core protocol and reference implementations.

As the project has progressed, we've built a variety of prototypes using existing libraries (go-libp2p, js-libp2p and Yggdrasil), demonstrating what an early P2P Matrix might feel like if it were running on a mobile device, in the web browser and so on using such an overlay network. Each of these demos has taught us something new, and so in October 2020 we decided to take this knowledge to build an experimental new overlay network of our own.

Pinecone is designed to provide end-to-end encrypted communications between devices, regardless of how they are connected to one another, in a lightweight and self-arranging fashion. The routing protocol is a hybrid, taking inspiration from Yggdrasil by building a global spanning tree, but rather than forwarding all traffic using the spanning tree topology, we use it as a bootstrap routing mechanism for a line/snake topology, ordered by their ed25519 public keys, which we have affectionately named SNEK (Sequentially Networked Edwards Key) routing.

Nodes seek out their closest keyspace neighbours on the network and paths are built between these pairs of nodes, similar to how a Chord DHT functions, populating the routing tables of intermediate nodes in the process. These paths are then used to forward traffic without having to perform up-front searches, allowing for very fast connection setups between overlay nodes. These paths are resilient to network topology changes and handle node mobility considerably better than any other name-independent routing scheme that we have seen — early results are very promising so far. We have also been experimenting with a combination of the μTP (Micro Transport Protocol) and TLS to provide stateful connection setup, congestion control and end-to-end encryption for all federation traffic carried over the Pinecone network.

Pinecone simulator showing line/snake logical network topology

If Pinecone works out, our intention is to collaborate with the libp2p and IPFS team to incorporate Pinecone routing into libp2p (if they'll have us!) while incorporating their gossipsub routing to improve Matrix federation... and get the best of both worlds :)

Today we're releasing the source code for our current early implementation of Pinecone — you can get it from GitHub right now! It's very experimental still and not very well optimised yet, but it is the foundation of our latest mobile P2P Matrix demos, which support P2P Matrix over both Bluetooth Low Energy mesh networks, multicast DNS discovery within a LAN, and/or by routing through static Pinecone peers on the Internet:

  • Android: https://appdistribution.firebase.dev/i/394600067ea8ba37
  • iOS: https://testflight.apple.com/join/Tgh2MEk6

Building a routing overlay is only the first step in the journey towards P2P Matrix. We will also be looking closely in the coming months at improving the Matrix federation protocol to work well in mixed-connectivity scenarios (rather than the full mesh approach used today) as well as decentralised identities, hybrid deployments with existing homeservers and getting Dendrite (the Matrix homeserver which is embedded into the current P2P demos) more stable and feature-complete.

The long-term plan could look something like this:

Diagram showing possible P2P Matrix stack

Most discussion around P2P Matrix takes place in #p2p:matrix.org, so if you are interested in what's going on, please join us there!

Synapse 1.33.0 released

05.05.2021 00:00 — ReleasesDan Callahan

Synapse 1.33.0 is out! Three main items of note:

  1. We plan to release 1.33.1 1.33.2 with a low severity security fix on Tuesday next week, and we're interested in your thoughts on decoupling routine security fixes from normal releases. Please weigh in on this discussion.

    Note: We shipped 1.33.1 with a small dependency fix when installing Synapse via pip. A security release is still planned for Tuesday, which will now be 1.33.2.

  2. If you use Synapse's optional account revalidation feature (see account_validity in config.yaml), you'll want to review the upgrading instructions as we've made a few small changes to the email templates it uses.

  3. Synapse now has very experimental support for moving presence off of the main process. This has not yet been extensively validated, so please proceed with caution. We expect to get this to a point where we can confidently recommend it in the coming weeks.

Otherwise, this is another release focused on internals. We're driving toward a goal of reducing excess memory consumption when joining large or complex rooms, and most of our effort (aside from the presence work) has been focused on measurement, instrumentation, and experimentation for that.

We did manage to slightly speed up room joins, improve the performance of the user directory, and refine our implementation of MSC3083. Additionally, thanks to work by ShadowJonathan, Synapse now passes all of flake8-bugbear's lints.

See the Upgrading Instructions and Release Notes for further information.

Thank You

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including rkfg, and ShadowJonathan.

This Week in Matrix 2021-04-30

30.04.2021 00:00 — This Week in MatrixBen Parsons

Matrix Live 🎙

Matthew presents the MSC process! Get slides here.

Dept of Status of Matrix 🌡️

Oda Nobunaga said:

The German IT platform Heise has conducted an interview with the university of Innsbruck which chose Matrix/Element over Microsoft Teams. Reasons: it's free, decentralized/federated, expandable and secure.

https://www.heise.de/news/Zusammenarbeit-MS-Teams-als-Lockangebot-in-eine-geschlossene-Microsoft-Umgebung-6030514.html

"free, decentralized/federated, expandable and secure" sounds like a great set of reasons!

Heise article, naturally, is in German. Element have some content in English discussing the deployment, and how well-suited Matrix is for universities. Finally, the best kind of feedback: an Innsbruck professor cheers the utility of federation:

And the best thing: The federation across different institutions really works. I can still participate in project related chats with my former team @ETH_en while seamlessly supporting lectures at @uniinnsbruck. All out of one tool, one login. pic.twitter.com/6LXHcu7Jjf

— Jan Beutel (@jan_beutel) April 30, 2021

Famedly Fellows on Linux Lounge

Nico offered:

Btw if anyone wanted to hear Krille and me live, we are live at the linux lounge now. It's in GERMAN! https://theradio.cc/blog/category/shownotes/linuxlounge/

Exciting! Possibly. For the lucky few who can understand it anyway!

Lazalatin added:

If anyone did miss the opportunity to listen to the live stream: Here you can find the recording, with show notes, as well: https://rec.theradio.cc/item/ll244/
We were happy to have Krille and Nico with us. 🥳
(website in german as well as the stream recording)

:D

Dept of Spec 📜

Spec

anoa offered:

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://spec.matrix.org/unstable/proposals.

MSC Status

New MSCs:

  • There were no new MSCs created this week.

MSCs with proposed Final Comment Period:

  • No MSCs entered proposed FCP state this week.

MSCs in Final Comment Period:

Merged MSCs:

Abandoned MSCs:

Spec Updates

We've opened up a new room for MSC authors to come and ask about the MSC process, as well as for review and feedback on their MSC ideas over at #sct-office:matrix.org. If you're unsure what the next steps for your MSC are, or aren't sure whether an idea would make sense for inclusion in the Matrix Specification, then this room is the perfect place to ask about it. Come on by!

In terms of MSC work, Spaces remains one of the most active topics in the spec this week, with MSC1772 finally reaching Final Comment Period! A lot of work went into writing and reviewing the MSC, as well as crafting the implementation - so congratulations to everyone involved! Of course, any concerns can still be raised over the next 5 days by anyone before the proposal is ultimately merged to the spec.

There are a number of changes being merged to the spec as we move swiftly towards the next major Spec release. Those that have merged this week are:

Otherwise there's been a smattering of discussion and updates on various MSCs, such as MSC2730 (verifiable forwarded events), MSC2249 (requiring a user to be able to see an event to report it) and MSC2516 (voice messages).

Thanks for reading!

Also - knocking is now published to the unstable spec, check it out at: https://spec.matrix.org/unstable/client-server-api/#knocking-on-rooms

2021-04-30-iLQCH-stacked_area_chart.png

Dept of P2P 👥

P2P Matrix

Neil Alexander announced:

We've just released the first public P2P Matrix demo using the experimental Pinecone protocol. There are builds available for Android right now and iOS is following shortly via Testflight as soon as it passes Apple's beta review:

  • Android 10+: https://appdistribution.firebase.dev/i/394600067ea8ba37

  • iOS: https://testflight.apple.com/join/Tgh2MEk6 (wait for build 73!)

If you enable the static peer in the peer-to-peer settings, you can join us in the #beachparty:3bf0258d23c60952639cc4c69c71d1508a7d43a0475d9000ff900a1848411ec7 room!

The demo will attempt to connect to other devices using Bluetooth Low Energy, even if internet connectivity isn't available. Local connectivity over Wi-Fi is also supported. It's still very rough around the edges and there are lots of bugs, but feel free to give it a go!

Join #p2p:matrix.org for more chat and to provide feedback!

Dept of Servers 🏢

Conduit

Conduit is a Matrix homeserver written in Rust https://conduit.rs

timokoesters said:

  • Feature: Implement /devices

  • Feature: Send invites over federation

  • Improvement: Verify signatures for incoming requests

  • Improvement: Room directory now loads a lot faster

  • Fix: Retrying transactions works correctly now

We managed to work on a few features the last two weeks, but there are some very annoying issues with major Matrix clients:

Synapse

Synapse is a popular homeserver written in Python.

callahad said:

We expect to release 1.33 on Tuesday; more on that next week. But generally speaking, much of our development focus has been on profiling the memory usage of small homeservers joining large, highly-federated rooms for the first time. For example, we see a graph like this when joining Matrix HQ:

2021-04-30-akzM5-Screenshotfrom2021-04-2918-33-43.png

At a first pass, we're aiming to significantly reduce the initial spikiness of that graph, and we think #9910 will help with the subsequent slow ramp up in memory.

We've also been discussing how to balance our relatively rapid release cadence with managing low severity security issues. If you have any thoughts, preferences, or relevant experiences to share, please comment here!

Dept of Bridges 🌉

Heisenbridge announced

hifi said:

Announcing a new bouncer style Matrix IRC bridge to bridge the gap between IRC and Matrix in a way that's easy to configure and manage by homeserver administrators for small scale deployment.

Meet Heisenbridge https://github.com/hifi/heisenbridge 🥳 (it's a Breaking Bad reference 🤦)

It's written in Python and started off as an experiment if it's possible to create a somewhat well performing stateless IRC bridge that's practically almost zero configuration. It quickly found a use case of being able to join !channels on IRCnet that hasn't been possible in the past.

Intended target audience are people who run their own homeserver and would like to transition from a combination like ssh+irssi to pure Matrix (with weechat-matrix, of course!) but still keep IRC close without losing the "fine details" of the protocol that happens when plumbing a room properly.

As the bridge is written by someone who doesn't really know Python help is much appreciated in the form of filing issues, sending pull requests or just discussing about it on Matrix to tell how you would like to use it!

Development and support in #heisenbridge:vi.fi

Thanks!

matrix-puppeteer-line updated

Fair offered:

matrix-puppeteer-line: A bridge for LINE Messenger based on running LINE's Chrome extension in Puppeteer.

Updates:

  • Inbound read receipts! But with a view caveats:

    • Receipts are only fetched for the "most recently active" room.

      • Example: if you send a message in room A, then room B, receipts will only be fetched for room B and not room A, until you (or someone else, from LINE) posts in room A.

      • This will be fixed in the next update. I have an idea for some Puppeteer trickery to get it working.

    • Read receipts in group chats are bridged as annotations (reactions) instead of "real" read receipts, until all members of the chat have read a message.

      • This is because LINE's read receipts for group chats don't tell show who read a message, but only how many people a message was read by.
      • To capture this in Matrix, instead of sending read receipts for a group chat message, the bridge puts a reaction of "Read by #" on it, with "#" matching how many people read the message so far. Once everyone in the chat has read a message, the reaction is removed and a "real" receipt is sent for all users.
  • Internal changes to message syncing that should hopefully make inbound messages more reliable, or the very least improve code maintenance.

Discussion:

#matrix-puppeteer-line:miscworks.net

Dept of Clients 📱

Nheko

Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE and intends to be full featured and nice to look at

Nico (@deepbluev7:neko.dev) offered:

Hey there, it is your friendly neighborhood cat client! Pardon, chat client!

Jedi18 implemented forwarding of messages. Just select a message and press Alt-F or use the context menu to send a message to the room you select in the popup! This encrypts and decrypts messages and media appropriately, but be aware that sending an encrypted file to an unencrypted room uploads the file unencrypted. (In all other cases the media is not uploaded again.)

Furthermore matrix.to links are now rewritten to matrix:-URIs internally. This means joining rooms and such should now work the same across both of them. It also makes navigation to events work now and even has a shiny highlight effect now! You can also create links to events now from the context menu. These are generated as matrix.to links, since currently the onboarding experience should still be better. Once the next version of the matrix.to website gets deployed, you can even just click there to open the event in Nheko! (Instead of having to copy a /join command).

Since "Copy link to event" was confusing to people and AppAraat opened his present box, only to realize it did not contain "Copy link location", we implemented that too as well as a copy action, to copy the event body for good measure. I hope everyone who got confused will forgive me!

Apart from that we also updated our screenshots on https://nheko-reborn.github.io/ (and all AppStream based store pages) to be higher resolution and reflect the current design. Furthermore we fixed the login on conduit homeservers and that no rooms were shown, if the server didn't support the groups API.

That's all, and keep on c(h)atting! <3

2021-04-30-zKbR8-nheko-message-fwd.png

NeoChat

Carl Schwan told us:

We rewrote the room managing code moving it from QML to C++ and unify how we manage rooms in NeoChat. This allowed us to support the matrix-URIs scheme, making it possible to open rooms and show user information when clicking on a matrix: link in your browser. Sverin Saji improved the look of the typing indicator and Jan Blackquill improved the keyboard navigation.

2021-04-30-JcL3Q-image.png

Watch The Matrix

2021-04-30-boDeS-Icon.png

Doug offered:

This week I published the first public build on TestFlight:

https://testflight.apple.com/join/a06n3gN4

There are a few reports of issues when signing in which I'm looking into. A few other changes include:

  • Round complications show [ ] instead of an empty icon (some have been disabled for now).

  • When signing in, the homeserver can be detected from the username.

  • Various performance and bug fixes for sync responses.

  • The app now has an icon and more purple accents.

As requested by benpa after my last update, here's a video of it running on a device:

/me doesn't need to buy an Apple Watch ... /me doesn't need to buy an Apple Watch

Element Clients

Updates from the teams.

Delight

  • We’ve been maturing and iterating on MSC1772 in anticipating of wider testing of Spaces in the near future
  • On Web, we’ve been iterating on designs & implementations for a beta, including tweaking filtering to display rooms from all Spaces
  • On Android, we’ve been doing the same, and have merged the initial Spaces implementation onto develop
  • On Synapse, we’ve been implementing restricting rooms based on the Space Summary API

Web

  • 1.7.26 released on Monday
    • Added persistence of unsent messages across app restart
    • Improved room list filtering performance
    • Improved the image detail view
  • On develop
  • In progress

iOS

  • 1.3.5 has been published to the App Store on Monday.
  • We have been polishing the new VoIP design this week. It should be merged into develop early next week and released in the App Store the week after.
  • We want to focus on code quality. We set up GitHub actions in our 3 repos. They run both unit tests and integration tests. We need to fix some of our 600+ tests in the SDK but we now have a CI able to complain about test failures.

Android

  • The Spaces have landed on Element Android, on the develop branch. We are working to stabilize the feature before the release in beta next week.
  • In parallel, we are working to stabilize the whole application, and provide a better support on Android 11.
  • Also it will be possible in the next release to compress video before sending, which is a very expected feature.

Hydrogen

A minimal Matrix chat client, focused on performance, offline functionality, and broad browser support. https://github.com/vector-im/hydrogen-web/

Bruno said:

Invites and other goodies have landed in 0.1.46 & 0.1.47!

2021-04-30-dyAS7-Screenshot_20210428-143543.png

If you spend any amount of time with Bruno, eventually the topic of sourdough will arise - now it's even sneaking into his updates!

Dept of VoIP 🤙

matrix-pstn-bridge

KB1RD told us:

I announced this project last week. It's exactly what it sounds like: A puppetting bridge to puppet a telephone number from Matrix. It's currently pre-alpha and not ready for use on public-facing HSes.

Not too much has happened in the last week as I've been a bit busy. The main thing is that there's now a dev branch where outbound voice calls work. Other than the 2014 SIP experiment, I think this is the first bridge to make voice calls work?

I hope having a reference VoIP bridge can help pave the way for more VoIP bridging. I'm already looking into how code can be generalized for other bridges. For example, I've opened an issue in matrix-bot-sdk to add helper classes and stuff for VoIP support. I'd be curious to see how other bridge builders think this could be accomplished. 👀

If you're interested in PSTN bridging specifically, there's a room for that.

If you're interested in VoIP bridging in general, there's a room for that, too

Dept of Services 🚀

etke.cc - like Element Matrix Services, but on your servers, under your control and without restrictions

Aine reported:

Some context: I like how simple EMS allows you to setup the Matrix homeserver, but it lacks any internal tuning or customization ability. Do you want more? OK, setup it yourself and try to stay normal while trying to configure coturn to properly run in docker cluster. Because of that, I did a service where you can order initial setup and configuration of homeserver (yes, with coturn in docker cluster!) and with full control of customization and fine-tuning of manual configuration (website even has step-by-step guide how the process works).

It's based on the amazing spantaleev/matrix-docker-ansible-deploy with additional stuff like system security hardening and maintenance

Announcements: #announcements:etke.cc

Source: https://gitlab.com/rakshazi/matrix-ansible/ It contains basic security stuff like ufw and fail2ban, system maintenance like updating packages, cleanup of old logs and docker images, configuration of swap, some additions like absurdly simply healthcheck page and website deployment to base domain. Documentation may be found in readme files for each role

GoMatrixHosting v0.4.5 is here!

Michael said:

GoMatrixHosting v0.4.5 is here!

https://gitlab.com/GoMatrixHosting

The FLOSS Matrix hosting system built on AWX, it enables you to manage multiple Matrix servers for many clients in both a commercial or non-commercial context. We are currently doing a free giveaway of on-premises server plans, if you're an experienced user or Matrix admin you can receive a free on-premises subscription and hopefully provide us with some feedback.

For more updates on this project visit our Matrix room: #general:gomatrixhosting.com

GoMatrixHosting was previously Oasis Hosting - looks like they're making great progress improving their product, may be time to have another look.

Dept of Interesting Projects 🛰️

Matrix_Stats Discoverer (bot) ( @server_stats:nordgedanken.dev )

MTRNord reported:

Some of you might already have seen that bot show up in rooms. To clear up some confusion around it I will briefly explain what it does and what it is supposed to do:

What does it do?

The Bot is simply existing to do the same task as Matrix Traveler (bot) already did for years now. It joins rooms, listens for an alias, tries to join it.

The main difference between these 2 bots is that my bot also searches for aliases in old messages.

What does it save?

It only saves the relations between rooms. This means it knows which room was posted where. It doesn't know who posted it, when it was posted, why it was posted or any other content of messages.

However due to matrix' nature it does need to get the data once onto a synapse to read trough it. This means it has the data on my synapse. (Running in helsinki at hetzner).

What can I do if I do not want it?

The easiest way is to just ban it. It will fully forget that room.

Kicking works too but it might again join if it gets found again.

Where is the source?

The source code is available at https://git.nordgedanken.dev/MTRNord/server_stats/-/blob/main/src/bot/mod.rs

For further questions feel free to join #server_stats:nordgedanken.dev

Also note this bot is still very early in development which is why it does not respond to you yet on any commands. This will be added in the near future as well as a readme in the repository

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server.

#ping:maunium.net

Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1maunium.net670.5
2matrix.org739.5
3matrix.vgorcum.com1068
4helsinki-systems.de1072
5kif.rocks1085
6dendrite.foxomy.com1092
7kittenface.studio1549
8thomcat.rocks1674
9mailstation.de2011
10chatcloud.net2311

#ping-no-synapse:maunium.net

Join #ping-no-synapse:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1dendrite.thomcat.rocks415
2dendrite.foxomy.com557
3dendrite.neilalexander.dev576
4dendrite01.fiksel.info1086

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

This Week in Matrix 2021-04-23

23.04.2021 00:00 — This Week in MatrixBen Parsons

Dept of Status of Matrix 🌡️

Open Tech Will Save Us news

Neil announced:

Folks, incredibly Open Tech Will Save Us is over a year old! Thanks to everyone who has made this possible, especially Ben. We’d like to figure out how to improve it and make it even better. If you have a few moments, please can take a look at this survey to share your thoughts? (Sorry it’s Google ...)

Matrix URI scheme

Jonathan noticed:

The matrix: URI scheme has just been officially accepted (safelisted) into the HTML standard! 🚀

https://github.com/whatwg/html/pull/6320

Nico (@deepbluev7:neko.dev) said:

What is the Matrix URI scheme?

You are probably familiar with mailto:[email protected] or tel:+1-555-438-3732. The former is usually used to open an email address in your mail client, while the latter is used for phone numbers. This allows applications to register themselves as a handler for it, like for example GMail or Thunderbird, and your browser and other applications can find them and tell them to open an editor or similar, with that identifier as the target. There are also a lot of other applications that have similar schemes like Telegram, Zoom and more.

In Matrix you have similar identifiers, most notably users (@user:example.org) and rooms (#room:example.org). The matrix URI scheme now allows you to open such identifiers in your favourite Matrix clients. You can now just click on matrix:u/user:example.org or matrix:r/room:example.org and start chatting with the user or join the room. (There is a bit more to it, you actually need an action after the identifier for it, otherwise it will just open the user profile or show the room, if you are joined to it, but that is the idea. For more details, check https://github.com/matrix-org/matrix-doc/blob/master/proposals/2312-matrix-uri.md )

As you can see the format of the matrix: scheme is a bit different that standard Matrix identifiers. This is mostly because they need to start with scheme:, where scheme is different depending on the application, but some characters are also treated differently in URIs, which is why the matrix scheme decided to avoid them where possible.

What does it mean, that the matrix: scheme is now part of the HTML standard?

In general desktop and mobile clients could already implement the Matrix scheme before, FluffyChat, Quaternion, gomuks and Nheko have even done so already. This means if you are using those clients, you can already use the Matrix scheme today!

On the web things are a bit more difficult though. Since you don't want arbitrary websites to intercept arbitrary schemes, schemes need to be safelisted in the HTML spec/browsers. So today web clients can't be registered to open matrix: URIs. But the first step is done now. The HTML spec now officially lists matrix: as a safelisted scheme.

Next steps

For web clients like Element to be able to open matrix: URIs, a few more things need to happen.

  1. Browsers need to implement the HTML standard change and allow web apps to register the Matrix scheme.
  • Status in Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=1688030
  • Status in Chrome(ium): https://bugs.chromium.org/p/chromium/issues/detail?id=1169258
  1. Web clients need to actually implement handling matrix: URIs.
  • Tracking issue for Element: https://github.com/vector-im/element-web/issues/16875

You may be wondering about matrix.to, which was used to share matrix identifiers before. In the long term matrix URIs will probably replace it. Watch the next few TWIMs to find out what happens and/or subscribe to the following issues: https://github.com/matrix-org/matrix.to/issues/191 https://github.com/matrix-org/matrix.to/issues/192

Closing words

As you can see, this was a big step for Matrix as a whole. This of course couldn't have been possible without Alexey writing such an awesome proposal and MayeulC pushing for the changes in browsers and the HTML standards. A lot of other people were also involved in implementing the scheme in their clients, sending the RFC to IANA or working on matrix.to, which was our solution until now and will probably bridge the gap into the future. The matrix scheme being in the HTML standard may look small, but a lot of stuff was blocked on it, that can now start making progress. Exciting, isn't it?

And it gets better...

Nico (@deepbluev7:neko.dev) offered:

MayeulC spotted, that Mozilla now safelisted the matrix: scheme in Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=1688030#c15

Expect this to land around Firefox 90 probably.

So we can check one more thing on our list. Apart from Chrome, we are now only waiting on webclients to implement it. This is moving faster than I expected!

If this truly lands in a Firefox version in the near future I'll be very, very happy.

kitsune added:

many thanks to MayeulC for making this happen.

Thanks to everyone involved!

Dept of Spec 📜

Spec

anoa told us:

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.

MSC Status

Merged MSCs:

MSCs in Final Comment Period:

New MSCs:

Spec Updates

Folks from the Matrix Core team continue to chip away at the various Spaces-related MSCs in order to further the initial implementation across all three Element clients. The Spec Core Team is cleaning up some deprecated APIs that have been hanging around for way too long now (Identity Service v1 APIs), as well as continuing to focus on getting various, implemented end-to-end encryption MSCs documented and into the spec.

We're also slowly inching towards a new release with a new spec-wide versioning scheme, as well as completing the move of the old documentation from matrix.org/docs/spec to spec.matrix.org. Slowly but surely.

2021-04-23-hRN19-stacked_area_chart.png

Dept of Servers 🏢

matrix-media-repo

matrix-media-repo is a highly customizable multi-domain media repository for Matrix

TravisR said:

v1.2.7 is out as a highly recommended security update. This release fixes a memory exhaustion issue regarding thumbnails, known as CVE-2021-29453.

See https://github.com/turt2live/matrix-media-repo/releases/tag/v1.2.7 for more details, and visit #media-repo:t2bot.io on Matrix for questions and support.

Synapse

callahad announced:

Synapse 1.32 is out! This release focused on internal cleanups and is our first release without support for Python 3.5, so nothing too groundbreaking, but we're pleased with the improvements we're making.

Unfortunately, we discovered a pair of issues late in the release, so we ended up with 1.32.2 rather than just 1.32.0, but we got there in the end. We'll adjust our processes for future releases to avoid repeating this issue.

Homeserver Deployment 📥️

Kubernetes

Ananace told us:

My regularly scheduled chart updates for Synapse 1.3.1 (the deprecated image got a bump too) and matrix-media-repo 1.2.7 have been pushed.

and then

Got the Synapse 1.32.2 update up on my chart too, replacing the pulled 1.32.1 version.

YunoHost

Mamie reported:

YunoHost is an operating system aiming for the simplest administration of a server, and therefore democratize self-hosting.

Synapse integration had been updated to 1.31.0 (1.32.0 available in branch testing)

Element Web integration had been updated to 1.7.24 (1.7.25 available in branch testing)

Dept of Bridges 🌉

The matrix.org gitter bridge is no more, long live gitter.im!

Half-Shot announced:

Hello folks! Today at 17:41, we sunset the matrix.org gitter bridge after many years of active service. It started out as a tiny little project, forked from another gitter<->IRC bridge before eventually becoming part of the matrix-appservice-bridge family of bridges (thanks Leo for putting in the hard work there!). It never quite gained all the nice features that the IRC and Slack bridges enjoyed, in part due to it's dependence on the use of one bot account (serving as a relay). However, it was still the entrypoint for Matrix users into the likes of NeoVIM and Typescript and continued to run for several years.

Years later, gitter.im gained the ability to speak Matrix, so the old bridge can finally be laid to rest. Thanks to everyone that used and supported the bridge, and we hope you enjoy using the new one!

Thanks from the bridge team 🌉!

matrix-puppeteer-line

Fair announced:

matrix-puppeteer-line: A bridge for LINE Messenger based on running LINE's Chrome extension in Puppeteer.

Updates:

  • Improvements to inbound messages syncing. Messages shouldn't be posted out-of-order anymore, and backfilling shouldn't skip any chats.

  • Fixes to internal state. Deleting rooms with clean-rooms should work properly now.

  • Experimental work on inbound read receipts (in the receipts-testing branch)

Discussion:

#matrix-puppeteer-line:miscworks.net

Dept of Clients 📱

Nheko

Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE and intends to be full featured and nice to look at

Nico (@deepbluev7:neko.dev) reported:

It's my pleasure to announce, that our testers didn't find the bugs I added and as such version 0.8.2 is out now! The changes are too much to fit in a TWIM, so you can read the changelog here: https://github.com/Nheko-Reborn/nheko/blob/master/CHANGELOG.md

Thank you to all the contributors, that made this release possible! (Check the changelog! So many new faces!)

Highlights from this release:

  • Edits ✏

    • If you made a typo, just press the Up key and edit what you wrote.
    • Messages other users edited will get updated automatically and have a small pen symbol next to them.
  • Privacy Screen 👁

    • Blur your messages, when Nheko looses focus, which prevents others from peeking at your messages.

    • You can configure the timeout of when this happens.

  • Improved notifications ‼ (contributed by lorendb)

    • No more breakage, because the message included a > on KDE based DEs.

    • Render html and images where possible in the notification.

    • Render if a message is a reply or someone sent an emote message more nicely where possible.

    • Encrypted notifications now show, that the content is encrypted instead of being empty.

  • Screenshare support in calls on X11 📺 (contributed by trilene)

    • Share your screen in a call!
    • Select if your mouse cursor should be shown or not and if your webcam should be included.
  • SEND MESSAGES AS RAINBOWS! 🌈 (contributed by LordMZTE)

    • YES MESSAGES, EMOTES AND NOTICES!

I hope you'll have a wonderful time with this release and if you find any bugs, bug us in #nheko:nheko.im or open an issue on our bug tracker!

Fractal

Alexandre Franke offered:

In keeping with our tradition of sending an update every two months, here’s what we’ve been up to since the last one.

Amongst a bunch of maintenance changes (dependency updates and the likes), we had a few ones that stand out:

  • Some views have been migrated to HdyStatusPage. This provides a more homogeneous look across GNOME applications and help them fit better on smaller screens.

  • We noticed regressions around message replacement and removal (edit display and redactions) and we started fixing them.

  • Another regression made it impossible to join rooms by alias or id. This was fixed as well.

  • We updated information in a couple places and we now have clearer contributing guidelines.

We rarely talk about new issues that are still open and tend to focus on fixed ones. We do have a milestone with the remaining known blockers for our next version, 4.5, which we hope to be able to release Really Soon Now™.

But as interesting as all that is, it pales in comparison to our bigger announcement. We are rewriting Fractal! Julian, Alexandre and Thib have documented the thought process behind this decision and the work done so far. For more details, have a look at Julian’s blog. The summary is that even though such rewrites are usually frowned upon (for good reasons), two major technological shifts (from a homegrown backend to matrix-rust-sdk and from GTK3 to GTK4) made it an option that we needed to consider… and so we did! This should result in a cleaner codebase, easier to maintain in the long run, and improvements that we can share with the rest of the Matrix community.

The work for the rewrite currently happens in a fractal-next branch. So far, we have the basic boilerplate in place. The app launches, offers a basic login form, does initial sync and while the interface then looks very similar to stable Fractal, it will just load an unsorted list of rooms for now. Not very impressive on the visible side, but Julian has also been very busy contributing to matrix-rust-sdk.

Hopefully we’ll be able to break the aforementioned tradition and report more progress soon. Watch this space.

Element Clients

Delight, a project to improve the Element experience

  • On Spaces, we’ve been continuing to implement MSC3083 (Restricting room membership based on space membership) on the Web, Android & Synapse, while also iterating on iOS.
  • Expect an announcement on more Spaces testing soon!

Web

  • 1.7.26-rc.1 on staging
    • Added persistence of unsent messages across app restart
    • Improved room list filtering performance
    • Improved the image detail view
  • 1.7.26 planned for release on Monday

iOS

  • The new room screen UI has been released (1.3.4) on the App Store. It contains several improvements and bug fixes. One major bug fix is encryption keys that failed to be shared between the notification service and the app.
  • 1.3.5 has been submitted to the app store. It contains another bug fix about encryption where the app failed to share new keys to all members of a room
  • Full story at: https://github.com/vector-im/element-ios/releases

Android

  • 1.1.6 version has been released, fixing several issues reported with 1.1.5.

Konheko

Nico (@deepbluev7:neko.dev) said:

I released version 0.0.5 of my Sailfish client: https://openrepos.net/content/deepbluev7/konheko

It adds an about page and more importantly fixes a bug where closing the window would not relaunch the app via the task icon! Shout out to Rudi Timmermans, who made the about page and is now working on some further design improvements! Small release, since I was working on Nheko most of the time, but I hope I can add persistence and E2EE soon to it!

2021-04-23-AAetC-konheko-sailfish-on-floating-xperia.jpe

Bruno commented:

This would be an excellent screenshot to for a messenger app in some hacker movie

Even in the movie they take a break to read TWIM!

Hydrogen

A minimal Matrix chat client, focused on performance, offline functionality, and broad browser support. https://github.com/vector-im/hydrogen-web/

Bruno offered:

Working on invites and leaving rooms, almost ready, should be released early next week.

Dept of SDKs and Frameworks 🧰

libkazv (and the Kazv Project)

tusooa told us:

libkazv is a sans-io C++ (gnu++17) client library built upon lager. Along with it there is kazv, a kirigami/qml client, and a forward bot between matrix and tencent qq. Talk to us on #kazv:tusooa.xyz .

Updates

In the past 2 week we ("tusooa and her longcat," we hope someday we can

remove this annotation :P):

  1. Supported encrypted attachments. https://lily.kazv.moe/kazv/libkazv/-/merge_requests/5

Ruma

Ruma is a set of Rust library crates around Matrix

Johannes Becker told us:

After the last update Ruma 0.0.3 was released 🎉

But progress didn't stop there and more things happened:

Dept of Interesting Projects 🛰️

Cactus Comments 🌵

Asbjørn told us:

Cactus Comments is a federated comment system for the open web built on Matrix.

We put out a blog post called MSCs We're Excited For.

It's about some of the spec change proposals that we're keeping an eye on for Cactus Comments.

We also released Web Client v0.8.0!

This release brings some improvements to the "<time> ago" text, as well as the ability to let the client fetch more messages "live" (i.e. without reloading the page). You can set a number of seconds in updateInterval to enable this, it is disabled by default.

  • Make comment time a semantic time element (Thanks to @hectorjsmith for !4).

  • Add hover text to comment time (Thanks to @hectorjsmith for !4).

  • Show "just now" instead of negative seconds if message timestamp is ahead of the client's time

  • Add ability to fetch new messages periodically.

  • New config option: updateInterval, which controls how often to fetch new messages.

  • Change thumbnail size from 32x32 to 64x64.

  • Stylesheet: allow linebreaks in comments.

/ipns/latest.cactus.chat is updated to point to the latest release, so sites linking there should already be using the new version.

Come play with the demo: https://cactus.chat/demo

Join our Matrix room: #cactus:cactus.chat

Matrix World from the point of view of my Synapse

MTRNord told us:

https://grafana.nordgedanken.dev/d/HDVgeCXGk/matrix-server-stats?orgId=1&refresh=15m&var-server_name=All&from=now-1h&to=now (Warning big dashboard)

This dashboard is a small experiment on what metadata you can get from matrix without any auth.

How?

It utilizes the synapse database to lookup alive and responding servers and on these currently just runs a query each 5m to check the version.

It allows a) to see the amount of servers and which servers my synapse at any point knows and b) to track the individual versions.

For my server at point of writing this results in 2348 Servers of which are 2269 running Synapse, 70 running Dendrite, 8 running Conduit and 1 running Construct.

Tech used

Used under the hood is a small rust script doing all the heavy lifting of many thousand requests each five minutes (around 2-3 per server currently).

The data is cached mostly in RAM as well as being written to a influxdb oss 2.x which allows for the grafana integration.

More information

If you want to opt out feel free to block requests with the "MTRNord/server_stats" User-Agent.

You can find the source code at https://git.nordgedanken.dev/MTRNord/server_stats

Very cool! Reminder to also check out this set of graphs showing Synapse versions over time from Chris.

Warning!

Timo from Conduit announced:

writes a script to boost the number of conduit servers 😉

Final Thoughts 💭

Several people shared this spicy article from fellow encrypted-messenger purveyors Signal: https://signal.org/blog/cellebrite-vulnerabilities/

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server.

#ping:maunium.net

Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1hmko.net414
2dendrite.thomcat.rocks867.5
3matrix.kootstra.frl897.5
4matrix.osl.frl1197.5
5halogen.city1225.5
6envs.net1371
7thomcat.rocks1398
8nordgedanken.dev1503.5
9maescool.be1674
10kittenface.studio2412

#ping-no-synapse:maunium.net

Join #ping-no-synapse:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1dendrite.foxomy.com445
2dendrite01.fiksel.info719

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Synapse 1.32.2 released

22.04.2021 18:16 — ReleasesDan Callahan

Synapse 1.32.2 is out! Synapse now requires Python 3.6 (or later) and we've made a few small changes which you should be aware of before upgrading. These are documented in the upgrade notes.

Note: We scrubbed the releases of Synapse 1.32.0 and 1.32.1 as we discovered a pair of regressions including a bug with Prometheus metrics after tagging the release. These have been resolved.

On Monday, humankind flew a helicopter on Mars. And while our pursuit of Space(s) is considerably more modest, it is nevertheless progressing apace: Synapse 1.32 includes an experimental implementation of MSC3083.

This release also includes a new Synapse module for routing of presence updates, which can allow devices to share presence information without requiring that they also share a room. Please note there are some nuances to worker configuration when using this module which we hope to iron out in a future release.

The Admin API is newly able to manage rate limits, and the user listing endpoint can finally sort its results by a variety of criteria.

Otherwise, this is again a very internals-focused release: many additional type hints, improvements to structured logging, and small cleanups, especially those possible now that we've left Python 3.5 behind. We've made changes to how we check whether accounts are exempt from rate limits to avoid cases where we mistakenly applied limits to Application Services which should have been exempt, and we've fixed a bug with sharded federation senders which could occasionally pin the CPU.

See the Upgrading Instructions and Release Notes for further information.

Thank You

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including dklimpel, languitar, ShadowJonathan, and xmunoz.