Disclaimer: Low bandwidth Matrix is experimental, not yet standardised, and subject to change without notice.
This guide is for Matrix developers who want to support
MSC3079: Low Bandwidth CS API in their
clients/servers. Please read the experimental MSC if you want to learn more about what is happening
at a protocol level. If you want a high level overview of low bandwidth Matrix and why you should
care, watch the 12 minute demo on Matrix Live.
Matrix currently uses HTTP APIs with JSON data to communicate from the client to the server. This is
widely supported but is not very bandwidth efficient. This means that the protocol is slower, more
costly and less able to be used on low bandwidth links (e.g 2G networks) which are common in certain
parts of the world. MSC3079 defines a low bandwidth protocol using CoAP and CBOR instead of HTTP and
JSON respectively. In the future homeservers will natively support some form of low bandwidth
protocol. However, at present, no homeserver natively supports MSC3079. Therefore, this guide will
set up a low bandwidth proxy server which can be put in front of any Matrix homeserver
(Synapse, Dendrite, Conduit, etc) to make it MSC3079-compatible. This guide will also configure an
Android device to speak MSC3079.
Low bandwidth Matrix currently does not support web browsers due to their inability to send UDP
traffic. You do not need to be running a homeserver to follow this tutorial.
Setting up a low bandwidth proxy for your homeserver
Prerequisites:
Go 1.13+
openssl to generate a self-signed DTLS certificate, or an existing certificate you want to use.
Linux or Mac user
Steps:
Clone the repo: git clone https://github.com/matrix-org/lb.git
Build the low bandwidth proxy: go build ./cmd/proxy
Generate a elliptic curve DTLS key/certificate: (we use curve keys as they are smaller than RSA
keys, but both work.)
openssl ecparam -name prime256v1 -genkey -noout -out private-key.pem
openssl req -new -x509 -key private-key.pem -out cert.pem -days 365
# you now have cert.pem and private-key.pem
INFO[0000] Listening on :8008/tcp to reverse proxy from http://127.0.0.1:8008 to https://matrix-client.matrix.org - HTTPS enabled: false
INFO[0000] Listening for DTLS on :8008 - ACK piggyback period: 5s
Mac users: If you are having trouble generating EC certificates, make sure you are using OpenSSL and
not LibreSSL which comes by default: openssl version. To use OpenSSL, brew install openssl which
then dumps the binary to /usr/local/opt/openssl/bin/openssl.
To test it is working correctly:
# build command line tools we can use to act as a low bandwidth client
go build ./cmd/jc
go build ./cmd/coap
# do a CoAP GET request to matrix.org via the proxy
./coap -X GET -k 'http://localhost:8008/_matrix/client/versions' | ./jc -c2j '-'
{"unstable_features":{"io.element.e2ee_forced.private":false,"io.element.e2ee_forced.public":false,"io.element.e2ee_forced.trusted_private":false,"org.matrix.e2e_cross_signing":true,"org.matrix.label_based_filtering":true,"org.matrix.msc2432":true,"org.matrix.msc3026.busy_presence":false,"uk.half-shot.msc2666":true},"versions":["r0.0.1","r0.1.0","r0.2.0","r0.3.0","r0.4.0","r0.5.0","r0.6.0"]}
If this doesn't work:
Check the proxy logs for errors (e.g bad hostname)
Try adding -v to ./coap (e.g bad method or path)
Run the proxy with SSLKEYLOGFILE=ssl.log and inspect the decrypted traffic using Wireshark.
Otherwise, congratulations! You now have a low bandwidth proxy! You can connect to your proxy just
like you would to matrix.org or any other homeserver.
Security considerations
The proxy acts as a man in the middle and can read all non-E2EE traffic, including login
credentials. DO NOT USE UNTRUSTED LOW BANDWIDTH PROXY SERVERS. Only use proxy servers run by
yourself or the homeserver admins.
We'll add low bandwidth matrix to Element Android and iOS by default once it's standardised - but
while things are still experimental, here's a guide for how to build Element Android to do it
yourself if you feel the urge. This can be used as inspiration for other Matrix clients too.
Prerequisites:
Android Studio
Steps:
Clone the repo: git clone https://github.com/vector-im/element-android.git
Checkout kegan/lb: git checkout kegan/lb. This branch replaces all HTTP traffic going to
/_matrix/client/* with LB traffic. /_matrix/media traffic is left untouched. This branch also
disables TLS checks entirely so self-signed certificates will work.
Clone the low bandwidth repo if you haven't already:
git clone https://github.com/matrix-org/lb.git
In the low bandwidth repo, build the mobile bindings:
go get golang.org/x/mobile/cmd/gomobile
cd mobile
# if gomobile isn't on your path, then ~/go/bin/gomobile
gomobile bind -target=android
Copy the output files to a directory in the Element Android repo which Gradle will pick up:
Configure the proxy's --advertise address. If you are running on a local device, restart the
proxy with an --advertise of your machines LAN IP e.g 192.168.1.2 instead of 127.0.0.1.
If you are running on an emulator, restart the proxy with an --advertise of the
host IP:
10.0.2.2. The URL scheme should be https not http, else image loading won't work as Element
Android won't download media over http.
Login to your matrix.org account via the proxy with the --advertise address as the HS URL
e.g https://192.168.1.2:8008 or https://10.0.2.2:8008. The port is important.
To verify it is running via low bandwidth:
Install Wireshark.
Restart the proxy with the environment variable SSLKEYLOGFILE=ssl.log.
Run tcpdump on the right interface e.g: sudo tcpdump -i en0 -s 0 -v port 8008 -w lb.pcap
Force stop the android app to forcibly close any existing DTLS connections.
Re-open the app.
Open lb.pcap in Wireshark and set ssl.log as the Pre-Master Secret log filename via
Preferences -> Protocols -> TLS -> Pre-Master Secret log filename.
Check there is DTLS/CoAP traffic.
Performance
To send a single 'Hello World' message to /room/$room_id/send/m.room.message/$txn_id
and receive the response, including connection setup:
Protocol
Num packets
Total bytes
HTTP2+JSON
43
6533
CoAP+CBOR
6
1440
Limitations
CoAP OBSERVE is not enabled by default.
This extension allows the server to push data to the client so the client doesn't need to
long-poll. It is not yet enabled because of the risk of state synchronisation issues between the
proxy and the client. If the proxy gets restarted, the client will not receive sync updates
until it refreshes its subscription, which happens infrequently. During this time the client
is not aware that anything is wrong.
CoAP uses Blockwise Transfer to download
large responses. Each block must be ACKed before the next block can be sent. This is less
efficient than TCP which has a Receive Window which allows multiple in-flight packets at
once. This means CoAP is worse at downloading large responses, requiring more round trips
to completely send the data.
The current version of /sync sends back much more data than is strictly necessary. This
means the initial sync can be slower than expected. On a low kbps link this can flood the
network with so much data that the sync stream begins to fall behind. Future work will look
to optimise the sync API.
I just learned that in a poll of 89 universities in Germany, Matrix ranked third place in the chat category already.
The source is in only available in German, unfortunately: https://zenodo.org/record/4817795
However, the conclusion list (first table) may be understandable from context. It lists the top three solutions per category with their respective number of universities using it.
The Instant Messaging part starts at slide 15, the first chart on that slide is "which solution do you use", the second is "How content are you with the solution?", with Matrix having the best average of all solutions, namely ~8.8/10.
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://spec.matrix.org/unstable/proposals.
This week the Spec Core Team has been reviewing various Spaces MSCs, most recently MSC3230 (Space ordering). We're also hoping to square away the aggregations MSCs (message editing, reactions, etc) once and for all, though this will likely take a concerted effort from a few members to pull off.
Finally MSC3231 is a (currently draft status) MSC from Callum, one of Matrix.org's GSoC students this year! His project aims to allow native token-based registration to homeservers (the idea is so that you can generate a few tokens from your registration-disabled homeserver and hand them out to a few trusted friends and family members).
And finally, work still continues on finishing up the technical portions of the new release process for the spec. As mentioned last week, we've attempted to split the work up over multiple people in order to get it done quicker. Slowly but surely...
Dept of Servers 🏢
Synapse
Synapse is a popular homeserver written in Python.
Hello from the Synapse team! A short update for a short workweek (thanks, bank holidays!):
Synapse 1.35 was released this week! The Spaces flag is on by default, a bunch of bugs were fixed, and we've landed many of the prerequisites to eliminating RAM spikes on room joins.
📚 We have new docs! 📚 Anoa converted our docs to build with mdbook (#10086), and you can now browse them at https://matrix-org.github.io/synapse/! Check it out and let us know what you think. (Note: Not all of the pages have been converted from reStructuredText to Markdown yet, so some might render a bit strangely, but the structure is there!)
Another round of updates on my smart caching media proxy. After refactoring a lot (as always), I implemented thumbnailing! Now the only big feature left to add is url previewing.
I also have a test deployment configured on media.pixie.town now, so you can try fetching a bit of remote media through there, or view this submissions screenshot
metrics
Got started implementing a Prometeus /metrics endpoint, with a rudimentary Grafana dashboard for my test installation.
While they both implement Matrix' media endpoints, they serve rather different niches, where matrix-media-repo fully decouples the media repo aspect, my proxy cooperates with Synapse's filesystem and database, to speed up operation while ultimately making it a seamless drop-in and removal process.
Security release for the matrix-appservice-irc and matrix-appservice-bridge library
Hello. This week we've released an update to the https://github.com/matrix-org/matrix-appservice-bridge/ library containing a security fix for room upgrade handling. The security report will come later, but for now we advise anyone using the room upgrade handler feature to upgrade to 2.6.1. By the same token, we would also advise all IRC bridge admins to update their bridge to 0.26.1.
The Libera.chat bridge is still ongoing
Howdy folks. As you've likely seen over the last few days, we're still hard at work getting the final pegs in place for the libera.chat bridge. As usual, you can start using the bridge now while it's in beta by going to #<foo>:libera.chat, but we're hoping to have the thing stable by next week. Catch us in #libera-matrix:libera.chat for the juicy gossip about it.
matrix-puppeteer-line: A bridge for LINE Messenger based on running LINE's Chrome extension in Puppeteer.
Send a bridge notice when getting unexpectedly logged out of LINE, to warn you to log in again.
Improve resiliency of LINE user avatar syncing.
Properly support syncing LINE rooms with participants who aren't in your LINE friends list (This was harder than it sounds...!)
These changes (and ones before it) will be merged to master once I reorganize some messy commits.
The next big task is still to fix outbound read receipts (i.e. to make it so that the bridge syncing a message doesn't make your > LINE contacts think you actually read that message). Once that is done, I'll consider the bridge to be in beta.
Thunderbird now has Matrix support based on matrix-js-sdk enabled in the Nightly builds.
The star feature is probably that we support multiple Matrix accounts in the same client. Right now all your unencrypted rooms with text messages should work fine. While we think we won't destroy your account's state, it's still recommended to use a testing account with it. During account setup, it will ask you for a password, even if the homeserver supports SSO. If you intend to log in through SSO, just leave the password field blank.
We're not quite at the point where we support all the things you love about chatting with Matrix. Many of the missing features and polish to make communication successful are tracked in this meta bug. The goal for that milestone is to enable Matrix in our Beta builds.
You can get a Thunderbird Nightly build at the bottom of thunderbird.net by switching from "Beta Channel" to "Nightly Channel". If you run into bugs with the Matrix integration, please report them through this form. When filing a bug, please include debug logs. You can copy the debug logs for the account by going to the "Show Accounts" dialog, right clicking the account and selecting "Copy Debug Log". Note that the debug log may contain information from any of your conversations, so you might want to check the contents before posting it anywhere.
NeoChat 1.2, our third major release, was released this week bringing many improvements to the timeline and text input component. If you missed it, you can read the announcement here: https://carlschwan.eu/2021/06/01/neochat-1.2-bubbles-better-text-editing-and-more/ and we even have a nice release video :) https://www.youtube.com/watch?v=4lcH4tm6uTk
Other than that, we started working on an integration with KDE web shortcuts functionality to quickly search selected text on the web: https://invent.kde.org/network/neochat/-/merge_requests/279.
Callum, our GSoC student, after spending some time on Synapse, had now his first go at Nheko's codebase. He implemented, that you can now just enter the server name on registration instead of the full URL. This means entering conduit.rs or matrix.org works now nicely, since those servers are actually hosted at a different URL. He's now working on the Token Registration MSC, which he will implement in Synapse and Nheko, so exciting times ahead!
We also had a small contribution from pcworld, who fixed that if you only viewed the room list in the narrow layout, you would not get notifications for the last selected room.
I'll leave you with some words, that you may have heard a few times already: "Watch this space for next weeks update!"
A dozen merge requests have been integrated in our fractal-next branch since last week.
Amongst the more trivial changes, Julian made sure rooms are added to the sidebar in batch (!737) to improve performances, added in-app notifications for invite errors (!760), added a menu entry to leave rooms (!769), and implemented display of user and room avatars (!770). We also gained a right-click-menu entry to display event sources thanks to Kévin (!766).
Element Clients
Updates provided by the teams.
Delight team
We’re continuing progress on implementing Blurhash on Web & Android to improve the image loading experience, especially on low bandwidth
On Spaces, we’ve started working on the ability to drag and drop to re-order Spaces, along with improving adding aliases to public Spaces
Web
1.7.30 RC on staging
Improved layout performance in the timeline and room list
Refined the message action bar UI
Continuing to improve application performance
Recent focus on minimising browser layout work when things change
Reducing DOM size
Working on Apple silicon desktop builds
iOS
1.4.0 is available on the public TestFlight. We expect to make it available on the App Store on Monday. It has:
Performance improvements
Crash fixes
New languages: Esperanto, Portuguese (Brazil), Kabyle, Norwegian, Swedish, Japanese and Welsh.
There are some API breaks in MatrixSDK due to those performance improvements.
We have now a MXLog module with log levels! It is now possible to disable all logs from MatrixSDK
1.1.8 has been released to production, and 1.1.9 has been released to beta on the PlayStore
We are currently working with the design team on the light and dark theme of the application, especially colors and text appearance. Lots of cleanup to do...
Hydrogen
A minimal Matrix chat client, focused on performance, offline functionality, and broad browser support. https://github.com/vector-im/hydrogen-web/
Released Hydrogen 0.1.56 this week, with redactions. In the meantime, I've been making good progress on reactions, which should hopefully get released early next week. Midhun has made good progress on the right panel, ironing out the last bugs.
Here's a sneak preview of reactions (with slow network to show off the local echo animation):
I guess it's a long time from our last twim. Here's what is going on in that time:
We used fluent for translations. https://lily.kazv.moe/kazv/kazv/-/merge_requests/1
We supported read and save client state. https://lily.kazv.moe/kazv/kazv/-/merge_requests/2
A work-in-progress, but we are displaying some common event types; there are even chat bubbles (>w<) Check out a screenshot below: (yes, and we got a new logo)
On August 21-22 the annual Free and Open Source Conference (short FrOSCon) will take place. Usually the conference takes place in a German University of applied Sciences Bonn Rhine Sieg. This year it will be virtual. On the positive side - we don't need to travel.
As German Matrix community grows this is a great opportunity to meet each other and hack together.
Matrix Dev Room
We are planing to do a virtual Dev Room this year. The idea is to exchange on the latest Matrix development and projects, get to know each other and drink <your_favorite_beverage> (virtually) together. 😉
To make it happen we need your help!
Dev Room is living from talks and workshops - this is your chance to present your Matrix project or to do a workshop!
Language: preferably German, but English is also ok
Submission is until 2021-06-11, but please give us feedback ASAP so we can create a plan now.
If it's your first talk or workshop some free of charge coaching is included. 😉
Also help in organizing the Dev Room (moderation, timekeeping) is needed.
Matrix Open Source booth
It was a great place to chit-chat and to get your in-depth answers regarding Matrix at FOSDEM this year. 👍️
We also planing to have a virtual booth at FrOSCon.
We need your support in answering questions about Matrix or just to have a good time.
Get in touch
If you want to take part please contact @oleg:fiksel.info (or oleg@fiksel.info) ASAP to add you to the Dev Room participants list.
Set up a new (virtual) EU homeserver in Frankfurt, so European folks can join the beta starting next week
Use StoreKit to detect when a user is in Europe and should therefore use the EU homeserver, both for GDPR and to minimize latency
Made some progress toward supporting m.video messages
Experimented with using the new iOS photo picker to better protect users' privacy
Homepage: https://kombuchaprivacy.com/circles/
Source code: https://github.com/KombuchaPrivacy/circles-ios
Synapse 1.35.0 is out! This release focused on improving internals as we drive toward better memory performance during room joins, but more on that below.
Update:Synapse 1.35.1 was published on Thursday, June 3rd. It resolves a bug (#10109) which mistakenly listed invite-only rooms in the Spaces summary.
We'd also like to call the attention of client developers to a deprecation: The unstable prefixes used during development of MSC2858: Multiple SSO Identity Providers will be removed from Synapse 1.38, due out in August. Please ensure your client supports the stable identifiers for this feature.
Spaces: On by Default
Following the successful release of Synapse 1.34, the experimental Spaces flag is now enabled by default. If you had manually enabled the experimental_features: { spaces_enabled: true } flag in your homeserver configuration, you may now remove it.
Bug Squashing
This release of Synapse fixes an issue which could cause federated room joins to fail when the join response exceeded a size limit which was too low (#10082). We've also improved what Synapse logs when it drops a connection in similar circumstances (#10091), which should aid diagnosis if a similar issue were to arise in the future.
GitHub user thermaq contributed a fix (#10014) for a bug which could cause user presence state to become stale.
Lastly our OpenTracing support now allows for profiling end-to-end performance on a per-user basis (#9978).
An Update on Room Joins
We've been hammering away at shrinking Synapse's memory footprint when joining large / complex rooms, and while we're not there yet, the end is in sight! In particular, this release includes many internal refactorings, including using ijson to parse the JSON response to /send_join (#9958), clearing the way for substantial improvements.
Memory usage still spikes because we're effectively doing the same work with a different library, but ijson's design allows for iterative parsing. This will pay dividends once we modify the code downstream of /send_join to take advantage of it.
Concretely, Erik Johnston has an experimental branch of Synapse which completely eliminates the memory spike:
The remaining work is centered on splitting that branch into self-contained, reviewable pull requests, like a rewrite of the Synapse Keyring class (#10035). After that's merged, we'll need to make one further change to properly batch up work, at which point we should attain the efficiency gains from Erik's experiment.
Everything Else
GitHub user savyajha contributed a security hardened systemd unit file which effectively sandboxes Synapse (#9803). While not enabled by default, we'd encourage security conscious users to review the example file and associated documentation.
Please see the Release Notes for a complete list of changes in this release.
Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including dklimpel, jerinjtitus, junquera, lonyeon, savyajha, and thermaq.
Here's your weekly spec update! The heart of Matrix is the specification - and this
is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://spec.matrix.org/unstable/proposals.
The Spec Core Team is working towards finalising the remaining spec PRs and the new
spec release process. Good news is there's only one final spec PR to go! Bad news is it's probably going to be one of the hardest :)
Otherwise the team was pleased to see that noticeable progress is being made on the
MSC backlog. But that doesn't mean we get to rest on our laurels!
Dept of Servers 🏢
Synapse
Synapse is a popular homeserver written in Python.
We have interns! In addition to Matrix.org's participation in Google Summer of Code, Element has funded two Outreachy interns to work on backend projects, and we're overjoyed to welcome the following folks to Matrix:
Shay (Outreachy) is focused on modernizing Sydent and Sygnal, our reference implementations of a Matrix Identity Server and a Matrix Push Gateway respectively.
Meenal (Outreachy) is helping us improve Complement, and modern replacement for SyTest, our homeserver integration test suite.
Callum (GSoC) is working on adding support for restricting homeserver registration to users with pre-shared invite codes.
Welcome Callum, Meenal, and Shay!
We're also working to unify and standardize the module API which is used to extend Synapse, and your feedback would be very welcome. Similarly, if you have any experience with adding custom modules to Docker images or Debian packages and have thoughts on how we should best support that (or just examples of projects which do that well), please weigh in at #9944.
Lastly, we're looking forward to the release of Synapse 1.35 which will bring significant improvements to memory use during room joins, but that's for next week. 😉
A very warm welcome to all of our Outreachy interns and Google Summer of Code students
this year! You can see the full list of GSoC students in our Google Summer of Code
2021 blogpost.
I've been working on a smart media worker/proxy, to offload initial spikes in traffic for homeservers behind slower network uplinks.
There's a <remote> component that sits on a small vps, handling the upload and download endpoints (and soon, thumbnails), with an in-memory cache to quickly respond to requests.
Media still gets forwarded to the <local> component where it gets stored in Synapse's media folder with an accompanying database entry, so all media is still stored long-term in the way Synapse expects it to be, so the proxy can be removed at any point.
Currently not production ready but you're welcome to snoop around the repo :)
https://git.pixie.town/f0x/synapse-media-proxy
Seems like quite a useful project! Other adventures in splitting out the media
server from the homeserver include TravisR's
matrix-media-repo project.
timokoesters reported the latest
updates since last week:
Feature: Implement /claim
Feature: Handle incoming to-device events
Feature: Implement /query
Improvement: Faster signing key storage
Improvement: Documentation for Appservices
Fix: State resolution bugs
Fix: Respond to unauthorized PDUs with FORBIDDEN
Fix: Forward errors from remote servers (e.g. unsupported room version)
Timo also shared with us a milestone dashboard for Conduit being production-ready:
https://gitlab.com/famedly/conduit/-/milestones/3. It's exciting to see the
multi-implementation Matrix homeserver landscape really beginning to take shape!
Hey folks, some of you (well, frankly most of you) wanted to hear about our plans for libera.chat. I've been hard at work
this week building a brand new bridge which is hosted on the libera.chat homeserver and it's nearly ready to go. We're just making a few final
preparations before taking it out of beta, but it's currently available as a portaling bridge. You can currently reach any channel on libera.chat by joining
#<channel_name>:libera.chat. You can also add the homeserver to your room directory in your Matrix client to search all the bridged rooms.
We're also hanging out in #libera-matrix:libera.chat, if you have any questions. Of course, the bridge is proving to be very popular so please be patient :)
Definitely one of the most sought-after Matrix bridges lately.
Props to Half-Shot, Libera.Chat and everyone else for getting
it all together in such short notice.
Somewhat experimental feature that has landed this week is the ability to use single puppeting in a public Matrix room to bridge IRC channels.
This support comes through a new PLUMB admin command for network rooms that will ask the bridge bot to join a public Matrix room and an IRC channel.
On IRC side there will be only a single user/bot that will relay messages from Matrix users. This is a stop-gap between fully puppeting Matrix users on IRC and doesn't need explicit permission from an IRC network to use by channel operators as long as bots are allowed.
On Matrix side puppets will join and leave based on their presence on the IRC channel. The bridge bot does not need any special permissions in a room as it doesn't do permission synchronization so it can be kept without any power levels.
Unplumbing a room is as simple as kicking the bridge out and it will take the puppets with it.
As this is a fairly new feature it does have some rough edges so feedback is greatly appreciated.
With all this said, if you're in the need to bridge any IRC channel on any network with little friction, Heisenbridge should have you covered!
SASL support 🔒
If you're using Heisenbridge to connect to Libera.Chat from a network block that has SASL enforcement enabled (like some VPS providers) you can now configure SASL credentials to authenticate at connection time to bypass this restriction.
This only implements SASL PLAIN for now. Certificate authentication may be looked at in the future.
But that's not all
Identd will now always provide a unique ident for everyone
Improved detection for stalled connections
IRC user ghosting issues fixed for good 🤞
Server TLS cert checking fixes, self-signed support
Initial bridge DM invites should go through more reliably
QUIT command to leave from all networks in one go
Automatic Docker Hub images from master (amd64 and arm64)
In addition Heisenbridge is now featured on matrix.org 🥳
matrix-puppeteer-line Updates
matrix-puppeteer-line is a
bridge for LINE Messenger based on running LINE's Chrome extension in Puppeteer.
This week brings a variety of assorted bugfixes & feature improvements in the testing branch. Included are:
Support for m.sticker events for LINE stickers (with an option to use m.image if preferred)
Clear resolution of LINE emojis, and a config option to scale them (since they're tiny by default)
Less frequent re-downloading of already-seen LINE stickers/emoji (there are limits to this, but it's still an improvement)
Option to disable syncing LINE stickers/emoji (in case re-hosting LINE imagery is a liability)
More reliable message syncing (should never get "Decrypting message..." or invisible images/stickers/emoji anymore)
Fix crashes during inbound message & read receipt syncing of multi-user rooms
Like last time, testing is much appreciated 😀
Remaining big tasks are:
Rework message syncing so that receiving a message doesn't require "viewing" the LINE chat in the Puppeteer-controlled browser, which will make LINE send a read receipt even though you may not actually have read the message yourself (I mentioned this a while ago; it's still a todo)
Support for multiple bridge users at once (A public instance of the bridge will only be considered once multi-user support is ready)
The past few weeks have been busy with fixing bugs. As a result, we plan to release NeoChat 1.2 next week. You can help by testing the Flatpak beta version of NeoChat: flatpak install https://flathub.org/beta-repo/appstream/org.kde.neochat.flatpakref.
Aside from bug fixing, Tobias worked a bit on E2EE encryption in Quotient and Noah added a fancy typing indicator.
Nheko Updates
Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE and intends to be full featured and nice to look at
Since last week, a couple things landed in fractal-next. Julian added support for accepting/rejecting invites (!754/#759). Kévin did an internal restructuring of how we handle the SyncResponse in !759. The visible consequence for users is that room categories (also known as tags) are updated directly in Fractal when the user changes it from an other client. He also switched from comrak to ruma’s markdown in !756. It shouldn’t have any impact for users, but should mean less code in the end on our side and more consistent code as well.
Spaces: We’ve been listening to, investigating, triaging and organising feedback from the beta so far. There’s a lot to get through, but thanks to everyone who has submitted feedback in-app in Element so far! The insight has been invaluable and is instrumental in shaping up our next milestones.
Papercuts: We’re also organising and fixing small issues throughout Element, biasing for highly visible issues which have the greatest impact and reach, affectionately titled ‘papercuts’. Expect more info on these soon, but for now a fun one to highlight is we’re implementing blurhash on Web & Android (with iOS to follow soon after) to improve previewing and viewing images, especially when on low bandwidth.
Web
1.7.29 released
Improved startup performance by focusing decryption on recent rooms
Fixed reaction duplication
Sorting out why we sometimes see "missing translations" hopefully for good this time
Continuing to improve application performance
Started work on Apple silicon desktop builds
iOS
We continued to work on technical subjects:
Stabilisation and performance improvements
New logging system. It will be possible to disable all logs for MatrixSDK, MatrixKit and Element-iOS
The code that manages application navigation in Element-iOS has been updated. It will allow us to add a slide menu to display things like user spaces.
Android
Element Android 1.1.8 has been released on the beta track of the Google PlayStore. It will probably be pushed to production at the beginning of next week. The Android matrix SDK2 has also been released.
This week, we implemented the ability to change the network when looking in the room directory. Also gitter.im has been added to the default network list. See some screenshots in https://github.com/vector-im/element-android/pull/3419
Lots of improvements regarding Spaces have also landed to develop.
Besides that, we are still fixing issues and perform regular maintenance on the project
Been a while since our last update! The biggest development is that we've hired some fantastic members of the community including SpiritCroc from SchildiChat to work on Beeper Android and Kilian from Nio to work on Beeper iOS.
Beeper Desktop reached 2.0.0 with much improved performance and UI. Video Walkthrough
If you want to use spaces to run an invite-only community, but can't wait for knocking and restricted room membership to make their way into a stable room version, you may be interested in Airlock, a very simple bot that I wrote to automatically invite people to a space's rooms and subspaces when they join.
It requires zero configuration—just invite the bot to your space and all of the rooms and subspaces within it, make sure it has invite permissions, and you're good to go! You're welcome to use my hosted version @airlock:townsendandsmith.ml, or run it yourself.
It's great to see people building useful tools to bridge the gap as Spaces inches closer out of beta.
About a month ago I came up with the idea that it would be interesting to use Matrix as an API for a blog. Sometime later it turned out that it's possible, and one thing led to another, and now I have a blog hosted on Matrix 😃
Long story short, the blog is a space and each post is a room within that space. This allows you to read the posts in your Matrix client of choice and also have discussions right underneath the content.
You can read more details on how it's done on the blog itself (incl. links to ugly code on my Github): https://evolved.systems/hosting-a-blog-on-matrix/. If you want to chat about this, join #blog:evolved.systems !
Longtime readers may remember Luke Barnard's take on the concept of Blogs built on
Matrix back in 2019 with
Journal. It's really
exciting to see a modern version of this concept - and using Spaces no less!
Server_Stats Updates
Server_Stats is a project to make
statistics about the Matrix network neatly catalogued and browsable.
The tool now has a separate list for rooms that are spaces at https://serverstats.nordgedanken.dev/spaces this allows you to find Spaces easily and fast :)
Other Updates
A bug in the sdk combined with appservices did initially have for some rooms the ban not recognized. This is now fixed and those rooms are hidden again from the API. Sorry for the inconvenience.
Safari should now also work after fixing a setting.
Check out the Sourse at: https://github.com/MTRNord/server_stats or join the room at #server_stats:nordgedanken.dev for questions :)
Circles
Circles markets itself as a private,
end-to-end encrypted social network built on top of Matrix.
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://spec.matrix.org/unstable/proposals.
Travis rolled out MSC3202 to aid in the quest for end-to-bridge encryption without having to rely on /sync. Otherwise the Spec Core Team has been reviewing MSCs and focusing on the last remaining spec PRs needed before cutting a new spec release. Watch this space!
Awesome to be able to VISIBLY see the progress in that graph!
Dept of Servers 🏢
Synapse
Synapse is a popular homeserver written in Python.
The big news this week was the launch of Spaces, supported by the release of Synapse 1.34.0.
However, there were quite a few other fixes, improvements, and additions: I'd encourage you to read the release announcement for those.
But more than anything, we're hard at work realizing the room-join memory improvements we talked about in last week's Matrix Live, and we expect to have some exciting news to share next week. Concretely: the pull request to support incremental parsing of join responses was merged, and several pull requests (#10017, #10018, #10035) to simplify and slim down the code for verifying event signatures are in review.
See you next week! 🚀
Conduit
Conduit is a Matrix homeserver written in Rust https://conduit.rs
And this week brings another instance of the regularly scheduled Kubernetes updates, with Element-web going to 1.7.28 and Synapse having been bumped to 1.34.0 on both the deprecated image and the chart.
Howdy folks! I'm sure many of you read the news about the new libera.chat IRC network starting up, and many of you also reached out (via pretty much every channel available) to ask about a potential bridge. libera.chat are doing a remarkably good job of getting spun up but are hyper busy with all the inbound users and need a bit more time, so in the meantime I would ask that everyone subscribe to https://github.com/matrix-org/matrix-appservice-irc/issues/1324 to listen for changes as they happen.
The matrix.org bridge team are on the case! Expect news in the near future :)
Dept of Clients 📱
Element Client Updates
Submitted by the teams
Delight
Spaces are now live! 🚀 Test them on Element Web, Desktop & Android (iOS is coming soon!)
In short, Spaces are a new way to group rooms and people together, and are slated to replace legacy groups/communities
Some performance improvements have been merged on develop:
Reduce the number of decryptions. A decryption takes about 5ms on iPhone X. On an account with 500 rooms this allows us to skip thousands of decryptions on an initial sync
Those decryptions do not happen anymore on the main thread
Android
Lots of dependency upgrade following the release with Space (1.1.7).
Next release candidate, 1.1.8 will also contain improvements on Spaces.
We have set up towncrier flow to better handle changelog generation.
Also Element Android project is now using GitHub actions, but it cannot run the integration tests for the moment.
As announced in the general Matrix GSoC post, we are lucky to get two interns this year again. Alejandro (whose previous work includes switching to matrix-rust-sdk for syncs) will implement multi-account support, while Kai (also a regular contributor by now) will help bring our application rewrite to the same functionality level as our current nightly and stable versions. Both projects will build on top of Fractal Next.
Julian came back from his vacation and immediately got quite busy reviewing and merging contributions from others. Kevin (also known as zecakeh) and him made good progress on several fronts:
Room handling was improved. We now keep track of whether the user joined them, left them, or has been invited to them, as well as their categories (favourites, low priority…).
The sidebar uses these categories and now uses a single list with GtkFilterListModels instead of one list per category. The greatly simplifies code for things like moving a room from one category to another.
History style has been tweaked, state events and timestamps have been added.
Markdown can be enabled from the newly implemented popover and the message composer offers syntax highlighting, although a bug currently makes it so that sent messages are still not sent with a formatted_body.
A persistent state store is used to load rooms on startup while sync is in progress.
Released 0.1.53 with support for linking to room this week, which should allow to add Hydrogen to matrix.to soon. Also started working on redactions, which is useful in itself and also lays the foundations for reactions and edits coming next.
Nheko reviewed
Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE and intends to be full featured and nice to look at
Check out this flattering portrait of Nheko on the Brodie Robertson "Linux Tips & Tricks" channel.
We've been quiet in the past weeks, but certainly not due to lack of activity! So many awesome things have happened, I don't even know where to start. I guess I'll go chronologically:
We published so many new crate releases. Most of our crates didn't see non-alpha releases for almost a year and now we've finally reset that counter! Now you can depend on ruma 0.1 and get bug fixes and new functionality through cargo update without having to worry about breaking changes. Huge thanks to @zecakeh for automating our release process, without that this would have taken so much longer.
We got initial spaces support! I'm super excited that we're now enabling client authors to take spaces into account, even if "taking into account" will initially just mean filtering out space rooms from their room lists. Code-wise, this was actually a relatively small change, with ruma-client-api receiving support for room types and ruma-events receiving support for the event types m.space.child and m.space.parent.
We're swapping out ruma-signatures' crypto library. Since ring, which is what we've been using so far, doesn't currently support cross-compilation to WASM, there's long been some interest to exchange it for something else for webbrowser homeserver experiments. A viable alternative library was identified a while ago and now @ShadowJonathan has taken on the task of implementing the switch.
In addition to unblocking the WASM usecase, this work revealed a bug in ring, which generated invalid PKCS#8 documents. We are pretty confident that once the switch away from ring (including a small compatibility shim to convert the invalid documents), a certain class of signature verification failures that Conduit has been getting will be fixed.
We're taking part in GSoC again. As you might have seen in the earlier blog post, two students are working on Ruma as part of GSoC this year: @Frinksy (for the first time) and @DevinR528 (for the second time).
By default, we still install Element, as Hydrogen is still not fully-featured. Still, people who'd like to try Hydrogen out can now install it via the playbook.
If you're trying out all those cool Spaces (Element Blog) and have some old Communities/Groups hanging around, I have the bot for you.
spacebot makes quick work of the conversion by translating as much as possible to the Space structure. Simply DM the bot and say !convert +group:example.org and it'll go off and make your Space.
The bot doesn't invite everyone from your Community to your Space, giving you a chance to configure the Space further before advertising it within your rooms.
The source can be found on GitHub if you feel like running your own or seeing how simple it is under the hood. #spacebot:t2bot.io is a great place to get help if you're having problems getting it going, and #help:t2bot.io would be glad to assist you if the t2bot.io deployment doesn't do the right thing.
P.S.: Spaces are currently considered Beta in Element, so they might not be perfect just yet. #spaces:riot.ovh is the room to join if you have questions about Spaces.
Hemppa
Hemppa the Bot is a multipurpose bot for writing modules super easily in Python.
Hemppa the bot is a general purpose Matrix bot written in Python. This week it gained two useful admin tools: Kick by wildcard (for cleaning up zombies after bridge decomission) and making tombstones to point room to a new one. https://github.com/vranki/hemppa
A little bit different, but matrix related, this week I added Matrix push support to the raspberry-noaa-v2 project. This is a project which runs on a raspberry pi to automate receiving, decoding and post processing of images over radio from weather satellites. It's easy to build a cheap ground station for this with little more than a USB SDR dongle and a metal coat hanger. If you want to see the pictures I am decoding you can join #weather:cadair.com.
Cadair, living in the north of England, will only see this symbol unfortunately 🌧️
The Project now has a proper webpage at https://serverstats.nordgedanken.dev/ !
Newly added Features are:
A List of all the rooms that are found
A list to find the links (either outgoing or incoming) for a room. So a 1 level deep view into the graph as a table.
An FAQ with some information about the project, known data issues and more
A websocket at wss://serverstats.nordgedanken.dev/ws that allows you to get the updates directly pushed as soon as a room lands in the db. Effectively the fastest way to get data updates.
Planned features for the webpage
AR and VR Graphs
A Space Finder that allows you to find the discovered spaces
Be aware that this page is very new and therefor might have bugs. It also isn't yet mobile optimized.
For questions, suggestions or fast updates check the #server_stats:nordgedanken.dev Room or for questions regarding issues with the bot in a specific room feel free to write a DM to MTRNord :)
Possible Bug that needs manual fixing
Due to a bug with my homeserver or synapse it happens for some rooms that synapse generates a lot of join events (Lots of "server_stats bot made no change" messages). If you see this please write a message to MTRNord so I can manually prevent my bot from trying to join the room over and over again as I cannot easily detect this kind of issue before joining.
The issue can be followed at https://github.com/matrix-org/synapse/issues/10021
This might be of interest for TWIM. It's a little web app that supports token-based registration in the Matrix UIAA.
Sort of like matrix-registration, but aiming for compliance with the UIAA spec
Circles is a new project to build an E2E encrypted social network on top of Matrix.
https://www.kombuchaprivacy.com/circles/
https://www.kickstarter.com/projects/cvwright/circles-a-secure-social-app-for-friends-and-family
It's really early days for this project, but please check it out!
The fedora project is discussing which domain to take in https://discussion.fedoraproject.org/t/so-what-domains-should-we-use-for-our-matrix-server/29842
Un-burying the lede: Fedora are making the jump to Matrix!
Dept of Ping 🏓
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server.
R Midhun Suresh: Right Sidebar for Hydrogen client
R Midhun Suresh from the Mar Baselios College of Engineering & Technology in Trivandrum, India will be working on Hydrogen this summer, mentored by Bruno Windels. He will be working on adding a right panel to the room view, including a member list and room information. He will be blogging at https://midhunsureshr.github.io throughout the project.
Devin Ragotzy: Ruma's Automated Checks
My name is Devin Ragotzy. I am a student at Western Michigan University in Kalamazoo, Michigan, studying computer science. I was lucky enough to work last summer on Ruma and have continued to contribute to the project. I was accepted to work on Ruma's automated checks project, mentored by Isaiah Inuwa, Jonas Platte, Timo Kösters. The goal of the project is to create a linter capable of enforcing Ruma-specific style and practices. I hope to get this tool to a working state by the end of this summers GSoC!
Abhinav Krishna C K: First-Class Email Bridge
Abhinav Krishna C K from NSS College of Engineering in Palakkad, India will be working on Building First-Class email bridge for Matrix this summer mentored by Half-Shot and tulir. This will enable Matrix to be connected with Email by translating incoming SMTP traffic to Matrix messages, and then bridging Matrix messages back into emails.
Frinksy: Extend Ruma's API coverage
My name is Adam Blanchet, and I am a student from the University of York in the UK.
I am happy to say that I have quite a few mentors: Isaiah Inuwa, Jonas Platte, Timo Kösters and Nico from Nheko.
My project is to extend Ruma's API coverage. I'll be doing a few things: finishing coverage of the Identity Service API, adding the "knock" feature to Ruma and finishing the implementation of QR code verification. If time allows it I will also work on implementing other MSCs or features such as "Event notification attributes and actions". I hope that my work will help enable other Rust-based Matrix projects, such as Conduit and Fractal, to implement more features.
Timo added:
Hello, I am Timo Kösters. I study Computer Science in Germany and spend most of the remaining time developing Conduit, a Matrix homeserver built on top of Ruma. I use Ruma all the time and will be mentoring Adam Blanchet to make it even better.
Vladyslav Hnatiuk: PyQuotient
My name is Vladyslav Hnatiuk, I'm a student of Vienna University of Technology and my project is PyQuotient.
The aim is to simplify creating of a Matrix Qt-based clients in Python by providing Qt-based SDK and avoid writing a large part of functionality manually. And to not reinvent the wheel PyQuotient will be bindings for the existing library libQuotient that provides SDK for Matrix for C++ applications. I'll be mentored by kitsune, the author of libQuotient and also libQuotient-based Matrix IM client Quaternion. I hope PyQuotient will facilitate the development of Matrix clients in Python with Qt, and it will be a small contribution to the promotion of Matrix, especially in Python world.
kitsune added:
If this experience proves to be successful, there’s a good chance Quaternion will eventually switch to Python.
Callum Brown: Token Authenticated Registration
Hi there, I'm Callum, a Londoner who'll be starting a physics degree in September. For GSoC I'll be working on adding Token Authenticated Registration to Matrix. This will allow homeserver admins to restrict who can sign-up by requiring a token to be submitted during registration. I run a small homeserver for friends and family, but don't have the resources to make registration public, so I have wanted this feature integrated into Matrix servers and clients for quite a while! I'll be working with Nico, anoa, and red_sky to write an MSC, implement the server side in Synapse, and the client side in Nheko. Thanks to the mentors and Matrix.org for the opportunity to work on this!
You can follow along with this project's progress throughout the program at https://calcuode.com/matrix-gsoc/.
Nico, mentor added:
Nico here, one of the Mentors. Personally I am super excited about this project! I have been using Matrix for a while now and I think Nheko is pretty good by now. But there is still a barrier, if I want my friends and family to use Matrix: They can't easily sign up! I have tried creating them accounts and telling them to change their passwords, having a dedicated registration page or just telling them to just use a different server, but nothing of that made me happy and it added friction to the already hard process of getting someone to try a new messenger! As such I am super excited for this, because it will make signing up your friends and family to your personal instance, without it having to be public, sooooo much simpler!
Jaiwanth: Exporting Conversations From Element
Jaiwanth Vemula from the IIT Kharagpur University in India will be working on Exporting Conversations in Element this summer, mentored by Michael (t3chguy). This work will enable users to easily export their conversations for archival or sharing, this is a feature which has been missed in Element for a very long time!
Alejandro Domínguez: Fractal: Multi account support
Kai A. Hiller: Fractal NEXT
Students become Mentors
I asked: how many of those who are mentors this year have ever been GSOC students? The answer is that this year four of the mentors were once GSoC students themselves!
Last week the UK government published a draft of the proposed Online Safety
Bill,
after having initially introduced formal proposals for said bill in early
2020.
With this post we aim to shed some light on its potential impacts and explain
why we think that this bill - despite having great intentions - may actually
be setting a dangerous precedent when it comes to our rights to privacy,
freedom of expression and self determination.
The proposed bill aims to provide a legal framework to address illegal and
harmful content online. This focus on “not illegal, but harmful” content is at
the centre of our concerns - it puts responsibility on organisations
themselves to arbitrarily decide what might be harmful, without any legal
backing. The bill itself does not actually provide a definition of harmful,
instead relying on service providers to assess and decide on this. This
requirement to identify what is “likely to be harmful” applies to all users,
children and adults. Our question here is - would you trust a service provider
to decide what might be harmful to you and your children, with zero input from
you as a user?
Additionally, the bill incentivises the use of privacy-invasive age
verification processes which come with their own set of problems. This
complete disregard of people’s right to privacy is a reflection of the
privileged perspectives of those in charge of the drafting of this bill, which
fails to acknowledge how actually harmful it would be for certain groups of
the population to have their real life identity associated with their online
identity.
Our view of the world, and of the internet, is largely different from the one
presented by this bill. Now, this categorically does not mean we don’t care
about online safety (it is quite literally our bread and butter) - we just
fundamentally disagree with the approach taken.
Whilst we sympathise with the government’s desire to show action in this space
and to do something about children’s safety (everyone’s safety really), we
cannot possibly agree with the methods.
Back in October of 2020 we presented our proposed approach to online safety -
ironically also in response to a government proposal, albeit about encryption
backdoors. In it, we briefly discussed the dangers of absolute determinations
of morality from a single cultural perspective:
We now find ourselves reading a piece of legislation that essentially demands
these determinations from tech companies. The beauty of the human experience
lies with its diversity and when we force technology companies to make calls
about what is right or wrong - or what is “likely to have adverse
psychological or physical impacts” on children - we end up in a dangerous
place of centralising and regulating relative morals. Worst of all, when the
consequence of getting it wrong is criminal liability for senior managers what
do we think will happen?
Regardless of how omnipresent it is in our daily lives, technology is still
not a solution for human problems. Forcing organisations to be judge and jury
of human morals for the sake of “free speech” will, ironically, have severe
consequences on free speech, as risk profiles will change for fear of
liability.
Forcing a “duty of care” responsibility on organisations which operate online
will not only drown small and medium sized companies in administrative tasks
and costs, it will further accentuate the existing monopolies by Big Tech.
Plainly, Big Tech can afford the regulatory burden - small start-ups can’t.
Future creators will have their wings clipped from the offset and we might
just miss out on new ideas and projects for fear of legal repercussions. This
is a threat to the technology sector, particularly those building on emerging
technologies like Matrix. In some ways, it is a threat to democracy and some
of the freedoms this bill claims to protect.
These are, quite frankly, steps towards an authoritarian dystopia. If Trust &
Safety managers start censoring something as natural as a nipple on the off
chance it might cause “adverse psychological impacts” on children, whose
freedom of expression are we actually protecting here?
More specifically on the issue of content moderation: the impact assessment
provided by the government alongside this
bill
predicts that the additional costs for companies directly related to the bill
will be in the billions, over the course of 10 years. The cost for the
government? £400k, in every proposed policy option. Our question is - why are
these responsibilities being placed on tech companies, when evidently this is
a societal problem?
We are not saying it is up to the government to single-handedly end the
existence of Child Sexual Abuse and Exploitation (CSAE) or extremist content
online. What we are saying is that it takes more than content filtering, risk
assessments and (faulty) age verification processes for it to end. More
funding for tech literacy organisations and schools, to give children (and
parents) the tools to stay safe is the first thing that comes to mind. Further
investment in law enforcement cyber units and the judicial system, improving
tech companies’ routes for abuse reporting and allowing the actual judges to
do the judging seems pretty sensible too. What is absolutely egregious is the
degradation of the digital rights of the majority, due to the wrongdoings of a
few.
Our goal with this post is not to be dramatic or alarmist. However, we want to
add our voices to the countless digital rights
campaigners,
individuals and organisations that have been raising the alarm since the early
days of this bill. Just like with coercive control and abuse, the degradation
of our rights does not happen all at once. It is a slippery slope that starts
with something as (seemingly) innocuous as mandatory content scanning for
CSAE content and ends with authoritarian surveillance
infrastructure.
It is our duty to put a stop to this before it even begins.
Twitter card image credit from Brazil, which feels all too familiar right now.
As many know, over the years we've experimented with how to let users locate
and curate sets of users and rooms in Matrix. Back in Nov
2017
we added 'groups' (aka 'communities') as a custom mechanism for this -
introducing identifiers beginning with a + symbol to represent sets of rooms
and users, like +matrix:matrix.org.
However, it rapidly became obvious that Communities had some major
shortcomings. They ended up being an extensive and entirely new API surface
(designed around letting you dynamically bridge the membership of a group
through to a single source of truth like LDAP) - while in practice groups
have enormous overlap with rooms: managing membership, inviting by email,
access control, power levels, names, topics, avatars, etc. Meanwhile the
custom groups API re-invented the wheel for things like pushing updates
to the client (causing a whole suite of
problems). So clients
and servers alike ended up reimplementing large chunks of similar
functionality for both rooms and groups.
And so almost before Communities were born, we started thinking about whether
it would make more sense to model them as a special type of room, rather than
being their own custom primitive.
MSC1215 had the first
thoughts on this in 2017, and then a formal proposal emerged at
MSC1772 in Jan 2019. We
started working on this in earnest at the end of 2020, and christened the new
way of handling groups of rooms and users as... Spaces!
Spaces work as follows:
You can designate specific rooms as 'spaces', which contain other rooms.
You can have a nested hierarchy of spaces.
You can rapidly navigate around that hierarchy using the new 'space summary'
(aka space-nav) API - MSC2946.
Spaces can be shared with other people publicly, or invite-only, or private
for your own curation purposes.
Rooms can appear in multiple places in the hierarchy.
You can have 'secret' spaces where you group your own personal rooms and
spaces into an existing hierarchy.
Today, we're ridiculously excited to be launching Space support as a beta in
matrix-react-sdk and matrix-android-sdk2 (and thus Element Web/Desktop and
Element Android) and Synapse
1.34.0 - so head
over to your nearest Element, make sure it's connected to the latest Synapse
(and that Synapse has Spaces enabled in its config) and find some Space to
explore! #community:matrix.org
might be a good start :)
The beta today gives us the bare essentials: and we haven't yet finished
space-based access controls such as setting powerlevels in rooms based on
space membership
(MSC2962)
or limiting who can join a room based on their space membership
(MSC3083) -
but these will be coming asap. We also need to figure out how to implement
Flair on top of Spaces rather than Communities.
This is also a bit of a turning point in Matrix's architecture: we are now
using rooms more and more as a generic way of modelling new features in
Matrix. For instance, rooms could be used as a structured way of storing
files (MSC3089);
Reputation data
(MSC2313) is stored in
rooms; Threads can be stored in rooms
(MSC2836); Extensible
Profiles are proposed as rooms too
(MSC1769). As such,
this pushes us towards ensuring rooms are as lightweight as possible in Matrix -
and that things like sync and changing profile scale independently of the
number of rooms you're in. Spaces effectively gives us a way of creating a
global decentralised filesystem hierarchy on top of Matrix - grouping the
existing rooms of all flavours into an epic multiplayer tree of realtime data.
It's like USENET had a baby with the Web!
For lots more info from the Element perspective, head over to the Element
blog.
Finally, the point of the beta is to gather feedback and fix bugs - so please
go wild in Element reporting your first impressions and help us make Spaces as
awesome as they deserve to be!
Synapse 1.34.0 is now available, and it's loaded with new features and performance improvements.
Note: This release deprecates and replaces the room_invite_state_types configuration option. If you've customized that for your homeserver, please review the Upgrade Notes.
We've also marked the v1 room deletion Admin API as deprecated. Instead of sending a POST to a path ending in /delete, administrators are encourage to instead send an HTTP DELETE to /_synapse/admin/v1/rooms/<room_id>. Thanks to ThibF for implementing this (#9889).
Spaces
The highlight of this release is support for Spaces, now that MSC1772: Matrix Spaces has merged into the Matrix spec!
Synapse also has support for MSC2946: Spaces Summary and MSC3083: Restricting room membership based on space membership, but these are off by default as they're still under development. To enable these experimental MSCs, set experimental_features: { spaces_enabled: true } in your homeserver configuration. These are enabled on the matrix.org homeserver, and we encourage you to experiment with Spaces there and let us know in the Spaces Feedback Room if you encounter any issues.
Memory and Caching
Memory consumption and caching have been a major focus of the Synapse team this quarter, and we've made significant strides:
Synapse has a new gc_min_intervalconfiguration option with reasonable defaults to prevent Python's garbage collector from running too frequently and thrashing when a large homeserver has its collection thresholds set too low.
For debugging, Synapse can optionally track the memory use of each LruCache.
We have a few more tricks up our sleeves; to learn more about how we're planning to improve the memory cost of joining large rooms, check out last week's Matrix Live.
Other Fixes and Improvements
We've also landed significant improvements to:
Sending events when Redis is available (#9905, #9950, #9951)
Joining large rooms when presence is enabled (#9910, #9916)
Prevented cross-account m.room_key_request messages from being delivered (#9961, #9965)
Incorrectly applied room creation / invitation rate limits to users and app services which should have been exempt (#9968)
The health check on our Docker images now responds more quickly upon successful startup thanks to improvements by maquis196 (#9913), and for especially privacy-conscious homeservers, device names can now be shielded over federation thanks to a contribution by aaronraimist (#9945).
So a token for @example:matrix.org might look like:
syt_ZXhhbXBsZQ_KfJetOcLWEKCvYdKnQLV_0i3W80
Existing tokens remain valid; this is just for new tokens. We hope the new format reduces network overhead while also making it easier identify misplaced tokens in logs and repositories.
Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including aaronraimist, maquis196, and ThibF.
Some combination of me, Kegan, Bruno and neilalexander have been working on v3 of the CS /sync API. (Today's /sync API in matrix is v2; v1 was the old /events API). We're not yet at the point of publishing a draft or MSC, but it's coming soon. It's really exciting work which flips Matrix around so that sync scales independently of the number of rooms you're in - and it's at last possible to write rich clients which only ever sync the bare minimum data needed to work: i.e. lazy loading eeeeeeeeverything. Watch this space :)
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://spec.matrix.org/unstable/proposals.
There's been activity from the Spec Core Team on a number of different MSCs, such as MSC3189 (per-room/space profile data). Additionally a spec PR for Matrix URI schemes has been getting feedback and is moving forward at a quick pace! There was also some feedback from the team on MSC2448 (blurhashes) which I'll get around to answering shortly 🙂.
Otherwise I think this week was a bit implementation-heavy for the team (the Spec Core Team is a task in addition to our full-time jobs). Hopefully next week will grant us more of a breather.
Hello TWiMmers! The bulk of our update is in Matrix Live today, so go check out the video above ☝️ to hear about how we're reducing the amount of memory it takes to join large rooms, and why joins take so much memory in the first place.
Otherwise, we're mainly getting ready for the public debut of Spaces as a beta feature, but more on that next week... 😉
Oh, and before we go: please make sure your Synapse is up to date! We released 1.33.2 on Tuesday, which contains a low severity security fix.
Another installation of the regularly scheduled Kubernetes Helm Chart updates (and another bump of the deprecated Synapse image). Now up to Synapse 1.33.2 and Element Web 1.7.27.
matrix-puppeteer-line: A bridge for LINE Messenger based on running LINE's Chrome extension in Puppeteer.
This week was spent on stability improvements & bug fixes.
Calling for testers!
The bridge is at a point where it's mostly usable, but it still has quite a few blindspots. If anyone is willing to try it out & report issues, it would be a great help!
Last time, we updated you on starting DM conversations with Gitter users from Matrix. Now we have the other side of this complete! From Gitter, you can now start a one to one conversation with someone you see from Matrix 😀
Just hover over their avatar to bring up the user popover, then press the "Chat privately" button. 🗣
Goooooooood afternoon folks and happy Friday! This week we're announcing the 0.26.0 release of the matrix-appservice-irc bridge which contains precious goodies:
You can now disable the kick behaviour of the bridge on Matrix users if you are running a personal bridge, so losing your IRC connection no longer results in a kick.
You can now remove bridges from rooms by using the admin room, so no need to use the provisioning API or modify the DB.
We've added a new feature to allow you to specify bridge options on a per room basis using room state. At the moment you can modify the limits of the automatic pastebin system but more features like reply formats are to come!
As always please come tell us about it in #irc:matrix.org and make sure to check out the new docs if you get a bit stuck.
Dept of Clients 📱
Hydrogen
A minimal Matrix chat client, focused on performance, offline functionality, and broad browser support. https://github.com/vector-im/hydrogen-web/
Hydrogen can now leave rooms and forget archived rooms. URLs are now also clickable in the timeline. Get the full details in the release notes!
Element Clients
Updates from the teams
Web
Element Web 1.7.28 is up on staging, targeting Monday for release.
New spaces Beta (new way of grouping rooms and people)
Added support for slash commands working in edits
On develop:
Voice messages are nearing completion - enable the labs flag and give it a go :)
Performance improvements to app startup time. Let us know if you run into any issues!
iOS
1.3.7 is available on TestFlight. It should be on the App Store on Monday. Spaces are not yet available on Element-iOS but the app offers minimal support. The release contains a fix for background crashed due to PushKit
At the platform level, we are still improving stability and performance:
Decryption operations to be moved outside the main thread
More robust on initial sync
etc
Android
1.1.7 is in open testing via playstore beta channel, Release candidate for Monday. Contains support for spaces beta, several improvements on attachment (video, compression…), as well as a bunch of bug fixes. All details here https://github.com/vector-im/element-android/releases/tag/v1.1.7
Delight
“Spaces are coming” (I had heard something about that - BP)
