Nico, one of the Nheko client maintainers, discusses the state of the project and shares various opinions about Matrix, chat clients, physics, the local weather and so on.
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
In terms of Spec Core Team MSC focus for this week, MSC1960 will be the only one on the list. The two MSCs from last time will be put aside for this week while we focus on implementation.
Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE (with the notable exception being device verification for now) and intends to be full featured and nice to look at
This week we added some more federation readers to matrix.org which should noticeably improve federation lag.
Patrick has continued on with the async/await-a-thon and we are starting to enter the home stretch. Of the things we care about only runInteraction remains. Once done we can get on with profiling the main process.
On the sharding side the last big milestone is to get the event persister to shard. I know this comes up every week, but we now have a rough working implementation for non-backfilled events. Next steps are to get backfill working, get the tests passing, go through review and then ship. I asked Erik if it would be live on matrix.org within 2 weeks. He didnβt say, at least I didnβt hear him say noβ¦>
Aside from that Rich is working on the push rules redesign, the next steps are to design the APIs in more detail and then make a formal spec proposal. Brendan is also in notifications land and his work on room unread counts is going through final review.
Finally, Andrew has been working with Sorunome to dust off the knock feature, both on the spec and the Synapse side.
πRomeo and Juliet benchmark to compare Matrix server implementations
This week I created the first(?) Matrix server benchmark and ran it on Synapse, Dendrite and Conduit. The test will go through the entire Romeo and Juliet play and create users for each character and sends messages for each line they say.
The results are very interesting:
Synapse:
timo neilalexander Half-Shot
# default (sqlite)
time 5m0.870s
# postgres:
time 1m46.319s 1m8s
Dendrite:
# default (sqlite)
time 6m8.802s 0m38.502s
# postgres:
time 2m45.387s 0m53.691s
Conduit:
# default (sled)
time 0m4.184s 0m2.776s 0m2.935s
There is some question here about how federation and state resolution will affect performance on these tests.
There is rather a lot in this section this week (see also Ops π section below.) I'll consider making this a more structured (table?) feature in future.
Hello! Just recently I made public an SELinux policy module I developed for Synapse. This policy is one I developed early on during my trip down the SELinux rabbit hole, and I have been using it in production for some time. At the moment, this policy module is built for the Gentoo SELinux policy (not RHEL/CentOS/Fedora) and does not currently have systemd support, nor has it been tested in all environments, i.e. with workers. I am publishing this policy in the hopes that it will be useful and I am open to any contributions, even those to help tighten the policy if possible. Finally, I have also made public a policy module for coturn, which I am also using in my own personal environment under the same conditions. Both the Synapse policy and the coturn policy can be found on Github. Thank you!
The ansible collection famedly.matrix has seen another release, 0.1.2, which updates the element role to 1.7.4 and the synapse role to 1.19.0. As always, get it from Ansible Galaxy, source from Gitlab, and for any questions join #ansible:famedly.de.
Just pushed the 1.19 tags for the K8s-optimized Synapse image, as well as an updated chart version for it that also takes advantage of the added /health endpoint.
As channels are really popular in Telegram, I wrote a quick and dirty program/bot which can mirror (text and image) public Telegram channels to Matrix rooms. This should make it easier for Telegram users to consider a switch.
Making the rooms read-only via permissions prevents messages from users.
It provides a good basis for someone trying to make a real 1to1 mirror. Please fork this project and host some channels publicly ;)
This project is functionally quite different from the more established mautrix-telegram: it is purely designed to mirror content from Telegram public rooms.
Hello all, I'm going to be quick off the mark and announce matrix-appservice-irc release 0.20.1! This release contains a ton of bug fixes and quality of life changes that you never knew you were missing. Please update and report back in the usual channels as always. Thanks!
Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE (with the notable exception being device verification for now) and intends to be full featured and nice to look at
I've been working on trying to finish the event store (when I wasn't distracted by video calls with some people). This should allow limited offline use and reduce memory use as well as allow some other features in the future. If you want to try it out, you can build this PR or try out the nightlies with new-event-store in their name.
If you decide to try it out, backup your cache or at least BACKUP YOUR ENCRYPTION KEYS! I won't be responding to feedback immediately, since I am currently hiking, but I should get to it this weekend or next week. Come discuss it in #nheko:ocean.joedonofry.com, if you have some feedback or issues.
We are still fixing issues on Element Android. Hopefully we will release version 1.0.5 of the application and of the Matrix SDK in the coming days!
later, fidèle à sa parole:
Element Android 1.0.5 has been published on the beta channel of the PlayStore. If everything is fine it will be pushed to production next week. I've also published SDK v1.0.5, and push translations of the PlayStore home page of Element to German, Chinese, English US, Hungarian, Portuguese (Brazil), Russian, Slovak, Swedish and Ukrainian.
This week, we improved customisation for forks. We add possibility to customize default HTTP headers (like "User-Agent") and improve login and room settings screens customisation. We started to improve project documentation. We also made some fixes like fixing registration validation link in email.
Hydrogen is a web-based client designed to be very lightweight, and with few dependencies.
Anyone who wants to discuss Hydrogen can join #hydrogen:matrix.org.
avatars and display names in the timeline and for rooms
implemented the logic for the member list, but no UI for this yet
room list sorting by last message timestamp, unread state and badge count
clear unread state when reading a room
some visual tweaks in the timeline and room list
bug fixes
If you have checked out hydrogen before, you'll need to clear your session (the little clear button in the session picker) to get correct avatars and display names everywhere.
Next week I'll get started on E2EE, which should make the project a lot more useful. I'll look into leveraging the nice rust matrix-crypto-sdk for this.
To add a new account, you will be asked first to pick one of the
listed public server (list data from anchel.nl) or to manually enter a server address
Typing in the server address field will also filter the public server list,
Up/Down or (Shift+)Tab and Enter can be used to navigate it by keyboard
If the address doesn't have a scheme://, auto-detect whether the server
supports HTTPS or only HTTP
Use the .well-known API if possible to resolve domains to the actual
homeserver's URL, e.g. matrix.org resolves to
https://matrix-client.matrix.org
The server address field will remember the last homeserver that was
connected to
Room members autocompletion:
Type @ followed by one or more characters in the composer,
or one or more characters and hit (Shift+)Tab to trigger username/user ID
autocompletion
Only autocompleted names will be turned into mentions, unlike before
where any word in a sent message that happened to be someone's name would
mention them
Full image viewer for matrix image messages and URL previews:
Click on a thumbnail in the timeline to open the image viewer
Middle click on a thumbnail (or use the option in the context menu)
to open the image externally
Left click on the image (mouse only): expand to window size if the
image's origin size is smaller than the window,
else expand to original size
Tap on the image (touch screen/pen only): reveal the info and button bars
when auto-hidden (bars will auto-hide only when they overlap with a big
enough displayed image)
Any mouse movement: reveal auto-hidden bars
Double click on the image: toggle full screen
Middle click anywhere: open externally
Right click anywhere: close the viewer, back to chat
Drag when displayed image is bigger than window to pan
Wheel to pan up/down, hold shift or alt to pan left/right
Ctrl+wheel to control zoom
Buttons to control rotation, scale mode, full screen, GIF play/pause
and GIF speed
New keyboard shortcuts are available for all these actions,
see keys.imageViewer in the config file (will be automatically updated
when you start Mirage 0.6.1)
Add media.openExternallyOnClick setting to swap the new
click and middle click on thumbnails behavior
Room and member filter fields now support (Shift+)Tab navigation, in addition
to Up/Down
Add a colored left border to the currently highlighted item in list views
(e.g. room list, members list, etc) to improve visibility
Themes:
Add controls.listView.highlightBorder and
controls.listView.highlightBorderThickness properties (can be set to 0)
Replace the confusing broken "Copy media address" entry with:
Copy media address: visible for non-encrypted media, always
copies the HTTP URL
Copy local path: always visible for already downloaded media, even if
they were downloaded before mirage was started
The openMessagesLinks keybind (default Ctrl+O) is renamed to
openMessagesLinksOrFiles and can now also open media message files
Using the openMessagesLinksOrFiles keybind on a reply will now ignore the
matrix.to links contained in the "In reply to XYZ" header
Pressing Ctrl+C to copy selected/highlighted non-encrypted media messages
will copy their HTTP URL instead of the filename
Retry downloading image thumbnails if they fail with a 404 or 500+ server
error (uploads sometimes take a few seconds to become available on the
server)
Non-encrypted media messages are now always downloaded on click and opened
with a desktop application (or the image viewer), instead of
being opened in a browser
Compress thumbnails and clipboard images in a separate process, to avoid
blocking every other backend operation while the compression is running
Reduce the level of optimization applied to clipboard images,
the previous setting was too slow for large PNG (10MB+)
Increase applied scrolling velocity when using the
scrollPageUp/scrollPageDown keybinds, now similar to how it was before
Mirage 0.6.0
Don't catch SIGQUIT (Ctrl+\ in terminal) and SIGTERM signals, exit immediately
Slightly increase the top/bottom padding to the multi-account bar in the
left pane
Dependencies: minimum nio version bumped to 0.15.0
We wanted to take a little bit of time to show off a state resolution library for Ruma, courtesy of @DevinR528 and @Timo. Ruma has a new crate to make resolving state easier, state-res! The goal of this crate is to be the base for federation, it provides the necessary utilities to resolve state, sort power events (reverse topological ordering), sort events (based on mainline depth), and a lot of event authentication functions. It is still in the early stages of development, but will be hopefully proved out in Conduit.
Other things that happened this week:
Added the invite endpoints from the Federation API
Started requiring strong types for event-sending endpoints
Augmented error types with error-specific fields (see the livestream here)
Speaking of livestreams, @jplatte streams coding sessions from time to time on Twitch. If you're interested in watching some coding streams to get more familiar with the Ruma codebase or learn Rust in general, follow us at [#ruma-livestreams:matrix.org](https://matrix.to/#/#ruma-livestreams:matrix.org)!
Very impressed that these features are landing in Ruma first, with the intention of getting them into Conduit.
SSSS and cross-signing are slowly making their way to mautrix-go! A WIP PR exists for both features and is being developed upon. This will allow any clients that use the library to eventually be able to use SSSS and verify the trustworthiness of other devices based on cross-signing.
Device verification was merged to neb! It needs human intervention to approve that the SAS match, of course.
A new service called "cryptotest" was added that allows other clients to exchange e2e-related events with neb to test their capabilities (such as encryption, room key forwarding, SAS verification)
Residents of Germany can apply for the Prototype Fund until the end of September. They are looking for new open source ideas or features and fund about 20 projects twice a year with 47,500 EUR for six months (starting in April for this round).
Previously funded chat-related projects include Briar and OpenPush.
It would be great to see a project building on Matrix.
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
1.19.0 contains lots of async/await changes, 27 separate PRs to be precise(thanks Patrick). These changes will provide some level of performance improvement, but the real value to us is that it will improve our ability to profile Synapse more effectively.
It also features improved logging performance, this is in part to reduce IO but also to reduce the total amount logged. Weβd be really interested to get some feedback on if the changes help. You may also want to take a look at our new logging defaults to see if they would help reduce total disk space usage.
Some admins will rejoice that we have added a /health end point to every configured HTTP listener which should make it easier to monitor multi-worker installs.
Finally 1.19.0 contains improvements to the Admin API allowing the admin to optionally purge rooms on deletion.
Get the new releases from any of the usual sources mentioned at https://github.com/matrix-org/synapse/blob/master/INSTALL.md. 1.19.0 is on github here.
Add a /health endpoint to every configured HTTP listener that can be used as a health check endpoint by load balancers. (#8048)
Allow login to be blocked based on the values of SAML attributes. (#8052)
Allow guest access to the GET /_matrix/client/r0/rooms/{room_id}/members endpoint, according to MSC2689. Contributed by Awesome Technologies Innovationslabor GmbH. (#7314)
In completely unrelated news to my earlier K8s updates, LinkΓΆping University has now soft-launched Matrix for use both internally - and hopefully with other universities and institutions in the future.
Se the k8s updates from Ananace below. Is this the first .se university offering Matrix? Also, great hostname!
We've finally opened up https://translate.riot.im/projects/matrix-doc/sas-emoji-v1 to translate the emoji used by clients to ensure that users are securely verifying each other. The idea is that clients should always have a consistent name for these emoji, even if developed by completely different teams. If your client supports encryption verification and has translations, please transition to using matrix-doc for your translations: https://github.com/matrix-org/matrix-doc/pull/2728 has some details, and #matrix-spec:matrix.org would love to help out.
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
Some of the team was on holiday last week, hence the noticeable lack of progress π In terms of Spec Core Team MSC focus for this week, we're keeping on with the same two as last week: MSC2674 (aggregations pt1: relationships) and MSC1544 (QR code verification).
lots of async/await changes, 27 separate PRs (thanks Patrick)
improved logging performance, this is in part to reduce IO but also to reduce the total amount logged. Weβd be really interested to get some feedback on if the changes help. You may also want to take a look at our new logging defaults to see if they would help reduce total disk space usage.
Further improvements to the Admin API allowing admin to optionally purge rooms on deletion.
Expect 1.19.0 to land early next week.
Aside from that Erik has been continuing with sharding the event persister, this is a big job, so weβre still a little way off having something to ship, but Iβll keep you updated.
Big news this week, after spending a good chunk of time trying to rework notifications to make the UX more intuitive, we have decided that the best (and only) way forward is to completely rethink how the push rules work. Rich is currently redesigning the whole thing, and once done weβll resume on the client side.
Having spent much of the past 3 months trying to improve the performance of large scale instances we will soon start looking at improving the out of the box experience for smaller installs and are in the early stages of thinking about what that might look like. Iβll update more over time, but the two obvious use cases to target are
Federating instance, with 1-5 users, resource constrained.
Federating instance with a few hundred users.
Our (relatively small) changes to logging in 1.19.0 are the first signs of this shift. We still need to ship event persistence sharding and also move more functionality from the main process (dependent on async/await), but once done it clears the way for new projects.
anoa brought it to our attention that this graph looks like a "ocean-beach waterline dropoff".
This week I worked a bit on federation! Last week we were able to query room directories over federation; This week I looked into what's needed to join one of those rooms:
Resolve room aliases over federation
Send make_join request when a user tries to join an unknown room
Use make_join response to create send_join request (insert origin, origin_server_ts and add hashes and signatures)
After implementing all those things, I could actually see Conduit's join event on Synapse!
Of course a lot more work is needed to make this really work,
like creating the room on the Conduit side and sending and receiving federation
transactions, but this is a start.
Other news:
Docker support (thanks to @weasy, @valkum and @paul)
Room upgrade support (almost done, thanks to @Faelar)
Thanks to everyone who supports me on Liberapay or Bitcoin!
Sytest has now been updated to allow skipping tests for endpoints that have been deprecated in the spec. This may be useful if you are writing a homeserver and have no plans to implement deprecated endpoints, e.g. /initialSync or /events.
Use the command line parameter --exclude-deprecated when starting Sytest to skip these tests!
Just pushed some WIP (v0.1) charts for both Synapse and the matrix-media-repo project, written with Redis, workers/sharding, PVC constraints, and other large deployment thoughts in mind.
Note that these charts - the Synapse one in particular - basically require you to have a fully featured cluster, with a regex-capable ingress, automated cert manager, and RBAC. They do not require you to have a ReadWriteMany capable storage class though, even when using workers.
Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE (with the notable exception being device verification for now) and intends to be full featured and nice to look at
So, we've been silent for a bit, but that doesn't mean we haven't been busy, so I have a little bit of a bigger announcement to make today:
Trilene implemented voice call support in Nheko! We just merged this and you can try it out now!
Some caveats apply at the moment, but will be resolved in the near future:
Currently you will need to have a patched version of GStreamer. The current releases of GStreamer have a bug when your turnserver uses a password, that contains a ':'. Since this is the default for coturn, I'm guessing this affects everyone.
You will need to build Nheko yourself. Currently we do not have GStreamer available in our CI pipeline, so we disabled support for voice calls in those builds (apart from the flatpak build, but that also needs some further validation and a new GStreamer release). We will release proper builds with VOIP support once GStreamer is actually released and available with the fixes we need.
There are still some other issues, when relying on a turnserver to connect a call. We will fix them one at a time.
Big shoutout to trilene for implementing all of this! I can finally call people from Nheko, which is super awesome!
This was the first of the bigger features, that have been cooking for a while, in the near future we should also finish the new event store, which reduced memory usage and provides some kind of offline support, and Chethan will probably finish the first iteration of cross-signing soon too. After that we'll focus on the smaller features again. ;-)
BIG update! Looking forward to trying out voice calls. βοΈ Maybe we'll be calling Nico for Matrix Live soon..?
We are still fixing issues following the Element release. We are also working to add Jitsi conference call support.
We have exported a first beta version of the SDK from Element Android. It is available here: https://github.com/matrix-org/matrix-android-sdk2 and can be integrated in any Android app as a regular gradle dependency. We will update the SDK after each release of Element Android.
It includes following changes since the last TWIM:
The single-list overview screen is now optional. If you prefer the tabbed layout for separating direct messages and rooms, as known from Element for Android, but still want SchildiChat's message bubbles, you can get that combination now too!
Push notifications have been enabled for the Play Store variant
Some regular readers might remember, in the past I've put a few entries here about a side-project called Brawl. Two weeks ago, we took Brawl and started building a new client on top of it code-named Hydrogen.
As the name implies, Hydrogen tries to be the lightest Element. It is written entirely in vanilla javascript (no React, no Webpack) for complete control, structured as an MVVM app, leveraging the raw performance of indexeddb. On huge accounts where Element-web uses up to 1GB of heap memory, Hydrogen sips away with a mere 15mb. It is very early days though, and Hydrogen has very little features: you can receive and send unencrypted messages and that's it.
Hydrogen also aims to support a wider range of browsers, from IE11 (particularly for orgs with locked-down machines) all the way to a fully featured PWA that works well on mobile. We should also be able to easily add progressive loading, which proved hard in the past.
Another goal is to make it easy to embed parts of the app (like the room view) into another app or website. It could also be really nice to have a place to play and experiment with new tech for element proper - much as Dendrite has influenced Synapse.
As mentioned, it's early days, and lots of things are missing or broken, so please don't file any issues yet. Feel free to play around with it at https://hydrogen.element.io/ though!
Lots of exciting things happened in Ruma this week:
We have officially implemented over half (18/31) of the Federation API endpoints!
Special thanks to @agraven and @q-b!
We've added the only new endpoint from r0.6.1 of the C2S spec.
We have begun work on adding support for borrowed types for requests. This
should reduce the memory allocations required for clients to send requests to
a server.
Exported some macros to make creating custom endpoints and events that
There's news from ansible land! Ansible recently introduced a new packaging format, the "collection", and we've now started moving our ansible content into collections. This means the matrix modules I twim'ed a few months back are now easier to install than ever before, and the synapse/element roles are included in there as well (thanks to madonius for the contributions there). The roles have been regularly updated for a while now, and the element role isn't even called riot anymore! You can get the collection on Ansible Galaxy, with the source being available over on GitLab. In case of any questions, feel free to hop over into #ansible:famedly.de
If Ansible were a film franchise "Ansible: The Collection" would be the long awaited boxset.
opsdroid is getting close to a release with the matrix connector rewritten to use the matrix-nio library. If you are feeling brave and want to help us iron out any bugs before the release, please install the master branch and test it out for your matrix bots. Please report any issues on the GitHub issue tracker or come talk in #opsdroid-developers:matrix.org. Hopefully, a release with this and lots of other goodies will be coming in a couple of weeks.
Iβve created a compact and portable home server based on Matrix protocol, itβs a piece of hardware which is easy to use and set up. Iβm working on a new client APP similar to other typical messenger GUI, but this server we named it βPriveeβ supports any messenger developed with Matrix protocol. Iβve launched a campaign on Kickstarter this week, please come and support us! Thank you!
https://www.kickstarter.com/projects/privee/privee?ref=beibnw
Privee is an interesting project alright! The project creators are based in Hong Kong, and are working from a desire to have more control over their communications. The plan is to run homeservers on a small computer physically located with the user, similar to plans outlined by NovaChat.
This is an early-stage project (hence starting on KickStarter.) While exciting and ambitious, I encourage everyone to do their own research! Learn more and chat with terry_hello (who has done a good job handling questions already!) in #Privee:matrix.org.
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
Handle SAS verification by adding an endpoint where the decimal SAS code generated by another device can be sent. If this code matches with go-neb's generated code, the verification succeeds and the device is marked as trusted.
Add a cryptotest service which can be used for other clients to test their functionalities related to e2ee. This currently includes a challenge / response, key forwarding functions (key request / key forward) and SAS verification. SSSS and cross-signing are planned for the future. (PR is under review)
v1.2.0 is out now, with a bunch of bug fixes and some added features:
Thumbnail support for apng and webp images (thanks Sorunome!)
Experimental (but pretty stable) support for Redis as a cache - instructions here
Optional oEmbed support for things like previewing tweets - turn it on here
Dynamic thumbnail support
Per-user quotas
v1.2.0 is also much faster at file uploads, however some areas of matrix-media-repo can still be improved - general performance work is planned for the next release.
We are happy to announce the beta milestone for our first versioned release of Dendrite, which we anticipate to land in the next few weeks. Once these issues are cleared, we'll have made the big breaking changes that we want to make and will have a migration mechanism in place for database schema upgrades.
This will make life significantly easier for anyone who wants to start experimenting with and running their own Dendrite homeservers without unexpected breakages. More updates to follow soon - join us in #dendrite:matrix.org for the latest!
This week, Dendrite's end-to-end encryption support has also been mostly rounded off with the exception of a few minor bugs and edge-cases.
Changes this week include:
TLS certificate verification is now enabled by default
A new federation_disable_tls_validation config option has been added to override the above
Added support for processing inbound device lists from federation
Local device list changes are now sent outbound over federation
One-time key counts are now sent in /sync
Direct server signing key fetches are now limited to 15 seconds
State events that fail signature checks on federated room joins are now properly ignored
A number of SQLite database lock bugs have been fixed using TransactionWriter
A bug when finding joined hosts in the federation sender in SQLite mode has been fixed
Logging levels have been adjusted in a number of areas
Spec compliance:
Client-Server APIs: 51%, same as last week
Server-Server APIs: 54%, up from 53% last week
Asked whether this progress means that matrix.org would be running on Dendrite next year, Neil said he was certain that it would definitely be ready:
This week in Synapse land we continued with our performance push making further progress on the async/await-athon. We will move into the storage layer this coming week which will leave the caching layer and a few other bits outstanding. Separate work continued to get the event persister into a shardable state. The first step being to replace the event stream token with its own type and then threading that through the code base.
We have also continued to improve on our push capabilities and released a new version of Sygnal (0.8.2) with support for pushing to iOS devices via Firebase.
We are also very close to getting a workable version of the new push rules. The idea is to test it out in the wild before getting it enshrined in the spec.
Next week will see a continued focus on sharding and async/awaiting, more notifications work, logging improvements, and a host of bug fixes!
I released v0.2.0 of the maubot RSS plugin. The most recent addition is support for JSON feeds, stolen frominspired by FantasyCookie17's jsonfeed-bot.
Other changes since v0.1.0:
Per-feed toggle to send posts as m.text instead of m.notice
Automatically moving subscriptions across room upgrades
Some fixes for handling weird feeds
maubot/rss is a maubot plugin that posts feed updates to Matrix rooms. It supports RSS/Atom/JSON feeds, customizing message templates per-feed/room, and more.
All my bridges now have options for automatic e2ee key sharing in portal rooms. Currently they don't have verification support (other than manually marking as verified in the database), so enabling key sharing isn't entirely secure yet. I'll probably invent some sort of web verification interface after nikofil adds cross-signing to mautrix-go.
This also means mautrix-python now supports sending and responding to key requests. I'll port the interactive verification and future cross-signing code from mautrix-go to mautrix-python at some point too.
4.4.0 is out as promised last week! Get it off Flathub now.
After we announced our intention to release, we discovered a regression that prevented joining new rooms because there was a mixup between room aliases and room id during a switch from internal structures to ruma-identifiers. Luckily we managed to fix it in time and make the release as intended.
Element Android 1.0.4 π has been released to the store, it contains several bug fixes, and is adding back some moderators functionality (see list of banned users for a room + actions)
The team has been mainly working on stabilisation + refactoring of the current code base.
We are also trying to give more visibility on what is coming next:
SchildiChat is a new fork of Element for Android (previously RiotX), which features some UI changes, to make it more comfortable to use for people who are used to other common instant messengers.
The most important changes are the following:
The overview screen lists all chats in one list instead of the tabbed layout in Element
Message bubbles (optional)
Theme choice based on the system-wide dark/light theme on recent Android versions
It is now available on Google Play, it might appear on F-Droid too in the future.
As per this issue comment, I was asked to inform this channel of the new AppImage for the Element Desktop Matrix Client. π. Interested users may get them on appimage.github.io or on my releases page, offering Continuous builds (refreshed every day) for stable and beta builds π
I have changed the archive feature. Added a splash screen. Further simplified settings. Login times work faster compared to FC. I am connecting it to my Jitsi instance. which operates p2p and is completely analytics free. Message notifications are analytics free. It will eventually be an extension to a videocall app developed by myself. It adds variety to the ecosystem. It strictly follows license rules, credits the developers and links to their wonderful work. And it is a great thing to do in my spare time βΊοΈ
Hey all! I built yet another bot library. matrixbz was built with the intention of making it easy to draft matrix bots. Check out the github - I've added some examples where you can build a bot in ~10 lines of python code. matrixbz features:
Auth - you can specify user(s) who are authorized to call commands. bot only accepts invites from those users.
Cache - you can cache results for particular command invocations.
Hemppa the bot is a general use bot that has lot of modules for various uses and super easy API to create more. Hemppa received support for Mastodon. Now you can login to your Mastodon account and send toots via Matrix. Next major feature would be to subscribe to hashtags and users and get notifications to Matrix rooms. https://github.com/vranki/hemppa#mastodon
The Hackers on Planet Earth conference went almost entirely self-hosted this year due to not being able to hold an in-person event, and used Matrix to go virtual! hopeless was the robot-heart of the conference, providing community+room ACLs for attendees. The bot also has a bunch of other features including:
configured admins sending notices to rooms
ability to manually and programmatically schedule announcements for scheduled events happening, like workshops or talks
the ability for users to re-join rooms and generate invites for themselves as necessary
admin commands for inviting attendees and presenters without tickets
an attempt to be friendly and solve people's common problems without a real person having to assist (surprisingly effective!)
Many thanks to the nio-template for existing, otherwise this wouldn't have been possible!
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
[Matrix will] be rolled out by the German education system to provide collaboration tools for half a million seats in the states of Schleswig-Holstein and Hamburg
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
In terms of Spec Core Team MSC focus for this week, we'll be continuing on with MSC2674 (aggregations pt1: relationships) and adding MSC1544 (QR code verification) on top.
E2E Dendrite work is progressing, and we're making a lot of breaking changes ahead of our first stable release which you can expect in the coming weeks.
As for the changes this week:
Add a bare bones user directory. This is mainly to benefit the iOS Yggdrasil demo.
The dendrite-config.yaml configuration format has been overhauled. It's still in review but expect big changes here!
Fixed a bug with SQLite when handling large rooms (thanks @HenrikSolver!)
Device lists now work locally, with federation support to come.
This week we shipped 1.18.0 which contained support for our worker sharding performance projects that youβve been hearing about recently.
We also shipped a security and feature release of Sygnal (actually two eachβ¦). The main new feature being the ability to run behind an HTTP proxy which is useful in environments that mandate that all traffic go through a proxy.
Aside from that week we continued our async/await-athon and Patrick provided a visual update
Weβve been documenting worker support and diving into sharding the event persisters. This is not a small job since it means every other part of the code base that assumes an integer stream id will need to be updated to be aware of multiple stream ids from multiple persisters.
Over in feature land weβve been working on the notifications project and most of the backend support is done. The project will mean more intuitive defaults as well as a much better client UX in configuring notification settings.
Just pushed the 1.18.0 K8s-optimized Synapse images, now only doing the debian-based version as like upstream - though still including jemalloc. So latest/v1.18.0 and *-debian will be the same in this version and going forwards. Additionally, the *-debian tags will only remain for a release or two more before disappearing, as they're now superfluous.
I finally added automatic backfilling options to mautrix-telegram. Like my other bridges, it can backfill old messages when creating portals and missed messages when restarting, and it can disable notifications in the room while backfilling.
I also added bridging for your own read receipts, so if you read a message with another Telegram client, the bridge will mark it as read on your Matrix account with double puppeting.
gomuks can now do interactive verification for e2ee. Most of it is thanks to nikofil's SAS verification PR to mautrix-go, I just made a UI for using it. Currently gomuks can only send verification requests, but I'll add some interface for accepting incoming requests soon.
Related to verification, there are some new commands for verifying/unverifying/blacklisting devices and viewing device lists. There's also a toggle to disable sending messages to unverified devices.
This is super exciting, raises gomuks to the level of a daily driver IMO.
SAS verification support has been merged! Support for it should also be landing in go-neb soon, both for verifying with the bot's device and for testing with go-neb's client testing service. Next up, looking at SSSS and eventually cross-verification support.
We, some German Synapse admins, have created a survey about the usage of Matrix-Synapse for private statistic purpose (results will be published alike). As suggested in #synapse:matrix.org, this could also be a topic for "TWIM".
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
The most important thing to know about 1.18.0 is that it contains support for sharding multiple workers. Specifically this means being able to run multiple federation senders, multiple client readers to handle registration and multiple push workers. This will be important for anyone running a large scale install of Synapse. You can read more about how to benefit from these changes in docs/workers.md. In the same spirit we also moved typing notifications from the main process.
Aside from that, we have new admin API support to list the users in a room, support for oEmbed for media previews (you can unfurl tweets again!) and a general slew of federation bug fixes.
Get the new releases from any of the usual sources mentioned at https://github.com/matrix-org/synapse/blob/master/INSTALL.md. 1.18.0 is on github here.
Fix detection of out of sync remote device lists when receiving events from remote users. (#7815)
Fix bug where Synapse fails to process an incoming event over federation if the server is missing too much of the event's auth chain. (#7817)
Fix a bug causing Synapse to misinterpret the value off for encryption_enabled_by_default_for_room_type in its configuration file(s) if that value isn't surrounded by quotes. This bug was introduced in v1.16.0. (#7822)
Fix bug where we did not always pass in app_name or server_name to email templates, including e.g. for registration emails. (#7829)
Errors which occur while using the non-standard JWT login now return the proper error: 403 Forbidden with an error code of M_FORBIDDEN. (#7844)
Fix "AttributeError: 'str' object has no attribute 'get'" error message when applying per-room message retention policies. The bug was introduced in Synapse 1.7.0. (#7850)
Fix a bug introduced in Synapse 1.10.0 which could cause a "no create event in auth events" error during room creation. (#7854)
Fix a bug which allowed empty rooms to be rejoined over federation. (#7859)
Fix 'Unable to find a suitable guest user ID' error when using multiple client_reader workers. (#7866)
Fix a long standing bug where the tracing of async functions with opentracing was broken. (#7872, #7961)
Fix "TypeError in synapse.notifier" exceptions. (#7880)
Fix deprecation warning due to invalid escape sequences. (#7895)
πNew guide for client e2ee implementation, including cross-signing
Thanks and congratulations to Sorunome this week for releasing a new guide: Implementing more advanced e2ee features, such as cross-signing. This is a hugely detailed guide detailing the necessary steps to enable Cross-Signing and verification. Thanks as well to uhoreg and poljar for their work in reviewing the text!
I wrote a master thesis that tries to create e2e encryption based on Ethereum on top of Matrix. It may not solve all problems but it's an interesting experiment
I have put the long-awaited MSC2312 out of WIP - this is about matrix: URIs in case you haven't heard - and it's already in active discussion at #uri-scheme-proposal:matrix.org! Join the fun to make it the most awesome! The upcoming Quotient 0.6 has an implementation of those so adventurous Quaternion users can start experimenting right away.
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
In terms of Spec Core Team MSC focus, we've reduced last week's list of 5 MSCs down to one, MSC2674 (aggregations part 1: relationships). Next week, we're going to focus entirely on that MSC to make some headway on finally getting aggregations into the spec.
The P2P Yggdrasil iOS demo has seen some improvements:
Federation sender blacklists are now persisted which is important as the iOS app will be frequently terminated.
SQLite3 now uses TransactionWriters in more places to reduce the amount of 'database is locked' issues.
Federation sender has had much of its storage code refactored and de-duplicated.
In addition, there has been much work getting Dendrite to support E2E rooms, which it now mostly does.
You can create and join E2E rooms and have a conversation locally, though there's issues with it over federation. In addition,
Dendrite currently lacks device lists so adding new devices may result in unable-to-decrypt errors:
Send-to-device events now work over federation.
Device key uploads and querying now work both locally and over federation.
One-time key uploads and claiming now work both locally and over federation.
Spec compliance is up from last week:
Client Server APIs: 49%, up from 48%.
Federation APIs: 53%, up from 51%.
Doesn't it feel like we'll be using Dendrite in the wild really soon now?
This week we continued on our async/await athon, you can track progress here, expect steady progress over the coming weeks.
We dusted off the notifications project working with our pals in the Element client teams to ensure that the push rule defaults make sense. On the flip side we made further improvements to Sygnal and will merge the ability to have it sit behind a proxy rsn.
We are also trying to figure out how media retention limits should work in Synapse.
Up next is to continue our performance push. We need to spend time between pulling things off the master process as well sharding the event persister. We will continue with notifications and maybe finally squash https://github.com/matrix-org/synapse/issues/2528
Hello people. It's been a while since I've talked about a new bridge...or actually any bridges. Fear not, now you can hear both! Today I'm announcing the birth of another project matrix-figma. Figma is an online collaborative design tool. It was created to scratch an itch with the Element design team, who wanted to see comments appear in their Matrix rooms in realtime when someone commented on a Figma file.
The bridge works by first joining it to an "admin room", which functions as a permitted user list. The bot can then be invited to any other room so long as the invitee is in that admin room, and can then ask the bot to start directing comment notifications into the room.
The room uses room state to hold configuration, so you can host the bot as a docker container anywhere without the need for any support files!
I can announce that the bridge is now functional for simple use cases such as these, with more interesting functionality in the future.
This Python project was initially intended for a full migration from Slack to a new Matrix homeserver and allows reusing the imported user accounts. We have some research groups using Slack that would like to import their Slack workspace history to our Matrix homeserver. This PR allows to import a (free) Slack export to a fresh and empty Synapse instance that is federated with our main homeserver. We do not need a full "migration": we kick all imported users and invite the existing Matrix users from our homeserver.
Two groups already imported the history and switched to Matrix, one more is in queue and probably more to come.
This is a really, really exciting development, and the context for which Sven is expecting to use it is BIG, I can't wait til we can share more about it!
The Twitter DM bridge I announced last week mostly works now. It does text and reaction bridging in both directions, twitter->matrix media bridging, end-to-bridge encryption and backfilling. I also added support for it in mautrix-manager for web-based login.
If it mostly works, I guess it's time to mostly start trying it out!
Fluffy 0.16.0 has been released! It is already available on F-Droid, Google Play and IOS Testflight will follow. You can also try it out in the webbrowser. Visit https://fluffychat.im
Last week, we forgot to announce that we renamed the Riot-iOS app to Element but this is not a surprise anymore. The app version is now 1.0.0. This change came with a lot of UI tweaks like new icons and new colors.
Since then, we made bug fixes we will release soon. This week, we also worked on PIN protection for the app, including support of Touch ID and Face ID.
Quotient 0.6 released! The release notes are somewhat long, since it's been 4 months since the last stable version update, and more than a year since the 0.5 release - thanking two heroes of this release, and giving a nod to the original Quaternion author who happened to contribute using an inverted time machine. Read here: https://github.com/quotient-im/libQuotient/releases/tag/0.6.0
Big news - first release in a year. Do take a look at the full notes, they're very readable. In particular, they call out contributions from Black Hat, Alexey Andreyev and the original project creator, Felix Rohrbach aka @fxrh.
We weren't in TWIM for a while, but that doesn't mean no progress! Apart from Devin's constant work on ruma-events, there was also some movement in ruma-identifiers and ruma-client-api, with the former seeing a new release, 0.17.0.
We're planning to do more releases next week, and hopefully a 0.1 release of the ruma crate that provides a simpler entry point by re-exporting everything else, in one of the weeks after that. Stay tuned!
It's been a while since it's come up, but matrix-wug is a bot capable of transcribing characters writable via normal keyboards to scripts from other languages/notation systems
It supports: x-sampa, z-sampa, proto-indo-european, Inuktitut, and IΓ±upiatun.
Now it also supports hiragana! The romanization system is something like WΔpuro rΕmaji, but sticks to Nihon-shiki when in doubt.
I want this transcription to be forgiving, so if you find it not transcribing something in a way you'd expect it to, feel free to DM me!
To use it use the hi key with a delimiter like /x/ or []
hi/konnichiwa maatorikusu!/
vΓΈgg is also behind pantalaimon now so it now works in encrypted rooms π
If you need to convert swiggles into a different type of swiggle, dandellion has you covered here.
This bot is not about adapting the current jitsi integration provided by Element IM as widget.
If you have a configured jitsi server to be authenticated with jwt tokens as in https://github.com/jitsi/lib-jitsi-meet/blob/master/doc/tokens.md , this bot will assist you when claiming for a jwt instead of going to jwt.io each time.
You'll need to create a user and invite you in each room where you want to create a jitsi room url respecting the matrix power levels in the room.
When the service is running invite the bot and send message !jitsi-jwt
If you are a moderator in the room (alias you have enough power levels to redact messages), then you will get a direct message from bot with the url to jitsi
room with the jwt get parameter.
New cody release. @cody:bordum.dk is a chat bot that works as a REPL for your matrix rooms. This week I worked on metrics. I host a Grafana instance with anonymous access allowed, so codys charts are now embedded the #cody:bordum.dk chat room.
I am proud to announce the very first release of sMythbot, the Matrix chatbot designed to control your Myth Tv DVR remotely. This release should be considered a tech preview. I welcome feedback and Bug reports. More information on installation and setup is available in the project wiki.
It currently supports the following commands
!smythbot help: Display this message
!smythbot set mythbackend address: Sets the Myth Tv backend address to use for this room.
!smythbot set mythbacked port: Sets the Myth Tv backend port to use for this room.
!smythbot view mythbackend address: Allows you to view the Myth Tv backend address set for this room
!smythbot view mythbackend port: Allows you to view the Myth Tv backend port set for this room
!smythbot view mythbackend info: Allows you to view various pieces of information for the Myth Tv backend connected to this room. It will not work if the address and port are not set.
!smythbot display upcoming recordings: Displays the upcoming recordings on your Myth Tv Backend.
!smythbot display recorded programs: Displays the recordings from the default recording group that are stored on your Myth Tv Backend.
The team behind Noteworthy (Matrix over WireGuard overlay networks) has started work on bringing better mobile web support for Element (riot-web).
Noteworthy Elements is a lightweight shim powered by the Ionic Framework thats goal is to bring first-class support for Element running on mobile devices.
Usable version of Element wrapped in native iOS / Android app
Ability to run multiple instances of Element in a native iOS / Android app (ie connect to multiple home servers simultaneously)
Our initial experimentation has been positive and our goal is to make Element the best mobile client for Matrix with minimal fuss. With minimal changes to Element's codebase our goal is to land (significantly better) support for mobile web in upstream Element. Join us over at #noteworthy:tincan.community to get involved!
πTrust-based Moderation Using Distributed Chat Systems for Transitive Trust Propagation
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
Let's hear from Christian about working on his dream projects, his thoughts on bridging, hosting, and the importance and nature of chatbots and assistants.
This week was the announcement of the much-awaited rebrand: Riot is now Element. In fact, three brands are coalescing into one: Riot and New Vector will be referred to as Element, while the SaaS platform known as Modular.im is now Element Matrix Services.
Note that Matrix is not involved in this change. Matrix is still Matrix, don't worry about that!
CommCon is an event dedicated to Real-Time Communications. In 2020 they made the difficult decision to go online-only, but had a fear of missing out on the "hallway-track" that is so important to industry events. Their solution was to include a live chat to run alongside their streamed talks!
Matrix was a natural choice for the crowd, but they wanted a way to encourage viewers to join the correct room from their own Matrix clients. To do this, they used the GSOC project from arnav-t - an HTML Embedded client - to present a live scrolling-view of the chat. You can see the result at https://2020.commcon.xyz/live/.
This week, the Spec Core Team will be focusing on MSC2610 (Removing m.login.oauth2 from User-Interactive Authentication), MSC2611 (Removing m.login.token from User-Interactive Authentication), MSC2663 (Errors for dealing with non-existent push rules), MSC2674 (Event Relationships), and MSC2689 (Fix E2EE for guests).
This week in the ruma/matrix Google Summer of Code project, I worked on refactoring both ruma-api and ruma-events. After moving some of the larger chunks of the ruma_api_macro::api::Api::to_tokens method to helper functions, I spent time removing repetition from the Request/Response code generated by the ruma_api! macro. For ruma-events, the input parsing was changed to only allow valid names for the Any*Event enums. Altering the input parsing had the added benefit of replacing all of the string comparison and manipulation with strongly typed comparison and manipulation.
The final few issues to be resolved before the next crates.io release for ruma-events can happen are related to redacted events. Support for redacted events was added to the Any*Event enums, they now have redacted variants of each event kind. A few follow-up PR's have been merged to fully integrate redacted events into ruma-events, fixing specific event deserialization issues and splitting the UnsignedData struct into Unsigned and RedactedUnsigned.
I also investigated why so many sytests fail and created this issue: https://github.com/matrix-org/sytest/issues/913
Apparently Sytest did not expect a server to optimize their /sync responses as much as Conduit does :P.
Hot on the heels of shipping shardable federation readers last week, this week we shipped shardable federation senders. There is still plenty of work to do, but together these changes has made a massive difference to federation lag overall and hopefully those of you not on matrix.org are noticing the difference when you talk to matrix.org users.
This graph shows the impact to outbound federation lag.
Aside from that we also shipped shardable push and frontend proxy workers as both were starting to max out on CPU as well as a shardable client reader, allowing us to shard registration which was especially important this week :)
Finally we moved typing notifications from the master process and optimised incoming replication queuing to buy us a little more head room.
Next steps are to revisit where all the remaining cycles are going on the master process. To help us profile we are migrating to async/await semantics and Patrick produced this natty graph to track progress.
Outside of performance we shipped a bug fix to prevent large initial syncs taking out the synchrotrons. The admin api sprouted an end point to list room members (thanks awesome-michaeland the ability to reactivate previously deactivated users.
Coming up weβll dust off the notifications project which has been put on hold while Riot transmogrified into Element and weβll continue with chipping away at the master process.
We only featured this a month ago, but I want to call attention again to this awesome Synapse version adoption tracking project from Chris . I wanted to know how quickly Synapse 1.17.0 would be the most deployed version (~36 hours), and the answer is right there!
I started working on a Twitter DM bridge that uses the internal API instead of the complicated official one. It should be easier to set up than bridges using the official API, since there won't be a need to get API keys and forward webhooks. The main risk is that Twitter doesn't like people using the internal API and starts blocking users like Facebook does, but hopefully that doesn't happen.
PlayStore users: Element should come as a regular upgrade of the application Riot-Android, and the upgrade will update your previous account and data, without the need to log in again.
F-Droid users: Element is a new app, you have to install it and log in again.
RiotX: RiotX will disappear from the PlayStore (it was only a beta application). We will provide a very last update to inform users to install Element Android
Please report any issues at https://github.com/vector-im/riotX-android (which will probably be renamed soon), because Element Android is actually RiotX code!
Thanks for all the contributors of RiotX, we still have lots to do to make the app even better and full featured.
After seeing all the Element room avatars, I thought to myself, "you know what, #radical-webext:matrix.org needs a new icon"β¦ so I sat down and combined a couple of icons. The result:
Radical also quickly (as always) got updates when Element 1.7.0 and 1.7.1 were released. Huge thank you to stoic for making Radical in the first place.
I updated my F-Droid repos which are containing the dev builds to reflect the latest naming change to Element.
The people who formerly used my repos for getting the dev repo of Riot-Android should update to the new repo. It wont be updated to get Element
As always you can grab the F-Droid or GPlay flavor and you can pick the repo which reflects your flavor on https://fdroid.krombel.de
Note: As "the new repo" is internally the same repo as the one of RiotX (and Riot.imX) the people who already used that don't need to update their packet source. They will get the updates to Element via the old repo urls.
As promised last week, maubot now supports end-to-end encryption. It's a bit bare at the moment, so you need to manually insert the device ID into the database for it to start working, but other than that it works. Plugins don't need to be changed at all, the framework will handle everything.
While I was adding e2ee to maubot, I also improved mautrix-python's crypto stuff so that it's easier to use it directly as a client library with e2ee. Some day I might even add docs :D
Related to my previous update here, I've just published an initial version of a Matrix Client-Server client library for the Crystal programming language: https://github.com/cvincent/matrix-client-cr
Is this the first time we've seen tooling for Crystal?
We have a few ways to handle the migration depending on how much change and breakage you're willing to tolerate. See our Migrating to Element documentation page.
Matrix-Alertmanager bot has a new release of v0.3.0. Highlights are compatibility with AWS Lambda, better error handling if message fails to send and the ability to do a @room mention on firing alerts. Find it here.
This week, we got support for javascript (nodejs) with !js. The code
complexity was considerably reduced, which should make adding new languages
much easier in the future. All Python sub-dependencies are now pinned, making
Noteworthy team (patrick and myself) have open sourced both components (spoke & hub) of Noteworthy (Matrix over Wireguard overlay networks -- https://github.com/decentralabs/noteworthy) we are actively welcoming testers/contributors and working with a handful of projects on incorporating our deployment model over in #noteworthy:tincan.community -- it is also the fastest way (less than a minute) to deploy your own home server! Also, working on incorporating Noteworthy into the popular ansible deployment repo to enable users who don't have access to a publicly accessible to deploy their own home servers.
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
Hot on the heels of Synapse 1.16.1, 1.17.0 is a bug fix release most notably containing a fix for 'stuck invites' which happen when we are unable to reject a room invite received over federation.
Get the new releases from any of the usual sources mentioned at https://github.com/matrix-org/synapse/blob/master/INSTALL.md. 1.17.0 is on github here.
Fix inconsistent handling of upper and lower case in email addresses when used as identifiers for login, etc. Contributed by @dklimpel. (#7021)
Fix "Tried to close a non-active scope!" error messages when opentracing is enabled. (#7732)
Fix incorrect error message when database CTYPE was set incorrectly. (#7760)
Fix to not ignore set_tweak actions in Push Rules that have no value, as permitted by the specification. (#7766)
Fix synctl to handle empty config files correctly. Contributed by @kotovalexarian. (#7779)
Fixes a long standing bug in worker mode where worker information was saved in the devices table instead of the original IP address and user agent. (#7797)
Fix 'stuck invites' which happen when we are unable to reject a room invite received over federation. (#7804, #7809, #7810)
