This release fixes a denial of service vulnerability (GHSA-hxmp-pqch-c8mm / CVE-2020-26257) in which a malicious homeserver could send malformed events into a room which would then break federation of that room.
This follows the disclosure of a denial of service vulnerability in OpenSSL (CVE-2020-1971). If you have installed Synapse from source, please ensure your host is up to date and then execute pip install 'cryptography>=3.3' inside your Synapse virtualenv.
We've also released Synapse 1.23.1 which includes that security fix and a small patch to maintain Python 3.5 compatibility. It is otherwise identical to 1.23.0. Note that Synapse 1.24.0 includes backwards incompatible changes which may affect a small number of users. See the notes on upgrading for more information.
Synapse 1.24.0 brings a pair of new Admin APIs, including a way to log in as users and to forcibly purge rooms when deleting them. We've also made numerous bug fixes and improvements to SSO support, especially around OpenID Connect and SAML providers.
This release includes an optional change to push notification badges: currently, the number in the badge is based on the count of rooms with unread messages. However, in some specialized cases you may want the badge to show the count of all unread messages, even if there are multiple unread messages in the same room. This behavior can now be toggled with a new configuration setting.
Additionally, for server admins, the deprecated /_matrix/client/*/admin Admin API endpoints have been removed. If you have tools which target these endpoints, please update them to use the /_synapse/admin URL prefix instead.
It’s been just over 2 months since we revealed that Gitter was going to join Matrix - and
we are incredibly proud to announce that Gitter has officially turned on true native Matrix connectivity:
all public Gitter rooms are now available natively via Matrix, and all Gitter users now natively exist on Matrix.
So, if you wanted to join the official Node.js language support room at https://gitter.im/nodejs/node
from Matrix, just head over to #nodejs_node:gitter.im and *boom*, you’re in!
This means Gitter is now running a Matrix homeserver at gitter.im which exposes all the active public rooms - so if you go to the the room directory in Element (for instance) and select gitter.im as a homeserver, you can jump straight in:
Once you’re in, you can chat back and forth transparently between users on the Gitter side and the Matrix side, and you no longer have the ugly “Matrixbot” user faking the messages back and forth - these are ‘real’ users talking directly to one another, and every public msg in every public room is now automatically exposed into Matrix.
So, suddenly all the developer communities previously living only in Gitter (Scala, Node, Webpack, Angular, Rails and thousands of others) are now available to anyone anywhere on Matrix - alongside communities bridged from Freenode and Slack; the native Matrix communities for Mozilla, KDE, GNOME communities etc. We’re hopeful that glueing everything together via Matrix will usher in a new age of open and defragmented dev collaboration, a bit like we used to have on IRC, back in the day.
This is also great news for mobile Gitter users - as the original mobile Gitter clients have been in a holding pattern for over a year, and native Matrix support for Gitter means they are now officially deprecated in favour of Element (or indeed any other mobile Matrix client).
Now, this is the first cut of native Matrix support in Gitter: much of the time since Gitter joined Element has been spent migrating stuff over from Gitlab to Element, and it’s only really been a month of work so far in hooking up Matrix. As a result: all the important features work, but there’s also stuff that’s yet to land:
Features ready today:
Ability to join rooms from Matrix via #org_repo:gitter.im
Bridging Edits, Replies (mapped to Threads on Gitter), Deletes, File transfer
Bridging Markdown & Emoji
What remains:
Ability to send/receive Direct Messages
Ability to plumb existing Matrix rooms into Gitter natively
Synchronising the full Gitter membership list to Matrix. Currently the membership syncs incrementally as people speak
Turning off the old Gitter bridge
Bridging emotes (/me support) (almost landed!)
Bridging read receipts
Synchronising room avatars
Bridge LaTeX
Stuff we’re not planning to support:
Ability to join arbitrary rooms on Matrix from Gitter. This could consume huge resources on Gitter, and we’re not in a rush to mirror all of Matrix into Gitter. This will get addressed when Gitter merges with Element into a pure Matrix client.
Bridging Reactions. Gitter doesn’t have these natively today, and rather than adding them to Gitter, we’d rather work on merging Gitter & Element together.
For more details, we strongly recommend checking out the native Matrix epic on Gitlab for the unvarnished truth straight from the coal-face!
🔗How do you make an existing chat system talk Matrix?
In terms of the work which has gone into this - Gitter has been an excellent case study of how you can easily plug an existing large established chat system into Matrix.
At high level, the core work needed was as simple as:
This can be accomplished by simply adding a virtualUser property to your chat message/post/tweet schema which holds the mxid, displayName, and avatar as an alternative to your author field. Then display the virtualUser whenever available over the author.
This "application service" comes pre-packaged for you in many cases, so for example you can simply drop in a library like matrix-appservice-bridge in a Node.js application, and all of the Matrix talking complexity is handled for you.
Polish it!
In practice, Eric (lead Gitter dev) laid out the waypoints of the full journey:
First big step was to add the concept of virtual users to Gitter. We could also have created a new Gitter user for every new matrix ID that appears, but tagging them as virtual users is a bit cleaner.
Figuring out how to balance the Matrix traffic coming into/out of Gitter.
Spreading the inbound load comes for free via our existing load-balancer setup (ELB) where we already have 8 webapp servers running the various services of gitter.im. We just run the Matrix bridge on those servers alongside each web and api process, and then the load-balancer’s matrix.gitter.im spreads out to the servers.
Events from Matrix then hit the load balancer and reach one of the servers (no duplication when processing events).
If something on Gitter happens, the action occurs on one server and we just propagate it over to Matrix (no duplication or locking needed).
We have realtime websockets and Faye subscriptions already in the app which are backed by Mongoose database hooks whenever something changes. We just tapped into the same thing to be able to bridge across new information to Matrix as we receive it on Gitter.
Hooking up the official Matrix bridging matrix-appservice-bridge library to use Gitter’s existing MongoDB for storage instead of nedb.
Figuring out how to namespace the mxids of the gitter users:
It’s nice to have the mxid as human readable as possible instead of just the numerical userId in your service.
But if people can change their username in your service, you can’t change your mxid on Matrix. In the future, we’ll have portable accounts in Matrix to support this (MSC2787) but sadly these are still vapourware at this point.
If you naively just switch the user’s mxid when they rename their username, then you could end up leaking conversation history between mxids(!)
So we went with @username-userid:gitter.im for the Matrix ID to make it a bit more human readable but also unique so any renames can happen without affecting anything.
For room aliases, we decided to change our community/room URI syntax to underscores for the room aliases, #community_room:gitter.im
Figuring out how to bridge features correctly;
Emoji - mapping between :shortcode: and unicode emojis
Mapping between Gitter threaded conversations <-> Matrix replies
Mapping between Matrix mentions and Gitter mentions
Keeping users and room data in sync
We haven’t gotten there yet, but the data comes through the same Mongoose hook and we can update the bridged data as they change on the Gitter end.
Meanwhile, the Matrix side of gitter.im is hosted by Element Matrix Services and is a plain old Synapse, talking through to Gitter via the Application Service API. An alternative architecture would be to have got Gitter directly federating with Matrix by embedding a “homeserver library” into it (e.g. embedding Dendrite). However, given Dendrite is still beta and assumes it is storing its data itself (rather than persisting in an existing backend such as Gitter’s mongodb), we went for the simpler option to start with.
It’s been really interesting to see how this has played out week by week in the Gitter updates in This Week in Matrix: you can literally track the progress and see how the integration came to life between Oct 9, Oct 23, Nov 6, Nov 27 and finally Dec 4.
Huge thanks go to Eric Eastwood, the lead dev of Gitter and mastermind behind the project - and also to Half-Shot and Christian who’ve been providing all the support and review from the Matrix bridging team.
First and foremost we’re going to be working through the “What remains” section of the list above: killing off the old bridge, sorting out plumbed rooms, hooking up DMs, importing old Gitter history into Matrix, etc. This should then give us an exceptionally low impedance link between Gitter & Matrix.
In the medium/long term, it’s simply not going to be efficient for the combined Element/Gitter team to split our efforts maintaining two high-profile Matrix clients. Our plan is instead to merge Gitter’s features into Element (or next generations of Element) itself and then - if and only if Element has achieved parity with Gitter - we expect to upgrade the deployment on gitter.im to a Gitter-customised version of Element. The inevitable side-effect is that we’ll be adding new features to Element rather than Gitter going forwards.
Now, that means implementing some features in Matrix/Element to match...
Instant live room peeking (less than a second to load the webapp into a live-view of a massive room with 20K users!!)
Seamless onboarding thanks to using GitLab & GitHub for accounts
Curated hierarchical room directory
Magical creation of rooms on demand for every GitLab and GitHub project ever
GitLab/GitHub activity as a first-class citizen in a room’s side-panel
Excellent search-engine-friendly static content and archives
KaTeX support for Maths communities
Threads!
...and this work is in full swing:
We have a proposal for fast peeking (via lazy-loading state over federation) at MSC2775 and the new peek APIs at MSC2753 and MSC2444 (and even implemented by Dendrite)
The only bits which aren’t already progressing yet are tighter GL/GH integration, and better search engine optimised static archives.
So, the plan is to get cracking on the rest of the feature parity, then merge Gitter & Element together - and meanwhile continue getting the rest of the world into Matrix :)
We live in exciting times: open standards-based interoperable communication is on the rise again, and we hope Gitter’s new life in Matrix is the beginning of a new age of cross-project developer collaboration, at last escaping the fragmentation we’ve suffered over the last few years.
Finally, please do give feedback via Gitter or Matrix (or mail!) on the integration and where you’d like to see it go next!
Taking place NEXT WEDNESDAY, the ninth edition of my favourite monthly Open Tech-themed live video broadcast! Subscribe to the calendar on https://matrix.org/open-tech-will-save-us/ and check out the lineup:
Angie Gaudion, Coordinatrice at CHATONS, presenting Mobilizon: a new libre events management tool https://joinmobilizon.org/en/.
Ben Francis, Founder at Krellian, presenting the relaunch of WebThings (previously from Mozilla)
Nad Chishtie, Product Lead at Element, presents Matrix Spaces: a new vision for room groups in Matrix
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
In terms of Spec Core Team MSC focus for this week, we've mostly been focusing on implementation. Thus the same 3 MSCs are in focus: MSC2844 (global versioning), MSC2366 (ready and done in key verification), and MSC2790 (modal widgets).
This week we released version 0.3.2, which contains some bug fixes. Changes this week include:
Checking if a server is allowed to see an event has been optimised somewhat
The ability to disable federation has been added
Application services are now sent correctly formatted client events
create-account has been simplified so that you can just point it to your Dendrite config
/unpeek support has been added (for MSC2753)
Event IDs are now calculated far less often in the roomserver, since this is an expensive operation
The federation sender now caches events in memory to reduce load on the database
The "since" parameter is now checked more thoroughly in the sync API, which led to a bug that could cause forgotten rooms to reappear (thanks to kaniini)
The polylith now proxies signing key requests through the federation sender correctly
⚠️ The OpenSSL project has pre-announced a high severity issue that will be disclosed on Tuesday. We will be regenerating Docker images and releasing Synapse 1.24.0 and 1.23.1 on Tuesday following that announcement. Note that these releases contain security fixes of their own, and you are encouraged to upgrade.
In other news, Synapse 1.24.0rc2 is out and includes numerous bug fixes and internal improvements, especially around our OpenID Connect (OIDC) and SAML providers. Of note, the OIDC provider can now re-try generating MXIDs when there's a collision, reaching parity with the SAML provider.
We've been talking about MSC2403: Add "knock" feature for a few weeks, but the end is in sight: the PR is reviewed and the tests pass, so we're on track for including it in 1.25.0! We're also on the cusp of a finishing a new algorithm for calculating the auth chain difference in State Resolution v2, which should significantly reduce worst case performance.
Lastly, the deprecated /_matrix/client/*/admin Admin API endpoints have been removed. If you have tools which target these endpoints, please update them to use the /_synapse/admin URL prefix instead.
The Docker ARM images for Dendrite have been updated for version 0.3.2. As always, they can be found in my docker repo at https://hub.docker.com/u/trslimey
We've soft-launched the new bridge to all public Gitter rooms! You can now find any public room on Gitter checking out the room directory for gitter.im or directly by replacing the forward slash in the URI part of the Gitter room with an underscore and joining #*:gitter.im(replace the *). If you come across any bugs, feel free check the list of known problems and create an issue. There will be a full announcement next week so stay tuned!
We've also added a few more features this week and now transform :emoji: syntax from Gitter to Unicode so they're compatible in Matrix, propagate the room topic/description, and some bug fixes around threaded conversations and replies. You can track our full progress from the GitLab epic for the native Gitter-Matrix bridge.
Massive milestone! Great work Eric, it's been such fast progress!
This was actually just the beginning, the ground work needed to lay the foundations for A Better Future. We already have a bunch of other merge requests to continue down that path. Hopefully there shouldn’t be such large ones in the (near) future. So what’s next? The work that landed this week was a switch to the SDK for almost all endpoint queries. Now need to handle the syncs with it. Later on, we will use it to handle storage, and then enable E2EE as well, but that will take quite some time.
Big progress from Fractal here! Will be good to hear more from the Fractal club to see what the ETA is for storage and E2EE.
Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE (with the notable exception being device verification for now) and intends to be full featured and nice to look at
While I've actually not had much time to work on Nheko, I did at least fix most of the UTD (Unable to decrypt) issues, when sending encrypted messages. This has been a long standing annoyance, but it should now finally not be an issue anymore (well... at least it should not be caused by Nheko as often anymore)! With this there are only a few tickets left to finally bring E2EE out of beta in Nheko (SSSS and bootstrapping cross-signing, as well as a general review for potential issues).
And while I have actually not been doing much work, we actually had more Pull Requests (and merged them too) this week than I have ever seen before in Nheko! Thank you very much to everyone who contributed. A short summary:
Nheko now uses the Nheko icon from your system theme, if available
You can finally look at avatars in their full resolution!
Megolm sessions are now lazily loaded from disk.
The registration page now also has nice inline error messages, hinting at why a specific input is unacceptable.
Updates to the russian translation
The README now lists all required GStreamer plugins for voice and video calls.
It feels super great, when you have so many people contributing, I love it! <3
Nheko really seems to be doing something right when it comes to attracting PRs!
We now have a public roadmap! We’ll be covering more details on it in this week’s Matrix Live, as well as a blog post on element.io soon.
Spaces
Work has started in earnest on the web and android implementations.
On the web side T3chguy completed the Spaces creation flows and is now looking at the Space landing page and Space post-creation prompts.
We have noticed that prefixing any word with ‘Space’ makes it sounds 20% cooler.
The mobile MVP will be read only and Valere will start with the ability to respond to invites and select Spaces.
So far it is looking very promising with lots of feedback from user testing informing the design.
Social login
We plan to ship this as a cross client feature this month. This week we completed the implementations for web and android with iOS and Synapse ongoing.
VoIP
Line 1 / 2 support is almost ready on web: ironing out some bugs and cosmetic issues, then PR will be ready for review. VoIP V1 support and hold/resume is getting close on Android & iOS, along with stability fixes.
We don't have much information about this one yet, other than that cryptovoxels have been interested in developing a Matrix client for their world for a while.
Hemppa the Bot is a multipurpose bot for writing modules super easily in Python. Last week Hemppa received a useful feature of printing files! You can set up a room for a specific printer and invite anyone you wish to it to allow printing. Any attachments uploaded to the room will be automatically printed. It's actually one of the easiest way to print from a phone and very useful if you have a printer that does not have drivers for modern Windows or MacOS versions. https://github.com/vranki/hemppa#printing
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
In terms of Spec Core Team MSC focus for this week, we're switching from QR Code verification to key verification additions. Otherwise things remain the same: MSC2844 (global versioning), MSC2366 (ready and done in key verification) and MSC2790 (modal widgets).
On behalf of the Americans on the Synapse team: Happy Thanksgiving! This past week saw us land some improvements to the OpenID Connect SSO provider, bringing it closer to parity with the SAML provider. In particular, #8801 made it possible to gracefully handle username collisions when creating a new account over SSO.
We've also merged a pull request (#8785) from @dklimpel which removes deprecated paths for the Synapse Admin APIs. As of the next Synapse release, all Admin APIs will be accessibly only under the /_synapse/admin/v1 path prefix.
Otherwise, much of our time has been spent on getting pull requests like #6739, implementing MSC2403: Add "knock" feature, ready to merge. It's coming to a Synapse release soon!
Speaking of releases, we're looking forward to cutting a first release candidate of 1.24.0 early next week; keep your eyes peeled for that!
I'd be remiss in not mentioning that we started the week by disclosing CVE-2020-26890, which is a denial of service affecting Synapse versions prior to 1.20.0. If you've not upgraded, please do.
Lots of features added this week including Gitter threaded conversations and Matrix reply bridging, image/file uploads, transforming mentions so notifications flow, and better looking flairs!
We'll be working through some of the bridge oddities and hoping to open up the bridge to all public rooms on Gitter soon. You can track our full progress from the GitLab epic for the native Gitter-Matrix bridge.
The MX-Puppet-VK bridge for VK (Vkontakte) social network via this commit have added initial support for User tokens. This means that we now can use it for puppeting the real user account, to automatically bridge all "VK Messenger" private dialogs and groups into Matrix. Previously bridge worked only with VK community tokens as bot, to bridge only conversation with specific communities.
The matrix-sms-bridge can finally be used with an Android smartphone as SMS-Gateway instead of gammu and a modem. @rebekka:imbitbu.de developed the app android-sms-gateway-server, that the bridge uses to send and receive sms messages. The new bridge version is used in combination with the app in production (medical care centre) for a week now. We have a throughput of 50-100 SMS message a day. The update is also integrated in matrix-docker-ansible-deploy.
Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE (with the notable exception being device verification for now) and intends to be full featured and nice to look at
The new completer is now on master. Please try it out and report any issues you may find. It now actually inserts a user link, keeps a per room message history and should look quite a bit nicer and be faster, when completing emojis or user names.
There were also quite a few cleanups, most of them done by Kirillpt! Some spacing here and there, saturation changes, the login page should now be less confusing, when you don't enter a valid mxid and toggles should now not get stuck in confusing intermediate states. There is also a russian translation now and some VOIP fixes. Pretty busy week, but I like it!
FluffyChat 0.23.0 is out now with password recovery options and a lot of minor fixes and UX improvements. Now also available on Flathub!! 🙂 Learn more at: https://fluffychat.im
On mobile, we’re busy adding better suggestions to empty/starting views of screens, improving searching for contacts, improving adding other users by QR code scanning, and improving deep link integrations on Android. Expect releases for iOS & Android soon!
Spaces
Spaces UI designs & user research testing are underway, with initial development for the Element Web client kicked off too. We've been doing research & testing in private to avoid advance bias from the people we're testing with, but hope to share things publicly soon.
Social login
Social login is in review for Element Web, with synapse & mobile support hot on the tails for next week prior to releasing.
VoIP
We’ve been focusing on implementing UI for call holding, and general UI polish across the VoIP experience, to be released soon!
Element Web 1.7.14 was released this week, which includes:
Several tweaks and improvements to the room list filter
Improved registration based on user feedback
Improved invite / create DM flow.
Future releases will no longer publish element-web to NPM, and the package is marked as deprecated
Renamed build artifacts from riot-.tar.gz to element-.tar.gz
Experimental LaTeX maths rendering support is available via a labs flag on develop. Thanks to akissinger, thosgood, uhoreg, and others who have worked on various pieces of this puzzle!
Element iOS
Element iOS 1.1.1 has been released:
It contains contains several major bug fixes, including end-to-end encryption bug fixes.
Full history available on GitHub viewing the 1.1.1 & 1.1.0 tags.
Element Android
The Element Android 1.0.11 release is pending:
We're working through final bug fixes.
Separately, we’re also working on adding missing features to the client like; managing room aliases, room history visibility, join rules, etc.
In the SDK, the migration of the services API from MatrixCallback to coroutines is going well, tracked here.
In addition to boring stuff like being able to persist events (hopefully) right, so we don't have to initial-sync every launch, I've been making a start towards multi-account support!
You can sign in as multiple users and switch between the "active" user you want to send messages as - it's a bit flaky for now, esp. when scrolling up to read past messages, but hey it's a start I guess
Asked if they were using an SDK:
Preeetty much doing it from scratch, though I am loosely basing the matrix-api side of it on Ruma
We gained a new contributor, kiwidu38, who added the server-default push rules, paving the way to Conduit supporting push notifications.
I improved compatibility with some commonly-broken DAG events and added rudimentary support for message edits to help Fractal finish their move onto matrix-rust-sdk.
I have not read this series so I must trust that Samuel is saying nice things! Anyway it seems like a useful resource for telling kameraden about getting started with Matrix.
Today there is a German article about Famedly on Gründerszene -> https://www.gruenderszene.de/health/famedly-kommunikation-app-gesundheitswesen. Famedly is developing a solution based on Matrix to enable healthcare providers to communicate securely with each other and to facilitate intersectoral processes such as patient transfers or the integration of IomT devices. We have also almost completed a successful financing round and hope to support the Matrix ecosystem more and more. It would be so cool to use FluffyChat, Element, or a Matrix Messenger of your choice to start a video consultation with the doctor of your choice, or to be integrated into existing clinical communication flows concerning you!
Congratulations to Famedly on the progress they're making! If you're ready to graduate from TLAs (Three-Letter-Abbreviations) to the next stage, you'll be please to learn "IomT", which means "Internet of medical Things".
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
Today we are disclosing CVE-2020-26890 / GHSA-4mp3-385r-v63f, a denial of service vulnerability affecting Synapse versions prior to 1.20.0. We strongly encourage all Synapse admins to upgrade as soon as possible. If you have not upgraded in a while, please refer to the upgrade notes, especially the latter portion of that document which covers any backwards incompatible changes which you may need to take into consideration.
As a best practice, we encourage Synapse admins to upgrade regularly, and either subscribe on GitHub or join #homeowners:matrix.org for low-traffic notifications of new releases.
We extend our thanks to Denis Kasak for reporting this issue, earning a second entry in the Matrix Security Hall of Fame.
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
In terms of Spec Core Team MSC focus for this week, we're sticking with the same three MSCs: MSC2844 (global versioning), MSC1544 (QR code verification) and MSC2790 (modal widgets). MSC2844 in particular I personally found quite interesting.
We started out this week by releasing Dendrite 0.3.0 and then ended the week with a bug-fix Dendrite 0.3.1 release.
I'd like to say thank you especially to our community contributors who have been adding new features and filling gaps!
Changes this week include:
Memory optimisation by reference passing, significantly reducing the number of allocations and duplications in memory
A concurrency bug has been fixed in the federation API that could cause Dendrite to crash
A hook API has been added for experimental MSCs, with an early implementation of MSC2836
Forgetting rooms is now supported (thanks to S7evinK)
The last seen timestamp and IP address are now updated automatically when calling /sync
The last seen timestamp and IP address are now reported in /_matrix/client/r0/devices (thanks to alexkursell)
An optional configuration option sync_api.real_ip_header has been added for specifying which HTTP header contains the real client IP address (for if Dendrite is running behind a reverse HTTP proxy)
Partial implementation of /_matrix/client/r0/admin/whois (thanks to DavidSpenler)
The error when registering a username with invalid characters has been corrected (thanks to bodqhrohro)
The -version command line flag has been added (thanks to S7evinK)
Backfilling should now work properly on rooms with world-readable history visibility (thanks to MayeulC)
Some more client event fields have been marked as omit-when-empty (thanks to S7evinK)
The build.sh script has been updated to work properly on all POSIX platforms (thanks to felix)
Spec compliance is unchanged, although some Synapse-specific tests have been removed and some new tests have been added:
We released Synapse 1.23.0 on Wednesday! Read all about it on the Matrix Blog. Otherwise, we'd like to highlight a few developments over the past week:
We're discussing a policy for ending support for old versions of Python and PostgreSQL. If you have opinions, please let us know on GitHub.
We've been looking at ways to improve the efficiency of state resolution, and Erik has managed to devise some algorithmic improvements that yield an order of magnitude speedup for a handful of pathologic cases. We hope to have a better idea of how this might work for real world workloads soon.
Lastly, we'd like to take this opportunity to remind you to please regularly upgrade your Synapse. Especially if you're not yet on 1.20.0, as we'll be disclosing a denial of service issue which affects older versions on Monday.
mewmew wanted to use my fork of Synapse, so I made the changes there configurable (instead of hardcoding to my user ID) and even added a readme to list the features. It might also be useful for other people who want custom room IDs and other fun stuff.
Support for receiving EDUs from matrix (typing, read indicators and presence) has been added. Protocol implementations can start using the new events, if they want to. Version 0.1.0 has been released along with this. You have to enable the de.sorunome.msc2409.push_ephemeral flag in your registration file
Hey, our team of colorful ponies proud to introduce you a new Matrix bridge to VK.com chats. It's powered by mx-puppet-bridge and so far supports almost all of core features people want to see, like replies, message edits and file attachments.
Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE (with the notable exception being device verification for now) and intends to be full featured and nice to look at
Profiles should now open again on older versions of Qt.
I've been slowly rewriting the message are to be qml, to make it easier to do fancy styling. As a result emojis in the emoji-completer are now colorful and the avatars in the username completer now match the avatars used in the timeline. It also open up a few other improvements, like finally sending mentions as links instead of plain text.
We have now CI running on our self hosted gitlab instance. We will probably switch of the travis CI, once this is more battle tested. Repositories are now also automatically mirrored between gitlab and github.
Fixed a bug, where we tried to read the internal world group on your server instead of keeping that to ourself.
I released 0.0.3 a few days ago bundling a few smaller improvements like redactions, copying messages and the ability to click on more links as well as some visual cleanups. Nothing radical, but also nothing breaking (I hope).
🔗Delight (Rich vdH, Michael (t3chguy), Valere, Steve, Nique, Nad)
Improving usability
Last week
Observed user tests of people trying to use Element for the first time for personal and professional use cases
This week:
Began work on fixing several of the issues observed, like:
adding an invite people button to new rooms, so users can more easily add people;
changing copy to help people understand what DMs are
Spaces
Communities are coming back with a bang! Last week we said we renamed them to Spaces, and this week, we’ve started designing what MSC1772 would look like for users on Element, to start user testing next week.
Social login
To make authentication easier, we’ve started initial implementations of SSO in Element, exploring how homeservers & Matrix clients can support multiple SSO providers. Most of the work so far is captured in MSC2858.
The release has been blocked because a bug has been found in the end to end encryption module. It has been fixed but we want to fix damages it created on one time keys before releasing the new app version.
The new background sync service mechanism PR has been updated
We started to integrate tuist to stop to be annoyed with merge conflicts on the Xcode project file
This week:
Release!
Merge the background sync service mechanism PR and make more people test it
We’ve just merged a lot of PRs, to improve room creation form and fix some bugs.
SDK side, Dominaezzz is converting some of the Service API methods to coroutines, for a cleaner code. See for instance https://github.com/vector-im/element-android/pull/2414 . 9 out of about 45 services have been migrated so far. We have about a 45 services in the SDK (!)
This week:
Release including a new way to invite friends to Matrix and to Element.
iinuwa created issues for all of the identity service api endpoints (w/ suggested module names!) 🎉
I updated lots of enumerated types to allow unspecced values, for future compatibility & robustness
Devin R made sure a custom Content-Type in responses overwrites the default rather than creating a duplicate header. #339
gnieto fixed a bunch of bugs in ruma-federation-api
Alejandro Domínguez added support for deserializing string power levels (requires the unstable-synapse-quirks feature because these events are invalid according to the spec but Synapse used to (?) accept them)
Just released version 2.2.0 of the Matrix Ruby SDK with the help of the community, with this release support for JRuby is improved - though still not perfect, in the higher-level abstraction direct messaging rooms are now exposed for all users - as well as all that the current user has, and the lower-level abstraction sees the addition of the CS API method to get an event context.
And just to reiterate; if you're using the gem - or have questions/comments about it. please do drop into the discussion room at #ruby-matrix-sdk:kittenface.studio.
!stats command which prints you the tracked durations based on a predefined or custom range of time
CSV Export based on a defined range
!break command. Putting a break in between a record or a in/out combination.
Better handling of multiple !in commands
Improved responses
Reminders to call !out and to take breaks
Fell free to join us at #timetracking-bot:famedly.de . As soon as the repo gets opened up to public we will announce it in TWIM. Feel free to give suggestions or wishes in our room :)
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
Reminder: On Monday, we will be announcing a denial of service vulnerability which affects Synapse versions prior to 1.20.0. If you have not upgraded recently, please do so.
For Synapse admins, this release support generating structured logs via the standard logging configuration (#8607, #8685). This may require changing your synapse configuration; see the upgrade notes for more information.
We've also added many new Admin APIs, contributed by @dklimpel:
Add API to get information about uploaded media (#8647)
Make it possible to delete files that were not used for a defined time (#8519)
Split API for reported events into detail and list endpoints. This is a breaking change to #8217 which was introduced in Synapse v1.21.0. Those who already use this API should check their scripts (#8539)
Allow server admins to list users' notification pushers (#8610, #8689)
Lastly, Synapse 1.23.0 addresses some significant bugs, including regressions in the SQLite-to-PostgreSQL database porting script (#8729, #8730, #8755) and an issue which could prevent Synapse from recovering after losing its connection to its database (#8726). Synapse will also reject ACL modifications from clients which would otherwise cause a server to ban itself from a room (#8708).
Installation instructions are available on GitHub, as is the v1.23.0release tag.
Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including @chagai95 and @dklimpel.
This release changes the way structured logging is configured. See the upgrade notes for details.
Note: We are aware of a trivially exploitable denial of service vulnerability in versions of Synapse prior to 1.20.0. Complete details will be disclosed on Monday, November 23rd. If you have not upgraded recently, please do so.
Add a push rule that highlights when a jitsi conference is created in a room. (#8286)
Add an admin api to delete a single file or files that were not used for a defined time from server. Contributed by @dklimpel. (#8519)
Split admin API for reported events (GET /_synapse/admin/v1/event_reports) into detail and list endpoints. This is a breaking change to #8217 which was introduced in Synapse v1.21.0. Those who already use this API should check their scripts. Contributed by @dklimpel. (#8539)
Support generating structured logs via the standard logging configuration. (#8607, #8685)
Add an admin API to allow server admins to list users' pushers. Contributed by @dklimpel. (#8610, #8689)
Add an admin API GET /_synapse/admin/v1/users/<user_id>/media to get information about uploaded media. Contributed by @dklimpel. (#8647)
Add an admin API for local user media statistics. Contributed by @dklimpel. (#8700)
Add displayname to Shared-Secret Registration for admins. (#8722)
Don't pull event from DB when handling replication traffic. (#8669)
Abstract some invite-related code in preparation for landing knocking. (#8671, #8688)
Clarify representation of events in logfiles. (#8679)
Don't require hiredis package to be installed to run unit tests. (#8680)
Fix typing info on cache call signature to accept on_invalidate. (#8684)
Fail tests if they do not await coroutines. (#8690)
Improve start time by adding an index to e2e_cross_signing_keys.stream_id. (#8694)
Re-organize the structured logging code to separate the TCP transport handling from the JSON formatting. (#8697)
Use Python 3.8 in Docker images by default. (#8698)
Remove the "draft" status of the Room Details Admin API. (#8702)
Improve the error returned when a non-string displayname or avatar_url is used when updating a user's profile. (#8705)
Block attempts by clients to send server ACLs, or redactions of server ACLs, that would result in the local server being blocked from the room. (#8708)
Add metrics the allow the local sysadmin to track 3PID /requestToken requests. (#8712)
Consolidate duplicated lists of purged tables that are checked in tests. (#8713)
Add some mdui:UIInfo element examples for saml2_config in the homeserver config. (#8718)
Improve the error message returned when a remote server incorrectly sets the Content-Type header in response to a JSON request. (#8719)
Speed up repeated state resolutions on the same room by caching event ID to auth event ID lookups. (#8752)
Heads up that we just cut another beta release of Dendrite - now at 0.3.0!
This is a really fun release given almost all the changes are contributed from
the wider community - so huge thanks to S7evinK,
MayeulC and felix!
The main new feature is full Read Receipt support thanks to S7evinK, which
makes an enormous perceptual improvement when using Dendrite - so especial
thanks are due there :)
So, if you're interested in helping us test, please spin up a copy from
https://github.com/matrix-org/dendrite and let us know how it goes - and if you're
already running one, now is an excellent time to upgrade!
Full changelog (including 0.2.1, which we forgot to blog about) follows:
Forward extremities are now calculated using only references from other extremities, rather than including outliers, which should fix cases where state can become corrupted (#1556)
Old state events will no longer be processed by the sync API as new, which should fix some cases where clients incorrectly believe they have joined or left rooms (#1548)
More SQLite database locking issues have been resolved in the latest events updater (#1554)
Internal HTTP API calls are now made using H2C (HTTP/2) in polylith mode, mitigating some potential head-of-line blocking issues (#1541)
Roomserver output events no longer incorrectly flag state rewrites (#1557)
This marks the start of Season 6 of Matrix Live (S06E01), a decision M+A apparently made on the fly just now. Incrementing a mostly-arbitrary counter... Imagine wielding that kind of raw power!
The IETF have been continuing to experiment with new chat systems, and #xmpp_bridgingmeta_jabber.ietf.org:matrix.org exists as a three-way bridge to discuss the initiative spanning Matrix, XMPP & Zulip. Really fun to see open chat converging like this :)
What type of consistency guarantees does it provide,
how many of the other homeservers you share a room with can be faulty or malicious without violating those guarantees (short answer: all of them!),
and how is it possible that a frayed-out event graph ever laces back to just a few current events,
while there is a probability larger than zero for growth to arbitrarily high limits?
While we are still in search of the right venue for peer-reviewed publication,
we went for a preprint this time to share our results with the scientific and the Matrix community in a timely manner.
In this figure, k is the number of participating servers, U_n is the current width of the event graph.
For example, with a current width of U_n = 1000 events and k = 200 servers,
we can expect that the width goes below E(U_{n+1}) < 600 events in just one step.*
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
In terms of Spec Core Team MSC focus for this week, MSC2744 has entered FCP. As such we're switching it out for MSC2844, which aims to define a global versioning system for all spec documents. Note that MSC2844 does not currently have FCP proposed. Its purpose for being in the focus this time around is for asking the team whether ideas proposed are plausible.
Thus this week's focus is MSC1544 (QR Code verification), MSC2790 (modal widgets) and MSC2844 (global versioning).
We've also started making use of a Github board to track the state of an MSC through the currently somewhat ambiguous process of putting up an MSC that is ready for review to starting FCP. You can find the board here: SCT Backlog.
Some explanation of the columns:
Awaiting SCT Input - These are MSCs that are deemed reviewable by the MSC author, and require review by a subsection of the Spec Core Team (usually one or two people) that are most familiar with the scope of Matrix that the MSC is relevant to.
Ready for FCP ticks - At least one individual from the Spec Core Team has signed off on the MSC. It is not time for others to give a cursory review - mainly sanity checking from the perspective of the part of Matrix that they work on. Note that fundamental flaws may still be found in this stage, requiring the MSC to be reworked.
In FCP - The MSC is in Final Comment Period, where everyone has a last chance to get their ideas in. This happens after most of the Spec Core Team have given their ticks from the previous column.
Done - These MSCs no longer need immediate attention by some portion of the Spec Core Team.
These definitions are all flexible as we continue to see what works for the limited resources we have available for review.
Do not be alarmed if a particular MSC is not on this board, we're placing things on here as deemed necessary, and more will certainly be added in time.
Working on the new spec platform: this week we've been simplifying the toolchain that builds and renders the spec, to make spec authoring easier and faster.
Note: We are aware of a trivially exploitable denial of service vulnerability in versions of Synapse prior to 1.20.0. Complete details will be disclosed on Monday, November 23rd. If you have not upgraded recently, please do so.
Happy RC day! The first release candidate of Synapse 1.23 is now available. For Synapse admins, we've added numerous new APIs thanks to contributions by @dklimpel, and we now support generating structured logs in a standard manner. The structured and drains logging options are now deprecated in Synapse's configuration.
Synapse 1.23.0rc1 also includes several significant bugfixes, especially around regressions in the SQLite-to-PostgreSQL database porting script, as well a fix to an issue that would prevent Synapse from recovering after losing its connection to the database. Synapse will also now reject modifications to access control lists which would otherwise result in a server blocking itself from a room. This is a particularly interesting case of unintended consequences in protocol design, and we hope to write more about it soon.
Next week should see the full release of 1.23, followed by a first implementation of MSC2403: Add "knock" feature landing in the development branch for 1.24. Look for more information in next week's edition of TWiM!
Neil is still working on P2P routing schemes and I am still working on Cerulean, but that doesn't mean nothing has happened this week! We've been busy reviewing PRs and have landed a big one this week:
Read receipts are now implemented (thanks @S7evinK!).
Usernames with = can now be created, in accordance with the spec.
Hello all. Hydrogen gained emotes (/me) this week, together with file uploads and downloads! Also, encryption keys are setup while typing instead of after sending, so you should see less momentary "the sender hasn't sent the keys yet". Also a good number of bugs were solved. Next up: image uploads!
As expected, /me commands brought a chorus of:
now that you support commands, you must add the single most important command in any decent Matrix client
Hello TWIM! From this week on we're changing how we're reporting updates on the Element clients. Each week, we'll include a section for cross-platform feature work the Element team is working on, and some sections for platform specific work too.
Spaces (previously known as Communities)
Communities are back! They've been renamed to Spaces, and on the Element team we've been busy exploring user experience and critical thinking for MSC1772, to allow users to organise rooms into useful groups.
Social Login
In an effort to reduce friction in authentication, we've been R&Ding how to allow homeservers & Matrix clients to support multiple SSO providers.. Most of the work so far is captured in MSC2858 and we're starting initial implementations in Element web next week.
VoIP: Voice & Video Calls
We've started foundational research, design and engineering to improve both the usability and stability of native matrix VoIP in Element. We've also started to look at how we can improve the continuity of the user experience between native Matrix calls and our current implementation of Jitsi.
Usability
We've been focusing on improving post-registration actions all round to make Element provide better prompts for first actions, and empty states for views on all platforms, with a multitude of fixes having landed on Element Web, iOS & Android in the last week. We've also started paid user testing to better understand specific pain points in the first time user experience, which we're looking to improve significantly in the coming weeks.
Web platform
v1.7.13 is now live! Which includes a bunch of fixes & improvements on VoIP.
Android platform
v1.0.10 is now available on the Play Store. We're also doing ongoing work to catch up with feature parity to Element Web, and Dominaezzz is migrating SDK APIs to suspend functions: no more MatrixCallback!
iOS platform
v1.1.0 is soon to be live on TestFlight for beta testers, and we'll soon be turning our attention to fixing end-to-end encryption regressions.
Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE (with the notable exception being device verification for now) and intends to be full featured and nice to look at
We've (finally) released a first version of matrix_app_service.ex, an Elixir library for the application service API (hex, GitLab)
uhoreg (known Elixir fan, asked):
that's cool. Do you have any projects that are using it?
Pierre:
I don't want to "sell" it before there's some kind of poc, but with a few friends we're trying to build a bridge for ActivityPub. we're using the library of the same name by the people behind CommonsPub, of which a lot of code comes from Pleroma, so the application service appears as an ActivityPub server. it's still the beginning, and I'm just starting to think about which information a bridge needs to keep (I guess sometime I'll ask Half-Shot a big list of questions 😁)
Today might be our highest-ever Pierre-ratio in TWIM!
A new release integrates an embedded h2 database into matrix-spring-boot-bot, which makes it a lot easier to write bots with persistency. You only need a few lines of code to write simple bots (e. g. a ping bot). I also changed a lot under the hood (e. g. using type-safe Matrix-Id-classes instead of Strings).
The matrix-sms-bridge now uses the h2 database from matrix-spring-boot-bot instead of neo4j, which makes it a lot more stable and fixes some bugs. We are using it in production with hundreds of target telephone numbers without problems. I also added an optional feature to use room aliases (e. g. #sms_4901234567:homeserver) to prevent multiple rooms with one target telephone number (which is still possible).
In a few weeks you can use this bridge with an old Android-Smartphone, which acts as an sms gateway with rest api.
I've written a small blog post about a Matrix chat bot I've developed with matrix-js-bot-sdk, it was started to address Mozilla specific features first, but then I've added a few other generally useful features too! Hope you'll like it and find it useful!
Hello Matrix friends. I've been asked to share this here. I'm currently building a platform for building communities. I'm focused on deeper matrix integration with communities, private messaging built on top of matrix, and custom domain support. Feel free to check out https://gurlic.com. The matrix community is at https://gurlic.com/matrix, but I haven't had the time to post much.
We have user pages (https://gurlic.com/bob -> https://bob.monster)
The UI is a little confusing now because we have longform articles (https://gurlic.com/homer/the-iliad-book-ii) and short social media posts (https://gurlic.com/privacy/post/324848317844422661).
I pretty much want all resources to have custom domain support, and matrix rooms for all communities, and nested communities. In the near future, I hope to have matrix servers for custom domains too. And of course, the source will also be open if folks want to host.
Right now, users on gurlic get a matrix account, and communities/galleries/publications get a room too, but i'm hoping to take some time to look at much deeper integration. Dendrite and hydrogen are great to build on top off...
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
Muhammad Radwan, on behalf of Tactical Tech, will present their work on The Glass Room Misinformation Edition.
What happens when we increasingly rely on social media and the web for nearly all our information? What information do we see, and what do we miss? How do we know if a picture or a tweet is genuine or truthful? And what can we do if we can’t be sure?
Popi from CHATONS will present the work done by CHATONS ("Collective of Alternative, Transparent, Open, Neutral and United Hosters")
CHATONS has been initiated by Framasoft, in the wake of the success of its campaign "De-google-ify Internet". A CHATONS member pledges to propose fully FLOSS-based online services, in an ethical and united spirit.
Hubert|Uhoreg Chathi will present on the state of encryption in Matrix, and more broadly
Why the PGP model is insufficient, what properties we want from encryption (e.g. why we want perfect forward secrecy), how the newer protocols (OTR, double-ratchet, megolm) give us some of the properties we want, and then what's coming in the future (MLS, and what that gives us)
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals
Happy November from the Synapse team! As mentioned last week, we pushed a small v1.22.1 release last Friday which fixed two regressions:
Fix a bug where an appservice may not be forwarded events for a room it was recently invited to. Broke in v1.22.0. (#8676)
Fix Object of type frozendict is not JSON serializable exceptions when using third-party event rules. Broke in v1.22.0. (#8678)
If you haven't upgraded your Synapse in a while, please do.
A major focus of Synapse is being able to meet the performance and reliability needs of massive homeservers like matrix.org. If you're curious about how Synapse's architecture has evolved over the years to meet these scaling challenges, check our our blog post from Tuesday: How we fixed Synapse's scalability!
Lastly, we anticipate releasing 1.23.0 in the next fortnight; keep your eyes peeled for release candidates and let us know if you have any feedback. For a preview of what's coming, check out GitHub for the new commits that have landed on the develop branch since our last release.
Synapse now horizontally scales across multiple python processes, as of 1.22: you can configure it so that events are no longer sent through the main proc, eliminating the single biggest bottleneck for large scale Synapse deployments. Read all about it at https://matrix.org/blog/2020/11/03/how-we-fixed-synapses-scalability
Exciting visual progress this week with actual bridging between Gitter and Matrix utilizing the virtualUser feature,
we've been iterating on the past couple weeks. Check out the image with all of the user avatars and display name goodness
to make both chats on Element and Gitter feel one in the same!
You can also check out the live demo in Matrix Live!
As mentioned in the sync on Monday, I was mostly distracted from Hydrogen this week. (sorry Bruno -BP) I did release the picture lightbox on Monday, and yesterday managed to close 4 bugs. There's also a community PR for better usability and accessibility in the login screen (keep those coming!) and after some work to make encryption more robust, I hope to do a release with all those goodies tonight.
This week, we came back to the background sync work to quickly display a notified message in the app. In parallel, we created a profiling tool at the SDK level to track performance like this one.
We are working through some low hanging fruit around post registration, blank screen interaction prompts and toast tweaks. We are also experimenting with SSO for matrix.org. Meaning that Element will give the user the option of either username/password or SSO. Finally, we are continuing our VoIP efforts and nailing down the designs, checkout Matrix Live for all the details. Next week we'll carry on with post registration UX, VoIP improvements.
Hi all! I think this is the right place to share this 🤞. I've been working on a Matrix SDK for Elixir over the last few months as a side project, it's very early days and it's currently just a bare-bones wrapper for the Client-Server API. I've also written the Elixir/Erlang bindings for Olm (currently missing group sessions) which should be added to the SDK soon. The repo can be found here: https://github.com/niklaslong/matrix-elixir-sdk and the Elixir bindings for Olm are linked in the readme.
It is the right place! Thanks cognitive_tea :D
Asked if there were big plans for use of the project
Not as yet, though a few people have reached out to me already and are building on top of it. I started it as a way to get going with Matrix dev and as a fun side project. That being said, I think providing the tools to Matrix-enable Elixir apps might lead to some interesting things.
If anyone has any precise ideas on how they would want to integrate their Elixir apps with Matrix, I'd be super happy to have a chat 👍️
Less precise ideas are also welcome, of course 🙂
Hemppa the bot is a generic bot for writing modules as easily as possible in Python. Thanks to issues with Freenode IRC bridge Hemppa got a new module for basic relaybot bridging of any Matrix rooms. Relaybots are stupid, but sometimes there's no working alternative. https://github.com/vranki/hemppa#relay-bridge
I did a talk at Arch Conf 2020 last month, on a generic introduction of Matrix and how to install a Matrix homeserver on Arch Linux. The recording has just been uploaded; it can be found on CCC's media site as well as YouTube 🙂
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
