One of the goals on the core team is to reduce friction for new users joining Matrix. A challenge we regularly face is all the factors which make Matrix flexible and powerful as an open, secure decentralised protocol also increase the difficulty of getting started.
As one example— to create a healthy, vibrant ecosystem and put power back into the hands of end users, it’s critical multiple clients exist, and that users ultimately have control over which one to use. However, needing to choose a client before getting going is counter-intuitive, and adds cognitive load which proprietary services simply don’t have.
Striking this balance is tricky, and one we’re aiming to improve with the latest version of Matrix.to.
Matrix.to is a simple URL redirection service which lets users share links to rooms, users and communities. It isn’t tied to any client, and for end users it serves as a landing page for many as their first touch to Matrix. And until the matrix:// URI scheme is deployed commonplace (we finally registered it with IANA a few months ago!) it is the only client-agnostic way to link to content in Matrix.
It’s privacy preserving, with no information stored server side, with all calculation done in client-side JavaScript. It started life years ago as a temporary minimum viable hack by Matthew which then hung around untouched for years being… minimally viable. But we’ve recently given it a huge upgrade in usability and functionality at last!
We’ve re-written matrix.to from scratch, giving it a visual upgrade and refocused the experience around:
Helping new users find the best client for them to easily get going. By default, links will prioritise showing clients which are compatible with the platform the link is being viewed from, including mobile platforms.
Optionally, remembering your preferred client for future visits. This also includes deeplinking into native apps for Element Desktop & Element Mobile (and in future, other clients too, of course).
Fetching useful previews. One of our main observations when testing with new users is a lack of confidence trying out a new service without personalised, contextual information on the rooms or people they’re heading to. Matrix.to now displays room metadata like avatars, names and topics directly (fetched via matrix.org by default, asking for permission).
We’ve baked in the ability to specify clients and deployments within links, allowing sharing to give the option of a specific destination to guarantee a predictable experience. For instance, Mozilla might share matrix.to links which recommend using chat.mozilla.org, if you don’t already have a preferred Matrix client configured. We’ve yet to implement this feature in Element, but we’ll be researching and experimenting with different implementations soon and will recommend best practises when we have them.
We plan to evolve matrix.to over time, including eventually evolving it to better support the Matrix URI scheme. As before, you can find the source code on GitHub and please go ahead and submit pull requests to get your Matrix client listed.
It's been a year since Dendrite development picked up again and it's certainly
been a busy one at that! We started off 2020 by sprinting to complete the FOSDEM P2P
demo and, since then, we have continued to develop Dendrite into a more featureful
and stable homeserver.
In October, we moved Dendrite into beta,
and have since released a number of releases. We've also seen quite a lot of interest
from the community, so I'm here today to write about some of the fun things that have
been going on in Dendrite-land.
I'm happy to announce that we've finally deployed our own public Dendrite instance at
dendrite.matrix.org! It's running the latest Dendrite code and is open for
registration, so if you have been looking for an opportunity to play with Dendrite
without hosting your own deployment, here's your chance!
There are still bugs and missing features, but overall the server is quite usable, so
please feel free to register and try it and let us know how you get on.
This is the first deployment that we've built for any kind of scale, so we are cautious
of the fact that there may be performance bottlenecks still. That said, over the last
few weeks, a number of performance-improving changes have been merged, including:
Around 20x performance improvement on the State Resolution v2 algorithm, which is used
in room versions 2 and above
Significantly reducing the amount of time spent recalculating event reference hashes
and event IDs in the roomserver and sync API, reducing CPU usage
Optimised memory usage and reduced database hits in the federation sender, which helps
particularly in large rooms with lots of resident servers
We're optimistic that running this deployment will help us to identify scaling pain
points and to make Dendrite leaner in the long run. Feel free to sign up and give it
a spin! :-)
Since the beginning of the year, we've added a number of new features, including but
not limited to:
Room versions support for all currently specced versions (v1-v6), including support
for State Resolution v2
SQLite storage support in addition to PostgreSQL, largely useful for the P2P demos
Support for database schema upgrades, making updating Dendrite significantly easier
Early end-to-end encryption support, including device list syncing and send-to-device
messages, although with key backup and cross-signing still to come
A number of federation features, including invites, retries and backing off unreachable
homeservers
User-interactive authentication (UIA), which is used in password changes and deleting
devices from your device list
Support for local pagination, backfilling over federation and fetching missing events
Redaction of events both locally and over federation
Entirely new microservices for managing server signing keys, E2E keys, user and device
management
Lots of great contributions from the community - including all of Read Receipts (thanks to S7evinK) and Read Markers (thanks to Lesterpig)!
Of course, Dendrite also needs to be able to fulfill the promise of being a usable
next-generation Matrix homeserver at the same time as being a sci-fi development
platform. We have spent much of the last year working specifically on this. Today,
Dendrite's Sytest compliance sits at:
59% compliance for Client-Server APIs, up from 33% in May 2020
83% compliance for Server-Server APIs, up from 26% in May 2020
As you can see, these are significant leaps in the numbers of tests passing against
Dendrite.
We have been increasingly trying to use Dendrite for the development and testing of
some new Matrix feature proposals. Recently we've seen early support added for
Peeking (MSC2753) and there
is work in progress on Peeking over Federation (MSC2444).
Peeking enables temporarily subscribing to a room for real-time events without joining
the room. This will only be possible with rooms that are world-readable, but it reduces
the overhead of looking into a room significantly as there is no need to update the room
state for each peeking user/device.
In addition to that, we've also been working on Threading (MSC2836)
support, which is the gateway to building some pretty new and interesting Matrix
experiences. Twitter-like or Reddit-like social prototypes like this have traditionally
been difficult to build on top of Matrix as the m.reference relation type from MSC1849
had never really been fleshed out.
Threading adds m.relationship fields for embedding these relationships, and also
specifies an additional /event_relationships API endpoint for finding other events
related to a given event in either direction. This makes it possible to build threads.
Dendrite has also been our primary development platform for P2P Matrix. This year we
have released multiple P2P Matrix demos, including:
p2p.riot.im, which uses libp2p rendezvous and embeds a full
Dendrite homeserver into an in-browser Service Worker
Element iOS P2P, available on TestFlight,
which embeds a full Dendrite homeserver into the Element iOS app, initially using
QUIC over Yggdrasil as a transport for federation
traffic, but with more recent versions using QUIC over the experimental Pinecone protocol
dendrite-demo-libp2p,
a standalone binary which formed the basis of the FOSDEM 2020 P2P demo, using libp2p and
local multicast discovery as a transport for federation traffic
dendrite-demo-yggdrasil,
another standalone binary like above, but using QUIC over Yggdrasil connectivity instead
of libp2p as the transport for federation traffic
Each experiment teaches us more about potential issues that need to be resolved in order
to bring P2P Matrix closer to being reality, and we are continuing to use Dendrite for
this work. We'll be announcing more information in the New Year about our latest efforts
and the Pinecone routing scheme that we are developing.
It's also worth highlighting that all of the other experimental work taking place right
now, including Threading and Peeking, also work over P2P!
We'll be taking a short break for Christmas, but will then be continuing work on
Dendrite in 2021, with the main aims being to add new features, improve spec compliance
further, fix bugs and eventually exit beta. We'll also be continuing further experimental work in the
P2P and Threading areas, as well as supporting the development of new MSCs such as
Portable Identities (MSC2787).
We'd like to say thank you for the community support and interest, and also to send
out a special thanks to our community contributors who have contributed a number of
fixes and features in recent months! We always welcome code contributions via
GitHub if you are an interested developer.
Lots of things to talk about! Firstly I'm very happy to say that spaces are happening, you can keep track of them here: https://github.com/vector-im/element-web/issues/15930.
Next Open Tech Will Save Us, was great this month, and you can also get a great introduction to Matrix Spaces there too.
Then, Marcus Schopen shared his love for Matrix by showing us the Matrix-branded mugs he has had printed:
If you'd like to buy some OFFICIAL Matrix merch (we don't have mugs just yet), then please go to The Matrix Shop! Marcus also added:
New spec platform: we've asked the spec core team to spend some proper time with the new spec so we can decide if it's ready to ship.
You can also file bugs at https://github.com/matrix-org/matrix-doc/labels/spec-redesign
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
In terms of Spec Core Team MSC focus for this week, we've continued to be quite busy with implementation during holiday crunch time. There were some constructive ideas for the team to carve out more time for spec work during this week's Spec Core Team retro, which will hopefully lead to an improved cadence this week and into the new year.
As such, the focus remains at MSC2844 (global versioning), MSC2366 (ready and done in key verification), and MSC2790 (modal widgets).
Hello, friends! On Wednesday we released Synapse 1.24.0 and 1.23.1, which include security fixes. Please upgrade! (Also, if you've installed Synapse from source, be sure to pip install 'cryptography>=3.3' in your virtualenv to address the recent OpenSSL CVE).
This is our last full release for the year, as we don't want to do anything that might risk the stability Santa's federated command and control centre at the North Pole, but we're looking forward to lots of new features and improvements in the new year — more details on that next week.
We'd also like to remind you that we're planning on removing support for Python 3.5 early next year (a formal announcement and timeline will accompany the 1.25.0 release), and we'd appreciate and comments or concerns in this GitHub issue.
Strikes me that Santa has a more centralised operation, but who can say?
Just updated my Synapse K8s-optimized image and Chart to 1.24.0, planning on deprecating the image at some point soon though, instead pointing people to upstream - or platform-optimized - images as I should be able to move all the Kubernetes-specific things into the chart itself to make it more compatible.
The image in question started off as a plain docker image, before there were official ones. I've since rebased it onto the official Matrix image with just the modified scripts and volume points.
My plan is to move all the Kubernetes-specific scripts into configmaps (which are a K8s way of storing text files/environment variables/etc), and then build my own launch arguments through the chart to get it to run on any image that has a Python interpreter and the Synapse modules available in the Python search path.
With all that done, it means I won't be a bottleneck in pushing updated image tags for people to be able to update, and I won't have to finagle multi-arch images 😃
As the PR from last TWIM is still awaiting review, I have once again updated the Docker ARM images for Dendrite for version 0.3.3. As always, they can be found in my docker repo at https://hub.docker.com/u/trslimey
If you didn't catch it already, we did a full announcement this week for Gitter now speaking Matrix 🎉! If you're curious about how we made it and some more technical details, the blog post is a great read: https://matrix.org/blog/2020/12/07/gitter-now-speaks-matrix
In terms of bridge progress, we added support for m.emote status messages which means your /me says hi messages will bridge back and forth properly. On the Gitter side, we fixed the browser notifications so they show properly from the virtualUser instead of the generic matrixbot.
I made an Instagram DM bridge: https://github.com/tulir/mautrix-instagram / #instagram:maunium.net
It's a bit barebones still, but text bridging and backfilling works. The main advantage over mx-puppet-instagram is that it uses the realtime MQTT API instead of polling.
There was a second update from tulir, he added more features as of today:
After the initial announcement on Monday, I added bridging of reactions, redactions and photos in both directions and also typing notifications, read receipts and all types of messages from Instagram to Matrix.
We’re making more progress on implementing Spaces.. On Web, we’ve made progress on creating spaces and switching between them. On mobile, we’ve laid down the foundations in the Android SDK. If you missed it, we gave a preview in this weeks Open Tech Will Save Us.
Social login
We’ve been working on server-side support in synapse for picking an MXID during login with SSO, and also implementing support in iOS to gain parity with Android & Web, to launch together in future.
VoIP
We’ve been working on implementing hold & resume (line 1 / 2) support on Web, iOS & Android. Web has something sneak previewable on develop.element.io!
The new background service has been merged. Messages from notifications will be displayed faster within the app. A TestFlight build for testing will be available tomorrow.
This is a first update regarding a very small project named "axon", aimed at operating the synapse admin API from Python, either as a library or a CLI tool. It is currently in use everyday on our synapse instance for managing users and looking for suspicious rooms, and is being integrated into a prototype Web interface for managing synapse.
Admin API coverage is currently about 50%, mostly focusing on users and rooms management (get, list, update, delete). It is still enough to delete a user, edit an offensive avatar, or run as a cron to periodically purge remote media.
Our main repository: https://forge.tedomum.net/tedomum/axon
Github project mirror for contributing: https://github.com/kaiyou/axon
I asked if there was a GUI:
Not currently, but we are integrating it with another of our tools named "hiboo", which is a general indie hoster user manager (provides SAML & OIDC authentication with config templates for many projects including synapse, and integrates the account workflows and moderation features). Currently it provides room navigation and purging, plus deleting accounts either on moderator decision or when the user deletes her SSO account.
Sounds promising! Synapse Admin would benefit from this kind of tooling...
... so it was a surprise to find another project designed to help in this area!
Well, maybe it's about time for "synadm - a CLI frontend to Matrix-Synapse admin APIs" to be announced more officially. As the name implies it's a CLI tool that eases the usage of admin API commands for Synapse admins. It currently covers everything the room and user API's can do: https://github.com/JOJ0/synadm#implementation-status--commands-list
It can be configured interactively and/or using a config file in yaml format: https://github.com/JOJ0/synadm#configuration
It's configurable whether the responds of the API are shown in pretty printed JSON or in a table format (using Python tabulate).
It's well documented via it's online help and supposed to be quite self-explanatory to the admin.
Any thoughts, ideas, feature-requests are very welcome in #synadm:peek-a-boo.at or as usual by filing PR's or issues in the project repo https://github.com/JOJ0/synadm.
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
This release fixes a denial of service vulnerability (GHSA-hxmp-pqch-c8mm / CVE-2020-26257) in which a malicious homeserver could send malformed events into a room which would then break federation of that room.
This follows the disclosure of a denial of service vulnerability in OpenSSL (CVE-2020-1971). If you have installed Synapse from source, please ensure your host is up to date and then execute pip install 'cryptography>=3.3' inside your Synapse virtualenv.
We've also released Synapse 1.23.1 which includes that security fix and a small patch to maintain Python 3.5 compatibility. It is otherwise identical to 1.23.0. Note that Synapse 1.24.0 includes backwards incompatible changes which may affect a small number of users. See the notes on upgrading for more information.
Synapse 1.24.0 brings a pair of new Admin APIs, including a way to log in as users and to forcibly purge rooms when deleting them. We've also made numerous bug fixes and improvements to SSO support, especially around OpenID Connect and SAML providers.
This release includes an optional change to push notification badges: currently, the number in the badge is based on the count of rooms with unread messages. However, in some specialized cases you may want the badge to show the count of all unread messages, even if there are multiple unread messages in the same room. This behavior can now be toggled with a new configuration setting.
Additionally, for server admins, the deprecated /_matrix/client/*/admin Admin API endpoints have been removed. If you have tools which target these endpoints, please update them to use the /_synapse/admin URL prefix instead.
It’s been just over 2 months since we revealed that Gitter was going to join Matrix - and
we are incredibly proud to announce that Gitter has officially turned on true native Matrix connectivity:
all public Gitter rooms are now available natively via Matrix, and all Gitter users now natively exist on Matrix.
So, if you wanted to join the official Node.js language support room at https://gitter.im/nodejs/node
from Matrix, just head over to #nodejs_node:gitter.im and *boom*, you’re in!
This means Gitter is now running a Matrix homeserver at gitter.im which exposes all the active public rooms - so if you go to the the room directory in Element (for instance) and select gitter.im as a homeserver, you can jump straight in:
Once you’re in, you can chat back and forth transparently between users on the Gitter side and the Matrix side, and you no longer have the ugly “Matrixbot” user faking the messages back and forth - these are ‘real’ users talking directly to one another, and every public msg in every public room is now automatically exposed into Matrix.
So, suddenly all the developer communities previously living only in Gitter (Scala, Node, Webpack, Angular, Rails and thousands of others) are now available to anyone anywhere on Matrix - alongside communities bridged from Freenode and Slack; the native Matrix communities for Mozilla, KDE, GNOME communities etc. We’re hopeful that glueing everything together via Matrix will usher in a new age of open and defragmented dev collaboration, a bit like we used to have on IRC, back in the day.
This is also great news for mobile Gitter users - as the original mobile Gitter clients have been in a holding pattern for over a year, and native Matrix support for Gitter means they are now officially deprecated in favour of Element (or indeed any other mobile Matrix client).
Now, this is the first cut of native Matrix support in Gitter: much of the time since Gitter joined Element has been spent migrating stuff over from Gitlab to Element, and it’s only really been a month of work so far in hooking up Matrix. As a result: all the important features work, but there’s also stuff that’s yet to land:
Features ready today:
Ability to join rooms from Matrix via #org_repo:gitter.im
Bridging Edits, Replies (mapped to Threads on Gitter), Deletes, File transfer
Bridging Markdown & Emoji
What remains:
Ability to send/receive Direct Messages
Ability to plumb existing Matrix rooms into Gitter natively
Synchronising the full Gitter membership list to Matrix. Currently the membership syncs incrementally as people speak
Turning off the old Gitter bridge
Bridging emotes (/me support) (almost landed!)
Bridging read receipts
Synchronising room avatars
Bridge LaTeX
Stuff we’re not planning to support:
Ability to join arbitrary rooms on Matrix from Gitter. This could consume huge resources on Gitter, and we’re not in a rush to mirror all of Matrix into Gitter. This will get addressed when Gitter merges with Element into a pure Matrix client.
Bridging Reactions. Gitter doesn’t have these natively today, and rather than adding them to Gitter, we’d rather work on merging Gitter & Element together.
For more details, we strongly recommend checking out the native Matrix epic on Gitlab for the unvarnished truth straight from the coal-face!
🔗How do you make an existing chat system talk Matrix?
In terms of the work which has gone into this - Gitter has been an excellent case study of how you can easily plug an existing large established chat system into Matrix.
At high level, the core work needed was as simple as:
This can be accomplished by simply adding a virtualUser property to your chat message/post/tweet schema which holds the mxid, displayName, and avatar as an alternative to your author field. Then display the virtualUser whenever available over the author.
This "application service" comes pre-packaged for you in many cases, so for example you can simply drop in a library like matrix-appservice-bridge in a Node.js application, and all of the Matrix talking complexity is handled for you.
Polish it!
In practice, Eric (lead Gitter dev) laid out the waypoints of the full journey:
First big step was to add the concept of virtual users to Gitter. We could also have created a new Gitter user for every new matrix ID that appears, but tagging them as virtual users is a bit cleaner.
Figuring out how to balance the Matrix traffic coming into/out of Gitter.
Spreading the inbound load comes for free via our existing load-balancer setup (ELB) where we already have 8 webapp servers running the various services of gitter.im. We just run the Matrix bridge on those servers alongside each web and api process, and then the load-balancer’s matrix.gitter.im spreads out to the servers.
Events from Matrix then hit the load balancer and reach one of the servers (no duplication when processing events).
If something on Gitter happens, the action occurs on one server and we just propagate it over to Matrix (no duplication or locking needed).
We have realtime websockets and Faye subscriptions already in the app which are backed by Mongoose database hooks whenever something changes. We just tapped into the same thing to be able to bridge across new information to Matrix as we receive it on Gitter.
Hooking up the official Matrix bridging matrix-appservice-bridge library to use Gitter’s existing MongoDB for storage instead of nedb.
Figuring out how to namespace the mxids of the gitter users:
It’s nice to have the mxid as human readable as possible instead of just the numerical userId in your service.
But if people can change their username in your service, you can’t change your mxid on Matrix. In the future, we’ll have portable accounts in Matrix to support this (MSC2787) but sadly these are still vapourware at this point.
If you naively just switch the user’s mxid when they rename their username, then you could end up leaking conversation history between mxids(!)
So we went with @username-userid:gitter.im for the Matrix ID to make it a bit more human readable but also unique so any renames can happen without affecting anything.
For room aliases, we decided to change our community/room URI syntax to underscores for the room aliases, #community_room:gitter.im
Figuring out how to bridge features correctly;
Emoji - mapping between :shortcode: and unicode emojis
Mapping between Gitter threaded conversations <-> Matrix replies
Mapping between Matrix mentions and Gitter mentions
Keeping users and room data in sync
We haven’t gotten there yet, but the data comes through the same Mongoose hook and we can update the bridged data as they change on the Gitter end.
Meanwhile, the Matrix side of gitter.im is hosted by Element Matrix Services and is a plain old Synapse, talking through to Gitter via the Application Service API. An alternative architecture would be to have got Gitter directly federating with Matrix by embedding a “homeserver library” into it (e.g. embedding Dendrite). However, given Dendrite is still beta and assumes it is storing its data itself (rather than persisting in an existing backend such as Gitter’s mongodb), we went for the simpler option to start with.
It’s been really interesting to see how this has played out week by week in the Gitter updates in This Week in Matrix: you can literally track the progress and see how the integration came to life between Oct 9, Oct 23, Nov 6, Nov 27 and finally Dec 4.
Huge thanks go to Eric Eastwood, the lead dev of Gitter and mastermind behind the project - and also to Half-Shot and Christian who’ve been providing all the support and review from the Matrix bridging team.
First and foremost we’re going to be working through the “What remains” section of the list above: killing off the old bridge, sorting out plumbed rooms, hooking up DMs, importing old Gitter history into Matrix, etc. This should then give us an exceptionally low impedance link between Gitter & Matrix.
In the medium/long term, it’s simply not going to be efficient for the combined Element/Gitter team to split our efforts maintaining two high-profile Matrix clients. Our plan is instead to merge Gitter’s features into Element (or next generations of Element) itself and then - if and only if Element has achieved parity with Gitter - we expect to upgrade the deployment on gitter.im to a Gitter-customised version of Element. The inevitable side-effect is that we’ll be adding new features to Element rather than Gitter going forwards.
Now, that means implementing some features in Matrix/Element to match...
Instant live room peeking (less than a second to load the webapp into a live-view of a massive room with 20K users!!)
Seamless onboarding thanks to using GitLab & GitHub for accounts
Curated hierarchical room directory
Magical creation of rooms on demand for every GitLab and GitHub project ever
GitLab/GitHub activity as a first-class citizen in a room’s side-panel
Excellent search-engine-friendly static content and archives
KaTeX support for Maths communities
Threads!
...and this work is in full swing:
We have a proposal for fast peeking (via lazy-loading state over federation) at MSC2775 and the new peek APIs at MSC2753 and MSC2444 (and even implemented by Dendrite)
The only bits which aren’t already progressing yet are tighter GL/GH integration, and better search engine optimised static archives.
So, the plan is to get cracking on the rest of the feature parity, then merge Gitter & Element together - and meanwhile continue getting the rest of the world into Matrix :)
We live in exciting times: open standards-based interoperable communication is on the rise again, and we hope Gitter’s new life in Matrix is the beginning of a new age of cross-project developer collaboration, at last escaping the fragmentation we’ve suffered over the last few years.
Finally, please do give feedback via Gitter or Matrix (or mail!) on the integration and where you’d like to see it go next!
Taking place NEXT WEDNESDAY, the ninth edition of my favourite monthly Open Tech-themed live video broadcast! Subscribe to the calendar on https://matrix.org/open-tech-will-save-us/ and check out the lineup:
Angie Gaudion, Coordinatrice at CHATONS, presenting Mobilizon: a new libre events management tool https://joinmobilizon.org/en/.
Ben Francis, Founder at Krellian, presenting the relaunch of WebThings (previously from Mozilla)
Nad Chishtie, Product Lead at Element, presents Matrix Spaces: a new vision for room groups in Matrix
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
In terms of Spec Core Team MSC focus for this week, we've mostly been focusing on implementation. Thus the same 3 MSCs are in focus: MSC2844 (global versioning), MSC2366 (ready and done in key verification), and MSC2790 (modal widgets).
This week we released version 0.3.2, which contains some bug fixes. Changes this week include:
Checking if a server is allowed to see an event has been optimised somewhat
The ability to disable federation has been added
Application services are now sent correctly formatted client events
create-account has been simplified so that you can just point it to your Dendrite config
/unpeek support has been added (for MSC2753)
Event IDs are now calculated far less often in the roomserver, since this is an expensive operation
The federation sender now caches events in memory to reduce load on the database
The "since" parameter is now checked more thoroughly in the sync API, which led to a bug that could cause forgotten rooms to reappear (thanks to kaniini)
The polylith now proxies signing key requests through the federation sender correctly
⚠️ The OpenSSL project has pre-announced a high severity issue that will be disclosed on Tuesday. We will be regenerating Docker images and releasing Synapse 1.24.0 and 1.23.1 on Tuesday following that announcement. Note that these releases contain security fixes of their own, and you are encouraged to upgrade.
In other news, Synapse 1.24.0rc2 is out and includes numerous bug fixes and internal improvements, especially around our OpenID Connect (OIDC) and SAML providers. Of note, the OIDC provider can now re-try generating MXIDs when there's a collision, reaching parity with the SAML provider.
We've been talking about MSC2403: Add "knock" feature for a few weeks, but the end is in sight: the PR is reviewed and the tests pass, so we're on track for including it in 1.25.0! We're also on the cusp of a finishing a new algorithm for calculating the auth chain difference in State Resolution v2, which should significantly reduce worst case performance.
Lastly, the deprecated /_matrix/client/*/admin Admin API endpoints have been removed. If you have tools which target these endpoints, please update them to use the /_synapse/admin URL prefix instead.
The Docker ARM images for Dendrite have been updated for version 0.3.2. As always, they can be found in my docker repo at https://hub.docker.com/u/trslimey
We've soft-launched the new bridge to all public Gitter rooms! You can now find any public room on Gitter checking out the room directory for gitter.im or directly by replacing the forward slash in the URI part of the Gitter room with an underscore and joining #*:gitter.im(replace the *). If you come across any bugs, feel free check the list of known problems and create an issue. There will be a full announcement next week so stay tuned!
We've also added a few more features this week and now transform :emoji: syntax from Gitter to Unicode so they're compatible in Matrix, propagate the room topic/description, and some bug fixes around threaded conversations and replies. You can track our full progress from the GitLab epic for the native Gitter-Matrix bridge.
Massive milestone! Great work Eric, it's been such fast progress!
This was actually just the beginning, the ground work needed to lay the foundations for A Better Future. We already have a bunch of other merge requests to continue down that path. Hopefully there shouldn’t be such large ones in the (near) future. So what’s next? The work that landed this week was a switch to the SDK for almost all endpoint queries. Now need to handle the syncs with it. Later on, we will use it to handle storage, and then enable E2EE as well, but that will take quite some time.
Big progress from Fractal here! Will be good to hear more from the Fractal club to see what the ETA is for storage and E2EE.
Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE (with the notable exception being device verification for now) and intends to be full featured and nice to look at
While I've actually not had much time to work on Nheko, I did at least fix most of the UTD (Unable to decrypt) issues, when sending encrypted messages. This has been a long standing annoyance, but it should now finally not be an issue anymore (well... at least it should not be caused by Nheko as often anymore)! With this there are only a few tickets left to finally bring E2EE out of beta in Nheko (SSSS and bootstrapping cross-signing, as well as a general review for potential issues).
And while I have actually not been doing much work, we actually had more Pull Requests (and merged them too) this week than I have ever seen before in Nheko! Thank you very much to everyone who contributed. A short summary:
Nheko now uses the Nheko icon from your system theme, if available
You can finally look at avatars in their full resolution!
Megolm sessions are now lazily loaded from disk.
The registration page now also has nice inline error messages, hinting at why a specific input is unacceptable.
Updates to the russian translation
The README now lists all required GStreamer plugins for voice and video calls.
It feels super great, when you have so many people contributing, I love it! <3
Nheko really seems to be doing something right when it comes to attracting PRs!
We now have a public roadmap! We’ll be covering more details on it in this week’s Matrix Live, as well as a blog post on element.io soon.
Spaces
Work has started in earnest on the web and android implementations.
On the web side T3chguy completed the Spaces creation flows and is now looking at the Space landing page and Space post-creation prompts.
We have noticed that prefixing any word with ‘Space’ makes it sounds 20% cooler.
The mobile MVP will be read only and Valere will start with the ability to respond to invites and select Spaces.
So far it is looking very promising with lots of feedback from user testing informing the design.
Social login
We plan to ship this as a cross client feature this month. This week we completed the implementations for web and android with iOS and Synapse ongoing.
VoIP
Line 1 / 2 support is almost ready on web: ironing out some bugs and cosmetic issues, then PR will be ready for review. VoIP V1 support and hold/resume is getting close on Android & iOS, along with stability fixes.
We don't have much information about this one yet, other than that cryptovoxels have been interested in developing a Matrix client for their world for a while.
Hemppa the Bot is a multipurpose bot for writing modules super easily in Python. Last week Hemppa received a useful feature of printing files! You can set up a room for a specific printer and invite anyone you wish to it to allow printing. Any attachments uploaded to the room will be automatically printed. It's actually one of the easiest way to print from a phone and very useful if you have a printer that does not have drivers for modern Windows or MacOS versions. https://github.com/vranki/hemppa#printing
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
In terms of Spec Core Team MSC focus for this week, we're switching from QR Code verification to key verification additions. Otherwise things remain the same: MSC2844 (global versioning), MSC2366 (ready and done in key verification) and MSC2790 (modal widgets).
On behalf of the Americans on the Synapse team: Happy Thanksgiving! This past week saw us land some improvements to the OpenID Connect SSO provider, bringing it closer to parity with the SAML provider. In particular, #8801 made it possible to gracefully handle username collisions when creating a new account over SSO.
We've also merged a pull request (#8785) from @dklimpel which removes deprecated paths for the Synapse Admin APIs. As of the next Synapse release, all Admin APIs will be accessibly only under the /_synapse/admin/v1 path prefix.
Otherwise, much of our time has been spent on getting pull requests like #6739, implementing MSC2403: Add "knock" feature, ready to merge. It's coming to a Synapse release soon!
Speaking of releases, we're looking forward to cutting a first release candidate of 1.24.0 early next week; keep your eyes peeled for that!
I'd be remiss in not mentioning that we started the week by disclosing CVE-2020-26890, which is a denial of service affecting Synapse versions prior to 1.20.0. If you've not upgraded, please do.
Lots of features added this week including Gitter threaded conversations and Matrix reply bridging, image/file uploads, transforming mentions so notifications flow, and better looking flairs!
We'll be working through some of the bridge oddities and hoping to open up the bridge to all public rooms on Gitter soon. You can track our full progress from the GitLab epic for the native Gitter-Matrix bridge.
The MX-Puppet-VK bridge for VK (Vkontakte) social network via this commit have added initial support for User tokens. This means that we now can use it for puppeting the real user account, to automatically bridge all "VK Messenger" private dialogs and groups into Matrix. Previously bridge worked only with VK community tokens as bot, to bridge only conversation with specific communities.
The matrix-sms-bridge can finally be used with an Android smartphone as SMS-Gateway instead of gammu and a modem. @rebekka:imbitbu.de developed the app android-sms-gateway-server, that the bridge uses to send and receive sms messages. The new bridge version is used in combination with the app in production (medical care centre) for a week now. We have a throughput of 50-100 SMS message a day. The update is also integrated in matrix-docker-ansible-deploy.
Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE (with the notable exception being device verification for now) and intends to be full featured and nice to look at
The new completer is now on master. Please try it out and report any issues you may find. It now actually inserts a user link, keeps a per room message history and should look quite a bit nicer and be faster, when completing emojis or user names.
There were also quite a few cleanups, most of them done by Kirillpt! Some spacing here and there, saturation changes, the login page should now be less confusing, when you don't enter a valid mxid and toggles should now not get stuck in confusing intermediate states. There is also a russian translation now and some VOIP fixes. Pretty busy week, but I like it!
FluffyChat 0.23.0 is out now with password recovery options and a lot of minor fixes and UX improvements. Now also available on Flathub!! 🙂 Learn more at: https://fluffychat.im
On mobile, we’re busy adding better suggestions to empty/starting views of screens, improving searching for contacts, improving adding other users by QR code scanning, and improving deep link integrations on Android. Expect releases for iOS & Android soon!
Spaces
Spaces UI designs & user research testing are underway, with initial development for the Element Web client kicked off too. We've been doing research & testing in private to avoid advance bias from the people we're testing with, but hope to share things publicly soon.
Social login
Social login is in review for Element Web, with synapse & mobile support hot on the tails for next week prior to releasing.
VoIP
We’ve been focusing on implementing UI for call holding, and general UI polish across the VoIP experience, to be released soon!
Element Web 1.7.14 was released this week, which includes:
Several tweaks and improvements to the room list filter
Improved registration based on user feedback
Improved invite / create DM flow.
Future releases will no longer publish element-web to NPM, and the package is marked as deprecated
Renamed build artifacts from riot-.tar.gz to element-.tar.gz
Experimental LaTeX maths rendering support is available via a labs flag on develop. Thanks to akissinger, thosgood, uhoreg, and others who have worked on various pieces of this puzzle!
Element iOS
Element iOS 1.1.1 has been released:
It contains contains several major bug fixes, including end-to-end encryption bug fixes.
Full history available on GitHub viewing the 1.1.1 & 1.1.0 tags.
Element Android
The Element Android 1.0.11 release is pending:
We're working through final bug fixes.
Separately, we’re also working on adding missing features to the client like; managing room aliases, room history visibility, join rules, etc.
In the SDK, the migration of the services API from MatrixCallback to coroutines is going well, tracked here.
In addition to boring stuff like being able to persist events (hopefully) right, so we don't have to initial-sync every launch, I've been making a start towards multi-account support!
You can sign in as multiple users and switch between the "active" user you want to send messages as - it's a bit flaky for now, esp. when scrolling up to read past messages, but hey it's a start I guess
Asked if they were using an SDK:
Preeetty much doing it from scratch, though I am loosely basing the matrix-api side of it on Ruma
We gained a new contributor, kiwidu38, who added the server-default push rules, paving the way to Conduit supporting push notifications.
I improved compatibility with some commonly-broken DAG events and added rudimentary support for message edits to help Fractal finish their move onto matrix-rust-sdk.
I have not read this series so I must trust that Samuel is saying nice things! Anyway it seems like a useful resource for telling kameraden about getting started with Matrix.
Today there is a German article about Famedly on Gründerszene -> https://www.gruenderszene.de/health/famedly-kommunikation-app-gesundheitswesen. Famedly is developing a solution based on Matrix to enable healthcare providers to communicate securely with each other and to facilitate intersectoral processes such as patient transfers or the integration of IomT devices. We have also almost completed a successful financing round and hope to support the Matrix ecosystem more and more. It would be so cool to use FluffyChat, Element, or a Matrix Messenger of your choice to start a video consultation with the doctor of your choice, or to be integrated into existing clinical communication flows concerning you!
Congratulations to Famedly on the progress they're making! If you're ready to graduate from TLAs (Three-Letter-Abbreviations) to the next stage, you'll be please to learn "IomT", which means "Internet of medical Things".
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
Today we are disclosing CVE-2020-26890 / GHSA-4mp3-385r-v63f, a denial of service vulnerability affecting Synapse versions prior to 1.20.0. We strongly encourage all Synapse admins to upgrade as soon as possible. If you have not upgraded in a while, please refer to the upgrade notes, especially the latter portion of that document which covers any backwards incompatible changes which you may need to take into consideration.
As a best practice, we encourage Synapse admins to upgrade regularly, and either subscribe on GitHub or join #homeowners:matrix.org for low-traffic notifications of new releases.
We extend our thanks to Denis Kasak for reporting this issue, earning a second entry in the Matrix Security Hall of Fame.
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
In terms of Spec Core Team MSC focus for this week, we're sticking with the same three MSCs: MSC2844 (global versioning), MSC1544 (QR code verification) and MSC2790 (modal widgets). MSC2844 in particular I personally found quite interesting.
We started out this week by releasing Dendrite 0.3.0 and then ended the week with a bug-fix Dendrite 0.3.1 release.
I'd like to say thank you especially to our community contributors who have been adding new features and filling gaps!
Changes this week include:
Memory optimisation by reference passing, significantly reducing the number of allocations and duplications in memory
A concurrency bug has been fixed in the federation API that could cause Dendrite to crash
A hook API has been added for experimental MSCs, with an early implementation of MSC2836
Forgetting rooms is now supported (thanks to S7evinK)
The last seen timestamp and IP address are now updated automatically when calling /sync
The last seen timestamp and IP address are now reported in /_matrix/client/r0/devices (thanks to alexkursell)
An optional configuration option sync_api.real_ip_header has been added for specifying which HTTP header contains the real client IP address (for if Dendrite is running behind a reverse HTTP proxy)
Partial implementation of /_matrix/client/r0/admin/whois (thanks to DavidSpenler)
The error when registering a username with invalid characters has been corrected (thanks to bodqhrohro)
The -version command line flag has been added (thanks to S7evinK)
Backfilling should now work properly on rooms with world-readable history visibility (thanks to MayeulC)
Some more client event fields have been marked as omit-when-empty (thanks to S7evinK)
The build.sh script has been updated to work properly on all POSIX platforms (thanks to felix)
Spec compliance is unchanged, although some Synapse-specific tests have been removed and some new tests have been added:
We released Synapse 1.23.0 on Wednesday! Read all about it on the Matrix Blog. Otherwise, we'd like to highlight a few developments over the past week:
We're discussing a policy for ending support for old versions of Python and PostgreSQL. If you have opinions, please let us know on GitHub.
We've been looking at ways to improve the efficiency of state resolution, and Erik has managed to devise some algorithmic improvements that yield an order of magnitude speedup for a handful of pathologic cases. We hope to have a better idea of how this might work for real world workloads soon.
Lastly, we'd like to take this opportunity to remind you to please regularly upgrade your Synapse. Especially if you're not yet on 1.20.0, as we'll be disclosing a denial of service issue which affects older versions on Monday.
mewmew wanted to use my fork of Synapse, so I made the changes there configurable (instead of hardcoding to my user ID) and even added a readme to list the features. It might also be useful for other people who want custom room IDs and other fun stuff.
Support for receiving EDUs from matrix (typing, read indicators and presence) has been added. Protocol implementations can start using the new events, if they want to. Version 0.1.0 has been released along with this. You have to enable the de.sorunome.msc2409.push_ephemeral flag in your registration file
Hey, our team of colorful ponies proud to introduce you a new Matrix bridge to VK.com chats. It's powered by mx-puppet-bridge and so far supports almost all of core features people want to see, like replies, message edits and file attachments.
Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE (with the notable exception being device verification for now) and intends to be full featured and nice to look at
Profiles should now open again on older versions of Qt.
I've been slowly rewriting the message are to be qml, to make it easier to do fancy styling. As a result emojis in the emoji-completer are now colorful and the avatars in the username completer now match the avatars used in the timeline. It also open up a few other improvements, like finally sending mentions as links instead of plain text.
We have now CI running on our self hosted gitlab instance. We will probably switch of the travis CI, once this is more battle tested. Repositories are now also automatically mirrored between gitlab and github.
Fixed a bug, where we tried to read the internal world group on your server instead of keeping that to ourself.
I released 0.0.3 a few days ago bundling a few smaller improvements like redactions, copying messages and the ability to click on more links as well as some visual cleanups. Nothing radical, but also nothing breaking (I hope).
🔗Delight (Rich vdH, Michael (t3chguy), Valere, Steve, Nique, Nad)
Improving usability
Last week
Observed user tests of people trying to use Element for the first time for personal and professional use cases
This week:
Began work on fixing several of the issues observed, like:
adding an invite people button to new rooms, so users can more easily add people;
changing copy to help people understand what DMs are
Spaces
Communities are coming back with a bang! Last week we said we renamed them to Spaces, and this week, we’ve started designing what MSC1772 would look like for users on Element, to start user testing next week.
Social login
To make authentication easier, we’ve started initial implementations of SSO in Element, exploring how homeservers & Matrix clients can support multiple SSO providers. Most of the work so far is captured in MSC2858.
The release has been blocked because a bug has been found in the end to end encryption module. It has been fixed but we want to fix damages it created on one time keys before releasing the new app version.
The new background sync service mechanism PR has been updated
We started to integrate tuist to stop to be annoyed with merge conflicts on the Xcode project file
This week:
Release!
Merge the background sync service mechanism PR and make more people test it
We’ve just merged a lot of PRs, to improve room creation form and fix some bugs.
SDK side, Dominaezzz is converting some of the Service API methods to coroutines, for a cleaner code. See for instance https://github.com/vector-im/element-android/pull/2414 . 9 out of about 45 services have been migrated so far. We have about a 45 services in the SDK (!)
This week:
Release including a new way to invite friends to Matrix and to Element.
iinuwa created issues for all of the identity service api endpoints (w/ suggested module names!) 🎉
I updated lots of enumerated types to allow unspecced values, for future compatibility & robustness
Devin R made sure a custom Content-Type in responses overwrites the default rather than creating a duplicate header. #339
gnieto fixed a bunch of bugs in ruma-federation-api
Alejandro Domínguez added support for deserializing string power levels (requires the unstable-synapse-quirks feature because these events are invalid according to the spec but Synapse used to (?) accept them)
Just released version 2.2.0 of the Matrix Ruby SDK with the help of the community, with this release support for JRuby is improved - though still not perfect, in the higher-level abstraction direct messaging rooms are now exposed for all users - as well as all that the current user has, and the lower-level abstraction sees the addition of the CS API method to get an event context.
And just to reiterate; if you're using the gem - or have questions/comments about it. please do drop into the discussion room at #ruby-matrix-sdk:kittenface.studio.
!stats command which prints you the tracked durations based on a predefined or custom range of time
CSV Export based on a defined range
!break command. Putting a break in between a record or a in/out combination.
Better handling of multiple !in commands
Improved responses
Reminders to call !out and to take breaks
Fell free to join us at #timetracking-bot:famedly.de . As soon as the repo gets opened up to public we will announce it in TWIM. Feel free to give suggestions or wishes in our room :)
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
Reminder: On Monday, we will be announcing a denial of service vulnerability which affects Synapse versions prior to 1.20.0. If you have not upgraded recently, please do so.
For Synapse admins, this release support generating structured logs via the standard logging configuration (#8607, #8685). This may require changing your synapse configuration; see the upgrade notes for more information.
We've also added many new Admin APIs, contributed by @dklimpel:
Add API to get information about uploaded media (#8647)
Make it possible to delete files that were not used for a defined time (#8519)
Split API for reported events into detail and list endpoints. This is a breaking change to #8217 which was introduced in Synapse v1.21.0. Those who already use this API should check their scripts (#8539)
Allow server admins to list users' notification pushers (#8610, #8689)
Lastly, Synapse 1.23.0 addresses some significant bugs, including regressions in the SQLite-to-PostgreSQL database porting script (#8729, #8730, #8755) and an issue which could prevent Synapse from recovering after losing its connection to its database (#8726). Synapse will also reject ACL modifications from clients which would otherwise cause a server to ban itself from a room (#8708).
Installation instructions are available on GitHub, as is the v1.23.0release tag.
Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including @chagai95 and @dklimpel.
This release changes the way structured logging is configured. See the upgrade notes for details.
Note: We are aware of a trivially exploitable denial of service vulnerability in versions of Synapse prior to 1.20.0. Complete details will be disclosed on Monday, November 23rd. If you have not upgraded recently, please do so.
Add a push rule that highlights when a jitsi conference is created in a room. (#8286)
Add an admin api to delete a single file or files that were not used for a defined time from server. Contributed by @dklimpel. (#8519)
Split admin API for reported events (GET /_synapse/admin/v1/event_reports) into detail and list endpoints. This is a breaking change to #8217 which was introduced in Synapse v1.21.0. Those who already use this API should check their scripts. Contributed by @dklimpel. (#8539)
Support generating structured logs via the standard logging configuration. (#8607, #8685)
Add an admin API to allow server admins to list users' pushers. Contributed by @dklimpel. (#8610, #8689)
Add an admin API GET /_synapse/admin/v1/users/<user_id>/media to get information about uploaded media. Contributed by @dklimpel. (#8647)
Add an admin API for local user media statistics. Contributed by @dklimpel. (#8700)
Add displayname to Shared-Secret Registration for admins. (#8722)
Don't pull event from DB when handling replication traffic. (#8669)
Abstract some invite-related code in preparation for landing knocking. (#8671, #8688)
Clarify representation of events in logfiles. (#8679)
Don't require hiredis package to be installed to run unit tests. (#8680)
Fix typing info on cache call signature to accept on_invalidate. (#8684)
Fail tests if they do not await coroutines. (#8690)
Improve start time by adding an index to e2e_cross_signing_keys.stream_id. (#8694)
Re-organize the structured logging code to separate the TCP transport handling from the JSON formatting. (#8697)
Use Python 3.8 in Docker images by default. (#8698)
Remove the "draft" status of the Room Details Admin API. (#8702)
Improve the error returned when a non-string displayname or avatar_url is used when updating a user's profile. (#8705)
Block attempts by clients to send server ACLs, or redactions of server ACLs, that would result in the local server being blocked from the room. (#8708)
Add metrics the allow the local sysadmin to track 3PID /requestToken requests. (#8712)
Consolidate duplicated lists of purged tables that are checked in tests. (#8713)
Add some mdui:UIInfo element examples for saml2_config in the homeserver config. (#8718)
Improve the error message returned when a remote server incorrectly sets the Content-Type header in response to a JSON request. (#8719)
Speed up repeated state resolutions on the same room by caching event ID to auth event ID lookups. (#8752)
