We are ridiculously excited to announce that Gitter is joining the Matrix ecosystem and will become the first major existing chat platform to switch to natively speaking Matrix!
If you’re reading this from the Gitter community and have no idea what Matrix is: we’re an open source project that provides an open protocol for secure, decentralised communication - effectively the missing real-time communication layer of the open Web. The open Matrix network has more than 20M users on it and is growing fast (adding another 1.7M or so with the arrival of Gitter!)
Gitter is easily one of the best developer community chat systems out there, used by the communities of some massive projects (Node, TypeScript, Angular, Scala etc) and is a custodian of huge archives of knowledge via their chat logs. Gitter is unique in specifically focusing on developers: their tagline is literally “Where developers come to talk” (unlike Slack, which has barely any community features - or Discord, with its ban on unofficial clients, where developers are a bit of an afterthought relative to the gamers). With Gitter natively joining Matrix, we’re super excited to see the global developer community converging on the open Matrix network - and Gitter’s community rooms should see a huge new lease of life as they’re properly made natively available to the wider network as first class citizens :)
We’ve always had a bit of a crush on Gitter ever since we ended up opposite each other in the exhibition hall at TechCrunch Disrupt Europe 2014 - particularly when they demoed us not only their sexy webapp but also their official IRC server bridge at irc.gitter.im :D Over the years we’ve been gently nudging them to consider fully embracing Matrix, but perhaps understandably they’ve been busy focusing on their own stuff. However, earlier this year, our friends at GitLab (who acquired Gitter in 2017) reached out to explore the opportunity of Gitter becoming a core part of Matrix rather than a non-core project at GitLab… and we’ve jumped on that opportunity to bring Gitter fully into Matrix.
In practice, the way this is happening is that Element (the company founded by the Matrix core team to fund Matrix development) is acquiring Gitter from GitLab, with a combined Gitter and Element dev team focusing on giving Gitter a new life in Matrix! You can read about it from the Element angle over on the Element blog.
Practically speaking, we have a pretty interesting plan here, which we’d like to be very transparent about given it’s a little unusual:
At first, Gitter will keep running as it always has - needless to say, we will be doing everything we can to delight the Gitter community and keep the service in good shape.
Then we’re going to build out native Matrix connectivity - running a dedicated Matrix homeserver on gitter.im with a new bridge direct into the heart of Gitter; letting all Gitter rooms be available to Matrix directly as (say) #angular_angular:gitter.im, and bridging all the historical conversations into Matrix via MSC2716 or similar. We will of course do this entirely as open source, just as Gitter itself is open source thanks to GitLab releasing it under the MIT license in 2017. The plan is to comprehensively document our progress as the flagship worked example case study of “how do you make an existing chat system talk Matrix.”
This will of course replace the old and creaky matrix-appservice-gitter bridge we’ve been running since 2016. Gitter users will also be able to talk to other users elsewhere in the open Matrix network - e.g. DMing them, and (possibly) joining arbitrary Matrix rooms. Effectively, Gitter will have become a Matrix client.
Now we come to the interesting bit. Gitter has some really nice features which are sorely lacking in Element today:
Instant live room peeking (less than a second to load the webapp into a live-view of a massive room with 20K users!!)
Seamless onboarding thanks to using GitLab & GitHub for accounts
Curated hierarchical room directory
Magical creation of rooms on demand for every GitLab and GitHub project ever
GitLab/GitHub activity as a first-class citizen in a room’s side-panel
Excellent search-engine-friendly static content and archives
KaTeX support for Maths communities
Threads!
...and we promise to do everything in our power to preserve and honour these features at all costs and continue to give the Gitter community the experience they’ve come to know and love.
However: in the medium/long term, it’s simply not going to be efficient for the combined Element/Gitter team to split our efforts maintaining two high-profile Matrix clients. Our plan is instead to merge Gitter’s features into Element (or next generations of Element) itself and then - if and only if Element has achieved parity with Gitter based on the above list - we expect to upgrade the deployment on gitter.im to a Gitter-customised version of Element. The inevitable side-effect is that we’ll be adding new features to Element rather than Gitter going forwards.
In practice, the main outcome in the end should be Element having benefited massively from levelling up with Gitter - and Gitter benefiting massively from all the goodies which Element and Matrix brings, including:
E2E Encryption
Reactions
Constantly improving native iOS & Android clients (which should be a welcome alternative to Gitter’s natives ones, which are already being deprecated)
VoIP and conferencing
All the alternative clients, bots, bridges and servers in Matrix
The full open standard Matrix API
Widgets (embedding webapps into rooms!)
...and of course participation in the wider decentralised Matrix network.
So, there you have it. It’s a new era for Gitter - and we look forward to reinvigorating Gitter’s communities over the coming months. We hope Gitter users will be blown away by the features arriving from Matrix… and we hope that Element users will be ecstatic with the performance and polish work that Gitter-parity will drive us towards. Imagine having guest access in Element that can launch and load a massive room in less than a second!
Finally, we would like to explicitly reassure the Gitter community again that we love and understand Gitter (it was one of the very first ever bridges we wrote for Matrix, for instance) - and we will be doing everything we can to not screw up our responsibility in looking after it. Please, please let us know if you have any concerns or if we ever fall short on this.
- Matthew, Amandine, and the whole Matrix, Element and Gitter teams.
Matthew and Amandine model 2014-vintage Matrix & Gitter swag in celebration :D
Bonus update - The Changelog Interview!
Sid Sijbrandij (CEO at GitLab) and Matthew had a chance to sit down with The Changelog to talk about Gitter's Big Adventure - so tune in to hear the story first hand! Warning: contains non-ironic use of the word "synergy" :D
Hello all, and welcome to this week's addition of This Week in Matrix!
Matrix is an open network protocol for secure,
decentralized communication on the web.
My name is Andrew (aka anoa), and I'm a Synapse developer at Element.
Thanks to Ben for letting me take over the reins of TWIM for this week! I'll
actually be doing the same for next week as well, so please adjust your
clocks to Anoa Nonstandard Time accordingly.
With that out of the way, let's jump right in!
Matrix Live 🎙
It's demos week again. This time around we've got the following lineup:
Michael shows off all the new widget goodies coming to Element Web!
Bruno shows off Hydrogen's new encrypted session backup support!
Ismail details the new room creation flow on Element iOS!
Jorik presents his work on revamping the UX of https://matrix.to that he completed for his second summer internship at Element!
Hubert fixes another class of failure-to-decrypt messages edge case with Device Dehydration!
Half-Shot closes off with the tale of Encrypted Bridges!
Don't forget that Matrix Live is also available in podcast form, if you're into that sort of thing. Search for "Matrix Live" wherever you get your podcasts.
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
In terms of Spec Core Team MSC focus for this week, we've been rather busy with implementation, so we'll be continuing on with the same focus as last week. As a reminder, that's MSC2414 (making reason and score optional on reports).
Dept of Servers 🏢
Dendrite / gomatrixserverlib
Dendrite is a next-generation homeserver written in Go.
This week has mostly been spent trying to improve stability and to fix bugs ahead of the beta release, which is so far still planned to go ahead in the next two weeks.
Changes this week include:
Room version 6 is now the default for newly created rooms
Soft-fail of events that aren't allowed by the current room state is now implemented
Support for configuring old_verify_keys has been added to the Dendrite config
Correct formatting for signing key IDs is now enforced in the configuration
Initial support for peeking over federation (MSC2444) is in progress and it "even works!" (thanks Matthew!)
Federated joins will now continue being processed even if the client gives up on the join due to a HTTP timeout
Backoff code has been refactored a bit more, and now correctly affects device list syncing
/make_join now errors correctly if a federated user tries to join a room which all members have left
A bug where a single user could start multiple simultaneous federated joins to the same room has been fixed
Some initial (but unfinished) support for the /key/v2/query notary endpoint has been added
Signature verification has been updated to not fail if the event origin field is missing (although it still requires a signature from the domain of the sender field)
A number of places where we use SQL transactions have been updated with safe wrappers (thanks samcday!)
A couple of error codes on invite endpoints for room version 6 JSON violations have been fixed
Spec compliance has improved slightly for federation:
Client-Server APIs: 56%, same as last week
Server-Server APIs: 77%, up from 74% last week
As always, if contributing to Dendrite sounds like something you would be interested in, please feel free to join us in #dendrite-dev:matrix.org ! There's also #dendrite:matrix.org
This week we released 1.20.0 (and 1.20.1) highlights include shadow banning support and including unread message counts in the sync response. This will help client developers and is a precursor to improving notification support.
We’ve also been looking at adding monitoring to get a better sense of which rooms are most expensive from a state resolution perspective, we also want a better way to track federation lag.
Up next we’ll move all background tasks away from the main process and try it out on matrix.org, we are hoping for a 10-15% saving in CPU. Event persistence sharding, specifically the new stream token format is on hold slightly while we work on a nasty race condition on start up of the event persister. We hope to get back to the main sharding project next week.
This week in Matrix concludes a summer of Construct where several transformative rounds of optimization took place. I'd like to talk about these achievements and why they are important for
future directions.
Over the summer, Construct introduced new vector extensions to the project. As server software, our primary target is the datacenter which (for now) is dominated by x86 hardware. The latest features of x86 chips include AVX-2, AVX-512, and on-die GPU. These advancements are important because they help mitigate current limitations of hardware, such as the cubic relationship between a processor's frequency and its power consumption. These limitations mean that CPU cores aren't completing more cycles-per-second every iteration of their development -- they're not getting much faster.
The problem with threads is that they can introduce a lot of complications to a project. Construct is able to forego multi-threading as a network server because it is primarily IO-bound. The benefits to a single-thread design in both performance and simplicity cannot be overstated. This is why Construct stands to benefit the most from features which allow more work to be accomplished at every individual computing cycle.
Construct undertook several endeavors over the summer which directly leverage platforms featuring SSE, AVX, AVX-512, etc. As an added bonus, our approach is designed to effortlessly port to ARM's Scalable Vector Extensions (SVE) as it becomes available (inside datacenters too!). Our focus has been JSON, Unicode, and finally Base64. Detailed explanations for each of these will need to be discussed in their own posts, but in summary:
Matrix specifies a canonical form of JSON which is not necessarily the same as the JSON the server receives. Therefor it is imperative that Construct is liberal with what JSON it accepts and correct with its transformation into canonical JSON. Over the summer, a portion of Construct's canonical transform received some optimization to go beyond the default method of character-by-character read-modify-write. As seen in [1], a streaming hardware-accelerated transform now processes at least 16 characters at a time (with more possible) without forward branching except for the parts where transformation needs to occur. One of these transformations is UTF-16 to UTF-8 surrogate conversion, which leads into the second endeavor.
Construct introduced a completely branch-free Unicode toolset in [2]. Among the functions offered is a custom transform of UTF-16 surrogate pairs to UTF-8 sequences. Using the new vector registers, Construct can transform two UTF-16 surrogates in parallel to output two UTF-8 sequences, or two UTF-16 surrogates in a pair to output one UTF-8 sequence. Unfortunately, these specific functions were written at fixed vector widths, so more work needs to be done to really take advantage of the widest hardware. Each surrogate is 6 bytes, and a surrogate pair is 12 bytes; therefor we cannot make use of the last 4 bytes of a 16 byte vector. However, with a little more work this approach can be extended to a 64 byte vector, capable of decoding 5 surrogate pairs and 10 individual surrogates in parallel!
Professor Daniel Lemire recently published a paper about fast Base64 from hardware acceleration in [3]. This approach is extremely elegant on the 64-byte-wide AVX-512 system. Prior to Construct's implementation of this in [4], the Boost library base64 encoding and decoding took roughly 20 and 25 cycles per character respectively. Our implementation on the same system, an old system with only SSE2 (not even AVX-512!) yields 5 and 6 cycles per character!
All of this helps lay a foundation for Construct to introduce Federated Media Rooms sometime in the future. Currently, Construct stores media in a separate database. Recently there's been work on a separate branch at [5] which stores actual file block inside events using Base64. It is for this reason sub-cycle and branchless JSON parsing and Base64 encoding is essential for maximum performance. The result is worthwhile, as the latency for querying the media database is slower than parsing and decoding the event content already in-hand.
It's really exciting to see Homeserver development ramping up from all
angles, and nice that the protocol warts are slowly getting ironed out in the
process.
Thanks to Arkaniad, v2.7.0 of my Matrix Helm chart for Kubernetes is released with support for exporting Prometheus metrics. It pairs well with the Synapse Grafana dashboard.
The synapse docker image from AVENTER (https://www.aventer.biz), does support PowerPC (ppc64le) and ARM64 architecture now. But at the moment only under the docker tag "ppc". https://hub.docker.com/r/avhost/docker-matrix/tags?page=1&name=ppc We will be happy to get feedback.
As a Synapse developer, it's great to see the community making personal and
enterprise Matrix deployments more accessible!
Jump to events in timeline: When tapping on a reply and when tapping a matrix.to link
Display messages with up to 10 emotes or emoji bigger
New design for the chat list and message bubbles
Implement reactions
Implement password change
Implement deactivate user account
Fixes
Timeline randomly resorting while more history is being fetched
Automatically request history if the "load more" button is on the screen
Sorunome briefly mentioned afterwards that there is no 0.19.0 due to some accidental messed up tagging, and that it was easiest to just call the new version 0.19.1.
Released 0.1.0 (and 0.1.1) this week with read-only session backup enabled 🎉 Also doing more work to make Hydrogen work on IE11 on Windows 7 (it does already for Windows 10), Safari and other browsers where you get TransactionInactiveError during login, hope to release that soon.
I need to give Hydrogren a shot myself, the quick speeds and low RAM usage
are really attractive.
This week, room settings have been updated with a new intermediate screen. The codebase saw the introduction of an AppCoordinator (in swift) which will help us to have a better control on navigation within the app AND to use swift from end to end.
Polyjuice Client v0.3.0 has been released. This release includes many breaking changes. ("Breaking" in the sense of API changes, rather than the sync process suddenly failing to work, which was already featured in a previous release.) This release also includes many changes, such as the client managing more bookkeeping, detecting if it got logged out, and supporting more Matrix features. See the release notes for more information.
I present you a new project I've been working on for some time. It's a bot that allows you to use the admin API through matrix, by typing commands to the bot. The inspiration comes from the OperServ-like bots that allow IRC operators to administrate an IRC server.
The bot exposes some of the available admin APIs and aims to provide some more high level commands by combining different APIs. There is currently one "high level command", !room garbage-collect, which allow you to purge from your HS all rooms without local users.
The project is written in crystal and is my first project in this language. It should be stable in normal conditions, but don't hesitate to fill issues. A Docker image is provided for more convenience.
benpa wanted a bot that replies with a songwhip.com link whenever someone sends a music link (youtube, spotify, apple music, etc), so I wrote a small maubot plugin to do that: https://github.com/maubot/songwhip
It's available at @songwhip:maunium.net
I'm also desperately in need of this for the 10 open.spotify.com links that get thrown at me every week. Thanks Tulir!
It supports E2EE and can be easily extended with Python plugins. Can be used e.g. for GitLab notifications, automated user invites, room creation, and of course can be programmed to react to any message posted in a room.
Soru made a new thing, matrix-gotify. It is a gotify plugin to receive matrix notifications. This means, you can now receive matrix push notifications on your android phone self-hosted without the need of google services! That is why putting the full event in the push notification is actually not a privacy leak in this case.
This plugin could also be used to receive push notifications on any other kind of device that gotify supports.
Please note that this is independent of any matrix client - you don't even need to run one to be able to use this!
I asked whether an OpenPush-like solution be
built on top of this , and Sorunome responded that someone had already
started working on just that! https://github.com/gotify/android/pull/115
This would allow other matrix clients on your phone to get their push
notifications through your gotify client, instead of needing to run a process
each. Great for battery life without proprietary Google blobs!
Homeservers on-the-go
js got Synapse running in a car on a German highway:
the setup is a cigarette lighter to 2x 230V converter, one being used to power the RockPro64, the other being used to power my notebook. My notebook is connected to the hotspot from my phone, while also being connected to a USB ethernet, the other end of which is plugged into the RockPro64.
The notebook then acts as a gateway, as well as SSHing to a tiny VPS with -R, to get a public port, and forwarding that traffic to the RockPro64.
Maybe one day we'll all have an embedded Synapse homeserver in our cars. P2P E2EE
car comms anyone?
I have been through the perils of setting up a TURN server and having VoIP continue to fail, whilst spending hours scratching my head and staring at Wireshark without getting anywhere. When I was given free reign over project choice last year during my internship, I chose to start a tool that tests your homeserver's VoIP configuration, hoping it would be useful to both me and the community at large.
It saw some progress but I never managed to iron out some of the 'last few things' or put a pretty front on it — until about 2 weeks ago, when I got some more time to do it.
There are known issues — such as the scores being very arbitrary, wrong and misleading; it crashes Chromium every time (on my machine) and it completely misbehaves in Brave Browser (on my machine). It may also not be the prettiest (but hopefully it is at least somewhat inoffensive). I hope to work on these annoying issues soon.
I have deployed a test instance at https://test.voip.librepush.net — it can be tried in a WebRTC-supporting browser (probably Firefox if you want half a chance). Please don't take it to heart if you get a Fail or Poor score — it's probably not your fault. :)
Oliver was interning with us at Element this Summer. Fixing up and releasing
this VoIP tester was part of it, but he's still working on it in his spare
time even though his internship has finished :)
Through this I found out that I forgot to turn on my turnserver on my
homeserver again. Thanks Oliver!
Dept of Guides 🧭
German-Language Guide for Getting Start with Matrix
Guides like these are the essential entrypoints to a project for a lot of users. The more, in different languages, the merrier!
Dept of Ping 🏓
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
I changed the formatting of the plot a bit to make it more readable. Note that it is now using a log scale which allows us to see more data at the same time.
Now that we have multiple somewhat federating Matrix homeserver implementations, I decided to make a ping room where all the echo bots are hosted on second-gen homeservers: #ping-no-synapse:maunium.net. Synapse users are still allowed to join and !ping, but all the pongs will come from non-Synapse servers.
Based on observations in that room so far, the new server implementations don't like eachother very much.
Unread message counts in the sync response to help our client developers, this is a precursor to improving notification support.
No less than 28 async/await PRs, so we can finally share all the hard work.
Also take note that in a future release, we will be dropping support for accessing Synapse's Admin API using the /_matrix/client/* prefixes. More details follow in the changelog.
Get the new releases from any of the usual sources mentioned at https://github.com/matrix-org/synapse/blob/master/INSTALL.md. 1.20.0 is on github here, and 1.20.0rc4 is here.
The changelog for 1.20.0 is as follows:
Synapse 1.20.0 (2020-09-22)
No significant changes since v1.20.0rc5.
Removal warning
Historically, the Synapse Admin API has been accessible under the /_matrix/client/api/v1/admin, /_matrix/client/unstable/admin, /_matrix/client/r0/admin and /_synapse/admin prefixes. In a future release, we will be dropping support for accessing Synapse's Admin API using the /_matrix/client/* prefixes. This makes it easier for homeserver admins to lock down external access to the Admin API endpoints.
Synapse 1.20.0rc5 (2020-09-18)
In addition to the below, Synapse 1.20.0rc5 also includes the bug fix that was included in 1.19.3.
Features
Add flags to the /versions endpoint for whether new rooms default to using E2EE. (#8343)
Bugfixes
Fix rate limiting of federation /send requests. (#8342)
Fix a longstanding bug where back pagination over federation could get stuck if it failed to handle a received event. (#8349)
Internal Changes
Blacklist MSC2753 SyTests until it is implemented. (#8285)
Synapse 1.20.0rc4 (2020-09-16)
Synapse 1.20.0rc4 is identical to 1.20.0rc3, with the addition of the security fix that was included in 1.19.2.
Synapse 1.20.0rc3 (2020-09-11)
Bugfixes
Fix a bug introduced in v1.20.0rc1 where the wrong exception was raised when invalid JSON data is encountered. (#8291)
Synapse 1.20.0rc2 (2020-09-09)
Bugfixes
Fix a bug introduced in v1.20.0rc1 causing some features related to notifications to misbehave following the implementation of unread counts. (#8280)
Synapse 1.20.0rc1 (2020-09-08)
Removal warning
Some older clients used a disallowed character (:) in the client_secret parameter of various endpoints. The incorrect behaviour was allowed for backwards compatibility, but is now being removed from Synapse as most users have updated their client. Further context can be found at #6766.
Features
Add an endpoint to query your shared rooms with another user as an implementation of MSC2666. (#7785)
Iteratively encode JSON to avoid blocking the reactor. (#8013, #8116)
Synapse now correctly enforces the valid characters in the client_secret parameter used in various endpoints. (#8101)
Fix a bug introduced in v1.7.2 impacting message retention policies that would allow federated homeservers to dictate a retention period that's lower than the configured minimum allowed duration in the configuration file. (#8104)
Fix a long-standing bug where invalid JSON would be accepted by Synapse. (#8106)
Fix a bug introduced in Synapse v1.12.0 which could cause /sync requests to fail with a 404 if you had a very old outstanding room invite. (#8110)
Return a proper error code when the rooms of an invalid group are requested. (#8129)
Fix a bug which could cause a leaked postgres connection if synapse was set to daemonize. (#8131)
Clarify the error code if a user tries to register with a numeric ID. This bug was introduced in v1.15.0. (#8135)
Fix a bug where appservices with ratelimiting disabled would still be ratelimited when joining rooms. This bug was introduced in v1.19.0. (#8139)
Fix logging in via OpenID Connect with a provider that uses integer user IDs. (#8190)
Fix a longstanding bug where user directory updates could break when unexpected profile data was included in events. (#8223)
Fix a longstanding bug where stats updates could break when unexpected profile data was included in events. (#8226)
Fix slow start times for large servers by removing a table scan of the users table from startup code. (#8271)
Updates to the Docker image
Fix builds of the Docker image on non-x86 platforms. (#8144)
Added curl for healthcheck support and readme updates for the change. Contributed by @maquis196. (#8147)
Improved Documentation
Link to matrix-synapse-rest-password-provider in the password provider documentation. (#8111)
Updated documentation to note that Synapse does not follow HTTP 308 redirects due to an upstream library not supporting them. Contributed by Ryan Cole. (#8120)
Explain better what GDPR-erased means when deactivating a user. (#8189)
Internal Changes
Add filter name to the /users admin API, which filters by user ID or displayname. Contributed by Awesome Technologies Innovationslabor GmbH. (#7377, #8163)
Reduce run times of some unit tests by advancing the reactor a fewer number of times. (#7757)
Don't fail /submit_token requests on incorrect session ID if request_token_inhibit_3pid_errors is turned on. (#7991)
The goal of this project was to port matrix-ircd from the outdated combinators style of futures 0.1 to the new-and-improved style of futures 0.3. As initially proposed, this greatly improves code readability and removes the convoluted compiler errors that were pervasive in futures 0.1
matrix-ircd
The matrix-ircd project functions as a bridge between two chat platforms: Internet relay chat (irc) and Matrix. matrix-ircd lets you use any standard IRC Client to communicate with Matrix chatrooms and direct messages.
Mentors
This project is mentored by Philipp Mandler (@phlmn) and Jonas Platte (@jplatte). I would like to thank them for the helpful advice and code reviews they have given me over the course of GSoC 2020.
Project Results
Since much of the code was written in late 2016, there were many portions of the code that were unidiomatic and produced compiler errors. The first step I made was to remove all compiler errors from cargo and clippy in #64.
Additional tests were then written in various parts of the codebase to ensure that the upgrade would produce the same results. These changes were made in #64 and #66.
Since the project master branch currently used a custom http implementation, I moved it to use hyper, a fast and correct http implementation that already supported async/await. Additionally, I moved the module that most utilized http, matrix, to async-await and the new hyper code. Not only did these changes shrink the code by ~700 lines, they also removed a lot of unnecessary complexity. These changes were made in #67.
The last large module remaining, irc, was then ported to irc, along with its dependencies in stream_fold.rs and its upstream user in the bridge module. The most exciting part of these changes was the removal of the futures 0.1 dependency. All code was now running in futures 0.3! These changes also moved away from tasked-futures which further improved code readability. This PR was in #71.
Since the bulk of the changes were now complete, I moved onto bug fixes. As @jplatte mentioned in #71, the current single threaded approach to the application was not ideal. In #72 I updated all code to be multithreaded-compatible.
Lastly, In #77 I included more tests to functions with heavy changes, added additional logging, removed unnecessary complexity that was introduced to keep code as "1:1" as possible, and added TLS support to hyper so that https would work. This patch also fixed a rather difficult bug regarding the irc TCP streams and the buffer they were reading into.
Remaining Work
Based on my personal testing there is no additional work to be done in the realm of updating to async/await. All tests pass and the IRC server and matrix bridge function as expected. @jplatte kindly announced in the This Week In Matrix blog that we will be conducting public testing of the async_await branch on github. Barring any issues the async/await code should be merged into the master branch in the next few weeks.
TechRadar have published a fairly thorough Element secure messenger review. This covers the Web, Android and iOS clients, and appears to use the matrix.org server for signup.
The review is very positive, awarding 4.5/5, and concludes:
The Element messenger platform scores highly for its approach to security and its commitment to decentralization, and it's definitely going to be of interest to businesses wanting control over their own chats – as well as plenty of individual users as well.
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
This week we put out two point releases to fix critical bugs. At the very least ensure that you have upgraded to 1.19.2 which is a security release but you may as well go the whole hog and upgrade to the hot off the press 1.19.3. We will release 1.20.0 early next week.
Aside from that we continue to work on performance analysis and the sharding of the event persister continues - it’s proving to be a really tough job but we are getting there. We’ve also been making progress on the room knocking implementation
This week we say goodbye to Oliver (reivilibre) as a leaving present he fixed a long standing bug that prevented servers catching up after a federating outage and as we speak is furiously trying to finish a service to test TURN configuration. Thanks Oliver!
Dendrite / gomatrixserverlib
Dendrite is a next-generation homeserver written in Go
Welcome back! This week we made some groundbreaking progress - this is not a joke, we found a bug in Synapse while breaking multiple Matrix rooms, causing the Matrix team a lot of unnecessary stress. Let me explain:
The good news is that Conduit is starting to federate now. This means that you should be able to join all public rooms of the Matrix network and exchange messages. Note that Conduit does not do all the checks it should be doing yet making it stop sending messages from time to time as well as that advanced features like loading the history, syncing temporary data like read receipts or accepting invites are not implemented yet.
The bad news is that, while Synapse is happy to accept Conduit's messages when it is already part of the room, joining into one of the rooms Conduit servers are part of didn't work because of an event validation bug. The Matrix team did an excellent job at fixing this bug and releasing a Synapse patch the same day, but the damage has been done making a few rooms inaccessible to old Synapse servers.
Thanks to everyone who supports me on "Liberapay" (https://liberapay.com/timokoesters) or Bitcoin!
I ported Synapse + its dependencies to NetBSD and it is now in pkgsrc as chat/matrix-synapse. This means that Synapse is now easily installable on any operating system that supports pkgsrc (which is many, e.g. NetBSD, Linux, macOS, Solaris, AIX, Haiku, …). And while at it, I also ported mautrix-hangouts + dependencies (in pkgsrc as chat/mautrix-hangouts).
Hey bridge developer enthusiasts! Myself and ChristianP (bridge crew of matrix.org) have been working hard on freshening up the matrix-appservice-bridge library. It's no secret that it was using "classic" Javascript rather than ES6 syntax, was poorly documented in places and there were no types at all to make use of.
But that's all changed now, the 2.0.0-rc1 release brings fresh types, new convenience methods, a total swichover from Bluebird Promises to native Promises and much much more. If you are looking to quickly get set up writing bridges, it's never been easier.
Check out the slack-starter project to get your first taste of bridge excellence and see what you think.
As this is an RC, feedback is valuable and logging bugs helps us all. Please do so if you encounter any issues 😄
Hi encryption fans. This week the bridge team has been churning away at adding support for encrypted rooms in the matrix-appservice-bridge library, to enable support in matrix-appservice-irc, matrix-appservice-slack, and anyone else using the library. This is probably the most exciting feature the library has seen in a long time!
The feature will be a simple config toggle, so existing bridges will have to do very little work to support it.
The first phase plans to use Pantalaimon with bridge users syncing like real users, but eventually the hope is that this will be all built into the library. Come check out this issue to keep track of progress.
Element Android 1.0.7 has been released (still stuck in Google pipes writing those lines), and will be quickly followed by 1.0.8 to fix a problem with cross signing verification.
We are working to implement search of messages in a room and we keep on fighting bugs and improving performance of the application.
We are also iterating on the home screen, to improve user experience, and try to satisfy users with a few rooms as well as users with several hundreds of rooms.
Released 0.037 on Monday with some polish for E2EE:
remove outdated devices when querying
update room summary with decrypted events, also when retrying decryption
various crash protections
keep megolm session with earliest index when receiving a room key we already know about
add new icon
fix issue where olm would be loaded twice when login failed
show decryption errors in timeline
show encryption enabled tile in timeline
The rest of the week was implementing session backup, which I hoped would be ready to release today but it needs some more polish unfortunately. In the meantime, here's a GIF teaser:
Hemppa gained a small but big feature for making fediverse better: per-room Mastodon tooting support. This means that you can set up a Mastodon account in your room and anyone with moderator rights can toot to that account. This way you and your roommates can super easily send toots to the world directly from Matrix. Mastodon (unlike the big alternative) supports RSS feeds out of the box so it's very easy to subscribe to users and hashtags with stock RSS bot. Toot #hemppathebot if you like it! Another new contribution worth mentioning is welcome which can send welcome messages to users registering on the server or joining a room. https://github.com/vranki/hemppa
Another week, another Ruby SDK release. This time adding a little fix to avoid duplicate state events being passed twice to the application if identical ones arrive in both state and timeline, also adds a global state event handler and moves room event handling to not depend on room instances themselves.
Asked how much the SDK is used in production, Ananace said:
a little, we've hooked Matrix into our server orchestration / configuration management system TheForeman - which is a Rails application. Also got a colleague who's doing a bot to linkify internal ticket IDs.
Thanks to @q-b and @jtescher we have implemented more of the federation endpoints (bringing us up to about 85% completion) and every (one) endpoint of the Push Gateway API!
We fixed a bug where federation endpoints expected access tokens instead of server signatures for authentication. In the future, we may add an API that requires signatures on requests.
Finally, we've also continued progress on the non-exhaustive type updates for backwards compatibility; we expect to finish this within the coming weeks.
v0.2.0 of matrix-reminder-bot has been released! This releases includes lots of bugfixes, updates and polishing. Find the list below:
Features
Better support for command prefixes other than the default !.
Just writing !silence now silences the currently active alarm.
The bot will now print the correct syntax of a command if the user fails to follow it.
The bot will reply to events if it cannot decrypt them, as well as offer helpful tips that both the user and bot operator can try to fix things.
Bugfixes
Timezones. They should finally work correctly! ...no 100% guarantees though.
Alarms were a bit broken. They're fixed now.
Fix commands with formatting and newlines not being picked up by the bot.
Fix non-latin characters preventing reminders from being deleted.
Internal changes
Add a dev-optimised Dockerfile for quicker iteration during development.
Better wording revolving alarms. They're just reminders that alarm repeatedly when they go off.
Log why the bot is unable to start.
Don't print "Unknown help topic" in case the user is trying to ask another bot for help.
The config dict is now a singleton.
Type hints everywhere!
Additionally, the minimum Python version is now 3.6. matrix-reminder-bot is made with nio-template.
Dept of Ping 🏓
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
Normally the ping shows how long other servers take to receive the sending-server's message. Some servers receive it quickly, some take hours. All this flows into the statistic of the sending server. I wrote a small program that's the other way around. Instead of measuring the sending-server's ping, it associates each measurement with an echo-bot (instead of the sending server), so this plot shows which maubots are the fastest to respond (it's probably not accurate at all, because most of the pings were started on conduit servers this week, but it's interesting nevertheless):
That's all I know 🏁
See you next week, and be sure to stop by #twim:matrix.org with your updates!
Wow the summer has flown by, it feels like just yesterday I was learning how to rebase and what exactly it is Ruma does. I exaggerate slightly, but it is a big library with lots of public API surface. I have learned more in the last few months than in two years of school. I have been able to observe and participate in a project with a community growing around it, been a part of discussions about design and best practices, given and received numerous code reviews as well as learned the process of addressing the feedback, and working from a specification. In short, this has been an amazing opportunity to gain experience in all the things that are hard to obtain in a classroom.
My project goal was to improve the existing macros in ruma-events-macros and ruma-api-macros. It became clear early on that this would include some major API changes and that improving the macros as they were was pointless without also moving to a new public API. While improving the durability and readability of the macro code I also rewrote entire sections to accommodate the new design.
A quick overview of the Matrix protocol for reference: a client sends content that is interpreted by the server as events. The server distributes those events out to other clients and other servers (the server case is known as federation). Ruma groups these events by kind Message, State, Ephemeral, ToDevice and Basic which are represented as generic structs (StateEvent<C>). Each event kind needs to be able to hold many different content types, for state events there is room creation, room name, and membership events to name a
few. Using the macros, enums are generated to represent all state event possibilities, so a variant for membership, room name, etc. These types exist to support the core API request and response types for each endpoint that is defined by the Matrix specification.
One of my personal goals was to become more familiar with git. With the help of my mentor I now feel more confident using this tool that is so essential to developers. I became fairly adept at merging, rebasing, and navigating all the headaches that come with that. I learned plenty of new commands. A few highlights: cherry-pick and specific uses of reset to avoid copy-pasting fixes and adding more commits. I used the reset command to craft good commits, splitting work into appropriate chunks. I am glad that I had the opportunity to hone my git skills. I feel like I have accomplished my goal and then some!
I am proud of the work that I have done: Being part of moving ruma-events much closer to the 0.22 release and creating macros to generate types specific to the Matrix specification. Working with the community that has grown around Ruma has been rewarding and I plan on sticking around.
Synapse 1.19.2 is a security patch. All federating instances should upgrade immediately.
Today we are releasing Synapse 1.19.2, which is a security patch release containing a fix to encountering invalid events over federation. We are also putting out a fourth release candidate for the upcoming Synapse 1.20.0 release with the same fix.
The bug prevents affected Synapse instances from joining rooms with invalid events. Server administrators running federating instances are strongly encouraged to update as soon as possible.
Those on Synapse 1.19.1 or earlier should upgrade to Synapse 1.19.2, while those who are running a release candidate of Synapse 1.20.0 should upgrade to 1.20.0rc4.
Get the new releases from any of the usual sources mentioned at https://github.com/matrix-org/synapse/blob/master/INSTALL.md. 1.19.2 is on github here, and 1.20.0rc4 is here.
The changelog for 1.19.2 is as follows:
Synapse 1.19.2 (2020-09-16)
Due to the issue below server admins are encouraged to upgrade as soon as possible.
Bugfixes
Fix joining rooms over federation that include malformed events. (#8324)
My name is Arnav Tiwari and I am a prefinal year undergraduate student from IIT Kharagpur and I wanted to share my amazing journey with Matrix. I am a budding open-source enthusiast and this was my first experience with Google Summer of Code. For the past few months, I had been working on a project to develop an HTML embeddable chat client under the GSoC program for Matrix. Matrix provides a highly versatile SDK for making custom clients that can be leveraged for a variety of applications, one of which is using Matrix to power an embeddable chat client. A chat client itself can have numerous forms, whether it being a live chat to a simple comments section. This project was intended to provide an easy-to-use and yet highly customizable client that can be deployed on a website with minimal effort.
My goal for the project was to have a useable project by the end of the coding period, however, as it turned out, the project was going to be tested in the real world far sooner than that. The need for an embeddable client and the feasibility of the project to fulfill this role was demonstrated during the second month of the coding period itself. The client was deployed on the website of CommCon 2020, a virtual conference on communication technologies (an apt place to be tested, coincidentally). On the days leading up to the conference and during the conference itself, I helped the organizers to set up, integrate, and troubleshoot the client when required. While the process went mostly without any hiccups, there was one small incident when the client broke during production. Since the project was still pretty early in development, I didn’t expect it to be bug-free and had anticipated the possibility of this happening. I was keeping an eye on things, which proved to be a prudent decision as I was able to fix this problem quickly and with minimal downtime. The rest of the conference went smoothly and the client performed quite well even when the number of users was quite large (A testament to Matrix’s scalability). Getting to experience this was a pleasant surprise since I never expected to have real users so soon, much less so many at once. Seeing the client being used out in the wild was a very fulfilling thing to witness. I also gained some very valuable feedback, courtesy of Dan, CommCon’s master of ceremonies.
Over the next month or so, I kept on steadily adding features and building up the client. The next big break for the project came in the form of another conference. KDE Akademy 2020. This was a big surprise as I genuinely didn’t expect to see another large conference using the project so soon. The conference was scheduled to be held after a week or so after the end of the coding period. This time, however, the integration was almost completely handled by the conference organizers themselves since it had to be integrated with their version of BigBlueButton, a web conferencing system. As the conference drew nearer, things seemed to be working out well and there was no sign of trouble. When the day of the conference finally came, however, many things seem to break simultaneously due to an apparent incompatibility with BBB. In the end, despite the numerous attempts by the conference organizers and myself to remedy the issues, they had to roll back to an older version of the chat since the risk would be too great. The organizers were understandably disappointed since they had spent a while working on the integration and had seen the great potential of using this client in place of their old chat system.. Even though It was a sad conclusion to the journey, there were still many lessons to be learned. Most importantly, even though the client might work well in standalone circumstances, ease of integration might have some room for improvement. Open-source development never truly stops and I don’t intend to give up on this project. I look forward to constantly improving it and seeing more people adopt it.
These past few months were a spectacular experience. Even before starting this journey, I knew I would learn a lot but this still managed to exceed all my expectations. I got to learn things I never would have thought I would be able to experience under GSoC. I met some awesome people along the way, I’m extremely grateful to my mentors Ben Parsons and Travis Ralston for being the best mentors anyone could ever ask for. They were always approachable and friendly throughout the entire program and I never hesitated before asking for their help. Without their guidance, all of this would’ve certainly not been possible. Lastly, I want to express my gratitude to Matrix for believing in me and giving me the opportunity to undertake this project. The Matrix community is full of talented people who will go the extra mile if you ask them for help. It has truly been a pleasure working with them and I hope to continue working with them in the future. Cheers and hope to see continue seeing you all!
Since we suddenly have multiple projects that've separately gone on to trial our new and shiny Matrix-on-location for multiple respective use-cases, from conferences to anonymous - encrypted - support chats, we've had to end our soft-launch period very quickly.
Oh dear.
For minutes, confusion reigned in the chat about just what this meant! Fortunately, clarity arrived:
To clarify; Ending the soft-launch period means that we've now gone on to a full launch, with public announcements and everything else that includes
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
In terms of Spec Core Team MSC focus for this week, the widgets spec and related MSCs are still being completed, however is formally removed from the focus list for this week. In its place we've put something in a very similar vein: MSC1960 (OpenID Connect exchange for widgets).
We had another long overdue Spec Core Team retro this week. On the whole it felt very productive, and we've hopefully helped identify some key issues and actions going forward to help internally organise the way the team spends time on MSCs (read: unblocking review progress from the team).
We're still working out our exact strategy asynchronously at the moment, so expect more details to follow soon.
This week we put out a new release candidate for Synapse 1.20.0, highlights include shadow ban support, more async/awaiting and unread message counts being added to sync responses to support upcoming notifications work.
Aside from that Erik has been continuing to get event persistence sharding ready for shipping. He already has a working version, but currently all workers will run at the speed for the slowest instance. Once that is fixed we will put it on matrix.org to see how it performs.
Patrick has finished up on asyncing all the things, and is now studying flame graphs to figure out where all the cycles on the main process are going.
Andrew has been taking a look at the knocking MSC, expecting concrete progress in the next few weeks.
Brendan has been working notifications support and is currently in VoIP land helping Dave on trying to level up our VoIP support.
Finally Oliver has dusted off his TURN tester from last Summer to get it working before he returns to university. The idea is that it will act a bit like the federation tester for but for TURN configuration.
Hi everyone, Devin R and I are working on state resolution, hopefully we have something exciting to show you next week. I also worked on the remaining key backup endpoints, so for example deleting backups works as expected now.
Thanks to everyone who supports me on Liberapay or Bitcoin!
Dendrite
Dendrite is a next-generation homeserver written in Go
Released 0.0.36 with end-to-end encryption enabled! 🎉
Most of this week was spent on performance improvements for IE11, where asm.js runs quite slow. Decryption is run in parallel on 4 workers in that browser, giving a massive speed-up and added responsiveness (the UI would freeze for 7 seconds when opening a room prior to this). There is still some polishing to do on sharing room keys, which I will get to next week, but all in all, it should work well. Next week I also want to get started on supporting key backup, so you can also read your encrypted history from before you logged in.
Element for Nextcloud (formerly Riot Chat for Nextcloud) has released v0.6.5 and v0.6.6. In this update, Element Web was updated to v1.7.5 and support for adding a custom integration server in the config through the settings UI and support for Nextcloud 20 was added.
We have released Element Android 1.0.6. Now we are working on sync mode for F-Droid version of the application (i.e. without FCM). We are also improving the experience with 1-1 calls.
This week, we released 1.0.10 which provides PIN protection and the return of the incoming native call screen. Then, we started to update room creation flows with new icons and behaviors for creation buttons and a new screen for room creation.
Maunium sticker picker
This isn't strictly a client, but it's in the clients section.
Getting stuff into the spec or even implemented in Elements is very slow, so while we wait for proper native sticker packs and pickers (i.e. not in an iframe/webview), I decided to make a better sticker picker widget: https://github.com/maunium/stickerpicker
The goal is to be as fast and simple as possible. The picker and sticker packs are just some static files that can be served anywhere (no compilation required). It doesn't require self-hosting anything except said static files, and it has been confirmed to work on all three Elements.
There's a Telegram import script that's used to reupload stickers to Matrix and generate the static sticker pack files that the picker widget uses. It's technically also possible to make the sticker pack files yourself, but there's no easy way to do that yet.
Due to the simplicity of the picker there's no authentication, which also means everyone sees the same sticker packs. One way to get around that limitation is to have a unique URL for each user. I'll probably write a bot to manage copies of the picker for multiple users in the future.
Other future improvements include remembering frequently used stickers (surprisingly, all three Elements keep localStorage for widgets), importing Telegram animated stickers and searching for stickers by emoji/name
later more words appeared from Tulir:
Since the initial announcement, my sticker picker has received some more features:
Added a slider to configure the number of stickers shown per row
Frequently used stickers now show up at the top
Imported all the packs from Scalar (the default integration manager)
Just bumped the Matrix Ruby SDK to version 2.1.2, fixing another bug with server discovery as well as the fact that state events haven't been provided for the application using the SDK. Oops.
libQuotient 0.6.1 is out, with fixes for bugs found during the work on the next Quaternion release and also an optimisation in the way profiles of all the users are managed - noticeable when opening large rooms (like our lovely HQ). Quaternion 0.0.9.5 beta is coming real soon now (TM), with some big features being coded in and merged as I write this.
I created mnotify which is a CLI based matrix client for automation tasks in shell scripts. Currently it supports login (via password), logout, sending messages, simple syncing, "get"ting the synapse admin api, creating a room, inviting users to a room, and joining a room.
https://publiclist.anchel.nl has been updated. The publiclist contains several public homeservers who can be used freely. Also listed some communities and roomdirectories which can be found at homeservers, last but not least public useable bridges can be found. The homeserver list contains now also uptime percentage and ip info. There is also a json that can be used by client builders.
And in completely unrelated news, I also just tagged version 1.0.0 of an invite bot for Matrix, which helps in doing bulk invites to communities and community-associated rooms.
A single bulk-invite (MXID or 3PID) to a main room will be propagated to invites into the community (if wanted), as well as invites to all rooms linked to the community, once the invited user joins the main room.
Michael Collins talks about the Matrix, covering the new features, perthchat.org, and ChatOasis an upcoming FOSS Matrix hosting company.
Dept of Ping 🏓
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
Rank
Hostname
Median MS
1
fairydust.space
325.5
2
helderferreira.io
614.5
3
neko.dev
661.5
4
kif.rocks
718.5
5
matrix.vgorcum.com
1785.5
6
an-atom-in.space
2034
7
urech.ca
2226.5
8
ragon.xyz
2436
9
im.kabi.tk
2478
10
utzutzutz.net
2588.5
That's all I know 🏁
See you next week, and be sure to stop by #twim:matrix.org with your updates!
I'm Tyagraj Desigar and I worked on Opsdroid with Stuart Mumford and Drew Leonard. Opsdroid is a python framework for creating platform agnostic bots usable with multiple chat services including matrix.
The Plan
The project's focus was on the interaction layer (called a connector) between opsdroid and matrix. As it stood, the connector used the deprecated matrix-python-sdk and had no support for encryption. The aim was to change this by moving over to matrix-nio.
I had planned to work on some other features as well:
A database module for opsdroid using matrix state events
Support for unused matrix events
Homeserver lookup using .well-known API requests
Device verification process for the bot
Porting to nio and adding encryption support was the bulk of the project, and the minimum I wanted to accomplish by the end of the project. I wanted to have a connector that worked as it did already with support for encrypted rooms which needed minimal extra configuration.
The Process
The process began with This PR which gave us a head start with the migration to matrix-nio. Through helping to review the PR I got to understand how matrix-nio and opsdroid work.
The implementation of encryption support was a little tricky in that it was tough to figure out the process required to do that in the context of the connector. One problem we faced was the installation of dependencies. Installing libolm, a C library which nio uses for encryption was a less than smooth process. This spawned a couple side projects that dealt with the CI, testing and installation of opsdroid. In the end we found a solution and had a working connector on our hands.
Next we shifted focus to the database module. It was based on this project. I rewrote it with nio and added a few features. The idea was straightforward but the implementation had many catches since we were working under some constraints for the interfacing with opsdroid. It went through several iterations before we settled on the final product.
The encryption and database took longer than I had initially expected which meant we didn't get to work on adding support for more events and homeserver lookups. I had a go at adding device verification steps while working on the encryption support but that turned out to be quite complicated and would introduce some breaking changes, besides cross-signing had just been introduced so we decided to drop that till nio implemented some way to leverage cross-signing.
Along the way some issues cropped up with the testing frameworks and CI that were hard to pinpoint and caused further delays but were solved eventually.
The Conclusion
I am extremely happy with what I've accomplished and hope that I have been able to achieve the standards set by the matrix community. It was a challenging and exhilarating experience. I have learned more in this project than I could've imagined and gained a ton of invaluable experience with software development thanks to my mentors who pushed me forward and guided me throughout. This journey was beyond amazing and I will be sure to contribute to matrix moving forward.
You can find complete details of everything I worked on during GSoC here.
