The most important thing to know about 1.18.0 is that it contains support for sharding multiple workers. Specifically this means being able to run multiple federation senders, multiple client readers to handle registration and multiple push workers. This will be important for anyone running a large scale install of Synapse. You can read more about how to benefit from these changes in docs/workers.md. In the same spirit we also moved typing notifications from the main process.
Aside from that, we have new admin API support to list the users in a room, support for oEmbed for media previews (you can unfurl tweets again!) and a general slew of federation bug fixes.
Get the new releases from any of the usual sources mentioned at https://github.com/matrix-org/synapse/blob/master/INSTALL.md. 1.18.0 is on github here.
Changelog for 1.18.0 follows:
Synapse 1.18.0 (2020-07-30)
Improved Documentation
Update worker docs with latest enhancements. (#7969)
Synapse 1.18.0rc2 (2020-07-28)
Bugfixes
Fix an AssertionError exception introduced in v1.18.0rc1. (#7876)
Fix experimental support for moving typing off master when worker is restarted, which is broken in v1.18.0rc1. (#7967)
Internal Changes
Further optimise queueing of inbound replication commands. (#7876)
Synapse 1.18.0rc1 (2020-07-27)
Features
Include room states on invite events that are sent to application services. Contributed by @Sorunome. (#6455)
Abort federation requests where the client disconnects before the ratelimiter expires. (#7930)
Cache responses to /_matrix/federation/v1/state_ids to reduce duplicated work. (#7931)
Bugfixes
Fix detection of out of sync remote device lists when receiving events from remote users. (#7815)
Fix bug where Synapse fails to process an incoming event over federation if the server is missing too much of the event's auth chain. (#7817)
Fix a bug causing Synapse to misinterpret the value off for encryption_enabled_by_default_for_room_type in its configuration file(s) if that value isn't surrounded by quotes. This bug was introduced in v1.16.0. (#7822)
Fix bug where we did not always pass in app_name or server_name to email templates, including e.g. for registration emails. (#7829)
Errors which occur while using the non-standard JWT login now return the proper error: 403 Forbidden with an error code of M_FORBIDDEN. (#7844)
Fix "AttributeError: 'str' object has no attribute 'get'" error message when applying per-room message retention policies. The bug was introduced in Synapse 1.7.0. (#7850)
Fix a bug introduced in Synapse 1.10.0 which could cause a "no create event in auth events" error during room creation. (#7854)
Fix a bug which allowed empty rooms to be rejoined over federation. (#7859)
Fix 'Unable to find a suitable guest user ID' error when using multiple client_reader workers. (#7866)
Fix a long standing bug where the tracing of async functions with opentracing was broken. (#7872, #7961)
Fix "TypeError in synapse.notifier" exceptions. (#7880)
Fix deprecation warning due to invalid escape sequences. (#7895)
Updates to the Docker image
Base docker image on Debian Buster rather than Alpine Linux. Contributed by @maquis196. (#7839)
Improved Documentation
Provide instructions on using register_new_matrix_user via docker. (#7885)
Change the sample config postgres user section to use synapse_user instead of synapse to align with the documentation. (#7889)
Reorder database paragraphs to promote postgres over sqlite. (#7933)
Update the dates of ACME v1's end of life in ACME.md. (#7934)
New guide for client e2ee implementation, including cross-signing
Thanks and congratulations to Sorunome this week for releasing a new guide: Implementing more advanced e2ee features, such as cross-signing. This is a hugely detailed guide detailing the necessary steps to enable Cross-Signing and verification. Thanks as well to uhoreg and poljar for their work in reviewing the text!
I wrote a master thesis that tries to create e2e encryption based on Ethereum on top of Matrix. It may not solve all problems but it's an interesting experiment
I have put the long-awaited MSC2312 out of WIP - this is about matrix: URIs in case you haven't heard - and it's already in active discussion at #uri-scheme-proposal:matrix.org! Join the fun to make it the most awesome! The upcoming Quotient 0.6 has an implementation of those so adventurous Quaternion users can start experimenting right away.
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
In terms of Spec Core Team MSC focus, we've reduced last week's list of 5 MSCs down to one, MSC2674 (aggregations part 1: relationships). Next week, we're going to focus entirely on that MSC to make some headway on finally getting aggregations into the spec.
The P2P Yggdrasil iOS demo has seen some improvements:
Federation sender blacklists are now persisted which is important as the iOS app will be frequently terminated.
SQLite3 now uses TransactionWriters in more places to reduce the amount of 'database is locked' issues.
Federation sender has had much of its storage code refactored and de-duplicated.
In addition, there has been much work getting Dendrite to support E2E rooms, which it now mostly does.
You can create and join E2E rooms and have a conversation locally, though there's issues with it over federation. In addition,
Dendrite currently lacks device lists so adding new devices may result in unable-to-decrypt errors:
Send-to-device events now work over federation.
Device key uploads and querying now work both locally and over federation.
One-time key uploads and claiming now work both locally and over federation.
Spec compliance is up from last week:
Client Server APIs: 49%, up from 48%.
Federation APIs: 53%, up from 51%.
Doesn't it feel like we'll be using Dendrite in the wild really soon now?
This week we continued on our async/await athon, you can track progress here, expect steady progress over the coming weeks.
We dusted off the notifications project working with our pals in the Element client teams to ensure that the push rule defaults make sense. On the flip side we made further improvements to Sygnal and will merge the ability to have it sit behind a proxy rsn.
We are also trying to figure out how media retention limits should work in Synapse.
Up next is to continue our performance push. We need to spend time between pulling things off the master process as well sharding the event persister. We will continue with notifications and maybe finally squash https://github.com/matrix-org/synapse/issues/2528
Hello people. It's been a while since I've talked about a new bridge...or actually any bridges. Fear not, now you can hear both! Today I'm announcing the birth of another project matrix-figma. Figma is an online collaborative design tool. It was created to scratch an itch with the Element design team, who wanted to see comments appear in their Matrix rooms in realtime when someone commented on a Figma file.
The bridge works by first joining it to an "admin room", which functions as a permitted user list. The bot can then be invited to any other room so long as the invitee is in that admin room, and can then ask the bot to start directing comment notifications into the room.
The room uses room state to hold configuration, so you can host the bot as a docker container anywhere without the need for any support files!
I can announce that the bridge is now functional for simple use cases such as these, with more interesting functionality in the future.
This Python project was initially intended for a full migration from Slack to a new Matrix homeserver and allows reusing the imported user accounts. We have some research groups using Slack that would like to import their Slack workspace history to our Matrix homeserver. This PR allows to import a (free) Slack export to a fresh and empty Synapse instance that is federated with our main homeserver. We do not need a full "migration": we kick all imported users and invite the existing Matrix users from our homeserver.
Two groups already imported the history and switched to Matrix, one more is in queue and probably more to come.
This is a really, really exciting development, and the context for which Sven is expecting to use it is BIG, I can't wait til we can share more about it!
The Twitter DM bridge I announced last week mostly works now. It does text and reaction bridging in both directions, twitter->matrix media bridging, end-to-bridge encryption and backfilling. I also added support for it in mautrix-manager for web-based login.
If it mostly works, I guess it's time to mostly start trying it out!
Fluffy 0.16.0 has been released! It is already available on F-Droid, Google Play and IOS Testflight will follow. You can also try it out in the webbrowser. Visit https://fluffychat.im
Features
Implement web notifications
Implement a connection status header
Changes
Switch out database engine for faster performance
Greatly improve startup time
Added languages: Galician, Croatian, Japanese, Russian, Ukrainian - Thanks a lot to all the weblate users!
Only show the microg toast once, if you have play services disabled
Homeserver URL input now strips trailing whitespace and slash - Thanks @Katerina
Also use prev_content to determine profile of a user: This allows the username and avatar of people who left a group to still be displayed
Fixes:
Fix not being able to initiate key verification properly
Fix message sending being weird on slow networks
Fix a few HTML rendering bugs
Various other fixes
Fix the 12h clock showing 00:15am, instead of 12:15am - Thanks @not_chicken
Fix an issue with replies and invalid HTML
Fix messages getting lost when retrieving chat history
Fix a bug where an incorrect string encoding from the server is assumed
Fix a bug where people couldn't log in if they had email notifications enabled
Last week, we forgot to announce that we renamed the Riot-iOS app to Element but this is not a surprise anymore. The app version is now 1.0.0. This change came with a lot of UI tweaks like new icons and new colors.
Since then, we made bug fixes we will release soon. This week, we also worked on PIN protection for the app, including support of Touch ID and Face ID.
Quotient 0.6 released! The release notes are somewhat long, since it's been 4 months since the last stable version update, and more than a year since the 0.5 release - thanking two heroes of this release, and giving a nod to the original Quaternion author who happened to contribute using an inverted time machine. Read here: https://github.com/quotient-im/libQuotient/releases/tag/0.6.0
Big news - first release in a year. Do take a look at the full notes, they're very readable. In particular, they call out contributions from Black Hat, Alexey Andreyev and the original project creator, Felix Rohrbach aka @fxrh.
Dept of SDKs and Frameworks ๐งฐ
ruma
Ruma is a Rust project to create a comprehensive set of APIs for Matrix. Previously there was a Ruma homeserver project.
We weren't in TWIM for a while, but that doesn't mean no progress! Apart from Devin's constant work on ruma-events, there was also some movement in ruma-identifiers and ruma-client-api, with the former seeing a new release, 0.17.0.
We're planning to do more releases next week, and hopefully a 0.1 release of the ruma crate that provides a simpler entry point by re-exporting everything else, in one of the weeks after that. Stay tuned!
Dept of Ops ๐
matrix-docker-ansible-deploy
This Ansible playbook is meant to easily let you run your own Matrix homeserver.
It's been a while since it's come up, but matrix-wug is a bot capable of transcribing characters writable via normal keyboards to scripts from other languages/notation systems
It supports: x-sampa, z-sampa, proto-indo-european, Inuktitut, and Iรฑupiatun.
Now it also supports hiragana! The romanization system is something like Wฤpuro rลmaji, but sticks to Nihon-shiki when in doubt.
I want this transcription to be forgiving, so if you find it not transcribing something in a way you'd expect it to, feel free to DM me!
To use it use the hi key with a delimiter like /x/ or []
hi/konnichiwa maatorikusu!/
vรธgg is also behind pantalaimon now so it now works in encrypted rooms ๐
If you need to convert swiggles into a different type of swiggle, dandellion has you covered here.
This bot is not about adapting the current jitsi integration provided by Element IM as widget.
If you have a configured jitsi server to be authenticated with jwt tokens as in https://github.com/jitsi/lib-jitsi-meet/blob/master/doc/tokens.md , this bot will assist you when claiming for a jwt instead of going to jwt.io each time.
You'll need to create a user and invite you in each room where you want to create a jitsi room url respecting the matrix power levels in the room.
When the service is running invite the bot and send message !jitsi-jwt
If you are a moderator in the room (alias you have enough power levels to redact messages), then you will get a direct message from bot with the url to jitsi
room with the jwt get parameter.
New cody release. @cody:bordum.dk is a chat bot that works as a REPL for your matrix rooms. This week I worked on metrics. I host a Grafana instance with anonymous access allowed, so codys charts are now embedded the #cody:bordum.dk chat room.
I am proud to announce the very first release of sMythbot, the Matrix chatbot designed to control your Myth Tv DVR remotely. This release should be considered a tech preview. I welcome feedback and Bug reports. More information on installation and setup is available in the project wiki.
It currently supports the following commands
!smythbot help: Display this message
!smythbot set mythbackend address: Sets the Myth Tv backend address to use for this room.
!smythbot set mythbacked port: Sets the Myth Tv backend port to use for this room.
!smythbot view mythbackend address: Allows you to view the Myth Tv backend address set for this room
!smythbot view mythbackend port: Allows you to view the Myth Tv backend port set for this room
!smythbot view mythbackend info: Allows you to view various pieces of information for the Myth Tv backend connected to this room. It will not work if the address and port are not set.
!smythbot display upcoming recordings: Displays the upcoming recordings on your Myth Tv Backend.
!smythbot display recorded programs: Displays the recordings from the default recording group that are stored on your Myth Tv Backend.
The team behind Noteworthy (Matrix over WireGuard overlay networks) has started work on bringing better mobile web support for Element (riot-web).
Noteworthy Elements is a lightweight shim powered by the Ionic Framework thats goal is to bring first-class support for Element running on mobile devices.
What we have so far
Usable version of Element wrapped in native iOS / Android app
Ability to run multiple instances of Element in a native iOS / Android app (ie connect to multiple home servers simultaneously)
Our initial experimentation has been positive and our goal is to make Element the best mobile client for Matrix with minimal fuss. With minimal changes to Element's codebase our goal is to land (significantly better) support for mobile web in upstream Element. Join us over at #noteworthy:tincan.community to get involved!
Trust-based Moderation Using Distributed Chat Systems for Transitive Trust Propagation
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
Rank
Hostname
Median MS
1
fairydust.space
309
2
bardiharborow.com
545
3
nicoll.xyz
647.5
4
grant.org
667.5
5
elsmussols.net
694
6
heitkoetter.net
871.5
7
ragon.xyz
876
8
acmelabs.space
906.5
9
opensuse.org
915.5
10
matrix.vgorcum.com
1017
That's all I know ๐
See you next week, and be sure to stop by #twim:matrix.org with your updates!
Let's hear from Christian about working on his dream projects, his thoughts on bridging, hosting, and the importance and nature of chatbots and assistants.
Dept of Status of Matrix ๐ก๏ธ
"Element"
This week was the announcement of the much-awaited rebrand: Riot is now Element. In fact, three brands are coalescing into one: Riot and New Vector will be referred to as Element, while the SaaS platform known as Modular.im is now Element Matrix Services.
Note that Matrix is not involved in this change. Matrix is still Matrix, don't worry about that!
CommCon Virtual 2020, with added Matrix
CommCon is an event dedicated to Real-Time Communications. In 2020 they made the difficult decision to go online-only, but had a fear of missing out on the "hallway-track" that is so important to industry events. Their solution was to include a live chat to run alongside their streamed talks!
Matrix was a natural choice for the crowd, but they wanted a way to encourage viewers to join the correct room from their own Matrix clients. To do this, they used the GSOC project from arnav-t - an HTML Embedded client - to present a live scrolling-view of the chat. You can see the result at https://2020.commcon.xyz/live/.
This week, the Spec Core Team will be focusing on MSC2610 (Removing m.login.oauth2 from User-Interactive Authentication), MSC2611 (Removing m.login.token from User-Interactive Authentication), MSC2663 (Errors for dealing with non-existent push rules), MSC2674 (Event Relationships), and MSC2689 (Fix E2EE for guests).
Dept of GSoC ๐๏ธ
Ruma/Matrix GSoC update
Ruma is a Rust project to create a comprehensive set of APIs for Matrix. Previously there was a Ruma homeserver project.
This week in the ruma/matrix Google Summer of Code project, I worked on refactoring both ruma-api and ruma-events. After moving some of the larger chunks of the ruma_api_macro::api::Api::to_tokens method to helper functions, I spent time removing repetition from the Request/Response code generated by the ruma_api! macro. For ruma-events, the input parsing was changed to only allow valid names for the Any*Event enums. Altering the input parsing had the added benefit of replacing all of the string comparison and manipulation with strongly typed comparison and manipulation.
The final few issues to be resolved before the next crates.io release for ruma-events can happen are related to redacted events. Support for redacted events was added to the Any*Event enums, they now have redacted variants of each event kind. A few follow-up PR's have been merged to fully integrate redacted events into ruma-events, fixing specific event deserialization issues and splitting the UnsignedData struct into Unsigned and RedactedUnsigned.
I also investigated why so many sytests fail and created this issue: https://github.com/matrix-org/sytest/issues/913
Apparently Sytest did not expect a server to optimize their /sync responses as much as Conduit does :P.
Hot on the heels of shipping shardable federation readers last week, this week we shipped shardable federation senders. There is still plenty of work to do, but together these changes has made a massive difference to federation lag overall and hopefully those of you not on matrix.org are noticing the difference when you talk to matrix.org users.
This graph shows the impact to outbound federation lag.
Aside from that we also shipped shardable push and frontend proxy workers as both were starting to max out on CPU as well as a shardable client reader, allowing us to shard registration which was especially important this week :)
Finally we moved typing notifications from the master process and optimised incoming replication queuing to buy us a little more head room.
Next steps are to revisit where all the remaining cycles are going on the master process. To help us profile we are migrating to async/await semantics and Patrick produced this natty graph to track progress.
Outside of performance we shipped a bug fix to prevent large initial syncs taking out the synchrotrons. The admin api sprouted an end point to list room members (thanks awesome-michaeland the ability to reactivate previously deactivated users.
Coming up weโll dust off the notifications project which has been put on hold while Riot transmogrified into Element and weโll continue with chipping away at the master process.
And another bump of the K8s-optimized Synapse images, this time to 1.17.0
Synapse adoption graphs
We only featured this a month ago, but I want to call attention again to this awesome Synapse version adoption tracking project from Chris . I wanted to know how quickly Synapse 1.17.0 would be the most deployed version (~36 hours), and the answer is right there!
I started working on a Twitter DM bridge that uses the internal API instead of the complicated official one. It should be easier to set up than bridges using the official API, since there won't be a need to get API keys and forward webhooks. The main risk is that Twitter doesn't like people using the internal API and starts blocking users like Facebook does, but hopefully that doesn't happen.
PlayStore users: Element should come as a regular upgrade of the application Riot-Android, and the upgrade will update your previous account and data, without the need to log in again.
F-Droid users: Element is a new app, you have to install it and log in again.
RiotX: RiotX will disappear from the PlayStore (it was only a beta application). We will provide a very last update to inform users to install Element Android
Please report any issues at https://github.com/vector-im/riotX-android (which will probably be renamed soon), because Element Android is actually RiotX code!
Thanks for all the contributors of RiotX, we still have lots to do to make the app even better and full featured.
After seeing all the Element room avatars, I thought to myself, "you know what, #radical-webext:matrix.org needs a new icon"โฆ so I sat down and combined a couple of icons. The result:
Radical also quickly (as always) got updates when Element 1.7.0 and 1.7.1 were released. Huge thank you to stoic for making Radical in the first place.
I updated my F-Droid repos which are containing the dev builds to reflect the latest naming change to Element.
The people who formerly used my repos for getting the dev repo of Riot-Android should update to the new repo. It wont be updated to get Element
As always you can grab the F-Droid or GPlay flavor and you can pick the repo which reflects your flavor on https://fdroid.krombel.de
Note: As "the new repo" is internally the same repo as the one of RiotX (and Riot.imX) the people who already used that don't need to update their packet source. They will get the updates to Element via the old repo urls.
As promised last week, maubot now supports end-to-end encryption. It's a bit bare at the moment, so you need to manually insert the device ID into the database for it to start working, but other than that it works. Plugins don't need to be changed at all, the framework will handle everything.
While I was adding e2ee to maubot, I also improved mautrix-python's crypto stuff so that it's easier to use it directly as a client library with e2ee. Some day I might even add docs :D
Related to my previous update here, I've just published an initial version of a Matrix Client-Server client library for the Crystal programming language: https://github.com/cvincent/matrix-client-cr
Is this the first time we've seen tooling for Crystal?
Dept of Ops ๐
matrix-docker-ansible-deploy
This Ansible playbook is meant to easily let you run your own Matrix homeserver.
We have a few ways to handle the migration depending on how much change and breakage you're willing to tolerate. See our Migrating to Element documentation page.
Matrix-Alertmanager bot has a new release of v0.3.0. Highlights are compatibility with AWS Lambda, better error handling if message fails to send and the ability to do a @room mention on firing alerts. Find it here.
This week, we got support for javascript (nodejs) with !js. The code
complexity was considerably reduced, which should make adding new languages
much easier in the future. All Python sub-dependencies are now pinned, making
Noteworthy team (patrick and myself) have open sourced both components (spoke & hub) of Noteworthy (Matrix over Wireguard overlay networks -- https://github.com/decentralabs/noteworthy) we are actively welcoming testers/contributors and working with a handful of projects on incorporating our deployment model over in #noteworthy:tincan.community -- it is also the fastest way (less than a minute) to deploy your own home server! Also, working on incorporating Noteworthy into the popular ansible deployment repo to enable users who don't have access to a publicly accessible to deploy their own home servers.
Dept of Ping ๐
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
Rank
Hostname
Median MS
1
fairydust.space
390
2
exp.farm
555.5
3
matrix.vgorcum.com
768
4
tchncs.de
810.5
5
settgast.org
1026
6
moritzdietz.com
1046
7
ragon.xyz
1288
8
aragon.sh
1383
9
elcyb.org
1755.5
10
kapsi.fi
2034
That's all I know ๐
See you next week, and be sure to stop by #twim:matrix.org with your updates!
Hot on the heels of Synapse 1.16.1, 1.17.0 is a bug fix release most notably containing a fix for 'stuck invites' which happen when we are unable to reject a room invite received over federation.
Get the new releases from any of the usual sources mentioned at https://github.com/matrix-org/synapse/blob/master/INSTALL.md. 1.17.0 is on github here.
Changelog for 1.17.0 follows:
Synapse 1.17.0 (2020-07-13)
Synapse 1.17.0 is identical to 1.17.0rc1, with the addition of the fix that was included in 1.16.1.
Synapse 1.17.0rc1 (2020-07-09)
Bugfixes
Fix inconsistent handling of upper and lower case in email addresses when used as identifiers for login, etc. Contributed by @dklimpel. (#7021)
Fix "Tried to close a non-active scope!" error messages when opentracing is enabled. (#7732)
Fix incorrect error message when database CTYPE was set incorrectly. (#7760)
Fix to not ignore set_tweak actions in Push Rules that have no value, as permitted by the specification. (#7766)
Fix synctl to handle empty config files correctly. Contributed by @kotovalexarian. (#7779)
Fixes a long standing bug in worker mode where worker information was saved in the devices table instead of the original IP address and user agent. (#7797)
Fix 'stuck invites' which happen when we are unable to reject a room invite received over federation. (#7804, #7809, #7810)
Updates to the Docker image
Include libwebp in the Docker file to properly handle webp image uploads. (#7791)
Improved Documentation
Improve the documentation of the non-standard JSON web token login type. (#7776)
Update doc links for caddy. Contributed by Nicolai Sรธborg. (#7789)
Internal Changes
Refactor getting replication updates from database. (#7740)
Send push notifications with a high or low priority depending upon whether they may generate user-observable effects. (#7765)
Use symbolic names for replication stream names. (#7768)
Add early returns to _check_for_soft_fail. (#7769)
Fix up synapse.handlers.federation to pass mypy. (#7770)
Convert the appserver handler to async/await. (#7775)
Allow to use higher versions of prometheus_client <0.9.0 which are expected to introduce no breaking changes. Contributed by Oliver Kurz. (#7780)
Update linting scripts and codebase to be compatible with isort v5. (#7786)
Ania M. Piotrowska (@aniampiotrowska), known for her work on Nym, Loopix, and at UCL presented "Building private future for the internet with the Nym mixnet".
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
Note that a majority of those new MSCs are split out from MSC1849 (the aggregations MSC) to make it easier to review ๐
Spec Core Team
In terms of Spec Core Team MSC focus for this week, we're sticking to implementation work. anoa did have some time this week to make the graphs more useful though:
work is still being done on the bot's functionality to test different crypto functionalities of other Matrix clients when thrown in a room with them
slight improvements to mautrix-go (sql store can now store multiple Olm accounts and their sessions, key rotation params taken from encryption event, other minor things)
add some simple instructions to the readme for enabling e2ee for neb as well as integrate these mautrix changes (PR #330)
room_key_requests will probably be the next thing to be developed for mautrix-go
Nheko (GSoC)
Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE (with the notable exception being device verification for now) and intends to be full featured and nice to look at
Over the last couple of weeks I have been working on the next P2P demo, using Yggdrasil as the transport instead of libp2p. Although libp2p is arguably more featureful in many ways, Yggdrasil does have one major benefit in that it provides full overlay routing. Participants in the network can carry traffic on behalf of other participants, resulting in something much closer to a true mesh network.
The mad science doesn't end there however. I've also built a custom Riot iOS build that includes the full Dendrite P2P demo (yes, that's a homeserver running locally on your phone). If you have an iPhone or iPad, it's available in public TestFlight right now and you can play with it!
The demo still is very experimental and has a number of bugs still (including but not limited to messages occasionally taking a while to deliver, the app crashing when going into the background sometimes etc.) but we'll be improving it further in the coming weeks and it's quite fun to play with, particularly if you can do so with other people nearby.
Thanks to the same technology that powers AirDrop, the demo will automatically find and connect to other nearby devices running the demo and build up a network automatically - even if you are not on the same Wi-Fi network (or indeed connected to Wi-Fi at all)! If you don't have any other nearby peers to test with then fear not - you can still configure a public static peer in the app settings and join the wider network that way.
With any luck the room directory will even work too, showing published rooms from other nearby devices. If not, try joining #beachparty:b5ae50589e50991dd9dd7d59c5c5f7a4521e8da5b603b7f57076272abc58b374 from within the app.
Questions, comments or feedback? Join us in #p2p:matrix.org!
Many of you know that I work on a Go homeserver called Dendrite, and so I've had to get familiar with SyTest: a black-box homeserver integration testing project. Unfortunately SyTest has a number of problems: from the dialect of perl, lack of documentation for federation bits, the inability to run a single test, and so on. Having a solid black-box integration testing project is crucial for the ecosystem to ensure spec compliance (and hence compatibility between server implementations) and for making all servers more stable and reliable.
As a side project I've been working on a modern Go rewrite called Complement which is now ready for a bit more exposure. It currently only has a handful of SyTests converted but I hope to rapidly expand the number in the coming weeks. Complement makes heavy use of Docker to agnostically run homeservers, and already includes a Dockerfile for configuring a Dendrite instance. To try it out, clone the repo and run:
(cd dockerfiles && docker build -t complement-dendrite -f Dendrite.Dockerfile .)
COMPLEMENT_BASE_IMAGE=complement-dendrite:latest go test -v ./tests
If you're interested in learning more and maybe adding some tests check out ONBOARDING.md.
My overall hope is that Complement will lower the barrier to entry when adding tests by so much that it encourages any bugfix/feature in any homeserver implementation to result in a new test. This will benefit everyone and create a feedback loop which will make Matrix even more reliable.
Asked about Sytest and SyTS, Kegan added:
Complement currently implements the same tests as sytest, so tests which pass sytest will pass complement (though that will diverge as complement will guard against more race conditions by default). There's about 9 different kinds of tests currently testing CS-API and Federation (outbound to a dummy server and inbound via a federation client)
The problem with SyTS was that the test tooling around it ultimately wasn't expressive enough. I was using Jest and kept finding myself fighting it by bumping up against issues like https://github.com/facebook/jest/pull/8751 and the multi-process parallelisation stuff was too opaque. The assertion helper functions also lacked context because they rely on behavioural testing names like describe("foo") { it("should say bar") { ... } etc so when your assertion fails (eg wrong value for a JSON key using assert equals) it just says unhelpfully "foo != bar" where what I really want is to know what it actually does say then, and other contextual info around the object (maybe the key name was typod). When I realised that the assertion lib didn't have this and I'd need to add it, coupled with parallelisation concerns and head-desking against Jest, I found myself wishing I just used something else. I stuck with it though and then realised that all the federation stuff (signature checking, canonical JSON, etc) would need to re-implemented when I knew I had working code in Go. Both of these things combined and I thought "you know what, I'm just going to do this in Go" but you'll note that the architecture is identical, so SyTS lives on as an early prototype for Complement ;)
Hi everyone, here are some things we worked on this week:
Work on to-device sending improvements
Work on room tags (thanks to @gnieto)
Merged /logout/all endpoint (thanks to @CapsizeGlimmer)
Work on /joined_members (thanks to @CapsizeGlimmer)
Last week we had a list of major features still missing. Here's a list of what already works:
Registering, logging in, creating rooms, room visibility, join rules, basic permission management, public room list, inviting, creating DMs, e2e encryption, key backups, device verification, cross signing, notifications, uploading media/files (also user/room avatars), lower-resolution media thumbnails, voip calls and a few other thing's I'm forgetting right now.
So it's useable for non-federating chats already.
Thanks to everyone who supports me on Liberapay or Bitcoin!
In Synapse land this week, we shipped 1.16.0 and err 1.16.1.
Lifting from the blog post, the highlights are:-
An important performance fix to improve room state resolution.
An option to enable e2e by default for new rooms.
Ability to run multiple media repo workers side by side.
Ability to mark specific content as being safe from quarantine.
Bug fixes to make migrating from SQLite to Postgres more reliable - if you are running sqlite for anything other than evaluation purposes then please migrate!
Continuing our matrix.org performance theme, having now got the CS API largely into a good state of responsiveness, we are looking at federation lag. Today we shipped a sharded Federation Reader to matrix.org which has reduced the average lag from seconds to milliseconds. We are just running two currently and are still tuning as we figure out how to get the best from it. Initial impressions seem promising. Watch this space.
We also have a PR out for review to shard the Federation Sender, which will have much the same effect in the other direction.
Once we have sending messages via the Client-Server API, much reduced federation lag and a sharded Pusher weโll take a look at room joins.
Hi everyone, my friend Patrick and I have been working on making Matrix more accessible in the context of personal overlay networks powered by WireGuard. Weโve built a 1-click deployment solution for Synapse & Riot based on docker. The interesting part is that it gives you public addressability via a distributed proxy service that we have been developing. We are free and open source and welcoming contributors and testers ASAP! We are imagining an ecosystem for building collaborative intranets and see them as foundational to a freer, more equitable internet. Please join us at #noteworthy:tincan.community See https://www.patrickdlg.com/personal-messaging/ and https://noteworthy.tech/start/
I fixed some bugs in matrix-sms-bridge and added a feature that allows delayed sending of sms messages with the sms send command. It runs very fast and stable in my production environment.
Dept of Clients ๐ฑ
Nheko
Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE (with the notable exception being device verification for now) and intends to be full featured and nice to look at
Not much to talk about currently, since I'm breaking my Nheko to replace the entire event store at this moment, but I have some heads up on an annoying bug that I fixed on my branch:
Sometimes we had a weird issue, where Nheko wouldn't load the language the user set on their system. This was especially annoying when I wanted to test a specific language, like Japanese, and I couldn't figure out the right LANG* variables to set the language to Japanese in Nheko. Turns out, this was a bug in Nheko and you wouldn't have guessed so, when looking at the examples in the Qt documentation. Anyway, KDAB wrote a nice blogpost explaining this issue: https://www.kdab.com/fixing-a-common-antipattern-when-loading-translations-in-qt/
If you are a Qt developer, you may want to check your applications, if you are also affected.
We will release a beta (0.91.5) of Riot-Android this evening, including all the recent changes and lots of fixes. We are still working to prepare the great release!
When I initially implemented end-to-bridge encryption in mautrix-python, I used matrix-nio in a slightly hacky way to get it done quickly. It worked fine in some cases, but also caused some undecryptable messages. I tried switching to using matrix-nio's crypto module directly, but the sans-I/O design made that difficult: there were too many parts that needed to be hooked up to the actual I/O and I wasn't able to find them all. In the end, I just decided to implement the basic e2ee stuff directly in mautrix-python the same way it's implemented in mautrix-go.
All my bridges based on mautrix-python (Telegram, Facebook and Hangouts) have been switched to use the new crypto stuff. After a few initial bugs that are already fixed, it seems to work better than the old system. I'll probably add native e2ee to maubot soon too, which is my last main project that's still missing e2ee.
Version 0.3 brings the ability to use room aliases instead of internal room ids.
Furthermore, you can configure it to accept invites and follow room upgrades.
Due to utter negligence and malpractice by myself, we missed the update for this project last week. Of course, 0.2 is now superseded by 0.3, but here is the 0.2 update anyway:
I wrote a pretty simple Matrix bot in Rust which is loosely based on Cat Disruptor 6000. It will react with ๐๏ธ to any message containing the string "cat" *. Since it is a small project, I hope it can be used as an example for other people interested in writing Matrix bots in Rust. It implements several important features (crypto/device store, auto-accepting invites, crafting custom events, etc.) You can host it yourself, or you can try it out by inviting the user "@catdisruptor:m.scd31.com" to your room! https://git.scd31.com/stephen/cat-disruptor-7000
* Cat Disruptor 6000 does not do this (it is used for disrupting monologues with cat pictures). However, many instances of Cat Disruptor 6000 also include a separate bot, which is what Cat Disruptor 7000 is mimicking.
Heya, we (famedly) are looking for about 2-3 more flutter developers. We are a german startup which works on revolutionizing communication in the medical area. We build on top of matrix, so having some matrix-knowledge would be very beneficial.
While we are based in Berlin we do allow remote work, and communication in english is fine, too.
If you are interested or have more questions, please message Niklas Zender.
Dept of Ping ๐
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
Rank
Hostname
Median MS
1
fairydust.space
334
2
matrix.vgorcum.com
637.5
3
lossy.network
657
4
asra.gr
795
5
swag.industries
1407.5
6
lo.hn
1481
7
finallycoffee.eu
1804
8
nzbr.de
1885.5
9
halogen.city
2275
10
utzutzutz.net
2462.5
That's all I know ๐
See you next week, and be sure to stop by #twim:matrix.org with your updates!
An important performance fix to improve room state resolution.
An option to enable e2e by default for new rooms.
Ability to run multiple media repo workers side by side.
Ability to mark specific content as being safe from quarantine.
Bug fixes to make migrating from SQLite to Postgres more reliable - if you are running sqlite for anything other than evaluation purposes then please migrate!
Note, we have deprecated the m.login.jwt login method in favour of org.matrix.login.jwt see the changelog for more details.
Enjoy!
Get the new releases from any of the usual sources mentioned at https://github.com/matrix-org/synapse/blob/master/INSTALL.md. 1.16.0 is on github here.
Changelog for 1.16.0 follows:
Synapse 1.16.0 (2020-07-08)
No significant changes since 1.16.0rc2.
Note that this release deprecates the m.login.jwt login method, renaming it to org.matrix.login.jwt, as m.login.jwt is not part of the Matrix spec. Otherwise the behaviour is identical. Synapse will accept both names for now, but this may change in a future release.
Synapse 1.16.0rc2 (2020-07-02)
Synapse 1.16.0rc2 includes the security fixes released with Synapse 1.15.2.
Please see the 1.15.2 release notes for more details.
Improved Documentation
Update postgres image in example docker-compose.yaml to tag 12-alpine. (#7696)
Internal Changes
Add some metrics for inbound and outbound federation latencies: synapse_federation_server_pdu_process_time and synapse_event_processing_lag_by_event. (#7771)
Synapse 1.16.0rc1 (2020-07-01)
Features
Add an option to enable encryption by default for new rooms. (#7639)
Add support for running multiple media repository workers. See docs/workers.md for instructions. (#7706)
Media can now be marked as safe from quarantined. (#7718)
Expand the configuration options for auto-join rooms. (#7763)
Bugfixes
Remove user_id from the response to GET /_matrix/client/r0/presence/{userId}/status to match the specification. (#7606)
In worker mode, ensure that replicated data has not already been received. (#7648)
Fix intermittent exception during startup, introduced in Synapse 1.14.0. (#7663)
Include a user-agent for federation and well-known requests. (#7677)
Accept the proper field (phone) for the m.id.phone identifier type. The legacy field of number is still accepted as a fallback. Bug introduced in v0.20.0. (#7687)
Fix "Starting db txn 'get_completed_ui_auth_stages' from sentinel context" warning. The bug was introduced in 1.13.0. (#7688)
Compare the URI and method during user interactive authentication (instead of the URI twice). Bug introduced in 1.13.0. (#7689)
Fix a long standing bug where the response to the GET room_keys/version endpoint had the incorrect type for the etag field. (#7691)
Fix logged error during device resync in opentracing. Broke in v1.14.0. (#7698)
Do not break push rule evaluation when receiving an event with a non-string body. This is a long-standing bug. (#7701)
Fixs a long standing bug which resulted in an exception: "TypeError: argument of type 'ObservableDeferred' is not iterable". (#7708)
The synapse_port_db script no longer fails when the ui_auth_sessions table is non-empty. This bug has existed since v1.13.0. (#7711)
Synapse will now fetch media from the proper specified URL (using the r0 prefix instead of the unspecified v1). (#7714)
Fix the tables ignored by synapse_port_db to be in sync the current database schema. (#7717)
Fix missing Content-Length on HTTP responses from the metrics handler. (#7730)
Fix large state resolutions from stalling Synapse for seconds at a time. (#7735, #7746)
Improved Documentation
Spelling correction in sample_config.yaml. (#7652)
Added instructions for how to use Keycloak via OpenID Connect to authenticate with Synapse. (#7659)
Move flake8 to the end of scripts-dev/lint.sh as it takes the longest and could cause the script to exit early. (#7738)
Explain the "test" conditional requirement for dependencies is not all of the modules necessary to run the unit tests. (#7751)
Add some metrics for inbound and outbound federation latencies: synapse_federation_server_pdu_process_time and synapse_event_processing_lag_by_event. (#7755)
Ania M. Piotrowska (@aniampiotrowska), known for her work on Nym, Loopix, and at UCL will present "Building private future for the internet with the Nym mixnet".
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
e2ee merged and some bugs fixed (pr #324)! You should be able to use neb normally in an encrypted room now, if that's not the case please let us know!
Added integration tests for both encrypted and unencrypted rooms to neb in order to catch things that might break some core functionality in a subtle way (issue #326)
Similarly, created a PR for mautrix-go for testing the Olm and Megolm session establishment and message encryption / decryption (pr #12)
Next up, working on a service for neb for testing the crypto functionality of other clients: neb will send messages to a room, rotate keys etc. and your client is supposed to respond!
Dept of Servers ๐ข
Dendrite / gomatrixserverlib
Dendrite is a next-generation homeserver written in Go
This week has seen some new architectural changes and a number of bugs fixed:
A new Current State Server component has been added, which tracks room state
The Public Rooms API has been deprecated in favour of the Current State Server
Event size limits are now honoured with the correct return code
Invites have seen some refactoring and retiring invites should work better now
The federation sender now persists unsent PDUs to the database and will automatically retry when Dendrite is restarted, making resend behaviour much more reliable (particularly in the P2P demos)
Some room checks in /state and /state_ids have been fixed
Some additional restrictions have been added to /send and we now return the correct error codes
A bug where current state was incorrectly used when getting missing/backfill events has been fixed
A bug where you couldn't rejoin a room that you created after wiping your database has been fixed
Synapse 1.15.2 shipped this week with some important security fixes - if you haven't already upgraded, please do asap! Meanwhile, we're also preparing a 1.16 release with the normal round of bugfixes and incremental improvements.
The big focus in the Synapse team continues to be on performance for larger Synapse instances, which is reflected in the support in 1.16 for running multiple media worker processes. We're working on giving other worker processes the same treatment, so expect more of the same in the weeks to come!
The avhost image including coturn, jemalloc, and mjolnir for synapse 1.15.2 was released at avhost/docker-matrix:v1.15.2, while the RCs images were released on my personal repo; mvgorcum/docker-matrix:v1.16.0rc1 and mvgorcum/docker-matrix:v1.16.0rc2
I wrote a very simple Twilio & Maubot based SMS bridge recently which I've been running for long enough with no delivery errors that I'd consider it stable. It's oriented towards bridging many individuals to single rooms via SMS. You'll need to have a Maubot instance in order to run it.
There has been a lot of under the hood work that users wonโt notice and which we wonโt delve into. In addition to them, we have a few nice things:
Matrix API tokens are redacted in logs for better privacy when submitting bug reports.
A send button was added next the message entry. This is especially relevant for touch screens.
We are less noisy with typing notifications that we send out.
We support SOCKS proxies.
Notifications for non text messages have been tweaked.
We gained support for edited message display. They are replaced with the newer content and a little icon is shown next to them. Editing messages is not supported yet though.
Fixed a ton of reliability bugs in the 7 bridges we support (changelog)
We are hiring (full-time or part-time) remote React developers to work on our (source available, still TBD on final license) Riot fork Send me a DM if interested.
RiotX is now published in the beta channel of the Riot-Android app: https://play.google.com/store/apps/details?id=im.vector.app. You can get the last beta version by becoming a tester if you want to. There will be no more update of RiotX on the PlayStore. Current RiotX users should signout from RiotX and signin again on Riot-Android. Riot-Android users do not have to do that. The changelog can be read here https://github.com/vector-im/riotX-android/releases/tag/v0.91.3-beta. And this week we are working on implementing what's remain on our list, and we are fixing as many bugs as we can before the great release.
Hello, I am announcing my first matrix bot - cody - that will evaluate code snippets in your Matrix rooms and return the result. Here is an example interaction:
This initial release only supports Python, but the plan is to add support for more languages in the next few weeks. The repo might also be of interest to some of you as it has CI, CD to DockerHub, complete dev environment with cody, pantalaimon, synapse and riot using docker-compose, simple metrics and Grafana-dashboards-as-code.
New version of logootish-js with a completely new algorithm
In particular, I should note that this new algorithm is tested extensively, so I think the bugs should be out
Corresponding updates to internal event structure
Fixed UI & CSS bugs
Improved UX for room title editor
Currently, conflicting text will be displayed, but conflicts will not be indicated in any way
I almost forgot to mention: The room list is now filtered using the typed rooms MSC (event ID org.matrix.msc1840). #test-document0:kb1rd.net is a room with that state event set.
I also made a flame graph of the internal algorithm and I'm surprised by some of the results. The good thing is that I think there's quite a few ways to speed up the algorithm. Here it is:
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
Folks, today we are releasing Synapse 1.15.2, which is a security release which contains fixes to two separate problems. We are also putting out the second release candidate for the forthcoming Synapse 1.16, including the same fixes.
Firstly, we have fixed a bug in the implementation of the room state resolution algorithm which could cause users to be unexpectedly ejected from rooms (Synapse issue #7742).
Secondly, we have improved the security of pages served as part of the Single-Sign-on login flows to prevent clickjacking attacks. Thank you to Quentin Gliech for reporting this.
We are not aware of either of these vulnerabilities being exploited in the wild, but we recommend that administrators upgrade as soon as possible. Those on Synapse 1.15.1 or earlier should upgrade to Synapse 1.15.2, while those who have already upgraded to Synapse 1.16.0rc1 should upgrade to 1.16.0rc2.
Get the new releases from any of the usual sources mentioned at https://github.com/matrix-org/synapse/blob/master/INSTALL.md. 1.15.2 is on github here, and 1.16.0rc2 is here.
Changelog for 1.15.2 follows:
Synapse 1.15.2 (2020-07-02)
Due to the two security issues highlighted below, server administrators are
encouraged to update Synapse. We are not aware of these vulnerabilities being
exploited in the wild.
Security advisory
A malicious homeserver could force Synapse to reset the state in a room to a
small subset of the correct state. This affects all Synapse deployments which
federate with untrusted servers. (96e9afe6)
HTML pages served via Synapse were vulnerable to clickjacking attacks. This
predominantly affects homeservers with single-sign-on enabled, but all server
administrators are encouraged to upgrade. (ea26e9a9)
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
In terms of Spec Core Team MSC focus for this week, unfortunately our three MSCs from last week ( MSC2366 (verification flows), MSC2403 (knocking), and MSC2630 (SAS security)) are still pre-FCP. Most of the team has been quite busy with implementation for the past few weeks. Instead of advertising those 3 MSCs again, we're going to switch the focus to "implementation" for a bit until we're freed up again.
So the focus for this week is: MSC implementation work. However, this should not discourage any MSC authors from responding to MSC feedback in the meantime ๐
This week in the ruma/matrix Google Summer of Code project, ruma-events was made ready for use! After adding stripped and sync event generation to the event_enum! macro there were only a few small tweaks needed to try it out in some dependent rust crates. I spent a few days converting matrix-rust-sdk to use the ruma-monorepo. Since ruma is used on both client and server-side, I also opened a PR to update Conduit, a homeserver implementation written in Rust. To test that everything worked together, I updated rumatui, my command-line client written in rust.
Then I could test that Conduit sent, and matrix-rust-sdk received the new ruma events successfully. While updating, I felt the pain of not having accessor methods for the Any*Event enums to get at the event fields held within. I have opened a pull request to add the generation of these methods to the event_enums! macro. Hopefully, the Conduit and matrix-sdk PR's will be merged and the ruma monorepo can be tested in the wild!
Moved matrix protocol, irc protocol, and bridge module to futures 0.3. Converted some utility functions to new futures, and updated the http implementation to use the standardized Hyper library instead of using a custom http implementation.
Created PR for enabling e2ee across all services! This required a few changes to how the bot client is initialised, as well as changes to all services to use the new functionality instead of directly sending messages to a room. https://github.com/matrix-org/go-neb/pull/324
Added code from another of tulir's projects to Mautrix to allow storing the crypto material (olm / megolm sessions, accounts etc.) in a SQL database, adding a second way besides using Gob storage. https://github.com/tulir/mautrix-go/pull/10
Future plans are to work on the library itself to add any features that might be missing.
Using the same design as Riot. However, I am just using flexbox and text instead of drawing on an HTML5 canvas. The HSL color is generated by hashing the user ID.
Quoting and replying
Added on-hover buttons and reply popup above message composer similar to Riot web. Quoting prepends the message (quoted) to the message composer similar to Riot web and replying would also work similar to Riot web. It will be fully functional once markdown parsing is added.
Guest access
The client now supports guest access. If no access token or user ID is provided in the configuration file, the client attempts to register a guest account on the home server. Room contents are viewed using peekInRoom.
If the guest attempts to send a message, only then joinRoom is called to avoid spam (each page load would lead to a guest joining the room).
This week has been mainly about testing to ensure that all the progress over the past few months is kept up-to-date and correct. We've also added a few features in order to get certain sytests working. To that end:
Invites can now be declined over federation and they will be reflected in /sync responses.
Errors encountered when joining a room over federation are now sent back to the client.
Errors encountered when accepting an invite over federation are now sent back to the client.
Dendrite will now check server names meet the server name grammar in the specification.
A bug which caused client-api-proxy to not actually proxy correctly has been fixed, thanks @fantashley !
/send now abides by the limits in the specification: 50 PDUs / 100 EDUs.
The docker-compose scripts now include appservice_api, thanks @fantashley !
Sending invites over federation will now fall back to v1 if v2 fails with a 404.
Dendrite now implements room.timeline.limit completely (in both in-line and stored filter formats).
Dendrite now sets the limited flag on /sync responses correctly.
In addition, we now have support for collecting code coverage output from SyTest. This indicates we are testing roughly 70% of the Dendrite codebase. The remaining 30% are hard to reach via integration tests (e.g database failures, communication problems between internal APIs).
Spec compliance:
Client-Server APIs: 45%, up from 40% last week
Server-Server APIs: 50%, up from 38% last week
In total, we've made an additional 45 sytests pass this week.
This week weโve been working on further improvements to event persistence and ironed out a nasty bug where an unusually long state resolution could block the reactor overall and impact send times. We seem to have got to the bottom of this and m.org has improved a lot as of today.
More generally weโve been trying to characterise matrix.org performance so that we can continue to improve over the coming months.
We are going to focus on:-
Client send event
Outbound Federation Latency
Inbound Federation Latency
Room joins
As well as tracking the CS API generally.
We are using apdex with a satisfied limit of 250 ms and a tolerating limit of 1000 ms. By the end of the Summer we will aim to hit an apdex score of 0.9 for each area.
For instance here is Federation Send Event Apdex graphed overtime. You can see that we are averaging about ~0.8 currently, so plenty to keeping us busy!
Next up will be to work on Outbound Federation Latency
Aside from that weโve been working hard on the upcoming Notifications improvements. Mapping the push rules to the demands of the UI has been challenging and weโve been through several iterations. If youโd like to learn more take a look at Michael and Richโs explanation in this weekโs Matrix Live. Richโs presentation has Sheltie pictures #justsayinโ
The Telegram bridge now supports logging in by scanning a QR code, although it requires using the master branch of Telethon instead of a release. I also fixed bridging captions in file messages, so they're now bridged as separate messages like with images
Hey all, I've got several releases to talk about this week!
First, matrix-appservice-slack has been updated to 1.4.0 with several quality of life changes such as automatically setting the bot profile on startup, supporting logging out of slack accounts and adding a health checkpoint. There are also quite a few bugfixes so make sure you update.
matrix-appservice-irc was also bumped to 0.18.0 with the headline features being Node 14.x support.
We've also shipped 0.19.0-rc1 which has stopped support for Node 10.x. Why? Because we're adding worker support to the bridge! This release starts to make use of the new(ish) worker_threads feature so that we can dish out processing to separate threads (running their own UV event loops, for node enthusiasts). The first thing to be workerized is metrics, so that metrics may still be reported should the bridge become saturated, but we plan to split out more work as things progress.
Finally matrix-appservice-bridge got a few fixes to support our new worker land, as well as being updated to support matrix-js-sdk 6.0.0. You can checkout the changes for 1.13.1 here.
Fluffychat Version 0.15.0 is released, and should be available in the Play Store, on F-Droid and in IOS Testflight soon! This makes Fluffychat the first non-Riot matrix client that supports Cross-Signing.
Features:
New room list app bar design
Chat app bar transparent
Implement web file picker
Minor design and UX improvements
Implement Cross Signing
Restore keys from online key backup
Added translations: Czech, Spanish, Slovakian
Changes:
Show presences of users sharing a direct chat
Big refactoring
Fixes:
Various fixes, including e2ee fixes and olm session recovery
Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE (with the notable exception being device verification for now) and intends to be full featured and nice to look at
Most of our time this week was spent trying out device verification with Chethan. It's fun and I could finally file bugs in Nheko against someone else!
We fixed some issues that should hopefully make text in the timeline less blurry again.
Fixed a bug which cause some clients like fluffychat to break in E2EE rooms (we accidentally sent a null relation when not replying...)
Fixed an issue, where Nheko didn't verify the format of html formatted messages correctly, causing it to render messages in a way which wasn't compliant to the spec.
Thanks to the work of a Pirate and his friends, current versions of Nheko should be available in backports for Debian Buster once again!
Riot Chat for Nextcloud 0.5.0 The new version updated the Riot.im version to 1.6.6 and added the ability for admins to set their own custom config for Riot rather then using the settings interface in addition to a few bug fixes.
Riot Chat for Nextcloud allows individuals and organizations with a Nextcloud instance to easily set up and configure their own Riot instance with just a few clicks on a web interface without the need to write a config file.
This week, we completed UX for cross-signing and secure backup. We made associated settings but we still need to polish them.
Sygnal and the new push extension have been updated to match Apple requirement and our privacy concerns. Events content are no more sent anymore.
We started to implement the new room notifications settings UI.
We will publish at the beginning of next week a beta version of the migration to RiotX codebase on the beta channel of the PlayStore, to be able to ensure the migration works fine, before release it to production.
joepie91 discovered this project, presented yesterday at Conference in the Cloud:
Net::Matrix::Webhook implements a webhook, so you can easily post messages to your matrix chat rooms via HTTP requests. It uses IO::Async to start a web server and connect as a client to matrix. It will then forward your messages.
Soru wrote a quick program that scrapes riot-web, riot-x and riot-ios for translations of the emoji names for emoji verification and combines them all into an easily-readable json file, so that other client developers can use it. Since all three riot versions have a different set of translations, it might also be helpful for them. You can find the source code along with the outputted json files here.
Like the other bots, you can self-host it, use my instance (@satw:maunium.net) in your own room or simply join #satwcomic:maunium.net to automatically get the latest comics in Matrix.
A bot that will allow to room administrators and moderators to generate room custom commands (in a similar way how telegram commands are thought).
When they are invoked it will post the message event object data and a predefined context object along with a token to a custom url. More information in: https://gitlab.com/communia/matrix-webhooker
Drupal module to receive links shared from a matrix room. Get links provided by a matrix bot in a room.
With this module one can get media from a matrix own bot to any drupal site. Once installed and bot is created through matrix-webhooker bot.
A blog entry about it (in spanish) in https://planet.communia.org/content/enlaces-desde-matrix
Dept of Interesting Projects ๐ฐ
Pollvis - new poll visualiser project
We have a couple of conferences coming up who are planning to use Matrix as part of their offering, much as we do with Open Tech Will Save Us. To help add some more features, I created an MVP "poll visualiser", which watches a room and works in tandem with the poll-bot from Brendan Abolivier . This project is still at the beginning, but might be interesting to some! Find the code at https://github.com/benparsons/pollvis.
Dept of Ping ๐
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
Rank
Hostname
Median MS
1
fairydust.space
331
2
services.pyrahex.com
537
3
heitkoetter.net
604
4
nitro.chat
646.5
5
mchus.pro
660
6
privacytools.io
671
7
matrix.vgorcum.com
727.5
8
eiselecloud.de
740
9
aruiz.io
855
10
neko.dev
923.5
That's all I know ๐
See you next week, and be sure to stop by #twim:matrix.org with your updates!
Are you emerging from lockdown? Dazed and wondering what to wear? Need some apparel which says "I'm cool and I have great taste in decentralised communications protocols"? Then you should visit The Matrix Merch Store!
T-Shirts, Hoodies, Zipped Hoodies and Stickers available now!
Matrix Decomposition: Analysis of an Access Control Approach on Transaction-based DAGs without Finality
We published another scientific paper on Matrix, this time it is called Matrix Decomposition: Analysis of an Access Control Approach on Transaction-based DAGs without Finality!
Its main topic is the question of how access control can work based on a structure that only provides a partial order on events and no consensus on the current state, i.e. the Matrix Event Graph. We found some concrete, non-critical security issues related to both incorrect specification as well as divergent homeserver behaviour. While the last remaining mitigations found their way into Synapse 1.14.0, we provide ideas on structural solutions to avoid both problem classes in the future using formal verification.
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
In terms of Spec Core Team MSC focus for this week, the team has been spread a bit thin with other work. As such, we're still on the same three MSCs: MSC2366 (verification flows), MSC2403 (knocking), and MSC2630 (SAS security). We're hoping this week will be more productive in terms of spec.
Dept of Servers ๐ข
Dendrite / gomatrixserverlib
Dendrite is a next-generation homeserver written in Go
This week has seen a lot of work put into fixing broken tests and refactoring parts of the codebase.
The media API has had quite a bit of attention this week and now media over federation works properly, as well as much better filename handling
A new user API component is now responsible for user accounts, devices, access tokens and account data
The server key API has been updated with more reliable validity checking and new tests
Logging into Dendrite using Riot iOS is now supported thanks to some minor tweaks/fixes
URL handling for version 3 rooms has been improved
A bug in the format of /v1/send_join has been fixed
The /joined_members response has now been fixed
In addition to that, we've made good progress on embedding the Dendrite-powered Yggdrasil demo into Riot iOS, which we'll hopefully be demoing in a couple of weeks!
We released 1.15.1, this is a bug fix release, sorry if you were bitten.
Changelog in full
Bugfixes
Fix a bug introduced in v1.15.0 that would crash Synapse on start when using certain password auth providers.
Fix a bug introduced in v1.15.0 which meant that some 3PID management endpoints were not accessible on the correct URL.
Outside of that
Implemented unread message counts (MSC 2625) - This is part of the a more general notifications improvement project in conjunction with the Riot teams. The aim being to make Notifications easier to configure. Weโve also been working on push rule definitions for the default behaviour. More on that next week.
Have the ability to shard the media repository worker. This means we can now run an arbitrary number in parallel, thereby improving upload reliability and performance, we are running this on Matrix.org.
Re-enabled Jaeger on matrix.org - previously we needed to switch it off due to the performance overhead, we needed a few tweaks but now it is working well and helping us to determine the cause of slow requests. It seems like the HTTP requests between the event creator and event persister workers can sometimes be slow - we are not sure of the cause yet, we are working on it but it will mean further performance improvements to message sending.
Next up performance wise is working on sharding the federation sender to improve federation lag. Weโll also try replacing simplejson with stdlib json which seems to be much faster at dumps nowadays.
Finally, weโve fixed a bug that meant that quarantining media would from time to time include quarantining sticker packs
Other interesting bugs
We fixed a spec compliance bug in fetching remote media raised by our pals in Dendrite-land. We fixed a bug causing federation .well-known requests to fail due to not including the user-agent header. We removed references to six (thanks @ilmari) and finally, fixed a bug that caused stream id to go backwards on the replication stream.
Wondering how quick this weeks synapse release was adopted by homeserver admins? Head over to https://graph.settgast.org/ and see yourself that it only took a little over 2 days for 1.15.1 to become the most used synapse version. I plot hourly distribution of synapse versions there. (Disclaimer: Only collects stats for homeservers that my homeserver sees in any of its rooms, which are around 2000 at the moment)
This is super cool! I love that this has been created, and also how well it reflects on homeserver admins! /me scurries to update to 1.15.1...
This week I worked on server-side key backups and cross-signing - the second most requested features (federation being the first one). Key backups are already working! Take a look:
Cross-signing should be working, but there is a bug where emoji verification gets stuck. I hope we can find the mistakes and finish the PR next week.
Thanks to everyone who supports me on Liberapay or (new!) Bitcoin!
To make the bridge info state events a bit more useful, all my bridges now include the room name and avatar URL in them. The events are also updated whenever the name or avatar changes, and there's a new option to re-send the event to all existing portals so old portals would get it too.
RiotX: Version 0.22.0 has been released on Tuesday, it includes integration manager support, sending stickers, modifying power levels, and lots of other things! See https://github.com/vector-im/riotX-android/releases/tag/v0.22.0 for a complete list of the changes.
This week we've been working on audio and video calls (The PR is in review). We are also trying to improve the performance of the application. We are adding room settings, and we have started to work on the migration Riot-Android -> RiotX.
PS check out Valere trying the new features:
Arch Linux (AUR) package for Revolution - a Riot fork by ponies.im
After some slow progress, rumatui, the rust command line chat client, is usable by anyone. You no longer need an account or have previously joined rooms. Registration and room search have been implemented! Using left/right arrow keys from the login screen will toggle to registration. Once registered hitting the left arrow will bring up the room search window. After typing your search term hit Enter and select the room to join by pressing Ctrl-d.
For the brave who would like to give it a trycargo install --git <https://github.com/DevinR528/rumatui> --branch room-search.
Added since last update
Room search is now available
Public rooms can be joined from the room search window
A user can register from the new register window
This features complete User Interactive Authentication by opening a web browser
Message edits are shown
When markdown is part of the message they are properly formatted
Reactions display under the respective message
Redaction events are handled for reactions (emoji) and messages
"Basically usable" is a standard we can all aspire to for our projects and endeavours.
This week, we worked on the new UX for cross-signing and secure backup. In parallel, we continued to improve the new push implementation. We are very closed to complete the two features.
This week we had an off-cycle 1.6.5 release to fix registration on some homeservers when email validation is required. After that, we've made a 1.6.6 RC including:
Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE (with the notable exception being device verification for now) and intends to be full featured and nice to look at
We released 0.7.2, which was a bit more messy than we would have liked...
As with every release we are currently working through the new bug reports. :D
I started working on SSSS and online key backup support. For this I wrote my own base64 and base58 encoders. As a result of this we dropped libsodium as a dependency, which should make Nheko a bit easier to package or build. It's also fun to write base conversions, although 58 is a horrible base!
From the changelog:
Reactions
React to a message with an emoji! ๐
Reactions are shown below a message in a small bubble with a counter.
By clicking on that, others can add to the reaction count.
It may help you celebrating a new Nheko Release or react with a ๐ to a failed build to express your frustration.
This uses a new emoji picker. The picker will be improved in the near future (better scrolling, sections, favorites, recently used or similar) and then probably replace the current picker.
Support for tagging rooms [tag]
Assign custom tags to rooms from the context menu in the room list.
This allows filtering rooms via the group list. This puts you in a focus mode showing only the selected tags.
You can assign multiple tags to group rooms however you like.
SSO Login
With this you can now login on servers, that only provide SSO.
Just enter any mxid on the server. Nheko will figure out that you need to use SSO and redirect your browser to the login page.
Complete the login in your browser and Nheko should automatically log you in.
Presence
Shows online status of the people you are talking to.
You can define a custom status message to tell others what you are currently up to.
The status message appears next to the usernames in the timeline.
Your server needs to have presence enabled for this to work.
after a couple of setbacks that didn't let us to make the releases earlier, the Quotient project has finally made two new releases:
GTAD (the piece of code magically producing readable C++ code from a Swagger API description) has achieved version 0.7 (https://github.com/KitsuneRal/gtad/releases/tag/v0.7.0) adding a few tricks in order to make...
...libQuotient 0.6 beta2 (https://github.com/quotient-im/libQuotient/releases/tag/0.6-beta2) rely entirely on the upstream matrix-doc specification, rather than a soft fork closely following the main sources! From now on it's "upstream first", in a sense that the original matrix-doc will be used to build Quotient codebase. Let's see how often it breaks ;)
In other news from the last few weeks:
some housekeeping and deprecation work in the API has been done in preparation to getting User Interactive Authentication along the next (post-0.6) release cycle of libQuotient.
also thanks to the updated code generator, the CS API code has been optimised, consolidating more code in the header files and making data deserialisation lazy; this helped significantly reduce compilation times, and runtime performance also improved a bit.
the number of configurations tested by CI has been extended, allowing to chase down a few more bugs that managed to go under the radar before.
when used with new enough Qt, CBOR is used to cache data locally - entirely transparently for clients.
Expect more news in the coming weeks, including continued work on matrix: URI proposal and its implementation in Quotient.
This week in ruma-events' Google Summer of Code project, after trying out the new Any*Event enums matrix-rust-sdk, we found a few big flaws. There was no easy way to go from StateEvent<AnyStateEventContent> to StateEvent<SpecificEventContent>, the other issue was one could create a StateEvent with differing content and prev_content fields using the AnyStateEventContent enum. The 0.22 ruma-events will be similar to the existing API; each event type has a corresponding event enum variant.
There were a few minor fixes during the week also. Unknown field deserialization is fixed, allowing deserialization of a JSON blob that has extra fields which are ignored. Custom events are now present in the Any*EventContent enums, although now they have to be moved up to be included in Any*Event enums. Benchmarks for deserialization have been added and used to increase performance.
Polyjuice Client, a Matrix library for Elixir, had a new release. This release fixes syncs, which apparently were completely broken in the last release (whoops). The wizarding community also welcomes a new contributor, Pierre de Lacroix.
I heard that you people like IoT?! With the Mozilla WebThings Matrix Adapter your Raspberry Pi sends you messages about your home. Want a log of when the front door has been opened? Need a low battery alert for your IoT devices? With the Mozilla IoT gateway and this adapter, you can send these events to a Matrix room of your choice!
Ever since m.room.aliases was replaced with alt_aliases, there hasn't really been any way to find aliases on other servers, since most room admins don't bother finding and publishing alternate aliases. To help solve that problem, I made a maubot plugin to let users publish alternate aliases in rooms: https://github.com/maubot/altalias
By default it lets users publish aliases with the same localpart as any already published alias. If that behavior is sufficient, you can simply invite it to a room and give it permissions to send m.room.canonical_alias events. Alternatively, it can be configured to allow aliases that match specific regexes.
Once it's set up, users can create a local alias and use !altalias publish <their alias> to publish it. The bot will make sure the alias points to the right room, check that the localpart is allowed and then add it to alt_aliases.
I have it running at @alias:maunium.net and I've added it to all my rooms already. Other rooms are also free to use that instance.
As of https://github.com/NixOS/nixpkgs/pull/89327 a go-neb module was added to nixos and will soon be available on nixos-unstable and in the 20.09 release. You can use it to declaratively install and configure go-neb matrix bots on nixos. I use it for prometheus alertmanager alerts and it works really cool! Thx to hexa- who did most of the work on it.
opsdroid 0.19 has been released, not many matrix specific features in this release, just making it slightly easier to get a connector or database instance by name. However, this release paves the way for the merge of the pull request transitioning opsdroid to use the matrix-nio Python library, so the next release should be packed full of matrix stuff ๐
Dept of Ping ๐
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
Rank
Hostname
Median MS
1
fairydust.space
384.5
2
maescool.be
390
3
gottliebtfreitag.de
453
4
construct.grin.hu
610
5
aragon.sh
639.5
6
privacytools.io
670
7
matrix.vgorcum.com
734
8
envs.net
756
9
settgast.org
865
10
intothecyber.space
1005
That's all I know ๐
Thanks to Alexander and Nico for their help editing this edition.
See you next week, and be sure to stop by #twim:matrix.org with your updates!
