This Week in Matrix 2020-07-10

Matrix Live 🎙

This week it's Open Tech Will Save Us!

• Jay Graber (@arcalinea, @arcalinea:matrix.org), creator of Happening and all-around decentralisation fan gave an overview of the decentralized social media ecosystem.
• Ania M. Piotrowska (@aniampiotrowska), known for her work on Nym, Loopix, and at UCL presented "Building private future for the internet with the Nym mixnet".
• Burak Nehbit (@nehbit, @burak:tomesh.net) gave a deep dive into his work on Aether P2P.

Dept of Spec 📜

anoa reported:

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.

MSC Status

Merged MSCs:

• No MSCs were merged this week.

MSCs in Final Comment Period:

• MSC2630: checking public keys in SAS verification (merge)

New MSCs:

• MSC2677: Annotations and reactions

• MSC2676: Message editing

• MSC2675: Serverside aggregations of message relationships

• MSC2674: Event Relationships

• MSC2673: Notification Levels

• MSC2666: Get rooms in common with another user

Note that a majority of those new MSCs are split out from MSC1849 (the aggregations MSC) to make it easier to review 🙂

Spec Core Team

In terms of Spec Core Team MSC focus for this week, we're sticking to implementation work. anoa did have some time this week to make the graphs more useful though:

Check out that graph!

Dept of GSoC 🎓️

HTML embeddable client GSoC project

arnav-t told us:

This week for my HTML embeddable client GSoC project:

• Added full markdown support and made replying and quoting fully functional

• Added a lot more configuration options to the client, including a read-only mode

• Fixed some bugs

go-neb

nikofil said:

• work is still being done on the bot's functionality to test different crypto functionalities of other Matrix clients when thrown in a room with them

• slight improvements to mautrix-go (sql store can now store multiple Olm accounts and their sessions, key rotation params taken from encryption event, other minor things)

• add some simple instructions to the readme for enabling e2ee for neb as well as integrate these mautrix changes (PR #330)

• room_key_requests will probably be the next thing to be developed for mautrix-go

Nheko (GSoC)

Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE (with the notable exception being device verification for now) and intends to be full featured and nice to look at

Chethan said:

• Added user cache which stores the encrypted user keys and updates them on sync

• Added Verification Cache which stores verified devices of a user

• Showing Verification Status of a device on Userprofile

Dept of P2P 👥

Riot iOS P2P Demo

Neil Alexander announced:

Over the last couple of weeks I have been working on the next P2P demo, using Yggdrasil as the transport instead of libp2p. Although libp2p is arguably more featureful in many ways, Yggdrasil does have one major benefit in that it provides full overlay routing. Participants in the network can carry traffic on behalf of other participants, resulting in something much closer to a true mesh network.

The mad science doesn't end there however. I've also built a custom Riot iOS build that includes the full Dendrite P2P demo (yes, that's a homeserver running locally on your phone). If you have an iPhone or iPad, it's available in public TestFlight right now and you can play with it!

The demo still is very experimental and has a number of bugs still (including but not limited to messages occasionally taking a while to deliver, the app crashing when going into the background sometimes etc.) but we'll be improving it further in the coming weeks and it's quite fun to play with, particularly if you can do so with other people nearby.

Thanks to the same technology that powers AirDrop, the demo will automatically find and connect to other nearby devices running the demo and build up a network automatically - even if you are not on the same Wi-Fi network (or indeed connected to Wi-Fi at all)! If you don't have any other nearby peers to test with then fear not - you can still configure a public static peer in the app settings and join the wider network that way.

With any luck the room directory will even work too, showing published rooms from other nearby devices. If not, try joining #beachparty:b5ae50589e50991dd9dd7d59c5c5f7a4521e8da5b603b7f57076272abc58b374 from within the app.

Questions, comments or feedback? Join us in #p2p:matrix.org!

Dept of Servers 🏢

A new way to test homeservers - Complement

kegan reported:

Many of you know that I work on a Go homeserver called Dendrite, and so I've had to get familiar with SyTest: a black-box homeserver integration testing project. Unfortunately SyTest has a number of problems: from the dialect of perl, lack of documentation for federation bits, the inability to run a single test, and so on. Having a solid black-box integration testing project is crucial for the ecosystem to ensure spec compliance (and hence compatibility between server implementations) and for making all servers more stable and reliable.

As a side project I've been working on a modern Go rewrite called Complement which is now ready for a bit more exposure. It currently only has a handful of SyTests converted but I hope to rapidly expand the number in the coming weeks. Complement makes heavy use of Docker to agnostically run homeservers, and already includes a Dockerfile for configuring a Dendrite instance. To try it out, clone the repo and run:

(cd dockerfiles && docker build -t complement-dendrite -f Dendrite.Dockerfile .)

COMPLEMENT_BASE_IMAGE=complement-dendrite:latest go test -v ./tests

If you're interested in learning more and maybe adding some tests check out ONBOARDING.md.

My overall hope is that Complement will lower the barrier to entry when adding tests by so much that it encourages any bugfix/feature in any homeserver implementation to result in a new test. This will benefit everyone and create a feedback loop which will make Matrix even more reliable.

Asked about Sytest and SyTS, Kegan added:

Complement currently implements the same tests as sytest, so tests which pass sytest will pass complement (though that will diverge as complement will guard against more race conditions by default). There's about 9 different kinds of tests currently testing CS-API and Federation (outbound to a dummy server and inbound via a federation client)

The problem with SyTS was that the test tooling around it ultimately wasn't expressive enough. I was using Jest and kept finding myself fighting it by bumping up against issues like https://github.com/facebook/jest/pull/8751 and the multi-process parallelisation stuff was too opaque. The assertion helper functions also lacked context because they rely on behavioural testing names like describe("foo") { it("should say bar") { ... } etc so when your assertion fails (eg wrong value for a JSON key using assert equals) it just says unhelpfully "foo != bar" where what I really want is to know what it actually does say then, and other contextual info around the object (maybe the key name was typod). When I realised that the assertion lib didn't have this and I'd need to add it, coupled with parallelisation concerns and head-desking against Jest, I found myself wishing I just used something else. I stuck with it though and then realised that all the federation stuff (signature checking, canonical JSON, etc) would need to re-implemented when I knew I had working code in Go. Both of these things combined and I thought "you know what, I'm just going to do this in Go" but you'll note that the architecture is identical, so SyTS lives on as an early prototype for Complement ;)

Find out more in the room: #complement:matrix.org.

Dendrite / gomatrixserverlib

Dendrite is a next-generation homeserver written in Go

Neil Alexander told us:

While I've been busy distracted with the P2P demo, Kegan has been on a mission to implement some new features. Changes this week include:

• Redaction is now mostly implemented

• User-interactive authentication is now implemented

• Device lists in the client API are now implemented

• Media APIs are now available on both /r0 and /v1 endpoints which makes Riot iOS a bit happier

• Some federation sender bugs have been fixed

• Some database locks in the federation sender in SQLite mode have been fixed

Spec compliance has improved a bit:

• Client-Server APIs: 48%, up from 45% last week

• Server-Server APIs: 51%, up from 50% last week

Conduit

Conduit is a Matrix homeserver written in Rust https://conduit.rs

timo said:

Conduit

Hi everyone, here are some things we worked on this week:

• Work on to-device sending improvements

• Work on room tags (thanks to @gnieto)

• Merged /logout/all endpoint (thanks to @CapsizeGlimmer)

• Work on /joined_members (thanks to @CapsizeGlimmer)

Last week we had a list of major features still missing. Here's a list of what already works:

Registering, logging in, creating rooms, room visibility, join rules, basic permission management, public room list, inviting, creating DMs, e2e encryption, key backups, device verification, cross signing, notifications, uploading media/files (also user/room avatars), lower-resolution media thumbnails, voip calls and a few other thing's I'm forgetting right now. So it's useable for non-federating chats already.

Thanks to everyone who supports me on Liberapay or Bitcoin!

Synapse

Neil reported:

In Synapse land this week, we shipped 1.16.0 and err 1.16.1.

Lifting from the blog post, the highlights are:-

• An important performance fix to improve room state resolution.

• An option to enable e2e by default for new rooms.

• Ability to run multiple media repo workers side by side.

• Ability to mark specific content as being safe from quarantine.

• Bug fixes to make migrating from SQLite to Postgres more reliable - if you are running sqlite for anything other than evaluation purposes then please migrate!

We also put out a release candidate for 1.17.0 which all being well we’ll release on Monday. 1.17.0 is really a bug fix release the most notable being finally squashing a long standing bug that caused locally rejected invites to get ‘stuck’ client side.

Continuing our matrix.org performance theme, having now got the CS API largely into a good state of responsiveness, we are looking at federation lag. Today we shipped a sharded Federation Reader to matrix.org which has reduced the average lag from seconds to milliseconds. We are just running two currently and are still tuning as we figure out how to get the best from it. Initial impressions seem promising. Watch this space.

We also have a PR out for review to shard the Federation Sender, which will have much the same effect in the other direction.

Once we have sending messages via the Client-Server API, much reduced federation lag and a sharded Pusher we’ll take a look at room joins.

Synapse Deployment 📥️

dacruz21/matrix-chart

Typo Kign reported:

v2.3.0 of dacruz21/matrix-chart has been published with Synapse 1.16.

Kubernetes

Just pushed the 1.16 K8s-optimized Synapse image tags, this time done from a tablet in a server room between moving a bunch of hardware.

New WireGuard-based deployment

balaa announced:

Hi everyone, my friend Patrick and I have been working on making Matrix more accessible in the context of personal overlay networks powered by WireGuard. We’ve built a 1-click deployment solution for Synapse & Riot based on docker. The interesting part is that it gives you public addressability via a distributed proxy service that we have been developing. We are free and open source and welcoming contributors and testers ASAP! We are imagining an ecosystem for building collaborative intranets and see them as foundational to a freer, more equitable internet. Please join us at #noteworthy:tincan.community See https://www.patrickdlg.com/personal-messaging/ and https://noteworthy.tech/start/

Dept of Bridges 🌉

mautrix-whatsapp

Tulir announced:

mautrix-whatsapp got some bugfixes and improvements:

• You can now create private chat portals by inviting a WhatsApp ghost user (e.g. from a group chat)

• WhatsApp users in groups are now synced to Matrix properly, including kicking users who left the group without the bridge noticing

matrix-sms-bridge

Benedict offered:

I fixed some bugs in matrix-sms-bridge and added a feature that allows delayed sending of sms messages with the sms send command. It runs very fast and stable in my production environment.

Dept of Clients 📱

Nheko

Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE (with the notable exception being device verification for now) and intends to be full featured and nice to look at

Nico (@deepbluev7:neko.dev) reported:

Not much to talk about currently, since I'm breaking my Nheko to replace the entire event store at this moment, but I have some heads up on an annoying bug that I fixed on my branch:

Sometimes we had a weird issue, where Nheko wouldn't load the language the user set on their system. This was especially annoying when I wanted to test a specific language, like Japanese, and I couldn't figure out the right LANG* variables to set the language to Japanese in Nheko. Turns out, this was a bug in Nheko and you wouldn't have guessed so, when looking at the examples in the Qt documentation. Anyway, KDAB wrote a nice blogpost explaining this issue: https://www.kdab.com/fixing-a-common-antipattern-when-loading-translations-in-qt/

If you are a Qt developer, you may want to check your applications, if you are also affected.

Riot-iOS

steve offered:

This week, we worked on the rebranding.

RiotX / Riot-Android

benoit announced:

We will release a beta (0.91.5) of Riot-Android this evening, including all the recent changes and lots of fixes. We are still working to prepare the great release!

Daydream

MTRNord told us:

While not much is happening on the master branch some things happened on the Redesign branch:

• Daydream does the sync now on a worker which is similar to using a thread

• jplatte made a PR for optimizing multiple Parts of the Code which significantly improved the overall Daydream performance ( https://github.com/daydream-mx/Daydream/pull/22 )

• Some rework of the Login logic happened to prepare for well-known support

Dept of SDKs and Frameworks 🧰

mautrix-python

Tulir offered:

When I initially implemented end-to-bridge encryption in mautrix-python, I used matrix-nio in a slightly hacky way to get it done quickly. It worked fine in some cases, but also caused some undecryptable messages. I tried switching to using matrix-nio's crypto module directly, but the sans-I/O design made that difficult: there were too many parts that needed to be hooked up to the actual I/O and I wasn't able to find them all. In the end, I just decided to implement the basic e2ee stuff directly in mautrix-python the same way it's implemented in mautrix-go.

All my bridges based on mautrix-python (Telegram, Facebook and Hangouts) have been switched to use the new crypto stuff. After a few initial bugs that are already fixed, it seems to work better than the old system. I'll probably add native e2ee to maubot soon too, which is my last main project that's still missing e2ee.

Dept of Internet of Things 💡

Mozilla IoT Matrix Adapter 0.3

Christian reported:

The Mozilla IoT Matrix Adapter can send posts based on events in your home.

Version 0.3 brings the ability to use room aliases instead of internal room ids. Furthermore, you can configure it to accept invites and follow room upgrades.

Due to utter negligence and malpractice by myself, we missed the update for this project last week. Of course, 0.2 is now superseded by 0.3, but here is the 0.2 update anyway:

Released v0.2.0 of the Mozilla IoT Matrix adapter. It's now possible to post in multiple rooms (still one per post).

Dept of Bots 🤖

Cat Disruptor 7000

Stephen D told us:

I wrote a pretty simple Matrix bot in Rust which is loosely based on Cat Disruptor 6000. It will react with 🐈️ to any message containing the string "cat" *. Since it is a small project, I hope it can be used as an example for other people interested in writing Matrix bots in Rust. It implements several important features (crypto/device store, auto-accepting invites, crafting custom events, etc.) You can host it yourself, or you can try it out by inviting the user "@catdisruptor:m.scd31.com" to your room! https://git.scd31.com/stephen/cat-disruptor-7000

* Cat Disruptor 6000 does not do this (it is used for disrupting monologues with cat pictures). However, many instances of Cat Disruptor 6000 also include a separate bot, which is what Cat Disruptor 7000 is mimicking.

matrix-imposter-bot

mr_johnson22 said:

matrix-imposter-bot - A bot that uses your account to repeat other people's messages. This gives relay-bot capabilities to puppet-only bridges.

Updates:

• Can be much more easily deployed with a production-level WSGI server. It comes with waitress but it's possible to use any other server.

• Member join/leave/rename now post messages

• Shuts down when receiving a term/exit/quit signal, instead of hanging

For more details, see https://github.com/mrjohnson22/matrix-imposter-bot

Dept of Interesting Projects 🛰️

cody

carl offered:

cody is a REPL for your matrix chat rooms.

This week, the big new feature was support for Ruby with the message prefix !rb.

Chat with cody: @cody:bordum.dk

Read the source: https://gitlab.com/carlbordum/matrix-cody

Dept of Jobs 💰️

Famedly is recruiting

sorunome announced:

Heya, we (famedly) are looking for about 2-3 more flutter developers. We are a german startup which works on revolutionizing communication in the medical area. We build on top of matrix, so having some matrix-knowledge would be very beneficial.

You might have seen us previously in TWIM with projects such as famedly-email-bridge or the famedly dart SDK.

While we are based in Berlin we do allow remote work, and communication in english is fine, too.

If you are interested or have more questions, please message Niklas Zender.

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

Rank	Hostname	Median MS
1	fairydust.space	334
2	matrix.vgorcum.com	637.5
3	lossy.network	657
4	asra.gr	795
5	swag.industries	1407.5
6	lo.hn	1481
7	finallycoffee.eu	1804
8	nzbr.de	1885.5
9	halogen.city	2275
10	utzutzutz.net	2462.5

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!