This Week in Matrix 2020-06-12

🔗Open Tech Will Save Us #3 🎙

If you somehow missed this, please take a look!

• 1:00 Rabble talking about Planetary and SSB
• 31:19 Annie on Ditto
• 1:10:20 Erik on Synapse Perf

🔗Dept of Spec 📜

🔗Spec

anoa announced:

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.

🔗MSC Status

Merged MSCs:

• MSC2265: Proposal for mandating case folding when processing e-mail address localparts

MSCs in Final Comment Period:

• MSC2399: Reporting that decryption keys are withheld (merge)

New MSCs:

• MSC2631: Add fallback_content to PusherData

• MSC2630: checking public keys in SAS verification

• MSC2625: Add mark_unread push rule action

• MSC2618: Helping others with mandatory implementation guides

🔗Spec Core Team

In terms of Spec Core Team MSC focus for this week, we're sticking with the same three from last week: MSC2366 (verification flows), MSC2403 (knocking), and MSC2630 (SAS security).

In possibly more exciting news, the Spec Core Team has decided to start separating the scattered casual implementation tips and notes in the spec out and putting them in a series of implementation guides! These will take the form of gitbook-style things, and will initially feature front-to-back implementation guides for a generic Matrix client and homeserver. For more details to provide feedback, check out the associated MSC2618!

🔗Dept of GSoC 🎓

This is the second week of the coding phase of GSoC 2020. Find updates from the students below:

🔗GSOC/Opsdroid

tyagdit told us:

• The port from matrix-python-sdk to matrix-nio is almost completed. Sending and receiving most events works properly. Changes are expected be merged in a few days. This PR itself doesn't implement E2EE, that will be a different PR

• Alongside that I've been working on implementing E2EE with nio which is partially done, sending and receiving messages works so far. some changes to the MatrixStore interaction and testing is left.

🔗Nheko

Chethan told us:

Device verification for Nheko

• Done
  • Adding Device List ( Although it already exists it is re-written in qml )
  • Detecting to_device device_verification messages
  • Part of the flow is done
• Yet to Complete
  • Complete the rest of the flow

🔗go-neb

nikofil offered:

• Implementing end-to-end encryption for go-neb turns out to be a lot easier than expected thanks to the https://github.com/tulir/mautrix library, which contains out of the box e2ee support!

• The first week of GSoC was spent switching go-neb's usage of gomatrix for mautrix, which was easy as the latter is a fork of the former. This caused a pretty big PR that touched most of go-neb's files, although the changes were simple and easy to review. The PR was merged shortly after into go-neb.

• During the second week, it was time to add some basic e2ee support to go-neb. This was complicated slightly due to the bot's ability to create multiple clients and its treatment of /sync responses, however in the end e2ee is now working.

• The code is still of course a mess and session keys are often lost in the void. The code will be cleaned up and tested further next week, bugs will be (hopefully) fixed and another PR will be submitted to go-neb then to officially add e2ee support!

• Code can be found at https://github.com/nikofil/go-neb/tree/e2ee until it can be cleaned up

🔗HTML-Embeddable rooms

From arnav-t:

Matrix has enormous potential to be used in many different use cases be it embedded chat rooms, comments section, chat boxes, etc. This project aims to fill that niche by providing an easy to use and easy to deploy client for users to embed on their webpages.
The client would mesh well wherever it is embedded by providing a postMessage interface to allow the parent webpage to interact with the client and dynamically modify it whenever needed (For example:- A comments section that is always the same theme as its parent webpage).

🔗Features

As of now, the following features have been added (with more on the way)

• List of joined rooms that can be selected
• Live room timeline events
• Support for image-based messages
• Message composer and ability to send messages to a room
• Dark and light themes for the client
• Changeable highlight colors
• Toggleable room header, room timeline, and message composer components
• postMessage interface for sending commands from the parent window

Currently, I'm working on adding more options to the postMessage interface as well as adding more interactivity to the client itself.

Find out more on the project page: https://github.com/arnav-t/riot-embedded.

🔗Ruma/Matrix GSoC update

Ruma is a Rust project to create a comprehensive set of APIs for Matrix. Previously there was a Ruma homeserver project.

devinr528 announced:

This week in ruma-events' Google Summer of Code project, I was able to finish the macros needed to generate the event content enums and trait implementations required for events. I started by defining the generic event structs (state, message, etc.) and manually writing the Serialize and Deserialize implementations. Over the next few days, this was moved into a custom derive macro called Event. The derive now implements all necessary traits with appropriate bounds, so a StateEvent<C> can not contain any ephemeral event content and so forth. I have removed the raw mod and related FromRaw and TryFromRaw traits, moving the validation into the deserialization and constructor for the few types that needed it. On the event content side of things, a function like procedural macro was used to allow declaring the enum using Matrix event type identifiers.

event_content_enum! {

    /// Any message event.
    name: AnyMessageEventContent,

    events: [
        "m.call.answer",

        "m.room.message",
        // ...

    ]
}

// Produces
pub enum AnyMessageEventContent {

    CallAnswer(CallAnswerEventContent),
    RoomMessage(MessageEventContent),

}

🔗Dept of Servers 🏢

🔗Dendrite / gomatrixserverlib

Dendrite is a next-generation homeserver written in Go

Neil Alexander offered:

Much of this week's work has been maintenance-related with refactoring, cleaning up etc.

• Key fetching/expiry behaviour has been fixed, improving the reliability of federation.

• The monolith and component setup has now been refactored.

• A number of unused internal APIs have now been removed.

• Other internal roomserver APIs have been streamlined.

We have also started building an Yggdrasil-powered P2P demo (as opposed to the previous demos built using libp2p) and even includes support for embedding Riot Web in the single binary. It's not very stable/usable yet but there will be more news on that to follow within the next couple of weeks.

We have also started building and releasing Docker images for Dendrite on Docker Hub for those who want to use Docker to test Dendrite deployment.

Spec compliance is unchanged from last week:

• Client-Server APIs: 36%

• Server-Server APIs: 35%

🔗Conduit

Conduit is a Matrix homeserver written in Rust https://conduit.rs

timo reported:

Welcome back! This week I focused my efforts on better error handling. My goal was that every problem would be forwarded to the client (e.g. InternalServerError response) and also be logged. Yesterday I merged the giant PR this resulted in. Hopefully this will make Conduit easier to debug in the future.

Here are some other things I finished this week:

• User-interactive authentication (e.g. you need to type your password again if you want to delete devices)

• Config option to disable registration

I also found the problem with Riot not showing notifications for new messages and created a bug report. With a few tweaks, I managed to solve this on my account and now mentions, DMs and more play the notification sound (it also works on Riot X!).

@gnieto fixed a bug that prevented Riot from loading history when you join a room and @PublicByte implemented the /whoami endpoint.

Thanks to my supporters on Liberapay!

🔗Synapse

Neil said:

This week we shipped 1.15.0, edited highlights include:

Features

• Add admin APIs to allow server admins to manage users' devices. Contributed by @dklimpel. (#7481)

• Add an option to disable autojoining rooms for guest accounts. (#6637)

• For SAML authentication, add the ability to pass email addresses to be added to new users' accounts via SAML attributes. Contributed by Christopher Cooper. (#7385)

• Add support for generating thumbnails for WebP images. Previously, users would see an empty box instead of preview image. Contributed by @WGH-. (#7586)

• Support the standardized m.login.sso user-interactive authentication flow. (#7630)

Bugfixes

• Allow new users to be registered via the admin API even if the monthly active user limit has been reached. Contributed by @dklimpel. (#7263)

• Fix a bug in automatic user creation during first time login with m.login.jwt. Regression in v1.6.0. Contributed by @olof. (#7585)

• Fix a bug causing the cross-signing keys to be ignored when resyncing a device list. (#7594)

• Fix bug where returning rooms for a group would fail if it included a room that the server was not in. (#7599)

• Fix duplicate key violation when persisting read markers. (#7607)

• Prevent an entire iteration of the device list resync loop from failing if one server responds with a malformed result. (#7609)

• Pass device information through to the login endpoint when using the login fallback. (#7629)

• Advertise the m.login.token login flow when OpenID Connect is enabled. (#7631)

• Fix bug in account data replication stream. (#7656)

Internal Changes

• Improve query performance for fetching state from a PostgreSQL database. Contributed by @ilmari. (#7567)

• Speed up processing of federation stream RDATA rows. (#7584)

• Refactor Ratelimiter to limit the amount of expensive config value accesses. (#7595)

• Clean up exception handling in SAML2ResponseResource. (#7614)

• Check if the localpart of a Matrix ID is reserved for guest users earlier in the registration flow, as well as when responding to requests to /register/available. (#7625)

Aside from that we continue out push towards improving performance both in terms of sharding workers but also improving tools to manage disk space usage. Specifically this week and next we are looking at db tuning and sharding the media repo worker. We are also noticing that the typing handler is chewing more CPU than would be expected. It could be a good candidate for moving away from the master process.

More broadly we are changing the defaults for notifications and making changes to support a more general overhaul of notifications.

Finally we are looking at how to not only delete messages after a certain time period, but also how to delete the associated media (which becomes difficult when considering e2e content along side long lived content such as avatars and sticker packs).

For more detail on what is coming next, take a look at our public task board.

🔗Synapse Deployment 📥

🔗Kubernetes

Ananace told us:

Just pushed version 1.15.0 of the K8s-optimized Synapse images

🔗Dept of Bridges 🌉

🔗mautrix-whatsapp

Tulir told us:

The WhatsApp bridge can now bridge location messages from WhatsApp to m.location on Matrix and contact messages as vCard files (until Matrix gets a native contact message type).

It also now encrypts media properly when bridging messages in encrypted rooms. I had apparently missed that when adding end-to-bridge encryption earlier.

🔗mautrix-facebook

Tulir said:

mautrix-facebook got a few improvements and bugfixes:

• Added timestamp massaging

• Added support for bridging video and file messages to Matrix

• Fixed backfilling with SQLite (it didn't work due to the python timezone mess, fix contributed by erdnaxeli)

• Improved options for automatic reconnecting. The bridge should now work fairly reliably when using all the auto-reconnect options and backfilling, it only breaks if facebook decides you've been hacked

🔗mautrix bridges in general

Tulir said:

Since MSC2190 and the related Synapse PR aren't moving much, all my bridges now generate appservice registration files with the workaround. The workaround is needed to avoid patching synapse or using other hacks, and I decided to enable it by default as many people had problems related to having to apply it manually.

The WhatsApp, Facebook and Hangouts bridges now have an option to use double puppeting to disable notifications when backfilling. It can be useful if you want to copy chat history to Matrix, but don't want to be spammed with hundreds of notifications while it's happening.

🔗Nova-slack

eric offered:

To get things ready for NovaChat launch, we added a bunch of new features to a fork of the mx-puppet-slack client that Sorunome created. You can try it out here. We'll be pushing our changes upstream soon.

New features:

• message backfilling at room creation and reconnect

• syncs read status to/from Slack.com

• adds all users to bridged room during initial creation

• automatically bridges all channels and DMs that you are part of during initial setup, so you don't have to wait for a message to arrive for it to appear in Matrix

• tons of little reliability fixes

🔗matrix-sms-bridge

Benedict announced:

My matrix-sms-bridge got an large update, which introduced an automated way to create and write messages to telephone numbers via commands in a bot room. See the docs for more information, how to use it.

🔗Dept of Clients 📱

🔗NovaChat v0.1.1

eric reported:

Introducing NovaChat - a desktop chat client that supports 9+ different chat networks, built on top of Matrix. Think of it as a modern Adium/Trillian.

How it works

NovaChat is a system with three parts designed to work together:

• a new desktop client, based on Riot-web

• a federating homeserver (nova.chat)

• a set of Matrix bridges configured to run on a private server called a bridgebox

Matrix bridges are awesome but generally require you to run own homeserver. NovaChat removes that restriction, without sacrificing security and privacy. Your remote chat network credentials never leave your bridgebox, and messages are encrypted* on the bridgebox before being sent to the NovaChat homeserver.

*Encryption is currently enabled on Telegram/FB/Hangouts/Whatsapp bridges, mx-puppet-bridges coming shortly)

Very rough system diagram:

> NovaChat desktop client <> NovaChat homeserver <> bridgebox <> remote chat networks

Features

• Fancy new UI, heavily inspired by Telegram. See photo below or see it in action

• Easy graphical bridge set up (no more fiddling with config files)

• Reliable bridging, with delivery confirmations and bridge disconnect warnings

• Includes 8 pre-configured bridges: Facebook/Whatsapp/Hangouts/Telegram/Slack/Instagram/Twitter/Skype. Planning to add more soon, or submit a PR!

• Works on Mac OS, Linux and Windows

You don't need to be technical to use NovaChat, it's designed to Just WorkTM. If you need help setting up a bridgebox, we are offering hosting and maintenance for USD $5 per month.

The vast majority of this software has been created by the talented tulir and dm0141. I personally am a terrible coder (you can tell which parts I did), but I do have experience building products and teams, and I'm excited to begin working in the Matrix universe. I wrote recently about how I believe Matrix is the Universal Communication Bus! Thanks goes to Sorunome for creating the mx-puppet-bridges!

We're developing the whole project in the open (though not all components are open source), check the issue backlog :)

Want to help? We're hiring Matrix developer to work on the project with us. Send me a message if you're interested! Particularly looking for a React dev to help with frontend on nova-web.

NovaChat chat room: #novachat:nova.chat

🔗Nheko

Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE (with the notable exception being device verification for now) and intends to be full featured and nice to look at

red_sky announced:

nheko 0.7.2 is ready for release. Here's the summary of changes:

🔗Highlights

• Reactions

  • React to a message with an emoji! 🎉

  • Reactions are shown below a message in a small bubble with a counter.

  • By clicking on that, others can add to the reaction count.

  • It may help you celebrating a new Nheko Release or react with a 👎 to a failed build to express your frustration.

  • This uses a new emoji picker. The picker will be improved in the near future (better scrolling, sections, favorites, recently used or similar) and then probably replace the current picker.

• Support for tagging rooms [tag]

  • Assign custom tags to rooms from the context menu in the room list.

  • This allows filtering rooms via the group list. This puts you in a focus mode showing only the selected tags.

  • You can assign multiple tags to group rooms however you like.

• SSO Login

  • With this you can now login on servers, that only provide SSO.

  • Just enter any mxid on the server. Nheko will figure out that you need to use SSO and redirect your browser to the login page.

  • Complete the login in your browser and Nheko should automatically log you in.

🔗Features

• Support for showing users presence and status message as well as setting your own status message"

• Respect exif rotation of images

• An italian translation (contributed by Lorenzo Ancora)

• Optional alerts in your taskbar (contributed by z33ky)

• Optional bigger emoji only messages in the timeline (contributed by lkito)

• Optional hover feedback on messages (contributed by lkito)

• /roomnick to change your displayname in a single room.

• Preliminary support for showing inline images.

• Warn about unencrypted messages in encrypted rooms.

🔗Improvements

• perf: Use less CPU to sort the room list.

• Limit size of replies. This currently looks a bit rough, but should improve in the future with a gradient or at some other transition.

• perf: Only clean out old messages from the database every 500 syncs. (There is usually more than one sync every second)

• Improve the login and register masks a bit with hints and validation.

• Descriptions for settings (contributed by lkito)

• A visual indicator, that nheko is fetching messages and improved scrolling (contributed by Lasath Fernando)

🔗Bugfixes

• Fix not being able to join rooms

• Fix scale factor setting

• Buildfixes against gcc10 and Qt5.15 (missing includes)

• Settings now apply immediately again after changing them (only exception should be the scale factor)

• Join messages should never have empty texts now

• Timeline should now fail to render less often on platforms with native sibling windows.

• Don't rescale images on every frame on highdpi screens.

🔗Matrix-Static release v0.3.0

Michael (t3chguy) reported:

New Matrix-Static release v0.3.0

With improved support for media and tombstone events and a bunch of bug fixes! https://github.com/matrix-org/matrix-static/releases

Should be deployed to view.matrix.org shortly

🔗Arch Linux package of radical-native

Peter Roberts told us:

I've made an Arch Linux package of radical-native.

🔗Riot X

valere announced:

• Several improvements on SSO login flow 🔒

• Support of power levels ⚡️

• Add remove favorites 🌟

• Multiple bugs and glitches fixes 🐛

The team is currently actively working on:

• Voip 1:1 Calls 📞

• Cross Signing bootstrap simplification 🔑

• Performances Improvement 🔥

• More room settings ⚙️

🔗Riot Web

Ryan told us:

There's been lots of work many fronts this week, including:

• Continued work on cross-signing bootstrapping robustness

• New room list implementation in progress

• Upgraded fonts for new emoji

• A refreshed look for matrix.to is underway

• The appearance tab is being tweaked and expanded

• Read marker consistency improvements

Most of this work is still in progress or has only landed on develop so far. Our next RC will happen next week.

🔗Riot-iOS

Manu offered:

This week, we continued to iterate on the implementation of the new push notification service thanks to user feedbacks. Those feedbacks also show that the app uses less battery.

On login, this is now possible to recover your cross-signing and your key backup from your recovery passphrase or key.

🔗Dept of SDKs and Frameworks 🧰

🔗libolm 3.1.5

uhoreg told us:

libolm 3.1.5 has been released. It mainly collects the various build system and documentation fixes that have accumulated since the last release. It also includes a new wasm build target (which apparently is used for running matrix-rust-sdk in the browser), as well as TypeScript definitions for the JavaScript bindings.

🔗Ruma

Ruma is a Rust project to create a comprehensive set of APIs for Matrix. Previously there was a Ruma homeserver project.

jplatte told us:

Most of the work this past week was on ruma-events, for more on that see the GSoC section! Devin got far enough with that so we now know it all works out and can replace the previous ruma-events API. When that was clear, ruma-events was also merged into our new monorepo; now the only remaining crate to be merged is ruma-client (where I'm also waiting for some pending changes).

🔗Dept of Ops 🛠

🔗Matrix Helm chart

Kubernetes Helm chart for easily deploying a Matrix homeserver stack

Typo Kign told us:

Version 2 of my Matrix Helm chart is out! It now allows for running a federating homeserver without needing an external load balancer for the federation service, potentially lowering costs (depending on your cloud provider), and definitely reducing the headache of managing TLS certificates!

dacruz21/matrix-chart is a Kubernetes Helm chart for easily deploying a Matrix homeserver stack, complete with Synapse, Riot, VoIP calls, bridges, and other backing services. It's ideal for anyone looking to run a large homeserver, or for corporate environments where security is paramount!

🔗Dept of Internet of Things 💡

🔗Tuple Weather Station Example

Alexander Eisele announced:

This week I've programmed a small POC for a Matrix weather station.

It's build using a ESP8266 Microcontroller and a BME280 Sensor for temperature, humidity and barometric pressure.

The data is send over the Tuple Matrix <-> MQTT bridge.

Code: https://github.com/derEisele/tuple-weather-example

You can join this project on #tuple:eiselecloud.de.

I'm personally really excited about this project!

🔗Dept of Bots 🤖

🔗Matrix bot for Redmine

progserega told us:

https://github.com/progserega/matrix_redmine_bot

1. Bot can create, accept (to user) new issues in Redmine from matrix

2. Bot can add comment or/and files to issue by it id

3. Bot can show messages (for user) from Redmine in matrix room

🔗Dept of Guides 🧭

Levans said:

Anyone running a federating instance of Synapse will likely have seen this: Synapse is database-hungry. It tends to take a lot of space. In this post, I'm documenting how I shrunk my homeserver database from 100GB to a little under 8GB, during a long maintenance cleanup. https://levans.fr/shrink-synapse-database.html

Very pleased to see this, I have a feeling I'm going to be running it in the near future...

🔗New Public Rooms 🏟

🔗New room "Matrix Outreach"

JulianF reported:

There's a new room "Matrix Outreach" #matrix-outreach:matrix.org for folks interested in "Helping each other explain and promote Matrix to the public: creating slogans, introduction leaflets, info-graphics, elevator pitch, Wikipedia articles, and so on."

🔗Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

Rank	Hostname	Median MS
1	fairydust.space	391
2	envs.net	443.5
3	maunium.net	506
4	tum.de	603.5
5	neko.dev	822
6	maclemon.at	887
7	cyllos.me	916
8	shortestpath.dev	1072.5
9	settgast.org	1079
10	maescool.be	1324

🔗That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!