Matrix logo

Dendrite development final report (GSoC 2019)

10.09.2019 18:21 — GSOC — Alex Chen

🔗Overview

During GSoC 2019, I participated in and pushed forward the development of Dendrite, a high-performance, scalable Matrix homeserver implementation written in Go.

Like many others passionate about Dendrite, I've been eager to see the day when it finally joins the decentralised, federated Matrix network, so I worked on the project to see how much I could do to bridge the gap between it and the current reference homeserver implementation, Synapse, in terms of feature completeness.

Working on Dendrite in just one area would be insufficient to propel it, so my work covers various components in the project, including the Client/Server API, Sync Server, Room Server, and Federation Component. I also spent time refining the project's documentation, improving its testing/continuous integration process, as well as reviewing pull requests from Matrix.org members and the community.

Below is the major portion of my work presented as pull requests, categorised by the components they belong to; a list of links to all the pull requests I created/reviewed is also available at the end of this report.

🔗Work in Component: Client/Server API

This component is the main handler of HTTP requests from the clients, except for /sync requests - see next part.

Work done:

Work left:

🔗Work in Component: Sync Server

The Sync Server is responsible for handling the long polling notification requests (/sync requests) and some other related requests from the clients.

Work done:

Work left:

  • Support for other types of EDUs

🔗Work in Component: Room Server

The Room Server is, as its name suggests, where room events, state, etc. are handled.

Work done:

Work left:

  • Support for alias deletion by admins
  • Discussion on and implementation of event retention policies

🔗Work in Component: Federation

The Federation Component sends requests to and receives requests from other Matrix homeservers.

Work done:

Work left:

  • Implementation of a "Most Recent Transaction Sender" In-Memory Cache
    • There is a delay in this part of work because some dependency problems took longer than expected to resolve, but this has been planned and will be worked on after GSoC.
  • Implementation of a component that handles backoff/retries for federation requests

🔗General Fixes, Improvements, Maintenance Work

Here is a list of links to all the pull requests I have created/reviewed (by 26 Aug, 2019):

RepositoryPull Requests
matrix-org/dendriteCreated (36) / Reviewed (27)
matrix-org/gomatrixserverlibCreated (12) / Reviewed (1)
matrix-org/sytestCreated (5)

🔗Appendix: On Dendrite Being a GSoC Project

I believe that Dendrite is a pretty special project to work on in the GSoC program.

At its current state, Dendrite isn't a complete homeserver yet. This has left a large portion of work on Dendrite wide open: As I worked on Dendrite, more than often I'd find myself in a place to answer questions related to topics like design or project architecture which, if not thoroughly thought through, might have consequences in the future development of the project. For this reason, I think working on Dendrite involves a set of challenges quite different from what attempting to improve an already functioning piece of software would have presented.

For the same reason, I think what Dendrite has taught me during the summer was less about how to become more fluent in Go or more familiar with various tools, but more about developing the mindset that powers software projects behind the scenes. But this mindset, to myself, seems more valuable than the former, indeed.

And at the end of this report, I'd like to thank my mentors anoa and Brendan, as well as everyone else at Matrix who has answered my questions, discussed with me, and helped me understand the code, and at the same time being super responsive. Without your help, my GSoC experience wouldn't have been this enjoyable! :)

Matrix Visualisations final report (GSoC 2019)

09.09.2019 00:00 — GSOC — Eisha Chen-yen-su

The Google Summer of Code 2019 is coming to an end for me, so it means that it’s time for the final report.

🔗The work done

I’ve been taking care of the project “Matrix Visualisations” during these past months. This project aimed at initiating the development of a tool which allows the real-time visualisation of the events DAG of a given Matrix room, as seen from the perspective of one or more homeservers (HS’s).

Regarding my initial proposal, I’ve completed every task proposed and even implemented some additional functionalities. The application is not finished yet and there could be a lot of improvements added to it (especially regarding the design of the UI) but the core functionalities have been implemented.
I am going to precise what has been accomplished and then give some ideas of features to improve.

During GSoC, I have used two separate repositories for the frontend and the backend. I will keep both of them because I’m referencing PRs from them (as PRs are easier to link than lengthy lists of commits).
However, this is the repository regrouping these two parts and this one will be moved to matrix-org for the continuation of this project.

🔗Complete the implementation of the CS API backend

During the application period, I wrote a prototype for this project. This prototype implemented some requests to the CS API (like /login, /logout or /sync) but there were more requests to implement in order to be able to fully use this API:

  • In order for the application to automatically join (and leave at the end) a room if the provided account hasn’t already joined it, I implemented the requests to /joined_rooms, /join and /leave.
  • The request /messages has been implemented to allow the application to fetch previous messages.
  • For retrieving in real-time new events, I used the field from in the /sync request.

I also did a lot of clean up in the source code from the prototype during this task. You can have more details in this PR.

🔗Implement the first UI to interact with the DAG

First of all, a lot of work had to be made in order to properly update the displayed DAG when adding new events to it. At this point, I previously used the setData method of the Network object of the visjs library (which is used for displaying the graph and interacting with it) each time a node was added, but it was resetting the display each time it was called.
The proper solution was to progressively add nodes and edges to the DataSet object passed to the constructor of the network (see the documentation of DataSet and Network for more details).

The DAG has been set to be displayed vertically, the events with the same depth are at the same level, the deepest events are at the bottom. The events which origin is the HS on which we are making the observation are in green, those which are coming from other servers are in orange. Here is a picture of what it looks like: Events display

In the graph, there is a special node just after the earliest event which allows us to ask the application to load more events. Here is what it looks like: More Events button

When you double click on the node of an event, there is a text area on the right which displays its complete JSON body, like this: JSON body display

I’ve also implemented the possibility to choose which particular fields of the event can be directly displayed in the labels of the nodes of the displayed graph: Fields selection

🔗Synapse PostgreSQL backend

Next, I implemented a backend for retrieving events from the PostgreSQL database of a Synapse HS. It is a small HTTP server which receives requests from the frontend application and then makes queries to the Synapse database to get the requested events.
You can find details about the API for using this backend in this readme, in the “HTTP REST API” section.
You can find more details about this initial implementation: here, here, here (my mentor helped me on this one, thanks to him), and here.
I’ve added the support of this backend in the frontend application, as well as a way to choose which backend to use (between this one and the CS API) in this PR.

🔗Multiple views

I implemented the ability to observe the same DAG of a room from multiple HS’s. This is done with “views”.

View selection

In the picture above, you can see that there is a drop down menu from which you can select the view. The fields under this line are used to control the selected view: indicate where it will be connecting, for which room (you could as well observe a different room in a different view), etc…
By default, there is only one view but you can add as many as you want by clicking “Add a view”.

All the DAGs from the different views are displayed side-by-side within the same canvas, like this: Multiple DAGs

You can have a look at the details of the implementation in this PR.

🔗Add Federation API support

The next major task I had to do was the implementation of a backend for retrieving events via the Federation API. I thought a lot about the possible options for the location of this backend and I decided to extend the web server created for the PostgreSQL Synapse backend, so that we could launch it in a “postgres mode” or “federation mode”. But the backend would offer the same HTTP API in both modes.

The backend uses the Federation API in the following way:

  • Before being able to retrieve events from a certain room, the backend must join it with /make_join and /send_join requests, it creates an “imaginary user” in this room. The join event created during this process will be the one returned by the endpoint /deepest.
  • To get earlier events, the backend uses /backfill requests.
  • In order to get new events, the backend listens for pushed events from other HS’s with the /send request.
  • When the observation is done, the backend makes the “imaginary user” leave the room by sending /make_leave and /send_leave requests.

The full details of the implementation are in this PR. My mentor also helped me get the usage of the Futures right thanks to this PR.
There has been a small modification in the frontend too, because of the addition of the /stop endpoint in the backend’s HTTP API, these modifications are in this PR.

🔗Display the state of the room for a given event

For each event, there is a state of the room associated with it, which describes what was the state of the room at the moment this event was accepted (the name of the room, its topic, its members and parameters, etc…).
So I’ve added a way to display this: when you have selected and displayed the JSON body of a given event, you can also request the associated room’s state. I have made it possible to use this feature with every backends: the CS API, the PostgreSQL database and the Federation API. You can have the full details of the implementation here (for the backend) and here (for the frontend).

You can see the result of this feature in the picture below (there is a button “Room state at the selected event”, which allows to ask the application to fetch the state, and the text area under this button where the state is displayed): Room's state display

🔗Additional fixes

Lastly, I’ve applied small fixes to the code of the backend, you can see them in this PR.

🔗Possible improvements

The objective of this project was to develop the core functionalities of this application, however there are a lot of improvements to bring to it, like:

  • Adding the possibility to start the observation of a room from any events (provided that we have the ID of this events) instead of the latest one.
  • These hasn’t been any UI/UX work design, the CSS style sheet is minimal and the overall look isn’t beautiful or correctly organised. So there would be a lot of work to be done in this area by people with better knowledge in this field than me.
  • The timestamps of the events are not displayed in a human readable format and would be written as dates and hours to greatly improve the readability.
  • The application has been tested a lot, especially in situations of misusage. I’ve fixed some bugs which occurred when I was testing it but it was far from being an exhaustive testing, so there could be many improvements regarding the overall robustness of this software.
  • The backend supports HTTPS connections but has no mechanism for controlling the access of the data behind it (in particular, it means that if you should not run on a production database as it would basically allow anyone to access any data on it). So more work would be needed to make it secure.

🔗Conclusion

This experience has been really rewarding for me. I could discover more about the Matrix community and how the Matrix ecosystem works (on a technical point of view). I want to thank my mentor, Erik Johnston, for his guidance during these past months, and the people in this community who gave me advice.

GSoC has also allowed me to further improve my programming skills in general and discover many various things: the WASM technology, how to use Rust in this context thanks to the various existing libraries/frameworks available, the practical usage of SQL requests as well as TLS certificates and how to apply cryptographic signatures.
It was sometimes challenging to use such experimental technologies (due to the lack of clear documentation) but also very exiting!

Mid-September, I will start my class for my second and final year of my master degree (software engineering, specialised in distributed systems and applications) at Sorbonne Université so I will definitely have less free time. So I don’t think I’ll be able to actively continue to contribute but I will do my best to help other people to continue the work I’ve initiated.

This Week in Matrix 2019-09-06

06.09.2019 00:00 — This Week in Matrix — Ben Parsons

🔗Matrix Live 🎙

🔗Dept of Servers 🏢

🔗Synapse

Neil from Synapseland:

This week privacy items have dominated, but we’ve also been edging towards getting our room directory improvements out, shipping a new configuration creator and making further progress on the federation side bus project to improve performance of smaller instances.

We also deployed open tracing on matrix.org, which will help a lot in tracking down all sorts of strange behaviour in the wild and fixing some bugs in the sqlite -> postgres migrator.

Next week we’ll land the room directory improvements, much of the privacy work, and have the configuration creator ready for testing.

🔗Construct

Jason:

Construct added user room listing support so that you can find out what public rooms a user is a member of should they choose to not be invisible. To find out the list of rooms for a user, simply open the room directory in Riot and type a user mxid starting with @...

Construct is implementing the concept of invisibility for users:

The exact mechanism for invisibility is not completely finished but there are obviously some limitations for this feature:

  • Only users who are local to the server can set any kind of invisibility. Remote users are always fully listed. One day when the spec has "user's rooms" we would be glad to adhere to an m.user.invisible-etc configuration state event in such a room
  • Since there is no federation support for user's rooms list publication, the rooms list for a user is limited to what the server knows (rooms it has seen the user in).

Construct is eager to implement any MSC's regarding user room and group membership listing over the federation.

Construct is written for maximum performance in C++, and is developing rapidly - join #construct:maunium.net for more info.

🔗matrix-media-repo updates for quarantining and purging media

TravisR announced:

matrix-media-repo has received a bunch of updates regarding quarantining and purging media - see the admin docs for more information. There's also an s3 upload bug fix if you're using that.

🔗Dept of Bridges 🌉

🔗GitLab bot to be re-written as a maubot

Tulir announced:

lorico has been rewriting my old GitLab bot to python/maubot: https://github.com/maubot/gitlab/pull/23. Progress on that is looking very good and it'll probably be ready within a week or two.

Check the maubot GH page for more.

🔗mautrix-telegram, small release

Tulir again:

I decided to make a minor mautrix-telegram release with some small fixes, since v0.7.0 will take a while still. v0.6.1-rc2 is out now, the full release will probably happen next week if nobody finds any problems. Main changes:

  • Invalid reply fallbacks sent by Riot web in edits are now ignored
  • User and portal sync errors are now caught so that the sync command wouldn't fail if one telegram chat disappears.

🔗Dept of Clients 📱

🔗Riot Web

Tidy update from Bruno this week:

This week we were doing the last bits of the privacy work, and fixing bugs and adding a format bar in the new composer (still behind labs flag).

We’ve also been planning the First Time User Experience work, which will start at full speed next week. This is an effort to address some issues that new users hit in the first days of using Riot. See https://github.com/vector-im/riot-web/projects/16#card-25137120 and Matthew talking over the details at https://youtu.be/JYsiHSL0lEA?t=710. Letterboxes in the left panel will likely only be removed as part of the new community project though, as we need parts of the new community UX to keep things usable for power users.

Also, if you're using https://riot.im/develop please enable the new composer in Labs, it works well and deserves testing.

🔗Continuum desktop client, many updates

yuforia reported:

Continuum desktop client written in pure Kotlin, version 0.9.20:

  • Updated login page appearance
  • Added validation to user ID and server address fields
  • The server address gets automatically filled with https:// followed by server name when user ID is being typed
  • After signing in, the token and server address are automatically saved
  • Added progress indicator when being signed in. It doesn't block user from controlling Continuum, pressing Esc will cancel the signing-in process.
  • Speed up application start-up by drawing the window and opening the database in parallel. Now the window typically appears within half a second with its size and position on screen restored to be the same as the last time it was opened. The most recently used accounts get loaded and automatically filled within a second.
  • Added drop shadow effect to scroll-to-latest-messages button to increase contrast
  • Fix: Always retry synchronization after getting a network error. Previous versions didn't retrieve new messages in a timely manner when the network is unreliable. A delay is added if the error is not a time-out to avoid stressing the homeserver.

continuum

koma the Kotlin library behind Continuum:

  • Fix suspend/hibernate detection. Synchronization is restarted automatically after the computer resumes from sleeping. The reason is that any previously opened connection is unlikely to be still alive, opening new connections reduces delay.

  • Fix incorrect interface declaration that caused login to fail

🔗Pattle 0.13.0 released

Wilko said:

A new version has been made available and should soon be on F-Droid, Google Play and TestFlight!

Expect the new version to be on most platforms in 2 days.

It's not the biggest release, but a release nonetheless.

🔗Changes

  • Separate public and personal chats
  • Use a bullhorn icon for public chats without an avatar
  • Fix sync being started multiple times on startup
  • Fix the room.member_count error (might require clearing of data)
  • Many SDK changes, for features in future releases

🔗Get Pattle

Please let me know what you think of the separation of personal and public chats! Currently it's implemented using a bottom navigation bar, but tabs could also be an option (which is closer to what WhatsApp does)

Pattle

🔗Riot Mobile versions

All teams when I asked responded:

Privacy, privacy and privacy.

🔗Dept of Services 🚀

🔗t2bot.io has 300k MAU

TravisR told us:

t2bot.io has surpassed 300k monthly active users (about 35k daily) 🎉

roughly 250k are (bridged) from Telegram, the rest is from Discord. (there's some very small numbers which include the bots, webhooks, and other tiny bridges, but together they form less than 100 users)

🔗room list viewer tool

joepie91 offered:

Inspired by a question in the Matrix HQ channel, I've built a (rudimentary) room list viewer at https://matrix-rooms.cryto.net/ - you can just enter a homeserver (hostname) and see all the public rooms on it, without needing to log into anything. This might be useful for showing other people what kinds of rooms exist on Matrix!

🔗Dept of Blockchain 🤷‍

🔗OI Chat, Matrix based chat uses blockchain for authentication

@friedger:matrix.org said:

OI Chat has been updated and now connects blockchain users on Blockstack and EOS (and the rest of the matrix universe): https://www.producthunt.com/posts/oi-chat-2 The sever has no passwords or requires permissions by other social account providers, it verifies the authenticity of the user using properties of the blockchain IDs, it names registered on the blockchain.

🔗Dept of Ops 🛠

🔗Matrixmon

Matrixmon new release 0.4.0, featuring Docker image hosted on Docker Hub and an option to specify custom config file location.

🔗Dept of Bots 🤖

🔗matrix-fly-paper: new project for room moderation

@serra:lost-angles.im reported:

matrix-fly-paper is a new bot project intended to help automate server and room moderation. It's currently not ready for any kind of use, as I'm trying out serious TDD and attempting a functional approach to break from my OO training. Feedback is appreciated, stop by #matrix-fly-paper:lost-angles.im to talk shop

If you're interested in such things, there is an existing tutorial on how to use matrix-bot-sdk for exactly this at https://matrix.org/docs/guides/matrix-bot-sdk-room-admin-features.

🔗Dept of Merch 👕

About a month ago we started selling merch at shop.matrix.org, and WOWEE ZOMG we sold a lot of stuff! If you are happy with your purchase I'd love to hear from you - share photos of where you put the stickers or you happily wearing the t-shirt in #twim:matrix.org and I'll make a little collage.

If you are not happy with your purchase then please message me at @benpa:matrix.org and I will try to make it right.

🔗Final thoughts 💭

You should complete this EU-related FOSS survey, and if you wanted to say nice things about Matrix that would be fine by us.

🔗That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

This Week in Matrix 2019-08-30

30.08.2019 00:00 — This Week in Matrix — Ben Parsons

🔗Matrix Live 🎙

🔗Dept of Spec 📜

Approved MSCs

Final Comment Period

In Progress MSCs

🔗Notes on the Spec from tulir

I made a few spec proposals: MSC2244: Mass redactions and MSC2246: Asynchronous media uploads. Mass redactions are useful for fighting spam (and bridging telegram deletions) more efficiently. Asynchronous media uploads would make it possible for bridges (like telegram) to handle messages synchronously and thus keep message ordering without blocking all messages behind some media upload.

🔗Dept of Servers 🏢

🔗Synapse

Erik reported:

  • Busy landing the remaining items for the privacy project.
  • Reviewing and landing the room stats work Oliver has been busy with before he finishes his internship this week (thanks Oliver!).
  • Jorik has been doing the finishing touches to the Synapse installer (which looks awesome fyi).
  • We’ve also been looking at a basic message retention implementation in Synapse.
  • Finally, Hawkowl had a lot of fun at Kiwi Pycon last week (woo keynote!), and is now back and looking at improving performance of outbound requests.

🔗Construct

Jason from Construct:

Construct added support for server notice rooms. As the server administrator I've placed quick access to the server control room for myself in this section.

Construct has also added support for the server-side commands \\read <room-tag>, for example \\read m.low_priority will mark all low-priority tagged rooms as read. The command \\read * also will mark all rooms as read.

Also, we now allow users to configure whether they transmit typing events by setting state events in their home-room. If a user creates an event with the type of ircd.typing.disable.send and the state_key of the room_id, typing events will not be sent to that room for them. Further variations exist, such as ircd.typing.disable.sync to prevent the client from receiving typing events for a particular room or all rooms.

Learn more in #construct:zemos.net

🔗Matrix Visualisations GSOC project 🎓

Eisha announced:

I've finished my GSoC project "Matrix Visualisations", you can view my final report at this address https://gist.github.com/Kagamihime/d0e7ac434b0d80eb201260d213dea0b7

🔗Dept of Bridges 🌉

🔗mx-puppet-twitter NEW

sorunome reported:

mx-puppet-twitter....do they ever stop?!

Now there is a twitter DM bridge based on mx-puppet-bridge! It uses twitters webhook API to receive direct messages instantly.

So far implemented:

  • basic message sending in both directions
  • Image, Gif and Video sending (twitter special-cases those) in both directions
  • twitter oauth to authenticate

Support room: #mx-puppet-bridge:sorunome.de Donate on liberapay: https://liberapay.com/Sorunome/donate

🔗Jojii email bridge NEW

Jojii announced:

For all those who want a new emailbridge, I have great news for you! The new bridge supports sending and writing emails, using multiple rooms for multiple email-addresses. In addition you can use markup to improve the style of your emailbody, which gets translated to HTML. Here you can find the repository: Matrix-EmailBridge

Go join #jojiiMail:matrix.jojii.de for more.

🔗Big news for the Slack bridges

Half-Shot offered:

The Slack bridge has had a whole week of care applied to it. We've switched the bridge entirely over to Typescript, and overhauled large parts of the codebase to make future work easier (for ex, using the Slack node library rather than request calls). If that's not exciting enough, we've also added support for the RTM API, which means the bridge can use websockets when talking to slack. This gives us nice things like typing updates, and the ability to deploy without needing a webserver to listen for events. Release 1.0 is imminent, but before then we are hoping to land Postgresql support so that the bridge can scale up to meet the high demands placed upon it.

🔗mautrix-whatsapp

Tulir told us:

mautrix-whatsapp got a bunch of small fixes and a dbms migration tool.

🔗Dept of Clients 📱

🔗Riot Web

Ironed the bugs out of the new composer, which is available on https://riot.im/develop behind a labs flag. Please test!

🔗Riot iOS

Privacy, privacy and privacy.

🔗Riot Android

A security release has been done on thursday. More details here: https://medium.com/@RiotChat/matrix-android-sdk-security-issue-affecting-riot-android-36b4792ea0d6

In parallel we continued working on privacy.

🔗RiotX (Android)

A release has been done today (v0.4.0). The main new feature is Read Receipts!

🔗Neo Renaming

f0x reported:

I'm considering renaming Neo, to help with getting a nice domain, and better findability in general. Any suggestions welcome in #neo:pixie.town.

🔗Pattle 0.12.0

Wilko told us:

A new version has been pushed to F-Droid, Google Play and TestFlight!

⚠️ This update requires a reinstall or clearing of data ⚠️: Because of some database changes which were not that easy to migrate, you'll have to reinstall the app or clear the data.

Expect the new version to be on most platforms in 2 days, and on Google Play in a few hours.

🔗Changes

  • Add a dark theme!
  • Add basic settings page
  • Add ability to change your own display name
  • Allow multiple lines while typing a message
  • Render line breaks in messages
  • Fix issue where homeserver setting was overridden by the user domain
  • Mark new messages as read while chat is opened
  • Fix chats being sorted by your own display name change
  • Fix multiple issues regarding the member count of room being null
  • Use Flutter beta, which means that you can paste in textfields again!

🔗Get Pattle

🔗Issues or suggestions

If you stumble upon any issues or would like to suggest features, please do so here! You can login via GitHub and Gitlab.com!

🔗Follow development

Follow development in #app:pattle.im!

🔗Continuum 0.9.19

yuforia reported:

Continuum the pure Kotlin desktop client

  • Provide builds of self-contained executables that have no external dependency. The Linux version has been done. AppImage is used, producing a binary of about 50 MB.
  • Adjust appearance. Remove alternating row high-lighting in list views. Remove extra padding between messages.

Continuum

🔗Quaternion has many visual improvements

kitsune reported:

Thanks to Roland Pallai, Quaternion master branch is now able to use customised fonts and sports a faster timeline, better copying and a few other improvements. Settings dialog window is coming.

🔗Dept of SDKs and Frameworks 🧰

🔗igor: a Matrix bot framework for Elixir

uhoreg offered:

for fans of strange programming languages or of monster movie clichés, I've created a Matrix bot framework for Elixir: https://gitlab.com/uhoreg/igor

🔗Dept of Encryption 🔐

poljar:

Initial code dump of Seshat, a Matrix event indexer/database that should enable search for encrypted rooms in riot-desktop. The dump contains a rust library as well as node bindings. The repo can be found here.

🔗Dept of Ops 🛠

🔗ARM docker images for synapse

Black Hat told us:

Apparently there are no ARM docker images for synapse, so I built a multi-arch synapse docker image. Supported CPU architectures include armv7, aarch64 and amd64.

🔗Dept of Bots 🤖

tulir:

There's a useless new maubot plugin that lets you run python and soon shell scripts from Matrix. Not intended as a public bot as it doesn't have any sandboxing, but it does have a "userbot" mode to run it under your own account so that you can use it in any room.

Slightly mean to call your own bot useless :(

🔗Final thoughts 💭

If you've suffered from invite spam lately, you can solve the problem by following these instructions:

There's 3 methods to block the room:

  1. Use the synapse admin API for it:
curl -s -X POST -H "Authorization: Bearer YOUR_ACCESS_TOKEN" -H "Content-Type: application/json" -d '{"new_room_user_id":"@abuse:YOUR_SERVER.org"}' 'https://YOUR_SERVER/_matrix/client/unstable/admin/shutdown_room/!BAD_ROOM:example.org'
  1. Block the room with an antispam module ( https://github.com/t2bot/synapse-simple-antispam )
  2. Ignore the user with /ignore

🔗That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

This Week in Matrix 2019-08-23

23.08.2019 00:00 — This Week in Matrix — Ben Parsons

🔗Dept of Status of Matrix 🌡

🔗Installing Matrix @ CCCamp 2019

It's CCCamp 2019 this weekend. Tonight, anoa will be leading an "Install Synapse" workshop in the Librehosters Village- keep decentralisation growing!

CCC poster

🔗Dept of Servers 🏢

🔗Synapse

Neil:

This week we’ve been working on getting our brand new room directory implementation ready, creating a new Synapse installer, and creating a TURN server testing tool. We’ve also been pushing on with the privacy project and fixing up some bugs and perf issues.
Next week it’s all about getting the room directory out of the door, shipping the installer and continuing on with the privacy project. We are in the early stages of a project designed to reduce the load caused by outbound SSL connections, it is looking very promising. We’re also going to give the sqlite->postgres porting script some love as well as working on some message retention ideas, and also ways to reduce the meta data we store.

🔗Dept of Bridges 🌉

🔗mautrix-facebook updated

Tulir announced:

the adding portal rooms to user-specific communities feature that appeared in mautrix-whatsapp and mautrix-hangouts last week is now also available in mautrix-facebook

🔗Gitter bridge is back

Half-Shot offered:

The gitter bridge is back to working order again today! We've made changes so that Gitter side admins can now issue a !matrix kick|ban name command to evict users who are causing issues, rather than relying on reports or matrix side admins.

🔗matrix-appservice-bridge library 1.10.0

Half-Shot reported:

The matrix-appservice-bridge library also saw its 1.10.0 release this week, which has experimental support for bridge errors and also has a bugfix related to room upgrades.

🔗Discord bridge will soon use matrix-bot-sdk

Half-Shot said:

Finally, the discord bridge has very nearly switched to Travis's bot SDK stuff. We are very close to a release, so please stand by :)

🔗Dept of Clients 📱

🔗Riot Web

From the team:

  • Work on privacy.
  • We’ve also been continuing the work on the new message composer implementation. Features are still missing, but this has now landed behind a labs flag on /develop. We’d be especially interested in feedback whether typing with an IME (Chinese, Japanese, …) works better than before.

🔗Riot iOS && Riot Android

From the team:

  • Working on privacy:
  • Working on identity server v2 API

🔗RiotX (Android)

From the team:

  • Read receipt work has been merged to develop. We will release next week, after a small UI iteration
  • François is working on Read Markers

🔗Riot with Math support

Tim offered:

I've hosted a version of riot with the changes from https://github.com/matrix-org/matrix-react-sdk/pull/3251 (for math support) built in so that people can test/use it if they want. It can be found at https://www.pigeon.digital — please do let me know about any bugs you may find!
there is now a room specifically for the LaTeX fork hosted on www.pigeon.digital where you can leave any feedback, report any bugs, or ask for any help: #pigeon.digital:matrix.org

🔗Pattle 0.11.0 released

Wilko reported:

A new version has been pushed to F-Droid, Google Play and TestFlight!

Pattle is now available on Google Play and the official F-Droid repository!

⚠️ F-Droid users ⚠️: The old repository (fdroid.pattle.im) is gone! Please use the official F-Droid repository.

Expect the new version to be on most platforms in 2 days (however, Google Play in a few hours).

🔗Changes

  • Fix UI not updating on sync!

  • Add ability to send images from your gallery!

  • Use Material Design on iOS (although with slight changes to fit iOS)

  • Reporting errors to Sentry is now optional, you can choose so at login time

    I do urge everyone to enable it to help me develop Pattle!

  • Show a dialog with the option to choose between register and login if the server does not have the ability to check for username availability

  • Show notification count in chat overview

    Default notifications will be grey, highlighted notifications will be red.

    Chats with unread notifications will always show on top.

  • Send read markers and receipts when a chat is opened for 2 seconds

  • Add ability to show password while logging in

  • Pattle will now show up as 'Pattle Android' or 'Pattle iOS' in the device list

  • Decrease spacing between items

  • Fix spacing of messages when sending a new message

  • Numerous other fixes

🔗Get Pattle

🔗Issues or suggestions

If you stumble upon any issues or would like to suggest features, please do so here! You can login via GitHub and Gitlab.com!

🔗Follow development

Follow development in #app:pattle.im!

🔗Support me

If you would like to support me, you can do so via Liberapay and Patreon!

Pattle

🔗'matrixcli', new client enables Matrix as a unix util

Very interesting project: matrixcli is a versatile command line client using the matrix-python-sdk. It's not a conventional IM client, rather it exposes access to Matrix rooms so you can pipe to them as you would other Unix utils.

🔗Continuum progress

yuforia offered:

Continuum

🔗Fractal Progress

Fractal, the GNOME client, continues to progress:

🔗Neo refactoring progress

f0x told us:

Neo's refactoring is almost done, with the latest changes being css-modulesifying the old stylesheet
We're also looking for nice domain name suggestions, to house the soon to be redesigned project page

🔗Dept of Encryption 🔐

🔗libQuotient E2E Progress (GSOC 🎓)

aa13q reported:

The GSoC is soon coming to the end this year and I'm happy no announce that receiving of the encrypted messages PR is here! https://github.com/quotient-im/libQuotient/pull/346 A significant amount of work still needs to be done (session storage, sending messages, device handling, attachments). And I'm interested to implement it after the summer period since I've already dived into the subject :) Here's the small demo of the receiving messages process: https://youtu.be/gGykLh4mVDg

🔗Dept of SDKs and Frameworks 🧰

🔗Matrix-ClientServer-API-java

Jojii created a new Java-based implementation of the Matrix CS API: Matrix-ClientServer-API-java, they call it "A small and simple java API for the Matrix ClientServer Protocol".

They created Matrix-Grep-Bot using the API.

🔗Dept of Ops 🛠

🔗flexible ansible role just for synapse

JCG offered:

For people looking for a rather flexible ansible role just for synapse, https://gitlab.com/famedly/ansible/synapse got your back. You can use it to deploy synapse with pip or docker, depending on what you prefer for your host. It's an actively maintained fork of https://github.com/matrix-org/matrix-ansible-synapse, which has been archived recently and was unmaintained for a while before that. It's just been updated to v1.3.1 too.

🔗New Bridges for matrix-docker-ansible-deploy

Slavi offered:

The matrix-docker-ansible-deploy Ansible playbook has gained new bridging capabilities:

This increases the number of services/protocols that the playbook can setup bridging for to 8: Telegram, Whatsapp, Facebook, Google Hangouts, IRC, Discord, Slack and Email. Learn more in Bridging other networks.

🔗matrix-docker-ansible-deploy gets support for all Synapse config options

Slavi offered:

matrix-docker-ansible-deploy can now harness the full power of Synapse, thanks to our newly-added support for Extensible Synapse configuration.

🔗Dept of Bots 🤖

🔗New /health maubot

Ananace announced:

Just built a quick maubot plugin to add a /health endpoint to maubot, so that database failures and other brokenness can be tracked and dealt with appropriately.

🔗That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Synapse 1.3.1 released

17.08.2019 00:00 — Releases — Neil Johnson

Some of you will have been bitten by a bug that prevented Synapse 1.3.0 from starting up correctly. If this is you, please upgrade.

Additionally we have taken the opportunity to fix a dependency bug for our intrepid packagers.

Apologies for the inconvenience.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Also, check out our Synapse installation guide page

The changelog since 1.3.0 follows:

🔗Synapse 1.3.1 (2019-08-17)

🔗Features

  • Drop hard dependency on sdnotify python package. (#5871)

🔗Bugfixes

  • Fix startup issue (hang on ACME provisioning) due to ordering of Twisted reactor startup. Thanks to @chrismoos for supplying the fix. (#5867)

This Week in Matrix 2019-08-16

16.08.2019 00:00 — This Week in Matrix — Ben Parsons

🔗Matrix Live 🎙

In which Matthew talks about:

Synapse

  • perf speedup
  • installer
  • turn tester
  • extremities fix by default
  • security work
  • privacy work

Riot

  • privacy; IS and IM permissions and management. Hashed lookup API. Metadata cleanup (GCing redactions; deleting media; history retention); TURN management
  • aggregations paused
  • FTUE next up
  • Cross-signing
  • Immutable DMs
  • ...then communities

RiotX

  • 0.3 came out last week - handles DMs; lots of polishing; reduces the APK down to ~10MB!

Bridges

  • moving stuff into proper managed infrastructure
  • bifrost should be back soon?
  • gitter kept breaking but we're actually working with their team to fix that
  • working out how to automate deploying them on modular hosts

🔗Dept of Spec 📜

🔗Approved MSCs

No MSCs have been approved.

🔗Final Comment Period

🔗In Progress MSCs

🔗Dept of Servers 🏢

🔗Ruma

Jimmy reported:

This Week in Ruma: https://ruma.dev/news/this-week-in-ruma-2019-08-11/

He was also a guest of the Rustacean Station podcast this week:

Ruma and the Matrix Communication Protocol: An Interview with Jimmy Cuadra (Rustacean Station podcast): https://rustacean-station.org/episode/001-ruma/

🔗Construct CS API progress

Jason said:

Construct implemented event relations from the client-server API. Now clients can view edit history of messages.

Learn more in #construct:zemos.net

🔗Dendrite

anoa:

Progress for Dendrite this week has been a bit starved due to ongoing privacy work from the backend team. However it continues to move forwards with the excellent help of cnly reviewing and merging PRs.

We had 3 authors have pushed 5 commits to master and 6 commits to all branches. On master, 14 files have changed and there have been 298 additions and 121 deletions.

Notably we had trion129’s work on providing a fallback webpage for recaptcha authentication finally merged! This was another early PR that was dragged up from the bottom of the list to be modernised and integrated into the current codebase.

Also this week was some work from cnly on refactoring gomatrixserverlib and exposing some more of its internal constants to deduplicate code between itself and Dendrite.

Finally, we had some more work on an unmerged PR from cnly, which adds some internal query APIs for which servers are currently joined to a room, for use by other components later down the line.

No new Sytests are passing this week. We’re still currently at 172 tests passing out of 761 tests in total.

🔗Matrixmon updated

Matrixmon, the perl C2S testing utility has been updated to v0.2.0 with support for SSL connections.

🔗Synapse

Synapse 1.3.0 was released - check out the blog post.

contains performance improvements to reduce disk I/O and reduce RAM usage. We’ve been running it on matrix.org for a week or so and are really pleased with the results.

Ananace's k8s tooling, Slavi's matrix-docker-ansible-deploy and andrewsh's Debian packages were all updated within an hour of release, which is a great community achievement. 👏

🔗Dept of Bridges 🌉

🔗Major mautrix-* bridges update

tulir said:

mautrix-telegram got some fixes to bugs that appeared during the move to mautrix-python. It also got a manhole similar to the one in synapse, which gives admins access to an interactive Python shell inside the bridge while it's running. The main part of the manhole (server and repl) is in mautrix-python, so you can expect the feature to appear in mautrix-facebook, mautrix-hangouts and possibly maubot in the future.

mautrix-whatsapp and mautrix-hangouts can now automatically add rooms to a user-specific community to help with filtering. mautrix-facebook is also getting that feature in the near future.

mautrix-facebook now has a search command, which means you can now actually initiate conversation with facebook users.

🔗Dept of Clients 📱

🔗Riot Web

From the team:

  • Privacy work continues
  • STUN fallback server for assisting with voice & video call negotiation (only used when your homeserver is not configured with it’s own TURN server) changed from Google to Matrix.org server and a prompt has been added to request permission before using the fallback server
  • Many small steps towards supporting user choice of identity servers and integration managers, including no identity server at all

🔗Bluepill (Sailfish client) status update

Cy8aer said:

Users can now download artifacts from my gitlab account since I got an SDK container from CoDerus running, cross compiling to Sailfish-RPMs in the Gitlab-Ci. But my programming progress on master looks a bit silent because I swap to matrix-nio.

🔗Fractal

Alexandre Franke told us:

Fractal nightlies are building again.

🔗Continuum

yuforia told us:

Continuum:

  • Reimplement ScrollBar for the message list view to make it more consistent with Riot Web. It has a flat appearance now and only appears when the mouse hovers over the list of messages. https://matrix.org/_matrix/media/r0/download/matrix.org/kVfBsoAsguMmESNThxnUmaTM
  • Rewrite the list view partially to handle lists with several types of items more efficiently. The message list has text messages, images, membership updates, etc. Each type needs to be rendered differently, but items of the same type could reuse GUI components when they scroll into and out of view.

🔗Dept of Ops 🛠

🔗K8s optimized images of Synapse RCs

Ananace offered:

Since the coming 1.3.0 release is bringing along a whole bunch of perf improvements, (and I want to test them out on my IO starved setup) I'm building K8s optimized images of the RCs. Only the debian version of the image though.

They expanded:

It's basically the regular docker image, only with some changes to make it fit nicer in Kubernetes
And there's no Helm chart at the moment, was poking at that for a bit but took a break from building one as I - back then - had trouble with the amount of data that had to be generated
You can only have one Synapse pod at any given time, as it's not scalable. Though you can scale it with workers (as I do myself)
Database-wise, I'm personally running three node HA clusters using Stolon

🔗Dept of Services 🚀

🔗New public instance: Permaweb.io

swedneck said:

We run a public instance on Permaweb.io for everyone to join. We’re a large community of developers focusing on the decentralized web, with lots of channels dedicated to IPFS. In fact, we even host Riot on IPFS! We make full use of Matrix’s bridging capabilities, to Discord, IRC, Slack, Gitter and Telegram too! Come join us at #general:permaweb.io or register at https://riot.permaweb.io/ (or any other compatible client) and say hi!

🔗Dept of Bots 🤖

🔗Notification/Remind-Me-Bot using matrix-bot-sdk

Anton (@antonivan:matrix.org) is working at Matrix Towers this summer for work experience. He's been learning about Matrix and matrix-bot-sdk:

This week I made a Notification/Remind-Me-Bot using Travis's bot sdk, it took me 1 and half days, and is able to - set reminder, check reminder, cancel reminder, set interval reminder, cancel interval reminder. It is also able to store multiple reminders and cancel them using the numbers 1 - ∞ . Here is a link to my Github repo - https://github.com/joakimvonanka/Matrix-Remind-me-bot

🔗Dept of Event Videos 📹

🔗Matrix talk @ FrOSCon

Oleg's talk at FrOSCon was apparently really well received:

Video recording from the Matrix talk @ FrOSCon. Video is in German slides in English.

He's also thinking of writing a guide to creating presentations:

If someone is interested on an efficient way of creating presentation using only open source tools I'm planing to write an article on that.
Small spoiler: matrix is one of them 😉

🔗Final Thoughts 💭

This week, I have been mostly thinking about... merch. We blew the cache on stickers, so if you are waiting for stickers specifically and have not heard from me, that is probably why. Please contact me with any questions!

🔗That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Synapse 1.3.0 released

15.08.2019 00:00 — Releases — Neil Johnson

Well now, Synapse 1.3.0 is here.

The main thing to know about 1.3.0 is that is contains performance improvements to reduce disk I/O and reduce RAM usage. We’ve been running it on matrix.org for a week or so and are really pleased with the results.

Check out our message send heat map.

Message Send

Other than that there are a bunch of bug fixes and tweaks to generally make things run more smoothly.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Also, check out our Synapse installation guide page

The changelog since 1.2.1 follows:

🔗Synapse 1.3.0 (2019-08-15)

🔗Bugfixes

  • Fix 500 Internal Server Error on publicRooms when the public room list was cached. (#5851)

🔗Synapse 1.3.0rc1 (2019-08-13)

🔗Features

  • Use M_USER_DEACTIVATED instead of M_UNKNOWN for errcode when a deactivated user attempts to login. (#5686)
  • Add sd_notify hooks to ease systemd integration and allows usage of Type=Notify. (#5732)
  • Synapse will no longer serve any media repo admin endpoints when enable_media_repo is set to False in the configuration. If a media repo worker is used, the admin APIs relating to the media repo will be served from it instead. (#5754, #5848)
  • Synapse can now be configured to not join remote rooms of a given "complexity" (currently, state events) over federation. This option can be used to prevent adverse performance on resource-constrained homeservers. (#5783)
  • Allow defining HTML templates to serve the user on account renewal attempt when using the account validity feature. (#5807)

🔗Bugfixes

  • Fix UISIs during homeserver outage. (#5693, #5789)
  • Fix stack overflow in server key lookup code. (#5724)
  • start.sh no longer uses deprecated cli option. (#5725)
  • Log when we receive an event receipt from an unexpected origin. (#5743)
  • Fix debian packaging scripts to correctly build sid packages. (#5775)
  • Correctly handle redactions of redactions. (#5788)
  • Return 404 instead of 403 when accessing /rooms/{roomId}/event/{eventId} for an event without the appropriate permissions. (#5798)
  • Fix check that tombstone is a state event in push rules. (#5804)
  • Fix error when trying to login as a deactivated user when using a worker to handle login. (#5806)
  • Fix bug where user /sync stream could get wedged in rare circumstances. (#5825)
  • The purge_remote_media.sh script was fixed. (#5839)

🔗Deprecations and Removals

  • Synapse now no longer accepts the -v/--verbose, -f/--log-file, or --log-config command line flags, and removes the deprecated verbose and log_file configuration file options. Users of these options should migrate their options into the dedicated log configuration. (#5678, #5729)
  • Remove non-functional 'expire_access_token' setting. (#5782)

🔗Internal Changes

  • Make Jaeger fully configurable. (#5694)
  • Add precautionary measures to prevent future abuse of window.opener in default welcome page. (#5695)
  • Reduce database IO usage by optimising queries for current membership. (#5706, #5738, #5746, #5752, #5770, #5774, #5792, #5793)
  • Improve caching when fetching get_filtered_current_state_ids. (#5713)
  • Don't accept opentracing data from clients. (#5715)
  • Speed up PostgreSQL unit tests in CI. (#5717)
  • Update the coding style document. (#5719)
  • Improve database query performance when recording retry intervals for remote hosts. (#5720)
  • Add a set of opentracing utils. (#5722)
  • Cache result of get_version_string to reduce overhead of /version federation requests. (#5730)
  • Return 'user_type' in admin API user endpoints results. (#5731)
  • Don't package the sytest test blacklist file. (#5733)
  • Replace uses of returnValue with plain return, as returnValue is not needed on Python 3. (#5736)
  • Blacklist some flakey tests in worker mode. (#5740)
  • Fix some error cases in the caching layer. (#5749)
  • Add a prometheus metric for pending cache lookups. (#5750)
  • Stop trying to fetch events with event_id=None. (#5753)
  • Convert RedactionTestCase to modern test style. (#5768)
  • Allow looping calls to be given arguments. (#5780)
  • Set the logs emitted when checking typing and presence timeouts to DEBUG level, not INFO. (#5785)
  • Remove DelayedCall debugging from the test suite, as it is no longer required in the vast majority of Synapse's tests. (#5787)
  • Remove some spurious exceptions from the logs where we failed to talk to a remote server. (#5790)
  • Improve performance when making .well-known requests by sharing the SSL options between requests. (#5794)
  • Disable codecov GitHub comments on PRs. (#5796)
  • Don't allow clients to send tombstone events that reference the room it's sent in. (#5801)
  • Deny redactions of events sent in a different room. (#5802)
  • Deny sending well known state types as non-state events. (#5805)
  • Handle incorrectly encoded query params correctly by returning a 400. (#5808)
  • Handle pusher being deleted during processing rather than logging an exception. (#5809)
  • Return 502 not 500 when failing to reach any remote server. (#5810)
  • Reduce global pauses in the events stream caused by expensive state resolution during persistence. (#5826)
  • Add a lower bound to well-known lookup cache time to avoid repeated lookups. (#5836)
  • Whitelist history visibility sytests in worker mode tests. (#5843)

Official Matrix Merch Now Available

09.08.2019 00:00 — Merch — Ben Parsons

Here at Matrix we’re frequently asked for official merch. While those who choose to sponsor Matrix development on Patreon have always been blessed with branded t-shirts, it was otherwise very difficult to obtain official merch.

Today, those dark times have ended, as we announce the Official Matrix Merch Store!

Now available from shop.matrix.org, we’re offering packs of Stickers, T-Shirts, and Hoodies!

Stickers are provided in three types: hexagonal Matrix and Riot stickers, and rectangular transparent Matrix stickers.

stickers

T-Shirts are available in all sizes and in Unisex and Women's editions. We’re using Gildan SoftStyle t-shirts, which are popular amongst printed shirts for being both soft and durable.

T-Shirt

Matrix-branded hoodies are something very special. Superbly embroidered, they are comfortable and stylish. On a recent trip to San Francisco, a passerby stopped a Matrix guardian in the street and asked where they could buy a Matrix hoodie without knowing what Matrix was.

You really need one of these.

Hoodie

All merchandise proceeds go to the Matrix.org Foundation, which helps fund Matrix development!

The Official Matrix Merch Store is open now: shop.matrix.org!

This Week in Matrix 2019-08-09

09.08.2019 00:00 — This Week in Matrix — Ben Parsons

🔗Matrix Live

Something a little different this week: Erik helped me install Prometheus logging on my own homeserver, and we recorded it for Matrix Live.

Interesting reading: https://github.com/matrix-org/synapse/blob/master/docs/metrics-howto.rst

🔗Dept of Status of Matrix 🌡

🔗The Official Merch Store

<blink>YOU NEED MATRIX MERCH</blink>

Yes, it's true: Official Matrix Merch is available now. There are stickers, t-shirts and hoodies all available now, go and claim yours from https://shop.matrix.org.

stickers

<blink>CONSUME</blink>

🔗Free Open Source Conference appearance from Oleg

Oleg:

I will be doing a talk on Matrix this weekend (tomorrow) at the annual Free Open Source Conference (FrOSCon) in Sankt Augustin (Germany). If you are around - come by and get some awesome [matrix] stickers! =)

https://programm.froscon.de/2019/events/2330.html

Visit #FrOSCon2019:froscon2019.fiksel.info to talk about the event.

🔗Dept of Spec 📜

Approved MSCs

[MSC 2134]: MSC2134: Identity Hash Lookups 🎉

Final Comment Period

No MSCs have entered FCP

In Progress MSCs

[MSC 2213]: Rejoinability of private/invite-only rooms [MSC 2214]: Joining upgraded private rooms

🔗Dept of Servers 🏢

🔗Dendrite

Huge update from Andrew:

This week’s stats for Dendrite are in! 4 authors have pushed 11 commits to master and 11 commits to all branches. On master, 48 files have changed and there have been 676 additions and 146 deletions.

Of note are the following merged PRs:

We also have a number of PRs that were moved forward but not merged. Add auth fallback endpoint (our oldest PR at the moment!) which we mentioned last week is so very close, but not quite there in time for this TWIM.

In terms of passing Sytests, Dendrite now passes 172 tests, up from 167 last week. This is out of 761 tests in total. So just a small bump (0.6%), but note that things that are crucial for Dendrite development at the moment, such as federating, only cover a small subset of tests, so each passing one can bring us much closer to getting Dendrite usable day-to-day.

🔗Synapse

From Neil:

The big news this week we’ve landed some massive DB improvements (https://github.com/matrix-org/synapse/pull/5706) which should make message sending on matrix.org (and any other server) feel noticeably snappier. :) It'll be in the next release.

Outside of perf, we’ve also been continuing with our privacy project, sydent now supports hashing third party identifiers instead of storing the raw form, and in the coming week we’ll look at minimising the meta data we store. You can track our progress across the full project here (https://vector-im.github.io/feature-dashboard/#/plan?label=privacy-sprint&repo=matrix-org/synapse&repo=vector-im/riot-web&repo=vector-im/riot-ios&repo=vector-im/riot-android&repo=vector-im/riotX-android&repo=matrix-org/matrix-doc&repo=matrix-org/sydent)

Work continues on our installer to make it easier to configure Synapse and we’ll be looking for some community feedback rsn. The room directory revamp is now very close and we hope to have something live on matrix.org in about a week.

Finally we’re working on improving the efficiency of smaller instances sending messages into large rooms, we’re still at the design stage, but it will make a huge difference for anyone self hosting.

andrewsh also informs us:

synapse in up-to-date in Debian unstable — and in buster-backports (uploaded last week)

🔗Ruma

This Week in Ruma: https://ruma.dev/news/this-week-in-ruma-2019-08-04/

Also, get live-streamed Ruma development: https://www.jimmycuadra.com/posts/twitch-live-streams-of-ruma-development/

Does the video include time spent waiting for async to land in Rust?

🔗signaller - new "lightweight server"

ChronosX88 announced that he is "developing a lightweight Matrix server" - check on their progress at https://github.com/signaller-matrix/signaller, also join #signaller_dev:netwhood.online.

🔗TravisR announces various unstable interesting things

TravisR:

If you fancy running highly experimental software on your homeserver which might break things, I have two things for you:

  1. matrix-key-server ( #matrix-key-server:t2bot.io ) is an implementation of a Matrix key server with notary support. Although it follows the specification, it has not been tested against Synapse. It is however fully featured: check it out on the federation tester ( https://federationtester.matrix.org/#keys.t2host.io ) or by querying it yourself ( https://keys.t2host.io/_matrix/key/v2/query/matrix.org ).

  2. matrix-room-directory-server ( #matrix-room-directory:t2bot.io ) is less experimental than the key server but is still very early days. Currently it only offers the ability to manipulate the federated public room directory for your server, but in future it is planned to be its own standalone directory server (room aliases without having to run a whole homeserver). Check it out by searching the t2bot.io room directory from your client.

🔗Dept of Clients 📱

🔗Continuum progressing!

I love getting updates from yuforia - they've been consistently working on Continuum for some time, and by increments are making a great client.

Reuse the ListView of messages across different rooms to reduce memory usage (Experimental) Remember and refocus the last read message, making it easier to go through all unread messages while switching chat rooms freely.

🔗QMatrixClient is now Quotient

kitsune:

Renaming of QMatrixClient to Quotient has been finally merged to the master branch; Quaternion master uses it from now, too. Packagers are welcome to make test builds and report bugs in #quotient:matrix.org. libQuotient 0.6 beta is coming close now!

🔗Riot iOS

  • Test on iOS 13 Beta. Beta 6 fixed most bugs discovered on Beta 5
  • Released 0.9.2 (Waiting for Apple review at the time of writing)
  • Working on privacy concern.

🔗Riot Android

  • Working on privacy concern.

🔗RiotX (Android)

🔗Dept of Identity 🛂

🔗ma1sd 2.1.1

ma1uta announced ma1sd release 2.1.0:

Changes: reworked unbind (MSC1915).

Now ma1sd is compliant with the MSC1915 (unbind).

and then 2.1.1:

ma1sd release 2.1.1 with the security fix and more strict request validation, allows only requests from ma1sd's matrix domains (mirroring bind). Download links: https://github.com/ma1uta/ma1sd/releases/tag/2.1.1 (archives and the deb package) and https://hub.docker.com/r/ma1uta/ma1sd for docker image.

🔗Dept of Bridges 🌉

🔗mautrix-facebook, mautrix-telegram and the tulirverse

tulir has been making big updates to two of his bridges:

mautrix-facebook can now bridge formatting, mentions, replies and reactions in both directions.

There is also a logout command now.

mautrix-telegram's switch to mautrix-python is nearly finished (i.e. it didn't cause any errors for the past few days when testing in production). The main reason for the switch is using one Matrix library for all my python bridges. It also means the bridging code like double puppeting and command handling I shared between mautrix-facebook and mautrix-hangouts is now also used in mautrix-telegram.

Visible changes directly caused by using mautrix-python:

  • Logs are now colorful.
  • Python 3.5 is no longer supported.
  • The bridge will refuse to start without access to the base config file.

Other changes that happened during the switch:

  • Telegram "Saved Messages" can now be bridged even when using double puppeting.
  • Mentions on Telegram are marked as read when using double puppeting (messages were already being marked as read, but mentions weren't).

Also, this actually happened last week and the week before that, but anyway: I've moved the CI and docker registry of my active projects to a self-hosted GitLab at mau.dev. Specifically, the CI/docker registry for all four mautrix bridges and maubot and automatic builds for gomuks have been moved. My maubot plugins also have automatic .mbp builds in the CI. For the docker registry, prepending dock.mau.dev/ to the existing image names will work. The old places (docker hub, dl.maunium.net) won't get new builds anymore. The repos on GitHub are still the "canonical" repos, but they're mirrored more or less instantly with maumirror.

He adds:

mautrix-telegram will probably get some sort of history filling in the near-ish future

Also,

I'm going to add some way to put bridged rooms into personal communities for filtering purposes. Not yet sure if it'll be fully built into the bridges or some kind of an external script

🔗IRC bridge RC

Half-Shot:

Hi folks, we've released a RC of the next irc bridge version. Check it out at https://github.com/matrix-org/matrix-appservice-irc/releases/tag/0.13.0-rc1

🔗Email2Matrix

Slavi, famed for his ansible playbooks, announced:

I've just released Email2Matrix - an SMTP server that can relay incoming messages over to Matrix based on mappings defined in a configuration file.

While it can also be installed standalone and work with any Matrix homeserver (see its documentation), the simplest way to install it is using matrix-docker-ansible-deploy and its guide for Email2Matrix installation.

The configuration process (users, rooms, mappings) is quite manual, but it's simple and has worked well for me for the past year and a half. I've only just polished it up a bit, released and integrated with the Ansible playbook, so others could benefit from it too.

🔗matrix-xmpp-filter

mijutu appeared, and announced:

I started a "matrix-xmpp-filter" project. It's like matrix-ircd, but with xmpp instead of irc. It can also filter messages by weekday, time and regular expressions. Target audience is Sailfish phone users, but it could be used with other xmpp clients too (no xmpp MUC support required). https://k2c42.dy.fi/matrix-xmpp-filter.git/ #matrix-xmpp-filter:ellipsis.fi.

🔗matrix-github (GH issues bridge)

Half-Shot:

https://github.com/Half-Shot/matrix-github has had quite a few updates. Notably:

  • Support for storing users access tokens securely, using a bot command.
  • Multi-process bridging: The webhook portion of the bridge can be run in a separate function if needed for performance.
  • Images and mentions now work in both directions
  • The readme is now actually helpful, and have also made *.sample.yaml files.
  • You can now use it via oauth

🔗Dept of Ops 🛠

Ananace:

Currently working on a Puppet module to manage Matrix Synapse installs (both directly and through docker), not quite ready for prime-time yet but watch this spot.

🔗Dept of Bots 🤖

🔗Voice to text bot

progserega announced:

I develop matrix Bot, which converting voice messages to text. It use Yandex Speech API and Yandex API cloud (some as aws) for temporary store voice-data before converting. https://github.com/progserega/voice2textMatrix Bot have such logic:

  1. user add bot to room (for example room with whatsapp bridge users - now bridge support adding bot to such rooms)
  2. bot listen room, and when get voice-message - send it to Yandex-cloud for translate (now support only Russian language).
  3. When translating is success - bot get result text and show it in room as notice, such: "Username said: text"
  4. Bot also receive some command, which allow disable it in this room, or disable translating for user, which send command..

Seeing this, I wondered, "y tho?", luckily progserega was able to explain with a graphic! They say a picture paints a thousand words, and it's clear from the conversation below why he'd want to have speech-to-text capabilities.

Speech to Text

🔗Dept of Services 🚀

The Modular Matrix -> Matrix migration tool now supports migration of power levels to the new user.

🔗That's all I know 🏁

So that's all I have to say to you right now! See you next week, and be sure to stop by #twim:matrix.org with your updates!