This Week in Matrix 2021-01-22

22.01.2021 00:00 — This Week in MatrixBen Parsons

Matrix Live 🎙

Dept of Status of Matrix 🌡️

Beeper launched

Eric Migicovsky announced:

NovaChat is now

Beeper

  • New website! beeperhq.com. Beeper is a Matrix server+client combo preconfigured with 12+ bridges. Our mission is to bring a lot of new users into the Matrix ecosystem and give them a fast and powerful client to message people on all networks. Beeper is a paid subscription service, but since all the bridges are open source, you can set the whole system up yourself for free if you choose

  • We're doing a lot of work to boost bridge reliability. Beeper now comes with bridges to Signal, Whatsapp, FB, Slack, Skype, Telegram, Instagram, Hangouts, Twitter, Discord. iMessage coming within a week or two.

  • Sneak peek below of our upcoming desktop UI

Looking for contract work?

  • We are offering bounties of $500-2,000 USD to build new open source bridges based on tulir's mautrix-python or mautrix-go libraries. In particular, we would love to commission Linkedin, MS Teams, Google Chat, GroupMe, WeChat, Snapchat (maybe run it in Anbox?), Line and Email (gmail specifically) bridges.

  • We are also looking for contract iOS and Android developers familiar with the Element codebase to reskin the app with our new UI. Part-time roles with potential for full time.

  • If you're interested, please send me a DM @eric:beeperhq.com

2021-01-22-KJ5nT-room.png

GSOC 2021

Google Summer of Code has come around again! Last year was really successful for Matrix: six projects, with members of the community coming forward to mentor students for their project. This year there is a slight tweak to the formula: the projects will now be 175 hours, rather than the traditional 350, over the summer.

Since we need to submit an application to Google within the next few weeks, I have a request: if you admin a Matrix-related Open Source project and would like to have a student work with you over the summer, get involved!

How to do this:

  • Read Google's own documentation on the setup. This is fairly thorough, and should get you most of the way there.

  • Think about a task, that you consider a student should be able to contribute to your project over a 10-week/175-hour period. Try to come up with 2-3 of these.

    • It should be a well-defined feature that genuinely contributes to the progress of your project, but maybe doesn't block other work on the critical path.
    • This is a very important step, and one that may need some time and discussion.
  • DM me (@benpa:bpulse.org) and we'll talk about next steps.

Dept of Spec 📜

kitsune (aka "fox-avatar guy") reported:

Matrix URI scheme proposal has FCP proposed now - if any of you, the readers of this, have anything to say about it that wasn't said in the past 2 years, speak soon or forever hold your peace. (Ok, you're allowed to write a separate MSC if you come late with revelations.)

This has been an labour of love for kitsune, kudos on his perseverance!

Spec

anoa said:

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.

MSC Status

Merged MSCs:

MSCs in Final Comment Period:

  • No MSCs are in FCP.

New MSCs:

Spec Core Team

We've decided to ditch the focus as it was clear that it wasn't a great indicator of progress. Once one team member has reviewed all of the focus items, they will move on to other MSCs. None of that work is reflected in the focus, and items in there often stick around while waiting on a small number of people to complete work on.

As such, we'd instead like to direct people to the Spec Core Team Backlog board, which is a better representation of the current state of things. And of course the weekly list of MSCs state changes above.

2021-01-22-VYFRA-stacked_area_chart.png

Dept of Servers 🏢

Dendrite / gomatrixserverlib

Dendrite is a next-generation homeserver written in Go

Neil Alexander told us:

At the beginning of this week we released Dendrite 0.3.6, which included a number of refactors in the sync API and a selection of fixes. It is also the first version to be released with multi-arch Docker images with ARMv7/ARM64 targets.

Since then, the following changes have been merged:

  • Sync filtering support for event types, senders as well as better-supported limits

  • Federation support for MSC2946 (Spaces Summary)

  • Fixed a bug where large well-known files could consume a lot of memory

  • Support for in-process DNS caching for federation traffic (although it is disabled by default)

  • Some preparatory work for tracking the most recent membership states in the sync API for future history visibility work

If you are running a Dendrite server, it is highly recommended that you upgrade to the latest version!

Spec compliance is at:

  • Client-server APIs: 60%, up from 58% last week

  • Server-server APIs: 83%, same as last week

As always, feel free to join us in #dendrite:matrix.org for general Dendrite chat or #dendrite-dev:matrix.org for development discussion.

Synapse

Synapse is a popular homeserver written in Python.

callahad reported:

twim:

Synapse

🎉 Synapse 1.26.0rc1 is here!

☠️ But don't install it yet; wait for rc2.

A few issues were reported with rc1 (#9187, #9193, #9208), and we'll issue an rc2 early next week once we're confident they've been resolved.

Otherwise, once 1.26.0 is released you can look forward to a new algorithm for calculating auth chain difference in v2 State Resolution which should significantly speed up some of the most expensive state resolution calculations (#8622).

We've also dramatically improved the performance of redactions in large rooms, helping Synapse's moderation tools scale to communities of any size (#9022). And speaking of scaling, Synapse 1.26 makes it possible to offload many different APIs to worker processes, including read receipts and account data persistence.

Lastly, Synapse 1.26 will include support for multiple Single Sign-On Identity providers, though we expect further fit-and-finish for that feature to land in 1.27.

You can preview the changelog and upgrade notes now, but remember: wait for rc2! 😉

Dept of Bridges 🌉

mautrix-facebook

Tulir said:

The switch to acting as a mobile app I mentioned last week has been merged and it's starting to be clear that it works better. So far logging in has worked fine for everyone at least after enabling 2FA.

Changes this week mostly included tons of bugfixes (thanks to everyone who already updated and found the bugs) and re-adding some old features like read receipt bridging. The SQLite migration script now exists too. Upgrade instructions can be found at https://github.com/tulir/mautrix-facebook/wiki/Upgrading-to-v0.2.0

Dept of Clients 📱

Nheko

Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE and intends to be full featured and nice to look at

Nico (@deepbluev7:neko.dev) told us:

Cheers everyone! On Wednesday we released version 0.8.0 of Nheko! The changelog is far too long, because so many people contributed (and it has been a while since the last release), so I'll just put some highlights here!

Highlights

  • Voice and Video Calls (contributed by trilene)

    • Call your friends right from within Nheko.

    • Use your camera if you want them to see your face!

    • This requires a somewhat new gstreamer, so our builds don't support it on all platforms yet.

  • Cross-Signing and Device/User Verification (contributed by Chethan)

    • Verify who you are talking to!

    • Ensure no malicious people eavesdrop on you!

    • Enable your connected devices to access key backup and your friends to see, which of your devices you trust!

    • Show devices in a users profile.

  • Separate profiles (contributed by lorendb)

    • Run multiple Nheko instances with separate profiles side by side.
    • Use multiple accounts at the same time in separate windows.

Features

  • Before a call select which audio device to use. (contributed by trilene)

  • Auto request unknown keys from your own devices.

  • Add a command to clear the timeline and reload it. (/clear-timeline).

  • Add a command to rotate the outbound megolm session. (/rotate-megolm-session).

  • React to messages instead of replying with arbitrary strings using /react.

  • Inline emoji and user completers. (contributed by Lurkki)

  • Show filename on hover over an image. (contributed by kamathmanu)

  • Mobile mode, that disables text selection and changes some dialogs.

  • Allow sending text after a /shrug command. (contributed by MayeulC)

  • Allow selecting a ringtone. (contributed by trilene)

  • View avatars fullscreen. (contributed by kamathmanu)

  • Request or download cross signing secrets in the settings.

  • Support 'matrix:' URIs. This works in app on all platforms and on Linux Nheko may be opened by clicking a 'matrix:' link.

  • Support inline replies on notifications on Linux.

Packaging is currently in progress, but it will probably take a bit. In the mean time you can download the release from here: https://github.com/Nheko-Reborn/nheko/releases/tag/v0.8.0

Please tell us, what you think and report bugs as you find them! And thanks again to all the contributors! It was a great pleasure to work with you and you really made an impact!

Over and out, the Nheko team!

Adrien added:

https://tracker.debian.org/news/1222370/accepted-nheko-080-1-source-into-unstable/
Nheko 0.8.0 was in Debian before TWIM.

❤️ for the packagers!

fluffychat 0.25.0

FluffyChat is a cute cross-platform matrix client. It is available for Android, iOS, Web and Desktop.

krille said:

fluffychat 0.25.0 is out with a lot of bug fixes, improvements, minor design tweaks and a new 3 column layout. iOS will skip this release until we can bring it to the AppStore next week or so

2021-01-22-Ctpne-bildschirmfotovon2021-01-1620-52-22.png

Element for Nextcloud

Gary Kim said:

Element for Nextcloud v0.6.19 has now been released. The new update contains fixes for Single sign-on users, the ability for admins to easily show or hide the labs settings, and an update to Element Web v1.7.17.

P.S. This will be the last version to support Nextcloud 17 and 18 if you're still on those versions.

Element Web instance on IPFS

TR_SLimey announced:

I've set up an Element Web instance on IPFS, which means that now, not even Matrix client downloads can be blocked - censorship resistance at its finest :P

It can be found at:

or

Element Clients

Updates supplied by the Teams

Web

  • Various widget related improvements
  • Changed guest access to uniformly only allow read access for now
  • Improved CI runtime when testing PRs

iOS

The release we built last week has been blocked due to push notifications issues. We are still on the case On the e2ee side, we have been working on an optimisation for the key re-share mechanism

Dept of SDKs and Frameworks 🧰

caridina, Crystal library

erdnaxeli reported:

Today a present you a Matrix client library I've been working for the last few months: caridina.

It is a Crystal library. Currently for reading it implements the sync API and supports the events to do text messaging (all state events + m.room.message). For writing it provides methods to send messages, edit them, and send receipts and typing notifications.

While I would not recommend this library to write a full featured client, it is enough to write a simple text bot. I currently use it for some of my own projects. The README provides some examples of how to use the library.

Feel free to try it, report bugs, ask for missing features if you need them or even contribute!

libkazv (and the Kazv Project)

tusooa reported:

libkazv is a sans-io C++ (gnu++17) client library

built upon lager. Along with it there is kazv, a kirigami/qml client, and a

forward bot between matrix and tencent qq. Talk to us on #kazv:tusooa.xyz.

Features

(Copying from readme:)

  • Logging in

  • Receiving room states

  • Receiving room messages

  • Receiving account data

  • Receiving presence

  • Sending messages

  • Send room state events

  • Create rooms

  • Room invites

  • Join rooms

  • Typing notifications

  • Receipts and fully-read markers

  • Leaving and forgetting rooms

  • Content repository

  • Send-to-device messages

  • E2EE (send and receive events only)

What goes here?

This week we ("tusooa and her longcat," we hope someday we can remove this annotation :P)

implemented sending and receiving encrypted events in rooms. Check a demo video below:

Opsdroid

Cadair said:

TWIM:

opsdroid

opsdroid is an easy to use chat-ops framework with excellent Matrix support.

0.20 Release

This week opsdroid released version 0.20. This is a massive release that's been a long time coming because of a complete rewrite of the internals of the matrix connector to use matrix-nio---thanks to @awesome-michael for the initial work on this.

Having ported the connector to nio, our GSOC student @Tyagdit got to work polishing the nio port and reimplementing the matrix database module in the core opsdroid library. Opsdroid has a concept of memory which is a key-value store which can be backed my many different kinds of databases. The matrix database stores the opsdroid memory in room state, meaning your bot's database and messages are all stored in the same place.

Finally, the other big feature added to the matrix support in this release is End to End Encryption support. Thanks to using matrix-nio as our matrix library it was possible to implement support for encrypted matrix rooms. This also includes support in the matrix database for encrypted memory which works by referencing event ids in room state (see here for more info). As this is the first release to support E2EE I would still consider it "beta"; please open issues if you have any trouble using it.

Highlights

  • End to End Encryption support in the Matrix connector.

  • Matrix database now included in core, with e2ee support.

  • Use of Python's install "extras" to allow for installations with fewer packages.

  • Improved documentation of the matrix connector and database (at least a little).

  • Support for Python 3.9 (and dropped support for 3.6).

Checkout the full release notes for all the gory details.

If you have any questions or want to get involved come say hi in #opsdroid:matrix.org, raise an issue on GitHub and checkout the docs.

trixnity (kotlin) progress

Benedict said:

A little sneak peek from trixnity, a crossplatform Matrix SDK. Currently I have some issues with ktor, which breaks my serialization.

trixnity

quotient

kitsune reported:

libQuotient has initiated transition from LGPL 2.1 to LGPL 3 as its distribution license. This is partially driven by the fact that newer Qt is (mostly) distributed under LGPL 3 so the "combined work" (legalese for libQuotient including and linked with specific Qt pieces) would be covered by LGPL 3 anyway. Boring, right?

kitsune added:

Those looking at my contributions to Quotient might have noticed the change of the primary committer's name. Don't fret, this is still me, I just shed the pseudonym (and those who've seen the interview knew my real name already).

Watch "the interview" here, where we discuss (then) progress on MSC2312 (mentioned earlier), and plenty more.

Dept of Ops 🛠

Matrix Corporal

matrix-corporal manages your Matrix server according to a configuration policy

Slavi reported:

Version 2.0 of matrix-corporal has been released.

It's a huge release bringing many improvements (read all of it in the release notes).

Here's a short summary:

  • an Event Hooks system has been introduced, so matrix-corporal can now also act like a "generic firewall" (similar to what mxgwd was doing)

  • device-free login is now used when impersonating users (thanks to a new API introduced by Synapse v1.24.0) - this is both faster and more resilient.

  • support for Interactive Authentication (and thus, End-to-End-Encryption) has been added. Users can now set up E2EE keys, manage devices, etc. (actions that require Interactive Authentication and were previously broken). Requires cooperation between matrix-corporal and the REST auth password provider module.

  • End-to-End-Encryption control. matrix-corporal can enforce whether rooms that users create are encrypted or unencrypted. You can force-disable or force-enable encryption for locally-created rooms.

  • fixes user-creation regression that happened with Synapse v1.24.0. matrix-corporal v1.12.0 is also out, with a backport of this fix (for users who don't want to jump to 2.0 yet).

Dept of Interesting Projects 🛰️

Matrix Notepad

KB1RD reported:

  • It's not dead! yet

  • There's been some code refactoring on the algorithm. Mainly, the search tree used has been split into a different package

  • I'm working on porting the Notepad to the new widget API. Ongoing work is on the nextgen branch and is tracked by this project. The widget is currently blocked on some stuff in the widget API, so I probably won't be releasing soon. The code in there needs a ton refactoring, too.

  • This means that the old notepad client will disappear on the next release and you'll need to add a widget for the Notepad for any documents that you want to edit. Currently, this means you would have to use Element, but I heard Ditto just might be doing some widget work

  • The widget API is planned to do everything and more that my "mxapps" project was supposed to do, so that project will be deprecated in favor of using the widget API

  • There's also a little demo/teaser video showing the old notepad talking to the new one. You can see the rate limiting issues that the Notepad hits there, which is on my todo list.

Dept of Guides 🧭

Superb documentation from TU Dresden, as they roll out their deployment

Marvin Dropp said:

doc.matrix.tu-dresden.de

We set up a bilingual (de, en) documentation for all of our university members, to help them get in touch with our matrix instance easily. This documentation is rich on screenshots, simple explanations and will be improved continuously.

Also we mirroring the documentation to GitHub to provide access of our sources to everyone.

(https://github.com/matrix-tu-dresden-de/Dokumentation)

We would appreciate to add further institutions of higher education with own matrix instances to our list.

Viva la federation!

Christian Bruchatz shared some additional context about their deployment growth:

2021-01-22-wC6rz-image.png

Watch Matrix Live (at the top) this week for a chat with Marvin and Christian.

Deploying a Matrix server on Fedora CoreOS, from fedoramagazine.org

Andy announced:

I couldn't find if someone else shared this, but Fedora just published an article about deploying a Matrix server on Fedora CoreOS.

https://fedoramagazine.org/deploy-your-own-matrix-server-on-fedora-coreos/

Matrix in the News 📰

EMS-Slack Bridge coverage

kim ([email protected]) told us:

EMS-Slack Bridge (matrix-appservice-slack) in German IT news

https://www.heise.de/news/Dezentraler-Firmenchat-mit-Slack-und-Matrix-5030856.html

The launch was also mentioned in The Register, in the UK. I used to read the Register every day...

Beeper news coverage

Beeper has had substantial coverage on their launch, much of it focused especially on the iMessage bridging feature. Some articles are linked below:

  • https://www.golem.de/news/beeper-universelle-chat-app-startet-betaversion-2101-153605.html
  • https://www.heise.de/news/Beeper-Chat-App-will-15-Messenger-in-einem-Interface-vereinen-5032049.html
  • https://hothardware.com/news/beeper-app-imessage-android-routing-trickery
  • https://www.macrumors.com/2021/01/21/beeper-brings-imessage-to-android-and-windows/
  • https://9to5google.com/2021/01/21/beeper-app-pebble-founder-imessage-android/
  • https://www.theverge.com/2021/1/21/22242143/beeper-universal-chat-app-imessage-whatsapp-signal-telegram-pebble-founder
  • https://www.protocol.com/beeper-messaging-app

Lots of people helped collect these links in #TWIM:matrix.org, thanks all!

Final Thoughts 💭

sorunome offered:

today is the four year m.anniversary of the @sorunome:sorunome.de account! 🎉

Some interesting reads of the last few days:

Cos announced:

btw i forgot to TWIM that there was once again long Matrix article in Finnish computer culture paper magazine Skrolli. December issue.

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1envs.net429.5
2maescool.be433
3midov.pl473
4juniorjpdj.pl545.5
5kawaiilo.li650.5
6fab.network768
7matrix.thedisco.zone810
8matrix.sp-codes.de843
9dendrite.neilalexander.dev976.5
10aria-net.org1111

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

This Week in Matrix 2021-01-15

15.01.2021 00:00 — This Week in MatrixBen Parsons

Matrix Live 🎙

Dept of Servers 🏢

Dendrite / gomatrixserverlib

Neil Alexander announced:

Earlier this week we released Dendrite 0.3.5, our first release of 2021, which includes a fairly significant refactor of parts of the sync API, along with other fixes. More work has since been merged into master too.

Changes in the last two weeks include:

  • Sync streams are now logically separate, with a lot of behaviour fixed

  • Forward extremities are now deep-checked properly, which should significantly reduce the peaks of CPU and RAM when handling the current room state

  • Pagination tokens in /messages have been fixed, which should improve the reliability of scrolling back in the timeline

  • A number of fixes have been made to the /sync response, largely avoiding nulls, which should make some clients happier

  • Send-to-device messaging has been refactored, which should improve E2EE stability

  • Well-known/DNS records for federated servers are now cached rather than performing lots of lookups, speeding up outbound federation

  • Device list requests to remote servers now time out quicker, so as to not block /send transactions

  • Experimental support for MSC2946 has been merged (gated behind the mscs configuration)

  • Request context is no longer reused for /send, which should help in cases where the remote sending server gives up waiting or the connection breaks

If you are running a Dendrite server, it is highly recommended that you upgrade to the latest version!

Spec compliance is at:

  • Client-server APIs: 58%

  • Server-server APIs: 83%

As always, feel free to join us in #dendrite:matrix.org for general Dendrite chat or #dendrite-dev:matrix.org for development discussion.

I asked:

Can you explain something: Dendrite federation is basically working fine, AIUI, what does it mean that Server-server APIs is <100%?

Neil helpfully replied:

There are still some edge-case tests that we haven't got passing yet, but I think that's only ~20 tests or so.
There are also a few tests which are quite sensitive to exactly what Synapse does/returns, even though Dendrite is probably doing the right thing but maybe taking a different amount of time to do it or returning something slightly different, so there'll be some tests we need to fix too

Synapse

Synapse is a popular homeserver written in Python.

callahad announced:

Happy Friday!

  • 🎉 Synapse 1.25.0 is out! It includes a pretty significant speedup for state resolution in bridged IRC rooms, as well as the usual assortment of bug fixes and improvements.

  • 🕵️ For a limited time, and thanks to funding from the European Commission, paid bug bounties are now available for Synapse and other Matrix.org projects via the Intigriti bug bounty platform.

  • 👋 If you're wondering about the face behind these updates, scroll up and check out Matrix Live!

We're on track for 1.26.0rc1 next week, which includes a massive speedup for message redaction in large rooms, as well as finally landing fundamental algorithmic improvements to state resolution.

Otherwise, we remain hard at work preparing for FOSDEM-on-Matrix, with special emphasis on improving our stability and moderation capabilities at scale. See y'all next week!

Homeserver Deployment 📥️

Kubernetes

Ananace offered:

Just pushed version 1.25.0 of Synapse for my Kubernetes image and chart. Haven't quite had the time to finish up the redo of the scripts, but expect that 1.25.0 will be the last version with a specific image, I'm hoping to have generalized the chart enough that it should work with any reasonable image at that point.

Dept of Bridges 🌉

Gitter

Eric Eastwood offered:

A few improvements to the Gitter bridge this week so that when a room updates and the avatar or topic changes, it now propagates across to Matrix automatically. We also handle updating the name whenever we have to rename a room or a group.

We're also thinking about how best to tackle self-service plumbing so you can pipe a Gitter room into your existing Matrix room. And the bigger idea of connecting various bridged portal rooms together. So you can connect your community on Gitter <-> Matrix <-> IRC seamlessly for example. The current thinking is introducing this as a native Matrix concept so you can easily connect any Matrix room to another Matrix room. We'd love to hear your thoughts in MSC2923.

mx-puppet-groupme

Robin said:

Version 2 of mx-puppet-groupme is here! This week I did some reverse engineering of the GroupMe Android app with mitmproxy so that I could work out the remaining undocumented features of GroupMe's API. As a result, typing notifications, read receipts, and videos are now working! Though note that due to limitations of the platform, read receipts are a DMs-only feature.

We also now have a Dockerfile (thanks, Trey B.!), double puppeting support was added, the code was ported to Typescript, and it should now work with older versions of Node. So at this point the bridge is basically feature-complete! As usual, send any questions, bugs, or feature requests to GitLab or Matrix. 💜

mautrix-facebook

Tulir told us:

I've been working on moving mautrix-facebook to act as a Messenger mobile app instead of the web app in order to hopefully make it more reliable. The initial version is starting to work now.

It's currently in the mobile branch. I'm not actually sure if it helps with facebook forcing password resets (for some reason they've never done that to my account), but if it does, I'll merge it into master in the near future.

The new version only supports Postgres like my newer bridges, but I'll invent a SQLite migration tool before merging to master.

Dept of Clients 📱

NeoChat 1.0.1 released

Carl Schwan told us:

NeoChat 1.0.1 was released with a few important bugfixes. https://carlschwan.eu/2021/01/13/neochat-1.0.1-first-bugfix-release/. On the unstable branch, we continued to improve the integration with Plasma. We now display the unread count in the taskbar and we are getting inline replies too thanks to the work of Kai Uwe Broulik in Knotifications. Another important change is that rooms can now be opened in a new separate window.

2021-01-15-1m-hE-image.png

2021-01-15-N_xQv-image.png

Nheko

Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE (with the notable exception being device verification for now) and intends to be full featured and nice to look at

Nico (@deepbluev7:neko.dev) offered:

Bugfixing for the next release is in progress, fixing various issues and instabilities introduced with the new features, but some smaller features also still made it in:

  • There is now a call invite screen more fitting for mobile devices, when you enable the "Touchscreen mode" in the settings.

  • Various clickable elements now have a ripple effect again to give feedback, when a click was registered.

  • Nheko now handles the Matrix URI scheme. This means opening a link starting with matrix: should open Nheko (at least on Linux) and redirect to the appropriate room and matrix URIs inside of Nheko are also handled. This currently does not support navigating to a specific event, but opening user profiles, starting chats with specific users and opening or joining rooms should work. Nheko also does not automatically linkify links starting with matrix:, so you need to do that manually.

  • Verification dialogs were cleaned up a bit and hopefully are easier to understand now.

  • The flatpak nightlies now support VOIP calls, but as a result are now a few MB more to download.

Since we are nearing the next release, it would be appreciated, if you could check the translations for your platform are up to date and try the current nightlies and check them for bugs your experience and want to have fixed before release. If you find anything, please report it on GitHub or discuss it in #nheko:ocean.joedonofry.com. Thank you!

Quaternion

kitsune told us:

I'd like to have announced the next beta release for Quaternion 0.0.9.5 - unfortunately, migrating from Travis CI took more than I wanted; but adventurous souls are welcome to test the master branch that's basically ready for beta 3.

Meanwhile, it surfaced that Quaternion 0.0.9.4e Windows binaries have been expecting (very old and no more supported) OpenSSL 1.0. This is now fixed in Quaternion 0.0.9.4f (hopefully this will be the last 0.0.9.4 rebuild and I won't chase the above-mentioned OpenSSL project in trying to expire Latin letters).

Element Clients

Various updates from the teams:

Spaces

On Web, we’re progressing on Spaces on our big checklist of doom, most recently on implementing invites to Spaces. On Android, we’re also progressing, iterating on the UX. Meanwhile, we’re also experimenting with different ways to explore, manage and navigate nested spaces.

Social Login

We’ve now merged support for multiple identity providers on Synapse, and are getting dangerously close to finishing the rest of the implementation. Watch this space for more news soon!

Web

Element Web 1.7.17-rc.1 is now available at https://staging.element.io, including:

  • Fixed avatar upload prompt layering issues
  • Added VoIP call transfer

iOS

Element iOS 1.1.4 has been submitted to the AppStore. A TestFlight build will be available during the week-end. Main things the release offers are:

  • Social login
  • New SSO login management
  • Several bug fixes

Dept of SDKs and Frameworks 🧰

libQuotient

kitsune announced:

Version 0.6.4 is out, with a few fixes around homeserver resolution (particularly when a .well-known record is not there). These has been made in tight collaboration with the folks behind Neochat - thanks a lot! 0.6.4 is also the best version to build the just-released Neochat 1.0.1 with.

Meanwhile, work on the next version (0.7) is proceeding in the unstable branch (not in the least pushed by Neochat activity) - expect more news in the next weeks.

Ruma

Ruma is a set of Rust library crates around Matrix

jplatte announced:

Since our last update on 2020-11-20,

Dept of Ops 🛠

PingPong: End-to-end latency monitoring for Matrix

p-e-w announced:

PingPong measures transport latencies on Matrix networks. It connects to two Matrix accounts simultaneously, and bounces messages back and forth between them. It aggregates all information in an intuitive terminal user interface, and automatically computes statistics. Source code and more information are available at https://github.com/p-e-w/pingpong.

I have been working on this for a while now and I believe it is ready for others to use at this point. No binary releases yet, the program must be built from source. I only have a Linux development system currently, so feedback on whether it works on macOS and especially Windows is very welcome.

2021-01-15-mZQHt-screenshot.png

Very interesting new project! I'm thinking of setting this up as I'd planned to write something similar to track perf on bpulse.org.

Dept of Interesting Projects 🛰️

Keymaker Serverlist Project

MTRNord said:

A small update on what is happening after a long time of nothing:

  • Bot has first part of the registration process (essentially the required automated tests) implemented. Next is the manual verification half.

  • Domains are bought and the web page is deployed (No servers listed yet)

  • Code of Conduct writing will start soonish.

As you will see there are only empty categories. This is due to the registration bot still being in work as well as missing documentation for it.

For the progress you can also take a look at https://github.com/keymaker-mx/keymaker/projects/1

Check the current page out at: https://homeservers.mx https://joinmatrix.rocks/

Join development and discussions at #serverlist:nordgedanken.dev Check out the Code at: https://github.com/keymaker-mx

2021-01-15-HNiKj-Screenshot_20210113_225623.png

This is really awesome progress! Not quite there but we'll keep a close watch. 👀

Dept of Guides 🧭

e2ee implementation guide

sorunome told us:

Soru worked on updating her e2ee implementation guide, so far it contains a new section on bootstrapping and hopefully soon on online key backup! You can find the WIP MR here.

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1maclemon.at318
2nicoll.xyz366
3imninja.net457
4aragon.sh637
5kif.rocks665
6matrix.sp-codes.de727.5
7matrix.vgorcum.com766
8envs.net775
9fairydust.space820
10mtx.liftm.de833

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Synapse 1.25.0 released

13.01.2021 00:00 — ReleasesDan Callahan

Synapse 1.25.0 is now available! With this release, we are deprecating Python 3.5 and PostgreSQL 9.5 and will cease producing binary packages for Debian 9 (Stretch) and Ubuntu 16.04 (Xenial) after a transition period which lasts through March 2021. See the changelog for further details.

We are also deprecating the Purge Room and Shutdown Room Admin APIs and will remove them in a future release. Please update your code to use the Delete Room Admin API instead.

Synapse 1.25.0 brings over a month's worth of improvements, including:

  • The ability for users to pick their own username when using Single Sign-On, right from within Synapse.
  • Support for async Python methods in custom spam checker modules.
  • New ways to restrict allowed IP address ranges for outgoing requests from Synapse.
  • Significantly faster v2 state resolution on rooms with large numbers of power level events, which are common in some types of bridged IRC rooms.

See the full changelog and upgrade notes for more.

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including @aaronraimist, @Bubu, @dklimpel, @edwargix, @fossterer, @jdreichmann, @jerinjtitus, and @MadLittleMods.

This Week in Matrix 2021-01-08

08.01.2021 00:00 — This Week in MatrixBen Parsons

Matrix Live 🎙

Dept of Status of Matrix 🌡️

FOSDEM 2021

Matthew reported:

FOSDEM 2021 is going to happen via Matrix: https://matrix.org/blog/2021/01/04/taking-fosdem-online-via-matrix

Dept of Spec 📜

New spec platform

wbamberg announced:

The spec core team has continued trying out the new spec (preview at: https://adoring-einstein-5ea514.netlify.app/). We've also finished applying design updates and implemented a fancy scrolling table of contents.

We have an outline for how to switch over to the new platform: https://github.com/matrix-org/matrix-doc/issues/2906 and have started work on that this week.

Dept of Servers 🏢

Synapse

Synapse is a popular homeserver written in Python.

callahad said:

Welcome back, TWiM readers! It's a new year, and the Synapse team is excited to get back to our regularly scheduled releases! Three main things this week:

  1. We published 1.25.0rc1! We'll have full release notes next week, but highlights include:
  • Deprecated Python 3.5 and Postgres 9.5 per #8782; more details to come with the formal release announcement.

  • People can pick their own username when using Single Sign-On, right from within Synapse itself.

  • Spam-checker modules can now use async Python methods.

  • New ways to restrict allowed IP address ranges for outgoing requests from Synapse.

  1. Matrix is hosting FOSDEM! Which means that most of the Synapse team is going to be focused on security, stability, and performance until February to ensure that our virtual Université Libre de Bruxelles is as reliable and welcoming the real thing.

  2. Speaking of security, we have some big (but good!) news to announce next week. Watch this space... 🕵️🇪🇺💶

Lastly, one of our goals for this year is to maintain a biweekly cadence of release candidates, so we should have a more consistent pace. At the same time, we realize that keeping up with biweekly releases of server-side software can be a bit of a chore, so we're also discussing ways that we could provide longer support for some of our releases. That's a project for much later this year, but consider it a preview of things to come.

If you have any feedback, please feel free to join us in #synapse:matrix.org!

Dept of Bridges 🌉

mx-puppet-steam

Icewind told us:

mx-puppet-steam has been receiving a bit of love over the past few weeks, including:

  • improved reliability of image uploads

  • syncing of read and typing status from matrix to steam

  • bridging of emotes and stickers from steam to matrix

Dept of Clients 📱

New NeoChat features

Carl Schwan announced:

NeoChat gained a few new features this week. The timeline displays stickers now, and edited messages don't get duplicated anymore. NeoChat also now lets you edit your messages. Additionally, Noah improved the support for attachments in the UI, more image formats are detected (and can then be edited in the built-in image editor), and mime type icons get added to non-image attachments. Tobias continues to work on the registration flow and SSO support.

You can meet Carl and Tobias from the NeoChat project by watching Matrix Live. \o/

2021-01-08-mp8Ia-image.png

Nheko

Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE (with the notable exception being device verification for now) and intends to be full featured and nice to look at.

Nico (@deepbluev7:neko.dev) reported:

  • d42 fixed SSO login with some more exotic SSO configurations.

  • Reenable showing a users status_msg, which got lost in a refactoring.

  • trilene cleaned up the design of our call dialogs and Nheko now picks up changes in your devices without a restart.

  • Typing notifications should now not flicker anymore.

  • The build is now more explicit, if call support is enabled.

  • You can now reply directly inline to a notification on Linux, if your notification manager supports it.

2021-01-08-0x46S-Screenshot_20210107_183454.png

This reply-in-notification feature is really cool!

SchildiChat for Android

SpiritCroc offered:

SchildiChat is a fork of Element, which focuses on UI changes such as message bubbles and a unified chat list for both direct messages and groups, which is a more familiar approach to users of other popular instant messengers.

During the last couple of weeks, SchildiChat-Android development focused mainly on staying up-to-date with the upstream Element codebase (URL previews took some time to make them fit nicely into the bubble layouts), and doing some smaller design improvements (like wider message bubbles for some scenarios, and an intelligent message timestamp placement at the bottom of the bubble, depending on the available space).

Furthermore, we now also have experimental support for MSC2867, which allows you to mark rooms as unread. Note that for now, this feature needs to be enabled in the labs settings, since the MSC is still unstable and not supported by many clients, which means other clients might ignore whether the user has manually marked a chat as unread and thus display it as read anyway.

Code on GitHub: https://github.com/SchildiChat/SchildiChat-android

Matrix room: #schildichat-android:matrix.org

Element Clients

Some updates below from the team. Not all teams are back from their Christmas breaks yet, and those who are may be a little busy with FOSDEM-specific projects, but we have some updates:

Spaces

We’re making more progress on the client implementations for Spaces on the web on Android. On Web, recent progress includes implementing notification badges, and on Android we’re progressing from SDK work to initial UI implementations.

Social Login

Support for Social Login on iOS is in review after resolving some issues around VoiceOver, which is the last Element client to implement support ahead of Synapse landing support for social login in the near future.

VoIP

Lots of things merged to web: dial pad, call forward, dtmf pad, although most won’t show up in normal use (yet). Other progress on web on hold for fosdem work. Dial pad & phone number lookup support incoming on iOS, and phone number lookup on its way on Android.

iOS

We progressed on several things this week. They are not yet merged but almost. They will be part of the next release we will ship next week: social login, app deadlock fix, encrypted message sending speed improvement, xcodegen usage, sending bug reports in background.

Dept of SDKs and Frameworks 🧰

MRSBFH - Matrix-Rust-SDK-Bot-Framework-Helper

MTRNord reported:

https://github.com/MTRNord/mrsbfh

I took the time and started a small (command)bot framework/utility crate which operates on top of the matrix rust sdk.

It currently is just some of the command logic and utilities extracted from the famedly timetrackingbot and offers a basic bot template.

Features

  • Proc macro based command definition

  • Auto generating of helptext

  • Full compatibility with the regular matrix rust sdk

  • Modularity

  • Helpers for session recovery and Configs

  • Helpers to minify boilerplate of the EventEmitter

Planned features

  • More utilities for bots that are not in scope of the main sdk.

Non-Goals

  • State management

  • Hiding the sdk behind another API (No automatic hooks into the event emitter)

  • usage as AS framework. (This however is possible as you can use the hook where ever needed.)

Possible goals

  • Early adoption of MSCs that are meant for bots (for example MSC2929)

Note this is very young and many is possible to change.

That is quite a project name!

mtxclient - the Matrix library Nheko uses

Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE (with the notable exception being device verification for now) and intends to be full featured and nice to look at.

Nico (@deepbluev7:neko.dev) announced:

I found some time to actually host the docs our library currently has. It is still severely lacking, but it may be interesting to some people like people contributing to Nheko. I'm going to write some higher level docs over the next few weeks, so that people can actually see how to use the library and have some inline examples and more extensive explanations. But for now you can find the barebones descriptions we always had in the source code here: http://nheko-reborn.pages.nheko.im/mtxclient/index.html

Providing docs always gets a big thumbs-up from me!

matrix-bot-sdk

TravisR said:

[email protected] has been released! This version contains early support for Identity Servers and Spaces (MSC1772), as well as easier functions for sending HTML messages and a bunch of other quality of life improvements. Feedback and bugs in #matrix-bot-sdk:t2bot.io.

Trixnity, Kotlin SDK announced

Benedict reported:

I'm currently working on a Kotlin cross-platform (JVM, JS, Native) Matrix SDK named trixnity

It's a very early version, but I can migrate many code and tests from matrix-spring-boot-sdk so that it's growing fast (for a one man project 😀).

Benedict explained:

matrix-spring-boot-sdk is my first attempt to write bots and appservices really fast, then matrix-sms-bridge using this sdk and now migrating the low-level stuff from matrix-spring-boot-sdk to trixnity, so it can be used independently from spring boot 🙂 My plan is to use trixnity for a an open source web client, that can do "Videosprechstunde" in Germany (I don't know the english word for it, maybe video doctor's consultation?).

More about Videosprechstunde later...

Dept of Ops 🛠

GitHub action for Matrix!

select said:

I recently started a new freelance project and got them to use Matrix as our main communication channel. Since they chose GitHub as our source code platform I got into writing continuous integrations and delivery scripts. But one thing was missing: notifications in our Matrix room. I looked into the GitHub Action Marketplace and found 2 actions that could do that, so that was nice ... but they could not send e2e encrypted messages. Therefore I took up my old issue on how to make e2e encryption work with the js-sdk (https://github.com/matrix-org/matrix-js-sdk/issues/731) with some new found energy and finally solved it. The result is this GitHub action:

https://github.com/select/matrix-message-e2e

While it's almost a full success story there is still one issue to solve: While the action successfully sends e2e encrypted messages the job is marked as failed. My guess it that while sending the messages the matrix client outputs on stderr due to some encryption errors (unknown device keys, ...)

Dept of Interesting Projects 🛰️

Battleship via Matrix

Christian announced:

Want to play the guessing game Battleship via Matrix? I'm building one to be ready for FOSDEM. Follow my progress, bring in ideas and play development versions:

https://matrix.to/#/#battleship:vector.modular.im?via=vector.modular.im&via=t2l.io&via=federator.dev

matrix-archive

I asked: "what's the best tool for dumping history from rooms (including e2e rooms)?", and Florian shared this great tool I don't think I'd seen before:

matrix-archive is the best currently-maintained tool I know of

A really useful project that generates "a YAML log of all room messages, including media".

Dept of Guides 🧭

easy-to-share comparison chart

joepie91 said:

Based on the original chart by hrjet, here's an easy-to-share comparison chart in image form, that compares a number of alternatives (Element/Matrix, Signal, Telegram) for people looking for an alternative to WhatsApp

2021-01-08-ayRQd-whatsapp-alternatives.png

Should be very useful for incoming "What do I use instead of WhatsApp?"-type conversations!

(Also you could share this tweet for even more visibility!)

Final Thoughts 💭

Videosprechstunde is quite hot right now

As promised, some more thoughts on the Videosprechstunde craze apparently sweeping Germany.

Niklas Zender announced:

Videosprechstunde is quite hot right now, and the technical requirements in terms of security are already handled with matrix.
It is regulated by the german government and is only working for centralised services right now. So there is no way to get it certified with matrix. It might be possible in future (fingers crossed) and then we (Famedly) are also ready to provide it :)
(Government is slightly wrong, it is more the KBV, but the Government also plans to regulate it starting in two years)

Videosprechstunde refers to having a video conference with your doctor, rather than going in-person.

Speed installations?

compu offered:

so one time I installed Matrix Synapse on a fresh debian install in 24 minutes do i get bragging rights?

If you have video of you installing Synapse or another matrix server at an alarming rate, we'll link to it from here!

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1matrix.org347
2the-apothecary.club351
3almum.de399
4envs.net485
5maescool.be634.5
6aria-net.org712
7casavant.org891
8mtx.liftm.de904
9cof100.dk1067
10tchncs.de1109

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Taking FOSDEM online via Matrix

04.01.2021 12:25 — GeneralMatthew Hodgson
FOSDEM

Imagine you could physically step into your favourite FOSS projects’ chatrooms, mailing lists or forums and talk in person to other community members, contributors or committers? Imagine you could see project leads show off their latest work in front of a packed audience, and then chat and brainstorm with them afterwards (and maybe grab a beer)? Imagine, as a developer, you could suddenly meet a random subset of your users, to hear and understand their joys and woes in person?

This is FOSDEM, Europe’s largest Free and Open Source conference, where every year thousands of people (last year, ~8,500) take over the Solbosch campus of the Université Libre de Bruxelles in Belgium for a weekend and turn it into both a cathedral and bazaar for FOSS, with over 800+ talks organised over 50+ tracks, hundreds of exhibitor stands, and the whole campus generally exploding into a physical manifestation of the Internet. The event is completely non-commercial and volunteer run, and is a truly unique and powerful (if slightly overwhelming!) experience to attend. Ever since we began Matrix in 2014, FOSDEM has been the focal point of our year as we’ve rushed to demonstrate our latest work and catch up with the wider community and sync with other projects.

This year, things are of course different. Thankfully FOSDEM 2020 snuck in a few weeks before the COVID-19 pandemic went viral, but for FOSDEM 2021 on Feb 6/7th the conference will inevitably happen online. When this was announced a few months back, we reached out to FOSDEM to see if we could help: we’d just had a lot of fun helping HOPE go online, and meanwhile a lot of the work that’s gone into Matrix and Element in 2020 has been around large-scale community collaboration due to COVID - particularly thanks to all the development driven by Element’s German Education work. Meanwhile, we obviously love FOSDEM and want it to succeed as much as possible online - and we want to attempt to solve the impossible paradox of faithfully capturing the atmosphere and community of an event which is “online communities, but in person!”... but online.

And so, over the last few weeks we’ve been hard at work with the FOSDEM team to figure out how to make this happen, and we wanted to give an update on how things are shaping up (and to hopefully reassure folks that things are on track, and that devrooms don’t need to make their own plans!).

Firstly, FOSDEM will have its own dedicated Matrix server at fosdem.org (hosted by EMS along with a tonne of Jitsi’s) acting as the social backbone for the event. Matrix is particularly well suited for this, because:

  • We’re an open standard comms protocol with an open network run under a non-profit foundation with loads of open source implementations (including the reference ones): folks can jump on board and participate via their own servers, clients, bridges, bots etc.
  • We provide official bridges through to IRC and XMPP (and most other chat systems), giving as much openness and choice as possible - if folks want to participate via Freenode and XMPP they can!
  • We’re built with large virtual communities in mind (e.g. Mozilla, KDE, Matrix itself) - for instance, we’ve worked a lot on moderation recently.
  • We’ve spent a lot of time improving widgets recently: these give the ability to embed arbitrary webapps into chatrooms - letting you add livestreams, video conferences, schedules, Q&A dashboards etc, augmenting a plain old chatroom into a much richer virtual experience that can hopefully capture the semantics and requirements of an event like FOSDEM.

We’re currently in the middle of setting up the server with a dedicated Element as the default client, but what we’re aiming for is:

  • Attendees can lurk as read-only guests in devrooms without needing to set up accounts (or they can of course use their existing Matrix/IRC/XMPP accounts)
  • Every devroom and track will have its own chatroom, where the audience can hang out and view the livestream of that particular devroom (using the normal FOSDEM video livestream system). There’ll also be a ‘backstage’ room per track for coordination between the devroom organisers and the speakers.
  • The talks themselves will be prerecorded to minimise risk of disaster, but each talk will have a question & answer session at the end which will be a live Jitsi broadcast from the speaker and a host who will relay questions from the devroom.
  • Each talk will have a dedicated room too, where after the official talk slot the audience can pop in and chat to the speaker more informally if they’re available (by text and/or by moderated jitsi). During the talk, this room will act as the ‘stage’ for the speaker & host to watch the livestream and conduct the question & answer session.
  • Every stand will also have its own chatroom and optional jitsi+livestream, as will BOFs or other adhoc events, so folks can get involved both by chat and video, to get as close to the real event as possible (although it’s unlikely we’ll capture the unique atmospheric conditions of K building, which may or may not be a bug ;)
  • There’ll also be a set of official support, social etc rooms - and of course folks can always create their own! Unfortunately folks will have to bring their own beer though :(
  • All of this will be orchestrated by a Matrix bot (which is rapidly taking shape over at https://github.com/matrix-org/conference-bot), responsible for orchestrating the hundreds of required rooms, setting up the right widgets and permissions, setting up bridges to IRC & XMPP, and keeping everything in sync with the official live FOSDEM schedule.

N.B. This is aspirational, and is all still subject to change, but that said - so far it’s all coming together pretty well, and hopefully our next update will be opening up the rooms and the server so that folks can get comfortable in advance of the event.

Huge thanks go to the FOSDEM team for trusting us to sort out the social/chat layer of FOSDEM 2021 - we will do everything we can to make it as successful and as inclusive as we possibly can! :)

P.S. We need help!

FOSDEM is only a handful of weeks away, and we have our work cut out to bring this all together in time. There are a few areas where we could really do with some help:

  • Folks on XMPP often complain that the Bifröst Matrix<->XMPP bridge doesn’t support MAMs - meaning that if XMPP users lose connection, they lose scrollback. We’re not going to have time to fix this ourselves in time, so this would be a great time for XMPP folks who grok xmpp.js to come get involved and help to ensure the best possible XMPP experience! (Similarly on other bifrost shortcomings).
  • It’d be really nice to be able to render nice schedule widgets for each devroom, and embed the overall schedule in the support rooms etc. The current HTML schedules at https://fosdem.org/2021/schedule/day/saturday/ and (say) https://fosdem.org/2021/schedule/room/vcollab/ don’t exactly fit - if someone could write a thing which renders them at (say) 2:5 aspect ratio so they can fit nicely down the side of a chatroom then that could be awesome!
  • While we’ll bridge all the official rooms over to Freenode, it’d be even nicer if people could just hop straight into any room on the FOSDEM server (or beyond) via IRC - effectively exposing the whole thing as an IRC network for those who prefer IRC. We have a project to do this: matrix-ircd, but it almost certainly needs more love and polish before it could be used for something as big as this. If you like Rust and know Matrix, please jump in and get involved!
  • If you just want to follow along or help out, then we’ve created a general room for discussion over at #fosdem-matrix:fosdem.org. It’d be awesome to have as many useful bots & widgets as possible to help things along.

This Week in Matrix 2020-12-30

30.12.2020 18:44 — This Week in MatrixBen Parsons

Dept of Status of Matrix 🌡️

The Matrix Holiday Special 2020

If you didn't read it already, I encourage you to read Matthew's Matrix Holiday Special 2020 post. So much has happened this year!

Oleg responded:

Great summary! Very inspiring!

I have a feeling experiencing a revolution in how people perceive messaging. Now, public sector and multiple communities adopting Matrix, one cannot think of a world without Matrix any more.

Kudos to the Element team and to the wonderful Matrix community! ♥️

I'm very excited for the FOSDEM on Matrix!

Andy added:

What a wonderful year has been for matrix, and it looks like next year will be equally as exciting. I'm HYPED.

Honestly, I'm hyped too. Simple statements like "loads of different universities have rolled out Matrix for collaboration" don't quite capture how much work is going on, and how much excitement there is to get the growth in the network we're soon to see.

Homeserver versions graphs

Chris offered:

For the first time (since recording the homeserver stats from Feb 2019) a non-synapse homeserver is now in the top 15 deployed homeserver versions: Welcome Dendrite 0.3.4: https://graph.settgast.org/d/z1nplqXik/matrix?orgId=1

How exciting is that! If you haven't already, do check out some of the history on these charts - big thanks to Chris for making this available.

Homeserver Deployment 📥️

Ansible-Dendrite

jaywink announced:

Created a new Ansible role to easily install a Dendrite server. Currently only tested on Ubuntu 20.04 with Ansible 2.9. Uses Docker to maintain a monolithic Dendrite. Requires but does not include PostgreSQL. Designed to easily be used with Traefik as reverse proxy.

https://git.feneas.org/jaywink/ansible-dendrite

Dept of Bridges 🌉

mx-puppet-groupme

Robin told us:

Not to be confused with matrix-puppet-groupme, mx-puppet-groupme is a fancy new GroupMe bridge with support for as many features as the web client would let me get my hands on. Please try it out, and let me know on GitLab or Matrix if you have any issues or suggestions! 💜

Thanks for sharing Robin!

Dept of Clients 📱

NeoChat 1.0

Tobias Fella said:

The KDE Community is excited to announce the first release of NeoChat, a Matrix Client based on Spectral and libQuotient. With the power of Qt and KDE Frameworks, NeoChat currently runs on mobile and desktop Linux devices, Android and Windows. You can read more about NeoChat and how to get it at https://carlschwan.eu/2020/12/23/announcing-neochat-1.0-the-kde-matrix-client/

Congrats on the 1.0! I think we'll hear much more from the team next year. 🎉

Nheko

Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE (with the notable exception being device verification for now) and intends to be full featured and nice to look at

Nico (@deepbluev7:neko.dev) reported:

We finally fixed the Windows build, so there are nightlies on Windows again! Also lorendb finished his profile work, which means that if you were using custom profiles before on one of the nightlies, you may need to login again, but the whole code is a lot cleaner now.

Dept of SDKs and Frameworks 🧰

libQuotient

kitsune reported:

Version 0.6.3 is out, another bugfix release on the stable branch. Nothing too significant, but .well-known-unaware homeservers should be treated better, and Matrix identifiers with special characters (ahem, slashes) can now be turned to valid URIs (matrix.to or proper Matrix URIs). Also, room tags starting with a . are no more considered valid, you'll get u. prepended to them.

Dept of Ops 🛠

matrix-docker-ansible-deploy

This Ansible playbook is meant to easily let you run your own Matrix homeserver.

Slavi offered:

matrix-docker-ansible-deploy now defaults to using the Postgres database engine for all bridges, bots and related services (ma1sd, Dimension, etc).

Previously we were only using Postgres for Synapse and couldn't reuse the same database server for other services.

With this huge pull request, we've added Postgres support to 17 other services.

Thanks to Johanna Dorothea Reichmann for starting the work on it and for providing great input!

Existing installations will get automatically migrated from SQLite/nedb to Postgres the next time you run the playbook.

Not only does this bring better performance and compatibility, but also, being able to reuse the same Postgres database for services other than Synapse paves the way for us to introduce other Postgres-only services such as Dendrite, the mautrix-signal bridge (existing pull request), etc.

For more information, refer to our changelog entry.

Dept of Bots 🤖

Timetracking Bot

MTRNord said:

We released v0.3.0 🎉 Feel free to follow development at #timetracking-bot:famedly.de or https://gitlab.com/famedly/bots/timetracking .

Also checkout the ansible role for the timetrackingbot: https://gitlab.com/famedly/ansible/collections/matrix/-/tree/main/roles/timetracking-bot

Changelog

Fixes

  • Print !in and !out responses in correct timezone

  • Make sure that we use the stores correctly

  • Make sure that "in" in the times table is not a primary key to allow multiple people at the same time to log in

  • Overall stability improvements

Features

  • Allow units in !record. For extended syntax see https://docs.rs/parse_duration/2.1.0/parse_duration/index.html

  • Better !stats command output (hours and minutes instead of pure minutes)

Chore

  • Dependency Updates

  • Add DB indices

Breaking changes

See https://gitlab.com/famedly/bots/timetracking/-/blob/v0.3.0/UPGRADING.md

Middleman

kapina-jaywink told us:

Common accounts for support are tricky to handle due to needing the people using those accounts to keep separate clients open to use the common account. For this at Elokapina we created Middleman, which acts as a proxy between an account and a room.

It's a bit basic but works pretty well for pure text messages. Any messages in rooms where the bot is are relayed to the management room and any replies to those messages (prefixed with !reply) in the management room are relayed back. Optionally senders can be anonymised to enable a feedback bot.

Coming up is more configuration on for example ignoring non-mentions in rooms with lots of members and hopefully support for images and reactions in the not too long future.

Find it here: https://github.com/elokapina/middleman (built with nio-template).

2020-12-30-MVAuG-demo.gif

GDQBot

daenney said:

With Awesome Games Done Quick 2021 Online starting on the 3rd of January the bot will now announce when an event is about to start in channels it's in.

If you don't want to run your own, you can invite @gdqbot:ecef.xyz or come hang out in #gdq:ecef.xyz.

Hopefully we'll also have donation tracking squared away before the start of the event.

Final Thoughts 💭

Nico (@deepbluev7:neko.dev) reported something interesting:

Threema published the source code for their apps this week: https://threema.ch/en/open-source

While that is not strictly Matrix related, it is great to see other E2EE enabled chat systems publishing the source code for their clients. It's the only way to verify their encryption actually works and is secure.

This move was announced a while ago, but I think that it actually happened now deserves a small shoutout! 🎉

Quite right! Though Matthew, thought the news needed to be put into proper context:

it’s almost like they’re scrabbling to keep up with matrix ;)

:D

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1tchncs.de259
2envs.net432
3maescool.be501
4matrix.sp-codes.de832
5flobob.ovh833
6aria-net.org1130
7libre.bzh1159.5
8uraziel.de1706
9matrix.weebl.me1728
10mailstation.de2017

That's all I know 🏁

Good grief, what a year. Exciting but I'm not sure I can fit it all in my head!

See you next week (2021-01-08: Friday!), and be sure to stop by #twim:matrix.org with your updates!

The Matrix Holiday Special 2020

25.12.2020 00:00 — GeneralMatthew Hodgson

Hi all,

Over the years it’s become a tradition to write an end-of-year wrap-up on Christmas Eve, reviewing all the things the core Matrix team has been up over the year, and looking forwards to the next (e.g. here’s last year’s edition). These days there’s so much going on in Matrix it’s impossible to cover it all (and besides, we now have This Week In Matrix and better blogging in general to cover events as they happen). So here’s a quick overview of the highlights:

Looking back at our plans for 2020 in last year’s wrap-up, amazingly it seems we pretty much achieved what we set out to do. Going through the bulletpoints in order:

  • We turned on End-to-end Encryption by default.
  • We have a dedicated team making major improvements to First-Time User Experience in Element (as of the last few months; hopefully you’ve been noticing the improvements!)
  • RiotX became Element Android and shipped.
  • Communities have been completely reinvented as Spaces (MSC1772) and while in alpha currently, they should ship in Jan.
  • Synapse scalability is fixed: we now shard horizontally by event - and Synapse is now pretty much entirely async/await!
  • Dendrite Beta shipped, as did the initial P2P Matrix experiments, which have subsequently continued to evolve significantly (although we haven’t implemented MSC1228 or MSC2787 portable accounts yet). Check out the Dendrite end-of-year update for more.
  • MLS experiments are in full swing - we got the first MLS messages passing over Matrix a few days ago, and Decentralised MLS work is back on the menu after an initial sprint in May.
  • There’s been a valiant mission to improve Bridge UX in the form of MSC2346 and its implementations in Element Web, although this has ended up failing to get to the top of the todo list (sorry Half-Shot! :/)
  • Spec progress has improved somewhat, and we are very excited to have welcomed Will Bamberg (formerly MDN) to support the spec from a professional tech writer perspective, with the all-new engine landing any day now! We’re still experimenting with ways to ensure the spec gets enough time allocated to keep up with the backlog, however - particularly community contributions.
  • ...and in terms of Abuse/Reputation - we properly kicked off our anti-abuse work and launched a first PoC implementation in the depths of Cerulean last week.

Perhaps more interesting is the stuff we didn’t predict (or at least didn’t want to pre-announce ;) for 2020:

  • Riot, Modular and New Vector got unified at last behind a single name: Element; hopefully the shock has worn off by now :)
  • Mozilla joined Matrix in force, turning off Moznet IRC in favour of going full Matrix.
  • We welcomed Gitter into the heart of the Matrix ecosystem (with Element acquiring Gitter from Gitlab in order to ensure Gitter’s Matrix integration acts as a reference for integrating future chat silos into Matrix) - with native Matrix support in Gitter going live shortly afterwards.
  • Automattic launched itself into the Matrix ecosystem with an investment in Element, and since then we’ve been working on getting Matrix better integrated and available to them (although all of Element’s Matrix-for-governments activity has ended up delaying this a bit). If you want to work for Automattic on integrating Matrix, they’re hiring!
  • We previewed Cerulean as a super-exciting proof-of-concept client, demonstrating how social media could work on Matrix, with native threading, profiles-as-rooms, decentralised reputation, and (shortly) peeking-over-federation.
  • We completely rewrote matrix.to and relaunched it as a much more capable and friendly permalink redirection service; a precursor to finally getting matrix:// URLs everywhere!
  • We certainly didn’t predict that the “how to install Synapse” video tutorial published at the beginning of the COVID-19 pandemic would end up with 25.5K views (and counting…)

Then, there’s whole new waves of exciting stuff going on. The most obvious has to be the amount of Government uptake we’ve seen with Matrix this year, following on from France embracing Matrix across the public sector last year. Firstly the German armed forces announced their transition to Matrix, and then the German states of Schleswig-Holstein and Hamburg announced a mammoth 500K user Matrix deployment for education and public administration. Meanwhile, North Rhine Westphalia (the biggest state in Germany) launched their own Matrix-powered messager for education; loads of different universities have rolled out Matrix for collaboration - and we hear Famedly is making good progress with Matrix-powered healthcare messaging solutions. Finally, outside of Germany, we’re seeing the first official deployments in the UK government and US federal government - we’ll share details where possible (but sometimes big deployments of encrypted communication systems want to remain discreet). It’s incredibly exciting to see Matrix spreading across the public sector and education, and we’re hoping this will follow a similar pattern to how the Internet, email or indeed the Web first developed: a mix of high profile public sector deployments, complemented by a passionate grass-roots technical community, eventually spreading to span the rest of society :).

Another exciting thing which emerged this year is the amazing academic work that Karlsruhe Institute of Technology’s Decentralized Systems and Network Services Research Group has been conducting on Matrix. This really came on the radar back in June when their Matrix Decomposition: Analysis of an Access Control Approach on Transaction-based DAGs without Finality paper was published - a truly fascinating analysis of how state resolution works in Matrix, and how we manage to preserve access control within rooms without using blockchain-style ‘sealed blocks’ (and has helped fix a few nasty bugs!). I’m not sure any of us realised that Matrix’s state resolution counts as a new field of research, but it’s been great to follow along with their independent work. Most recently, and even more excitingly, they’re circulating a preview of their Analysis of the Matrix Event Graph Replicated Data Type paper - a deep analysis of the properties of Matrix DAGs themselves. We highly recommend reading the papers (what better way to spend the holiday break!). To give a taste, the final paragraph of the paper concludes:

MEG summary

2020 has also seen the arrival and maturation of a whole new generation of Matrix clients - Hydrogen is really impressive as an experimental next-generation Web (and Mobile Web) client; an account with 3000 rooms that uses 1.4GB of RAM on Element Web uses 14MB of RAM on Hydrogen and launches instantly, complete with excellent E2EE implementation. It even works on MSIE! The whole app, including dependencies, is about 70KB of code (200KB including Olm). Meanwhile, matrix-rust-sdk is coming along well, providing a general purpose native library for writing excellent native Matrix clients. Fractal merged initial matrix-rust-sdk a few weeks ago, and we’ll be experimenting with switching to it in Element iOS and Element Android (for its e2ee) in the coming year. It’s not inconceivable to think of a world where matrix-rust-sdk ends up being the no-brainer official SDK for native apps, and Hydrogen’s SDK becomes the no-brainer official SDK for JS apps.

Meanwhile, in the community, there’s been so much activity it’s untrue. But on the subject of maturing apps, it’s been incredibly exciting to see NeoChat emerge as an official KDE Matrix client (built on libQuotient and Kirigami, forked from Spectral), FluffyChat going from strength to strength; Nheko continuing to mature impressively; Mirage appearing out of nowhere as a fully featured desktop client; Fractal merging matrix-rust-sdk etc. On the serverside, Conduit was the big community story of the year - with an incredibly fast Rust + Sled server appearing out of the blue, with viable federation coming up on the horizon. The best bet for an overview of all things community is to checkout the TWIM backlogs however - there’s simply way too much stuff to mention it all here.

Obviously, no 2020 wrap-up post would be complete without acknowledging the COVID-19 pandemic - which increased focus on Matrix and other remote collaboration technology more than anyone could have predicted (especially given all the privacy missteps from Zoom, Teams and others). One of the highlights of the year was seeing the HOPE (Hackers On Planet Earth) conference shift their entire proceedings over to Matrix - turning the conference into a 10 day television station of premium hacking content, with Matrix successfully providing the social glue to preserve a sense of community despite going virtual. Similarly, we’re incredibly excited that FOSDEM 2021 is highly likely to run primarily via Matrix (with bridges to IRC and XMPP, of course) - our work is going to be cut out for us in January to ensure the amazing atmosphere of FOSDEM is preserved online for the >8,500 participants and ~800 talks. And if any other event organisers are reading this - please do reach out if you’re interested in going online via Matrix: we want Matrix to be the best possible ecosystem for online communities, including virtual events, and we’ll be happy to help :)

Talking of FOSDEM, a really fun bit of work which landed in Element this year was to (finally!) polish Widgets: the ability to embed arbitrary webapps into Matrix chatrooms. This includes being able to embed widgets in the RightPanel on Element Web, the LeftPanel too, add as many as you like to a room, resize them(!), and generally build much more sophisticated dashboards of additional content. Modal and fullscreen widgets are coming too, as are ways to simplify and unify access control. It turns out that these have arrived in the nick of time for events like FOSDEM, where we’re expecting to very heavily use widgets to embed video streams, video conferences, schedules, and generally automate the workflow of the conference via adding in web UIs as widgets wherever necessary. The work for this has been driven by the various German education deployments, where the same tricks are invaluable for automating online learning experiences. We originally wrote Widgets back in 2017 as a proof-of-concept to try to illustrate how chatrooms could be used to host proper custom UIs, and it's fantastic to see that dream finally come of age.

Finally, it’s been really exciting to see major progress in recent months on what’s essentially a whole new evolution of Matrix. Two years ago, a quiet patch during the Christmas holidays gave birth to a whole bunch of wild science fiction Matrix Spec Changes: MSC1772: Spaces (groups as rooms), MSC1769: Profiles as rooms, MSC1767: Extensible events, MSC1776: Peeking over /sync, MSC1777: Peeking over federation, etc. This was in part trying to ensure that we had something to look forward to when we emerged from the tunnel of launching Matrix 1.0, and in part trying to draw a coherent high-level sketch of what the next big wave of Matrix features could look like. Inevitably the MSCs got stuck in limbo for ages while we exited beta, launched Matrix 1.0, turned on E2EE by default etc - but in the latter half of this year they’ve hit the top of the todo list and it’s been incredibly exciting to see entirely new features landing once again. Implementation for Spaces is in in full swing and looking great; Profiles-as-rooms are effectively being trialled in Cerulean; Peeking over /sync has landed in Dendrite and peeking over federation is in PR (and unlocks all sorts of desirable features for using rooms more generically than we have today, including Spaces). Only Extensible events remains in limbo for now (we have enough to handle getting the others landed!)

Of these, Spaces has turned out to be exciting in wholly unexpected ways. While prototyping the UX for how best to navigate hierarchies of spaces, we had a genuine epiphany: the ability for anyone to define and share arbitrary hierarchies of rooms makes Matrix effectively a global decentralised hierarchical file system (where the ‘files’ are streams of realtime data, but can obviously store static content too). The decentralised access controls that KIT DSN wrote about could literally be file-system style access controls; enforcing access on a global decentralised hierarchy. We obviously have shared hierarchical filesystems today thanks to Dropbox and Google Drive, but these of course are centralised and effectively only store files - whereas Spaces could potentially scale to the whole web. In fact, you could even think of Spaces as flipping Matrix entirely on its head: the most defining building block going forwards could be the Spaces themselves rather than the rooms and events - just as directories are intrinsic to how you navigate a conventional filesystem. How has Matrix got this far without the concept of folders/directories?!

Right now these thoughts are just overexcited science fiction, but the potential really is mindblowing. It could give us a global read/write web for organising any arbitrary realtime data - with the social controls via ACLs to delegate and crowdsource curation of hierarchies however folks choose. The Matrix.org Foundation could seed a ‘root’ hierarchy, go curate all the rooms we know about into some Linnean-style taxonomy, delegate curation of the various subspaces to moderators from the community, and hey presto we’ve reinvented USENET… but with modern semantics, and without the rigid governance models. Hell, we could just mount (i.e. bridge) USENET straight into it. And any other hierarchical namespace of conversations you can think of - Google Groups, Stackoverflow, Discourse, IMAP trees…

Of course, the initial Spaces implementation is going to be focused of on letting communities publish their existing rooms, and users organise their own rooms, rather than managing an infinite ever-expanding global space hierarchy - but given we’ve been designing Spaces to support government (and inter-government) scales of Spaces, it’s not inconceivable to think we could use it to navigate gigantic public shared Spaces in the longer term.

Anyway, enough Space scifi - what’s coming up in 2021?

2021

Our current hit list is:

  • Spaces - see above :)
  • Social Login - we’re going to be making Single Sign On (SSO) a proper first-class citizen in Matrix (and Synapse and Element) in the coming weeks, and enabling it on the matrix.org homeserver, so users can do single-click logins via Github/Gitlab/Google and other SSO providers. Obviously this means your Matrix identity will be beholden to your identity provider (IdP), but this may well be preferable for many users who just want a single-click way to enter Matrix and don’t care about being tied to a given IdP.
  • VoIP - we have a lot of work in flight at the moment to make 1:1 VoIP super robust. Some of it has already landed in Element, but the rest will land in the coming weeks - and then we’re hoping to revisit Matrix-native group voice/video.
  • Voice messaging - we’re hoping to finally add voice messaging to Element (and Matrix)
  • Location sharing - ...and this too.
  • **P2P **- Lots of P2P work on the horizon, now Dendrite is increasingly stable. First of all we need to iterate more on Pinecone, our pre-alpha next-generation P2P overlay network - and then sort out account portability, and privacy-preserving store-and-forward. We’re hoping to see the live P2P Matrix network turn on this year, however, and ideally see homeservers (probably Dendrite) multihoming by default on both today’s Matrix as well as the P2P network, acting as gateways between the two.
  • Threads - Cerulean is excellent proof for how threading could work in Matrix; we just need to get it implemented in Element!
  • Peeking - Peeking is going to become so much more important for participating in non-chat rooms, such as Spaces, Profiles, Reputation feeds, etc. We’ll finish it in Dendrite, and then implement it in Synapse too.
  • **Decentralised Reputation **- Cerulean has the first implementation of decentralised reputation for experimentation purposes, and we’ll be working solidly on it over the coming year to empower users to counter abuse by applying their own subjective reputation feeds to their content.
  • **Incremental Signup **- Once upon a time, Element (Riot) had the ability to gradually sign-up without the user even really realising they’d signed up. We want to bring it back - perhaps this will be the year?
  • DMLS - with the first MLS messages flowing over Matrix, we want to at least provide MLS as an option alongside Megolm for encryption. It should be radically more performant in larger rooms (logarithmic rather than linear complexity), but lacks deniability (the assurance that you cannot prove a user said something in retrospect, in order to blackmail them or similar), and is still unproven technology. We’ll aim to prove it in 2021.
  • E2EE improvements - We improved E2EE immeasurably in 2020; turning it on by default, adding cross-signing, QR code verification etc. But usability and reliability can still be improved. We’ll be looking at further simplifying the UX, and potentially combining together your login password and recovery/security passphrase so you only have one password to remember going forwards.
  • Hydrogen - We’ll keep polishing Hydrogen, bringing it towards feature parity with Element, ensure its SDK is available for other clients, and start seeing how we can use it in Element itself. For instance, the Spaces-aware RoomList in Element may well end up stealing alien technology from Hydrogen.
  • matrix-rust-sdk - Similarly, we’ll keep polishing matrix-rust-sdk; stealing inspiration from Hydrogen’s state model, and start migrating bits of the native mobile Element apps to use it.
  • The Spec - get Will’s new spec website live, and get improving all the surrounding material too.

I’m sure I’m missing lots here, but these are the ones which pop immediately to mind. You can also check Element's public roadmap, which covers all the core Matrix work donated by Element (as well as everything else Element is getting up to).

As always, huge huge thanks goes to the whole Matrix community for flying Matrix and keeping the dream alive and growing faster than ever. It’s been a rough year, and we hope that you’ve survived it intact (and you have our sincere sympathies if you haven’t). Let’s hope that 2021 will be a massive improvement, and that the whole Matrix ecosystem will continue to prosper in the new year.

-- Matthew, Amandine, and the whole Matrix team.

Introducing Cerulean

18.12.2020 00:00 — GeneralMatthew Hodgson

Hi all,

We have a bit of an unexpected early Christmas present for you today…

Alongside all the normal business-as-usual Matrix stuff, we’ve found some time to do a mad science experiment over the last few weeks - to test the question: “Is it possible to build a serious Twitter-style decentralised microblogging app using Matrix?”

It turns out the answer is a firm “yes” - and as a result we’d like to present a very early sneak preview of Cerulean: a highly experimental new microblogging app for Matrix, complete with first-class support for arbitrarily nested threading, with both Twitter-style (“vertical”) and HN/Reddit-style (“horizontal”) layout… and mobile web support!

Cerulean screenie

Cerulean is unusual in many ways:

  • It’s (currently) a very minimal javascript app - only 2,500 lines of code.
  • It has zero dependencies (other than React).
    • This is to show just how simple a fairly sophisticated Matrix client can be...
    • ...and so the code can be easily understood by folks unfamiliar with Matrix...
    • ...and so we can iterate fast while figuring out threading...
    • ...and because none of the SDKs support threading yet :D
  • It relies on MSC2836: Threading - our highly experimental Matrix Spec Change to extend relationships (as used by reaction & edit aggregations) to support free-form arbitrary depth threading.
  • As such, it only works on Dendrite, as that’s where we’ve been experimenting with implementing MSC2836. (We’re now running an official public Dendrite server instance at dendrite.matrix.org though, which makes it easy to test - and our test Cerulean instance https://cerulean.matrix.org points at it by default).

This is **very much a proof of concept. **We’re releasing it today as a sneak preview so that intrepid Matrix experimenters can play with it, and to open up the project for contributions! (PRs welcome - it should be dead easy to hack on!). Also, we give no guarantees about data durability: both Cerulean and dendrite.matrix.org are highly experimental; do not trust them yet with important data; we reserve the right to delete it all while we iterate on the design.

What can it do?

So for the first cut, we’ve implemented the minimal features to make this something you can just about use and play with for real :)

  • Home view (showing recent posts from folks you follow)
  • Timeline view (showing the recent posts or replies from a given user)
  • Thread view (showing a post and its replies as a thread)
  • Live updating (It’s Matrix, after all! We’ve disabled it for guests though.)
  • Posting plain text and images
  • Fully decentralised thanks to Matrix (assuming you’re on Dendrite)
  • Twitter-style “Vertical” threading (replies form a column; you indent when someone forks the conversation)
  • HN/Reddit/Email-style “Horizontal” threading (each reply is indented; forks have the same indentation)
  • Basic Registration & Login
  • Guest support (slightly faked with non-guest users, as Dendrite’s guest support isn’t finished yet)
  • Super-experimental proof-of-concept support for decentralised reputation filtering(!)

Obviously, there’s a huge amount of stuff needed for parity with a proper Twitter-style system:

  • Configurable follows. Currently the act of viewing someone’s timeline automatically follows them. This is because Dendrite doesn’t peek over federation yet (but it’s close), so you have to join a room to view its contents - and the act of viewing someone’s timeline room is how you follow them in Cerulean.
  • Likes (i.e. plain old Matrix reactions, although we might need to finally sort out federating them as aggregations rather than individually, if people use them like they use them on Twitter!)
  • Retweets (dead easy)
  • Pagination / infinite scrolling (just need to hook it up)
  • Protect your posts (dead easy; you just switch your timeline room to invite-only!)
  • Show (some) replies to messages in the Home view
  • Show parent and sibling context as well as child context in the Thread view
  • Mentions (we need to decide how to notify folks when they’re mentioned - perhaps Matrix’s push notifications should be extended to let you subscribe to keywords for public rooms you’re not actually in?)
  • Notifications (although this is just because Dendrite doesn’t do notifs yet)
  • Search (again, just needs to be implemented in Dendrite - although how do you search beyond the data in your current homeserver? Folks are used to global search)
  • Hashtags (it’s just search, basically)
  • Symlinks (see below)
  • Figure out how to handle lost unthreaded messages (see below)
  • Offline support? (if we were using a proper Matrix SDK, we’d hopefully get this for free, but currently Cerulean doesn’t store any state locally at all).

How does it work?

Every message you send using Cerulean goes into two Matrix rooms, dubbed the "timeline" room and the "thread" room. The "timeline" room (with an alias of #@matthew:dendrite.matrix.org or whatever your matrix id is) is a room with all of your posts and no one else's. The "thread" room is a normal Matrix room which represents the message thread itself. Creating a new "Post" will create a new "thread" room. Replying to a post will join the existing "thread" room and send a message into that room. MSC2836 is used to handle threading of messages in the "thread” room - the replies refer to their parent via an m.relationship field in the event.

These semantics play nicely with existing Matrix clients, who will see one room per thread and a flattened chronological view of the thread itself (unless the client natively supports MSC2836, but none do yet apart from Cerulean). However, as Cerulean only navigates threaded messages with an m.reference relationship (eg it only ever uses the new /event_relationships API rather than /messages to pull in history), normal messages sent by Matrix into a thread or timeline room will not yet show up in Cerulean.

In this initial version, Cerulean literally posts the message twice into both rooms - but we’re also experimenting with the idea of adding “symlinks” to Matrix, letting the canonical version of the event be in the timeline room, and then the instance of the event in the thread room be a ‘symlink’ to the one in the timeline. This means that the threading metadata could be structured in the thread room, and let the user do things like turn their timeline private (or vice versa) without impacting the threading metadata. We could also add an API to both post to timeline and symlink into a thread in one fell swoop, rather than manually sending two events. It’d look something like this:

Cerulean diagram

We also experimented with cross-room threading (letting Bob’s timeline messages directly respond to Alice’s timeline messages and vice versa), but it posed some nasty problems - for instance, to find out what cross-room replies a message has, you’d need to store forward references somehow which the replier would need permission to create. Also, if you didn’t have access to view the remote room, the thread would break. So we’ve punted cross-room threading to a later MSC for now.

Needless to say, once we’re happy with how threading works at the protocol level, we’ll be looking at getting it into the UX of Element and mainstream Matrix chat clients too!

What’s with the decentralised reputation button?

Cerulean is very much a test jig for new ideas (e.g. threading, timeline rooms, peeking), and we’re taking the opportunity to also use it as an experiment for our first forays into publishing and subscribing to reputation greylists; giving users the option to filter out content by default they might not want to see… but doing so on their own terms by subscribing to whatever reputation feed they prefer, while clearly visualising the filtering being applied. In other words, this is the first concrete experimental implementation of the work proposed in the second half of Combating Abuse in Matrix without Backdoors. This is super early days, and we haven’t even published a proto-MSC for the event format being used, but if you’re particularly interested in this domain it’s easy enough to figure out - just head over to #nsfw:dendrite.matrix.org (warning: not actually NSFW, yet) and look in /devtools to see what’s going on.

So, there you have it - further evidence that Matrix is not just for Chat, and a hopefully intriguing taste of the shape of things to come! Please check out the demo at https://cerulean.matrix.org or try playing with your own from https://github.com/matrix-org/cerulean, and then head over to #cerulean:matrix.org and let us know what you think! :)

This Week in Matrix 2020-12-18

18.12.2020 00:00 — This Week in MatrixBen Parsons

Matrix Live 🎙

This week we're all about video content! So you get Matrix Live AND Demos - you are so lucky! 🥰

Demos

uhoreg added a late improvement to his MLS demo:

This is how my demo would have looked if I had managed to finish debugging in time.

2020-12-18-mls.png

So, now you need to watch the video to make sense of this image!

Dept of Status of Matrix 🌡️

Introducing Cerulean

Matthew reported:

Introducing Cerulean - highly experimental threaded microblogging for Matrix. Read all about it at https://matrix.org/blog/2020/12/18/introducing-cerulean

Cerulean is being discussed on the front page of Hacker News.

Community Demos

The last set of community demos were well received, so if you have an idea for a demo, get it ready for 2021-01-15, which will be the next demo session!

Dept of Spec 📜

New spec platform

wbamberg said:

The spec core team has been trying out the new spec this week to decide if it's ready to ship. And we've been applying Dean's design updates (currently in a separate branch at https://friendly-yonath-3de225.netlify.app/).

Spec

anoa told us:

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.

MSC Status

Closed MSCs:

Spec Core Team

In terms of Spec Core Team MSC focus for this week, we made a healthy amount of progress on the MSCs in focus. A few members have also been iterating quite a bit on MSC1772 for Spaces. Look for these MSCs to start landing come the first few weeks of the New Year as implementation work starts to relax a bit. The focus for next week is... Christmas! 🎄

But first week of January will be MSC2844, MSC2366 (ready and done in key verification) and MSC2790 (modal widgets).

2020-12-18-7E797-stacked_area_chart.png

Dept of Servers 🏢

Dendrite / gomatrixserverlib

Dendrite is a next-generation homeserver written in Go

neilalexander announced:

If you haven't already seen it, we wrote a Dendrite 2020 Progress Update blog post detailing our year in Dendrite. It also includes an announcement for our new public dendrite.matrix.org homeserver, which is open for registration!

Today we released Dendrite 0.3.4, which is a performance and bug-fix release, including the following changes from the last two weeks:

  • The stream tokens for /sync have been refactored, giving PDUs, typing notifications, read receipts, invites and send-to-device messages their own respective stream positions, greatly improving the correctness of sync

  • A new roominfo cache has been added, which results in less database hits in the roomserver

  • Prometheus metrics have been added for sync requests, destination queues and client API event send perceived latency

  • Event IDs are no longer recalculated so often in /sync, which reduces CPU usage

  • Sync requests are now woken up correctly for our own device list updates

  • The device list stream position is no longer lost, so unnecessary device updates no longer appear in every other sync

  • A crash on concurrent map read/writes has been fixed in the stream token code

  • The roomserver input API no longer starts more worker goroutines than needed

  • The roomserver no longer uses the request context for queued tasks which could lead to send requests failing to be processed

  • A new index has been added to the sync API current state table, which improves lookup performance significantly

  • The client API /joined_rooms endpoint no longer incorrectly returns null if there are 0 rooms joined

  • The roomserver will now query appservices when looking up a local room alias that isn't known

  • The check on registration for appservice-exclusive namespaces has been fixed

If you are running a Dendrite server, it is highly recommended that you upgrade to the latest version!

Spec compliance is unchanged:

  • Client-server APIs: 58%, same as last week

  • Server-server APIs: 83%, same as last week

As always, feel free to join us in #dendrite:matrix.org for general Dendrite chat or #dendrite-dev:matrix.org for development discussion.

Dendrite also powers Cerulean, mentioned above!

Synapse

Synapse is a popular homeserver written in Python.

callahad announced:

Hello again from Synapse land! This is our last TWiM of 2020, but we're hard at work landing our last patches before going on holiday. Expect 1.25.0rc1 in early January with loads of fixes, performance improvements, and a few new features. In particular:

You can look forward to all of this and more next year, but until then, happy holidays from all of us over here at Element and in the broader Synapse community!

Conduit

Conduit is a Matrix homeserver written in Rust https://conduit.rs

timokoesters said:

Hey, I'm back! I was busy with University. I also forgot to TWIM this last week, so here you go:

  • Conduit is beginning to support appservices
  • Devin R and I are currently working on a better state store and improving federation

Homeserver Deployment 📥️

YunoHost

Pierre said:

YunoHost is an operating system aiming for the simplest administration of a server, and therefore democratize self-hosting.

Synapse integration had been updated to 1.23.1

Element Web integration had been updated to 1.7.15

Dept of Bridges 🌉

GitHub / Gitlab bridge

Half-Shot told us:

Hello everyone! It's been ages since I've talked about a personal project on here. I've been working on the matrix-github bridge, which is an all singing all dancing service to connect you to Github AND Gitlab. (Yes, it's now poorly named).

I've landed the initial bits to make GitLab work, so that you can join issues and MRs as rooms in the UI but there is more work to do to make that solid and more interactive. The thing I am most looking forward to is notifications for GitLab so I can get away from emails.

ALSO, off the back of TravisR's fantastic widget work, I've landed initial support for widgets in admin rooms and soon issue/PR rooms. The idea here being that widgets give you a nicer UI to control the bridge or render details like commits going past or diffs. All the groundwork for authenticating widgets against the bridge and rendering them is done, now it just needs features.

That's the highlights this week. I'm hoping to really tighten the documentation to get more people spun up on it as the notifications features have turned out to be a killer feature for me :)

2020-12-18-x7YaI-image.png

I've been using this (the bridge, rather than the upcoming widgets) for a couple of months now and find it really really powerful. I'm not in GitHub most of the day, so getting notifs piped direct into Matrix is great!

mautrix-instagram

Tulir said:

I added bridging of typing notifications and read receipts from Matrix to Instagram and fixed a bunch of bugs. Only thing that's still missing is presence and Matrix->Instagram non-image files.

This is now almost feature complete! Nice one Tulir.

Gitter

Eric Eastwood said:

A few updates to the Gitter bridge itself this week handling Matrix edits outside of the 5 minute edit window on Gitter and fixing up @mentions being mangled by : colons on the Gitter side when someone mentions you from Element. We also strip the redundant Matrix reply quote when it goes into a Gitter threaded conversation because it's already easily viewable in the thread.

We're also getting started on some of the Element feature parity with Gitter issues. First directly tackling backfilling existing message history.

Other teams are already underway working on Spaces, Social login for Github which will ship in the new year!

Dept of Clients 📱

Element Clients

Compiled by the team.

  • Spaces
    • We’ve been progressing spaces on Web, focusing on managing spaces, members and rooms. Meanwhile, we’ve also been iterating on designs for all platforms.
  • Social login
    • Synapse support for picking a username should be landing in develop today focusing next on supporting multiple identity providers. On iOS, we’ve implemented SSO redirect authentication mechanisms which also should land soon.
  • VoIP
    • We’ve been making progress all round on all platforms, with the next web release including improvements to holding & resuming calls and multiple line support!

Web

  • 1.7.16-rc.1 available on staging
    • Added ability to put a VoIP call on hold and answer another
    • Added a special visual effect for confetti
    • Improved the login flow
  • Investigating changelog options in the background

Android

iOS

  • We’re prepping 1.1.3 after ironing out final kinks relating to the new background service, hoping to release before Christmas.

Nheko

Nico (@deepbluev7:neko.dev) offered:

In celebration of the cross-signing MSC being finally merged, Nheko master now supports cross signing other users and your own devices. 🎉 The UX is currently not final, you have to manually fetch the cross signing keys in the settings, if you don't have them after a verification. You are also not yet prompted to verify new logins and can't bootstrap cross-signing from Nheko, but the functional bits and bobs are there now. We are also still investigating, if prompting for your recovery key/passphrase is less confusing by using just one input field or if we should prompt for them separately. Currently Nheko just asks you to unlook and will accept whatever you input (if it can decrypt the secrets with it). FluffyChat does that and I think it is really nice!

There are also a few smaller features added recently:

  • Nheko now uses the icon from your theme on the login page

  • You can react instead of reply with a (very) short text using /react

  • A few improvements(?) regarding whitespace and code rendering.

SchildiChat Web/Desktop

qg reported:

Today, we can announce the web/desktop variant of SchildiChat is out of beta!

SchildiChat is a fork of Element, which focuses on UI changes such as message bubbles and a unified chat list for both direct messages and groups, which is a more familiar approach to users of other popular instant messengers.

Since the last announcement on TWIM, we have not only fixed most of the layout bugs caused by the message bubbles, but can also report the following:

  • Settings for layouts work, so you can also use SchildiChat if you only want the unified overview and no message bubbles

  • All information about desktop is now on our new website: https://schildi.chat/desktop

  • The hosted instance is now here: https://app.schildi.chat

  • For Linux, SchildiChat is now available from AUR, apt repo for debian, flathub and as AppImage

  • There's a portable Windows version now too

Code on GitHub: https://github.com/SchildiChat/schildichat-desktop

Matrix room: #schildichat-web:matrix.org

2020-12-18-bIasy-2.png

FluffyChat

FluffyChat is a cute cross-platform matrix client. It is available for Android, iOS, Web and Desktop.

krille reported:

FluffyChat 0.24.0 is released with new features and fixes.

What is new:

  • A new public room discovery page, where you can find and join public rooms

  • Experimental bootstrap support for cross-signing and keybackup and something really weird that is named SSSS

  • Enhanced power level managing for rooms

  • Minor design changes like a new icon set

Beside of that we have of course a lot of bug fixes and stability improvements. All languages should now also work on iOS.

Also we are looking forward to publish the app in the AppStore for iOS but we will do this under the hood of Famedly. Btw. FluffyChat will soon move to the Famedly GitLab project and we will move the app in the PlayStore to the Famedly account too. This doesn't mean that anything will change there. It is just that Famedly will now handle the legal stuff for us and may offer paid customer support for businesses, which have interest in FluffyChat. But the app will still be available under the same AGPL3 license. This will just make it easier for us to publish the app in the official stores as we don't need to pay so much for the developer accounts.

2020-12-18-JruOC-screenshot_20201218-144554.png

Dept of Bots 🤖

Timetracking Bot by famedly

MTRNord reported:

Find it at: https://gitlab.com/famedly/bots/timetracking or join us at #timetracking-bot:famedly.de

Numerous new features and fixes arrived this week. Getting closer to production readiness. Famedly also started daily internal testing for the bot to find more issues and fixing them.

v0.1.5

New Features

  • Implement !setTimezoneDefault

  • Add timezone conversion to !in and !out

  • Write admin status to table in Database

  • Add default_timezone field to Config which defines the bot wide default. If not set UTC is assumed.

  • Add !stats command

  • Use Unix timestamps in database (migrated automatically)

Fixes

  • Make sure Version sha actually gets generated in CI

  • Make tests work again in the CI

  • Fix that !help and !version work without the database in case of issues

!record command polish

  • Only accept positive durations greater than 1

  • Only accept durations less than 24h

  • Check if duration is numeric

Refactoring

  • Move message templates where needed to templates folder

  • Use color-eyre instead of anyhow

v0.2.0

Refactoring

  • All new command definition. Allowing for much easy adding of commands and helptext

  • New !help response

Fixes

  • Add the missing default for !stats

  • Check for required amount of arguments in !delete

Chore

  • Fix typos in responses

  • Remove some dead code

GDQBot

daenney told us:

It's the most wonderful time of the year! The ⛄️ winter edition of Games Done Quick is almost upon us. This recurring speedrunning event helps raise money for charity each year. As usual, the proceeds of the week long AGDQ are going to the Prevent Cancer Foundation. With some luck we might break last year's $3.13 million record.

To help you keep track of the event I've created GDQBot. Right now you can use it to query for the schedule. Next week should see the code land to add the announcer routine so the bot will announce events as they happen to channels its invited to.

Also a big thank you to tulir for patiently answering my questions as I was trying to find my way around mautrix-go.

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1neko.dev322
2bridges.matrix.linux.pizza420
3almum.de474
4envs.net598.5
5saces.de627
6kb1rd.net722
7matrix.sp-codes.de905
8casavant.org943
9aria-net.org1034
10flobob.ovh1050

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Matrix.to: Reloaded

17.12.2020 15:08 — GeneralNad Chishtie
Last update: 17.12.2020 14:11

One of the goals on the core team is to reduce friction for new users joining Matrix. A challenge we regularly face is all the factors which make Matrix flexible and powerful as an open, secure decentralised protocol also increase the difficulty of getting started.

As one example— to create a healthy, vibrant ecosystem and put power back into the hands of end users, it’s critical multiple clients exist, and that users ultimately have control over which one to use. However, needing to choose a client before getting going is counter-intuitive, and adds cognitive load which proprietary services simply don’t have.

Striking this balance is tricky, and one we’re aiming to improve with the latest version of Matrix.to.

What's Matrix.to?

Matrix.to is a simple URL redirection service which lets users share links to rooms, users and communities. It isn’t tied to any client, and for end users it serves as a landing page for many as their first touch to Matrix. And until the matrix:// URI scheme is deployed commonplace (we finally registered it with IANA a few months ago!) it is the only client-agnostic way to link to content in Matrix.

It’s privacy preserving, with no information stored server side, with all calculation done in client-side JavaScript. It started life years ago as a temporary minimum viable hack by Matthew which then hung around untouched for years being… minimally viable. But we’ve recently given it a huge upgrade in usability and functionality at last!

What's new?

The matrix.to room, on the new matrix.to!

We’ve re-written matrix.to from scratch, giving it a visual upgrade and refocused the experience around:

  1. Helping new users find the best client for them to easily get going. By default, links will prioritise showing clients which are compatible with the platform the link is being viewed from, including mobile platforms.

  2. Optionally, remembering your preferred client for future visits. This also includes deeplinking into native apps for Element Desktop & Element Mobile (and in future, other clients too, of course).

  3. Fetching useful previews. One of our main observations when testing with new users is a lack of confidence trying out a new service without personalised, contextual information on the rooms or people they’re heading to. Matrix.to now displays room metadata like avatars, names and topics directly (fetched via matrix.org by default, asking for permission).

What's next?

We’ve baked in the ability to specify clients and deployments within links, allowing sharing to give the option of a specific destination to guarantee a predictable experience. For instance, Mozilla might share matrix.to links which recommend using chat.mozilla.org, if you don’t already have a preferred Matrix client configured. We’ve yet to implement this feature in Element, but we’ll be researching and experimenting with different implementations soon and will recommend best practises when we have them.

We plan to evolve matrix.to over time, including eventually evolving it to better support the Matrix URI scheme. As before, you can find the source code on GitHub and please go ahead and submit pull requests to get your Matrix client listed.

Let us know what you think over in the matrix.to room, linked to via matrix.to!