As you may have seen from the previous blog post, we have a new drive to advance the Matrix Specification itself. Part of this is https://matrix.org/docs/spec/proposals, which lists all the spec change proposals we’ve accumulated so far, and describes the flow for getting new proposals merged. There is a new room, #matrix-spec:matrix.org for discussion, please join if you want to get involved in this process. Check out the page and the blog post for more detail.
Next up: try to turn some of the many WIP proposals into Spec PRs…
A major topic at the Hackfest was a discussion of splitting the Fractal client into two UIs for the different behaviours of messaging apps. For anyone interested in product design thinking this is a genuinely fascinating topic. I encourage you to read “Banquets and Barbecues”, Tobias’ excellent coverage of the latest thinking. The different chat personas are very well explained and the post brings up some of the immediate technical challenges too.
Projects and Products
mxisd v1.1 RC1 available
Max reports on mxisd, a Federated Matrix Identity server for self-hosted Matrix infrastructures:
mxisd v1.1 RC1 is out, addressing various privacy issues and being more GDPR-friendly overall. Testing and feedback from the community is very much appreciated
Dimension, an open source integrations manager for matrix clients from TravisR, now supports sticker packs.
Lots of excitement at the variety of independent clients and servers able to interact over the matrix protocol. The images above show The Construct (server) and gomuks (client), and then mxhsd and Fractal. A fundamental part of matrix is to be an open protocol, so it’s great to see entirely independent implementations liaising together! While implementing mxhsd, Max has been documenting spec omissions in a branch of the spec – we’re hoping he will contribute these back!
Honourable mention for mujx, who was sending messages with nheko and Ruma a year ago!
Matrix Core team expansion
Stève – 17th May (yesterday)
Amber – 21st (Monday)
Anoa – 21st (Monday)
Hubert – 28th May
Half-Shot – 4th June
…and one more community member, hopefully (just sorting paperwork currently!)
Heads up that we’re consciously trying to hire a mix of folks from the Matrix community as well as those outside it – and avoid hiring the whole community, both to ensure diversity of viewpoint & experience in the core team, and also to avoid cannibalising folks who working on their own commercial projects on top of Matrix. We’d prefer Matrix to be as decentralised and heterogenous as possible, needless to say – and instead try to support folks in building on Matrix without hiring them into the core team (where we’d expect them to focus on the core project for everyone’s benefit). This may change once we have Matrix set up as a separate foundation, once we’ve got out of beta, of course.
#matrix-spec:matrix.org is a new room dedicated to discussions on specific matrix spec proposals, as part of the process we’re building around contributing to the matrix spec.
The talk of the town in Strasbourg this week was the arrival of Fractal Hackfest 2018! Event is still ongoing, and I’m sure they will provide a report of the progress on https://wiki.gnome.org/Hackfests/Fractal2018, though Alexandre kindly sent us a photo of the group in action
Wonderful things are happening and being discovered regarding IoT and Home Automation. uhoreg was the first to point us to tinloaf’s project to build a Matrix Chatbot component for Home Assistant:
This component allows you to send messages to matrix rooms, as well as to react to messages in matrix rooms. Reacting to commands is accomplished by firing an event when one of the configured commands is triggered.
Enthusiasm for this work led to jfred discussing his past adventures in Matrix, including a component for sibyl, ‘a python chatbot with a focus on XBMC’ allowing Matrix communication.
All this excitement led to Cadair creating #homeautomation:cadair.com, which has started a more thorough discussion. I’m eager to see more non-chat applications of Matrix, #twim:matrix.org came up with others with projects in progress.
It’s worth noting that we feel that GDPR is an excellent piece of legislation from the perspective of forcing us to think more seriously about our privacy – it has forced us to re-prioritise all sorts of long-term deficiencies in Matrix (e.g. dependence on DNS; improving User Interactive authentication; improving logout semantics etc). There’s obviously a lot of work to be done here, but hopefully it should all be worth it!
TravisR on GDPR
TravisR has also been thinking about GDPR, and how it relates to his Voyager bot. In his words:
TWIM: I’ve mostly been working on figuring out how GDPR affects t2bot.io for the last couple weeks. One of the things running on t2bot.io is Voyager – a bot that tries to join rooms it sees mentioned in people’s messages, graphing them on https://voyager.t2bot.io. With the increase in talk about GDPR and more bots starting to wander the federation, the recurring topic of whether Voyager should change its approach to finding and listing rooms.
With the current approach, Voyager reads messages and tries to find room aliases to try and join. Individual people can opt-out of this tracking to stop Voyager from reading/parsing their messages (opting back in at a later time, if desired). The room moderators can kick or ban the bot to completely remove their room from the graph, and can invoke a ‘soft kick’ if they’d like to have their room remain listed, but don’t want the bot in the room. Voyager will make sure to only show information for public rooms and will update the graph if the room flips between public and private.
If anyone has feedback on how this approach could be improved (or if it should be left as-is), please come by #voyager:t2bot.io on matrix to start the conversation.
I was surprised and excited to learn that a Russian translation of the Matrix FAQ has been produced by a group of Russian-lanugage users. ma1uta reported:
They’ve provided a PR which I will presently merge (though of course, as I don’t speak Russian I will need to trust that it’s really a translation of the FAQ!)
Projects and Updates
Matrix Ruby SDK
Ananace reports that work has begun on a Matrix SDK for Ruby ‘with a design based heavily on the Python one‘. Doing a lot of sysadmin work, Ananace has been working a lot with Ruby, and also wants to get going using the SDK to write bots.
Public catalog for matrix rooms announced: matrixstats.org. The place where you can find a lot of rooms and sort them by ratings or categories. Presented rooms are collected from different homeservers; some of rooms have detailed statistics. The homeservers itself can be explored without the registration. The project is currently in beta stage, so some features may be missing. We would be glad to receive any feedback and ideas for further improvement. Additional info available at https://matrixstats.org/about, related discussions at #matrixstats:matrix.org.
Lots of work implementing chunking for DAGs to properly solve the depth vulnerability
Much-needed work has begun to classify and present the spec proposals for the Matrix specification. We’ve tagged up the all the issues in GitHub, new page will appear on matrix.org at the start of next week if I can just stop preening the generator.
Do you have a suggestion for this series? What could we be doing more of? I have a nascent plan to do ‘deeper’ conversations with people or projects that aren’t necessarily in the normal run of things, but are interesting uses of Matrix. Does this sound like something you’d want to read on a Friday afternoon? Drop a line in #twim:matrix.org or ping benpa.
One of the riot developers, luke has a fun side-project called Journal, this being a blogging platform built on matrix.
The big news this week being that I’m going to redesign the interface to focus on the personal blog use-case, optimising for easy setup and easy blog post sharing.
And hopefully push a 1.0 release that I’d be happy to use as my own personal blog.
Worth noting that the linked project page (Journal) is itself a blog using journal (the url might give you a hint of this!)
This week kitsune has been focused on ‘GTAD (Generate Things from API Description)’, which is a code generator for C++, taking API description in Swagger/OpenAPI as it’s source. Now at version 0.5, apparently GTAD
can generate correct buildable (and runnable) code to convert data structures used in CS API between JSON and C++ – for the entirety of CS API calls. That basically means that libqmatrixclient gains (so far low-level) C++ CS API for all calls in The Spec and will follow updates to it.
This is super-exciting, especially as we are going to see discussion and progress on the spec…
Trying to work our way through the regressions which keep stacking up
Lots of work on improved UTs for Groups and Replies; discussion about flux stuff
Next up is E2E verification (at last).
Android is now Kotlin enabled!
Handling abuse of the depth parameter; short-term fix deployed and longer term coming along shortly.
This destroyed progress on the algorithmic perf improvements.
Half-Shot PRs for negotiating size limits
Amber is inbound!
We’re behind on PRs – sorry Thibaut :(
Ansible stuff is being refactored based on our experiences trying to use it in the wild
status.matrix.org is coming soon!
Loads of work happening to build the Spec Proposals website, tracking workflow for all the proposals in flux and putting them into a formal RFC-style process. It should help community participation in the spec process massively whilst we finalise the longer term governance model for Matrix.org
Also looking at publishing formal roadmaps for Synapse, Dendrite and Riot (at last!) – we have them internally these days but need to just chuck them up on the web and maintain them.
Finally, GDPR work is in full swing.
This section is scraped manually from #newrooms:matrix.org, though there has not been much activity there this week. Meanwhile, there are a couple of rooms suggested by Creak which deserve some love:
Amber Brown of the Twisted project will be joining the Matrix core team in a few weeks. She’ll be focusing on Synapse implementation work, and will bring a lot of Python experience with her. Having someone working full time on synapse will increase others bandwidth for homeserver and spec work.
Matrix Live is now available, where among other things you can see this blog post being written!
After the blog post on matrix.org yesterday, lots of attention developed around the news. Take a look at this Hacker News thread with lots of discussion, which has stayed on the front page for nearly 24 hours, and a megathread on Reddit/r/linux: nearly 1000 upvotes and growing.
Staying in QMatrix-land, there is a pre-release of Quaternion available. Take a look at v0.0.9, improvements include “redactions, file downloading, room creation and settings editing, better timeline visualisation and much more”!
f0x has been beavering away on neo, and has announced that Alpha 0.01 is available now. neo is a webclient in React, it feels really light and fast. I asked f0x about the uptick in work, and he mentioned he got started with React recently, but likes it a lot.
Says TravisR: “I’ve whipped together a simple Trello notification bot for matrix: matrix-trello-bot. Future plans include the ability to create/update/delete cards and more granular control of what is notified about. Currently it acts very similar to how the Github notifications bot works.”
MTRNord is working on a “matrix DSL”, basically a config file language that generates a project Template in multiple languages. The project is just now at the discussion stage, so join #matrix_dsl:matrix.ffslfl.net to check out what’s going on. (DSL = Domain Specific Language.)
Missed this one last week, but matrix-stfu from xwiki is a message filtering tool released recently, it can “mass remove everything which was said by a particular user in a particular room”. XWiki use matrix as their internal chat system for the ~30 people at the company.
Massive experimental work on GDPR in progress – doing the thought experiment of pseudonymising MXIDs throughout Matrix on a per-room basis so that it’s possible to redact MXIDs in the event of a “right-to-erasure” GDPR event. Rich vdH is leading the work.
The good news is that introducing an MXID abstraction layer like this could help us enormously with some of Matrix’s longest-term architectural issues – i.e. account migration and portability; improving on PERSPECTIVEs for managing server identity; future support for P2P Matrix; solving the domain name reuse problem; etc.
The bad news is that it would obviously be a very significant change to the Matrix spec, although we’d be doing it in such a way which minimises impact to client implementers and keep the CS API looking as similar as possible.
We’re not sure whether we need to rush this through or not yet; still waiting for final GDPR clarification from lawyers, but it feels like this might be a good opportunity to force us to finally tackle some of these harder problems.
Meanwhile, Erik’s Delta State Resolution algorithm work is continuing well; we’re hoping to finish & merge it asap in order to get back headroom on the server.
Thankfully the matrix.org synapse itself has been relatively stable this week, other than being completely overloaded causing Freenode to be almost unusable.
We’ve started using Ansible in production, although the playbooks need some iteration before we’re fully happy to announce them & recommend folks use them as an official way of running Synapse in production.
There is a update out on try-matrix-now, matrix.org’s central listing of projects. Try filtering and see whether benpa mangled the metadata for your project. Submit any changes as markdown PRs on the repo.
benpa is currently working on a Proposals page for the Spec to properly stack spec proposal status, at last!
Articles around the web
Riot: A Distributed Way of Having IRC and VOIP Client and Home Server
#matrix_dsl:matrix.ffslfl.net As mentioned above, MTRNord is looking at creating a DSL for matrix. In his words: “A room to discuss about making a Matrix DSL to allow non coders to write simple as and bots. I would love to see some people discussing about syntax with me as this is my very first DSL and very first time trying JetBrains MSP. Anyone with Ideas what it should allow to do and how to have the syntax a welcome to join (and people who want to follow of course too)”
#trellobot:t2bot.io (from above) TravisR: “a discussion/developer room for Matrix Trello Bot (https://github.com/turt2live/matrix-trello-bot). This is a bot that notifies rooms of changes to tracked Trello boards, similar to how the Github bot works. Future enhancements include being able to create, update, and delete cards from within Matrix.”
This Week in Matrix is printed fresh every week! There will be another post before you know it, so if you’d like to be included join us in #twim:matrix.org and let us know what you’ve been working on. See you next week!
Fractal 0.1.26 was released this week, which includes an animated scroll to bottom animation, fixed MacOS compilation, and bug fixes. Since then markdown support and many other commits have been spotted…
kitsune released 0.2 of libqmatrixclient a couple of weeks ago, which he says is “the first one more or less functional and stable”!
Highlights include redactions support, files transfer (both ways), rooms creation, account data (tags, direct chat flags). Also there are a few things making libqmatrixclient clients stand out compared to, e.g., Riot: one can set per-room names/avatars, and also the best estimate for the number of unread messages can be taken from the library and shown in the client application.”
It’s able to report metrics to Prometheus and has it’s own built-in dashboard (see https://lag.t2bot.io/ for some horrendous times). In the coming weeks it’ll become more production-ready, however in the meantime if people are looking to help iron out the scalability issues, please run an instance on your homeserver.
thrrgilag wrote on behalf of uMatriks, to say that v0.10 was published to the openstore this last week which now includes unread counts for rooms. Those not already familiar, uMatriks is a native matrix client being developed for Ubuntu Touch.
There looks to be lots of activity on Construct (jzk’s fork of the charybdis ircd which adds in a C++ homeserver implementation) based on the commit log at https://github.com/jevolk/charybdis. jzk’s work there has been highlighting some of synapse’s nastier problems (e.g. thundering presence bugs), and it looks like the federation support is progressing steadily, including handling EDUs (presence & typing data).
(jzk) – https://github.com/jevolk/charybdis
tyler shared his “horrifying” project, for people that want to use Matrix to send an SMS via an XMPP bridge to their Android phone, in order to provide SMS and call notifications inside Riot.
remaeus wrote in to say “we (+rpg:verse.im) just launched an initial build of #idlerpg:verse.im (now also available at #idlerpg:matrix.org) — an automated RPG that rewards you for staying online (via the presence API). It’s an initial test case for prototyping out a bot framework called “doorman” with multi-service support, but focused on Matrix as we are using it as a “backbone” for our p2p app framework.”
Official Ansible playbook repository dev is progressing on the develop branch at https://github.com/matrix-org/matrix-ansible thanks to massive work from Michael K. We’ll blog properly about this when ready for use in earnest.
Dave has been doing an interesting experiment in syncing user profile data from HSes into ISes (useful for folks running a private federation with its own IS who want to use the IS as a global user directory); almost complete – we’ll see how this goes.
GDPR work is in progress. We’re seeing how far we can push the envelope on improving privacy, including a very interesting thought experiment on pseudonyimsing MXIDs to provide ‘perfect’ right-to-erasure. vdH is on the case. Warning: this may well require a major federation protocol bump if we go through with it.
More Python3 port PRs are landing from NotAFile – thanks! They’re getting slightly lost behind everything above.
Still stuck behind Synapse work.
Barring disasters we have at least 2 new folks starting fulltime in the core team (both synapse & dendrite) to help backend progress at the end of May however!
Lots of PRs coming in from APWhitehat though – thanks!!
uhoreg has been looking at the .well-known URI proposal again
Spec work on the core team is currently stuck behind Riot/Web and Synapse work, sadly, on the basis that burning implementation problems are more visible and harming to the project than burning spec problems.
Matrix in the News
Tom’s Hardware provided a great article on about Matrix in the French Government.
Next INpact produced two articles, one from last week introducing a project from the French government to use an open messaging platform, then a follow up provided an extremely thorough story covering Riot & Matrix following the announcement of the French Government encrypted messaging project. (Article is in French, but will Google Translate nicely!)
Huge thank you to nouts who produced a great intro to Riot and Matrix! Check it out! (Note that is also in French.)