This Week in Matrix

We have a LOT of updates this week, so let's dive straight in!

matrix-docker-ansible-deploy

We've covered the growth of this project several times in TWIM, but I wanted to give a little more attention to the work Slavi has been doing with matrix-docker-ansible-deploy. Synapse is a large Python project with many configurable options, and many optional components, so installing it has sometimes been a challenge. I have seen many reports that using Ansible and Docker, and in particular using these playbooks from Slavi, is a more sane way to install Synapse. The tools get a lot of attention and updates. This week, Slavi reports:

matrix-docker-ansible-deploy now has a self-check command that can help diagnose various configuration problems with the setup (DNS records being misconfigured, firewall ports not being open, etc).

For support and more info, come join the associated room: #matrix-docker-ansible-deploy:devture.com >

Late breaking news from Slavi:

One more matrix-docker-ansible-deploy update: the playbook can now set up Whatsapp-bridging via mautrix-whatsapp. Thanks to @izissise!

Dimension

Dimension is an integration manager for Matrix. It's written and maintained by TravisR, and allows you to an a pre-defined selection of widgets, bots and bridges to improve your self-hosted Matrix experience. Check out: https://dimension.t2bot.io/. This week, TravisR reports:

Dimension has received quite a lot of updates over the last week. Here's what's hot off the press:

• 4 new bridges can be self-hosted and managed in Dimension: Telegram, Webhooks, Slack, and Gitter.
• 3 new widgets are available: Grafana, TradingView, and Spotify.
• Add your own custom bots for people to add to their rooms.
• A dark theme has been added and is automatically applied if you use the dark theme in Riot.
• The overall UI has been updated to be slightly more modern and less bright orange.
• Various bug fixes and improvements (is it even possible to have a changelog without this?)

As per usual, if you find any bugs or have ideas for things to add to Dimension feel free to come by #dimension:t2bot.io

mautrix-telegram

tulir has been working on mautrix-telegram, and has made some massive performance improvements:

mautrix-telegram now uses the non-ORM part of SQLAlchemy for database tables that are used often. That change made the CPU usage on the t2bot.io instance drop from ~100% to ~7%

We have graphs to illustrate the improvements:

TravisR, who hosts the bridge on t2bot.io, reports that the bridge is now effectively instantaneous!

The bottleneck has returned to being synapse instead of the bridge

Discord Bridge 0.3.0-rc1

A tonne of work has happened on the Discord Bridge, and it has all been brought together in 0.3.0-rc1.

From the release notes, this is just a subset of the features:

• #251 Support for Postgresql and a newer SQLite3 backend!
• #182 Replace npmlog with winston, for logging to files and better logging overall.
• #221 Add support for m.sticker.
• #210 Discord-side moderation of matrix users.
• #259 Show Matrix replies as Discord embeds.
• #164 Bot will now mention name, topic and membership changes on Discord.
• #175 Add special discord keys onto m.room.member for ghosts

Go check out the full release notes if you're interested in the Bridge as there are many more changes. Half-Shot also noted:

Shoutout to our new member of the team, @Sorunome who did a lot of the review work behind the scenes for this release. Also, thank you to everyone who submitted a PR or an issue!

Slack Bridge 0.2.0 released!

We covered progress on the Slack Bridge previously, but Half-Shot has now declared it ready for 0.2.0 final! The bridge is reportedly running and very stable - go try it out now!

https://github.com/matrix-org/matrix-appservice-slack/releases/tag/0.2.0

Spectral

We just missed out on this update from Spectral last week. Black Hat says:

Spectral now provides an AppImage along with Flatpak build. Also, thanks to the notification codes from nheko, Spectral can show icons in notifications, and now enters corresponding room when clicking on the notification. It also gains several UI/UX improvements.
P.s. I have resubmitted Spectral to Flathub.

FAQBot

Coffee featuring two weeks in a row in TWIM:

FAQBot has been set free at last! Find its code at https://gitlab.com/Matrixcoffee/FAQBot, and its room at #faqbot:matrix.org.

FAQBot sits in various public rooms and answers common Matrix questions. Very useful for demonstrating the product to newcomers!

Also, if anyone wants to help write answers for FAQBot, https://gitlab.com/Matrixcoffee/matrix-knowledge-base is the place.

Strongly encourage people to go take a look at the knowledge base and see what can be improved.

New communities

swedneck has created a new gaming community on Matrix:

we just bridged the linux-gaming community from discord to matrix, with a matrix community and individually bridged rooms/channels and all main room is #general_linuxgaming:matrix.org community is +linux-gaming:matrix.org

i've set up an instance of matrix-appservice-discord, which is bridging some select rooms from the linux-gaming discord server to +linux-gaming:matrix.org

The Linux Gaming community has gotten a proper matrix community (+linux-gaming:matrix.org) with a fair few rooms in it, all of which are bridged to a discord channel via my matrix-appservice-discord instance.

#prismo:matrix.org and #anfora:matrix.org have also been bridged to discord.

Matrix for Ansible notifications

If you are using ansible, jcgruenhage has a useful addition that will allow you to get notifications from matrix:

After over a month of waiting, the matrix notification module has been merged on to ansible devel which will be released as ansible 2.8 early next year. Src: https://github.com/ansible/ansible/pull/45823

mxisd v1.2.0-beta.3

Max:

mxisd v1.2.0-beta.3 is out with the start of a brand new Identity store based on arbitrary executable, to connect to anything and everything. Authentication is implemented at the moment (see doc). Feedback is very welcome to improve as much as possible for v1.2.0

MSC1695 Message Edits

Discussion of message editing, in particular for how message edits from Bridges are handled has progressed. Nothing is final yet so check out https://github.com/matrix-org/matrix-doc/pull/1695 for the latest.

Quaternion translations: German and Polish now available

Last week we had an update from kitsune to say that there was a new Lokalise project to allow Quaternion translations. This week, we learn that the first translations are now available:

First couple of translations - German and Polish - have made it to the released Quaternion 0.0.9.3 - thanks to krombel and krkk for their contributions! Swedish and Russian translations are in the works.

Fluffychat

We anticipated it last week, but here it is:

The first stable release of the #matrix messenger #fluffychat is out now. ? Get it from: https://www.fluffy.chat
Thanks to all who have helped me. Thanks to regionetz for hosting the ubports.chat homeserver, thanks to @matrix for the awesome work, thanks to @Ubports for this awesome platform and to fabiyamada, advocatux, wayneoutthere, lionelb, Diogo, mithgarthsormr, mark, and all the awesome people from the community!!
With your help, Ubuntu Touch is still alive and has got a new stable messenger!

Informo: new bot for spec changes

Informo is an ambitious project hoping to be a "decentralised news network, making information accessible". The project lives at https://github.com/Informo, but for now you can join #discuss:weu.informo.network to get their latest news.

This week, vabd reports:

We made a Matrix bot that shouts about updates to change submissions to Informo's specifications. It basically processes all changes made to the list of labels for each issue and PR of a GitHub repository's, and generates a notice message that it sends to the configured room(s). We made it because we wanted the people that are interested in Informo to know in real time about any change made to the state of proposals, along with the calls for public reviews. We just set it up in #discuss:weu.informo.network, and published its source code along with a built binary here: https://github.com/Informo/specs-bot
It might also be worth noticing that, although we designed it to shout about updates to Informo's specifications proposals, we also made it compatible with other projects, e.g. the Matrix specs

Push-to-talk functionality with Jitsi on Riot

anoa has been making improvements to Video Calling on Riot:

I've been working on global push-to-talk functionality with Jitsi on Riot. I've got toggle on/off functionality working, but still trying for walkie-talkie mode. To do so, I need to get this library working with Riot: https://github.com/WilixLead/iohook
If anyone has experience with native Node modules and/or Electron, please hit me up! @andrewm:amorgan.xyz

Greetings from Mozfest

Mozfest is a tech-focused event happening this weekend in London. Neil and I have been along tonight and we've been chatting to a lot of people about Matrix, decentralisation and all those things we love! Check out our very short and sweet video below!

This is the start of Season Three of Matrix Live. Matrix Live seasons are variable in length, based on the data available so far. From this season, Matrix Live with change the format slightly, based on feedback. The videos will try to be a bit more interesting, varied, and deep. With this video being the start of a new season, it was meant to be more substantive with us talking to Mozfest-ites, but I lost track of time, so this shorter but still energetic video will hopefully convey the idea.

Bridges and SDKs

Slack Bridge 0.2.0-rc1

Half-Shot, master of the bridge, has been working with Cadair to produce a new RC for bridging Matrix and Slack.

We've got a Slack RC out at long last! Rejoice! It's absolutely massive and full of features.

Half-Shot even provided this explanation of how it works:

Integrations manager > sign into slack > click your channel > ??? > channel bridged :)

See release notes:

This is the first release of the Slack bridge. 0.1.0 has been the version number for previous efforts but was never an official release.

To test this integration, use Riot on https://riot.im/develop and select the "Event Bridging" when adding the integration.

maubot Python

tulir's maubot bot system:

The maubot Python rewrite I twimmed two weeks ago is now complete. The plugin system seems to work well and I'm pretty sure I'll be able to implement proper plugin reloading now. Next I'll implement plugin config storage and some way to manage maubot and plugins (maybe a plugin to manage plugins?)

Clients

Riot Web 0.17, Riot Android 0.8.18: Lazy Loading

This week saw the launch of Riot Web 0.17, and two bug-release updates (0.17.1 and 0.17.2) to fix issues on the Windows desktop app. This version is substantially faster due to its use of Lazy Loading room members. Read more here.   Meanwhile, Travis continues his foray into 'first impressions' bugs - including an initial implementation of .well-known URI support!

Riot Android 0.8.18 is also available from the Play Store and F-Droid, with Lazy Loading option available in the Labs menu (but still has a few bugs left).

Riot iOS meanwhile is busy implementing incremental server-side E2E key backups, and there's generally been a huge amount of work on E2E encryption UX across the board in preparation for all-new E2E on Web and iOS.  More details will be coming soon!

With this done, Riot is now getting a lot more attention on the impending redesign, with Bruno starting to merge code to the experimental branch.

Quaternion 0.0.9.3

After being in RC for a week, Quaternion 0.0.9.3 is ready and will be released over the weekend. Most importantly, you can now translate it into your language! Just head over to https://lokalise.co/public/730769035bbc328c31e863.62506391/, register (you can optionally reuse your GitHub account), ask in #qmatrixclient:matrix.org to add your language to the list (if it's not there yet) and start translating!

Coffee on Matrix Console for Android

Coffee rolled a natural twenty for bravery this week.

I have decided to take up maintainership of the Matrix Console for Android client. This is still the only multi-account capable Android client, but it is unmaintained and growing long in tooth. In particular, the API endpoints it uses may be removed from Synapse soon. I will not be developing new features for it, but I will integrate reasonable patches if others want to take that on. My own goals are to remove GCM and analytics, and get it added to F-Droid. And of course to switch to the latest API. As part of this work, I have been fighting Gradle and its bugs to get matrix-android-sdk to build together with matrix-android-console as a git submodule, so it's no longer necessary to inject the precompiled sdk into the source code. I did not win yet.

FluffyChat stable release creeps closer

Ubuntu Touch fans will know that FluffyChat has been progressing rapidly, and the project is approaching a first stable release. You can find current features being tested as part of the release here.

nheko

mujx has decided to stop maintaining nheko. Since many people are using it, we hope that someone will step in to continue his work. Thanks to mujx for his work so far.

Featured on Mozilla Hacks blog

If you like blog posts about Matrix (and if you're reading this, you may well do), then you'll be interested to see that Matrix was featured on the Mozilla Hacks blog: Decentralised, Real-Time, Interoperable Communication with Matrix. The article was also picked up by Hacker News.

Raiden Network uses Matrix for transport - this article explains it

The Raiden Network enables fast payments for Ethereum. From their website:

The Raiden Network is an off-chain scaling solution, enabling near-instant, low-fee and scalable payments. It's complementary to the Ethereum blockchain and works with any ERC20 compatible token.

To help explain their use of Matrix in their solution, they have prepared an article: Raiden Transport Explained.

Synapse 0.33.7

Soon, so soon, there will be a full Python 3 release for those running Synapse in worker configuration, but this one is not it. Check out Synapse 0.33.7 changes here.

Synapse on kubernetes update

Ananace:

tracking the latest synapse release on my K8s optimized image. Got no real time to work on things due to deadlines at work, but that should end come November, so expect more odd Ruby stuff after that point.

"Odd Ruby stuff" will hopefully include a return to the Matrix Ruby SDK!

mxisd v1.2.0-beta.2

Max continues work on mxisd:

mxisd v1.2.0-beta.2 is out, fixing bugs found during beta.1

Cadair at GSOC

Cadair was at the GSOC 2018 Mentors Summit in Mountain View last week:

I attended the GSOC mentor summit. I had some great conversations with people who are using matrix and with people about bridging in different chat services. A lot of matrix stickers all vanished off the overloaded sticker table. I have lots of ideas for GSOC next year, and plan to try and get many more community projects involved. Finally, I dont need to eat chocolate for a month.

GSOC chocolate: GCHOC

Until next week

Next Friday /me is going to be at MozFest 2018, promoting Matrix, Open Source, decentralisation etc. (all the stuff we know and love), so I may change the schedule a little next week.  We're also going to reboot Matrix Live, so consider this a hiatus before Season 3 begins next week!!

Matrix Spec updates

• MSC1695 Half-Shot has a newly released proposal relating to Message Edits: "The key difference to this proposal is that it's only the event schema which is based off the relates stuff. It does NOT do any kind of aggregations but simply is a format to allow bridges/clients to specify edits in the metadata."
• MSC1693 Erik has been thinking about state res v2 in the case of rejected events

libQMatrixClient 0.4.0

We mentioned libQMatrixClient progress last week, but this week kitsune reports:

libQMatrixClient 0.4.0 has just been released: CS API 0.4.0, local echo, caching avatars to disk and more - check the release notes at https://github.com/QMatrixClient/libqmatrixclient/releases/v0.4.0!

Thanks for the work also go to Black Hat (who is using libQMatrixClient to author Spectral) and delijati.

Informo

vadb, back again with news about Informo:

The Informo specifications documentation is out! ? It now lives right here: https://specs.informo.network
As a reminder, we were working on "phase 1" of this documentation, which goal was to outline how Informo was going to work. This is basically the foundation for a more complete technical documentation, which shall come later.
The idea is to enable people to give it a look as soon as possible, and to enable them to contribute to Informo as early in the design process as possible. The spec is entirely open, with a process for contributions documented and all its source files available on GitHub here: https://github.com/Informo/specs
The online version I initially linked is a live version from the repo's master branch, which is updated each time commits are pushed to it (including on merging a PR).
If people have any question or remark regarding the specs, or Informo in a general manner, we'll gladly answer them in #discuss:weu.informo.network (or on the #informo IRC channel on Freenode, which is bridged to the Matrix room)! ?

mxisd v1.2.0-beta.1 (beta release)

Max has continued work on mxisd, an Identity Server:

mxisd got a new beta release v1.2.0-beta.1.
It adds the ability to send email notification about room invites to existing accounts in Identity store that might not have been provisioned in the homeserver yet as users never logged in, or for users' profile that are not auto-populated with 3PIDs. This is especially targeted for onboarding/enrolling times when adopoting Matrix in groups/corporations or for custom setups.
We would love to hear about your experience with it!

matrix-trello-bot

TravisR has been working on matrix-trello-bot:

matrix-trello-bot has received quite the overhaul. It's gone from a small bot that had pre-determined notifications for board activity to a much more capable bot. The bot now supports managing boards (yes, you can create a board from within Matrix!), lists, and cards including creating, moving, archiving, assigning, and querying.

On top of all that, the bot respects that not every room will want to receive every possible notification and offers commands to pick and choose which events it should notify about on a per-board basis. Feel free to give it a spin and check out the massive help menu with !trello help. Please send bugs and suggestions to GitHub :)

The help menu really is massive! Travis provided an image to highlight the scope of functionality provided (click for larger version):

Clients

Riot 0.16.6 Released

Fixes only these two issues:

• Firefox private mode being broken > (https://github.com/matrix-org/matrix-react-sdk/pull/2195)
• Breakage when mixing /app and /develop use (issue #7432)

As both fixes are not relevant for electron, we didn't release 0.16.6 as the electron app, but rest assured 0.17 will of course be released as electron.

Fluffychat

We missed this last week, but FluffyChat (client for Ubuntu Touch) v0.6.0 was released, with LOTS of new features:

• User profiles
• Design improvements
• Audioplayer in chat
• Videoplayer in chat (only audio at the moment)
• Imageviewer
• Edit chat aliases
• Edit chat settings and permissions
• Kick, ban and unban users
• Edit user permissions
• New invite page
• Display and edit chat topics
• Change chat avatar
• Change user avatar
• Edit phone numbers
• Edit email addresses
• Display and edit archived chats
• New add-chat and add-contact pages
• Display contacts and find contacts with their phone number or email address
• Discover public chats on the user's homeserver
• Registration (currently only working with ubports.chat and NOT with matrix.org due captchas)
• Register and login with phone number
• Edit identity-server
• Add in-app viewer for the privacy policy

Construct Docker image

mujx has created a new Docker image for Construct, the C++ homeserver:

Construct has a new docker image (https://github.com/matrix-construct/docker-construct) which will ease the process of deployment

Translations for Quaternion

If it didn't seem that kitsune was busy enough, he also announced translations for Quaternion (the original client based on libQMatrixClient):

Quaternion is now localisable! The web platform for translation will be set up in the nearest days; meanwhile those who can't wait can just take the repo, look how the German translation is made and copy!

That's all we can tell you!

Safe travels to Cadair, who is off to represent Matrix at the GSOC 2018 mentors summit. Check out Matrix Live below:

Bridges

Half-Shot is supposedly back at university, but he's productive enough to have THREE updates this week (delivered in ascending order of interestingness):

matrix-appservice-irc 0.11.2:

https://github.com/matrix-org/matrix-appservice-irc/releases/tag/0.11.2 is just a hotfix release containing a few things we fixed in develop so yay.

matrix-appservice-node v0.3.5:

https://github.com/matrix-org/matrix-appservice-node/blob/master/CHANGELOG.md#v035 is a long overdue update that just updates dependencies so we are no longer exposing lots of vulnerabilities
Updated body-parser, express, morgan and request packages to fix security vulnerabilities.

matrix-appservice-bridge 1.7.0 released:

matrix-appservice-bridge is out! Bundled in this version are major updates to dependencies (to stop the warnings), a logging component for quick setup of pretty winston logs and a provisoner "validator" to allow bridge admins to curate which bridges can be linked into the same room to stop the dreaded double bridge issue.
More details can be found at https://github.com/matrix-org/matrix-appservice-bridge/releases/tag/1.7.0

Informo news project

Informo is an ambitious project hoping to be a "decentralised news network, making information accessible". The project lives at https://github.com/Informo, but for now you can join #discuss:weu.informo.network to get their latest news.

vadb provided an update:

Hey there, here's a quick update to let you know that we're making great progress towards the completion of phase 1 of Informo's specifications' documentation (i.e. general outline without going to much in depth into the technical specifications)! I've also updated the board to cut the big "Basic spec" task down into smaller tasks to give a more precise insight of our current progress
Once that phase 1 is done we'll publish the whole thing and migrate our work to a public GitHub repository so people can get to know Informo a bit better and contribute towards making the doc better if they want

Riot

riot-web 16.5 released, with matrix-js-sdk 0.11.1:

Go see the latest changes and make sure to start using room member Lazy Loading with the latest matrix-js-sdk.

Libraries

Black Hat is writing an Olm wrapper for Qt:

I am writing a Qt wrapper for Olm. It has APIs similar to python-olm

kitsune and Black Hat are working on libQMatrixClient:

libQMatrixClient is nearing its release 0.4.0, with low-level support of CS API (coincidentally) 0.4.0 added this week. Another very useful addition is caching avatar images to the disk, not only in memory, relieving the network from dozens of extra requests when a client powered by the library is restarted. All thanks for the avatar caching PR go to Black Hat!

SDKs

tulir on mautrix-go and mautrix-python:

My gomatrix fork is now mautrix-go and the basic client API stuff in mautrix-python is starting to be usable enough for me to start the rewrite of maubot in Python.
For a bit of background, maubot is a dynamic plugin-based bot system. It's currently written in Go, but due to many limitations in the Go plugin system, I decided to rewrite it in Python. matrix-python-sdk doesn't have asyncio or a maintainer, so I decided to make a new framework that combines my existing mautrix-appservice-python framework with a more generic client API layer. (tulir)

Homeservers

Max introduces his new server, Gridepo:

mxhsd work has resumed via a new project, Gridepo, which will be a Matrix/Grid dual-stack server. While mxhsd focused on researching the protocol and reverse-engineering the spec, Gridepo will follow whatever is specced in its Matrix compatible mode.

Synapse 0.33.6 released: >

It's literally the previous blog post!

we've been focusing on fixing a whole host of federation bugs to improve reliability and latency. Additionally we've squashed some py3 bugs, improved lazy loading and been working hard in the background to improve our CI infrastructure. Finally, we cleaned up the Docker file, the image is now half the size of 0.33.5.1's standing at 58 MB.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI.

DevOps and Hosting

Slavi on matrix-docker-ansible-deploy:

matrix-docker-ansible-deploy has received some more improvements lately. They're mostly about the ability to tweak things affecting performance: Synapse cache factor configuration, the ability to enable/disable user presence tracking. The logging situation has also been improved for all Docker containers, so that their output would not get logged twice (once by systemd's journal, once by Docker) - something which was previously causing unpredictably-high disk usage for long-running containers.

ananace, Synapse for kubernetes:

Updated the K8s tweaked packaging for Synapse 0.33.6 (And added a preliminary Python 3 version there as well)

swedneck (literally) on IPFS:

swedneck is hosting riot on IPFS: https://riot.swedneck.xyz, see also https://riot-all.swedneck.xyz for all the versions currently hosted there.

Bots

TravisR, always working on bots:

Through use of the https://emoncms.org API, the makerspace I help run now has power monitoring statistics available from within matrix.

Rooms

Coffee, maintaining #synapse:matrix.org:

#synapse:matrix.org became an official Matrix room! It also had a small face lift, changing from "Synapse Community" to "Synapse Admins", hopefully making its purpose and intended audience clearer.

That's all for now

That's all for now. No Matrix Live this week, but we'll be back next week with more Matrix news and another episode of Matrix Live!

DSN Traveller

If you've spent any time using Matrix public rooms, you've probably seen the bot DSN Traveller. This is a post-grad project from Florian Jacob, an informatics student at the Karlsruhe Institute for Technology. This week, Florian handed in his thesis on Matrix!

In summary, I could show that Matrix has few large but many small servers. Large servers reduce the overall network load, but a significant fraction of the load is concentrated in them. Introducing more small servers would further increase the load concentration. The Matrix event graph as a Conflict-Free Replicated Data Type showed to be well-suited for reliable messaging and history synchronization, and is applicable beyond Matrix.
I'm now working on a scientific paper on the results, which will boil down the more than 80 pages of the thesis to something much more digestible. ? You'll hear it in TWIM when that is finished!

Room for discussions: #dsn-traveller:matrix.org
Website with more information on the DSN Traveller: https://dsn-traveller.dsn.scc.kit.edu/

This really is exciting stuff! The thesis will be made available online in the future (we'll post it here.) Florian is also hoping to continue his work into Matrix research:

I'm currently in the process of trying to secure funding for a doctorate with Matrix as the topic, as that's where I can proof experience.

#open-science:matrix.vgorcum.com

All this talk of academia led Mathijs and Cadair to create #open-science:matrix.vgorcum.com, room for discussion of Open Science topics.

Half-Shot IRC Connection Tracker

As you may know, although he's now back studying for the final year of his Computer Science degree, Half-Shot will continue to dedicate some time to bridge maintenance. He's been working on IRC Connection Tracker, the next gen bridge for Matrix-IRC:

The IRC connection tracker has had yet more code and love applied to it. The headline changes are:

• We now have a fully working IRC client that can connect to an IRCd, join channels and chat. These client's persist over > the lifetime of the service.
• There is a tool included with the service ircctl which allows you to spawn and use connections en masse. It also lets > you list the state of the currently connected clients.
• Work has just begun on a client library for connecting this up to the bridge, but should be swiftly completed thanks to…
• A brand new spec website in the works for describing the protocol (thanks Brendan for pointing me in the right direction)

Spec docs: https://half-shot.github.io/irc-conntrack/ Repo: https://github.com/Half-Shot/irc-conntrack/ Room: #irc-conntrack:half-shot.uk

Fractal

Alexandre Franke has the handle on Fractal, the Matrix client for GNOME:

This week in Fractal, more refactoring and small bugfixes. About 50 commits by 5 people, one of which made their first contributions this week (congrats Rasmus!).

Julian Sparber, who was part of GSOC 2018 on the Fractal team, has been working on Room History:

The room history refactor I was working on for fractal is upstream, now we can start to improve how messages are displayed and make the loading of older messages better.

Julian is also pleased to still be heavily involved with Fractal outside GSOC.

Synapse

Synapse 0.33.5.1 was released with Py3 beta support:

0.33.5.1 is an interesting release. On the one hand it contains the usual bug fixes and performance improvements of a point release, but it also our first versioned release where monolith installs can be run under Python 3.5 and 3.6! Python 3 support is very much in beta, so please be cautious but if you would like to try running under a py3 environment we'd love to get your feedback.

Check out https://hub.docker.com/r/avhost/docker-matrix/tags/ for Python 3 docker images of Synapse, look for the v0.33.5.1.dev.py3 tag.
Says Mathijs:

it is functional, but much like python3 support it is still a work in progress, hence the larger size

Matrix on Hyperboria

jcg has set up hypertrix.ovh:

I spent last night setting up hypertrix.ovh, a matrix server only listening on Hyperboria, a cjdns based end-to-end encrypted IPv6 overlay mesh network. I'd be glad if someone could be found to peer and federate with me there! Registration is open, but your client needs to be connected to Hyperboria to be able to talk to the server.

If you are currently using Hyperboria, go join hypertrix.ovh, or start your matrix server listening on it, and go chat to JC!

Seaglass

Lots of discussion about this project, specifically the question of how to efficiently render Rich Text. macOS does not make this easy, so a solution being considered is to use a WebKit for room rendering:

WebKit has the advantage of being super super fast on macOS, and also very low overheads
The current approach uses Cocoa NSTableViews and it's horrible because Apple clearly couldn't decide how they wanted them to work and therefore it's not very optimised
Moving to WebKit only adds about 16mb to the RAM usage and redraws far faster than the NSTableViews can when resizing etc, and we'll save a lot on the text formatting too which currently is a bit of a mess

Cadair and Half-Shot on the Slack Bridge

Cadair has been helping Half-Shot with bridge maintenance, specifically by contributing to matrix-appservice-slack.

matrix-appservice-slack is a Node project, built on matrix-appservice-bridge, which is designed to bridge messages and metadata between Slack and Matrix. These updates:

• Improves pills substitutions to use the new slack mentions
• Add Gitter style edit bridging
• Adds a config option to specify the directory the user-store.db and room-store.db files go in
• Converts slack snippets to inline code dumps in matrix

ma1uta tests for client

ma1uta has been adding tests to jmsdk.

Spectral

Black Hat has been working on the client formerly-known-as-Matrique: Spectral:

I added elevation shadows for some components, such as message bubble, panels, etc.

Native Tor onion service enabled for matrix.org and riot.im

Cloudflare now provide Onion routing, this service has been enabled for matrix.org and riot.im. Cloudflare have a thorough explanation which is worth reading: https://blog.cloudflare.com/cloudflare-onion-service/.

We just turned on the new native Tor onion service support for https://t.co/vidAnPoIo2 & https://t.co/UIjS6gDkvf in cloudflare; feedback welcome! https://t.co/keXC4bjo5F pic.twitter.com/nTRmGHCt8P

— Matrix (@matrixdotorg) September 28, 2018

Spec

September was mainly spent cleaning up loose ends on the Spec after all the releases at the end of August, and catching up on the never-ending maintenance burden of improving Synapse.  However, in October the plan is to to go back again to working full out on the S2S r0 release. Wish us luck...

Riot

• Lazy loading members is now on by default on riot.im/develop - reducing Riot's RAM by 3-5x.  Please give it a go and test it before we ship it in Riot 0.17 (probably next week) so we can iron out any last bugs (which will probably look like user profiles going missing)
• Lazy loading also ships by default in Riot/iOS in Testflight 0.7.4 - if you want in on Testflight let us know in #riot-ios:matrix.org and we'll share an invite link!
• Lazy loading in Riot/Android coming real soon now; it's behind a labs flag on the develop branch if you want to experiment.
• Travis has started attacking the Riot/Web 'First Impressions' project (starting with unbreaking onboarding in Riot/Web when GDPR consent is enabled)
• Lots and lots of UX work from Nad on E2E, Communities, Onboarding and the overall redesign, complete with a redesign workshop with Jouni.
• Aiming for end of Oct for first cut of redesign to be live as an experimental branch on riot.im.
• Lots and lots of E2E implementation work in general; backups, cross-signing, and verification.

The end is nigh!

But only for this blog post! Check out Matrix Live below, and we'll see you back here next week. :D

https://youtu.be/zo4IH4nUQ9w

Nheko 0.6.0 released!

Get latest stable releases from GitHub.

Features

• Support for sending & receiving markdown formatted messages.
• Import/Export of megolm session keys. (Incompatible with Riot)
• macOS: The native emoji picker can be used.
• Context menu option to show the raw text message of an event.
• Rooms with unread messages are marked in the room list.
• Clicking on a user pill link will open the user profile.

Spec Proposals: E2E Cross-signing and bi-directional key verification

uhoreg has written up a new work-in-progress proposal for E2E cross-signing.

Also, although it wasn't this week, I don't think that we have previously announced the proposal for bi-directional key verification using QR codes: https://github.com/matrix-org/matrix-doc/pull/1544

Jeon

Massive update from ma1uta about his Jeon project! This update brings Jeon into line with the most recent updates to the Client-Server, Application Service, Push and Identity APIs.

In ma1uta's words:

Jeon is a set of the java interfaces and classes which describes the Matrix API.

• client-api: r0.4.0-1 corresponds to the r0.4.0 C2S API.
• application-api: r0.1.0-1 corresponse to the r0.1.0 AS API,
• push-api: r0.1.0-1 corresponds to the r0.1.0 PUSH API,
• identity-api: r0.1.0-1 corresponds to the r0.1.0 IS API.
• All artefacts available from the Maven Central Repository.

Major changes:

• Full support for the corresponding Matrix api.
• Changed version for displaying the Matrix api version.
• Added support to the asynchronous responses.

Also updated the swagger schemas generated from the code:

• https://ma1uta.github.io/swagger/c2s.json - C2S API
• https://ma1uta.github.io/swagger/as.json - AS API
• https://ma1uta.github.io/swagger/push.json - PUSH API
• https://ma1uta.github.io/swagger/id.json - IS API

And the first hotfix: application-api r0.1.0-2 with fixed url (/transactions has been changes to the _matrix/app/v1/transactions). And this release will break all AS because synapse sends transactions to the old url.

Join #jeon:matrix.org to discuss the progress of this product more.

VoIP signalling support has landed in libQMatrixClient

Exciting times for libQMatrixClient!! Thanks kitsune, developer of libQMatrixClient and Quaternion:

After some pretty long time of being in a PR/fork, VoIP signalling support has landed in libQMatrixClient! Many thanks to mariogrip (the founder of UBports) for the initial code and to delijati (a developer behind uMatriks) for getting it to work with the most recent library.
The actual VoIP stack does not come included, client developers have to take whatever WebRTC implementation is available for their platform and glue the pieces together. However, as the example of uMatriks shows, this now becomes relatively easy if your platform is on good terms with WebRTC (like UBports). I look forward to further work with UBports community on keeping this platform a first-class Matrix citizen.

Go chat in #quaternion:matrix.org to see the ways libQMatrixClient is developing.

Matrique is now Spectral

After intense discussion, there is a new name for Matrique: Spectral. The repo now sites at https://gitlab.com/spectral-im/spectral, there is a new room at #spectral:encom.eu.org, and a new logo:

FluffyChat featured in OpenStore

FluffyChat is getting some love from OpenStore, the official Ubuntu Touch app store: this week it was the featured app.

Matrix Corporal

@slavi:devture.com, creator of Matrix Corporal (a Matrix server configuration tool - "Kubernetes for Matrix"):

Matrix Corporal has received some updates over the past few weeks since its initial release: a couple of additional HTTP APIs for retrieving/destroying user access tokens; more consistency with the Matrix Client-Server specification when it comes to error responses; faster reconciliation for user accounts that are joined to many/large rooms.

matrix-docker-ansible-deploy

Another project from @slavi:devture.com, for those who prefer their DevOps ansible-flavoured:

matrix-docker-ansible-deploy now also helps you set up service discovery as per the .well-known Matrix specification.

jcg ansible PR for matrix notifications

jcg has an upstream PR to have matrix notifications in ansible. Combined with Slavi's matrix-docker-ansible-deploy above, you can get Matrix notifications about issues with your Matrix deployment…

Seaglass E2E + self-update

neilalexander:

Seaglass end-to-end encryption support is now complete, including device verification and blacklisting, key sharing requests, key import and export (which should be compatible with Riot) and re-requesting keys

This is really exciting news for macOS matrix users!

I'm also working on auto-updating Seaglass with Aaron Raimist's help in addition to finishing E2E support :-)

:-) is right!

ma1uta jmsdk

ma1uta must have been in a work-on-Matrix mode this week, because he has also updated jmsdk:

I have released a new version of the java client (https://github.com/ma1uta/jmsdk/tree/master/client-sdk). The new client works is asynchronous mode, each method doesn't block the thread and return the CompletableFuture (promise in java). Then you can block thread to get the response or build a asynchronous promises chain.

Finally:
ma1uta is also looking forward to the release of Java 11:

with the Curve25519 key agreement (http://openjdk.java.net/jeps/324) and will try make a pure java implementation of the olm/megolm. Just for fun. :)

synapse-purge

Maze, seeing that his synapse database was already at several gigabytes, decided to produce a tool to shrink it:

The synapse-purge tool allows homeserver admins to free disk space by purging old room events from the synapse database. It is an alternative for synpurge which currently does not work correctly.

Configuration is minimalistic at the moment. Meaning it purges all remote rooms on the server with a globally configured preservation period.

synapse

andrewsh: 0.33.4 uploaded to Debian's stretch-backports, pending approval.

0.33.5rc1 is now available, with the big news being the inclusion of support for Python 3.5 and 3.6! Hawkowl's Py3 has merged for monoliths and is working pretty well, looking like 2-3x RAM improvement. Please help us test!

Erik's state compressor tool is pretty much finished, we've been starting to run it on things and it reduces disk usage for the state group table by at least 10x.

The only catch is that it's quite DB heavy whilst it runs, so we haven't run it on Matrix.org yet.

Fractal

Alexandre Franke and the Fractal team:

refactoring of the history and other parts is going on in the master branch of Fractal. We also cleaned up build and dependency related bits.

maubot and sedbot

tulir:

I made some updates to maubot and fixed most of the sedbot (S. Edbot) issues people had reported.

tulir used maubot to create a factorial bot: >

I might also make some useful bots soon

And so it was - late breaking news that maubot has been used to develop a Dictionary-definition-bot! Not available for public use yet but it proves that the project is useful!

Riot Web

Lazy Loading remains the focus, we're getting closer with more bugs solved this week! To enable Lazy Loading room members and get speed and memory benefits in Riot, use the develop branch and enable "Lazy Loading" under "Labs" in the settings.

Lots of final bug hunting for lazy loading - this is taking longer than you might expect because we're doing end-to-end CI everywhere.

Lots of work on E2E, Dave has been working on:

UI for e2e key backup that's waiting for some lower level bits
and hopefully our e2e core code is moving from asm.js to webassembly making it, by current estimations, significantly faster.

Redesign work continuing as well - Bruno has been working on it this week, Jouni the designer will visit next week to continue the process.

Nad has joined us to help with design bandwidth and is working on the onboarding flows for the redesign as well as fixing all the UX issues in Communities!

Riot Mobile + Mobile SDKs

Lots of work on Lazy Loading - to be released along with Riot Web.

Bridges

Half-Shot is joining us to work part-time on bridges going forwards - this is great news, especially for his connection-based IRC bridging antics as well as catching up on the PR and maintenance backlog for the IRC bridge and Slack bridge.

Modular

Modular (Hosted Homeservers) has first customers; if you want to give it a go please let us know!

Finally

Thanks for reading, take a look at Matrix Live below!

Dimension

Update (this got lost in the original post; sorry Travis!): Dimension received a security update - if you run your own Dimension instance it is strongly recommended you update right away. Telegram bridge support in Dimension is underway, with more updates expected for next week in Matrix.

Clients

Fluffychat

It's been some months since we checked in with FluffyChat. If you're a Ubuntu Touch user, or have a device running it, you should see the progress that has been made recently on this Matrix client. Collected changelog 0.5.0 to latest (0.5.4):

• Search chats
• Chat avatars
• Search users in chats
• Security & Privacy settings:

1. Disable typing notifications
2. Auto-accept invitations

• New message status:

1. Sending: Activity indicator
2. Sent: Little cloud
3. Received: Tick
4. Seen by someone: Usericon

• Display stickers
• Minor UI improvements
• FluffyChat now automatically opens the link to the matrix.org consens
• Updated translations

Seaglass

Neil has been keeping up the pace with Seaglass development:

Seaglass has had a substantial rewrite to the room cache to help improve reliability and reduce crashes, better thumbnail behaviour on inline images, various tiny visual tweaks, in-window blending, support for encryption key sharing requests for E2E rooms.

Rendering performance has been massively increased (if you ignore the occasional bug). Resizing the window shouldn't be so slow anymore and a lot of avatar image operations are no longer repeated unnecessarily

Other than that this week has mostly featured lots and lots of bug fixes, hopefully lots of crashes fixed.

Screenshot below shows the new E2E UI:

Quaternion

When not escaping typhoons, kitsune has found some time to continue work on Quaternion:

Quaternion's master branch is alive again - it's been prone to crashes in the last two weeks, now it shouldn't. Feel free to try the new room list organised by tag!

SimpleMatrix

MTRNord has been working on SimpleMatrix:

SimpleMatrix now supports Basic messages sending (with Commonmark) and basic receiving of messages.

miniVector

Marcus has re-packaged miniVector for F-Droid:

There's now a second matrix client available in F-Droid: https://f-droid.org/packages/com.lavadip.miniVector/

This is a fork of Riot Android done by hrjet, f-droid release done by me. It's removing mostly jitsi group call functionality and some other smaller stuff. In doing so it manages to require far less permissions and is also only 12 MB in size instead of riots 20 MB.

Matrique

Black Hat:

Matrique gained alpha support for multiple accounts

This is thanks to leaning on libqmatrixclient's native multiple account support!

Riot Web 0.16.4 released

This is pretty much a maintenance release - fixing the DM avatar regression that crept in with 0.16.3, adding better support for warning users when their client hasn't yet synced with the server, and the final bits of work needed before we can turn on membership Lazy Loading in the upcoming Riot 0.17.

Full changelogs as always are split over the three projects which make up Riot/Web:

• https://github.com/vector-im/riot-web/blob/master/CHANGELOG.md#changes-in-0164-2018-09-10
• https://github.com/matrix-org/matrix-react-sdk/blob/master/CHANGELOG.md#changes-in-0134-2018-09-10
• https://github.com/matrix-org/matrix-js-sdk/blob/master/CHANGELOG.md#changes-in-0110-rc1-2018-09-07

SDKs and Libraries

libQMatrixClient ecosystem

As you may know, Matrique, led by Black Hat, and Quaternion, led by kitsune, are both projects build using libQMatrixClient, a Qt5 library from kitsune designed for writing Matrix clients. While kitsune has been working on the library for some time, Black Hat has also now started making contributions:

libQMatrixClient now has a pkg-config file to further ease clients building on Linux systems, as well as more convenient API to track read markers if all users, not just of the local one.

matrix-js-sdk v0.11.0 released

This release contains support for lazy loading room members, and also some breaking changes relating to startClient().

• Support for lazy loading members. This should improve performance for users who joined big rooms a lot. Pass to lazyLoadMembers = true option when calling startClient.
• MatrixClient::startClient now returns a Promise. No method should be called on the client before that promise resolves. Before this method didn't return anything.
• A new CATCHUP sync state, emitted by MatrixClient#"sync" and returned by MatrixClient::getSyncState(), when doing initial sync after the ERROR state. See MatrixClient documentation for details.
• RoomState::maySendEvent('m.room.message', userId) & RoomState::maySendMessage(userId) do not check the membership of the user anymore, only the power level. To check if the syncing user is allowed to write in a room, use Room::maySendMessage() as RoomState is not always aware of the syncing user's membership anymore, in case lazy loading of members is enabled.

Synapse

Synapse 0.33.4 was released, with a whole host of bug fixes, some enhancements to resource usage management and a bunch of internal changes in readiness for room member state lazy loading and our ongoing port to Python 3.

Meanwhile, Python 3 support for monolithic (non-worker) Synapses has finally landed on the develop branch, thanks to massive work from hawkowl and notafile - if you want to help us test and flush out any remaining byte/utf8 style errors, please create a virtualenv for python 3.6 or 3.5 (twisted doesn't support 3.7 yet) and point the develop branch of Synapse at it, tail the logs for ERRORs and report them via Github if/when you see them.  In practice it seems pretty stable though, and noticeably reduces RAM and speeds things up thanks to improved GC and general performance work in Python.

We've also discovered that jemalloc works very well at improving RAM usage on Python 2 under Linux (we haven't tried it on Python 3 yet) by providing a more fragmentation-resistant malloc implementation; if you are having problems with your Synapse RAM spiking up we recommend giving it a go.  All of the Matrix.org server is using it now.

Also, lots of ops work this week; Erik has prototyped a new storage strategy for state groups which shrinks storage requirements by 10x, we'll be applying this shortly to Matrix.org otherwise we're going to run out of disk space.  There was also a regression on Synapse develop on federation, where outbound requests would get stuck and never retry - this impacted the matrix.org server badly over the course of the week, but as of Friday night we have a workaround in place.  We're not aware of it affecting anyone other than the matrix.org deployment (and we haven't got to the root cause yet).

Construct homeserver progress

This week:

Added notification counts which show up in Riot now, and expanded support for g++-7 and 8 instead of just g++-6. Construct repository is found at: https://github.com/matrix-construct/construct.

IRC Connection Tracker

Half-Shot is continuing to work on the project to split out IRC connection management from the IRC bridge, letting the bridge be restarted without interrupting IRC connections!

The project is going quite well, and is going to be used on matrix.org once production ready which will really speed up upgrades and give us near zero downtime indifferent to the size of the bridge.

At the moment the project has the ability to spin up and maintain connections, however the connections are not supporting IRC fully yet as there are bits to do on the parsing and maintaining state side. There is also work on a top-like tool to visualise and control the service outside of the bridge so we can quickly handle any oddities if they come up. Finally, it allows you to hot reload the configuration without dropping existing connections!

On the work done to support this on matrix-appservice-bridge, there is basic support for stating connections on the bridge but it's in early stages at the moment.

Spec

Travis has been tidying up loose bits on the Matrix spec this week:

• experimenting with splitting out the "Room" specific bits of the spec into a dedicated section rather than being split between the CS and SS APIs;
• trying to pin down a formal grammar for room aliases
• ...and identifiers in general.

In practice, finalising the S2S API is now blocked on proving the implementation on Synapse; work on this will resume next week and then we'll document the end result and ship the r0 at last.  Timings are going to be completely determined by available manpower and what level of ops distractions we face (c.f. the Synapse section above...).  Whilst we're waiting for the final S2S details to get hashed out, Travis is going to be helping on Riot dev, to try to stop stuff like this, as there's no point in having the platonic ideal of a perfect spec if actual users are unable to benefit from it.

#matrix-dev

#matrix-dev:matrix.org was reborn as a new room a couple of weeks ago to flush out old corrupted events, but maybe not everyone knows. Come join #matrix-dev:matrix.org, it's a starting point for all developers looking to build on the platform.  We're also rebuilding #test:matrix.org and #riot:matrix.org, although once we ship the new state resolution

A sneak peek at Modular...

Finally, there's been a massive amount of work on the New Vector side of things to soft-launch Modular - a paid hosting platform for Matrix servers (and, in future, paid integrations).  At this point we're looking for early adopters who want a dedicated Riot+Synapse for communities or companies of 50 or more users - but don't want to have to run it themselves.  Modular takes the homeserver hosting we've already been providing for Status, TADHack and others, and turns it into a mass-market product.  The pricing for early adopters is over 5x cheaper than Slack, so if you've been dying to have a reliable, fast and expertly maintained homeserver without any of the headaches of admining one yourself, please head over to https://modular.im and give it a whirl and let us know how it goes!  This is also a great way to support Matrix development in general, as money from Modular will directly keep the core Matrix team funded to work on Matrix.  Once we're happy with the soft-launch and have incorporated any feedback we'll start yelling about it as loud as we can :)

Matrix Live

We've had a bit of an accidental hiatus on Matrix Live thanks to getting submerged all the different project endgames happening atm (spec releases, lazy loading, Modular, Riot redesign etc), and for the last few Fridays we've got to midnight and beyond with too much still on the todo list to justify recording a video.  But to avoid completely falling behind, here's a slightly exhausted Saturday morning update instead (warning: Matthew is not a morning person).

---

Hi all,

Ben's away today, so this TWIM is brought to you mainly in association with Cadair's TWIMbot!

Spec Activity

Since last week's sprint to get the new spec releases out, focus on the core team has shifted exclusively to the remaining stuff needed to cut the first stable release for the Server-Server API.  In practice this means fleshing out the MSCs in flight and implementing them - work has progressed on both improving auth rules, switching event IDs to be hashes and others.  Whilst implementing this in Synapse we're also doing a complete audit and overhaul of the current federation code, hence the 0.33.3.1 security release this week.

Meanwhile, in the community, ma1uta reports:

I am working on the jeon (java matrix api) to update it to the latest stable release. Also I changed versions of api to form rX.Y.Z-N where rX.Y.Z is a API version and N is a library version within API. So, I have prepared Push API (r0.1.0-1), Identity API (r0.1.0-1) and Appservice API (r0.1.0-1) for the first release and current updating the C2S API to the r0.4.0 version.

XMPP Bridging

Are you in the market for a Matrix-XMPP bridge? When I say "market", I mean it because this week we have two announcements for bridging to XMPP! You can choose whether you'd prefer your bridge to be implemented as a puppet, or a bot.

Ma1uta has a new version of his Matrix-Xmpp bridge:

It is a double-puppet bridge which can connects the Matrix and XMPP ecosystems. Just invite the @_xmpp_master:ru-matrix.org and tell him: @_xmpp_master: connect [email protected] to connect current room with the specified conference.
You can ask about this bridge in the #matrix-jabber-java-bridge:ru-matrix.org room.
Currently supports only conferences and only m.text messages. 1:1 conversations and other message types will be later.

maze appeared this week and announced MxBridge, a new Matrix-XMPP bridge:

It works as a bot, so it is non-puppeting. Rooms can be mapped dynamically by the bot administrator(s). There is no support for 1-1 chats (yet). MxBridge is written as a multi-process application in Elixir and it should scale quite well (but don't tie me down on it ;)). https://github.com/djmaze/mxbridge

Seaglass

Neil powers onwards with Seaglass, with updates this week including:

• Displaying stickers
• Lazy-loading room history on startup to help with performance
• Scrollback support (both forwards and backwards)
• Support for Matthew's Account (aka retries on initial sync for those of us with massive initial syncs, and general perf improvements to nicely support >2000 rooms)
• Better avatar support & cosmetics on macOS Mojave
• Encryption verification support, device blacklisting and message information
• Ability to turn encryption on in rooms
• Responding to encryption being turned on in rooms
• Paranoid mode for encryption (only send to verified devices)
• Invitation support (both in UI and /invite)

Matrique

Blackhat announces that Matrique's new design is almost done, along with GNU/Linux, MacOS and Windows nightly build!

Fractal

Alexandre Franke says:

Fractal 3.30 got release alongside the rest of GNOME. It includes a bunch of new and updated translations, and redacted messages are now hidden.

Meanwhile, hidden in this screenshot, uhoreg noted that E2E plans are progressing...

Riot

Bruno has been hacking away on Riot/Web squashing the remaining Lazy Loading Members defects and various related optimisations and fixes. We also released Riot/Web 0.16.3 as a fairly minor point release (which unfortunately has a regression with DM avatars, which is fixed in 0.16.4, for which a first RC was cut a few hours ago and should be released on Monday).  Meanwhile the first cut of Lazy Loading also got implemented on Android as well. Both are hidden behind labs flags, but we're almost at a point where we can turn it on now!  Otherwise, the Riot team has got sucked into working on commercial Matrix stuff, for better or worse (all shall be revealed shortly though!)

Construct

Jason has been working heavily on Construct, and has new test users.  Construct is able to federate with Synapse and the rest of the Matrix ecosystem.  mujx has created a docker for Construct which streamlines its deployment.

Construct development is still occurring here https://github.com/jevolk/charybdis but we are now significantly closer to pushing the first release to https://github.com/matrix-construct. Also feel free to stop by in #test:zemos.net / #zemos-test:matrix.org as well -- a room hosted by Construct, of course.

tulir has now deployed using the standalone install instructions on a very small LXC VM using ZFS. Unfortunately ZFS does not support O_DIRECT (direct disk IO) which is how Construct achieves maximum performance using concurrent reads. This is not a problem though when using an SSD or for personal deployments. I'll be posting more about how Construct hacked RocksDB to use direct IO, which can get the most out of your hardware with multiple requests in-flight (even with an SSD).

Synapse

Work was split this week into spec/security work, with the critical update for 0.33.3.1 - if you haven't upgraded, please do so immediately.

Otherwise, Hawkowl has been on a mission to finish the Python 3 port, which is now almost merged.  Testers should probably wait until it fully merges to the develop branch and we'll yell about it then, but impatient adrenaline enthusiasts may want to check out the hawkowl/py3-3 branch (although it may explode in your face, mangle your DB and format your cat, and probably misses lots of recent important PRs like the 0.33.3.1 stuff).  However, i've been running a variant on some servers for the last few days without problems - and it seems (placebo effect notwithstanding) incredibly snappy...

Meanwhile, the Lazy Loaded Member implementation got sped up by 2-3x, which makes /sync roughly 2-3x faster than it would be without Lazy Loading.  This hasn't merged yet, but was the main final blocker behind Lazy Loading going live!

matrix-docker-ansible-deploy

Slavi reports:

matrix-docker-ansible-deploy now supports bridging to Telegram by installing tulir's mautrix-telegram bridge. This feature is contributed by @izissise.

In addition, Matrix Synapse is now more configurable from the playbook, with support for enabling stats-reporting, event cache size configurability, password peppering.

Matrix Python SDK needs a maintainer

We should say a huge Thank You to &Adam for his work leading the Python SDK over the previous months! Unfortunately due to a busy home life (best of luck for the second child!) he has decided to step down as lead maintainer. Anyone interested in this project should head to https://github.com/matrix-org/matrix-python-sdk/issues/279, and also come and chat in #matrix-python-sdk:matrix.org.

MatrixToyBots!

Coffee reports that:

A new bot appears! Are you a pedantic academic who likes to correct others' misuse of Latin-derived plurals? This task can now be automated for you by means of SingularBot! Also for people who just like to have some fun. Free PongBot and SmileBot included.

kitsune on Hokkaido island

I ended up being on Hokkaido island right when a major earthquake struck it; so no activity on Matrix from me in the nearest couple of days. Also, donations to GlobalGiving for the disaster relief are welcome because people are really struggling here (abusing the communication channel, sorry).

Matrix Live

...has got delayed again; sorry - we're rather overloaded atm. We'll catch up as soon as we can.

The Spec

As many know, we've been aiming for the end of August to cut the first ever stable releases of the remaining APIs in the spec which have up to now been marked unstable - i.e. providing a snapshot of the spec which correctly matches the reference implementations (other than implementation bugs) and which can be used in isolation to develop production grade implementations of clients, servers, etc without need to reference any other implementations. There's been a massive sprint to pull this together, and as of the time of writing there are still PRs and commits landing every few minutes.  We'll post a full update on our progress on Monday; meanwhile you can see a sneak peek over at the August 2018 r0 project board.

Spec work has completely precluded any other backend dev this week.

Half-Shot, gone but not really gone

This week we say farewell to Half-Shot, who has been working fulltime on bridges over the summer. He has managed the matrix.org bridges largely single-handedly, with a big focus on the often-volatile IRC bridge(s).

Bridges

matrix-appservice-irc 0.11.0

It's a big deal, and it's rolling out to IRC networks this week. Half-Shot released matrix-appservice-irc 0.11.0, with the following included:

New features & improvements:

• Cache modes internally
• Replace nicks with user pill mentions
• Kick users if we fail to create an IRC client for them on join (aka ILINE kicks)
• SASL support
• Add err_nononreg so we can announce PMs that failed
• Formatting of replies

Bug Fixes:

• Fix invalidchar nick
• Don't answer any msgtypes other than text in an admin room.
• Fix provisoner leaving users on unlink
• Fixed a bug where content of events the bridge hadn't cached were not being used in replies.
• We were calling authedRequest but the request was not mocked out.
• There was a bug involving intents in m-a-b so it was bumped

Metrics:

• Metrics for MatrixHandler - Iline Kicks
• Idle connection metrics
• QueuePool.waitingItems should use it's internal queue size

Misc:

• Section out tests, linting and coverage into separate stages for Travis

WhatsApp

tulir has been working on mautrix-whatsapp bridge.

now bridges a lot more stuff, such as formatting, media and replies. I'm also almost done with desegregating users so that Matrix users join the same group chat portals rather than everyone having their own portal to the same chat

Zulip chat, bridged by Zulip

Matthew discovered there is a Matrix-Zulip bridge on the Zulip side. So if you're running a Zulip server (for some reason), and want to bridge with Matrix check out the integration docs here.

IRC Connection Tracker

Half-Shot created a new component to enhance the reliability of IRC-Matrix bridging:

IRC Connection Tracker is a thing now. It's a project to separate out the IRC connections from the appservice so the two can be run independently, so that restarting the appservice doesn't affect the IRC connections. It's hopefully going to allow bridge stuff to run faster when it's done.

This project is still really early stage. You can take a look at the Proposal document here.

Clients

Nheko 0.5.4

Not technically this week, but Nheko 0.5.4 was recently released

• The settings page now includes the device id & device fingerprint (thanks @valkum )
• The Polish translation has been updated (thanks @m4sk1n )

Get the latest builds of Nheko from bintray.

Fractal

Alexandre Franke and the GNOME crew have been working on Fractal, gearing up for their upcoming 3.30 release.

Fractal 3.29.92 got released and we are freezing strings to give GNOME translators a bit of time to complete translations for 3.30 next week. Latest developments include tweaks for the title bar, misc bug fixes, a new presentation for uploaded files (that are not images, those are still displayed inline) with buttons to download or open them. Development builds are now parallel installable for easier testing and CI has been improved.

Seaglass

neilalexander:

Seaglass now has some early support for inline images and attachments, and supports Quick Look. Also handles emotes and notices better. It also has version numbers now, various other little fixes and Aaron Raimist has been working on auto-update support.

Version numbers! Now when you go to download the tarball from bintray, you can see what's going on!

There's also been some work on supporting dark mode on Mojave (which looks particularly sexy) and even Touch Bar support!

Riot Android v0.8.15

Riot Android v0.8.15 is on it's way to the Play Store.

Changes in Riot Android 0.8.15 (2018-08-30)

MatrixSdk:

• Upgrade to version 0.9.9.

Improvements:

• Improve intent to open document (#2544)
• Avoid useless dialog for permission (#2331)
• Improve wording when exporting keys (#2289)

Other changes:

• Upgrade lib libphonenumber from v8.0.1 to 8.9.12
• Upgrade Google firebase libs

Bugfix:

• Handle \\/ at the beginning of a message to send a message starting with / (#658)
• Escape nicknames starting with a forward slash / in mentions (#2146)
• Improve management of Push feature
• MatrixError mResourceLimitExceededError is now managed in MxDataHandler (vector-im/riot-android#2547 point 2)

Changes in Riot Android 0.8.14 (2018-08-27)

MatrixSdk:

• Upgrade to version 0.9.8.

Features:

• Manage server quota notices (#2440)

Improvements:

• Do not ask permission to write external storage at startup (#2483)
• Update settings icon and transparent logo for notifications and navigation drawer (#2492)
• URL previews are no longer requested from the server when displaying URL previews is disabled (PR #2514)
• Fix some plural and puzzle strings, and remove other unused ones (#2444)
• Manage System Alerts in a dedicated section

Other changes:

• Upgrade olm-sdk.aar from version 2.2.2 to version 2.3.0
• move PieFractionView from the SDK to the client (#2525)

Bugfix:

• Fix media sharing (#2530)
• Fix notification sound issue in settings (#2524)
• Disable app icon badge for "listen for event" notification (#2104)

Riot iOS 0.7.3

Changes in 0.7.3 (2018-08-27)

Improvements:

• Upgrade MatrixKit version (v0.8.3).

Bug fix:

• Fix input toolbar reset in RoomViewController on MXSession state change (#2006 and #2008).
• Fix user interaction disabled in master view of UISplitViewController when selecting a room (#2005).

Changes in 0.7.2 (2018-08-24)

Improvements:

• Upgrade MatrixKit version (v0.8.2).
• Server Quota Notices in Riot (#1937).

Bug fix:

• User defaults: the preset application language (if any) is ignored.
• Recents: Avoid to open a room twice (it crashed on room creation on quick HSes).
• Riot-bot: Do not try to create a room with it if the user homeserver is not federated.

Riot Web

There's been lots of work debugging and optimising Lazy Loading, which is edging closer to being turned on by default.  We've also been working away at improving E2E UX - starting with finishing key backup, and then improved verification, and then finally cross-signing (at last!)

SDKs

Ruby Matrix SDK

ananace is working on the Ruby Matrix SDK "fixing issues and documenting as I go".

Matrix Android SDK

Changes to Matrix Android SDK in 0.9.9 (2018-08-30)

Improvements:

• Clear unreachable Url when clearing media cache (vector-im/riot-android#2479)
• "In reply to" is not clickable on Riot Android yet. Make it a plain text (vector-im/riot-android#2469)

Bugfix:

• Removing room from 'low priority' or 'favorite' does not work (vector-im/riot-android#2526)
• MatrixError mResourceLimitExceededError is now managed in MxDataHandler (vector-im/riot-android#2547)

API Change:

• MxSession constructor is now private. Please use MxSession.Builder() to create a MxSession

Changes to Matrix Android SDK in 0.9.8 (2018-08-27)

Features:

• Manage server_notices tag and server quota notices (vector-im/riot-android#2440)

Bugfix:

• Room aliases including the '@' and '=' characters are now recognized as valid (vector-im/riot-android#2079, vector-im/riot-android#2542)
• Room name and topic can be now set back to empty (vector-im/riot-android#2345)

API Change:

• Remove PieFractionView class from the Matrix SDK. This class is now in Riot sources (#336)
• MXMediasCache.createTmpMediaFile() methods are renamed to createTmpDecryptedMediaFile()
• MXMediasCache.clearTmpCache() method is renamed to clearTmpDecryptedMediaCache()
• Add MXMediasCache.moveToShareFolder() to move a tmp decrypted file to another folder to prevent deletion during sharing. New API MXMediasCache.clearShareDecryptedMediaCache() can be called when the application is resumed. (vector-im/riot-android#2530)

Matrix iOS SDK

Changes in Matrix iOS SDK in 0.11.3 (2018-08-27)

Bug fix:

• MXJSONModel: Manage m.server_notice empty tag sent due to a bug server side (PR #556).

Changes in Matrix iOS SDK in 0.11.2 (2018-08-24)

Improvements:

• MXSession: Add the supportedMatrixVersions method getting versions of the specification supported by the homeserver.
• MXRestClient: Add testUserRegistration to check earlier if a username can be registered.
• MXSession: Add MXSessionStateSyncError state and MXSession.syncError to manage homeserver resource quota on /sync requests (vector-im/riot-ios/issues/1937).
• MXError: Add kMXErrCodeStringResourceLimitExceeded to manage homeserver resource quota (vector-im/riot-ios/issues/1937).
• MXError: Define constant strings for keys and values that can be found in a Matrix JSON dictionary error.
• Tests: MXHTTPClient_Private.h: Add method to set fake delay in HTTP requests.

Bug fix:

• People tab is empty in the share extension (vector-im/riot-ios/issues/1988).
• MXError: MXError lost NSError.userInfo information.

Tools

matrix-to-riot

Half-Shot shared a handy Firefox extension: matrix-to-riot

This is a tiny webextension to forward matrix.to links to an open Riot tab.

Really useful if you often click matrix.to links and find yourself needing to URL-hack to get to where you need to be.

That's all for now

We're going to shift this week's Matrix Live to Monday, alongside the upcoming blog post on the spec release progress. Have a good weekend!

Synapse 0.33.3

Big week for Synapse: v0.33.3 was released this week. You'll find preparation for support for Lazy Loading and Room Versioning, lots of bugfixes, and a great contribution from vojeroen: support for sending SNI over federation for vhosted servers. More complete change log here, features below:

• Add support for the SNI extension to federation TLS connections. Thanks to @vojeroen! (#3439)
• Add /_media/r0/config (#3184)
• speed up /members API and add at and membership params as per MSC1227 (#3568)
• implement summary block in /sync response as per MSC688 (#3574)
• Add lazy-loading support to /messages as per MSC1227 (#3589)
• Add ability to limit number of monthly active users on the server (#3633)
• Support more federation endpoints on workers (#3653)
• Basic support for room versioning (#3654)
• Ability to disable client/server Synapse via conf toggle (#3655)
• Ability to whitelist specific threepids against monthly active user limiting (#3662)
• Add some metrics for the appservice and federation event sending loops (#3664)
• Where server is disabled, block ability for locked out users to read new messages (#3670)
• set admin uri via config, to be used in error messages where the user should contact the administrator (#3687)
• Synapse's presence functionality can now be disabled with the "use_presence" configuration option. (#3694)
• For resource limit blocked users, prevent writing into rooms (#3708)

Ananace has upgraded his K8s-optimized Synapse image to 0.33.3

Spec

• A large chunk of e2e has landed in the spec, largely thanks to Zil0's work. There are still some things left, but the main bits related to sending encrypted messages is in there. Attachments and key sharing are being reviewed today
• TravisR progressed on Push, Application Services, and Client-Server APIs.
• To follow along with progress, the best place to look is the August 2018 r0 tracker.

E2E

Key backups are being worked on, and we've had some new thoughts on key verification.

ru-matrix.org

Well are you?

Alexey and ma1uta have a new Russian-language community wiki available at https://ru-matrix.org. This:

accumulates Russian translates of Matrix FAQ and manuals, and other info about Matrix in Russian. Also we starts a public Matrix Synapse server for Russian users with free open registration

Russian Matrix users: feel free to add and complement articles in our wiki to make Matrix more friendly for Russian users.

Client News

Seaglass

neilalexander has been on a roll the last few weeks working on Seaglass, the macOS-native client:

Seaglass now has support for creating, joining and leaving rooms, and accepting/rejecting room invites, redacting messages, improvements to the room settings pages, timestamps, some visual tweaks, lots of bug fixes, early support for detecting failed message sends, scrolling improvements

Matrique

Black Hat continues work on Matrique, a glossy Qt client:

Typing notification and desktop notification of Matrique are working, plus other improvements

See the latest commits here.

Riot Web

Lazy Loading (LL) is available on develop behind the LABS flag, and is being polished. It will probably not make it to next release, but it's progressing!

Riot Mobile

New releases are imminent for both Android and iOS:

• iOS: bug fixes and Lazy Loading in settings
• Android: bug fixes, better management of permission requests at startup

Shared Secret Auth and Matrix Corporal

Slavi has been working on several operations tools for Synapse. Firstly Shared Secret Auth password provider module for Matrix Synapse.

It allows external systems (which know the shared secret) to easily obtain access tokens for any user hosted on the homeserver. This is incredibly useful for all sorts of automation (no more pre-generated access tokens or plain-text passwords).

With this tool, you can pass a mxid, and get back an access_token for that user. With this, you can do whatever actions would normally be done with a token. For example, you for a new starter at a company, you could join a set of rooms known to be needed by new starters.

Support room for Shared Secret Auth: #matrix-synapse-shared-secret-auth:devture.com

Next, he has released Matrix Corporal, reconciliator and gateway for a managed Matrix server:

It's a way to take control of Matrix (Synapse) in a corporate (or other such) environment.
Based on a configuration policy (generated by an external system, like your intranet), it can auto-create and auto-disable users, keep their profile and authentication details to date, auto-join/leave users to communities and rooms, etc.
It's also meant to sit in front of the Matrix Synapse Client API, inspecting and allowing/denying requests, in accordance with the configuration policy (preventing people from leaving certain rooms/communities, preventing them from messing around with their profile details, etc.)

Support room for Matrix Corporal: #matrix-corporal:devture.com

Note that Slavi also maintains matrix-docker-ansible-deploy, which can deploy these projects along with Synapse itself.

Bots

twimbot

I was extremely young when I first read and understood Larry Wall's explanation about the virtue of laziness, but I did not expect it to lead here. Cadair has created (I think inspired by an idea from TravisR), a "twimbot". This bot works in conjunction with opsdroid and the opsdroid connector-matrix - it works by storing selected messages from #twim:matrix.org in room state. In this way, it either assists the author by making it easier to produce the blog post you're reading, or it coldly replaces the author, leading to his sacking and penury. To view updates currently held by the bot, join #twim_updates:cadair.com.

Hashtag bot

When he's not working away on the Matrix Spec, TravisR continues to work on less [essential|sane] Matrix projects. This week we have @hashtag:t2bot.io, a bot that converts #hashtags to Matrix groups.

Why? Because why not. Source available on github: https://github.com/turt2live/matrix-hashtag-bot

Bridges

matrix-appservice-bridge and matrix-appservice-irc

Half-Shot will only be working full time with the Matrix Core team for another week (!!!), but until then he's going full-speed with bridge work. First matrix-appservice-irc:

Just pushed out matrix-appservice-irc 0.11.0-rc3 which contains a lot of nice things. The full changelog is there, hoping to get some of this tested and running on the matrix.org servers soon, but initial testing looks good :)

Having worked so much on IRC bridging the last few months, Half-Shot has this week managed to make improvements to matrix-appservice-bridge:

Released matrix-appservice-bridge 1.6.0 which contains mostly things I was writing for the irc bridge and factored out instead. There's a couple of goodies like automatic caching of profiles/events, and getEvent as a way to fetch events from the homeserver without fiddling with context.

Full changelog: https://github.com/matrix-org/matrix-appservice-bridge/blob/1.6.0/CHANGES.md

Some Intent operations now cache requests that would otherwise fall through to the homeserver which can be expensive. This is configurable for Intents via the opts.caching.ttl and size options. AgeCounters now allow you to set your own time period buckets. Added a new function Intent.getEvent which will fetch events from the homeserver without any context information, which should be quick. MembershipCache is now exposed to let folks read and write to the cache while also letting the bridge access it.

a release: Fix issue where roomState would fail. b release: Fix issue where we were calling this.intent inside intent like fools! c release: Fix issue where some stole js-sdk code was not checked thoroughly.

WhatsApp

tulir announces that mautrix-whatsapp "is now somewhat usable for basic chatting".

It uses the go-whatsapp library to talk with the WhatsApp Web API. Using the web API means that you'll still need WhatsApp on your phone connected to the internet, but it also means you won't get banned for using the bridge.
The bridge is still very alpha, but basic message bridging and some advanced features already work. There's a features & roadmap document about that.

To come chat about the WhatsApp bridge: #whatsapp:maunium.net.

mxtoot

mxtoot is a Matrix-Mastodon bot from ma1uta, creator of Jeon and the ecosystem around it.

There were two releases last week:

0.4.6 with additional commands (follow, unfollow, mute, unmute, block, unblock users, show followers and followings of the specified user) and a little bugfix.

0.4.7 is out now, which increases the length of messages to 4000 characters.

GSOC is finished!

GSOC 2018 has been valuable for everyone involved!

• Thank you zil0 for all your work on the Matrix Python SDK, E2E and and spec work.
• Thank you APWhiteHat: lots of work on dendrite, including a bunch of stuff on federation, as well as typing notifications!
• and thank you to Cadair for helping to organising it!

The Construct using asynchronous disk IO

Jason writes:

the Construct has advanced to fully asynchronous disk IO in a single-threaded environment.
RocksDB is a highly configurable and customizable database with a very large callback surface to support many different environments, which is good to connect it with something like AIO.
Normally RocksDB gets its performance from concurrent standard POSIX system calls (blocking IO) with multiple threads. The callback surface and its internals are not specifically suited for deviating from the pthread model. However I hacked RocksDB and tricked it into believing that my userspace contexts, which mimic the std::thread/pthread interface, are threads.
This is an example of a RocksDB callback to read from a file, it expects this callback to return immediately (i.e as soon as the kernel returns from a blocking read(2).) Except, where I call fs::read() that is actually a stack-suspension which makes a request to linux AIO where the stack resumes after AIO has called back with the data (see: https://github.com/jevolk/charybdis/blob/master/ircd/aio.cc)
No modifications to the internals of RocksDB were necessary. The Construct is being developed here: https://github.com/jevolk/charybdis.

Community Guides Index

Some weeks ago I asked for suggestions for content to add to a "Community Guides Index". There was not a wave of feedback, but we have enough collected content to be able to share a first pass of this Index. Check out the content there, and please please contact me if you have articles, videos or anything else you'd like to see included. Of course, if you'd like to produce some content and have it included, that would be great too! Ideas for missing articles:

• any content in a language other than English
• DevOps, how to use k8s or ansible to install Synapse
• how to use the Application Services API to make a bot

OggCamp 2018

OggCamp last weekend was great - massive thank you to Jon for organising and running the show - we love your work!

Presentations from Ben and Half-Shot seemed to go down well, we all chatted with Open Source fans and generally flew the flag for Matrix.

New Rooms

• tulir has enough projects that he has created #:maunium.net, for any project of his that doesn't have it's own room
• Support room for Shared Secret Auth: #matrix-synapse-shared-secret-auth:devture.com
• Support room for Matrix Corporal: #matrix-corporal:devture.com
• If you prefer the cold output of a machine, #twim_updates:cadair.com shows you the updates as they're posted in #twim:matrix.org.
• WhatsApp bridge: #whatsapp:maunium.net

So long…

Hope you had a good week, hope you enjoyed reading this post! Amandine is back after some needed vacation time, and has this week's Matrix Live below.