This week you’re stuck with me, but I’m chatting to Ryan, who works on Riot web. Having previously worked at Mozilla, Ryan has a LOT of interesting things to say about Firefox, the browser market, the importance of decentralisation, Matrix being GREAT, and more.
It’s all about 1.0 for the Synapse gang this week. This means performance improvements across the board in the form of read receipt batching, user directory (room directory coming soon!) and rate limiting on log in and registration APIs.
Brendan shipped the low bandwidth CoAP proxy we demo’d at FOSDEM
As well as a bunch of spec implementation projects to ensure that Synapse (and Sydent) are ready for Synapse 1.0.
Synapse workers projects
Turn-of-phrase of the week from Half-Shot (“make it more performant and less crashy”):
synapse-netcore-workers is progressing as strongly as ever. This week has mainly been supporting a couple of users trying to use the fed sender, and also trying to make it more performant and less crashy. I’ve been using it solidly for two weeks now, and by and large it’s been working nicely :)
Support for replication protocol in Cortex is mostly complete. A federation sender worker implementation is being worked on. If anyone is interested or wants to contribute, please go to #cortex:encom.eu.org
Lots of progress this week. I’d like to give a special thanks to Yan Minari for another great week of testing, bugfixes and feature requests.
We implemented features related to room directory lists, reporting content, ignoring users, VoIP turnServer, prev_content in state events.
Improving SSL: allowing configurable lists of ciphers, and sending/receiving SNI.
At the lower level: adding support for Linux AIO features that are present in newer kernels, giving a nice performance boost.
Hey all, I’ve been working on my Matrix Android app, Pattle! The goal of Pattle is to be an easy to use app for Matrix, with it’s design inspired by popular apps such as WhatsApp and Telegram. Development happens here, and contributions are encouraged! The app is not currently suited for daily use, but some functionality is there, such as registering, logging in and viewing chats.
Currently it’s an Android only app using the official Matrix Android SDK, but the plan is to support iOS and web too, in the future.
There’s also a sort of design document available, stating how Pattle differs from standard Matrix apps and what it’s goals are. The intent of the design document is to make development easier later on for other platforms
Quaternion 0.0.9.4 beta 2 (too many numbers? That too shall pass) is out, with bugfixes and translation updates. Notably, Quaternion won’t crash on upgraded rooms in some cases, and won’t cry in #gsoc:matrix.org and other v3 rooms. Translators are still strongly encouraged to push forward – due to all the features and fixes, there are many untranslated strings across the board! Also, some bugfixes are still in order before we can call the release RC, and some of them are really easy – so if you’d like to contribute, it’s a great time to start!
Preparing for 1.0.4 release with lots of small polish fixes
Planning our roadmap for the next few quarters
iOS released 2 times. Last release was to fix an issue with invalid scalar token.
Review of one PR from the community for iOS10 notifications.
Started implementing device verification with emoji.
We’ve released v0.8.25 on Thursday, containing refresh of invalid scalar token, and some bugfixes. Links on m.notice messages are now clickable again.
Started implementing device verification with emoji.
We started to setup build tools and CI configuration.
I just realized that I haven’t had lazy loading activated by default in the Ruby SDK, despite having had lazy loading code in place since ages back, so now that’s going to be the default value going forward.
matrix-appservice-bridge got a 1.8.0 release last night, featuring automatic handling of room upgrades for all your room upgrade needs. Providing your bridge uses the RoomStore as designed, it’s literally a few lines to enable :). Changelog here
What is this? A matrix-appservice-irc release? No, it’s a release candidate. Announcing that 0.12.0-rc1 is now out and about for folks to play with. More IRC updates to come in the future :)
I wrote a small bot that takes a kick/ban policy from room state from all rooms it’s a member of and tries to enact that policy. In practice that means it applies a regex to all MXIDs and tries to kick/ban them based on that. It’s been a request of TravisR , source code is available at https://gitlab.com/jcgruenhage/banhammer, documentation is still lacking but will hopefully soon be added
So that’s all I have to say! I hope you enjoyed this edition of This Week in Matrix, and whether you did or you didn’t, I’d love too hear from you in #twim:matrix.org. If you have Matrix news to share, that’s the place to come and do so!
Folks, in the run up to Synapse 1.0, if you are running your own homeserver now would be an excellent time to check that your TLS certificates are up to date. Point your server name at https://matrix.org/federationtester/ and if there are errors check our handy FAQ on how to fix it. If you do not have valid TLS certificates Synapse 1.0 will refuse to federate with you.
benpa has put together a federation checker to quantify how many homeservers are 1.0 ready – https://www.arewereadyyet.com/ – It currently stands at 50.5% let’s try and get that to 60% over the weekend.
Aside from all that, the team have been working on preparing for Synapse 1.0, you can track our progress here. We promise not to just land 1.0 out of the blue – we’ll give everyone a 2 week warning to give stragglers a chance to get their certificates in order.
And this week we have Neil and Erik talking about this in more detail on Matrix Live
JeonServer and related project updates
ma1uta has been working on Jeon – Java interfaces to the various Matrix APIs – and is now getting ready to start work on a homeserver. He was previously asking for a name for this project, but might now have settled on “JeonServer“.
First Release Candidate of the Jeon Project with upcoming Client-Server API 0.5, Server-Server API 0.1.1, Application API 0.1, Identity API 0.2 and Push API 0.1.
Also the RC of the jmsdk has been prepared with Java Matrix Client for Client-Server API 0.5.
Changes in the C2S: Added the m.push_rules event, removed presence list methods and other minor fixes.
Added S2S API of the 0.1.1 version.
I prepared a very simple page https://ma1uta.github.io/ with links to the swagger schemas (json and yaml) for all Matrix API which generated from the Jeon code.
There’s a lot of progress, a few endpoints and features have been implemented this week such as Room Tags and all of the spec features for /createRoom. Most of the progress has been with testing and bugfixes thanks to Yan Minari, and tulir and mujx.
We’ve fixed several interactions with synapse such as invite accept/deny and synapse’s ability to join and leave construct created rooms without any issues.
Lastly and most important, we’ve generated an official issues list thanks again to our star tester Yan Minari available here https://github.com/matrix-construct/construct/issues
pztrn has created a new mechanism for relaying apps that use Slack webhooks into Matrix:
To everyone who wonders how to connect his application to Matrix (at least for notifications of some kind) – use OpenSAPS! It just reached v0.1.0. OpenSAPS stands for Open Slack API Server and able to retransmit messages from applications (like Gitlab or everything that can send data to Slack) to somewhere else. Right now these “somewhere else” is a Telegram (with HTTP proxy support) and Matrix! Written in Golang to ensure minimal memory footprint. Take a look at https://gitlab.com/pztrn/opensaps
Tested with Gitlab and Gitea but should work with almost any service.
Join #opensaps:pztrn.name to talk with developers or get help. BTW, there is OpenSAPS instance in our room that transmits everything from gitlab.com into room!
(almost) immediately after 0.1.0 comes 0.1.1, with fixed URLs parsing and fixed inability to login into servers which use .well-known for delegation.
It should work [with other webhooks]. If something strange happens there is also a possibility to write own parser to make everything work :)‚ Tested with Gitlab and Gitea ATM. Share application names that work, I’ll start to make a list of them. :D
matrix-puppet-bridge: matrix-puppet-slack and matrix-puppet-hangouts updates
Minivector, a minimalistic fork of riot-android had a new release last week, getting rid of a few more unused dependencies. This brings the final apk size down to 13mb vs riot android’s 25mb. This work was done by @hrjet:matrix.org. The project room is here: #miniVectorAndroid:matrix.org
matrix-docker-ansible-deploy, now with Discord and email templates
The volume of discussion about installing/configuring Synapse and other Matrix-related components is like a subculture in itself. Standing tall within this is Slavi‘s matrix-docker-ansible-deploy collection of Ansible playbooks. They’re a great way to quickly and reliably get a Synapse instance running.
between chasing bugs in Quaternion 0.0.9.4 beta (translators, your help is hugely needed to catch up with new and updated strings) there happens almost literal bikeshedding in #qmatrixclient:matrix.org, under an excuse of discussing The Universal Algorithm to Colour Usernames. Join the fun!
Bifrost is now starting to comfortably support gatewaying. For those that don’t know, gateways allow a remote user to participate in the matrix network without prior bridging, it’s very fancy. The latest changes are that XMPP clients can now ask for the public room list by querying the bridge component. There is a video on this using the Yaxim XMPP client on Android (Credit to Ge0rG). Come chat with us in #bifrost:half-shot.uk
You can check out the video of the bridge in operation here:
I made a room for !pinging echo maubots: #ping:maunium.net. The room has a bunch of echo bots, currently on maunium.net, c.mau.dev and matrix.vgorcum.com. It also has a new maubot plugin called pingstat, which collects the pong data and makes a leaderboard website. The website is linked to the room as a widget.
Not much news to report on the netcore-workers other than it’s maturing and we have nice things now like logging, metrics and a docker image you can run at home. I’m running the federation sender fulltime on half-shot.uk to dogfood it and will announce when I think it’s good for general consumption :)
Related: Black Hat is investigating a similar project in Rust. Anyone interested in that please do go chat, and take a look at the repo they’ve created.
matrix-wug, IPA rendering bot, gets support for cherokee
This is “script” or written language has a very interesting story behind it, where the creator of it actually couldn’t read and write. Regardless, he wanted to write down his language, and developed his own writing system.
The characters look a lot like latin characters because he tried to imitate the characters of a bible. Just a fun history lesson!
Modular.im is a Matrix-as-a-SaaS offering. It’s suitable for users who want the benefits of having their own Matrix homeserver, but don’t want to host and run one.
There are some new announcements this week:
Today we are pleased to announce that you can now customise your Modular hosted Riot at the touch of a button, through the Modular host admin interface. Better yet, this is available to all Modular customers at no additional cost.
Today we are also launching a proof of concept for purchasing additional content via the Modular integration manager.
For the initial experiment we’re offering a set of “snazzy”, limited edition Matrix stickers for the princely sum of $0.50. These are digital versions of the Matrix and Riot hexagons that some of you may have seen in real life.
decred.org are an organisation concerned with blockchain technology, and they also use Matrix for their communications! They now have a stickerpack available in Riot, so if you’d like to use their stickers you can add the pack and get going with blockchain-related memes.
If you have a stickerpack you’d like to see included, please let me know.
I’ve made a room for anyone interested in drums and percussion. Acoustic, Electronic, Played or Programmed, anything goes as long as it’s rhytm related. The room is focused on both playing/programming drums and equipment.
Either way, you will soon have a functioning Matrix Synapse homeserver and mautrix-whatsapp installed with it. Next, we will set up an Android VM.
Set up an Android VM
The best way to run an Android Virtual Machine is to use the Android Studio tools from Google. First, install Android Studio, making sure to follow the post-install steps, as they will install additional tools we need, including AVD Manager.
Once installed, run AVD manager by choosing Tools -> AVD Manager from the menu.
Follow the steps to create a new virtual machine, in this example I have a Nexus 5X running Android 9, but almost any configuration is fine here. Make sure that you give the device access to the Play Store.
Install WhatsApp and sign-in
Launch the Virtual Device, the open the Play Store and sign in. Now use the Play Store to install WhatsApp on the Virtual Device.
You will be asked to verify your phone number, use your number on another device to complete this step.
Setup mautrix-whatsapp bridge
Now that you have WhatsApp working in a VM, and Matrix working on your server, it’s time to bridge them togther!
Per the instructions at mautrix-whatsapp/wiki, you must start a new chat with @whatsappbot:<yourdomain>. Type login to begin the authentication process.
mautrix-whatsapp operates by using the WhatsApp Web feature of WhatsApp – which means it uses a QR code that you must now scan on the device running WhatsApp – which in your case is the AVD. In order to scan the presented QR code, set your AVD camera to passthrough the camera device on your host machine – see the images below.
Once this is complete, you can type sync, to start bridging contacts, and sync --create to automatically create room invites.
And that’s it! You may need to take a little time to watch the sync happen, particularly if you have a very large number of chats on the WhatsApp side, but there is no further configuration needed.
The remainder of 0.6.x is expected to be bug fixes. This includes bug fixes in mtxclient as well as nheko.
0.7.0 will be the next feature release. The end goal here is to add some of the capabilities that people have been requesting, and other things that people haven’t been requesting.
AFTER 0.7.0 releases, I will then focus on updating the encryption to include more than just text messages.
libQMatrixClient project, Quaternion
kitsune has been writing the codes this week, Quaternion now supports room upgrades and more:
Master branch of Quaternion can now store access token in your secure storage/keychain, better integrating into frameworks of GNOME and KDE, as well as macOS and Windows.
Also, you can see the room version and upgrade rooms in Quaternion (just like Matthew said on FOSDEM ;) ) – this feature is enabled by libQMatrixClient, so other clients are welcome to support room versions/upgrades too!
All this is within the work on the upcoming release of libQMatrixClient 0.5 and Quaternion 0.0.9.4 – stay tuned!
Oh we will. Spectral support for these new features when?
With all the interest generated through Riot 1.0 we’ve seen a surge of new and returning users on matrix.org. So we’ve spent much of the week investing in stability and performance. We’ll ship a new point release (0.99.2) next week.
Once again a big flashing advert that Synapse admins should no longer be using self-signed certificates for their federation API. Synapse 1.0 is scheduled in early March and will not accept self signed certs – so servers that do not update will lose the ability to federate with 1.0 servers. Updating your certs is straight forward, see our guide and FAQ for more info.
Just tagged and pushed a new version (0.0.4) of the Ruby SDK, since I’ve completely forgotten to do that for a while now.
A whole bunch of small stuff I did, mostly just additional exposed methods and parameters, and some fixed issues as well. I did add support for HS URL discovery using both SRV and .well-known though, which I think is probably the largest new feature.
mxisdv1.3.1 is out. It is a maintenance release that fixes a set of regressions following the changes in v1.3.0 to massively improve performance. If you haven’t updated to the v1.3 branch, now is the time!
Construct progresses toward Matrix 1.0 Spec
Over the past three weeks Construct has made rapid progress with covering the Matrix 1.0 specification!
Last week we implemented room version 1 and room version 2 authentication rules.
This week we implemented device support and management, and have nearly wrapped up successful End-To-End Encryption testing.
The Construct is a community-driven server implementation written in C++ for maximum performance. To all experienced C++ developers out there: we need your contributions to accelerate and test the 1.0 release!
Make the c++ code less heavy, we should focus on speed
Offload js code to direct qml binding, with this i mean instead of having ablock if-else code that modifies qml object that runs on component load, we do this directly in qml: see 0a800fc and https://github.com/uMatriks/uMatriksblob/0a800fcdc0af4fa2e08526dbff88e06bcb591779/uMatriks/componentsChatItem.qml as an example for a cleaner code
Make all ui depended calls async, we should NEVER block ui
Offload heavy js logic to c++
Cleaner UI, with less crust
Merge our WIP call support
Finish impl of call support (maybe move to a standalone webrtc module)
Slowly start moving to QQC2
Implement sending of media
Finalising version and delivery of Riot 0.8.23 on the Play Store.
RiotX: testing library to render events in the timeline.
matrix-archive-keyring (from last week’s TWIM) is ready to be uploaded to Debian. Attempting to get it uploaded to unstable right now as I speak.
We linked to the annoucement last week, and have been using it for even longer, but somehow I missed a great labs feature that shipped with Riot 1.0: “room breadcrumbs”, which you can enable from the Labs page of Riot settings. This gives you a series of quick links with the history of rooms you were most recently in. So if you’re like me, clicking between rooms all day, you can get where you’re going a little faster.
There is no big news about mautrix-whatsapp this week, other than that I installed it on my own server and found that it works really really well. I’d love to switch to this as a main interface for my less-decentralised friends, but like others, I’ve been stung by this issue in an upstream project. If there are go-fans reading (and I know there are), I wonder if it’s an issue which can be fixed?
We also have Synapse v0.99.1.1 available, as the race to Synapse v1.0 gets closer and closer and closer!
Brendan also said, regarding low-bandwidth:
Some of the week has been spent finalising the components used in the low-bandwidth demo we did at FOSDEM. This required polishing some internal developments, which are now almost ready to be published to the wide world! This should happen in the next few days so stay tuned :)
Riot 1.0.0, and hotfix 1.0.1
If you’re following Matrix much at all, you will surely have seen it by now: Riot, the most popular client, has announced the v1.0 milestone. I won’t reiterate the features and improvements here, but do check out the blog post, or just head over to https://riot.im/app and see for yourself!
Particularly exciting is the new encryption verification process, involving sharing a list of emojis out-of-band:
thank you @matrixdotorg@RiotChat – with the new update the office is full of people yelling names of emoji at each other
Since we haven’t featured their work here at all I wanted to give some space to poljar. They’ve been working on a Weechat-Matrix protocol script, but as part of this project they created a new Matrix client library, matrix-nio, “designed according to sans I/O principles”.
As if this wasn’t enough! python-olm is a library providing Python bindings for Olm (enabling E2E encryption for Matrix.)
FluffyChat v11 preview
FluffyChat is a pure QML Matrix client, designed to be used on smartphones running Ubuntu Touch. Krille, the author is working on:
boring refactoring stuff :-( at the moment to prepare everything for e2e encryption work
. . .which seems to me not boring at all – we can expect, in the next few months, another Matrix Client with E2EE support, and on mobile!
In my role as editor of this particular blog post, I enjoy certain luxuries that others might envy. For example, today, I tried a new preview build of FluffyChat. This I installed on my barely-repaired OnePlus One running Ubuntu Touch 16.04.
Over the past week, Fractal has adapted its main view and room settings views for mobile phones, and implemented long-press for right-clicks on messages for touch devices. It’s also received some changes in the backend to use the serde_derive Rust crate instead of parsing JSON by hand.
To celebrate tags being available again in riot-web, TravisR has created matrix-tag-manager:
If you’re a power user which uses room tagging to organize your rooms, I’ve made a web UI for you: https://tags.t2bot.io/
mxisdv1.3.0 is out! This release brings the massive performance improvements, bug fixes and enchancements. This release also marks the two years anniversary of the project. Thank you to all our users for your undying support!
QMatrixClient project has been busy merging PRs from contributors (close to tray and middle-click in room list and user list, to mention a couple) and streamlining the translation process before the upcoming releases of libQMatrixClient 0.5 and Quaternion 0.0.9.4. At the same time the feature branch to support room versions and room upgrades is at an advanced stage; expect the features to land in master next week.
Linda has been working on new Debian package called matrix-archive-keyring:
I’ve paused porting Synapse to OpenBSD this week, while I attempt to both fix and revert some security issues caused by sudo apt-key add for Matrix.org and Riot.im packages for Debian-based GNU/Linux distribution users.
matrix-org/synapse PR #4609 was my first attempt to fix the problem, however I fell short this time in ease-of-use and my implementation had other issues. (Read: The repo-key.asc downloaded from Matrix.org Debian repository should go to /usr/share/keyrings, and must not be added by apt-key add or added to /etc/apt/trusted.gpg.d directly. I attempted the latter with my first PR; but that would’ve allowed the Matrix.org signing key to also be used to verify packages from Debian’s main repositories. Yikes!)
As my second attempt, I’m now creating a new Debian package called matrix-archive-keyring. There’s multiple things this package intends to do:
A post-install script to remove any previously added Matrix.org package signing keys from /etc/apt/trusted.gpg. (This is the file apt-key add most commonly uses, and the most commonly found bad advice on the Internet.)
Convert repo-key.asc to a de-armorized .gpg format, understood better by apt(8). (Essentially, gpg --dearmor.)
Install the converted .gpg key to /usr/share/keyrings.
Make sure apt(8) uses the newly added key for Matrix.org repository only?
Deliver any updates to repo-key.asc directly to you via apt upgrade. Your operating system might then also do it automatically for you if unattended-upgrades(8) is installed on your system. 😊
Make sure everything goes away when this keyring package is uninstalled. No more sudo sh -c 'apt-key del C35EB17E1EAE708E6603A9B3AD0592FE47F0DF61; rm /etc/apt/sources.list.d/matrix-org.list to get rid of the repository.
I have one more interesting enhancement to disclose.
We can ask you with debconf if you want to install deb https://matrix.org/packages/debian $(lsb_release -cs) main to /etc/apt/sources.list.d/matrix-org.list while you’re installing the new keyring with apt install matrix-archive-keyring.
You’ll most likely notice the difference in install instructions, as a result of this package. Before and after:
sudo apt install matrix-archive-keyring
sudo sh -c 'apt update && apt install matrix-synapse-py3
If you are a Debian or Ubuntu user and Synapse homeserver operator using Matrix.org repositories, I’ll be towncrying when this package becomes available as much as I can. All you should need to do then is apt install matrix-archive-keyring and it should patch the security holes in your operating system if you’ve formerly installed packages from Matrix.org or Riot.im! (The command may change a bit, because next Debian operating system release “Buster” has entered soft-freeze just today (2019-02-12) and not accepting new packages into that release.)
For the eager early testers not afraid of building from source, matrix-archive-keyring version 2015.12.09+debian0.10 is available for early testing (attached).
# Something like this!
sudo apt install devscripts
tar -xfvJ matrix-archive-keyring_2015.12.09+debian0.10.tar.xz
debuild -us -uc
sudo dpkg -i dpkg -i ../matrix-archive-keyring_2015.12.09+debian0.10_all
Please follow Debian Bug#922155 for updates. Debian binary packages will be available later sometime this/next week from Debian’s unstable distribution (contrib), if all goes well and there are no remaining Debian policy violations or serious bugs.
Gridepo, as a Matrix Homeserver, is gaining basic support for Matrix Client API. We have basic login, room creationg, message sending and sync working! A little screen recording of the progress can be viewed here for live action with Riot. As a reminder, Gridepo is a clean rewrite of mxhsd.
This is not usable yet outside of a dev environment but we should have a first usable version with federation in a couple of weeks!