Welcome to the new blog!

Check out the new digs! We're happy with this newly deployed blog, and all the old and loveable content is right down there. If you find issues, let me know. You may remember Nad, from previous editions of Matrix Live - huge thanks to him for his work on the design and upkeep of this new deployment.

Notes from the downtimes

Github tokens

jaywink:

Due to the security incident, all GitHub access tokens for the Scalar GitHub integration were cleared. This means that if you have a GitHub bot in the channel and want to use the !github bot commands, you need to re-login to github via the integration manager menu. Note, existing webhooks are untouched and should work fine without re-authenticating.

Bridges

Half-Shot:

From the matrix.org bridge team, we are ressurecting bridges as fast as possible. Currently running are the freenode, slack, gitter and gimpnet (now hosted on gnome.org) with more to come today and next week.
We have the snoonet and oftc irc bridges back. Mozilla is coming soon hopefully this weekend too!

Whoop! Mozilla is actually up now!

During the intermittent matrix.org downtime last week it was heartening to see people embracing the benefits of decentralisation, for example this article from Alex Gleason explaining to a less-technical audience how they can make their rooms more resilient by adding homeservers.

Pattle new release on F-Droid

Wilko:

A big release of Pattle has just been pushed to the F-droid repo! Changes include:

• Display names are now shown
• You can now click on chats and view them!
• Messages are grouped by time and sender (see screenshots)
• Add fancy transition animation and ripples to chat messages (see video)
• Use Sentry for error reporting (only Android version and device model is sent, along with the stacktrace of the error)

Also, please note that if you have a matrix.org you probably have reinstall the app if you're logged in because of the recent matrix.org incident (because there's no logout button yet and no detection for invalidated access tokens)

There has actually been a release since, which includes:

message sending and viewing image history!

libQMatrixClient 0.5.1.2 && Quaternion 0.0.9.4

kitsune is talking about the long road to 0.0.9.4:

libQMatrixClient 0.5.1.2 has been released, with all the remaining bugfixes for Quaternion 0.0.9.4 that's coming any day soon now. The release notes are here: https://github.com/QMatrixClient/libqmatrixclient/releases/tag/0.5.1.2

Quaternion 0.0.9.4 RC3, the last one before the release that will happen in the nearest days, is out. Release notes can be found at https://github.com/QMatrixClient/Quaternion/releases/tag/0.0.9.4-rc3. Translators, you literally have hours to add your translations for 0.0.9.4!

neo

I reimplemented the matrix sdk into Neo, so it works nicer. Colors and font look nicer (base16-tomorrow, Open Sans), and there's text message sending, with localecho!
I also fixed a bug where react would recycle displayname-components across rooms, attributing them to the wrong messages

a video at https://lain.haus/_matrix/media/v1/download/lain.haus/VfshWRfaNUnpGQbdkyYczxvd

Go test Neo at: https://neo.lain.haus/neo

matrix-registration update

ZerataX:

it's been a long while, but I've finally come around to improving on matrix-registration
For those of you, who have forgotten what this project is about, it basically lets you invite people to your homeserver with tokens, e.g. https://homeserver.tld/register?token=DoubleWizardSky
This whole update was about making the project more user friendly.
I made a new default registration page that requires 0 setup and you can install the project right from pypi with pip, so you don't even need to clone the repo any longer.
check a live example here: https://chat.dmnd.sh/register
and to play around with the api you can can go over to the github page: https://zeratax.github.io/matrix-registration/demo.html?token=ColorWhiskeyExpand
channel: #matrix-registration:dmnd.sh
github: https://github.com/zeratax/matrix-registration

Video available here.

matrix-media-repo now has s3 support

TravisR:

matrix-media-repo now has s3 (and s3-like) support, making it easier to archive older media or use minimal disk space. See the new datastores option in the config and the admin docs ( https://github.com/turt2live/matrix-media-repo/blob/master/docs/admin.md#datastore-management ) for more information.

Dimension

TravisR:

Dimension has been updated to more safely handle when upstream integration managers (like Scalar) are offline. Instead of crashing or breaking in various ways, it'll report which integrations are not accessible.
As well, due to recent events, if you use matrix.org bots or bridges in Dimension then go the the admin section and log everyone out using the red button. Dimension caches upstream tokens and isn't smart enough to realize that they are no longer valid, which means they need clearing. Clients should automatically handle getting new tokens in the background.

Riot iOS

• Interactive device verification is still in progress
• Update WebRTC and Jitsi libs (in progress)

Riot Android

We are finalizing SAS device verification feature We are preparing a release to fix some issues

RiotX (Android)

• We can now clear the cache of the application (Riot legacy feature)
• Sync filter has been implemented
• François is working on thumbnail of video (upload and preview) in the timeline

That's all I know

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Overview

We became aware today of a flaw in sydent’s validation of email addresses which can lead to a failure to correctly limit registration to a given email domain. This only affects people who run their own sydent, and are relying on allowed_local_3pid in their synapse config. We’d like to thank @fs0c131y for bringing it to our attention on Twitter this morning. We are not aware of this being exploited in the wild other than the initial report.

If you are running your own sydent, and limiting signup for your server using the allowed_local_3pids configuration option, then you need to upgrade your sydent immediately to Sydent 1.0.2.

Meanwhile, if you have been relying on the allowed_local_3pids configuration option to restrict access to your homeserver, you may wish to check your homeserver’s user_threepids table for malformed email addresses and your sydent’s database as follows:

$ sqlite3 sydent.db 
sqlite> select count(*) from global_threepid_associations where address like '%@%@%';
0
$ psql matrix
matrix=> select count(*) from user_threepids where address like '%@%@%';
 count 
-------
     0

If the queries return more than 0 results, please let us know at [email protected] - otherwise you are fine.

Details

A flaw existed in sydent whereby it was possible to bypass the requirement specified in synapse’s allowed_local_3pids option, which restricts that users may only register with an email address matching a specific format.

This relied on two things:

1. sydent uses python's email.utils.parseaddr function to parse the input email address before sending validation mail to it, but it turns out that if you hand parseaddr an malformed email address of form [email protected]@c.com, it silently discards the @c.com prefix without error. The result of this is that if one requested a validation token for '[email protected]@important.com', the token would be sent to '[email protected]', but the address '[email protected]@important.com' would be marked as validated. This release fixes this behaviour by asserting that the parsed email address is the same as the input email address.
2. synapse's checking of email addresses relies on regular expressions in the home server configuration file. synapse does not validate email addresses before checking them against these regular expressions, so naive regular expressions will detect the second domain in email addresses such as the above, causing them to pass the check.

You can get sydent 1.0.2 from https://github.com/matrix-org/sydent/releases/tag/v1.0.2.

Matrix Live featuring Slavi, creator of matrix-docker-ansible-deploy

This week, as you may have seen, there was some downtime on matrix.org. With matrix.org being such a large homeserver, there was some disruption to the ecosystem, but Matrix was not "down", since so many people are running their own homeservers. In fact - many rooms were not heavily affected by the downtime.

If you'd like to run your own server, now is a great time to get started because of projects like matrix-docker-ansible-deploy, which allow you to use Docker and Ansible to massively simplify installation and maintenance.

Famedly, start-up with investment making use of Matrix

jcgruenhage shared this exciting news:

Some of you might have heard that there's a Berlin startup called famedly (see https://famedly.com if you can read German), that employed some people from the German matrix community (including MTRNord from SimpleMatrix, krille from Fluffychat and myself), working on getting matrix into the German health sector. We won startup competition yesterday, giving us more funding for continuing our work on matrix and connecting to potential users and investors.

Our main work right now is writing a matrix client focussed on simplicity and the specific needs of the health sector and building the surrounding infrastructure. On the client side we look forward to working with the two other Dart/Flutter clients, to reduce duplicated work on the SDK side.

For questions and general chat, please join #famedly:famedly.de Code will follow later once we have more to show off. Contributions to existing codebases will be licensed under their license, new stuff that we write will be (A)GPLv3.

The two people on the left are larodar and phill.short, the two doctors who originally came up with the concept after being very dissatisfied with the existing comms solutions in that sector.
The one on the right is one of the people organizing the competition from a German bank.

Synapse

• More progress against server key validity periods
• MSC1711 (ensure valid S2S CA certs) support ready for final review
• Started work on reactions
• Started work on optimising small homeserver instances

Construct update

Construct made lots of progress this week toward the 1.0 release. One highlight of interest is the new experimental approach to initial sync called phased initial sync or "crazy loading" for short. Construct sends a very small amount of data to the client at first so it loads very quickly. It then feeds a sequence of since tokens which act to update the client live with small chunks until the initial sync is complete. This allows client developers to build a progress meter for initial sync.
This is just one of the many innovations for performance to enhance your user experience that we love building here. We need your support and contributions at https://github.com/matrix-construct/construct and in #test:zemos.net

Introducing bluepill, a new client for Sailfish OS

If you're using Sailfish, you might be interested in this very early announcement from Cy8aer about bluepill, a new Matrix Client for Sailfish written in Python.

hi benpa it is getting into pre-alpha state and let the community on it. It is far away from release but now I cannot cancel it anymore

YOU CANNOT CANCEL IT ANYMORE

If you're not using Sailfish on a device and want to try bluepill, Cy8aer has you covered:

you can already test it with the sdk emulator https://sailfishos.org/wiki/ApplicationDevelopment#Gettingstarted and the Readme from the repo

Ananace at foss-north 2019

Ananace has been at foss-north 2019, sharing the love and knowledge about Matrix:

I've now held a lightning talk about Matrix at FOSS-North, and am discussing with the Gothenburg FOSS community about setting up a homeserver for them

Major gomuks update

tulir has been storming work on gomuks, his Go CLI client this week:

While waiting for metaolm to be functional, I've been developing gomuks again for the past couple of weeks. So far the changes are mostly internal, but there were a few visible changes too.

• Switched to my own TUI component library. The main visible change from this is that the input bar now supports multiline input. Sadly most terminals don't send ctrl/shift modifiers for enter and alt+enter is being used by the quick room switcher, so the temporary-ish way to input newlines is ctrl+n.
• Changed history storage to store raw event data instead of parsed data.
• Improved HTML parser/renderer. The old parser parsed HTML into a string with style information, whereas the new parser parses it into my custom renderable structs. The new style should make it possible to add scrolling to code blocks and other such things in the future.
• Added syntax highlighting support for code blocks.
• Improved reply rendering. The old reply rendering just put some HTML into the new event before parsing. Now it actually renders the original event separately.
• Added commands for inviting, kicking, banning, unbanning and sending raw events in the current room (thanks to nepugia)

Riot Web

• Finishing touches to breadcrumbs
• More redesign activity
• Planning for Message Editing and Reactions

Riot iOS

• Grouped notifications have landed
• Fix upgraded rooms that appeared twice in room
• Interactive device verification is still in progress

Riot Android

• We have fixed several issue related to the integration of the last Jitsi library.
• SAS PRs are in review

RiotX (Android)

• It’s now possible to create a room
• Send file to a room is now supported
• Some slash commands are now supported: send emote, invite people to a room
• Slash command and people completion is here, but there are still a few issue with it

yuforia has continued work on continuum

continuum completely removed plaintext file-based storage; now working to complete database-based local storage

f0x working on neo

I did some bugfixes in the Neo login process, testing is welcomed. There's not much of a functional client after login yet, but there's full .well-known discovery :)

See a working version of neo here.

libaqueous

Black Hat:

I implemented more strongly-typed event types in libaqueous. Also applied some performance tweaks in the sqlite store. I am still stabilizing and documenting the API for now.
Dart's serialization framework is still a bit inconvenient to use, but the language's flexibility is great. Also fast prototyping!

Still more

Alexandre Franke gave a talk last Sunday at Journées Du Logiciel Libre in Lyon. He gave a tour of Matrix and encouraged attendees to switch to it. In future we'll link to a video of the talk and his blog of the event!

The talk was recorded and I would have given a link, but they haven’t been published yet.

GSOC student applications ended on Tuesday - Matrix received some great entries and we're looking forward to welcoming them to the project this summer!

That's all I know

If you have something to say here, something to add, come chat to us in #twim:matrix.org - I love that this is a supportive and engaged project ecosystem, so come share what you have!

Here's what you need to know.

TL;DR: An attacker gained access to the servers hosting Matrix.org. The intruder had access to the production databases, potentially giving them access to unencrypted message data, password hashes and access tokens. As a precaution, if you're a matrix.org user you should change your password now.

The matrix.org homeserver has been rebuilt and is running securely; bridges and other ancillary services (e.g. this blog) will follow as soon as possible. Modular.im homeservers have not been affected by this outage.

The security breach is not a Matrix issue.

The hacker exploited a vulnerability in our production infrastructure (specifically a slightly outdated version of Jenkins). Homeservers other than matrix.org are unaffected.

How does this affect me?

We have invalidated all of the active access tokens for users on Matrix.org - all users have been logged out.

Users with Matrix.org accounts should:

• Change your password now - no plaintext Matrix passwords were leaked, but weak passwords could still be cracked from the hashed passwords
• Change your NickServ password (if you're using IRC bridging) - there's no evidence bridge credentials were compromised, but if you have given the IRC bridges credentials to your NickServ account we would recommend changing this password

And as a reminder, it's good practice to:

• Review your device list regularly - make sure you recognise all of the devices connected to your account
• Always make sure you enable E2E encryption for private conversations

What user data has been accessed?

Forensics are ongoing; so far we've found no evidence of large quantities of data being downloaded. The attacker did have access to the production database, so unencrypted content (including private messages, password hashes and access tokens) may be compromised.

What has not been affected?

• Source code and packages have not been impacted based on our initial investigations. However, we will be replacing signing keys as a precaution.
• Modular.im servers are not affected, based on our initial analysis
• Identity server data does not appear to have been compromised

The target appeared to be internal credentials for onward exploits, not end user information from the matrix.org homeserver.

You might have lost access to your encrypted messages.

As we had to log out all users from matrix.org, if you do not have backups of your encryption keys you will not be able to read your encrypted conversation history. However, if you use server-side encryption key backup (the default in Riot these days) or take manual key backups, you’ll be okay.

This was a difficult choice to make. We weighed the risk of some users losing access to encrypted messages against that of all users' accounts being vulnerable to hijack via the compromised access tokens. We hope you can see why we made the decision to prioritise account integrity over access to encrypted messages, but we're sorry for the inconvenience this may have caused.

What happened?

We were using Jenkins for continuous integration (automatically testing our software). The version of Jenkins we were using had a vulnerability (CVE-2019-1003000, CVE-2019-1003001, CVE-2019-1003002) which allowed an attacker to hijack credentials (forwarded ssh keys), giving access to our production infrastructure. Thanks to @jaikeysarraf for drawing this to our attention.

Timeline

March 13th Updated 2019-04-12 11:00 UTC

• Attacker compromises Jenkins CI server

April 4th Updated 2019-04-12 11:00 UTC

• Attacker gains access to production infrastructure by hijacking a forwarded SSH agent logging into the compromised Jenkins worker

April 9th

• Jenkins vulnerability brought to our attention by @jaikeysarraf

April 10th

• Investigation identified the compromised machines and the full scope of the attack
• Jenkins was removed
• Attacker's access to compromised machines was removed

April 11th

• Matrix.org was taken offline and production infrastructure fully rebuilt
• Having fully flushed out the attacker, external communication was published informing users and advising on next steps
• Matrix.org homeserver restored, with bridges and ancillary services (e.g. this blog) following as soon as possible

Update 2019-04-12

At around 5am UTC on Apr 12, the attacker used a cloudflare API key to repoint DNS for matrix.org to a defacement website (https://github.com/matrixnotorg/matrixnotorg.github.io). The API key was known compromised in the original attack, and during the rebuild the key was theoretically replaced. However, unfortunately only personal keys were rotated, enabling the defacement. We are currently doublechecking that all compromised secrets have been rotated.

The rebuilt infrastructure itself is secure, however, and the DNS issue has been solved without further abuse. If you have already changed your password, you do not need to do so again.

The defacement confirms that encrypted password hashes were exfiltrated from the production database, so it is even more important for everyone to change their password. We will shortly be messaging and emailing all users to announce the breach and advise them to change their passwords. We will also look at ways of non-destructively forcing a password reset at next login.

The attacker has also posted github issues detailing some of their actions and suggested remediations at https://github.com/matrix-org/matrix.org/issues/created_by/matrixnotorg.

This confirms that GPG keys used for signing packages were compromised. These keys are used for signing the synapse debian repository (AD0592FE47F0DF61), and releases of Riot/Web (E019645248E8F4A1). Both keys have now been revoked. The window of compromise for the keys started from April 4th; there have been no Synapse releases since then. There has been one release of Riot/Web (1.0.7), however as the key was passphrased and based on our initial analysis of the release, we believe it to be secure.

What are we doing to prevent this in future?

Once things are back up and running we will retrospect on this incident in detail to identify the changes we need to make. We will provide a proper postmortem, including follow-up steps; meanwhile we are obviously going to take measures to improve the security of our production infrastructure, including patching services more aggressively and more regular vulnerability scans.

Folks, this is an update to explain that we will be shortly deprecating Synapse support for Postgres 9.4 and Python 2.x.

What are we doing?

When is this happening?

For Python, we shared that we would discontinue to Python 2.x support from April 1st 2019, so for the first release that follows 1.0 we do not guarantee Python 2.x support.

For Postgres, will give server admins 6 weeks to upgrade to a newer version, and will guarantee support up until 20th May 2019.

Why would you do this to us?

• We want to make use of new language features not supported in old versions. This will enable us to continue to improve the performance and maintainability of Synapse.
• Python 2.x overall will be end of life'd at the end of the year. Postgres 9.4's final release will follow 2 months later on 13th February 2020.
• Since very few server admins still use these technologies on the wild, providing support is costly and we want to reduce our overall maintenance load.

La la la I am ignoring you - what will happen?

Okay, but I have questions, where should I go?

Matrix Live S3E20 FluffyChat creator Krille on Matrix, Ubuntu Touch and Matrix-for-Healthcare

Synapse, the road to 1.0

Last week we carried on with Synapse 1.0 work, in particular server key signing validity, a config option to verify federation certificates to support MSC1711 and support for 3PID unbinding APIs (MSC1915). Outside of that we've been thinking about how to improve room upgrades for private rooms and landed the preparatory work for a refactor of the room directory.

Coming up next, we'll continue to bash out 1.0 blockers, Hawkowl has started work on optimising for small hosted homeservers, and anoa will be working on the new super fast room directory. Finally Erik has started work on aggregations support so clients will be able to offer things like edits and reactions ?

https://www.arewereadyyet.com/ is now just shy of 64%, keep going.

Dendrite

Dockerfile for sytest running against Dendrite approved. Close to being able to run Dendrite against sytest. Also using sytest's new test white/blacklist functionality to include a list of tests that Dendrite is known to pass in the repo. When people make new PRs that allow Dendrite to pass new tests, they can also append the names of the tests to the testfile to help automatically track Dendrite's improving progress. Look forward to seeing further progress post Synapse 1.0.

libQMatrixClient 0.5.1 released,

libQMatrixClient 0.5.1 is out! This is a minor release with bugfixes and small improvements, a base for Quaternion 0.0.9.4, RC of which is coming this weekend. Full release notes here

Quaternion 0.0.9.4 Release Candidate is available for all those who don't need a "release" seal of approval to use applications. Get out, grab it, report bugs that could still sneak in: https://github.com/QMatrixClient/Quaternion/releases/tag/0.0.9.4-rc. A separate call out to translators - it's a great moment to teach Quaternion a few more phrases in your language!

.NET Core Synchrotron

This weekend I took a brief pause from doing matrix work, and then went back into it by finishing the feature set for the .NET Core Synchrotron. It's "feature complete" meaning all specced features of sync are in this branch, excluding legacy endpoints. There are a few bugs logged against it and it's really only useful as a super early test, while I fix some of the p1 bugs.

The federation sender continues to be reliable and stable, and we are prepping for the release of synapse 0.99.3 which made a few changes to the replication stream. For more info, please see https://github.com/turt2live/synapse-netcore-workers

Pattle has now moved to Flutter

After some silence here but a lot of development, Pattle has now moved to Flutter. The new app is available in the same F-droid repository, you can just update. Note that you will have to log in again. If there are issues with updating, try removing the old app and installing the new one.

Also note that this version is still very basic and does not have all features of the old Pattle unfortunately. The reason that I chose to already replace it on the repo is that the old app will not receive updates anymore.

The current features are logging in and an overview of all chats. What is new is that data is now stored locally, so you won't have to do an initial sync everytime you open the app (which was the case with old app). What's also new is that I've decided to use more red, because.. I like red.

There is still a lot to do! Please notifiy me of anything that you think is missing, even though some things may be obvious (sorting chats based on date, chat avatars), some may not be as obvious to me!

If you'd like to contribute, the new repo is here. Note that many contributions will probably start at the SDK level. Pattle uses my own Dart SDK, which you can find here.

There is at the moment a very specific bug in Flutter, where on Android 7.1 with release builds, the app crashes when building a list widget. If you use Android 7.1 (like me) and you crash after logging in, that's the reason. It seems I can't do anything about it, sorry.

libaqueous (Matrix Dart SDK) and Aqueous client

Aqueous now comes with room categories and history messages support. Also, a lot of code refactoring is going on to change the backend store from plain json file to sqlite, which should improve the performance a lot. The room is at #libaqueous:encom.eu.org

Riot Web

• Working towards bringing breadcrumbs out of labs
• Adding image preview to file upload UX
• Stability fixes for timeline
• Defending against issues introduced when using Riot Web on a machine which is running out of disk space

Riot iOS

• Grouped notifications are coming! We have added some logs to track why the notification setting is sometimes disables (#2348).
• SAS device verification is still in progress but it should be released next week.

Riot Android

• Three releases this week, to fix several bugs with the new EventStreamServiceX, and to fix issue with the Jitsi conference call (Riot.im 0.8.29). FDroid users should receive the upgrade soon.
• Users are happy with the new notifications!
• Valere is still working on SAS, we are reaching the end :).

RiotX (Android)

• It's now possible to signout from RiotX.
• The whole setting screen has been imported from Riot, as well as the Themes and some other small features (launcher icon, etc.).
• Benoit is working on room creation, sdk side, and François is working on image upload.
• We should start to integrate new design from Nad next week.

continuum (JavaFX desktop client)

in continuum, moving storage of messages from plain text files to a database, simplifying code and improving performance

neo

Neo now has .well-known implemented for login :D
video showcasing the different outcomes:

Synapse K8s images

I'll be writing a Helm chart for 1.0 as well, just haven't had time to do so yet.
I need to update all the references to TLS as well, as that's supposed to be left to kube-lego or cert-manager for a modern install

Federation Tester

Following up on last week's effort, a number of issues on the Matrix federation tester, both on the backend and the frontend side, have been fixed, including fixing certificate validity checks, not following CNAME records in SRV records, fixing display of the SRV lookup in the UI, preventing some crashes, and a few more.

All outstanding issues on both the federation tester's backend and its frontend that could have preventing people to test efficiently how their server is performing in the context of Synapse's 1.0 release have now been fixed and deployed on https://matrix.org/federationtester/.

Synapse and Riot bounties

A few bounties have been added to Synapse and Riot recently to encourage community memebers to work on those projects. Although they aren't large enough to actually pay for development, I believe they can serve as a small push to help an already interested developer justify spending some time working on these projects.

• $400 bounty on threading in Riot thanks to GitHub user @fallerOfFalls, expires November 21, 2019
• $200 bounty to fixing stuck invites, expires on January 1, 2020
• $200 bounty to fix Riot forgetting the name of rooms, AKA the "Aaron and 5246 others" bug, expires on January 1, 2020

Still more

jj announced a new Python SDK for Matrix:

new, alpha quality, matrix python client for asyncio and with support for e2e encryption: https://github.com/SFTtech/aiomatrix
Polishing is needed but the basics do work already :)

Matrix listed as one of the "Top Ten Tools" for Open Science practitioners

the whole point of Matrix is that its a protocol which many different suppliers can use so you can avoid nasty lock ins

That's all I know