Matrix 1.0 – Are We Ready Yet?

TL;DR

  • If you run a Synapse ensure that your federation certificates are valid here.
  • If they are not valid check out the FAQ.
  • Follow along with progress at https://arewereadyyet.com
  • Tell all your admin friends.

Folks, as you know we are now very close to achieving Matrix 1.0 and finally being in a position to shed our ‘beta’ tag. It has been a long time coming and speaks to the huge effort from hundreds of people over the past 5 years.

A critical step towards this goal is the release of Synapse 1.0. We want to ship Synapse 1.0 as soon as possible but can’t do so without your help!

We’d like to introduce AreWeReadyYet.com – a quick and easy way for everyone to track the progress and check if their federation is ready for Matrix 1.0!!

Are we ready yet?

Synapse 1.0 is good news for anyone running a Synapse installation – it contains critical bug fixes, security patches, a new room algorithm version and dramatically improved user and room search. However, as part of the security work it also contains a breaking change from previous Synapse versions. From 1.0 onwards it will necessary to ensure a valid TLS certificate on the federation API. Self signed certificates will no longer be accepted. Why would we do such a thing?

In anticipation for this, everyone currently running a homeserver must ensure that they have checked their federation certificate (check yours here). Failure to do so will mean being unable to federate with any Matrix 1.0 compliant server. If your server fails the check, our FAQ has all the details on what you need to do.

This post is a call to arms to try and get as many admins to upgrade their certificates as possible. We are tracking adoption at https://arewereadyyet.com – currently this sits at about 55% – we need this figure to be higher before we can pull the lever.  

So what are you waiting for? Check that your server has valid certs – then tell all your admin pals to do the same. Friends don’t let friends miss out on Synapse 1.0, send them to arewereadyyet.com (or tweet here to remind them!) We really need the community to help us here because at some point soon, we will need to pull the lever and release.

Once we make more progress on adoption, we will announce an official release date and finally get Synapse out of beta!

Synapse 0.99.2 released!

Well now, what have we here? Synapse 0.99.2 is the latest in the 0.99.x series as we step ever closer to 1.0.

0.99.2 is an incremental release including a bunch of performance improvements, enhancements to room upgrades and generally a plethora of bug fixes.

The most important thing that admins should know is that prior to 1.0 landing later this month, it is essential that the federation API has a valid TLS certificate – self signed certificates will no longer be accepted. For more details see our handy guide. Failure to do this will result in being unable to federate with other 1.0 servers.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.

 

Synapse 0.99.2 changelog

Features

  • Added an HAProxy example in the reverse proxy documentation. Contributed by Benoît S. (“Benpro”). (#4541)
  • Add basic optional sentry integration. (#4632#4694)
  • Transfer bans on room upgrade. (#4642)
  • Add configurable room list publishing rules. (#4647)
  • Support .well-known delegation when issuing certificates through ACME. (#4652)
  • Allow registration and login to be handled by a worker instance. (#4666#4670#4682)
  • Reduce the overhead of creating outbound federation connections over TLS by caching the TLS client options. (#4674)
  • Add prometheus metrics for number of outgoing EDUs, by type. (#4695)
  • Return correct error code when inviting a remote user to a room whose homeserver does not support the room version. (#4721)
  • Prevent showing rooms to other servers that were set to not federate. (#4746)

Bugfixes

  • Fix possible exception when paginating. (#4263)
  • The dependency checker now correctly reports a version mismatch for optional
    dependencies, instead of reporting the dependency missing. (#4450)
  • Set CORS headers on .well-known requests. (#4651)
  • Fix kicking guest users on guest access revocation in worker mode. (#4667)
  • Fix an issue in the database migration script where the
    e2e_room_keys.is_verified column wasn’t considered as
    a boolean. (#4680)
  • Fix TaskStopped exceptions in logs when outbound requests time out. (#4690)
  • Fix ACME config for python 2. (#4717)
  • Fix paginating over federation persisting incorrect state. (#4718)

Internal Changes

  • Run black to reformat user directory code. (#4635)
  • Reduce number of exceptions we log. (#4643#4668)
  • Introduce upsert batching functionality in the database layer. (#4644)
  • Fix various spelling mistakes. (#4657)
  • Cleanup request exception logging. (#4669#4737#4738)
  • Improve replication performance by reducing cache invalidation traffic. (#4671#4715#4748)
  • Test against Postgres 9.5 as well as 9.4. (#4676)
  • Run unit tests against python 3.7. (#4677)
  • Attempt to clarify installation instructions/config. (#4681)
  • Clean up gitignores. (#4688)
  • Minor tweaks to acme docs. (#4689)
  • Improve the logging in the pusher process. (#4691)
  • Better checks on newsfragments. (#4698#4750)
  • Avoid some redundant work when processing read receipts. (#4706)
  • Run push_receipts_to_remotes as background job. (#4707)
  • Add prometheus metrics for number of badge update pushes. (#4709)
  • Reduce pusher logging on startup (#4716)
  • Don’t log exceptions when failing to fetch remote server keys. (#4722)
  • Correctly proxy exception in frontend_proxy worker. (#4723)
  • Add database version to phonehome stats. (#4753)

This Week in Matrix 2019-03-01

No Ben this week, apparently he is allowed to go on holiday from time to time. Also no Matrix Live because we are terrible people – bring back Ben, that’s what I say.

Quaternion

kitsune:

Quaternion (master branch, and upcoming v0.0.9.4) can now open rooms by their aliases or ids upon pressing Ctrl+O, as long as those rooms are already in your room list (opening arbitrary public rooms will come in later versions). You can even paste matrix.to URIs for users (will open direct chat) and rooms in the same dialog. Navigation to known rooms inside Quaternion also works.

libQMatrixClient 0.5 is out – with members lazy-loading, room versions and upgrades and much more: https://github.com/QMatrixClient/libqmatrixclient/releases/tag/0.5.0

Which is, err, pretty amazing actually. As far as I know this is the only independent SDK supporting lazy loading and room upgrades. Congratulations.

weechat-matrix

Poljar has been hard at work on weechat-matrix

Matthew reports:-

weechat-matrix’s e2e support is really impressive (via matrix-nio and python-olm). It can only read rather than send right now, but otherwise looks to be massively on the right track. It even does fingerprint-based verification!

Synapse

We shipped 0.99.2 this week, it’s a point release containing all the usual bug fixes and perf improvements. We have also been taking a look at our docs and trying to improve where we can.

Hawkowl has spent some time improving CI so that we don’t get queued up for hours waiting for builds (woo).

Admins – your weekly reminder that if you’ve not already done so, you must ensure the TLS certificate on your federation endpoints is no longer self signedsee our handy guide for all the details.

matrix-puppet-bridge

tom:

matrix-puppet-bridge v1.17.0 has been released!

This release:

Matrix rooms representing remote rooms being joinable by anyone who knows the room ID (which is generated, at least in part, from the remote room ID in all the matrix-puppet-bridge applications) was a big deficiency, and it’s finally resolved. It wasn’t known whether or not we could do this, and have our ghost users still be able to join the rooms (they need to be invited instead of just joining themselves), until it was attempted and tested in a few of the bridge applications.

Because it is a big deal, new minor versions of matrix-puppet-slackmatrix-puppet-facebookmatrix-puppet-hangoutsmatrix-puppet-signalmatrix-puppet-imessage and matrix-puppet-groupme, bumping the matrix-puppet-bridge version to 1.17.0, have been released.

Riot iOS

  • Support .well-known data from login response
  • Create right permalinks for room v3
  • Fix builder to Xcode 10.1 in order to still be able to release
  • Update swift version. Add swiftLint

matrix-media-repo

TravisR reports that:

matrix-media-repo has alpha-quality support for s3 (and s3-like services) on the travis/s3 branch. Intrepid testers are encouraged to give it a shot, and report bugs. Caution: may upload your cat.

Scylla

It’s been a little while since we saw any news about Scylla, a web-based client written in Elm. VaNilLa said:

Scylla, the Elm-based Matrix client, has received a few improvements. Included in this update:

  • Files that are uploaded now keep their original name (as opposed to being called File)
  • Sent messages are displayed right away, instead of when they are delivered to the server
  • Usernames are only shown once per several consecutive messages, to avoid clutter Some CSS changes have been made to keep the interface looking clean

Almost forgot! You can find Scylla’s source code here and a live instance here!

Riot Android

  • Implementation of .well-known support (SDK and Riot)
  • Minor change on some colors of the themes (link, home badges)
  • Many issue will be fixed regarding linkification
  • KeysBackup: improvement on recovery process: importing keys step is 8 times faster, and user get more feedback during the process which can take several seconds
  • We will prepare a new release for the beginning of next week.
  • PlayStore new descriptions have been updated for the following languages: Bulgarian, German, English (US), French, Hungarian, Russian and Chinese (Taiwan).

RiotX (Android)

  • Timeline:
    • HTML formatting
    • Implementation of pills (need optimization)
  • Many Github issues have been created to track parity with the Riot Android

Riotic: new fork with updates

Aaron Raimist has made some updates to Riotic, which was a good chance for me to revisit it. It works nicely and is a great alternative to the Electron version of Riot. I also like being able to use https://riot.im/develop as an app.

I’ve slightly tweaked Joakim Ahlen’s Riot wrapper for macOS, riotic, which uses the native WKWebView instead of Electron. I updated the app to be sandboxed so it has very limited access to your system. I also updated the interface to follow macOS conventions and updated it to use the latest version of Swift.

riotic does have some limitations though. Riot doesn’t support VoIP on Safari so riotic can’t support VoIP either and WKWebView doesn’t support notifications as far as I can tell. Right now it uses a really old Riot icon, maybe I’ll ask about using one of these community made icons https://github.com/vector-im/riot-web/pull/4474.

It does have some advantages over the official Riot Electron app though. The app is only ~12 MB compared to Riot which is ~180 MB, it also uses significantly less RAM. riotic also allows you to pick what Riot URL to use so you can run /develop as a desktop app.

You can download riotic from https://github.com/aaronraimist/riotic/releases/tag/v1.1.

 

synapse-netcore-workers

Half-Shot has been working on synapse-netcore-workers:

The synapse-netcore-worker project has continued to evolve. You can now federate with other servers using the federation sender implementation. It supports everything except device lists at the moment, so it supports PDUs/EDUs and can just be connected up to one of your existing synapse instances. It’s not been battletested enough yet to be put in production (hence no dockerfile), but it’s very fast.
Oh and for those of you who don’t know, “synapse-netcore-worker” is Travis’s .NET implementation of synapse workers, the room can be found at #synapse-netcore-workers:t2bot.io.

If you’re as uninitiated as I was three days ago, this project is a replacable worker component for Synapse, which just happens to be written in .NET.

But wait, there’s more! Half-Shot continues with

To give an update to the earlier exploits of synapse-netcore-worker’s federation sender, we’ve still got a few more things to iron out before we can suggest people use it actively. The hit list of remaining things to fix is in https://github.com/turt2live/synapse-netcore-workers/pull/5

Mautrix Whatsapp

That Ben guy eh? When he’s not swanning off on holiday and having fun, he’s writing super groovy guides to getting whatsapp bridging up and running. You don’t even need a real device. Check it out.

Ruby SDK

Ananace reports that:

Just merged the protocol split branch I’ve been working on for the Ruby SDK, including a first PoC for an application service base. Not tested in any actual use as of yet, but expect Things TM in the next release.

Koma

yuforia reports that:

in project koma, a new bot picsay is created. It like the classic easter-egg program cowsay, but it uses actual photos instead of ASCII art. It configured to use any image just by editing a json file. So you can run your own version for fun.

TChap

Willem reports that:

I took some time to hack on Tchap again. This time I disabled virus scanning of thumbnails and downloads in Tchap to be able to see avatars. I also wrote a non-scanning virus scanner API implementation to be able to see files, pictures and videos that are sent. The updated Tchap can be found at https://github.com/14mRh4X0r/tchap-android, the virus scanner API implementation at https://git.snt.utwente.nl/14mRh4X0r/tchap-media-scanner.

Debian

andrewsh reports that:

Unless things get seriously on fire in the coming months, Debian buster will have Synapse 0.99.2 plus something newer in backports

Matrix WUG

Dandellion’s Wug now supports Inuktitut Syllabics and Iñupiatun Orthography. In the bot’s own words.

Hi I can help you translate X-SAMPA, Z-SAMPA to IPA, and transcribe into proto-indo european notation! Use (x/z/p) together with either / or [] as delimeters x/”hEloU/ z[or` 5aIk DIz] p/mreghnom/

I also can transcribe to Inuktitut Syllabics like this: i[tusaumaqattautijjutinik aulattijiit]. Find my source at https://github.com/Dali99/matrix-wug

New Rooms

Aaron Raimist has created a new room for Formula 1 fans:

Now that the Formula 1 season is getting underway it’s probably a good time to announce @CIA:matrix.org‘s new(ish) room: #f1:matrix.org When this blog post is released there will be 16 days left before the first race of the 2019 season
There was previously a Formula 1 room but it was merged with Snoonet’s IRC channel which tends to be extremely busy. This is a matrix only room.

That’s it folks, your normal Ben orientated programming will continue next week. Bring back Ben, bring back Ben.

Publishing the Backend Roadmap

Good people,

2019 is a big year for Matrix, in the next month we will have shipped:

  • Matrix spec 1.0 (including the first stable release of the Server to Server Spec)
  • Synapse 1.0
  • Riot 1.0

This is huge in itself, but is really only the beginning, and now we want to grow the ecosystem as quickly as possible. This means landing a mix of new features, enhancing existing ones, some big performance improvements as well as generally making life easier for our regular users, homeserver admins and community developers.

Today we are sharing the Matrix core team’s backend roadmap. The idea is that this will make it easier for anyone to understand where the project is going, what we consider to be important, and why.

To see the roadmap in its full glory, take a look here.

What is a roadmap and why is it valuable?

A roadmap is a set of high level projects that the team intend to work on and a rough sense of the relative priority. It is essential to focus on specific goals, which inevitably means consciously not working on other initiatives.

Our roadmap is not a delivery plan – there are explicitly no dates. The reason for this is that we know that other projects will emerge, developers will be needed to support other urgent initiatives, matrix.org use continues to grow exponentially and will require performance tweaking.

So simply, based on what we know now, this is the order we will work on our projects.

Why are we sharing it?

We already share our day to day todo list, and of course our commit history, but it can be difficult for a casual observer to see the bigger picture from such granular data. The purpose of sharing is that we want anyone from the community to understand where our priorities lie.

We are often asked ‘Why are you not working on X, it is really important’ where the answer is often ‘We agree that X is really important, but A, B and C are more important and must come first’.

The point of sharing the roadmap is to make that priority trade off more transparent and consumable.

How did we build it?

The core contributors to Synapse and Dendrite are 6 people, of 5 nationalities spread across 3 locations. After shipping the r0 release of the Server to Server spec last month we took some time to step back and have a think about what to do after Synapse 1.0 lands. This meant getting everyone in one place to talk it through.

We also had Ben (benpa) contribute from a community perspective and took input from speaking to so many of you at FOSDEM.

In the end we filled a wall with post-its, each post-it representing a sizeable project. The position of the post-it was significant in that the vertical axis being a sense of how valuable we thought the task would be, and the horizontal axis being a rough guess on how complex we considered it to be.

We found this sort of grid approach to be really helpful in determining relative priority.

After many hours and plenty of blood, sweat and tears we ended up with something we could live with and wrote it up in the shared board.

And this is written in blood right?

Not at all (it’s written in board marker). This is simply a way to express our plan of action and we are likely to make changes to it dynamically. However, this means that at any given moment, if someone wants to know what we are working on then the roadmap is the place to go.

But wait I want to know more!

Here is a video of myself and Matthew to talk you through the projects

Interesting, but I have questions …

Any feedback gratefully received, come and ask questions in #synapse or #dendrite or feel free to ping me direct at @neilj:matrix.org

 

Synapse 0.99.1.1 Released!

Hey, everyone, today is the day we release Synapse 0.99.1.1

This release contains improved ACME support to make it even easier to get going with TLS certs on your federation end points, plus some tweaks to make the room version upgrade path easier.

Just as a reminder that the 0.99.x series is precursor for our 1.0 release (which will land in early March, exact date to be confirmed) – it is really important that all server admins are aware that self signed certificates on the Server to Server API will no longer be accepted by >= Synapse 1.0. If you have not already done so, now is the time to configure your certificate. For more info see our FAQ and if you get stuck come and join us in #Synapse.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.

Synapse 0.99.1.1 Changelog

Bugfixes

  • Fix “TypeError: ‘>’ not supported” when starting without an existing certificate.
    Fix a bug where an existing certificate would be reprovisoned every day. (#4648)

Synapse 0.99.1 Changelog

Features

  • Include m.room.encryption on invites by default (#3902)
  • Federation OpenID listener resource can now be activated even if federation is disabled (#4420)
  • Synapse’s ACME support will now correctly reprovision a certificate that approaches its expiry while Synapse is running. (#4522)
  • Add ability to update backup versions (#4580)
  • Allow the “unavailable” presence status for /sync.
    This change makes Synapse compliant with r0.4.0 of the Client-Server specification. (#4592)
  • There is no longer any need to specify no_tls: it is inferred from the absence of TLS listeners (#4613#4615#4617#4636)
  • The default configuration no longer requires TLS certificates. (#4614)

Bugfixes

  • Copy over room federation ability on room upgrade. (#4530)
  • Fix noisy “twisted.internet.task.TaskStopped” errors in logs (#4546)
  • Synapse is now tolerant of the tls_fingerprints option being None or not specified. (#4589)
  • Fix ‘no unique or exclusion constraint’ error (#4591)
  • Transfer Server ACLs on room upgrade. (#4608)
  • Fix failure to start when not TLS certificate was given even if TLS was disabled. (#4618)
  • Fix self-signed cert notice from generate-config. (#4625)
  • Fix performance of user_ips table deduplication background update (#4626#4627)

Internal Changes

  • Change the user directory state query to use a filtered call to the db instead of a generic one. (#4462)
  • Reject federation transactions if they include more than 50 PDUs or 100 EDUs. (#4513)
  • Reduce duplication of synapse.app code. (#4567)
  • Fix docker upload job to push -py2 images. (#4576)
  • Add port configuration information to ACME instructions. (#4578)
  • Update MSC1711 FAQ to calrify .well-known usage (#4584)
  • Clean up default listener configuration (#4586)
  • Clarifications for reverse proxy docs (#4607)
  • Move ClientTLSOptionsFactory init out of refresh_certificates (#4611)
  • Fail cleanly if listener config lacks a ‘port’ (#4616)
  • Remove redundant entries from docker config (#4619)
  • README updates (#4621)