Neil Johnson

Well now, what have we here? Synapse 0.99.2 is the latest in the 0.99.x series as we step ever closer to 1.0.

0.99.2 is an incremental release including a bunch of performance improvements, enhancements to room upgrades and generally a plethora of bug fixes.

The most important thing that admins should know is that prior to 1.0 landing later this month, it is essential that the federation API has a valid TLS certificate - self signed certificates will no longer be accepted. For more details see our handy guide. Failure to do this will result in being unable to federate with other 1.0 servers.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.

Synapse 0.99.2 changelog

Features

• Added an HAProxy example in the reverse proxy documentation. Contributed by Benoît S. (“Benpro”). (#4541)
• Add basic optional sentry integration. (#4632, #4694)
• Transfer bans on room upgrade. (#4642)
• Add configurable room list publishing rules. (#4647)
• Support .well-known delegation when issuing certificates through ACME. (#4652)
• Allow registration and login to be handled by a worker instance. (#4666, #4670, #4682)
• Reduce the overhead of creating outbound federation connections over TLS by caching the TLS client options. (#4674)
• Add prometheus metrics for number of outgoing EDUs, by type. (#4695)
• Return correct error code when inviting a remote user to a room whose homeserver does not support the room version. (#4721)
• Prevent showing rooms to other servers that were set to not federate. (#4746)

Bugfixes

• Fix possible exception when paginating. (#4263)
• The dependency checker now correctly reports a version mismatch for optional dependencies, instead of reporting the dependency missing. (#4450)
• Set CORS headers on .well-known requests. (#4651)
• Fix kicking guest users on guest access revocation in worker mode. (#4667)
• Fix an issue in the database migration script where the e2e_room_keys.is_verified column wasn't considered as a boolean. (#4680)
• Fix TaskStopped exceptions in logs when outbound requests time out. (#4690)
• Fix ACME config for python 2. (#4717)
• Fix paginating over federation persisting incorrect state. (#4718)

Internal Changes

• Run black to reformat user directory code. (#4635)
• Reduce number of exceptions we log. (#4643, #4668)
• Introduce upsert batching functionality in the database layer. (#4644)
• Fix various spelling mistakes. (#4657)
• Cleanup request exception logging. (#4669, #4737, #4738)
• Improve replication performance by reducing cache invalidation traffic. (#4671, #4715, #4748)
• Test against Postgres 9.5 as well as 9.4. (#4676)
• Run unit tests against python 3.7. (#4677)
• Attempt to clarify installation instructions/config. (#4681)
• Clean up gitignores. (#4688)
• Minor tweaks to acme docs. (#4689)
• Improve the logging in the pusher process. (#4691)
• Better checks on newsfragments. (#4698, #4750)
• Avoid some redundant work when processing read receipts. (#4706)
• Run push_receipts_to_remotes as background job. (#4707)
• Add prometheus metrics for number of badge update pushes. (#4709)
• Reduce pusher logging on startup (#4716)
• Don't log exceptions when failing to fetch remote server keys. (#4722)
• Correctly proxy exception in frontend_proxy worker. (#4723)
• Add database version to phonehome stats. (#4753)

No Ben this week, apparently he is allowed to go on holiday from time to time. Also no Matrix Live because we are terrible people - bring back Ben, that's what I say.

Quaternion

kitsune:

Quaternion (master branch, and upcoming v0.0.9.4) can now open rooms by their aliases or ids upon pressing Ctrl+O, as long as those rooms are already in your room list (opening arbitrary public rooms will come in later versions). You can even paste matrix.to URIs for users (will open direct chat) and rooms in the same dialog. Navigation to known rooms inside Quaternion also works.

libQMatrixClient 0.5 is out - with members lazy-loading, room versions and upgrades and much more: https://github.com/QMatrixClient/libqmatrixclient/releases/tag/0.5.0

Which is, err, pretty amazing actually. As far as I know this is the only independent SDK supporting lazy loading and room upgrades. Congratulations.

weechat-matrix

Poljar has been hard at work on weechat-matrix

Matthew reports:-

weechat-matrix's e2e support is really impressive (via matrix-nio and python-olm). It can only read rather than send right now, but otherwise looks to be massively on the right track. It even does fingerprint-based verification!

Synapse

We shipped 0.99.2 this week, it's a point release containing all the usual bug fixes and perf improvements. We have also been taking a look at our docs and trying to improve where we can.

Hawkowl has spent some time improving CI so that we don't get queued up for hours waiting for builds (woo).

Admins - your weekly reminder that if you've not already done so, you must ensure the TLS certificate on your federation endpoints is no longer self signed - see our handy guide for all the details.

matrix-puppet-bridge

tom:

matrix-puppet-bridge v1.17.0 has been released! This release:

• Handles m.notice messages sent from Matrix as text, instead of ignoring them (thanks go to https://matrix.to/#/@cdknight1:matrix.org !)
• Sets rooms invite-only when they're touched, instead of relying on others not knowing the room ID (thanks to https://matrix.to/#/@AndrewJDR:matrix.org , from all of us who federate on the homeservers we use to bridge!)

Matrix rooms representing remote rooms being joinable by anyone who knows the room ID (which is generated, at least in part, from the remote room ID in all the matrix-puppet-bridge applications) was a big deficiency, and it's finally resolved. It wasn't known whether or not we could do this, and have our ghost users still be able to join the rooms (they need to be invited instead of just joining themselves), until it was attempted and tested in a few of the bridge applications.
Because it is a big deal, new minor versions of matrix-puppet-slack, matrix-puppet-facebook, matrix-puppet-hangouts, matrix-puppet-signal, matrix-puppet-imessage and matrix-puppet-groupme, bumping the matrix-puppet-bridge version to 1.17.0, have been released.

Riot iOS

• Support .well-known data from login response
• Create right permalinks for room v3
• Fix builder to Xcode 10.1 in order to still be able to release
• Update swift version. Add swiftLint

matrix-media-repo

TravisR reports that:

matrix-media-repo has alpha-quality support for s3 (and s3-like services) on the travis/s3 branch. Intrepid testers are encouraged to give it a shot, and report bugs. Caution: may upload your cat.

Scylla

It's been a little while since we saw any news about Scylla, a web-based client written in Elm. VaNilLa said:

Scylla, the Elm-based Matrix client, has received a few improvements. Included in this update:

• Files that are uploaded now keep their original name (as opposed to being called File)
• Sent messages are displayed right away, instead of when they are delivered to the server
• Usernames are only shown once per several consecutive messages, to avoid clutter Some CSS changes have been made to keep the interface looking clean

Almost forgot! You can find Scylla's source code here and a live instance here!

Riot Android

• Implementation of .well-known support (SDK and Riot)
• Minor change on some colors of the themes (link, home badges)
• Many issue will be fixed regarding linkification
• KeysBackup: improvement on recovery process: importing keys step is 8 times faster, and user get more feedback during the process which can take several seconds
• We will prepare a new release for the beginning of next week.
• PlayStore new descriptions have been updated for the following languages: Bulgarian, German, English (US), French, Hungarian, Russian and Chinese (Taiwan).

RiotX (Android)

• Timeline:
  • HTML formatting
  • Implementation of pills (need optimization)
• Many Github issues have been created to track parity with the Riot Android

Riotic: new fork with updates

Aaron Raimist has made some updates to Riotic, which was a good chance for me to revisit it. It works nicely and is a great alternative to the Electron version of Riot. I also like being able to use https://riot.im/develop as an app.

I've slightly tweaked Joakim Ahlen's Riot wrapper for macOS, riotic, which uses the native WKWebView instead of Electron. I updated the app to be sandboxed so it has very limited access to your system. I also updated the interface to follow macOS conventions and updated it to use the latest version of Swift.

riotic does have some limitations though. Riot doesn't support VoIP on Safari so riotic can't support VoIP either and WKWebView doesn't support notifications as far as I can tell. Right now it uses a really old Riot icon, maybe I'll ask about using one of these community made icons https://github.com/vector-im/riot-web/pull/4474.

It does have some advantages over the official Riot Electron app though. The app is only ~12 MB compared to Riot which is ~180 MB, it also uses significantly less RAM. riotic also allows you to pick what Riot URL to use so you can run /develop as a desktop app.

You can download riotic from https://github.com/aaronraimist/riotic/releases/tag/v1.1.

 

synapse-netcore-workers

Half-Shot has been working on synapse-netcore-workers:

The synapse-netcore-worker project has continued to evolve. You can now federate with other servers using the federation sender implementation. It supports everything except device lists at the moment, so it supports PDUs/EDUs and can just be connected up to one of your existing synapse instances. It's not been battletested enough yet to be put in production (hence no dockerfile), but it's very fast.
Oh and for those of you who don't know, "synapse-netcore-worker" is Travis's .NET implementation of synapse workers, the room can be found at #synapse-netcore-workers:t2bot.io.

If you're as uninitiated as I was three days ago, this project is a replaceable worker component for Synapse, which just happens to be written in .NET.

But wait, there's more! Half-Shot continues with

To give an update to the earlier exploits of synapse-netcore-worker's federation sender, we've still got a few more things to iron out before we can suggest people use it actively. The hit list of remaining things to fix is in https://github.com/turt2live/synapse-netcore-workers/pull/5

Mautrix Whatsapp

That Ben guy eh? When he's not swanning off on holiday and having fun, he's writing super groovy guides to getting whatsapp bridging up and running. You don't even need a real device. Check it out.

Ruby SDK

Ananace reports that:

Just merged the protocol split branch I've been working on for the Ruby SDK, including a first PoC for an application service base. Not tested in any actual use as of yet, but expect Things TM in the next release.

Koma

yuforia reports that:

in project koma, a new bot picsay is created. It like the classic easter-egg program cowsay, but it uses actual photos instead of ASCII art. It configured to use any image just by editing a json file. So you can run your own version for fun.

TChap

Willem reports that:

I took some time to hack on Tchap again. This time I disabled virus scanning of thumbnails and downloads in Tchap to be able to see avatars. I also wrote a non-scanning virus scanner API implementation to be able to see files, pictures and videos that are sent. The updated Tchap can be found at https://github.com/14mRh4X0r/tchap-android, the virus scanner API implementation at https://git.snt.utwente.nl/14mRh4X0r/tchap-media-scanner.

Debian

andrewsh reports that:

Unless things get seriously on fire in the coming months, Debian buster will have Synapse 0.99.2 plus something newer in backports

Matrix WUG

Dandellion's Wug now supports Inuktitut Syllabics and Iñupiatun Orthography. In the bot's own words.

Hi I can help you translate X-SAMPA, Z-SAMPA to IPA, and transcribe into proto-indo european notation! Use (x/z/p) together with either / or [] as delimiters x/"hEloU/ z[or` 5aIk DIz] p/mreghnom/

I also can transcribe to Inuktitut Syllabics like this: i[tusaumaqattautijjutinik aulattijiit]. Find my source at https://github.com/Dali99/matrix-wug

New Rooms

Aaron Raimist has created a new room for Formula 1 fans:

Now that the Formula 1 season is getting underway it's probably a good time to announce @CIA:matrix.org's new(ish) room: #f1:matrix.org When this blog post is released there will be 16 days left before the first race of the 2019 season
There was previously a Formula 1 room but it was merged with Snoonet's IRC channel which tends to be extremely busy. This is a matrix only room.

That's it folks, your normal Ben orientated programming will continue next week. Bring back Ben, bring back Ben.

Good people,

2019 is a big year for Matrix, in the next month we will have shipped:

• Matrix spec 1.0 (including the first stable release of the Server to Server Spec)
• Synapse 1.0
• Riot 1.0

This is huge in itself, but is really only the beginning, and now we want to grow the ecosystem as quickly as possible. This means landing a mix of new features, enhancing existing ones, some big performance improvements as well as generally making life easier for our regular users, homeserver admins and community developers.

Today we are sharing the Matrix core team's backend roadmap. The idea is that this will make it easier for anyone to understand where the project is going, what we consider to be important, and why.

To see the roadmap in its full glory, take a look here.

What is a roadmap and why is it valuable?

A roadmap is a set of high level projects that the team intend to work on and a rough sense of the relative priority. It is essential to focus on specific goals, which inevitably means consciously not working on other initiatives.

Our roadmap is not a delivery plan - there are explicitly no dates. The reason for this is that we know that other projects will emerge, developers will be needed to support other urgent initiatives, matrix.org use continues to grow exponentially and will require performance tweaking.

So simply, based on what we know now, this is the order we will work on our projects.

Why are we sharing it?

We already share our day to day todo list, and of course our commit history, but it can be difficult for a casual observer to see the bigger picture from such granular data. The purpose of sharing is that we want anyone from the community to understand where our priorities lie.

We are often asked ‘Why are you not working on X, it is really important' where the answer is often ‘We agree that X is really important, but A, B and C are more important and must come first'.

The point of sharing the roadmap is to make that priority trade off more transparent and consumable.

How did we build it?

The core contributors to Synapse and Dendrite are 6 people, of 5 nationalities spread across 3 locations. After shipping the r0 release of the Server to Server spec last month we took some time to step back and have a think about what to do after Synapse 1.0 lands. This meant getting everyone in one place to talk it through.

We also had Ben (benpa) contribute from a community perspective and took input from speaking to so many of you at FOSDEM.

In the end we filled a wall with post-its, each post-it representing a sizeable project. The position of the post-it was significant in that the vertical axis being a sense of how valuable we thought the task would be, and the horizontal axis being a rough guess on how complex we considered it to be.

We found this sort of grid approach to be really helpful in determining relative priority.

After many hours and plenty of blood, sweat and tears we ended up with something we could live with and wrote it up in the shared board.

And this is written in blood right?

Not at all (it's written in board marker). This is simply a way to express our plan of action and we are likely to make changes to it dynamically. However, this means that at any given moment, if someone wants to know what we are working on then the roadmap is the place to go.

But wait I want to know more!

Here is a video of myself and Matthew to talk you through the projects

Interesting, but I have questions ...

Any feedback gratefully received, come and ask questions in #synapse or #dendrite or feel free to ping me direct at @neilj:matrix.org

Hey, everyone, today is the day we release Synapse 0.99.1.1

This release contains improved ACME support to make it even easier to get going with TLS certs on your federation end points, plus some tweaks to make the room version upgrade path easier.

Just as a reminder that the 0.99.x series is precursor for our 1.0 release (which will land in early March, exact date to be confirmed) - it is really important that all server admins are aware that self signed certificates on the Server to Server API will no longer be accepted by >= Synapse 1.0. If you have not already done so, now is the time to configure your certificate. For more info see our FAQ and if you get stuck come and join us in #Synapse.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.

Synapse 0.99.1.1 Changelog

Bugfixes

• Fix "TypeError: '>' not supported" when starting without an existing certificate. Fix a bug where an existing certificate would be reprovisoned every day. (#4648)

Synapse 0.99.1 Changelog

Features

• Include m.room.encryption on invites by default (#3902)
• Federation OpenID listener resource can now be activated even if federation is disabled (#4420)
• Synapse's ACME support will now correctly reprovision a certificate that approaches its expiry while Synapse is running. (#4522)
• Add ability to update backup versions (#4580)
• Allow the "unavailable" presence status for /sync. This change makes Synapse compliant with r0.4.0 of the Client-Server specification. (#4592)
• There is no longer any need to specify no_tls: it is inferred from the absence of TLS listeners (#4613, #4615, #4617, #4636)
• The default configuration no longer requires TLS certificates. (#4614)

Bugfixes

• Copy over room federation ability on room upgrade. (#4530)
• Fix noisy "twisted.internet.task.TaskStopped" errors in logs (#4546)
• Synapse is now tolerant of the tls_fingerprints option being None or not specified. (#4589)
• Fix 'no unique or exclusion constraint' error (#4591)
• Transfer Server ACLs on room upgrade. (#4608)
• Fix failure to start when not TLS certificate was given even if TLS was disabled. (#4618)
• Fix self-signed cert notice from generate-config. (#4625)
• Fix performance of user_ips table deduplication background update (#4626, #4627)

Internal Changes

• Change the user directory state query to use a filtered call to the db instead of a generic one. (#4462)
• Reject federation transactions if they include more than 50 PDUs or 100 EDUs. (#4513)
• Reduce duplication of synapse.app code. (#4567)
• Fix docker upload job to push -py2 images. (#4576)
• Add port configuration information to ACME instructions. (#4578)
• Update MSC1711 FAQ to clarify .well-known usage (#4584)
• Clean up default listener configuration (#4586)
• Clarifications for reverse proxy docs (#4607)
• Move ClientTLSOptionsFactory init out of refresh_certificates (#4611)
• Fail cleanly if listener config lacks a 'port' (#4616)
• Remove redundant entries from docker config (#4619)
• README updates (#4621)

Hey hey, Synapse 0.99.0 is here!

You may have heard that we recently published the first stable release of the Server to Server Spec (r0.1). The spec makes some changes which are not compatible with the protocol of the past - particularly, self-signed certificates are no longer valid for homeservers. Synapse 1.0.0 will be compliant with r0.1 and the goal of Synapse 0.99.0 is to act as a stepping stone to Synapse 1.0. Synapse 0.99.0 supports the r0.1 release of the server to server specification, but is compatible with both the legacy Matrix federation behaviour (pre-r0.1) as well as post-r0.1 behaviour, in order to allow for a smooth upgrade across the federation.

It is critical that all admins upgrade to 0.99.0 and configure a valid TLS certificate. Admins will have 1 month to do so, after which 1.0.0 will be released and those servers without a valid certificate will no longer be able to federate with >= 1.0.0 servers.

First of all, please don't panic :) We have taken steps to make this process as simple as possible - specifically implementing ACME support to allow servers to automatically generate free Let's Encrypt certificates if you choose to. What's more, it is not necessary to add the certificate right away, you have at least a month to get set up.

For more details on exactly what you need to do (and also why this change is essential), we have provided an extensive FAQ as well as the Upgrade notes for Synapse

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.

This was a huge effort! Congratulations to all involved, especially those of you in the community who contributed to spec MSCs and tested our release candidates. Thank you for bearing with us as we move the whole public Matrix Federation onto r0.1 compliant servers.

Onwards!

Changelog

Synapse v0.99.x is a precursor to the upcoming Synapse v1.0 release. It contains foundational changes to room architecture and the federation security model necessary to support the upcoming r0 release of the Server to Server API.

Features

• Synapse's cipher string has been updated to require ECDH key exchange. Configuring and generating dh_params is no longer required, and they will be ignored. (#4229)
• Synapse can now automatically provision TLS certificates via ACME (the protocol used by CAs like Let's Encrypt). (#4384, #4492, #4525, #4572, #4564, #4566, #4547, #4557)
• Implement MSC1708 (.well-known routing for server-server federation) (#4408, #4409, #4426, #4427, #4428, #4464, #4468, #4487, #4488, #4489, #4497, #4511, #4516, #4520, #4521, #4539, #4542, #4544)
• Search now includes results from predecessor rooms after a room upgrade. (#4415)
• Config option to disable requesting MSISDN on registration. (#4423)
• Add a metric for tracking event stream position of the user directory. (#4445)
• Support exposing server capabilities in CS API (MSC1753, MSC1804) (#4472, 81b7e7eed))
• Add support for room version 3 (#4483, #4499, #4515, #4523, #4535)
• Synapse will now reload TLS certificates from disk upon SIGHUP. (#4495, #4524)
• The matrixdotorg/synapse Docker images now use Python 3 by default. (#4558)

Bugfixes

• Prevent users with access tokens predating the introduction of device IDs from creating spurious entries in the user_ips table. (#4369)
• Fix typo in ALL_USER_TYPES definition to ensure type is a tuple (#4392)
• Fix high CPU usage due to remote devicelist updates (#4397)
• Fix potential bug where creating or joining a room could fail (#4404)
• Fix bug when rejecting remote invites (#4405, #4527)
• Fix incorrect logcontexts after a Deferred was cancelled (#4407)
• Ensure encrypted room state is persisted across room upgrades. (#4411)
• Copy over whether a room is a direct message and any associated room tags on room upgrade. (#4412)
• Fix None guard in calling config.server.is_threepid_reserved (#4435)
• Don't send IP addresses as SNI (#4452)
• Fix UnboundLocalError in post_urlencoded_get_json (#4460)
• Add a timeout to filtered room directory queries. (#4461)
• Workaround for login error when using both LDAP and internal authentication. (#4486)
• Fix a bug where setting a relative consent directory path would cause a crash. (#4512)

Deprecations and Removals

• Synapse no longer generates self-signed TLS certificates when generating a configuration file. (#4509)

Improved Documentation

• Update debian installation instructions (#4526)

Internal Changes

• Synapse will now take advantage of native UPSERT functionality in PostgreSQL 9.5+ and SQLite 3.24+. (#4306, #4459, #4466, #4471, #4477, #4505)
• Update README to use the new virtualenv everywhere (#4342)
• Add better logging for unexpected errors while sending transactions (#4368)
• Apply a unique index to the user_ips table, preventing duplicates. (#4370, #4432, #4434)
• Silence travis-ci build warnings by removing non-functional python3.6 (#4377)
• Fix a comment in the generated config file (#4387)
• Add ground work for implementing future federation API versions (#4390)
• Update dependencies on msgpack and pymacaroons to use the up-to-date packages. (#4399)
• Tweak codecov settings to make them less loud. (#4400)
• Implement server support for MSC1794 - Federation v2 Invite API (#4402)
• debian package: symlink to explicit python version (#4433)
• Add infrastructure to support different event formats (#4437, #4447, #4448, #4470, #4481, #4482, #4493, #4494, #4496, #4510, #4514)
• Generate the debian config during build (#4444)
• Clarify documentation for the public_baseurl config param (#4458, #4498)
• Fix quoting for allowed_local_3pids example config (#4476)
• Remove deprecated --process-dependency-links option from UPGRADE.rst (#4485)
• Make it possible to set the log level for tests via an environment variable (#4506)
• Reduce the log level of linearizer lock acquirement to DEBUG. (#4507)
• Fix code to comply with linting in PyFlakes 3.7.1. (#4519)
• Add some debug for membership syncing issues (#4538)
• Docker: only copy what we need to the build image (#4562)

On Thursday Jan 10th we released a Critical Security Update (Synapse 0.34.0.1/0.34.1.1), which fixes a serious security bug in Synapse 0.34.0 and earlier. Many deployments have now upgraded to 0.34.0.1 or 0.34.1.1, and we now consider it appropriate to disclose more information about the issue, to provide context and encourage the remaining affected servers to upgrade as soon as possible.

In Synapse 0.11 (Nov 2015) we added a configuration parameter called “macaroon_secret_key” which relates to our use of macaroons in authentication. Macaroons are authentication tokens which must be signed by the server which generates them, to prevent them being forged by attackers. “macaroon_secret_key” defines the key which is used for this signature, and it must therefore be kept secret to preserve the security of the server.

If the option is not set, Synapse will attempt to derive a secret key from other secrets specified in the configuration file. However, in all versions of Synapse up to and including 0.34.0, this process was faulty and a predictable value was used instead.

So if, your homeserver.yaml does not contain a macaroon_secret_key, you need to upgrade to 0.34.1.1 or 0.34.0.1 or Debian 0.34.0-3~bpo9+2 immediately to prevent the risk of account hijacking.

The vulnerability affects any Synapse installation which does not have a macaroon_secret_key setting. For example, the Debian and Ubuntu packages from Matrix.org, Debian and Ubuntu include a configuration file without an explicit macaroon_secret_key and must upgrade. Anyone who hasn't updated their config since Nov 2015 or who grandfathered their config from the Debian/Ubuntu packages will likely also be affected.

We are not aware of this vulnerability being exploited in the wild, but if you are running an affected server it may still be wise to check your synapse's user_ips database table for any unexpected access to your server's accounts. You could also check your accounts' device lists (shown under Settings in Riot) for unexpected devices, although this is not as reliable as an attacker could cover their tracks to remove unexpected devices.

We'll publish a full post-mortem of the issue once we are confident that most affected servers have been upgraded.

We'd like to apologise for the inconvenience caused by this - especially to folks who upgraded since Friday who were in practice not affected. Due to the nature of the issue we wanted to minimise details about the issue until people had a chance to upgrade. We also did not follow a planned disclosure procedure because Synapse 0.34.1 already unintentionally disclosed the existence of the bug by fixing it (causing the logout bug for affected users which led us to pull the original Synapse 0.34.1 release).

On the plus side, we are approaching the end of beta for Synapse, and going forwards hope to see much better stability and security across the board.

Thanks again for your patience,

The Matrix.org Team

Welcome!

Do not panic, Benpa is away, I repeat, Benpa is away. Nonetheless TWIM lives on!

Spec

Lots of spec work this week, and a shout out to anoa for his magical mscbot that provides pokes, nudges and updates on all things spec. Here's what mscbot had to say about the past week.

Approved MSCs

[MSC 1497]: Advertising support of experimental features in the CS API [MSC 1339]: Proposal to add a GET method to read account data [MSC 1501]: Room version upgrades

Final Comment Period

MSC 1708: .well-known support for server name resolution MSC 1711: X.509 certificate verification for federation connections

New and In Progress MSCs

[MSC 1794]: Federation v2 Invite API [MSC 1796]: Improved e2e notifications [MSC 1797]: Proposal for more granular profile error codes [MSC 1640]: Replace event IDs with hashes [MSC 1776]: Implementing peeking via /sync [MSC 1777]: peeking over federation [MSC 1779]: Proposal for Open Governance for Matrix.org (v2)

(A few may be missing as we're still tweaking mscbot :)

Dendrite

Brendan had this to say:-

The Dendrite audit is over! A bunch of issues have been created on the Dendrite GitHub repository, as well as a project board in order to keep track of everything: https://github.com/matrix-org/dendrite/projects/2 There's a fair amount of issues that have been labeled as “good first issue”, so feel free to pick them up and open pull requests if you're looking into hacking on Dendrite! :)

And whilst we have your attention - here's Brendan & Matthew talking through the audit in this week's Matrix Live!

Synapse

Neil says:-

both contain critical security updates so please update asap for more details, we'll be able to share a bit more about the vuln once admins have had a chance to upgrade.

Meanwhile Hawkowl has been cranking out bug fixes and perf improvements and in particular taking a look at taming the user_ips table.

While Debian packager Andrewsh adds:-

latest synapse (0.34.1.1, Python 3) in Debian, fixing CVE-2019-5885; an update to a previous release fixing this CVE uploaded to stretch-backports, using Python 2. Dependencies for a Python 3 upload approved in stretch-backports, a Python 3 upload of 0.34.1.1 will be following later this week

Riot/iOS

Riot-iOS 0.7.11 has been released, with lots of bug fixes.

We have been working on e2e new screens (like key backup setup) and the re-skinning of the app.

Riot/Android

Working to improve notifications style.

Split screen mode will be supported on next release!

Continuous autofocus on the Camera has been enabled.

Also fighting bugs on registration.

Bridges

Halfshot has this to say:

Matrix-appservice-purple is being renamed to matrix-bifröst, on the basis that we now bridge to things and "burning rainbow bridge" seemed like a good description.

Other things that have happened: Performance improvements, as always. XMPP -> Matrix typing notifications XMPP -> Matrix avatars XMPP -> Matrix uploads * Matrix -> XMPP uploads (via oob)

and then follows up with this:-

As promised, we've got a discord bridge release out today. v0.4.0-rc1 has landed! See the change notes https://github.com/Half-Shot/matrix-appservice-discord/releases/tag/v0.4.0-rc1 . Thank you to Sorunome for doing a huge amount of work on this!
@swedneck reports that:

linuxgaming.life is now running matrix-appservice-discord v0.4.0-rc1.

Matrix.org Foundation

Matthew has a final draft of the Matrix.org Foundation governance document ready: https://github.com/matrix-org/matrix-doc/blob/matthew/msc1779/proposals/1779-open-governance.md. Comments on https://github.com/matrix-org/matrix-doc/pull/1779 would be much appreciated!  We expect to propose merging it next week, and then incorporating it into the final Articles of the foundation.

Riot Web

Loads and loads of work happening on https://riot.im/experimental which is now where all new development is happening as we race towards launching the new design.  Highlights include:

• All new key verification is implemented! (in olm & matrix-js-sdk).  We're currently hooking up the UX.
• Online key backup is pretty much finished.
• Cross-signing is up next.
• Redesign backlog is progressing (slightly stuck on making the RoomList resizing work nicely, but almost there)
• Finalising the all new registration/login screens
• ...and loads of other stuff too.

Meanwhile...

kitsune reports that:

Sending files landed in master branches of libQMatrixClient and Quaternion. Finally you can send your Quaternion screenshots (as any other images, jingles, cat videos etc.) to Matrix using Quaternion ;)

Also, libQMatrixClient is available as a Conan repository, for developers who'd like to use Conan to track dependencies.

progserega reports that:

Hello to all! I am write matrix bot for bridge messages between matrix and social network vk.com (russian analog of facebook). https://github.com/progserega/MatrixVkBot

alphapapa reports that:

matrix-client.el gained a room-list buffer, which can be sorted by unread status, name, number of members, etc, and has a right-click context menu like the room-list sidebar.

matrix-client.el gained right-click context menus in the room sidebar, allowing to set room priority, notifications, etc.

The matrix-client.el git repository has moved to: https://github.com/alphapapa/matrix-client.el

Stanislav N. aka pztrn reports that:

Hey guys, joined here to post another thing that works in Matrix https://gitlab.com/pztrn/check_mk_matrix_notifications it is a script that sends check_mk notifications to Matrix. Check_mk is a "plugin" for Nagios NMS.

Cadair reports that:

It's not my update but I saw this HomeAssistant addon for matrix (https://github.com/hassio-addons/addon-matrix) and wanted to make sure it got a shoutout on TWIM. [Seeing how nobody else has posted it in here, just on twitter etc.]

Morgan McMillian (thrrgilag) reports that:

I published v1.0.1 of the pnut-matrix bridge this week which brings public pnut.io chat rooms to the matrix network. Features include syncing of pnut.io names and avatars, matrix users ability to authorize their pnut.io accounts, and administrative controls for managing linked rooms. Project can be found at https://gitlab.dreamfall.space/thrrgilag/pnut-matrix and discussion is at #pnut-matrix:monkeystew.net

MMJD reports that:

ma1uta's MXToot deserves mention in the blog, and in https://matrix.org/docs/projects/try-matrix-now.html . People should not be wanting of Twitter over Decentralized-Federated F(L)OSS feeds in their Matrix room.

uforia reports that:

in the koma project, the desktop client continuum now does a full sync when the user account doesn't seem to have joined any chat rooms, this way, it can recover from some disk IO errors, or more commonly, unclean shutdowns. A ca-certificates issue with Java 11 on Debian stable was found while running a bot on a headless server, more details and the solution is in the README

vabd reports that:

Our first specs proposal of 2019 just landed in the form of SCS #16, which specifies the data/event structure for trust authorities. This is a big step as TAs play a key role in Informo's trust/reputation system!

In the meantime, we've also opened SCS #19, which proposes a rework of the specs' introduction with the idea to give newcomers a more accessible and immediate way to figure out what Informo is about, and give them some starting points so they can dive deeper into it if interested. It's a rather small one and we'd love people to give it a look so we can aim for the most newcomer-friendly version possible

We've also just opened SCS #21 which specifies a way for a source to change the Matrix user it uses to publish articles (e.g. if it was previously using a server managed by non trustworthy people). As with all of our proposals introducing changes in behaviour, it's open for people to share their comments on it for the next 7 days.

Maximus reports that:

The first alpha release for mxisd v1.3.0 has been released with already major performance improvements. Early testing and reporting about success/failure would be very much appreciated as v1.3.0 will break backward compatibility. We have been running it on our own servers for about a week now and feels really good and stable.

Friedger Müffke reports that:

I just launched OI Chat, a matrix service dedicated to Blockstack users (https://www.producthunt.com/posts/oi-chat).

It is a home server that does not rely on any passwords but on cryptography and user-owned storage.

OI Chat uses one-time logins to verify the ownership of a username that can only be created by the user if they control the blockstack account.

...and that's all this week, folks!  Your normal hand-crafted artisanal benpa confectionery will be back next week.

After releasing Synapse v0.34.1, we have become aware of a security vulnerability affecting all previous versions (CVE-2019-5885). v0.34.1 closed the vulnerability but, in some cases, caused users to be logged out of their clients, so we do not recommend v0.34.1 for production use.

Today we release two mitigating versions v0.34.0.1 and v0.34.1.1. Both versions close the vulnerability and will not cause users to be logged out. All installations should be upgraded to one or other immediately.

• Admins who would otherwise upgrade to v0.34.1 (or those that have already done so) should upgrade to v0.34.1.1.
• Admins on v0.34.0, who do not wish to bring in new non-security related behaviour, should upgrade to v0.34.0.1.

You can get the new updates for v0.34.0.1 and v0.34.1.1 here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. See also our Synapse installation guide page.

We will publish more details of the vulnerability once admins have had a chance to upgrade. To our knowledge the vulnerability has not been exploited in the wild.

Many thanks for your patience, we are moving ever closer to Synapse reaching v1.0, and fixes like this one edge us ever closer.

Thanks also to the package maintainers who have coordinated with us to ensure distro packages are available for a speedy upgrade!

• Synapse v0.34.0.1 Changelog
• Synapse v0.34.1.1 Changelog

Matrix's reference homeserver, Synapse, is written in Python and uses the Twisted networking framework to power its bitslinging across the Internet. The Python version used has been strictly Python 2.7, the last supported version of Python 2, but as of this week that changes! Since Twisted and our other upstream dependencies now support the newest version of Python, Python 3, we are now able to finish the jump and port Synapse to use it by default. The port has been done in a backwards compatible way, written in a subset of Python that is usable in both Python 2 and Python 3, meaning your existing Synapse installs still work on Python 2, while preparing us for a Python 3 future.

Why port?

Porting Synapse to Python 3 prepares Synapse for a post-Python 2 world, currently scheduled for 2020. After the 1st of January in 2020, Python 2 will no longer be supported by the core Python developers and no bugfixes (even critical security ones) will be issued. As the security of software depends very much on the runtime and libraries it is running on top of, this means that by then all Python 2 software in use should have moved to Python 3 or other runtimes.

The Python 3 port has benefits other than just preparing for the End of Life of Python 2.7. Successive versions of Python 3 have improved the standard library, provided newer and clearer syntax for asynchronous code, added opt-in static typing to reduce bugs, and contained incremental performance and memory management improvements. These features, once Synapse stops supporting Python 2, can then be fully utilised to make Synapse's codebase clearer and more performant. One bonus that we get immediately, though, is Python 3's memory compaction of Unicode strings. Rather than storing as UCS-2/UTF-16 or UCS-4/UTF-32, it will instead store it in the smallest possible representation giving a 50%-75% memory improvement for strings only containing Latin-1 characters, such as nearly all dictionary keys, hashes, IDs, and a large proportion of messages being processed from English speaking countries. Non-English text will also see a memory improvement, as it can be commonly stored in only two bytes instead of the four in a UCS-4 “wide” Python 2 build.

Editor's note: If you were wondering how this fits in with Dendrite (the next-gen golang homeserver): our plan is to use Synapse as the reference homeserver for all the current work going on with landing a 1.0 release of the Matrix spec: it makes no sense to try to iterate and converge on 1.0 on both Synapse and Dendrite in parallel. In order to prove that the 1.0 spec is indeed fit for purpose we then also need Synapse to exit beta and hit a 1.0 too, hence the investment to get it there. It's worth noting that over the last year we've been plugging away solidly improving Synapse in general (especially given the increasing number of high-profile deployments out there), so we're committed to getting Synapse to a formal production grade release and supporting it in the long term. Meanwhile, Dendrite development is still progressing - currently acting as a place to experiment with more radical blue-sky architectural changes, especially in low-footprint or even clientside homeservers. We expect it to catch up with Synapse once 1.0 is out the door; and meanwhile Synapse is increasingly benefiting from performance work inspired by Dendrite.

When will the port be released?

The port is has been released in a “production ready” form in Synapse 0.34.0, supporting Python 3.5, 3.6, and 3.7. This will work on installations with and without workers.

What's it like in the real world?

Beta testers of the Python 3 port have reported lower memory usage, including lower memory “spikes” and slower memory growth. You can see this demonstrated on matrix.org:

See 10/15, ~20:00 for the Python 3 migration. This is on some of the Synchrotrons on matrix.org.

See ~11/8 for the Python 3 migration. This is on the Synapse master on matrix.org.

We have also noticed some better CPU utilisation:

See 21:30 for the migration of federation reader 1, and 21:55 for the others. The federation reader is a particular pathological case, where the replacement of lists with iterators internally on Python 3 has given us some big boosts.

See 10/15, 4:00.The CPU utilisation has gone down on synchrotron 1 after the Python 3 migration, but not as dramatically as the federation reader. Synchrotron 3 was migrated a few days later.

As some extra data-points, my personal HS consumes about 300MB now at initial start, and grows to approximately 800MB -- under Python 2 the growth would be near-immediate to roughly 1.4GB.

Where to from here?

Python 2 is still a supported platform for running Synapse for the time being. We plan on ending mainstream support on 1st April 2019, where upon Python 3.5+ will be the only officially supported platform. Additionally, we will give notice ahead of time once we are ready to remove Python 2.7 compatibility from the codebase (which will be no sooner than 1st April). Although slightly inconvenient, we hope that this gives our users and integrators adequate time to migrate, whilst giving us the flexibility to use modern Python features and make Synapse a better piece of software to help power the Matrix community.

How can I try it?

The port is compatible with existing homeservers and configurations, so if you install Synapse inside a Python 3 virtualenv, you can run it from there. Of course, this differs based on your installation method, operating system, and what version of Python 3 you wish to use. Full upgrade notes live here but if you're having problems or want to discuss specific packagings of Synapse please come ask in #synapse:matrix.org.

Thanks

Many thanks go to fellow Synapse developers Erik and Rich for code review, as well as community contributors such as notafile and krombel for laying the foundations many months ago allowing this port to happen. Without them, this wouldn't have happened.

Happy Matrixing,

Amber Brown (hawkowl)

Folks this is a big day for us at Matrix Towers, because today we release 0.34.0.

The big news for 0.34.0 is that we now recommend Python 3 for production use and have been running matrix.org under Python 3 for the past month.

Performance improvements have been marked, in some contexts we have seen 50% reductions in RAM and CPU usage. Here are some illustrative graphs to get you going but look out for a dedicated post delving into much more detail on the port. You can also see a Matrix Live interview with the project lead Amber (hawkowl) here.

Matrix.org federation reader workers, the big drops signify roll over to python 3

Synapse master on matrix.org, again the drop in RAM signifies the roll over to python 3

Many thanks to Amber for leading the effort, Rich and Erik for providing support as well as Notafile and Krombel from the community for pushing this effort right from the early days of the project.

If that wasn't enough, 0.34.0 also all the usual bug fixes and perf improvements. In particular the media repository now no longer fails to decode UTF-8 filenames when downloading remote media and auto joining rooms now work on servers with consent requirements enabled.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.

In particular, if you want to run Synapse 0.34.0 on Python 3 take a look at the upgrade notes.

Synapse 0.34.0 changelog

Synapse 0.34.0 is the first release to fully support Python 3. Synapse will now run on Python versions 3.5 or 3.6 (as well as 2.7). Support for Python 3.7 remains experimental.

We recommend upgrading to Python 3, but make sure to read the upgrade notes when doing so.

Features

• Add 'sandbox' to CSP for media reprository (#4284)
• Make the new landing page prettier. (#4294)
• Fix deleting E2E room keys when using old SQLite versions. (#4295)
• Add a welcome page for the client API port. Credit to @krombel! (#4289)
• Remove Matrix console from the default distribution (#4290)
• Add option to track MAU stats (but not limit people) (#3830)
• Add an option to enable recording IPs for appservice users (#3831)
• Rename login type m.login.cas to m.login.sso (#4220)
• Add an option to disable search for homeservers that may not be interested in it. (#4230)

Bugfixes

• Pushrules can now again be made with non-ASCII rule IDs. (#4165)
• The media repository now no longer fails to decode UTF-8 filenames when downloading remote media. (#4176)
• URL previews now correctly decode non-UTF-8 text if the header contains a <meta http-equiv="Content-Type" header. (#4183)
• Fix an issue where public consent URLs had two slashes. (#4192)
• Fallback auth now accepts the session parameter on Python 3. (#4197)
• Remove riot.im from the list of trusted Identity Servers in the default configuration (#4207)
• fix start up failure when mau_limit_reserved_threepids set and db is postgres (#4211)
• Fix auto join failures for servers that require user consent (#4223)
• Fix exception caused by non-ascii event IDs (#4241)
• Pushers can now be unsubscribed from on Python 3. (#4250)
• Fix UnicodeDecodeError when postgres is configured to give non-English errors (#4253)

Internal Changes

• Debian packages utilising a virtualenv with bundled dependencies can now be built. (#4212)
• Disable pager when running git-show in CI (#4291)
• A coveragerc file has been added. (#4180)
• Add a GitHub pull request template and add multiple issue templates (#4182)
• Update README to reflect the fact that #1491 is fixed (#4188)
• Run the AS senders as background processes to fix warnings (#4189)
• Add some diagnostics to the tests to detect logcontext problems (#4190)
• Add missing jpeg package prerequisite for OpenBSD in README. (#4193)
• Add a note saying you need to manually reclaim disk space after using the Purge History API (#4200)
• More logcontext checking in unittests (#4205)
• Ignore __pycache__ directories in the database schema folder (#4214)
• Add note to UPGRADE.rst about removing riot.im from list of trusted identity servers (#4224)
• Added automated coverage reporting to CI. (#4225)
• Garbage-collect after each unit test to fix logcontext leaks (#4227)
• add more detail to logging regarding "More than one row matched" error (#4234)
• Drop sent_transactions table (#4244)
• Add a basic .editorconfig (#4257)
• Update README.rst and UPGRADE.rst for Python 3. (#4260)
• Remove obsolete verbose and log_file settings from homeserver.yaml for Docker image. (#4261)