We are happy to launch The Matrix Conference on Oct 15-18 in Strasbourg, France. Learn more about it, buy a ticket!

Critical Security Update: Synapse 0.34.0.1/Synapse 0.34.1.1

2019-01-10 — Releases, SecurityNeil Johnson

After releasing Synapse v0.34.1, we have become aware of a security vulnerability affecting all previous versions (CVE-2019-5885). v0.34.1 closed the vulnerability but, in some cases, caused users to be logged out of their clients, so we do not recommend v0.34.1 for production use.

Today we release two mitigating versions v0.34.0.1 and v0.34.1.1. Both versions close the vulnerability and will not cause users to be logged out. All installations should be upgraded to one or other immediately.

  • Admins who would otherwise upgrade to v0.34.1 (or those that have already done so) should upgrade to v0.34.1.1.
  • Admins on v0.34.0, who do not wish to bring in new non-security related behaviour, should upgrade to v0.34.0.1.

You can get the new updates for v0.34.0.1 and v0.34.1.1 here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. See also our Synapse installation guide page.

We will publish more details of the vulnerability once admins have had a chance to upgrade. To our knowledge the vulnerability has not been exploited in the wild.

Many thanks for your patience, we are moving ever closer to Synapse reaching v1.0, and fixes like this one edge us ever closer.

Thanks also to the package maintainers who have coordinated with us to ensure distro packages are available for a speedy upgrade!

This Week in Matrix 2019-01-04

2019-01-04 — This Week in MatrixBen Parsons

🔗Welcome to 2019

It's been 2019 for several days now, plenty of time to get used to it! Let's get started with the first TWIM of the year!

🔗Matrix Live S3E09 from 35C3

Several Matrix-ers attended 35C3 in Leipzig last month, you can check out Matrix Live recorded from the conference below (also includes some screenshots and other clips of the event), and also watch a talk given by this author titled Matrix, the current status and year to date.

🔗Welcoming Jason Robinson to New Vector

Decentralisation lover and Python fan Jason Robinson joins New Vector!

one of my dreams of working for a company that is a driver and leader in open source and open standards is coming true

🔗matrix-appservice-purple

Hey Half-Shot, what bridges have you worked on this week?

matrix-appservice-purple got soft launched on matrix.org and is happily bridging XMPP and matrix communities together. I am on full bug and feature fixing duty for it and the consensus from both sides is that it's looking pretty awesome.
The matrix-appservice-purple bridge is coming along leaps and bounds, with formatting fixes, presence handling and speedier message delivery both ways. Also a shoutout to the XMPP community for guidling me through the XEP landscape and giving the bridge a thorough testing. :)

🔗Riot Web Experimental

Riot Web Experimental was announced in a blog post last month, and is ready to get some more testing! Note that the name is still "experimental", but to get an image of where things are going, please give it a try!

http://riot.im/experimental

🔗Riot iOS

Photo sharing via the app share extension has been fixed this week.
At the time of writing, a new Riot iOS is baking with all bug fixes made the last month.

🔗Riot Android

Riot 0.8.21 has been released on 01/02 on the PlayStore and on F-Droid.

This version contains:

  • A new notification troubleshoot screen with the possibility to run a diagnostic and to submit bug report. Feedbacks are already coming and we improve this screen incrementally to help users.
  • A new invitations counter on the group icon in the home screen
  • Other bug fixes
We are still working on push/notification reliability. Riot Play Store resources have been translated into 8 languages so far: Basque, Bulgarian, Chinese (Traditional), French, German, Hungarian, Italian and Portuguese (Brazil).

🔗matrix-client.el's many updates

alphapapa provided many updates for matrix-client.el this week, I recommend chatting in #matrix-client.el:matrix.org where the cultists Emacs users and client devs hang out.

matrix-client.el gained more room sorting options and a /priority command to set room priority. It also includes a workaround for a Google Chrome drag-and-drop bug on Linux, so now Chrome users can drag-and-drop URLs, files, and images directly into room buffers to upload them.

matrix-client.el gained a new notifications-buffer feature that shows notifications from multiple rooms in a single list, allowing you to easily monitor multiple rooms at once and jump to events in them.
e.g. I can see messages from #matrix and #twim in the same window, and reply to messages in both rooms from the same prompt

🔗koma project: now continuum-desktop (client) and koma library

uforia from koma announced that the client formerly known as koma is now continuum-desktop:

in the koma project, the desktop client now has continuous integration and prebuilt packages for Mac and Linux; and you can click on image messages to zoom in. A simple weather bot is created reusing the same implementation of matrix client api. Send it the name of a city, and it will fetch the current weather using openweathermap

🔗mxisd

Max, creator of mxisd, announced:

All versions of mxisd dropped support for Riot v0.17.8, introducing a bug affecting many of its features. Any new release integrating this PR will also be dropped of support. mxisd users are strongly encouraged to roll back to v0.17.7

This design concern is noted by the riot-web team and is under investigation.

🔗Dendrite audit progress

Brendan on the progress of the Dendrite audit:

Dendrite's audit is finally coming to an end! I'm happy to say I just finished the “data collection” phase, in which I looked at everything that needs to be either fixed or implemented in Dendrite. This represents 90% of the work and around 3 weeks of full-time work. All that's left to do now is some triaging in the data (which is available here, by the way), into order to have a clear view on what's left to do in the audit. Expect a lot of new issues and a shiny project board appearing on the Dendrite repository next week ?

🔗Informo news

Informo is a project intended to enable information sharing, especially for vulnerable activists. It is enabled by Matrix. vabd, the mystery individual behind the project announced:

Our specs bot, which shouts in Matrix rooms when the state of a proposal to a specs project changes, got an upgrade: it now handles concurrency better, and can now send multiple messages if multiple matching labels are added to the proposal in the same action (before, it just wouldn't know what to do in such an event and would fail silently).

🔗msc-chatbot

The MSC process is the formal process by which changes are submitted to become part of the Matrix specification. anoa has been working on a bot to help with the process:

msc-chatbot now exists. It has commands that let you view the status of current MSCs, as well as a daily summary of MSCs to keep people up to date.

🔗Matrix-Minecraft bridge

TravisR, as if he has time to be working on such things, has announced the revival of his Matrix-Minecraft bridge:

I've brought my Matrix<-->Minecraft bridge back to life in the form of a Bukkit plugin. It's still in the very early stages of development and requires you to compile it yourself to get it, but it is a thing. Check it out on GitHub: https://github.com/turt2live/matrix-minecraft
It'll be designed to work as a public hosted bridge, so someone could use t2bot.io to bridge their minecraft server (for example)

🔗stickerpack dimension migration tool

Dandellion has created a tool for stickerpack creators:

I've finished https://dali99.github.io/stickerpack-dimension-migration-tool/ for a niceish way to make migrations files for custom stickerpacks in dimension

🔗XMPP (as Jabber) turns 20

As noted in The XMPP Newsletter:

Today is Jabber's 20th anniversary! Jabber would later be standardized and renamed to XMPP.

On this subject, it's always worth thinking about the importance of openness and interoperability in messaging. This recent article in Linux Journal is a reminder of the need to avoid proprietary vendor lock in, and mentions both XMPP and Matrix.

🔗We'll meet again…

Come chat in #twim:matrix.org with your Matrix news to be featured in this post. Next Friday there will be another weekly edition, but before then expect to see an edition to the effect "benpa's best-of-the-community 2018".

This Week in Matrix 2018-12-28

2018-12-28 — This Week in MatrixBen Parsons

🔗Welcome to the last TWIM of the year

It's the 28th of December as I write this, and I hope you had a good year. Long as it is, I recommend reading Matthew's write-up of the year, as it covers a lot of ground!

Many of the core team have been out of the office this week, but there are still plenty of updates to share from the Matrix ecosystem!

🔗mxisd v1.2.2

Max released mxisd v1.2.2:

mxisd had a Holiday-special release: v1.2.2 before v1.3.0.

This release introduces two new big features:

  • Username login rewriting via 3PID to allow advanced flows, like bypassing the synapse restriction of having numerical usernames for non-guest users
  • Support for multiple Base DNs for LDAP backends
Work has started on v1.3.0 so this is definitely the last release before a non-backward compatible release.

🔗Scylla

VaNilLa has started a new client, a web app build in Elm:

Hi all! I am working on a Matrix client in Elm, and I was recommended to share it here: Scylla

🔗koma-desktop

druig:

koma-desktop is updated to JavaFx 11 and installation is simplified. Dependencies, including native modules can be packaged into one single file, which only needs to be downloaded and run. Java Runtime 11 is the only runtime dependency. Now it's just cross-compilation that needs to be set up before packaged releases can be provided for Mac, Windows, and Linux users.

🔗Prometheus Alertmanager bot for Matrix

Jason Robinson:

I started work on a Prometheus Alertmanager bot for Matrix. The basic idea is that Alertmanager can send webhook alert events to the bot which will then send the formatted events to configured rooms based on the alert receiver. It works, but is still early work in progress. See code and info here: https://git.feneas.org/jaywink/matrix-alertmanager.

Also mirrored on GitHub: https://github.com/jaywink/matrix-alertmanager

🔗Fractal 4.0

Alexandre Franke:

As expected the Fractal team released 4.0 and is already hard at work on the next micro version. We recommend getting it from Flathub like we usually do.

From the release notes:

New features:

  • Enhanced history view with adaptive layout, day divider
  • Reorganised headerbar, app menu merged with user menu
  • Larger display of emoji-only messages
  • Some performance improvements
  • Opening a room jumps to first unread message
Bugfixes:
  • More reliable notifications
  • Fixed display bug for avatars
Under the hood:
  • Large code refactor
  • Logging infrastructure
  • Continuous integration
  • More informative build output

🔗maubot

tulir working on maubot:

The new command handling system in maubot is ready. The new system should be much nicer to use when developing plugins.

Previously maubot had a system that was designed after the improved bot support spec proposal, but it wasn't very nice or pythonic. If/when the proposal or something similar goes through, I'll probably add support for it in the new command handling system.

Next I'll make some developer docs so that other people could actually make their own plugins.

The code is at https://github.com/maubot/maubot and you can ask about maubot in #maubot:maunium.net.

🔗matrix-client.el new features

alphapapa:

Emacs makes it so easy to integrate things. Now you can send org-mode syntax messages with the /org command in matrix-client.el.

Tab-completion of usernames and IDs was added to matrix-client.el.

🔗matrix-to GitHub app

t3chguy has come out of hiding to announce:

https://github.com/apps/matrix-to is a Github App which makes use of their shiny and new Content Attachments API/Webhook. When a matrix.to or view.matrix.org URL is used this app is activated. It adds a little snippet with the Room Title and Topic (if the room is peekable from matrix.org). In future it'll work for event permalinks, but currently there is no support for peeking context/event in Matrix API.

Example can be seen at https://github.com/matrix-org/matrix.to/issues/52#issuecomment-449878490. Idea courtesy of TravisR.

🔗See you next year

So there you have it.

I'm at 35c3 with some known characters from the Matrix world (as well as 15,000 others.) If you're here too, come visit us in our assembly, and also make sure to come to Dijkstra tomorrow to watch me present a look back at on the last year: https://fahrplan.events.ccc.de/congress/2018/Fahrplan/events/9400.html. We have recorded a message for Matrix Live from 35c3, but will post tomorrow with some more footage from the event.

Otherwise, see you next year Matrix fan!

The 2018 Matrix Holiday Special!

2018-12-25 — General, Holiday SpecialMatthew Hodgson

Hi all,

It's that time again where we break out the mince pies and brandy butter (at least for those of us in the UK) and look back on the year to see how far Matrix has come, as well as anticipate what 2019 may bring!

🔗Overview

It's fair to say that 2018 has been a pretty crazy year.  We have had one overriding goal: to take the funding we received in January, stabilise and freeze the protocol and get it and the reference implementations out of beta and to a 1.0 - to provide a genuinely open and decentralised mainstream alternative to the likes of Slack, Discord, WhatsApp etc.  What's so crazy about that, you might ask?

Well, in parallel with this we've also seen adoption of Matrix accelerating ahead of our dev plan at an unprecedented speed: with France selecting Matrix to power the communication infrastructure of its whole public sector - first trialling over the summer, and now confirmed for full roll-out as of a few weeks ago.  Meanwhile there are several other similar-sized projects on the horizon which we can't talk about yet.  We've had the growing pains of establishing New Vector as a startup in order to hire the core team and support these projects.  We've launched Modular to provide professional-quality SaaS Matrix hosting for the wider community and help fund the team.  And most importantly, we've also been establishing the non-profit Matrix.org Foundation to formalise the open governance of the Matrix protocol and protect and isolate it from any of the for-profit work.

Continue reading…

Porting Synapse to Python 3

2018-12-21 — TechNeil Johnson

Matrix's reference homeserver, Synapse, is written in Python and uses the Twisted networking framework to power its bitslinging across the Internet. The Python version used has been strictly Python 2.7, the last supported version of Python 2, but as of this week that changes! Since Twisted and our other upstream dependencies now support the newest version of Python, Python 3, we are now able to finish the jump and port Synapse to use it by default. The port has been done in a backwards compatible way, written in a subset of Python that is usable in both Python 2 and Python 3, meaning your existing Synapse installs still work on Python 2, while preparing us for a Python 3 future.

🔗Why port?

Porting Synapse to Python 3 prepares Synapse for a post-Python 2 world, currently scheduled for 2020. After the 1st of January in 2020, Python 2 will no longer be supported by the core Python developers and no bugfixes (even critical security ones) will be issued. As the security of software depends very much on the runtime and libraries it is running on top of, this means that by then all Python 2 software in use should have moved to Python 3 or other runtimes.

The Python 3 port has benefits other than just preparing for the End of Life of Python 2.7. Successive versions of Python 3 have improved the standard library, provided newer and clearer syntax for asynchronous code, added opt-in static typing to reduce bugs, and contained incremental performance and memory management improvements. These features, once Synapse stops supporting Python 2, can then be fully utilised to make Synapse's codebase clearer and more performant. One bonus that we get immediately, though, is Python 3's memory compaction of Unicode strings. Rather than storing as UCS-2/UTF-16 or UCS-4/UTF-32, it will instead store it in the smallest possible representation giving a 50%-75% memory improvement for strings only containing Latin-1 characters, such as nearly all dictionary keys, hashes, IDs, and a large proportion of messages being processed from English speaking countries. Non-English text will also see a memory improvement, as it can be commonly stored in only two bytes instead of the four in a UCS-4 “wide” Python 2 build.

Editor's note: If you were wondering how this fits in with Dendrite (the next-gen golang homeserver): our plan is to use Synapse as the reference homeserver for all the current work going on with landing a 1.0 release of the Matrix spec: it makes no sense to try to iterate and converge on 1.0 on both Synapse and Dendrite in parallel. In order to prove that the 1.0 spec is indeed fit for purpose we then also need Synapse to exit beta and hit a 1.0 too, hence the investment to get it there. It's worth noting that over the last year we've been plugging away solidly improving Synapse in general (especially given the increasing number of high-profile deployments out there), so we're committed to getting Synapse to a formal production grade release and supporting it in the long term. Meanwhile, Dendrite development is still progressing - currently acting as a place to experiment with more radical blue-sky architectural changes, especially in low-footprint or even clientside homeservers. We expect it to catch up with Synapse once 1.0 is out the door; and meanwhile Synapse is increasingly benefiting from performance work inspired by Dendrite.

🔗When will the port be released?

The port is has been released in a “production ready” form in Synapse 0.34.0, supporting Python 3.5, 3.6, and 3.7. This will work on installations with and without workers.

🔗What's it like in the real world?

Beta testers of the Python 3 port have reported lower memory usage, including lower memory “spikes” and slower memory growth. You can see this demonstrated on matrix.org:

See 10/15, ~20:00 for the Python 3 migration. This is on some of the Synchrotrons on matrix.org.

See ~11/8 for the Python 3 migration. This is on the Synapse master on matrix.org.

We have also noticed some better CPU utilisation:

See 21:30 for the migration of federation reader 1, and 21:55 for the others. The federation reader is a particular pathological case, where the replacement of lists with iterators internally on Python 3 has given us some big boosts.

See 10/15, 4:00.The CPU utilisation has gone down on synchrotron 1 after the Python 3 migration, but not as dramatically as the federation reader. Synchrotron 3 was migrated a few days later.

As some extra data-points, my personal HS consumes about 300MB now at initial start, and grows to approximately 800MB -- under Python 2 the growth would be near-immediate to roughly 1.4GB.

🔗Where to from here?

Python 2 is still a supported platform for running Synapse for the time being. We plan on ending mainstream support on 1st April 2019, where upon Python 3.5+ will be the only officially supported platform. Additionally, we will give notice ahead of time once we are ready to remove Python 2.7 compatibility from the codebase (which will be no sooner than 1st April). Although slightly inconvenient, we hope that this gives our users and integrators adequate time to migrate, whilst giving us the flexibility to use modern Python features and make Synapse a better piece of software to help power the Matrix community.

🔗How can I try it?

The port is compatible with existing homeservers and configurations, so if you install Synapse inside a Python 3 virtualenv, you can run it from there. Of course, this differs based on your installation method, operating system, and what version of Python 3 you wish to use. Full upgrade notes live here but if you're having problems or want to discuss specific packagings of Synapse please come ask in #synapse:matrix.org.

🔗Thanks

Many thanks go to fellow Synapse developers Erik and Rich for code review, as well as community contributors such as notafile and krombel for laying the foundations many months ago allowing this port to happen. Without them, this wouldn't have happened.

Happy Matrixing,

Amber Brown (hawkowl)

This Week in Matrix 2018-12-21

2018-12-21 — This Week in MatrixBen Parsons

🔗Matrix Live: Half-Shot talks bridges, and working on libpurple bridging to Matrix

You may have seen that Half-Shot been working fearlessly and tirelessly on bridges for many, many months. In this episode of Matrix Live Half-Shot chats with Matthew about his work, progress and achievements, with a focus on recent matrix-appservice-purple and XMPP work. Audio is not amazing, but worth listening to get acquainted with recent work.

As a note from Half-Shot:

The matrix-appservice-purple bridge gained a XMPP specific backend for better performance when you want to do just xmpp bridging. It's rather quick right now, and needs dogfooding.

Chat in #purple-bridge:half-shot.uk!

🔗Matrix badges on shields.io

TWIM

Brendan:

A couple of weeks ago, fr1kin PR'd a nice Matrix badge to the Shields project that tells you how many users are in a public room. There were a few issues with it so I PR'd some changes to make the badge more usable, and it's now merged and live (as of yesterday evening), with examples available here! ?
For instance, here's a badge for TWIM: https://img.shields.io/matrix/twim:matrix.org.svg ?

🔗Spec update

New MSC for cross-signing, which has different (hopefully better) semantics from the previous cross-signing MSC.

🔗matrix-bot-sdk support for application services

TravisR, author of matrix-bot-sdk:

matrix-bot-sdk has received early support for application services. Similar to the official matrix-js-sdk, the bot-sdk uses an Intent-based model for making bridges easier to write. Check out the simple example here for more information on how it works.

🔗koma-library

druig continues work on their JavaFX/kotlin client project:

The matrix client API implementation in koma is being extracted into a new repo, which is going to be a lightweight library for Kotlin.

🔗Seaglass now available on homebrew

Aaron Raimist reports that Seaglass is available on homebrew for macOS:

Installing Seaglass is now easier than ever. If you already use Homebrew to manage other packages, you can now install Seaglass with a simple brew cask install seaglass.

If you don't use Homebrew, you can still download Seaglass from GitHub.

🔗matrix-client.el is reborn!

alphapapa reports from a team who have forked and are maintaining a Matrix client for Emacs:

Many additions and improvements to matrix-client.el (https://github.com/jgkamat/matrix-client-el) recently, including a "standalone client" mode that launches Emacs to look like this.

Chat in #matrix-client-el:matrix.org.

🔗matrix-docker-ansible-deploy

Slavi:

the matrix-docker-ansible-deploy playbook has received some bugfixes and improvements lately. Most importantly, it's now running the freshly-released Synapse 0.34.0 under Python 3, so memory usage should be much better.

🔗libQMatrixClient/Quaternion

kitsune:

I started work on matrix: scheme support in libQMatrixClient/Quaternion. Expect more news on this around New Year.

🔗Fractal road to 4.0

Fresh from their hackfest in Seville last week, Alexandre Franke reports:

The Fractal crew spent the week chasing last minutes bugs and made two beta releases (3.99.0 and 3.99.1) in preparation for the big new stable release, 4.0, which is right around the corner.

🔗New Rooms for Space Launches and Aviation

Aaron Raimist has been creating some new rooms:

#space:im.kabi.tk for anyone interested in space, rocket launches, satellites, etc.
Are you wondering what NASA's new Mars Rover is doing? Maybe you live on the west coast of the United States, and you saw that meteor on Wednesday night that came within minutes of a scheduled rocket launch and just after three astronauts left the space station. If any of that sounds interesting to you, feel free to join the room.
A Matrix bot is being tested to send updates about upcoming rocket launches to the room.

#aviation:matrix.org for anyone interested in aviation. Whether you are a pilot, someone who visits an airshow once in a while, or if https://www.youtube.com/watch?v=CXv1j3GbgLk piqued your interest, come join.

🔗t2bot.io upgrade

TravisR:

Just a head's up that I've increased the storage capacity of the database. With current projections, the server should be good for another year or two.

The database is also 1ms closer to Synapse and has a faster CPU in it. It probably won't make a dent in speed on federation, but it is a step forward.

I'll be rolling out python 3 to the homeserver this week too, which should help a little bit.

https://riot.im/experimental/#/room/#help:t2bot.io/$154511414912clELm:t2l.io

🔗FluffyChat 8.0 RC announced

Krille and his Ubuntu Touch fans are looking forward to FluffyChat 8.0:

Hey guys, in order to release the FluffyChat 8.0 Christmas Edition, the FluffyChat 8.0 release candidate is ready for you! :-)
Also the weblate translations are ready: https://hosted.weblate.org/projects/fluffychat/

🔗Riot iOS

From the Riot iOS team:

  • We have fixed and improved some e2e stuff.
  • Back to reskin. We start to implement e2e backup screens.

🔗Riot Android

From the Riot Android team:

  • New screen to troubleshoot notification issue has been merged on /develop.
  • Splitting current Android SDK to separate crypto part is on it's way. We're doing this in order to be able to integrate crypto faster in the Riot reboot.

🔗Synapse

Neil from Synapse reports:

We released 0.34.0! This release recommends Python 3 for production and brings with it huge performance improvements. If you've been putting upgrading off upgrading your Synapse, now is the time to do so. For more details here is a post that explains what you should expect and a recent Matrix Live interviews Amber (hawkowl) on the subject.

Aside from that we are working furiously towards federation R0 and have a bunch of MSCs to get us ever closer. You can track our progress here.

andrewsh notes that 0.34.0 is also available in the Debian repos:

Synapse 0.34.0~rc2 in Debian since Tuesday, 0.34.0 uploaded today; both use Python 3 only

🔗Dendrite

Brendan reports:

Dendrite's internal audit is progressing very well and is getting very close to its end.
What's left to do for me is check the implementation status of a few Matrix features, and translate those into tagged GitHub issues so that everyone can have as clear of a view as possible on what's left to work on.
I'm on holiday all of next week, but hopefully will have some good news about that the following week.

🔗That really is it for now

Did you get to the end? What was your favourite section? Come tell us in #twim:matrix.org! Do you have your own update you'd like to add? Same place, come chat in #twim:matrix.org.

Next week is various things. It's Christmas, which means there will be more hacking and coding happening than usual I expect! It's also 35c3, which I will be attending, and might affect scheduling next week. Stay tuned in #twim:matrix.org for news, and come join us in #matrix-35c3:matrix.org if you'll be there and want to meet up!

Merry Christmas!

Synapse 0.34.0 released!

2018-12-20 — ReleasesNeil Johnson

Folks this is a big day for us at Matrix Towers, because today we release 0.34.0.

The big news for 0.34.0 is that we now recommend Python 3 for production use and have been running matrix.org under Python 3 for the past month.

Performance improvements have been marked, in some contexts we have seen 50% reductions in RAM and CPU usage. Here are some illustrative graphs to get you going but look out for a dedicated post delving into much more detail on the port. You can also see a Matrix Live interview with the project lead Amber (hawkowl) here.

Matrix.org federation reader workers, the big drops signify roll over to python 3

Synapse master on matrix.org, again the drop in RAM signifies the roll over to python 3

Many thanks to Amber for leading the effort, Rich and Erik for providing support as well as Notafile and Krombel from the community for pushing this effort right from the early days of the project.

If that wasn't enough, 0.34.0 also all the usual bug fixes and perf improvements. In particular the media repository now no longer fails to decode UTF-8 filenames when downloading remote media and auto joining rooms now work on servers with consent requirements enabled.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.

In particular, if you want to run Synapse 0.34.0 on Python 3 take a look at the upgrade notes.

🔗Synapse 0.34.0 changelog

Synapse 0.34.0 is the first release to fully support Python 3. Synapse will now run on Python versions 3.5 or 3.6 (as well as 2.7). Support for Python 3.7 remains experimental.

We recommend upgrading to Python 3, but make sure to read the upgrade notes when doing so.

🔗Features

  • Add 'sandbox' to CSP for media reprository (#4284)
  • Make the new landing page prettier. (#4294)
  • Fix deleting E2E room keys when using old SQLite versions. (#4295)
  • Add a welcome page for the client API port. Credit to @krombel! (#4289)
  • Remove Matrix console from the default distribution (#4290)
  • Add option to track MAU stats (but not limit people) (#3830)
  • Add an option to enable recording IPs for appservice users (#3831)
  • Rename login type m.login.cas to m.login.sso (#4220)
  • Add an option to disable search for homeservers that may not be interested in it. (#4230)

🔗Bugfixes

  • Pushrules can now again be made with non-ASCII rule IDs. (#4165)
  • The media repository now no longer fails to decode UTF-8 filenames when downloading remote media. (#4176)
  • URL previews now correctly decode non-UTF-8 text if the header contains a <meta http-equiv="Content-Type" header. (#4183)
  • Fix an issue where public consent URLs had two slashes. (#4192)
  • Fallback auth now accepts the session parameter on Python 3. (#4197)
  • Remove riot.im from the list of trusted Identity Servers in the default configuration (#4207)
  • fix start up failure when mau_limit_reserved_threepids set and db is postgres (#4211)
  • Fix auto join failures for servers that require user consent (#4223)
  • Fix exception caused by non-ascii event IDs (#4241)
  • Pushers can now be unsubscribed from on Python 3. (#4250)
  • Fix UnicodeDecodeError when postgres is configured to give non-English errors (#4253)

🔗Internal Changes

  • Debian packages utilising a virtualenv with bundled dependencies can now be built. (#4212)
  • Disable pager when running git-show in CI (#4291)
  • A coveragerc file has been added. (#4180)
  • Add a GitHub pull request template and add multiple issue templates (#4182)
  • Update README to reflect the fact that #1491 is fixed (#4188)
  • Run the AS senders as background processes to fix warnings (#4189)
  • Add some diagnostics to the tests to detect logcontext problems (#4190)
  • Add missing jpeg package prerequisite for OpenBSD in README. (#4193)
  • Add a note saying you need to manually reclaim disk space after using the Purge History API (#4200)
  • More logcontext checking in unittests (#4205)
  • Ignore __pycache__ directories in the database schema folder (#4214)
  • Add note to UPGRADE.rst about removing riot.im from list of trusted identity servers (#4224)
  • Added automated coverage reporting to CI. (#4225)
  • Garbage-collect after each unit test to fix logcontext leaks (#4227)
  • add more detail to logging regarding "More than one row matched" error (#4234)
  • Drop sent_transactions table (#4244)
  • Add a basic .editorconfig (#4257)
  • Update README.rst and UPGRADE.rst for Python 3. (#4260)
  • Remove obsolete verbose and log_file settings from homeserver.yaml for Docker image. (#4261)

This Week in Matrix 2018-12-14

2018-12-14 — This Week in MatrixBen Parsons

🔗Fractal Hackfest

Developers and creators of Fractal, the GNOME Matrix client, have been holding a Hackfest this week in Seville. Matthew and I caught up with them on video for Matrix Live this week, and discussed the product improvements they've been making and their plans for the next release (due next week!)

🔗Matrix Spec

🔗This Week in Ruma

We don't often get to feature news from Ruma, but this week there is an updated This Week in Ruma.

Ruma is not dead, however, and small improvements have continued over the last year. The Matrix spec has advanced quite a bit and many of the blocking issues for Ruma have been resolved. Rust's maturity is another story. async/await is still under development and this is the most significant blocking issue to progress on Ruma.

🔗maubot new tooling

tulir has been working on tooling for maubot:

maubot got a command-line tool for building plugins and managing maubot instances. I also added some server-side stuff for easy registration of accounts in the management interface using synapse shared secrets, but the UI for that isn't ready. Also, I'm planning on adding some kind of small Matrix client in the management interface for manually managing the added bot clients. That might lead to a separate library that could be used in other projects or embedded in websites.

Max with a new release of mxisd:

mxisd hits v1.2.1 with a new maintenance release, fixing bugs and regressions from v1.2.0. Updating to v1.2.1 is strongly encouraged as v1.3.x will contain breaking changes and will not be a straight-forward update like v1.x has been until now.

🔗Clients

🔗Lazy-loading lands in QMatrixClient

kitsune has reported that lazy-loading, a Matrix feature that entails only loading room member details as needed, is now in libQMatrixClient master.

reference implementation in Quaternion will follow up

Black Hat has been testing the feature in Spectral, which uses the library, and says there is a 30%-50% reduction in RAM usage at startup.

🔗Riot Web progress on /experimental and v0.17.8 released

Bruno has been working on /experimental, and I recommend taking a look at the progress there! The next version of Riot Web is closer than ever.

Improved read markers now available on /experimental, needs further tweaking though. Brought back community UX on redesign, other small improvements.

v0.17.8 was released with several bugfixes and improvements.

🔗Riot iOS

Release made with these notes:

This new version supports the consent of matrix servers terms of service (including GDPR) in the registration flow. It also contains fixes for the "Empty room" bug, the registration issue on iOS 10, etc.

Get more information:

🔗Riot Android

Release made:

This new version supports the consent of matrix servers terms of service (including GDPR) in the registration flow. Many bugfixes SDK contains KeyBackup

Get more information:

🔗Servers

🔗Synapse

A lot of focus on getting debian packages ready for python 3 - this is a blocker for our official python 3 release 0.34.0. Aside from that, finalising some outstanding state resolution behaviour (https://github.com/matrix-org/matrix-doc/pull/1693) and dusting off event ids as hashes (https://github.com/matrix-org/matrix-doc/issues/1127.)

🔗Dendrite

Brendan:

My internal audit of Dendrite is continuing, drawing a more and more precise picture of what's left to fix and implement. I aim to have it over by the end of the year, or the very early days of 2019 at the latest. Folks can track its progress through https://cloud.abolivier.bzh/index.php/s/qXi2KFjCQk2c6eG

🔗modular.im now has Extra-Large instances available

Due to demand, modular.im Hosted Homeservers now has Extra-Large instances available. If you need to service 1,000+ users on a Matrix homeserver, this is the product for you!

🔗linuxgaming.life homeserver is the number one Matrix homeserver focused on Linux gaming

swedneck has continues his work on linuxgaming.life:

I've added some bots, health monitoring, and dimension integration manager to linuxgaming.life, and made the website dark.

The homeserver is open for new registrations.

🔗That's all I know!

If you'd like see your Matrix-related project featured in this blog post, come chat to us in #twim:matrix.org, and we'll see you next week!

This Week in Matrix 2018-12-07

2018-12-07 — This Week in MatrixBen Parsons

🔗Matrix Live S03E06

In which Matthew & Amandine discuss The Matrix.org Foundation, go-live for the French Government deployment for Matrix, and some exciting random diversions into post-1.0 Matrix features which should land once once 1.0 is out the door!

🔗Purism sponsoring Fractal development

We are proudly sponsoring the work of Julian Sparber on Fractal, the @matrixdotorg client for GNOME. Read his latest update - https://t.co/q4ubjzETxC #LibreDesktop #DemandFreedom #gnome

— Purism (@Puri_sm) December 5, 2018

Purism announced they are sponsoring Fractal development, starting with Julian Sparber working last month to improve the message view. You can read about his progress here. Highlights:

Before, there was a jarring cut when new messages were loaded, but now you can just scroll upward and older messages are loaded continuously.
The part I'm most excited about is the new "new message divider". When the user opens a room they can directly start reading the conversation from the last seen message and they don't need to search for the new message divider. Not only has the UX gotten a lot better, but also the underlying code is much cleaner now.
I also spent some time on making message rendering faster. I replaced the RegExp with more efficient code, this made the rendering much faster (from ~10ms to ~ 1ms) for every single message.
In summary, all of these things improve Fractal's UX a lot and make it feel more like a modern messaging app.

🔗Synapse 0.34.0rc1

0.34.0rc1 is out - please test it! 0.34.0 will be the first to officially support python 3. We're still working on the debian packaging but 0.34.0 proper should be out next week.

🔗Dendrite

Brendan has been getting on with Dendrite development:

Started auditing Dendrite's codebase to identify what is left to implement, along with what hasn't been implemented correctly. Still a lot of work left to do on this, but it's looking promising so far.

🔗Modular

Modular is a Hosted Homeservers product, check out modular.im. This week:

Matrix -> Matrix account migration tool available at https://www.modular.im/tools/matrix-migration. It's still beta, so please test it and let us know if you experience issues.

🔗New Homeserver: linuxgaming.life

swedneck has set up a new public homeserver at linuxgaming.life! There is also a Riot installation at https://riot.linuxgaming.life, served via IPFS.

Currently working on the website (based on t2bot.io, thank you travis for making that available on github!) and making things nicer in general.
I will be adding more bots shortly. Please let me know of any issues with the HS or anything surrounding it.

🔗Informo

Informo is a project to create a specification and implementation for distributing information and news. This week they have been working on merging changes to the specification:

Work on the Informo specifications has slowed down a bit in the last couple of weeks, though since the last update we did manage to get some relatively big SCSs merged into the specs, including SCS #9 (rendered here) which specifies how information sources must publish their information through the federation, and SCS #11 (rendered here) which describes how sources must register themselves in order to be picked up by clients, and handle localisation. ?
There's still a couple of big items to take care of before we can cut a 1.0 release of the Informo specs but this is definitely a huge step towards this goal.

🔗Minecraft Bridge

Dandellion:

I started working on a Minecraft bridge pretty heavily based on Travis's old project, but using Minecraft-protocol instead of mineflayer.
You need a bot Minecraft account that can join and idle in the server, it then uses /tellraw to post messages.

🔗libQMatrixClient 0.4.1 released

libQMatrixClient, which powers Spectral and Quaternion was released by kitsune, Lazy Loading coming soon:

libQMatrixClient 0.4.1 has been released today, with small fixes in the stable branch. Meanwhile, active work is ongoing on lazy-loading support in the library, with ETA for the feature landing in master being in about a week or so.

🔗Riot Web

🔗Riot iOS

  • Reskin still ongoing
  • A new release is coming
  • Less activity because of POSS

🔗Riot Android

  • A new developer joined us to help maintaining Riot: Valere Fedronic
  • Keys backup PR in review
  • Privacy Terms acceptance in login flow in review
  • FCM issue investigation/troubleshooting by Valere
  • François Off (sick :()
  • New theme is coming soon

🔗Koma

druig has been working on Koma, a JavaFX Matrix client:

This week in koma: implementation of json library switched from runtime reflection to compile-time code generation
The matrix api is implemented manually in the project, retrofit and moshi are used to interact with the rest api.

🔗SimpleMatrix

As we mentioned in TWIM last week, MTRNord has been working on designs for SimpleMatrix, a Matrix client for Android in development. This week he has made a video showcasing the new design.

🔗Redecentralize meetup in London

Last night Half-Shot & Neil & Brendan went to see the Redecentralize folk at their meetup:

Beer! And pizza! And also Redecentralize
We chatted to a nice bunch of folks both demoing their decentralized projects and talked to the likes of scuttlebutt, BBC, IPFS and more. Was a interesting experience having around 8 minutes to quickly explain to newcomers what Matrix is and why they should use it in a speed dating format. We chatted (and demoed) bridges, new-riot and generally how it all fits together. There were a lot of very interesting people with different profiles and backgrounds, and diversified questions which lead to a lot of interesting discussions.

Note for the confused: the format of the meetup was comparable to speed dating. There is no suggestion that Matrix should be used at regular speed dating. If you do find such a use for Matrix, come tell us in #twim:matrix.org.

🔗matrix-wug, X-SAMPA to IPA bot

Last month Dandellion introduced a bot designed to convert between formats for pronunciation notation, this week he released the source code: https://github.com/dali99/matrix-wug

🔗The end of the post, and nearly the end of 2018

This week it has been winter-in-Europe kind of weather, but that's ok, it happens every year. TWIM on the other hand happens every week, so if you have something to share, and would like to share what you've been working on, come chat to us in #twim:matrix.org.

This Week in Matrix 2018-11-30

2018-11-30 — This Week in MatrixBen Parsons

🔗Porting to Python 3 - Matrix Live S3E5

Longer than usual episode of Matrix Live this week. Amber talks to Neil about the works she's been doing over the last few months to port Synapse from Python 2 to Python 3. Recommended for anyone who's been following along the progress of Synapse, or who wants a good intro on the benefits of Python 3 over Python 2.

🔗ZeroPhone

This week we learned about ZeroPhone, a fairly mature project aiming to build a working cellphone on top of a Raspberry Pi Zero.

an open-source smartphone that can be assembled for 50$ in parts. It is Linux-powered, with UI software written in Python. Currently, ZeroPhone is based on Raspberry Pi Zero, SIM800L GSM modem and 1.3" OLED screen and button interface.

&Adam was the first to bring it to our attention:

I stumbled across this blog post which mentions a matrix app in a project to create an open, hackable phone out of a raspberry pi. It's old news at this point I guess, but I don't think I've seen it mentioned here, so I thought it might be a neat thing worth featuring: https://zerophone.github.io/newsletter/ZeroPhone-Weekly-No.-16/

The creator of the Matrix app, derivmug also arrived to tell us about his work:

Hey everyone, I have been suggested to post about the Zerophone project here. I have written a basic matrix client for it. It's basically a simple phone based on a raspberry pi zero, focused on privacy, security and hackability.

The client app is built using the matrix-python-sdk, I'm also hoping to chat to derivmug as part of a future Matrix Live. Take a look at the code here.

Just to prove it's running on real hardware!

🔗Spec / Foundation

  • Work continues on MSC1730 - Mechanism for redirecting to an alternative server during login https://github.com/matrix-org/matrix-doc/pull/1730
  • Foundation: Working with the Foundation lawyers on the CIC37 to register the Foundation as a Community Interest Company proper

🔗The Federation (the-federation.info)

Jason Robinson is passionate about Federated technology, and has been working on his site at https://the-federation.info/ for some time. He recently added Matrix to the list of protocols he scrapes and lists:

Matrix (or more specifically, Synapse) servers are now visible on the The Federation info website at https://the-federation.info/matrix%7Csynapse. The website collects lists of servers for various federation protocols (like Matrix, ActivityPub, Diaspora, etc). If a server outputs usage information, it also shows some historical information for servers and projects. Currently the information available from Synapse on the site is just version, availability of registrations and (obviously) domain name. If you want to register your server, check https://the-federation.info/info for more details. Tips also welcome on how to register other Matrix server projects and how to get more data out of servers.

🔗maubot

tulir has continued work on maubot:

  • I made a rss plugin for maubot and added a fancy log viewer in the maubot management UI.
  • mautrix-telegram 0.4.0 was released, though it has no changes since the release candidate. I also copied mautrix-telegram's HTML parser into mautrix-python. Hopefully it's now generic enough so that when mautrix-telegram switches to mautrix-python, it can drop most of the built-in HTML parser.

🔗matrix-appservice-discord

Bridge-of-the-week this week is matrix-appservice-discord. Anyone in the dev room (#discord:half-shot.uk) will be very aware of this as Half-Shot and Sorunome have spent the day testing @room notifications.

Sorunome:

matrix-appservice-discord received a lot of work these past days. Not only the usual small bugfixes, but also some rather big changes: The parsers for both Discord->Matrix sending and Matrix->Discord sending were re-written completely to properly handle things (and thus, at the same time, crunch a bunch of edge-case bugs!). In addition, the long-awaited highlighting issues with @ everyone and @ here (on discord) and @ room (on matrix) were addressed. Some of these things are still in PR stage, but expected to be merged soon!

Half-Shot also provided a great screenshot of everything working well:

is just showing off having IRC+Matrix+Discord+XMPP plugged together. XMPP being the one purple is bridging

🔗matrix-docker-ansible-deploy

Slavi reports that:

matrix-docker-ansible-deploy now supports configuring the matrix-synapse-ldap3 LDAP auth password provider. Thanks to @tvo6 for contributing this!

🔗Riot iOS

  • User Agreement (GDPR) in now part of the registration flow
  • Reskin has started
  • New fixes for unexpected “Empty room”
  • We have PR templates

🔗Riot Android

  • Realm store for e2e keys has landed
  • Riot reboot: support timelines merge in DB to avoid to trash data as less as possible when navigating through permalinks
  • New settings to choose ringtone for incoming call
  • We have PR templates also

🔗SimpleMatrix

MTRNord, is working on the design for SimpleMatrix:

As I currently have less time for actual coding I am planning out the SimpleMatrix Design. You can find the Images of the Design (and a PDF file) at https://gitlab.com/Nordgedanken/simplematrixredesignimages Feel free to also join #SimpleMatrix:matrix.ffslfl.net to give feedback.

🔗Synapse

  • This week our focus has been to work through some final bugs blocking 0.34.0 which will be our first official python 3 release. Expect a RC rsn. For a sneak peak see this week's Matrix Live video at the top!

🔗Auf Wiedersehen…

That's it for this week. If you have anything to share with us, and you'd like to be included on the blog, please come talk in #twim:matrix.org!

In one month from now, many Matrix-acolytes, including myself, will be at 35c3. If you'll be there, come chat to us in #matrix-35c3:matrix.org!