The 2018 Matrix Holiday Special!

25.12.2018 00:00 — General — Matthew Hodgson

Hi all,

It's that time again where we break out the mince pies and brandy butter (at least for those of us in the UK) and look back on the year to see how far Matrix has come, as well as anticipate what 2019 may bring!

Overview

It's fair to say that 2018 has been a pretty crazy year. We have had one overriding goal: to take the funding we received in January, stabilise and freeze the protocol and get it and the reference implementations out of beta and to a 1.0 - to provide a genuinely open and decentralised mainstream alternative to the likes of Slack, Discord, WhatsApp etc. What's so crazy about that, you might ask?

Well, in parallel with this we've also seen adoption of Matrix accelerating ahead of our dev plan at an unprecedented speed: with France selecting Matrix to power the communication infrastructure of its whole public sector - first trialling over the summer, and now confirmed for full roll-out as of a few weeks ago. Meanwhile there are several other similar-sized projects on the horizon which we can't talk about yet. We've had the growing pains of establishing New Vector as a startup in order to hire the core team and support these projects. We've launched Modular to provide professional-quality SaaS Matrix hosting for the wider community and help fund the team. And most importantly, we've also been establishing the non-profit Matrix.org Foundation to formalise the open governance of the Matrix protocol and protect and isolate it from any of the for-profit work.

However: things have just about come together. Almost all the spec work for 1.0 is done and we are now aiming to get a 1.0 released in time by the end of January (in time for FOSDEM). Meanwhile Synapse has improved massively in terms of performance and stability (not least having migrated over to Python 3); Riot's spectacular redesign is now available for testing right now; E2E encryption is more stable than ever with the usability rework landing as we speak. And we've even got a full rewrite of Riot/Android in the wings.

But it's certainly been an interesting ride. Longer-term spec work has been delayed by needing to apply band-aids to mitigate abuse of the outstanding issues. Riot redesign was pushed back considerably due to prioritising Riot performance over UX. The E2E UX work has forced us to consider E2E holistically… which does not always interact well with structuring the dev work into bite-sized chunks. Dendrite has generally been idling whilst we instead pour most of our effort into getting to 1.0 on Synapse (rather than diluting 1.0 work across both projects). These tradeoffs have been hard to make, but hopefully we have chosen the correct path in the end.

Overall, as we approach 1.0, the project is looking better than ever - hopefully everyone has seen both Riot and Synapse using less RAM and being more responsive and stable, E2E being more reliable, and anyone who has played with the Riot redesign beta should agree that it is light-years ahead of yesterday's Riot and something which can genuinely surpass today's centralised proprietary incumbents. And that is unbelievably exciting :D

We'd like to thank everyone for continuing to support Matrix - especially our Patreon & Liberapay supporters, whose donations continue to be critical for helping fund the core dev team, and also those who are supporting the project indirectly by hosting homeservers with Modular. We are going to do everything humanly possible to ship 1.0 over the coming weeks, and then the sky will be the limit!

Before going into what else 2019 will hold, however, let's take the opportunity to give a bit more detail on the various core team projects which landed in 2018…

France

DINSIC (France's Ministry of Digital, IT & Comms) have been busy building out their massive cross-government Matrix deployment and custom Matrix client throughout most of the year. After the announcement in April, this started off with an initial deployment over the summer, and is now moving towards the full production rollout, as confirmed at the Paris Open Source Summit a few weeks ago by Mounir Mahjoubi, the Secretary of State of Digital. All the press coverage about this ended up in French, with the biggest writeup being at CIO Online, but the main mention of Matrix (badly translated from French) is:

Denouncing the use of tools such as WhatsApp; a practice that has become commonplace within ministerial offices, Mounir Mahjoubi announced the launch in production of Tchap, based on Matrix and Riot: an instant messaging tool that will be provided throughout the administrations. So, certainly, developing a product can have a certain cost. Integrating it too. "Free is not always cheaper but it's always more transparent," admitted the Secretary of State.

The project really shows off Matrix at its best, with up to 60 different deployments spread over different ministries and departments; multiple clusters per Ministry; end-to-end encryption enabled by default (complete with e2e-aware antivirus scanning); multiple networks for different classes of traffic; and the hope of federating with the public Matrix network once the S2S API is finalised and suitable border gateways are available. It's not really our project to talk about, but we'll try to share as much info as we can as roll-out continues.

The Matrix Specification

A major theme throughout the year has been polishing the Matrix Spec itself for its first full stable release, having had more than enough time to see which bits work in practice now and which bits need rethinking. This all kicked off with the creation of the Matrix Spec Change process back in May, which provides a formal process for reviewing and accepting contributions from anyone into the spec. Getting the balance right between agility and robustness has been quite tough here, especially pre-1.0 where we've needed to move rapidly to resolve the remaining long-lived sticking points. However, a process like this risks encouraging the classic “Perfect is the Enemy of Good” problem, as all and sundry jump in to apply their particular brand of perfectionism to the spec (and/or the process around it) and risk smothering it to death with enthusiasm. So we've ended up iterating a few times on the process and hopefully now converged on an approach which will work for 1.0 and beyond. If you haven't checked out the current proposals guide please give it a look, and feel free to marvel at all the MSCs in flight. You can also see a quick and dirty timeline of all the MSC status changes since we introduced the process, to get an idea of how it's all been progressing.

In August we had a big sprint to cut stable “r0” releases of all the APIs of the spec which had not yet reached a stable release (i.e. all apart from the Client-Server API, which has been stable since Dec 2015 - hence in part the large number of usable independent Matrix clients relative to the other bits of the ecosystem). In practice, we managed to release 3 out of the 4 remaining APIs - but needed more time to solve the remaining blocking issues with the Server-Server API. So since August (modulo operational and project distractions) we've been plugging away on the S2S API.

The main blocking issue for a stable S2S API has been State Resolution. This is the fundamental algorithm used to determine the state of a given room whenever a race or partition happens between the servers participating in it. For instance: if Alice kicks Bob on her server at the same time as Charlie ops Bob on his server, who should win? It's vital that all servers reach the same conclusion as to the state of the room, and we don't want servers to have to replicate a full copy of the room's history (which could be massive) to reach a consistent conclusion. Matrix's original state resolution algorithm dates back to the initial usable S2S implementation at the beginning of 2015 - but over time deficiencies in the algorithm became increasingly apparent. The most obvious issue is the “Hotel California” bug, where users can be spontaneously re-joined to a room they've previously left if the room's current state is merged with an older copy of the room on another server and the ‘wrong' version wins the conflict - a so-called state-reset.

After a lot of thought we ended up proposing an all new State Resolution algorithm in July 2018, nicknamed State Resolution Reloaded. This extends the original algorithm to consider the ‘auth chain' of state events when performing state resolution (i.e. the sequence of events that a given state event cites as evidence of its validity) - as well as addressing a bunch of other issues. For those wishing to understand in more detail, there's the MSC itself, the formal terse description of the algorithm now merged into the unstable S2S spec - or alternatively there's an excellent step-by-step explanation and guided example from uhoreg (warning: contains Haskell :) Or you can watch Erik and Matthew try to explain it all on Matrix Live back in July.

Since the initial proposal in July, the algorithm has been proofed out in a test jig, iterated on some more to better specify how to handle rejected events, implemented in Synapse, and is now ready to roll. The only catch is that to upgrade to it we've had to introduce the concept of room versioning, and to flush out historical issues we require you to re-create rooms to upgrade them to the new resolution algorithm. Getting the logistics in place for this is a massive pain, but we've got an approach now which should be sufficient. Clients will be free to smooth over the transition in the UI as gracefully as possible (and in fact Riot has this already hooked up). So: watch this space as v2 rooms with all-new state resolution in the coming weeks!

Otherwise, there are a bunch of other issues in the S2S API which we are still working through (e.g. changing event IDs to be hashes of event contents to avoid lying about IDs, switching to use normal X.509 certificates for federation and so resolve problems with Perspectives, etc).

Once these land and are implemented in Synapse over the coming weeks, we will be able to finally declare a 1.0!

Also on the spec side of things, it's worth noting that a lot of effort went into improving performance for clients in the form of the Lazy Loading Members MSC. This ended up consuming a lot of time over the summer as we updated Synapse and the various matrix-*-sdks (and thus Riot) to only calculate and send details to the clients about members who are currently talking in a room, whereas previously we sent the entire state of the room to the client (even including users who had left). The end result however is a 3-5x reduction in RAM on Riot, and a 3-5x speedup on initial sync. The current MSC is currently being merged as we speak into the main spec (thanks Kitsune!) for inclusion in upcoming CS API 0.5.

The Matrix.org Foundation (CIC!)

Alongside getting the open spec process up and running, we've been establishing The Matrix.org Foundation as an independent non-profit legal entity responsible for neutrally safeguarding the Matrix spec and evolution of the protocol. This kicked off in June with the “Towards Open Governance” blog post, and continued with the formal incorporation of The Matrix.org Foundation in October. Since then, we've spent a lot of time with the non-profit lawyers evolving MSC1318 into the final Articles of Association (and other guidelines) for the Foundation. This work is basically solved now; it just needs MSC1318 to be updated with the conclusions (which we're running late on, but is top of Matthew's MSC todo list).

In other news, we have confirmation that the Community Interest Company (CIC) status for The Matrix.org Foundation has been approved - this means that the CIC Regulator has independently reviewed the initial Articles of Association and approved that they indeed lock the mission of the Foundation to be non-profit and to act solely for the good of the wider online community. In practice, this means that the Foundation will be formally renamed The Matrix.org Foundation C.I.C, and provides a useful independent safeguard to ensure the Foundation remains on track.

The remaining work on the Foundation is:

Update and land MSC1318, particularly on clarifying the relationship between the legal Guardians (Directors) of the Foundation and the technical members of the core spec team, and how funds of the Foundation will be handled.
Update the Articles of Association of the Foundation based on the end result of MSC1318
Transfer any Matrix.org assets over from New Vector to the Foundation. Given Matrix's code is all open source, there isn't much in the way of assets - we're basically talking about the matrix.org domain and website itself.
Introduce the final Guardians for the Foundation.

We'll keep you posted with progress as this lands over the coming months.

Riot

2018 has been a bit of a chrysalis year for Riot: the vast majority of work has been going into the massive redesign we started in May to improve usability & cosmetics, performance, stability, and E2E encryption usability improvements. We've consciously spent most of the year feature frozen in order to polish what we already have, as we've certainly been guilty in the past of landing way too many features without necessarily applying the needed amount of UX polish.

However, as of today, we're super-excited to announce that Riot's redesign is at the point where the intrepid can start experimenting with it - in fact, internally most of the team has switched over to dogfooding (testing) the redesign as of a week or so ago. Just shut down your current copy of Riot/Web or Desktop and go to https://riot.im/experimental instead if you want to experiment (we don't recommend running both at the same time). Please note that it is still work-in-progress and there's a lot of polish still to land and some cosmetic bugs still hanging around, but it's definitely at the point of feeling better than the old app. Most importantly, please provide feedback (by hitting the lifesaver-ring button at the bottom left) to let us know how you get on. See the Riot blog for more details!

Meanwhile, on the performance and stability side of things - Lazy Loading (see above) made a massive difference to performance on all platforms; shrinking RAM usage by 3-5x and similarly speeding up launch and initial sync times. Ironically, this ended up pushing back the redesign work, but hopefully the performance improvements will have been noticeable in the interim. We also switched the entire rich text composer from using Facebook's Draft.js library to instead use Slate.js (which has generally been a massive improvement for stability and maintainability, although took ages to land - huge thanks to t3chguy for getting it over the line). Meanwhile Travis has been blitzing through a massive list of key “First Impression” bugs to ensure that as many of Riot's most glaring usability gotchas are solved.

We also welcomed ever-popular Stickers to the fold - the first instance of per-account rather than per-room widgets, which ended up requiring a lot of new infrastructure in both Riot and the underlying integration manager responsible for hosting the widgets. But judging by how popular they are, the effort seems to be worth it - and paves the way for much more exciting interactive widgets and integrations in future!

An unexpectedly large detour/distraction came in the form of GDPR back in May - we spent a month or so running around ensuring that both Riot and Matrix are GDPR compliant (including the necessary legal legwork to establish how GDPR even applies to a decentralised technology like Matrix). If you missed all that fun, you can read about it here. Hopefully we won't have to do anything like that again any time soon...

And finally: on the mobile side, much of the team has been distracted helping out France with their Matrix deployment. However, we've been plugging away on Riot/Mobile, keeping pace with the development on Riot/Web - but most excitingly, we've also found time to experiment with a complete rewrite of Riot/Android in Kotlin, using Realm and Rx (currently nicknamed Riot X). The rewrite was originally intended as a test-jig for experimenting with the redesign on mobile, but it's increasingly becoming a fully fledged Matrix client… which launches and syncs over 5x faster than today's Riot/Android. If you're particularly intrepid you should be able to run the app by checking out the project in Android Studio and hitting ‘run'. We expect the rewrite to land properly in the coming months - watch this space for progress!

E2E Encryption

One of the biggest projects this year has been to get E2E encryption out of beta and turned on by default. Now, whilst the encryption itself in Matrix has been cryptographically robust since 2016 - its usability has been minimal at best, and we'd been running around polishing the underlying implementation rather than addressing the UX. However, this year that changed, as we opened a war on about 6 concurrent battlefronts to address the remaining issues. These are:

Holistic UX. Designing a coherent, design-led UX across all of the encryption and key-management functionality. Nad (who joined Matrix as a fulltime Lead UI/UX designer in August) has been leading the charge on this - you can see a preview of the end result here. Meanwhile, Dave and Ryan are working through implementing it as we speak.
Decryption failures (UISIs). Whenever something goes wrong with E2E encryption, the symptoms are generally the same: you find yourself unable to decrypt other people's messages. We've been plugging away chasing these down - for instance, switching from localStorage to IndexedDB in Riot/Web for storing encryption state (to make it harder for multiple tabs to collide and corrupt your encryption state); providing mechanisms to unwedge Olm sessions which have got corrupted (e.g. by restoring from backup); and many others. We also added full telemetry to track the number of UISIs people are seeing in practice - and the good news is that over the course of the year their occurrence has been steadily reducing. The bad news is that there are still some edge cases left: so please, whenever you fail to decrypt a message, please make sure you submit a bug report and debug logs from *both* the sender and receiving device. The end is definitely in sight on these, however, and good news is other battlefronts will also help mitigate UISIs.
Incremental Key Backup. Previously, if you only used one device (e.g. your phone) and you lost that phone, you would lose all your E2E history unless you were in the habit of explicitly manually backing up your keys. Nowadays, we have the ability to optionally let users encrypt their keys with a passphrase (or recovery key) and constantly upload them to the server for safe keeping. This was a significant chunk of work, but has actually landed already in Riot/Web and Riot/iOS, but is hidden behind a “Labs” feature flag in Settings whilst we test it and sort out the final UX.
Cross-signing Keys. Previously, the only way to check whether you were talking to a legitimate user or an imposter was to independently compare the fingerprints of the keys of the device they claim to be using. In the near future, we will let users prove that they own their devices by signing them with a per-user identity key, so you only have to do this check once. We've actually already implemented one iteration of cross-signing, but this allowed arbitrary devices for a given user to attest each other (which creates a directed graph of attestations, and associated problems with revocations), hence switching to a simpler approach.
Improved Verification. Verifying keys by manually comparing elliptic key fingerprints is incredibly cumbersome and tedious. Instead, we have proposals for using Short Authentication String comparisons and QR-code scanning to simplify the process. uhoreg is currently implementing these as we speak :)
Search. Solving encrypted search is Hard, but t3chguy did a lot of work earlier in the year to build out matrix-search: essentially a js-sdk bot which you can host, hand your keys to, and will archive your history using a golang full-text search engine (bleve) and expose a search interface that replaces Synapse's default one. Of all the battlefronts this one is progressing the least right now, but the hope is to integrate it into Riot/Desktop or other clients so that folks who want to index all their conversations have the means to do so. On the plus side, all the necessary building blocks are available thanks to t3chguy's hacking.

So, TL;DR: E2E is hard, but the end is in sight thanks to a lot of blood, sweat and tears. It's possible that we may have opened up too many battlefronts in finishing it off rather than landing stuff gradually. But it should be transformative when it all lands - and we'll finally be able to turn it on by default for private conversations. Again, we're aiming to pull this together by the end of January.

Separately, we've been keeping a close eye on MLS - the IETF's activity around standardising scalable group E2E encryption. MLS has a lot of potential and could provide algorithmic improvements over Olm & Megolm (whist paving the way for interop with other MLS-encrypted comms systems). But it's also quite complicated, and is at risk of designing out support of decentralised environments. For now, we're obviously focusing on ensuring that Matrix is rock solid with Olm & Megolm, but once we hit that 1.0 we'll certainly be experimenting a bit with MLS too.

Homeservers

The story of the Synapse team in 2018 has been one of alternating between solving scaling and performance issues to support the ever-growing network (especially the massive matrix.org server)... and dealing with S2S API issues; both in terms of fixing the design of State Resolution, Room Versioning etc (see the Spec section above) and doing stop-gap fixes to the current implementation.

Focusing on the performance side of things, the main wins have been:

Splitting yet more functionality out into worker processes which can scale independently of the master Synapse process.
Yet more profiling and optimisation (particularly caching). Between this and the worker split-out we were able to resolve the performance ceiling that we hit over the summer, and as of right now matrix.org feels relatively snappy.
Lazy Loading Members.
Python 3. As everyone should have seen by now, Synapse is now Python 3 by default as of 0.34, which provides significant RAM and CPU improvements across the board as well as a path forwards given Python 2's planned demise at the end of 2019. If you're not running your Synapse on Python 3 today, you are officially doing it wrong.

There are also some major improvements which haven't fully landed yet:

State compression. We have a new algorithm for storing room state which is ~10x more efficient than the current one. We'll be migrating to it in by default in future. If you're feeling particularly intrepid you can actually manually use it today (but we don't recommend it yet).
Incremental state resolution. Before we got sucked into redesigning state resolution in general, Erik came up with a proof that it's possible to memoize state resolution and incrementally calculate it whenever state is resolved in a room rather than recalculate it from scratch each time (as is the current implementation). This would be a significant performance improvement, given non-incremental state res can consume a lot of CPU (making everything slow down when there are lots of room extremities to resolve), and can consume a lot of RAM and has been one of the reasons that synapse's RAM usage can sometimes spike badly. The good news is that the new state res algorithm was designed to also work in this manner. The bad news is that we haven't yet got back to implementing it yet, given the new algorithm is only just being rolled out now.
Chunks. Currently, Synapse models all events in a room as being part of a single DAG, which can be problematic if you can see lots of disconnected sections of the DAG (e.g. due to intermittent connectivity somewhere in the network), as Synapse will frantically try to resolve all these disconnected sections of DAG together. Instead, a better solution is to explicitly model these sections of DAG as separate entities called Chunks, and not try to resolve state between disconnected Chunks but instead consider them independent fragments of the room - and thus simplify state resolution calculations significantly. It also fixes an S2S API design flaw where previously we trusted the server to state the ordering (depth) of events they provided; with chunks, the receiving server can keep track of that itself by tracking a DAG of chunks (as well as the normal event DAG within the chunks). Now, most of the work for this happened already, but is currently parked until new state res has landed.

Meanwhile, over on Dendrite, we made the conscious decision to get 1.0 out the door on Synapse first rather than trying to implement and iterate on the stuff needed for 1.0 on both Synapse & Dendrite simultaneously. However, Dendrite has been ticking along thanks to work from Brendan, Anoa and APWhitehat - and the plan is to use it for more niche homeserver work at first; e.g. constrained resource devices (Dendrite uses 5-10x less RAM than Synapse on Py3), clientside homeservers, experimental routing deployments, etc. In the longer term we expect it to grow into a fully fledged HS though!

Bridging

2018 was a bit of a renaissance for Bridging, largely thanks to Half-Shot coming on board in the Summer to work on IRC bridging and finally get to the bottom of the stability issues which plagued Freenode for the previous, uh, few years. Meanwhile the Slack bridge got its first ever release - and more recently there's some Really Exciting Stuff happening with matrix-appservice-purple; a properly usable bridge through to any protocol that libpurple can speak… and as of a few days ago also supports native XMPP bridging via XMPP.js. There'll probably be a dedicated blogpost about all of this in the new year, especially when we deploy it all on Matrix.org. Until then, the best bet is to learn more is to watch last week's Matrix Live and hear it all first hand.

Modular

One of the biggest newcomers of 2018 was the launch of Modular.im in October - the world's first commercial Matrix hosting service. Whilst (like Riot), Modular isn't strictly-speaking a Matrix.org project - it feels appropriate to mention it here, not least because it's helping directly fund the core Matrix dev team.

So far Modular has seen a lot of interest from folks who want to spin up a super-speedy dedicated homeserver run by us rather than having to spend the time to build one themselves - or folks who have yet to migrate from IRC and want a better-than-IRC experience which still bridges well. One of the nice bits is that the servers are still decentralised and completely operationally independent of one another, and there have been a bunch of really nice refinements since launch, including the ability to point your own DNS at the server; matrix->matrix migration tools; with custom branding and other goodness coming soon. If you want one-click Matrix hosting, please give Modular a go :)

Right now we're promoting Modular mainly to existing Matrix users, but once the Riot redesign is finished you should expect to see some very familiar names popping up on the platform :D

TWIM

Unless you were living under a rock, you'll hopefully have also realised that 2018 was the year that brought us This Week In Matrix (TWIM) - our very own blog tracking all the action across the whole Matrix community on a weekly basis. Thank you to everyone who contributes updates, and to Ben for editing it each week. Go flip through the archives to find out what's been going on in the wider community over the course of the year! (This blog post is already way too long without trying to cover the rest of the ecosystem too :)

Shapes of Things to Come

Finally, a little Easter egg (it is Christmas, after all) for anyone crazy enough to have read this far: The eagle-eyed amongst you might have noticed that one of our accepted talks for FOSDEM 2019 is “Breaking the 100bps barrier with Matrix” in the Real Time Communications devroom. We don't want to spoil the full surprise, but here's a quick preview of some of the more exotic skunkworks we've been doing on low-bandwidth routing and transports recently. Right now it shamelessly assumes that you're running within a trusted network, but once we solve that it will of course be be proposed as an MSC for Matrix proper. A full write-up and code will follow, but for now, here's a mysterious video…

(If you're interested in running Matrix over low-bandwidth networks, please get in touch - we'd love to hear from you...)

2019

So, what will 2019 bring?

In the short term, as should be obvious from the above, our focus is on:

r0 spec releases across the board (aka Matrix 1.0)
Implementing them in Synapse
Landing the Riot redesign
Landing all the new E2E encryption UX and features
Finalising the Matrix.org Foundation

However, beyond that, there's a lot of possible options on the table in the medium term:

Reworking and improving Communities/Groups.
Reactions.
E2E-encrypted Search
Filtering. (empowering users to filter out rooms & content they're not interested in).
Extensible events.
Editable messages.
Extensible Profiles (we've actually been experimenting with this already).
Threading.
Landing the Riot/Android rewrite
Scaling synapse via sharding the master process
Bridge UI for discovery of users/rooms and bridge status
Considering whether to do a similar overhaul of Riot/iOS
Bandwidth-efficient transports
Bandwidth-efficient routing
Getting Dendrite to production.
Inline widgets (polls etc)

Improving VoIP over Matrix.

Adding more bridges, and improving the current ones..
Account portability
Replacing MXIDs with public keys

In the longer term, options include:

Shared-code cross-platform client SDKs (e.g. sharing a native core library between matrix-{'{'}js,ios,android{'}'}-sdk)
Matrix daemons (e.g. running an always-on client as a background process in your OS which apps can connect to via a lightweight CS API)
Push notifications via Matrix (using a daemon-style architecture)
Clientside homeservers (i.e. p2p matrix) - e.g. compiling Dendrite to WASM and running it in a service worker.
Experimenting with MLS for E2E Encryption
Storing and querying more generic data structures in Matrix (e.g. object trees; scene graphs)
Alternate use cases for VR, IoT, etc.

Obviously we're not remotely going to do all of that in 2019, but this serves to give a taste of the possibilities on the menu post-1.0; we'll endeavour to publish a more solid roadmap when we get to that point.

And on that note, it's time to call this blogpost to a close. Thanks to anyone who read this far, and thank you, as always, for flying Matrix and continuing to support the project. The next few months should be particularly fun; all the preparation of 2018 will finally pay off :)

Happy holidays,

Matthew, Amandine & the whole Matrix.org team.

Porting Synapse to Python 3

21.12.2018 00:00 — Tech — Neil Johnson

Matrix's reference homeserver, Synapse, is written in Python and uses the Twisted networking framework to power its bitslinging across the Internet. The Python version used has been strictly Python 2.7, the last supported version of Python 2, but as of this week that changes! Since Twisted and our other upstream dependencies now support the newest version of Python, Python 3, we are now able to finish the jump and port Synapse to use it by default. The port has been done in a backwards compatible way, written in a subset of Python that is usable in both Python 2 and Python 3, meaning your existing Synapse installs still work on Python 2, while preparing us for a Python 3 future.

Why port?

Porting Synapse to Python 3 prepares Synapse for a post-Python 2 world, currently scheduled for 2020. After the 1st of January in 2020, Python 2 will no longer be supported by the core Python developers and no bugfixes (even critical security ones) will be issued. As the security of software depends very much on the runtime and libraries it is running on top of, this means that by then all Python 2 software in use should have moved to Python 3 or other runtimes.

The Python 3 port has benefits other than just preparing for the End of Life of Python 2.7. Successive versions of Python 3 have improved the standard library, provided newer and clearer syntax for asynchronous code, added opt-in static typing to reduce bugs, and contained incremental performance and memory management improvements. These features, once Synapse stops supporting Python 2, can then be fully utilised to make Synapse's codebase clearer and more performant. One bonus that we get immediately, though, is Python 3's memory compaction of Unicode strings. Rather than storing as UCS-2/UTF-16 or UCS-4/UTF-32, it will instead store it in the smallest possible representation giving a 50%-75% memory improvement for strings only containing Latin-1 characters, such as nearly all dictionary keys, hashes, IDs, and a large proportion of messages being processed from English speaking countries. Non-English text will also see a memory improvement, as it can be commonly stored in only two bytes instead of the four in a UCS-4 “wide” Python 2 build.

Editor's note: If you were wondering how this fits in with Dendrite (the next-gen golang homeserver): our plan is to use Synapse as the reference homeserver for all the current work going on with landing a 1.0 release of the Matrix spec: it makes no sense to try to iterate and converge on 1.0 on both Synapse and Dendrite in parallel. In order to prove that the 1.0 spec is indeed fit for purpose we then also need Synapse to exit beta and hit a 1.0 too, hence the investment to get it there. It's worth noting that over the last year we've been plugging away solidly improving Synapse in general (especially given the increasing number of high-profile deployments out there), so we're committed to getting Synapse to a formal production grade release and supporting it in the long term. Meanwhile, Dendrite development is still progressing - currently acting as a place to experiment with more radical blue-sky architectural changes, especially in low-footprint or even clientside homeservers. We expect it to catch up with Synapse once 1.0 is out the door; and meanwhile Synapse is increasingly benefiting from performance work inspired by Dendrite.

When will the port be released?

The port is has been released in a “production ready” form in Synapse 0.34.0, supporting Python 3.5, 3.6, and 3.7. This will work on installations with and without workers.

What's it like in the real world?

Beta testers of the Python 3 port have reported lower memory usage, including lower memory “spikes” and slower memory growth. You can see this demonstrated on matrix.org:

See 10/15, ~20:00 for the Python 3 migration. This is on some of the Synchrotrons on matrix.org.

See ~11/8 for the Python 3 migration. This is on the Synapse master on matrix.org.

We have also noticed some better CPU utilisation:

See 21:30 for the migration of federation reader 1, and 21:55 for the others. The federation reader is a particular pathological case, where the replacement of lists with iterators internally on Python 3 has given us some big boosts.

See 10/15, 4:00.The CPU utilisation has gone down on synchrotron 1 after the Python 3 migration, but not as dramatically as the federation reader. Synchrotron 3 was migrated a few days later.

As some extra data-points, my personal HS consumes about 300MB now at initial start, and grows to approximately 800MB -- under Python 2 the growth would be near-immediate to roughly 1.4GB.

Where to from here?

Python 2 is still a supported platform for running Synapse for the time being. We plan on ending mainstream support on 1st April 2019, where upon Python 3.5+ will be the only officially supported platform. Additionally, we will give notice ahead of time once we are ready to remove Python 2.7 compatibility from the codebase (which will be no sooner than 1st April). Although slightly inconvenient, we hope that this gives our users and integrators adequate time to migrate, whilst giving us the flexibility to use modern Python features and make Synapse a better piece of software to help power the Matrix community.

How can I try it?

The port is compatible with existing homeservers and configurations, so if you install Synapse inside a Python 3 virtualenv, you can run it from there. Of course, this differs based on your installation method, operating system, and what version of Python 3 you wish to use. Full upgrade notes live here but if you're having problems or want to discuss specific packagings of Synapse please come ask in #synapse:matrix.org.

Thanks

Many thanks go to fellow Synapse developers Erik and Rich for code review, as well as community contributors such as notafile and krombel for laying the foundations many months ago allowing this port to happen. Without them, this wouldn't have happened.

Happy Matrixing,

Amber Brown (hawkowl)

This Week in Matrix 2018-12-21

21.12.2018 00:00 — This Week in Matrix — Ben Parsons

Matrix Live: Half-Shot talks bridges, and working on libpurple bridging to Matrix

You may have seen that Half-Shot been working fearlessly and tirelessly on bridges for many, many months. In this episode of Matrix Live Half-Shot chats with Matthew about his work, progress and achievements, with a focus on recent matrix-appservice-purple and XMPP work. Audio is not amazing, but worth listening to get acquainted with recent work.

As a note from Half-Shot:

The matrix-appservice-purple bridge gained a XMPP specific backend for better performance when you want to do just xmpp bridging. It's rather quick right now, and needs dogfooding.

Chat in #purple-bridge:half-shot.uk!

Matrix badges on shields.io

Brendan:

A couple of weeks ago, fr1kin PR'd a nice Matrix badge to the Shields project that tells you how many users are in a public room. There were a few issues with it so I PR'd some changes to make the badge more usable, and it's now merged and live (as of yesterday evening), with examples available here! ?
For instance, here's a badge for TWIM: https://img.shields.io/matrix/twim:matrix.org.svg ?

Spec update

New MSC for cross-signing, which has different (hopefully better) semantics from the previous cross-signing MSC.

matrix-bot-sdk support for application services

TravisR, author of matrix-bot-sdk:

matrix-bot-sdk has received early support for application services. Similar to the official matrix-js-sdk, the bot-sdk uses an Intent-based model for making bridges easier to write. Check out the simple example here for more information on how it works.

koma-library

druig continues work on their JavaFX/kotlin client project:

The matrix client API implementation in koma is being extracted into a new repo, which is going to be a lightweight library for Kotlin.

Seaglass now available on homebrew

Aaron Raimist reports that Seaglass is available on homebrew for macOS:

Installing Seaglass is now easier than ever. If you already use Homebrew to manage other packages, you can now install Seaglass with a simple brew cask install seaglass.

If you don't use Homebrew, you can still download Seaglass from GitHub.

matrix-client.el is reborn!

alphapapa reports from a team who have forked and are maintaining a Matrix client for Emacs:

Many additions and improvements to matrix-client.el (https://github.com/jgkamat/matrix-client-el) recently, including a "standalone client" mode that launches Emacs to look like this.

Chat in #matrix-client-el:matrix.org.

matrix-docker-ansible-deploy

Slavi:

the matrix-docker-ansible-deploy playbook has received some bugfixes and improvements lately. Most importantly, it's now running the freshly-released Synapse 0.34.0 under Python 3, so memory usage should be much better.

libQMatrixClient/Quaternion

kitsune:

I started work on matrix: scheme support in libQMatrixClient/Quaternion. Expect more news on this around New Year.

Fractal road to 4.0

Fresh from their hackfest in Seville last week, Alexandre Franke reports:

The Fractal crew spent the week chasing last minutes bugs and made two beta releases (3.99.0 and 3.99.1) in preparation for the big new stable release, 4.0, which is right around the corner.

New Rooms for Space Launches and Aviation

Aaron Raimist has been creating some new rooms:

#space:im.kabi.tk for anyone interested in space, rocket launches, satellites, etc.
Are you wondering what NASA's new Mars Rover is doing? Maybe you live on the west coast of the United States, and you saw that meteor on Wednesday night that came within minutes of a scheduled rocket launch and just after three astronauts left the space station. If any of that sounds interesting to you, feel free to join the room.
A Matrix bot is being tested to send updates about upcoming rocket launches to the room.

#aviation:matrix.org for anyone interested in aviation. Whether you are a pilot, someone who visits an airshow once in a while, or if https://www.youtube.com/watch?v=CXv1j3GbgLk piqued your interest, come join.

t2bot.io upgrade

TravisR:

Just a head's up that I've increased the storage capacity of the database. With current projections, the server should be good for another year or two.

The database is also 1ms closer to Synapse and has a faster CPU in it. It probably won't make a dent in speed on federation, but it is a step forward.

I'll be rolling out python 3 to the homeserver this week too, which should help a little bit.

https://riot.im/experimental/#/room/#help:t2bot.io/$154511414912clELm:t2l.io

FluffyChat 8.0 RC announced

Krille and his Ubuntu Touch fans are looking forward to FluffyChat 8.0:

Hey guys, in order to release the FluffyChat 8.0 Christmas Edition, the FluffyChat 8.0 release candidate is ready for you! :-)
Also the weblate translations are ready: https://hosted.weblate.org/projects/fluffychat/

Riot iOS

From the Riot iOS team:

We have fixed and improved some e2e stuff.
Back to reskin. We start to implement e2e backup screens.

Riot Android

From the Riot Android team:

New screen to troubleshoot notification issue has been merged on /develop.
Splitting current Android SDK to separate crypto part is on it's way. We're doing this in order to be able to integrate crypto faster in the Riot reboot.

Synapse

Neil from Synapse reports:

We released 0.34.0! This release recommends Python 3 for production and brings with it huge performance improvements. If you've been putting upgrading off upgrading your Synapse, now is the time to do so. For more details here is a post that explains what you should expect and a recent Matrix Live interviews Amber (hawkowl) on the subject.

Aside from that we are working furiously towards federation R0 and have a bunch of MSCs to get us ever closer. You can track our progress here.

andrewsh notes that 0.34.0 is also available in the Debian repos:

Synapse 0.34.0~rc2 in Debian since Tuesday, 0.34.0 uploaded today; both use Python 3 only

Dendrite

Brendan reports:

Dendrite's internal audit is progressing very well and is getting very close to its end.
What's left to do for me is check the implementation status of a few Matrix features, and translate those into tagged GitHub issues so that everyone can have as clear of a view as possible on what's left to work on.
I'm on holiday all of next week, but hopefully will have some good news about that the following week.

That really is it for now

Did you get to the end? What was your favourite section? Come tell us in #twim:matrix.org! Do you have your own update you'd like to add? Same place, come chat in #twim:matrix.org.

Next week is various things. It's Christmas, which means there will be more hacking and coding happening than usual I expect! It's also 35c3, which I will be attending, and might affect scheduling next week. Stay tuned in #twim:matrix.org for news, and come join us in #matrix-35c3:matrix.org if you'll be there and want to meet up!

Merry Christmas!

Synapse 0.34.0 released!

20.12.2018 00:00 — Releases — Neil Johnson

Folks this is a big day for us at Matrix Towers, because today we release 0.34.0.

The big news for 0.34.0 is that we now recommend Python 3 for production use and have been running matrix.org under Python 3 for the past month.

Performance improvements have been marked, in some contexts we have seen 50% reductions in RAM and CPU usage. Here are some illustrative graphs to get you going but look out for a dedicated post delving into much more detail on the port. You can also see a Matrix Live interview with the project lead Amber (hawkowl) here.

Matrix.org federation reader workers, the big drops signify roll over to python 3

Synapse master on matrix.org, again the drop in RAM signifies the roll over to python 3

Many thanks to Amber for leading the effort, Rich and Erik for providing support as well as Notafile and Krombel from the community for pushing this effort right from the early days of the project.

If that wasn't enough, 0.34.0 also all the usual bug fixes and perf improvements. In particular the media repository now no longer fails to decode UTF-8 filenames when downloading remote media and auto joining rooms now work on servers with consent requirements enabled.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.

In particular, if you want to run Synapse 0.34.0 on Python 3 take a look at the upgrade notes.

Synapse 0.34.0 changelog

Synapse 0.34.0 is the first release to fully support Python 3. Synapse will now run on Python versions 3.5 or 3.6 (as well as 2.7). Support for Python 3.7 remains experimental.

We recommend upgrading to Python 3, but make sure to read the upgrade notes when doing so.

Features

Add 'sandbox' to CSP for media reprository (#4284)
Make the new landing page prettier. (#4294)
Fix deleting E2E room keys when using old SQLite versions. (#4295)
Add a welcome page for the client API port. Credit to @krombel! (#4289)
Remove Matrix console from the default distribution (#4290)
Add option to track MAU stats (but not limit people) (#3830)
Add an option to enable recording IPs for appservice users (#3831)
Rename login type m.login.cas to m.login.sso (#4220)
Add an option to disable search for homeservers that may not be interested in it. (#4230)

Bugfixes

Pushrules can now again be made with non-ASCII rule IDs. (#4165)
The media repository now no longer fails to decode UTF-8 filenames when downloading remote media. (#4176)
URL previews now correctly decode non-UTF-8 text if the header contains a <meta http-equiv="Content-Type" header. (#4183)
Fix an issue where public consent URLs had two slashes. (#4192)
Fallback auth now accepts the session parameter on Python 3. (#4197)
Remove riot.im from the list of trusted Identity Servers in the default configuration (#4207)
fix start up failure when mau_limit_reserved_threepids set and db is postgres (#4211)
Fix auto join failures for servers that require user consent (#4223)
Fix exception caused by non-ascii event IDs (#4241)
Pushers can now be unsubscribed from on Python 3. (#4250)
Fix UnicodeDecodeError when postgres is configured to give non-English errors (#4253)

Internal Changes

Debian packages utilising a virtualenv with bundled dependencies can now be built. (#4212)
Disable pager when running git-show in CI (#4291)
A coveragerc file has been added. (#4180)
Add a GitHub pull request template and add multiple issue templates (#4182)
Update README to reflect the fact that #1491 is fixed (#4188)
Run the AS senders as background processes to fix warnings (#4189)
Add some diagnostics to the tests to detect logcontext problems (#4190)
Add missing jpeg package prerequisite for OpenBSD in README. (#4193)
Add a note saying you need to manually reclaim disk space after using the Purge History API (#4200)
More logcontext checking in unittests (#4205)
Ignore __pycache__ directories in the database schema folder (#4214)
Add note to UPGRADE.rst about removing riot.im from list of trusted identity servers (#4224)
Added automated coverage reporting to CI. (#4225)
Garbage-collect after each unit test to fix logcontext leaks (#4227)
add more detail to logging regarding "More than one row matched" error (#4234)
Drop sent_transactions table (#4244)
Add a basic .editorconfig (#4257)
Update README.rst and UPGRADE.rst for Python 3. (#4260)
Remove obsolete verbose and log_file settings from homeserver.yaml for Docker image. (#4261)

This Week in Matrix 2018-12-14

14.12.2018 00:00 — This Week in Matrix — Ben Parsons

Fractal Hackfest

Developers and creators of Fractal, the GNOME Matrix client, have been holding a Hackfest this week in Seville. Matthew and I caught up with them on video for Matrix Live this week, and discussed the product improvements they've been making and their plans for the next release (due next week!)

Matrix Spec

MSC1466 merged. MSC1466 from Erik describes "Soft Remote Logout", which avoids loss of encryption keys in circumstances of remote logout.
MSC1730 entered FCP (final comment period, the last step before being merged.) read the proposal

This Week in Ruma

We don't often get to feature news from Ruma, but this week there is an updated This Week in Ruma.

Ruma is not dead, however, and small improvements have continued over the last year. The Matrix spec has advanced quite a bit and many of the blocking issues for Ruma have been resolved. Rust's maturity is another story. async/await is still under development and this is the most significant blocking issue to progress on Ruma.

maubot new tooling

tulir has been working on tooling for maubot:

maubot got a command-line tool for building plugins and managing maubot instances. I also added some server-side stuff for easy registration of accounts in the management interface using synapse shared secrets, but the UI for that isn't ready. Also, I'm planning on adding some kind of small Matrix client in the management interface for manually managing the added bot clients. That might lead to a separate library that could be used in other projects or embedded in websites.

mxisd v1.2.1 released, updating recommended

Max with a new release of mxisd:

mxisd hits v1.2.1 with a new maintenance release, fixing bugs and regressions from v1.2.0. Updating to v1.2.1 is strongly encouraged as v1.3.x will contain breaking changes and will not be a straight-forward update like v1.x has been until now.

Clients

Lazy-loading lands in QMatrixClient

kitsune has reported that lazy-loading, a Matrix feature that entails only loading room member details as needed, is now in libQMatrixClient master.

reference implementation in Quaternion will follow up

Black Hat has been testing the feature in Spectral, which uses the library, and says there is a 30%-50% reduction in RAM usage at startup.

Riot Web progress on `/experimental` and v0.17.8 released

Bruno has been working on /experimental, and I recommend taking a look at the progress there! The next version of Riot Web is closer than ever.

Improved read markers now available on /experimental, needs further tweaking though. Brought back community UX on redesign, other small improvements.

v0.17.8 was released with several bugfixes and improvements.

Riot iOS

Release made with these notes:

This new version supports the consent of matrix servers terms of service (including GDPR) in the registration flow. It also contains fixes for the "Empty room" bug, the registration issue on iOS 10, etc.

Get more information:

Riot Android

Release made:

This new version supports the consent of matrix servers terms of service (including GDPR) in the registration flow. Many bugfixes SDK contains KeyBackup

Get more information:

Servers

Synapse

A lot of focus on getting debian packages ready for python 3 - this is a blocker for our official python 3 release 0.34.0. Aside from that, finalising some outstanding state resolution behaviour (https://github.com/matrix-org/matrix-doc/pull/1693) and dusting off event ids as hashes (https://github.com/matrix-org/matrix-doc/issues/1127.)

Dendrite

Brendan:

My internal audit of Dendrite is continuing, drawing a more and more precise picture of what's left to fix and implement. I aim to have it over by the end of the year, or the very early days of 2019 at the latest. Folks can track its progress through https://cloud.abolivier.bzh/index.php/s/qXi2KFjCQk2c6eG

modular.im now has Extra-Large instances available

Due to demand, modular.im Hosted Homeservers now has Extra-Large instances available. If you need to service 1,000+ users on a Matrix homeserver, this is the product for you!

linuxgaming.life homeserver is the number one Matrix homeserver focused on Linux gaming

swedneck has continues his work on linuxgaming.life:

I've added some bots, health monitoring, and dimension integration manager to linuxgaming.life, and made the website dark.

The homeserver is open for new registrations.

That's all I know!

If you'd like see your Matrix-related project featured in this blog post, come chat to us in #twim:matrix.org, and we'll see you next week!

This Week in Matrix 2018-12-07

07.12.2018 00:00 — This Week in Matrix — Ben Parsons

Matrix Live S03E06

In which Matthew & Amandine discuss The Matrix.org Foundation, go-live for the French Government deployment for Matrix, and some exciting random diversions into post-1.0 Matrix features which should land once once 1.0 is out the door!

Purism sponsoring Fractal development

We are proudly sponsoring the work of Julian Sparber on Fractal, the @matrixdotorg client for GNOME. Read his latest update - https://t.co/q4ubjzETxC #LibreDesktop #DemandFreedom #gnome

— Purism (@Puri_sm) December 5, 2018

Purism announced they are sponsoring Fractal development, starting with Julian Sparber working last month to improve the message view. You can read about his progress here. Highlights:

Before, there was a jarring cut when new messages were loaded, but now you can just scroll upward and older messages are loaded continuously.
The part I'm most excited about is the new "new message divider". When the user opens a room they can directly start reading the conversation from the last seen message and they don't need to search for the new message divider. Not only has the UX gotten a lot better, but also the underlying code is much cleaner now.
I also spent some time on making message rendering faster. I replaced the RegExp with more efficient code, this made the rendering much faster (from ~10ms to ~ 1ms) for every single message.
In summary, all of these things improve Fractal's UX a lot and make it feel more like a modern messaging app.

Synapse 0.34.0rc1

0.34.0rc1 is out - please test it! 0.34.0 will be the first to officially support python 3. We're still working on the debian packaging but 0.34.0 proper should be out next week.

Dendrite

Brendan has been getting on with Dendrite development:

Started auditing Dendrite's codebase to identify what is left to implement, along with what hasn't been implemented correctly. Still a lot of work left to do on this, but it's looking promising so far.

Modular

Modular is a Hosted Homeservers product, check out modular.im. This week:

Matrix -> Matrix account migration tool available at https://www.modular.im/tools/matrix-migration. It's still beta, so please test it and let us know if you experience issues.

New Homeserver: linuxgaming.life

swedneck has set up a new public homeserver at linuxgaming.life! There is also a Riot installation at https://riot.linuxgaming.life, served via IPFS.

Currently working on the website (based on t2bot.io, thank you travis for making that available on github!) and making things nicer in general.
I will be adding more bots shortly. Please let me know of any issues with the HS or anything surrounding it.

Informo

Informo is a project to create a specification and implementation for distributing information and news. This week they have been working on merging changes to the specification:

Work on the Informo specifications has slowed down a bit in the last couple of weeks, though since the last update we did manage to get some relatively big SCSs merged into the specs, including SCS #9 (rendered here) which specifies how information sources must publish their information through the federation, and SCS #11 (rendered here) which describes how sources must register themselves in order to be picked up by clients, and handle localisation. ?
There's still a couple of big items to take care of before we can cut a 1.0 release of the Informo specs but this is definitely a huge step towards this goal.

Minecraft Bridge

Dandellion:

I started working on a Minecraft bridge pretty heavily based on Travis's old project, but using Minecraft-protocol instead of mineflayer.
You need a bot Minecraft account that can join and idle in the server, it then uses /tellraw to post messages.

libQMatrixClient 0.4.1 released

libQMatrixClient, which powers Spectral and Quaternion was released by kitsune, Lazy Loading coming soon:

libQMatrixClient 0.4.1 has been released today, with small fixes in the stable branch. Meanwhile, active work is ongoing on lazy-loading support in the library, with ETA for the feature landing in master being in about a week or so.

Riot Web

Redesign progress: all resizeable sections are now persisted, (slow) progress on improving read markers
Communities and community invitations should be more resilient to errors (on /develop only so far)

Riot iOS

Reskin still ongoing
A new release is coming
Less activity because of POSS

Riot Android

A new developer joined us to help maintaining Riot: Valere Fedronic
Keys backup PR in review
Privacy Terms acceptance in login flow in review
FCM issue investigation/troubleshooting by Valere
François Off (sick :()
New theme is coming soon

Koma

druig has been working on Koma, a JavaFX Matrix client:

This week in koma: implementation of json library switched from runtime reflection to compile-time code generation
The matrix api is implemented manually in the project, retrofit and moshi are used to interact with the rest api.

SimpleMatrix

As we mentioned in TWIM last week, MTRNord has been working on designs for SimpleMatrix, a Matrix client for Android in development. This week he has made a video showcasing the new design.

Redecentralize meetup in London

Last night Half-Shot & Neil & Brendan went to see the Redecentralize folk at their meetup:

Beer! And pizza! And also Redecentralize
We chatted to a nice bunch of folks both demoing their decentralized projects and talked to the likes of scuttlebutt, BBC, IPFS and more. Was a interesting experience having around 8 minutes to quickly explain to newcomers what Matrix is and why they should use it in a speed dating format. We chatted (and demoed) bridges, new-riot and generally how it all fits together. There were a lot of very interesting people with different profiles and backgrounds, and diversified questions which lead to a lot of interesting discussions.

Note for the confused: the format of the meetup was comparable to speed dating. There is no suggestion that Matrix should be used at regular speed dating. If you do find such a use for Matrix, come tell us in #twim:matrix.org.

matrix-wug, X-SAMPA to IPA bot

Last month Dandellion introduced a bot designed to convert between formats for pronunciation notation, this week he released the source code: https://github.com/dali99/matrix-wug

The end of the post, and nearly the end of 2018

This week it has been winter-in-Europe kind of weather, but that's ok, it happens every year. TWIM on the other hand happens every week, so if you have something to share, and would like to share what you've been working on, come chat to us in #twim:matrix.org.

This Week in Matrix 2018-11-30

30.11.2018 00:00 — This Week in Matrix — Ben Parsons

Porting to Python 3 - Matrix Live S3E5

Longer than usual episode of Matrix Live this week. Amber talks to Neil about the works she's been doing over the last few months to port Synapse from Python 2 to Python 3. Recommended for anyone who's been following along the progress of Synapse, or who wants a good intro on the benefits of Python 3 over Python 2.

ZeroPhone

This week we learned about ZeroPhone, a fairly mature project aiming to build a working cellphone on top of a Raspberry Pi Zero.

an open-source smartphone that can be assembled for 50$ in parts. It is Linux-powered, with UI software written in Python. Currently, ZeroPhone is based on Raspberry Pi Zero, SIM800L GSM modem and 1.3" OLED screen and button interface.

&Adam was the first to bring it to our attention:

I stumbled across this blog post which mentions a matrix app in a project to create an open, hackable phone out of a raspberry pi. It's old news at this point I guess, but I don't think I've seen it mentioned here, so I thought it might be a neat thing worth featuring: https://zerophone.github.io/newsletter/ZeroPhone-Weekly-No.-16/

The creator of the Matrix app, derivmug also arrived to tell us about his work:

Hey everyone, I have been suggested to post about the Zerophone project here. I have written a basic matrix client for it. It's basically a simple phone based on a raspberry pi zero, focused on privacy, security and hackability.

The client app is built using the matrix-python-sdk, I'm also hoping to chat to derivmug as part of a future Matrix Live. Take a look at the code here.

Just to prove it's running on real hardware!

Spec / Foundation

Work continues on MSC1730 - Mechanism for redirecting to an alternative server during login https://github.com/matrix-org/matrix-doc/pull/1730
Foundation: Working with the Foundation lawyers on the CIC37 to register the Foundation as a Community Interest Company proper

The Federation (the-federation.info)

Jason Robinson is passionate about Federated technology, and has been working on his site at https://the-federation.info/ for some time. He recently added Matrix to the list of protocols he scrapes and lists:

Matrix (or more specifically, Synapse) servers are now visible on the The Federation info website at https://the-federation.info/matrix%7Csynapse. The website collects lists of servers for various federation protocols (like Matrix, ActivityPub, Diaspora, etc). If a server outputs usage information, it also shows some historical information for servers and projects. Currently the information available from Synapse on the site is just version, availability of registrations and (obviously) domain name. If you want to register your server, check https://the-federation.info/info for more details. Tips also welcome on how to register other Matrix server projects and how to get more data out of servers.

maubot

tulir has continued work on maubot:

I made a rss plugin for maubot and added a fancy log viewer in the maubot management UI.

mautrix-telegram 0.4.0 was released, though it has no changes since the release candidate. I also copied mautrix-telegram's HTML parser into mautrix-python. Hopefully it's now generic enough so that when mautrix-telegram switches to mautrix-python, it can drop most of the built-in HTML parser.

matrix-appservice-discord

Bridge-of-the-week this week is matrix-appservice-discord. Anyone in the dev room (#discord:half-shot.uk) will be very aware of this as Half-Shot and Sorunome have spent the day testing @room notifications.

Sorunome:

matrix-appservice-discord received a lot of work these past days. Not only the usual small bugfixes, but also some rather big changes: The parsers for both Discord->Matrix sending and Matrix->Discord sending were re-written completely to properly handle things (and thus, at the same time, crunch a bunch of edge-case bugs!). In addition, the long-awaited highlighting issues with @ everyone and @ here (on discord) and @ room (on matrix) were addressed. Some of these things are still in PR stage, but expected to be merged soon!

Half-Shot also provided a great screenshot of everything working well:

is just showing off having IRC+Matrix+Discord+XMPP plugged together. XMPP being the one purple is bridging

matrix-docker-ansible-deploy

Slavi reports that:

matrix-docker-ansible-deploy now supports configuring the matrix-synapse-ldap3 LDAP auth password provider. Thanks to @tvo6 for contributing this!

Riot iOS

User Agreement (GDPR) in now part of the registration flow
Reskin has started
New fixes for unexpected “Empty room”
We have PR templates

Riot Android

Realm store for e2e keys has landed
Riot reboot: support timelines merge in DB to avoid to trash data as less as possible when navigating through permalinks
New settings to choose ringtone for incoming call
We have PR templates also

SimpleMatrix

MTRNord, is working on the design for SimpleMatrix:

As I currently have less time for actual coding I am planning out the SimpleMatrix Design. You can find the Images of the Design (and a PDF file) at https://gitlab.com/Nordgedanken/simplematrixredesignimages Feel free to also join #SimpleMatrix:matrix.ffslfl.net to give feedback.

Synapse

This week our focus has been to work through some final bugs blocking 0.34.0 which will be our first official python 3 release. Expect a RC rsn. For a sneak peak see this week's Matrix Live video at the top!

Auf Wiedersehen…

That's it for this week. If you have anything to share with us, and you'd like to be included on the blog, please come talk in #twim:matrix.org!

In one month from now, many Matrix-acolytes, including myself, will be at 35c3. If you'll be there, come chat to us in #matrix-35c3:matrix.org!

This Week in Matrix 2018-11-26

26.11.2018 00:00 — This Week in Matrix — Ben Parsons

A little delay to the release of this week's TWIM, expect us to resume normal schedule on Friday.

Spec & Foundation

Two workshops happened with the Foundation lawyers to work on MSC1318 - basically fleshing out a proposal for the details of the foundation governance in terms of how the Guardians (i.e. directors of the foundation board) should interact with the Core Spec Team to act as a back-stop - and working out how to port the MSC (once finished) into formal Articles of Association & adjunct documents. We'll get the workshop notes written up as a rewrite of MSC1318 very shortly.
New MSCs from vdH on:
- MSC1721 Rename m.login.cas to m.login.sso
- MSC1730 Mechanism for redirecting to an alternative server during login
…and an alternate to MSC1730: MSC1731 Mechanism for redirecting to an alternative server during SSO login.
These three SSO-related MSCs are derived from work being done for a large-scale Matrix deployment for a New Vector customer which we should be able to talk about in future. Improving SSO and other auth/identity work in Matrix is vital, and needless to say we're trying to ensure all the resulting work ends up in the spec to the benefit of the whole ecosystem :)

matrix-media-repo

TravisR reports that:

matrix-media-repo has received a lot of updates in the last few weeks. Although mostly bug fixes and improvements, there's added support for streaming downloads from remote servers. This is mostly useful when someone posts a 100mb video in a room: previously the media repo would download the entire thing then send it over to you. Now it'll send you the video while it also caches it.

matrix-docker-ansible-deploy

Big update from Slavi this week:

matrix-docker-ansible-deploy has received some updates in the past few weeks - fixing minor bugs, improving documentation, adding support for additional Synapse/Riot features, etc.

Most notably (in oldest to most-recent order):

Cadair heard you like playbooks, so he made it possible to run the playbook from another playbook. Thanks Cadair!

some configuration renaming to make things more consistent. Most people would likely be unaffected.

support has been added for Postgres 11. Upgrading is possible and easy (as a related success story, Aaron Raimist shared that he recently upgraded his ~40GB database)

more flexible and more secure-by-default TLS configuration. Thanks Aaron Raimist!

the ability to customize more parts of riot-web. Thanks Aaron Raimist!

better control over Synapse logging. Thanks Aaron Raimist!

the ability to set up a public server and the ability to make use of Synapse's \\"auto-join\\" feature. Until now, the playbook was only suited for setting up a private server, but with this, setting up public servers is now a possibility. Thanks @anadahz

I'm quite happy that the playbook is getting more contributions from others now and is improving in such diverse ways!

Additionally, the #matrix-docker-ansible-deploy:devture.com support room in Matrix is now bridged to the #matrix-docker-ansible-deploy IRC Freenode channel, making it easier for people to get support when they don't have a working homeserver.

libQMatrixClient

kitsune has been working on libQMatrixClient:

the work on lazy-loading in libQMatrixClient has advanced to the point when client authors may start trying to integrate it in their applications (welcome to #qmatrixclient:matrix.org for details - it's a bit rough around the edges yet).

Spectral

This week Black Hat has been working to give Spectral a more Material design oriented UI.

SMSMatrix

swedneck reports that:

SMSMatrix now has a room (#smsmatrix:matrix.org). SMSMatrix is a simple bridge between Matrix and SMS, developed by tijder. You install an android app and register a bot account which will invite you to direct chats under the name of anyone who sends you an SMS, which will then act as a normal SMS conversation.

I had a very quick play with this tool and it works great - a good way to quickly bridge SMS to Matrix.

Riot

News from the Riot teams:

Android rewrite is continuing in Kotlin - lots of infrastructure work over the last few weeks
E2E key backups landed in Riot/iOS develop (but no UI yet, as we finalise the implementation first on Riot/Web)
E2E key backups also landed in Riot/Web develop (but with placeholder UI)
E2E key backups in dev on Riot/Android too (using Realm for storage!)
Lots of work from uhoreg on E2E cross-signing
Work on the Riot/Web redesign paused for a week due to other commitments but as of Nov 26th we're back on the case again, making up for lost time.

Synapse

Most of the focus from the Synapse team this week has been on the new version using Python 3:

Py3 is finished, other than Debian packages - ongoing debate on whether they should use a virtualenv or depend on debian dependencies, and who should maintain them in a Py3 world…
Lots of running around on final Py3 bugs

matrix-appservice-purple

Half-Shot has continued work on matrix-appservice-purple.

Modular

Modular.im provides Hosted Homeservers for Matrix.

Modular now has full support for Custom DNS (letting you point your own SRV records at it)
Lots of work on a migration tool to allow folks to easily shift Matrix accounts between servers, as well as (in future) migrations from HipChat, Slack & others.

That's it!

Apologies for the delayed release this week - please expect an episode of Matrix Live in the week, and back to a normal schedule for TWIM as of Friday. As always, come talk to us in #thisweekinmatrix:matrix.org and tell us what you've been doing.

Synapse 0.33.9 is here!

20.11.2018 00:00 — Releases — Neil Johnson

Well here you are then, the 9th episode in the Synapse 0.33.x series.

Features wise, 0.33.9 contains a change to the way that GDPR consent works under the hood. It is now plumbed in to the login flow (rather than following immediately afterwards) such that it does not inadvertently break on-boarding. This is part of a broader set of changes that span Synapse and Riot to improve initial first impressions of using matrix.

Separately we now have support for room version upgrades which is pre-requisite for rolling out the new state resolution algorithm, come and join us in #teststateresv2:jki.re if you would like to help us test.

Finally we've spent a bunch of time further improving perf especially in and around reducing device ids federation traffic.

I know I say it every time, but full python 3 support is really really close now, matrix.org is now running entirely on py3 and seeing some amazing perf improvements - the remaining blocker is getting py3 deb packages ready and then we'll ship an official python 3 release. There will also be a blog post to explain what we've been up to and what to expect perf wise.