This Week In Matrix 2018-09-07


Hi all,

Ben’s away today, so this TWIM is brought to you mainly in association with Cadair’s TWIMbot!

Spec Activity

Since last week’s sprint to get the new spec releases out, focus on the core team has shifted exclusively to the remaining stuff needed to cut the first stable release for the Server-Server API.  In practice this means fleshing out the MSCs in flight and implementing them – work has progressed on both improving auth rules, switching event IDs to be hashes and others.  Whilst implementing this in Synapse we’re also doing a complete audit and overhaul of the current federation code, hence the 0.33.3.1 security release this week.

Meanwhile, in the community, ma1uta reports:

I am working on the jeon (java matrix api) to update it to the latest stable release. Also I changed versions of api to form rX.Y.Z-N where rX.Y.Z is a API version and N is a library version whithin API. So, I have prepared Push API (r0.1.0-1), Identity API (r0.1.0-1) and Appservice API (r0.1.0-1) for the first release and current updating the C2S API to the r0.4.0 version.

XMPP Bridging

Are you in the market for a Matrix-XMPP bridge? When I say “market”, I mean it because this week we have two announcements for bridging to XMPP! You can choose whether you’d prefer your bridge to be implemented as a puppet, or a bot.

Ma1uta has a new version of his Matrix-Xmpp bridge:

It is a double-puppet bridge which can connects the Matrix and XMPP ecosystems. Just invite the @_xmpp_master:ru-matrix.org and tell him: @_xmpp_master: connect [email protected] to connect current room with the specified conference.
You can ask about this bridge in the #matrix-jabber-java-bridge:ru-matrix.org room.
Currently supports only conferences and only m.text messages. 1:1 conversations and other message types will be later.

maze appeared this week and announced MxBridge, a new Matrix-XMPP bridge:

It works as a bot, so it is non-puppeting. Rooms can be mapped dynamically by the bot administrator(s). There is no support for 1-1 chats (yet). MxBridge is written as a multi-process application in Elixir and it should scale quite well (but don’t tie me down on it ;)). https://github.com/djmaze/mxbridge

Seaglass

Neil powers onwards with Seaglass, with updates this week including:

  • Displaying stickers
  • Lazy-loading room history on startup to help with performance
  • Scrollback support (both forwards and backwards)
  • Support for Matthew’s Account (aka retries on initial sync for those of us with massive initial syncs, and general perf improvements to nicely support >2000 rooms)
  • Better avatar support & cosmetics on macOS Mojave
  • Encryption verification support, device blacklisting and message information
  • Ability to turn encryption on in rooms
  • Responding to encryption being turned on in rooms
  • Paranoid mode for encryption (only send to verified devices)
  • Invitation support (both in UI and /invite)

Matrique

Blackhat announces that Matrique’s new design is almost done, along with GNU/Linux, MacOS and Windows nightly build!

Fractal

Alexandre Franke says:

Fractal 3.30 got release alongside the rest of GNOME. It includes a bunch of new and updated translations, and redacted messages are now hidden.

Meanwhile, hidden in this screenshot, uhoreg noted that E2E plans are progressing…

Riot

Bruno has been hacking away on Riot/Web squashing the remaining Lazy Loading Members defects and various related optimisations and fixes. We also released Riot/Web 0.16.3 as a fairly minor point release (which unfortunately has a regression with DM avatars, which is fixed in 0.16.4, for which a first RC was cut a few hours ago and should be released on Monday).  Meanwhile the first cut of Lazy Loading also got implemented on Android as well. Both are hidden behind labs flags, but we’re almost at a point where we can turn it on now!  Otherwise, the Riot team has got sucked into working on commercial Matrix stuff, for better or worse (all shall be revealed shortly though!)

Construct

Jason has been working heavily on Construct, and has new test users.  Construct is able to federate with Synapse and the rest of the Matrix ecosystem.  mujx has created a docker for Construct which streamlines its deployment.

Construct development is still occurring here https://github.com/jevolk/charybdis but we are now significantly closer to pushing the first release to https://github.com/matrix-construct. Also feel free to stop by in #test:zemos.net / #zemos-test:matrix.org as well — a room hosted by Construct, of course.

tulir has now deployed using the standalone install instructions on a very small LXC VM using ZFS. Unfortunately ZFS does not support O_DIRECT (direct disk IO) which is how Construct achieves maximum performance using concurrent reads. This is not a problem though when using an SSD or for personal deployments. I’ll be posting more about how Construct hacked RocksDB to use direct IO, which can get the most out of your hardware with multiple requests in-flight (even with an SSD).

Synapse

Work was split this week into spec/security work, with the critical update for 0.33.3.1 – if you haven’t upgraded, please do so immediately.

Otherwise, Hawkowl has been on a mission to finish the Python 3 port, which is now almost merged.  Testers should probably wait until it fully merges to the develop branch and we’ll yell about it then, but impatient adrenaline enthusiasts may want to check out the hawkowl/py3-3 branch (although it may explode in your face, mangle your DB and format your cat, and probably misses lots of recent important PRs like the 0.33.3.1 stuff).  However, i’ve been running a variant on some servers for the last few days without problems – and it seems (placebo effect notwithstanding) incredibly snappy…

Meanwhile, the Lazy Loaded Member implementation got sped up by 2-3x, which makes /sync roughly 2-3x faster than it would be without Lazy Loading.  This hasn’t merged yet, but was the main final blocker behind Lazy Loading going live!

matrix-docker-ansible-deploy

Slavi reports:

matrix-docker-ansible-deploy now supports bridging to Telegram by installing tulir‘s mautrix-telegram bridge. This feature is contributed by @izissise.

In addition, Matrix Synapse is now more configurable from the playbook, with support for enabling stats-reporting, event cache size configurability, password peppering.

Matrix Python SDK needs a maintainer

We should say a huge Thank You to &Adam for his work leading the Python SDK over the previous months! Unfortunately due to a busy home life (best of luck for the second child!) he has decided to step down as lead maintainer. Anyone interested in this project should head to https://github.com/matrix-org/matrix-python-sdk/issues/279, and also come and chat in #matrix-python-sdk:matrix.org.

MatrixToyBots!

Coffee reports that:

A new bot appears! Are you a pedantic academic who likes to correct others’ misuse of Latin-derived plurals? This task can now be automated for you by means of SingularBot! Also for people who just like to have some fun. Free PongBot and SmileBot included.

kitsune on Hokkaido island

I ended up being on Hokkaido island right when a major earthquake struck it; so no activity on Matrix from me in the nearest couple of days. Also, donations to GlobalGiving for the disaster relief are welcome because people are really struggling here (abusing the communication channel, sorry).

Matrix Live

…has got delayed again; sorry – we’re rather overloaded atm. We’ll catch up as soon as we can.

Pre-disclosure: Upcoming critical security fix for Synapse

Hi all,

During the ongoing work to finalise a stable release of Matrix’s Server-Server federation API, we’ve been doing a full audit of Synapse’s implementation and have identified a serious vulnerability which we are going to release a security update to address (Synapse 0.33.3.1) on Thursday Sept 6th 2018 at 12:00 UTC.

We are coordinating with package maintainers to ensure that patched versions of packages will be available at that time – meanwhile, if you run your own Synapse, please be prepared to upgrade as soon as the patched versions are released.  All previous versions of Synapse are affected, so everyone will want to upgrade.

Thank you for your time, patience and understanding while we resolve the issue,

signed_predisclosure.txt

Matrix Spec Update August 2018

Introducing Client Server API 0.4, and the first ever stable IS, AS and Push APIs spec releases!

Hi folks,

As many know, we’ve been on a massive sprint to improve the spec – both fixing omissions where features have been implemented in the reference servers but were never formalised in the spec, and fixing bugs where the spec has thinkos which stop us from being able to ratify it as stable and thus fit for purpose .

In practice, our target has been to cut stable releases of all the primary Matrix APIs by the end of August – effectively declaring Matrix out of beta, at least at the specification level.  For context: historically only one API has ever been released as stable – the Client Server API, which was the result of a similar sprint back in Jan 2016. This means that the Server Server (SS) API, Identity Service (IS) API, Application Service (AS) API and Push Gateway API have never had an official stable release – which has obviously been problematic for those implementing them.

However, as of the end of Friday Aug 31, we’re proud to announce the first ever stable releases of the IS, AS and Push APIs!


To the best of our knowledge, these API specs are now complete and accurately describe all the current behaviour implemented in the reference implementations (sydent, synapse and sygnal) and are fit for purpose. Any deviation from the spec in the reference implementations should probably be considered a bug in the impl. All changes take the form of filling in spec omissions and adding clarifications to the existing behaviour in order to get things to the point that an independent party can implement these APIs without having to refer to anything other than the spec.

This is the result of a lot of work which spans the whole Spec Core Team, but has been particularly driven by TravisR, who has taken the lead on this whole mission to improve the spec.  Huge thanks are due to Travis for his work here, and also massive thanks to everyone who has suffered endured reviewed his PRs and contributed to the releases.  The spec is looking unrecognisably better for it – and Matrix 1.0 is feeling closer than ever!

Alongside the work on the IS/AS/Push APIs, there has also been a massive attempt to plug all the spec omissions in the Client Server API.  Historically the CS API releases have missed some of the newer APIs (and of course always miss the ones which postdate a given release), but we’ve released the APIs which /have/ been specified as stable in order to declare them stable.  However, in this release we’ve tried to go through and fill in as many remaining gaps as possible.

The result is the release of Client Server API version 0.4. This is a huge update – increasing the size of the CS API by ~40%. The biggest new stuff includes fully formalising support for end-to-end encryption (thanks to Zil0!), versioning for rooms (so we can upgrade rooms to new versions of the protocol), synchronised read markers, user directories, server ACLs, MSISDN 3rd party ids, and .well-known server discovery (not that it’s widely used yet), but for the full picture, best bet is to look at the changelog (now managed by towncrier!).  It’s probably fair to say that the CS API is growing alarmingly large at this point – Chrome says that it’d be 223 A4 pages if printed. Our solution to this will be to refactor it somehow (and perhaps switch to a more compact representation of the contents).

Some things got deliberately missed from the CS 0.4 release: particularly membership Lazy Loading (because we’re still testing it out and haven’t released it properly in the wild yet), the various GDPR-specific APIs (because they may evolve a bit as we refine them since the original launch), finalising ID grammars in the overall spec (because this is surprisingly hard and subtle and we don’t want to rush it) and finally Communities (aka Groups), as they are still somewhat in flux.

Meanwhile, on the Server to Server API, there has also been a massive amount of work.  Since the beginning of July it’s tripled in size as we’ve filled in the gaps, over the course of >200 commits (>150 of which from Travis).  If you take a look at the current snapshot it’s pretty unrecognisable from the historical draft; with the main changes being:

  • Adding the new State Resolution algorithm to address flaws in the original one.  This has been where much of our time has gone – see MSC1442 for full details.  Adopting the new algorithm requires rooms to be recreated; we’ll write more about this in the near future when we actually roll it out.
  • Adding room versioning so we can upgrade to the new State Resolution algorithm.
  • Everything is now properly expressed as Swagger (OpenAPI), just like the CS API
  • Adding all the details for E2E encryption (including dependencies like to-device messaging and device-list synchronisation)
  • Improvements in specifying how to authorize inbound events over federation
  • Document federation APIs such as /event_auth and /query_auth and /get_missing_events
  • Document 3rd party invites over federation
  • Document the /user/* federation endpoints
  • Document Server ACLs
  • Document read receipts over federation
  • Document presence over federation
  • Document typing notifications over federation
  • Document content repository over federation
  • Document room directory over federation
  • …and many many other minor bug fixes, omission fixes, and restructuring for coherency – see https://github.com/matrix-org/matrix-doc/issues/1464 for an even longer list :)

However, we haven’t finished it all: despite our best efforts we’re running slightly past the original target of Aug 31.  The current state of play for the r0 release overall (in terms of pending issues) is:…and you can see the full breakdown over at the public Github project dashboard.

The main stuff we still have remaining on the Server/Server API at this point is:

  • Better specifying how we validate inbound events. See MSC1646 for details & progress.
  • Switching event IDs to be hashes. See MSC1640 for details and progress.
  • Various other remaining security considerations (e.g. how to handle malicious auth events in the DAG; how to better handle DoS situations).
  • Merging in the changes to authoring m.room.power_levels (as per MSC1304)
  • Formally specifying the remaining identifiers which lack a formal grammar – MSC1597 and particularly room aliases (MSC1608)

The plan here is to continue speccing and implementing these at top priority (with Travis continuing to work fulltime on spec work), and we’ll obviously keep you up-to-date on progress.  Some of the changes here (e.g. event IDs) are quite major and we definitely want to implement them before speccing them, so we’re just going to have to keep going as fast as we can. Needless to say we want to cut an r0 of the S2S API alongside the others asap and declare Matrix out of beta (at least at the spec level :)

In terms of visualising progress on this spec mission it’s interesting to look at the rate at which we’ve been closing PRs: this graph shows the total number of PRs which are in state ‘open’ or ‘closed’ on any given day:

…which clearly shows the original sprint to get the r0 of the CS API out the door at the end 2015, and then a more leisurely pace until the beginning of July 2018 since which the pace has picked up massively.  Other ways of looking at include the number of open issues…


…or indeed the number of commits per week…


…or the overall Github Project activity for August.  (It’s impressive to see Zil0 sneaking in there on second place on the commit count, thanks to all his GSoC work documenting E2E encryption in the spec as part of implementing it in matrix-python-sdk!)


Anyway, enough numerology.  It’s worth noting that all of the dev for r0 has generally followed the proposed Open Governance Model for Matrix, with the core spec team made up of both historical core team folk (erik, richvdh, dave & matthew), new core team folk (uhoreg & travis) and community folk (kitsune, anoa & mujx) working together to review and approve the changes – and we’ve been doing MSCs (albeit with an accelerated pace) for anything which we feel requires input from the wider community.  Once the Server/Server r0 release is out the door we’ll be finalising the open governance model and switching to a slightly more measured (but productive!) model of spec development as outlined there.

Meanwhile, Matrix 1.0 gets ever closer.  With (almost) all this spec mission done, our plan is to focus more on improving the reference implementations – particularly performance in Synapse, UX in matrix-{react,ios,android}-sdk as used by Riot (especially for E2E encryption), and then declare a 1.0 and get back to implementing new features (particularly Editable Messages and Reactions) at last.

We’d like to thank everyone for your patience whilst we’ve been playing catch up on the spec, and hope you agree it’s been worth the effort :)

Matthew & the core spec team.

Synapse 0.33.0 is here!!

Hi all,

We’ve just released Synapse 0.33.0!  This is a major performance upgrade which speeds up /sync (i.e. receiving messages) by a factor of almost 2x!  This has already made a massive difference to the CPU usage and snappiness of the matrix.org homeserver since we rolled it out a few days ago – you can see the drop in sync worker CPU just before midday on July 17th; previously we were regularly hitting the CPU ceiling (at which point everything grinds to a halt) – now we’re back down hovering between 40% and 60% CPU (at the current load).  This is actually fixing a bug which crept in around Synapse 0.31, so please upgrade – especially if Synapse has been feeling slower than usual recently, and especially if you are still on Synapse 0.31.

Meanwhile we have a lot of new stuff coming on the horizon – a whole new algorithm for state resolution (watch this space for details); incremental state resolution (at last!) to massively speed up state resolution and mitigate extremities build up (and speed up the synapse master process, which is now the bottleneck again on the matrix.org homeserver); better admin tools for managing resource usage, and all the Python3 porting work (with associated speedups and RAM & GC improvements).  Fun times ahead!

The full changelog follows below; as always you can grab Synapse from https://github.com/matrix-org/synapse.   Thanks for flying Matrix!

Synapse 0.33.0 (2018-07-19)

Bugfixes

  • Disable a noisy warning about logcontexts. (#3561)

Synapse 0.33.0rc1 (2018-07-18)

Features

  • Enforce the specified API for report_event. (#3316)
  • Include CPU time from database threads in request/block metrics. (#3496#3501)
  • Add CPU metrics for _fetch_event_list. (#3497)
  • Optimisation to make handling incoming federation requests more efficient. (#3541)

Bugfixes

  • Fix a significant performance regression in /sync. (#3505#3521#3530#3544)
  • Use more portable syntax in our use of the attrs package, widening the supported versions. (#3498)
  • Fix queued federation requests being processed in the wrong order. (#3533)
  • Ensure that erasure requests are correctly honoured for publicly accessible rooms when accessed over federation. (#3546)

Misc

This Week in Matrix 2018-07-13

Welcome to TWIM for Friday the 13th! Nothing too scary, but lots of client updates and news.

Quaternion

kitsune:

Quaternion has gained a new /html command that allows one to send raw HTML. Note that an actually displayed HTML subset entirely depends on the receiving client; no HTML validation or sanitation is done; a plaintext version is automatically created by stripping all the HTML tags.

neo

f0x is working on Neo again:

added emotes, working localecho and /rainbow

rainbows :D

matrix-logger

A useful tool from CromFr for anyone who wants to archive logs of rooms.

I wrote a small matrix “client” for logging messages in joined rooms, and store them in a format very similar to Weechat logs.
Source code (Rust) Binary release (x86_64 windows & linux)
Please open issues if encounter any bug or need specific features.\nAlso I’m new to rust, so any review / advices would be appreciated :)

Matrique client progress

Black Hat has been working on Matrique, and it’s looking really good! Screenshot below.

Matrique now has support for sending/receiving messages (plaintext, markdown, HTML, rainbow), emotes and notices. It also supports receiving images, videos, files and states. I am still working on file receiving functions.

I love that all clients include rainbow messages.

fluffychat for Ubuntu Touch

New to me, fluffychat is a “Simple Matrix Messenger for Ubuntu Touch“. The design philosophy for this app is focused on simplicity, and the hope is for it to complement uMatriks, another Ubuntu Touch client.

Why are you not just contributing to uMatriks?
uMatriks is great and it’s superb, that someone has created a Matrix Client for Ubuntu Touch. But sometimes you have a so detailed vision of a user interface, which you want to implement, that you can not just contribute to an existing project. However, I would like to work with the uMatriks developers together. We could use the same push gateway for example.

Riot/Web

  • Welcomed Bruno onto the Riot/Web team! (and said farewell to Luke)
  • Lots of bug blitzing
  • Riot 0.16 due next week, with Jitsi-everywhere (at last) and Slate for composer.
  • We now have an end-to-end test harness (via puppeteer!) at last – being applied at first to fix onboarding bugs.
  • Lazy loading members in progress

Riot/Mobile

  • Major perf work ongoing
  • Decoupling members from state…
  • …and then lazyloading members.

Synapse 0.32.2 available

v0.32.2 is now available, which includes the m.room.server_acl security feature discussed in a previous blog post.

  • If you’re not running 0.32, please upgrade!!
  • Python 3 progresses!
  • Quite a lot of ops work to keep matrix.org alive. Status page coming soon! (We promise!!)
  • Looking at metrics and quotas for better experiences on limited-resource homeservers.

Spec

  • State res fix
  • Version bump
  • Lots of omission PRs landing
  • TravisR S2S work
  • Work on both IS & AS API too
  • Uhoreg work – .well-known, and 3pid logins
  • Some internal discussions about improving the proposal process; particularly having a 5 day ‘final comment period’ similar to Rust’s. Once we’ve got alignment within the core spec team we’ll MSC it.

Dendrite

  • Anoa & APwhitehat continue to blitz onwards on AS and federation
  • thirdparty protocol support for application services in the works.
  • continued progress on getting the AS query APIs supported. PR for /alias still being reviewed, PR for /users happened this week and now being reviewed
  • Timestamp massaging for application services ready and in review
  • APWhiteHat did a good chunk on federation, now migrating to supporting the various EDUs, starting with m.typing events.
  • …running out of low hanging fruit…

AS

  • matrix-appservice-irc 0.9.1 is here with perf improvements!
  • Lots of work on trying to make it start up sooner

E2E

  • UISI stats
  • Device list syncing
  • Encrypted backup

Debian Packages

andrewsh has packaged synapse 0.32.2 for Debian

uhoreg has packaged quaternion 0.0.9.2 for Debian

That’s all folks

Watch Matrix Live below, and see you next week!