---

Hi all,

Ben's away today, so this TWIM is brought to you mainly in association with Cadair's TWIMbot!

Spec Activity

Since last week's sprint to get the new spec releases out, focus on the core team has shifted exclusively to the remaining stuff needed to cut the first stable release for the Server-Server API.  In practice this means fleshing out the MSCs in flight and implementing them - work has progressed on both improving auth rules, switching event IDs to be hashes and others.  Whilst implementing this in Synapse we're also doing a complete audit and overhaul of the current federation code, hence the 0.33.3.1 security release this week.

Meanwhile, in the community, ma1uta reports:

I am working on the jeon (java matrix api) to update it to the latest stable release. Also I changed versions of api to form rX.Y.Z-N where rX.Y.Z is a API version and N is a library version within API. So, I have prepared Push API (r0.1.0-1), Identity API (r0.1.0-1) and Appservice API (r0.1.0-1) for the first release and current updating the C2S API to the r0.4.0 version.

XMPP Bridging

Are you in the market for a Matrix-XMPP bridge? When I say "market", I mean it because this week we have two announcements for bridging to XMPP! You can choose whether you'd prefer your bridge to be implemented as a puppet, or a bot.

Ma1uta has a new version of his Matrix-Xmpp bridge:

It is a double-puppet bridge which can connects the Matrix and XMPP ecosystems. Just invite the @_xmpp_master:ru-matrix.org and tell him: @_xmpp_master: connect [email protected] to connect current room with the specified conference.
You can ask about this bridge in the #matrix-jabber-java-bridge:ru-matrix.org room.
Currently supports only conferences and only m.text messages. 1:1 conversations and other message types will be later.

maze appeared this week and announced MxBridge, a new Matrix-XMPP bridge:

It works as a bot, so it is non-puppeting. Rooms can be mapped dynamically by the bot administrator(s). There is no support for 1-1 chats (yet). MxBridge is written as a multi-process application in Elixir and it should scale quite well (but don't tie me down on it ;)). https://github.com/djmaze/mxbridge

Seaglass

Neil powers onwards with Seaglass, with updates this week including:

• Displaying stickers
• Lazy-loading room history on startup to help with performance
• Scrollback support (both forwards and backwards)
• Support for Matthew's Account (aka retries on initial sync for those of us with massive initial syncs, and general perf improvements to nicely support >2000 rooms)
• Better avatar support & cosmetics on macOS Mojave
• Encryption verification support, device blacklisting and message information
• Ability to turn encryption on in rooms
• Responding to encryption being turned on in rooms
• Paranoid mode for encryption (only send to verified devices)
• Invitation support (both in UI and /invite)

Matrique

Blackhat announces that Matrique's new design is almost done, along with GNU/Linux, MacOS and Windows nightly build!

Fractal

Alexandre Franke says:

Fractal 3.30 got release alongside the rest of GNOME. It includes a bunch of new and updated translations, and redacted messages are now hidden.

Meanwhile, hidden in this screenshot, uhoreg noted that E2E plans are progressing...

Riot

Bruno has been hacking away on Riot/Web squashing the remaining Lazy Loading Members defects and various related optimisations and fixes. We also released Riot/Web 0.16.3 as a fairly minor point release (which unfortunately has a regression with DM avatars, which is fixed in 0.16.4, for which a first RC was cut a few hours ago and should be released on Monday).  Meanwhile the first cut of Lazy Loading also got implemented on Android as well. Both are hidden behind labs flags, but we're almost at a point where we can turn it on now!  Otherwise, the Riot team has got sucked into working on commercial Matrix stuff, for better or worse (all shall be revealed shortly though!)

Construct

Jason has been working heavily on Construct, and has new test users.  Construct is able to federate with Synapse and the rest of the Matrix ecosystem.  mujx has created a docker for Construct which streamlines its deployment.

Construct development is still occurring here https://github.com/jevolk/charybdis but we are now significantly closer to pushing the first release to https://github.com/matrix-construct. Also feel free to stop by in #test:zemos.net / #zemos-test:matrix.org as well -- a room hosted by Construct, of course.

tulir has now deployed using the standalone install instructions on a very small LXC VM using ZFS. Unfortunately ZFS does not support O_DIRECT (direct disk IO) which is how Construct achieves maximum performance using concurrent reads. This is not a problem though when using an SSD or for personal deployments. I'll be posting more about how Construct hacked RocksDB to use direct IO, which can get the most out of your hardware with multiple requests in-flight (even with an SSD).

Synapse

Work was split this week into spec/security work, with the critical update for 0.33.3.1 - if you haven't upgraded, please do so immediately.

Otherwise, Hawkowl has been on a mission to finish the Python 3 port, which is now almost merged.  Testers should probably wait until it fully merges to the develop branch and we'll yell about it then, but impatient adrenaline enthusiasts may want to check out the hawkowl/py3-3 branch (although it may explode in your face, mangle your DB and format your cat, and probably misses lots of recent important PRs like the 0.33.3.1 stuff).  However, i've been running a variant on some servers for the last few days without problems - and it seems (placebo effect notwithstanding) incredibly snappy...

Meanwhile, the Lazy Loaded Member implementation got sped up by 2-3x, which makes /sync roughly 2-3x faster than it would be without Lazy Loading.  This hasn't merged yet, but was the main final blocker behind Lazy Loading going live!

matrix-docker-ansible-deploy

Slavi reports:

matrix-docker-ansible-deploy now supports bridging to Telegram by installing tulir's mautrix-telegram bridge. This feature is contributed by @izissise.

In addition, Matrix Synapse is now more configurable from the playbook, with support for enabling stats-reporting, event cache size configurability, password peppering.

Matrix Python SDK needs a maintainer

We should say a huge Thank You to &Adam for his work leading the Python SDK over the previous months! Unfortunately due to a busy home life (best of luck for the second child!) he has decided to step down as lead maintainer. Anyone interested in this project should head to https://github.com/matrix-org/matrix-python-sdk/issues/279, and also come and chat in #matrix-python-sdk:matrix.org.

MatrixToyBots!

Coffee reports that:

A new bot appears! Are you a pedantic academic who likes to correct others' misuse of Latin-derived plurals? This task can now be automated for you by means of SingularBot! Also for people who just like to have some fun. Free PongBot and SmileBot included.

kitsune on Hokkaido island

I ended up being on Hokkaido island right when a major earthquake struck it; so no activity on Matrix from me in the nearest couple of days. Also, donations to GlobalGiving for the disaster relief are welcome because people are really struggling here (abusing the communication channel, sorry).

Matrix Live

...has got delayed again; sorry - we're rather overloaded atm. We'll catch up as soon as we can.

Hi All,

As referenced in yesterday's pre-disclosure, today we are releasing Synapse 0.33.3.1 as a critical security update.

We have patched two security vulnerabilities we identified whilst working on the upcoming r0 spec release for the Server-Server API (see details below). We do not believe either have been exploited in the wild, but strongly recommend everybody running a federated Synapse upgrades immediately.

As always you can get the new update here or from any of the sources mentioned at https://github.com/matrix-org/synapse/

Many thanks for your patience and understanding; with fixes like this we are moving ever closer to Synapse reaching a 1.0 Thanks also to the package maintainers who have coordinated with us to ensure distro packages are available for a speedy upgrade!

Note, for anyone running Debian Jessie, we have prepared a 0.33.2.1 deb (as 0.33.3 dropped support for Jessie).

Synapse 0.33.3.1 (2018-09-06)

SECURITY FIXES

• Fix an issue where event signatures were not always correctly validated (#3796)
• Fix an issue where server_acls could be circumvented for incoming events (#3796)

Internal Changes

• Unignore synctl in .dockerignore to fix docker builds (#3802)

Hi all,

During the ongoing work to finalise a stable release of Matrix's Server-Server federation API, we've been doing a full audit of Synapse's implementation and have identified a serious vulnerability which we are going to release a security update to address (Synapse 0.33.3.1) on Thursday Sept 6th 2018 at 12:00 UTC.

We are coordinating with package maintainers to ensure that patched versions of packages will be available at that time - meanwhile, if you run your own Synapse, please be prepared to upgrade as soon as the patched versions are released.  All previous versions of Synapse are affected, so everyone will want to upgrade.

Thank you for your time, patience and understanding while we resolve the issue,

signed_predisclosure.txt

Recently I've been working to improve some of the content on the matrix.org website.

Firstly the FAQ now has updated content and a more presentable menu.

We have a Guides Index, which includes a clarified guide list, plus links to off-site contributions from the community. It's possible to click "recommend" on these items if you've had a good experience with them. If you have documentation or guides you'd like to see added to the list, contact me on Matrix or make a pull request on the site repo.

Finally, as part of a programme to improve visibility on projects in the Matrix ecosystem, we are introducing the "Matrix Clients Matrix". This is a list of some of the most popular current Matrix clients in the ecosystem today, and should shed some light on current feature statuses! The list is not exhaustive, and if you would like to see your client project included, please contact me at the same address, or come chat in the Matrix Client Developers community room. Pretty green Features grid:

Introducing Client Server API 0.4, and the first ever stable IS, AS and Push APIs spec releases!

Hi folks,

As many know, we've been on a massive sprint to improve the spec - both fixing omissions where features have been implemented in the reference servers but were never formalised in the spec, and fixing bugs where the spec has thinkos which stop us from being able to ratify it as stable and thus fit for purpose .

In practice, our target has been to cut stable releases of all the primary Matrix APIs by the end of August - effectively declaring Matrix out of beta, at least at the specification level.  For context: historically only one API has ever been released as stable - the Client Server API, which was the result of a similar sprint back in Jan 2016. This means that the Server Server (SS) API, Identity Service (IS) API, Application Service (AS) API and Push Gateway API have never had an official stable release - which has obviously been problematic for those implementing them.

However, as of the end of Friday Aug 31, we're proud to announce the first ever stable releases of the IS, AS and Push APIs!

To the best of our knowledge, these API specs are now complete and accurately describe all the current behaviour implemented in the reference implementations (sydent, synapse and sygnal) and are fit for purpose. Any deviation from the spec in the reference implementations should probably be considered a bug in the impl. All changes take the form of filling in spec omissions and adding clarifications to the existing behaviour in order to get things to the point that an independent party can implement these APIs without having to refer to anything other than the spec.

This is the result of a lot of work which spans the whole Spec Core Team, but has been particularly driven by TravisR, who has taken the lead on this whole mission to improve the spec.  Huge thanks are due to Travis for his work here, and also massive thanks to everyone who has suffered endured reviewed his PRs and contributed to the releases.  The spec is looking unrecognisably better for it - and Matrix 1.0 is feeling closer than ever!

Alongside the work on the IS/AS/Push APIs, there has also been a massive attempt to plug all the spec omissions in the Client Server API.  Historically the CS API releases have missed some of the newer APIs (and of course always miss the ones which postdate a given release), but we've released the APIs which /have/ been specified as stable in order to declare them stable.  However, in this release we've tried to go through and fill in as many remaining gaps as possible.

The result is the release of Client Server API version 0.4. This is a huge update - increasing the size of the CS API by ~40%. The biggest new stuff includes fully formalising support for end-to-end encryption (thanks to Zil0!), versioning for rooms (so we can upgrade rooms to new versions of the protocol), synchronised read markers, user directories, server ACLs, MSISDN 3rd party ids, and .well-known server discovery (not that it's widely used yet), but for the full picture, best bet is to look at the changelog (now managed by towncrier!).  It's probably fair to say that the CS API is growing alarmingly large at this point - Chrome says that it'd be 223 A4 pages if printed. Our solution to this will be to refactor it somehow (and perhaps switch to a more compact representation of the contents).

Some things got deliberately missed from the CS 0.4 release: particularly membership Lazy Loading (because we're still testing it out and haven't released it properly in the wild yet), the various GDPR-specific APIs (because they may evolve a bit as we refine them since the original launch), finalising ID grammars in the overall spec (because this is surprisingly hard and subtle and we don't want to rush it) and finally Communities (aka Groups), as they are still somewhat in flux.

Meanwhile, on the Server to Server API, there has also been a massive amount of work.  Since the beginning of July it's tripled in size as we've filled in the gaps, over the course of >200 commits (>150 of which from Travis).  If you take a look at the current snapshot it's pretty unrecognisable from the historical draft; with the main changes being:

• Adding the new State Resolution algorithm to address flaws in the original one.  This has been where much of our time has gone - see MSC1442 for full details.  Adopting the new algorithm requires rooms to be recreated; we'll write more about this in the near future when we actually roll it out.
• Adding room versioning so we can upgrade to the new State Resolution algorithm.
• Everything is now properly expressed as Swagger (OpenAPI), just like the CS API
• Adding all the details for E2E encryption (including dependencies like to-device messaging and device-list synchronisation)
• Improvements in specifying how to authorize inbound events over federation
• Document federation APIs such as /event_auth and /query_auth and /get_missing_events
• Document 3rd party invites over federation
• Document the /user/* federation endpoints
• Document Server ACLs
• Document read receipts over federation
• Document presence over federation
• Document typing notifications over federation
• Document content repository over federation
• Document room directory over federation
• ...and many many other minor bug fixes, omission fixes, and restructuring for coherency - see https://github.com/matrix-org/matrix-doc/issues/1464 for an even longer list :)

However, we haven't finished it all: despite our best efforts we're running slightly past the original target of Aug 31.  The current state of play for the r0 release overall (in terms of pending issues) is: ...and you can see the full breakdown over at the public Github project dashboard.

The main stuff we still have remaining on the Server/Server API at this point is:

• Better specifying how we validate inbound events. See MSC1646 for details & progress.
• Switching event IDs to be hashes. See MSC1640 for details and progress.
• Various other remaining security considerations (e.g. how to handle malicious auth events in the DAG; how to better handle DoS situations).
• Merging in the changes to authoring m.room.power_levels (as per MSC1304)
• Formally specifying the remaining identifiers which lack a formal grammar - MSC1597 and particularly room aliases ( MSC1608)

The plan here is to continue speccing and implementing these at top priority (with Travis continuing to work fulltime on spec work), and we'll obviously keep you up-to-date on progress.  Some of the changes here (e.g. event IDs) are quite major and we definitely want to implement them before speccing them, so we're just going to have to keep going as fast as we can. Needless to say we want to cut an r0 of the S2S API alongside the others asap and declare Matrix out of beta (at least at the spec level :)

In terms of visualising progress on this spec mission it's interesting to look at the rate at which we've been closing PRs: this graph shows the total number of PRs which are in state ‘open' or ‘closed' on any given day:

...which clearly shows the original sprint to get the r0 of the CS API out the door at the end 2015, and then a more leisurely pace until the beginning of July 2018 since which the pace has picked up massively.  Other ways of looking at include the number of open issues...

...or indeed the number of commits per week…

...or the overall Github Project activity for August.  (It's impressive to see Zil0 sneaking in there on second place on the commit count, thanks to all his GSoC work documenting E2E encryption in the spec as part of implementing it in matrix-python-sdk!)

Anyway, enough numerology.  It's worth noting that all of the dev for r0 has generally followed the proposed Open Governance Model for Matrix, with the core spec team made up of both historical core team folk (erik, richvdh, dave & matthew), new core team folk (uhoreg & travis) and community folk (kitsune, anoa & mujx) working together to review and approve the changes - and we've been doing MSCs (albeit with an accelerated pace) for anything which we feel requires input from the wider community.  Once the Server/Server r0 release is out the door we'll be finalising the open governance model and switching to a slightly more measured (but productive!) model of spec development as outlined there.

Meanwhile, Matrix 1.0 gets ever closer.  With (almost) all this spec mission done, our plan is to focus more on improving the reference implementations - particularly performance in Synapse,  UX in matrix-{'{'}react,ios,android{'}'}-sdk as used by Riot (especially for E2E encryption), and then declare a 1.0 and get back to implementing new features (particularly Editable Messages and Reactions) at last.

We'd like to thank everyone for your patience whilst we've been playing catch up on the spec, and hope you agree it's been worth the effort :)

Matthew & the core spec team.

The Spec

As many know, we've been aiming for the end of August to cut the first ever stable releases of the remaining APIs in the spec which have up to now been marked unstable - i.e. providing a snapshot of the spec which correctly matches the reference implementations (other than implementation bugs) and which can be used in isolation to develop production grade implementations of clients, servers, etc without need to reference any other implementations. There's been a massive sprint to pull this together, and as of the time of writing there are still PRs and commits landing every few minutes.  We'll post a full update on our progress on Monday; meanwhile you can see a sneak peek over at the August 2018 r0 project board.

Spec work has completely precluded any other backend dev this week.

Half-Shot, gone but not really gone

This week we say farewell to Half-Shot, who has been working fulltime on bridges over the summer. He has managed the matrix.org bridges largely single-handedly, with a big focus on the often-volatile IRC bridge(s).

Bridges

matrix-appservice-irc 0.11.0

It's a big deal, and it's rolling out to IRC networks this week. Half-Shot released matrix-appservice-irc 0.11.0, with the following included:

New features & improvements:

• Cache modes internally
• Replace nicks with user pill mentions
• Kick users if we fail to create an IRC client for them on join (aka ILINE kicks)
• SASL support
• Add err_nononreg so we can announce PMs that failed
• Formatting of replies

Bug Fixes:

• Fix invalidchar nick
• Don't answer any msgtypes other than text in an admin room.
• Fix provisoner leaving users on unlink
• Fixed a bug where content of events the bridge hadn't cached were not being used in replies.
• We were calling authedRequest but the request was not mocked out.
• There was a bug involving intents in m-a-b so it was bumped

Metrics:

• Metrics for MatrixHandler - Iline Kicks
• Idle connection metrics
• QueuePool.waitingItems should use it's internal queue size

Misc:

• Section out tests, linting and coverage into separate stages for Travis

WhatsApp

tulir has been working on mautrix-whatsapp bridge.

now bridges a lot more stuff, such as formatting, media and replies. I'm also almost done with desegregating users so that Matrix users join the same group chat portals rather than everyone having their own portal to the same chat

Zulip chat, bridged by Zulip

Matthew discovered there is a Matrix-Zulip bridge on the Zulip side. So if you're running a Zulip server (for some reason), and want to bridge with Matrix check out the integration docs here.

IRC Connection Tracker

Half-Shot created a new component to enhance the reliability of IRC-Matrix bridging:

IRC Connection Tracker is a thing now. It's a project to separate out the IRC connections from the appservice so the two can be run independently, so that restarting the appservice doesn't affect the IRC connections. It's hopefully going to allow bridge stuff to run faster when it's done.

This project is still really early stage. You can take a look at the Proposal document here.

Clients

Nheko 0.5.4

Not technically this week, but Nheko 0.5.4 was recently released

• The settings page now includes the device id & device fingerprint (thanks @valkum )
• The Polish translation has been updated (thanks @m4sk1n )

Get the latest builds of Nheko from bintray.

Fractal

Alexandre Franke and the GNOME crew have been working on Fractal, gearing up for their upcoming 3.30 release.

Fractal 3.29.92 got released and we are freezing strings to give GNOME translators a bit of time to complete translations for 3.30 next week. Latest developments include tweaks for the title bar, misc bug fixes, a new presentation for uploaded files (that are not images, those are still displayed inline) with buttons to download or open them. Development builds are now parallel installable for easier testing and CI has been improved.

Seaglass

neilalexander:

Seaglass now has some early support for inline images and attachments, and supports Quick Look. Also handles emotes and notices better. It also has version numbers now, various other little fixes and Aaron Raimist has been working on auto-update support.

Version numbers! Now when you go to download the tarball from bintray, you can see what's going on!

There's also been some work on supporting dark mode on Mojave (which looks particularly sexy) and even Touch Bar support!

Riot Android v0.8.15

Riot Android v0.8.15 is on it's way to the Play Store.

Changes in Riot Android 0.8.15 (2018-08-30)

MatrixSdk:

• Upgrade to version 0.9.9.

Improvements:

• Improve intent to open document (#2544)
• Avoid useless dialog for permission (#2331)
• Improve wording when exporting keys (#2289)

Other changes:

• Upgrade lib libphonenumber from v8.0.1 to 8.9.12
• Upgrade Google firebase libs

Bugfix:

• Handle \\/ at the beginning of a message to send a message starting with / (#658)
• Escape nicknames starting with a forward slash / in mentions (#2146)
• Improve management of Push feature
• MatrixError mResourceLimitExceededError is now managed in MxDataHandler (vector-im/riot-android#2547 point 2)

Changes in Riot Android 0.8.14 (2018-08-27)

MatrixSdk:

• Upgrade to version 0.9.8.

Features:

• Manage server quota notices (#2440)

Improvements:

• Do not ask permission to write external storage at startup (#2483)
• Update settings icon and transparent logo for notifications and navigation drawer (#2492)
• URL previews are no longer requested from the server when displaying URL previews is disabled (PR #2514)
• Fix some plural and puzzle strings, and remove other unused ones (#2444)
• Manage System Alerts in a dedicated section

Other changes:

• Upgrade olm-sdk.aar from version 2.2.2 to version 2.3.0
• move PieFractionView from the SDK to the client (#2525)

Bugfix:

• Fix media sharing (#2530)
• Fix notification sound issue in settings (#2524)
• Disable app icon badge for "listen for event" notification (#2104)

Riot iOS 0.7.3

Changes in 0.7.3 (2018-08-27)

Improvements:

• Upgrade MatrixKit version (v0.8.3).

Bug fix:

• Fix input toolbar reset in RoomViewController on MXSession state change (#2006 and #2008).
• Fix user interaction disabled in master view of UISplitViewController when selecting a room (#2005).

Changes in 0.7.2 (2018-08-24)

Improvements:

• Upgrade MatrixKit version (v0.8.2).
• Server Quota Notices in Riot (#1937).

Bug fix:

• User defaults: the preset application language (if any) is ignored.
• Recents: Avoid to open a room twice (it crashed on room creation on quick HSes).
• Riot-bot: Do not try to create a room with it if the user homeserver is not federated.

Riot Web

There's been lots of work debugging and optimising Lazy Loading, which is edging closer to being turned on by default.  We've also been working away at improving E2E UX - starting with finishing key backup, and then improved verification, and then finally cross-signing (at last!)

SDKs

Ruby Matrix SDK

ananace is working on the Ruby Matrix SDK "fixing issues and documenting as I go".

Matrix Android SDK

Changes to Matrix Android SDK in 0.9.9 (2018-08-30)

Improvements:

• Clear unreachable Url when clearing media cache (vector-im/riot-android#2479)
• "In reply to" is not clickable on Riot Android yet. Make it a plain text (vector-im/riot-android#2469)

Bugfix:

• Removing room from 'low priority' or 'favorite' does not work (vector-im/riot-android#2526)
• MatrixError mResourceLimitExceededError is now managed in MxDataHandler (vector-im/riot-android#2547)

API Change:

• MxSession constructor is now private. Please use MxSession.Builder() to create a MxSession

Changes to Matrix Android SDK in 0.9.8 (2018-08-27)

Features:

• Manage server_notices tag and server quota notices (vector-im/riot-android#2440)

Bugfix:

• Room aliases including the '@' and '=' characters are now recognized as valid (vector-im/riot-android#2079, vector-im/riot-android#2542)
• Room name and topic can be now set back to empty (vector-im/riot-android#2345)

API Change:

• Remove PieFractionView class from the Matrix SDK. This class is now in Riot sources (#336)
• MXMediasCache.createTmpMediaFile() methods are renamed to createTmpDecryptedMediaFile()
• MXMediasCache.clearTmpCache() method is renamed to clearTmpDecryptedMediaCache()
• Add MXMediasCache.moveToShareFolder() to move a tmp decrypted file to another folder to prevent deletion during sharing. New API MXMediasCache.clearShareDecryptedMediaCache() can be called when the application is resumed. (vector-im/riot-android#2530)

Matrix iOS SDK

Changes in Matrix iOS SDK in 0.11.3 (2018-08-27)

Bug fix:

• MXJSONModel: Manage m.server_notice empty tag sent due to a bug server side (PR #556).

Changes in Matrix iOS SDK in 0.11.2 (2018-08-24)

Improvements:

• MXSession: Add the supportedMatrixVersions method getting versions of the specification supported by the homeserver.
• MXRestClient: Add testUserRegistration to check earlier if a username can be registered.
• MXSession: Add MXSessionStateSyncError state and MXSession.syncError to manage homeserver resource quota on /sync requests (vector-im/riot-ios/issues/1937).
• MXError: Add kMXErrCodeStringResourceLimitExceeded to manage homeserver resource quota (vector-im/riot-ios/issues/1937).
• MXError: Define constant strings for keys and values that can be found in a Matrix JSON dictionary error.
• Tests: MXHTTPClient_Private.h: Add method to set fake delay in HTTP requests.

Bug fix:

• People tab is empty in the share extension (vector-im/riot-ios/issues/1988).
• MXError: MXError lost NSError.userInfo information.

Tools

matrix-to-riot

Half-Shot shared a handy Firefox extension: matrix-to-riot

This is a tiny webextension to forward matrix.to links to an open Riot tab.

Really useful if you often click matrix.to links and find yourself needing to URL-hack to get to where you need to be.

That's all for now

We're going to shift this week's Matrix Live to Monday, alongside the upcoming blog post on the spec release progress. Have a good weekend!

Thank you to Half-Shot for all your work on Bridges over the last months and beyond. Today is your last day, but I'm sure we'll see you again before long. Text below is from Half-Shot.

---

Today marks my last day of my 3 month internship at New Vector (the startup which hires many of the core Matrix team). For those of you who haven't been reading Ben's fabulous blog posts, I've been working exclusively on bridges; in particular the IRC bridge.

Tasked with the goal of making it crash less and run faster, I hope that the evidence is visible and people are generally having a better experience on it!

Some stats pulled from the matrix-appservice-irc repo:

• 39 PRs closed (4 remain open)
• 27 issues closed, 27 issues opened.
• 334 commits, averaging 7.6 commits a PR.

Commits this year:

---

But aside from showing off some stats, I wanted to mention all the new features:

• Replies on Matrix translate well to IRC, or as well as IRC allows.
• People mentioning your IRC nick now ping your matrix user, finally!
• So. Many. Metrics. Everything you wanted to know about the internals of the bridge, but were too afraid to --inspect.
• Not spamming homeservers with join requests on startup (it makes for a happy ops team).
• No longer are IRC users shackled to a "(IRC)" extension on their displayname, you can be who you want with group flairs!
• Support for node 4 has been dropped, and support for 6,8 and 10 has been assured.

On the matrix-appservice-bridge side, I optimised some calls to cache locally and avoid hitting the homeserver too often, and disabling presence for homeservers that don't support it.

There are future plans to make bridging more visible to Matrix Clients as a first class citizen. Ideas like speccing a state event (MSC1410) so that bridges can interact with each other properly and clients can create full bridge management views which are still decentralised from an integration manager.

I'd like to give a shoutout to Travis who has reviewed nearly all my changes that have made their way into the bridge, on top of all the other tasks he has on his plate. And of course a thank you to all of the Matrix team who have been very supportive during my time here.

Synapse 0.33.3

Big week for Synapse: v0.33.3 was released this week. You'll find preparation for support for Lazy Loading and Room Versioning, lots of bugfixes, and a great contribution from vojeroen: support for sending SNI over federation for vhosted servers. More complete change log here, features below:

• Add support for the SNI extension to federation TLS connections. Thanks to @vojeroen! (#3439)
• Add /_media/r0/config (#3184)
• speed up /members API and add at and membership params as per MSC1227 (#3568)
• implement summary block in /sync response as per MSC688 (#3574)
• Add lazy-loading support to /messages as per MSC1227 (#3589)
• Add ability to limit number of monthly active users on the server (#3633)
• Support more federation endpoints on workers (#3653)
• Basic support for room versioning (#3654)
• Ability to disable client/server Synapse via conf toggle (#3655)
• Ability to whitelist specific threepids against monthly active user limiting (#3662)
• Add some metrics for the appservice and federation event sending loops (#3664)
• Where server is disabled, block ability for locked out users to read new messages (#3670)
• set admin uri via config, to be used in error messages where the user should contact the administrator (#3687)
• Synapse's presence functionality can now be disabled with the "use_presence" configuration option. (#3694)
• For resource limit blocked users, prevent writing into rooms (#3708)

Ananace has upgraded his K8s-optimized Synapse image to 0.33.3

Spec

• A large chunk of e2e has landed in the spec, largely thanks to Zil0's work. There are still some things left, but the main bits related to sending encrypted messages is in there. Attachments and key sharing are being reviewed today
• TravisR progressed on Push, Application Services, and Client-Server APIs.
• To follow along with progress, the best place to look is the August 2018 r0 tracker.

E2E

Key backups are being worked on, and we've had some new thoughts on key verification.

ru-matrix.org

Well are you?

Alexey and ma1uta have a new Russian-language community wiki available at https://ru-matrix.org. This:

accumulates Russian translates of Matrix FAQ and manuals, and other info about Matrix in Russian. Also we starts a public Matrix Synapse server for Russian users with free open registration

Russian Matrix users: feel free to add and complement articles in our wiki to make Matrix more friendly for Russian users.

Client News

Seaglass

neilalexander has been on a roll the last few weeks working on Seaglass, the macOS-native client:

Seaglass now has support for creating, joining and leaving rooms, and accepting/rejecting room invites, redacting messages, improvements to the room settings pages, timestamps, some visual tweaks, lots of bug fixes, early support for detecting failed message sends, scrolling improvements

Matrique

Black Hat continues work on Matrique, a glossy Qt client:

Typing notification and desktop notification of Matrique are working, plus other improvements

See the latest commits here.

Riot Web

Lazy Loading (LL) is available on develop behind the LABS flag, and is being polished. It will probably not make it to next release, but it's progressing!

Riot Mobile

New releases are imminent for both Android and iOS:

• iOS: bug fixes and Lazy Loading in settings
• Android: bug fixes, better management of permission requests at startup

Shared Secret Auth and Matrix Corporal

Slavi has been working on several operations tools for Synapse. Firstly Shared Secret Auth password provider module for Matrix Synapse.

It allows external systems (which know the shared secret) to easily obtain access tokens for any user hosted on the homeserver. This is incredibly useful for all sorts of automation (no more pre-generated access tokens or plain-text passwords).

With this tool, you can pass a mxid, and get back an access_token for that user. With this, you can do whatever actions would normally be done with a token. For example, you for a new starter at a company, you could join a set of rooms known to be needed by new starters.

Support room for Shared Secret Auth: #matrix-synapse-shared-secret-auth:devture.com

Next, he has released Matrix Corporal, reconciliator and gateway for a managed Matrix server:

It's a way to take control of Matrix (Synapse) in a corporate (or other such) environment.
Based on a configuration policy (generated by an external system, like your intranet), it can auto-create and auto-disable users, keep their profile and authentication details to date, auto-join/leave users to communities and rooms, etc.
It's also meant to sit in front of the Matrix Synapse Client API, inspecting and allowing/denying requests, in accordance with the configuration policy (preventing people from leaving certain rooms/communities, preventing them from messing around with their profile details, etc.)

Support room for Matrix Corporal: #matrix-corporal:devture.com

Note that Slavi also maintains matrix-docker-ansible-deploy, which can deploy these projects along with Synapse itself.

Bots

twimbot

I was extremely young when I first read and understood Larry Wall's explanation about the virtue of laziness, but I did not expect it to lead here. Cadair has created (I think inspired by an idea from TravisR), a "twimbot". This bot works in conjunction with opsdroid and the opsdroid connector-matrix - it works by storing selected messages from #twim:matrix.org in room state. In this way, it either assists the author by making it easier to produce the blog post you're reading, or it coldly replaces the author, leading to his sacking and penury. To view updates currently held by the bot, join #twim_updates:cadair.com.

Hashtag bot

When he's not working away on the Matrix Spec, TravisR continues to work on less [essential|sane] Matrix projects. This week we have @hashtag:t2bot.io, a bot that converts #hashtags to Matrix groups.

Why? Because why not. Source available on github: https://github.com/turt2live/matrix-hashtag-bot

Bridges

matrix-appservice-bridge and matrix-appservice-irc

Half-Shot will only be working full time with the Matrix Core team for another week (!!!), but until then he's going full-speed with bridge work. First matrix-appservice-irc:

Just pushed out matrix-appservice-irc 0.11.0-rc3 which contains a lot of nice things. The full changelog is there, hoping to get some of this tested and running on the matrix.org servers soon, but initial testing looks good :)

Having worked so much on IRC bridging the last few months, Half-Shot has this week managed to make improvements to matrix-appservice-bridge:

Released matrix-appservice-bridge 1.6.0 which contains mostly things I was writing for the irc bridge and factored out instead. There's a couple of goodies like automatic caching of profiles/events, and getEvent as a way to fetch events from the homeserver without fiddling with context.

Full changelog: https://github.com/matrix-org/matrix-appservice-bridge/blob/1.6.0/CHANGES.md

Some Intent operations now cache requests that would otherwise fall through to the homeserver which can be expensive. This is configurable for Intents via the opts.caching.ttl and size options. AgeCounters now allow you to set your own time period buckets. Added a new function Intent.getEvent which will fetch events from the homeserver without any context information, which should be quick. MembershipCache is now exposed to let folks read and write to the cache while also letting the bridge access it.

a release: Fix issue where roomState would fail. b release: Fix issue where we were calling this.intent inside intent like fools! c release: Fix issue where some stole js-sdk code was not checked thoroughly.

WhatsApp

tulir announces that mautrix-whatsapp "is now somewhat usable for basic chatting".

It uses the go-whatsapp library to talk with the WhatsApp Web API. Using the web API means that you'll still need WhatsApp on your phone connected to the internet, but it also means you won't get banned for using the bridge.
The bridge is still very alpha, but basic message bridging and some advanced features already work. There's a features & roadmap document about that.

To come chat about the WhatsApp bridge: #whatsapp:maunium.net.

mxtoot

mxtoot is a Matrix-Mastodon bot from ma1uta, creator of Jeon and the ecosystem around it.

There were two releases last week:

0.4.6 with additional commands (follow, unfollow, mute, unmute, block, unblock users, show followers and followings of the specified user) and a little bugfix.

0.4.7 is out now, which increases the length of messages to 4000 characters.

GSOC is finished!

GSOC 2018 has been valuable for everyone involved!

• Thank you zil0 for all your work on the Matrix Python SDK, E2E and and spec work.
• Thank you APWhiteHat: lots of work on dendrite, including a bunch of stuff on federation, as well as typing notifications!
• and thank you to Cadair for helping to organising it!

The Construct using asynchronous disk IO

Jason writes:

the Construct has advanced to fully asynchronous disk IO in a single-threaded environment.
RocksDB is a highly configurable and customizable database with a very large callback surface to support many different environments, which is good to connect it with something like AIO.
Normally RocksDB gets its performance from concurrent standard POSIX system calls (blocking IO) with multiple threads. The callback surface and its internals are not specifically suited for deviating from the pthread model. However I hacked RocksDB and tricked it into believing that my userspace contexts, which mimic the std::thread/pthread interface, are threads.
This is an example of a RocksDB callback to read from a file, it expects this callback to return immediately (i.e as soon as the kernel returns from a blocking read(2).) Except, where I call fs::read() that is actually a stack-suspension which makes a request to linux AIO where the stack resumes after AIO has called back with the data (see: https://github.com/jevolk/charybdis/blob/master/ircd/aio.cc)
No modifications to the internals of RocksDB were necessary. The Construct is being developed here: https://github.com/jevolk/charybdis.

Community Guides Index

Some weeks ago I asked for suggestions for content to add to a "Community Guides Index". There was not a wave of feedback, but we have enough collected content to be able to share a first pass of this Index. Check out the content there, and please please contact me if you have articles, videos or anything else you'd like to see included. Of course, if you'd like to produce some content and have it included, that would be great too! Ideas for missing articles:

• any content in a language other than English
• DevOps, how to use k8s or ansible to install Synapse
• how to use the Application Services API to make a bot

OggCamp 2018

OggCamp last weekend was great - massive thank you to Jon for organising and running the show - we love your work!

Presentations from Ben and Half-Shot seemed to go down well, we all chatted with Open Source fans and generally flew the flag for Matrix.

New Rooms

• tulir has enough projects that he has created #:maunium.net, for any project of his that doesn't have it's own room
• Support room for Shared Secret Auth: #matrix-synapse-shared-secret-auth:devture.com
• Support room for Matrix Corporal: #matrix-corporal:devture.com
• If you prefer the cold output of a machine, #twim_updates:cadair.com shows you the updates as they're posted in #twim:matrix.org.
• WhatsApp bridge: #whatsapp:maunium.net

So long…

Hope you had a good week, hope you enjoyed reading this post! Amandine is back after some needed vacation time, and has this week's Matrix Live below.

All the threes, Synapse 0.33.3!

This release brings together a lot of bugfixes, and also some preparation for support for Lazy Loading and Room Versioning.

We also have, as a great contribution from @vojeroen, SNI extension support! With v0.33.3, Synapse now supports sending SNI over federation for vhosted servers, which resolves this long-standing request.

As always, you can get the new update from https://github.com/matrix-org/synapse/releases/tag/v0.33.3 or any of the sources mentioned at https://github.com/matrix-org/synapse.

Features

• Add support for the SNI extension to federation TLS connections. Thanks to @vojeroen! (#3439)
• Add /_media/r0/config (#3184)
• speed up /members API and add at and membership params as per MSC1227 (#3568)
• implement summary block in /sync response as per MSC688 (#3574)
• Add lazy-loading support to /messages as per MSC1227 (#3589)
• Add ability to limit number of monthly active users on the server (#3633)
• Support more federation endpoints on workers (#3653)
• Basic support for room versioning (#3654)
• Ability to disable client/server Synapse via conf toggle (#3655)
• Ability to whitelist specific threepids against monthly active user limiting (#3662)
• Add some metrics for the appservice and federation event sending loops (#3664)
• Where server is disabled, block ability for locked out users to read new messages (#3670)
• set admin uri via config, to be used in error messages where the user should contact the administrator (#3687)
• Synapse's presence functionality can now be disabled with the "use_presence" configuration option. (#3694)
• For resource limit blocked users, prevent writing into rooms (#3708)

Bugfixes

• Fix occasional glitches in the synapse_event_persisted_position metric (#3658)
• Fix bug on deleting 3pid when using identity servers that don't support unbind API (#3661)
• Make the tests pass on Twisted < 18.7.0 (#3676)
• Don't ship recaptcha_ajax.js, use it directly from Google (#3677)
• Fixes test_reap_monthly_active_users so it passes under postgres (#3681)
• Fix mau blocking calculation bug on login (#3689)
• Fix missing yield in synapse.storage.monthly_active_users.initialise_reserved_users (#3692)
• Improve HTTP request logging to include all requests (#3700, #3723)
• Avoid timing out requests while we are streaming back the response (#3701)
• Support more federation endpoints on workers (#3705, #3713)
• Fix "Starting db txn 'get_all_updated_receipts' from sentinel context" warning (#3710)
• Fix bug where state_cache cache factor ignored environment variables (#3719)

Deprecations and Removals

• The Shared-Secret registration method of the legacy v1/register REST endpoint has been removed. For a replacement, please see the admin/register API documentation. (#3703)

Internal Changes

• The test suite now can run under PostgreSQL. (#3423)
• Refactor HTTP replication endpoints to reduce code duplication (#3632)
• Tests now correctly execute on Python 3. (#3647)
• Sytests can now be run inside a Docker container. (#3660)
• Port over enough to Python 3 to allow the sytests to start. (#3668, #3732)
• Update docker base image from alpine 3.7 to 3.8. (#3669)
• Rename synapse.util.async to synapse.util.async_helpers to mitigate async becoming a keyword on Python 3.7. (#3678)
• Synapse's tests are now formatted with the black autoformatter. (#3679)
• Implemented a new testing base class to reduce test boilerplate. (#3684)
• Rename MAU prometheus metrics (#3690)
• add new error type ResourceLimit (#3707)
• Logcontexts for replication command handlers (#3709)
• Update admin register API documentation to reference a real user ID. (#3712)

This feels like a bumper (extra big) post, so let's get straight into it!

Spec

TravisR continues to plow through work in documenting, clarifying and confirming the spec. You can check out his project on Github: August 2018 r0, which should give an idea of both the scale of the project and the amount of work which has been done. Rather than list out individual items as I have been, please take a look at that project, and come chat in #matrix-spec:matrix.org if you'd like to contribute.

Clients

Riot web

Some exciting things on Riot Web this week!

Bruno has merged his work on lazy loading room members lists, which should mean we see some big performance improvements, especially in larger rooms. You can expect to see that released on /develop soon.

He has also just started to work on the riot redesign, and has begun on some of the more visible changes like resizeable panels:

More news on the redesign to come soon!

nheko 0.5.3 released

nheko 0.5.3 was released this week. From the changelog:

Features

• Add option to disable desktop notifications
• Allow user to configure join rules for a room
• Add tab-completion for usernames

Improvements

• Remove the space gap taken by the typing notifications
• Remove hover event from emoji picker
• Add tooltips for the message indicators
• Fix compilation on FreeBSD
• Update Polish translation
• Small modal improvements

Seaglass builds being generated

Seaglass, the MacOS-native client, now has automatically generated builds available on bintray.

Seaglass is now a very usable and stable client on macOS. neilalexander has been very active this week:

Seaglass has had various updates, including the groundwork for joining and leaving channels, LOADS of crashes fixed, some more work done on avatars and E2E encryption has been fairly well road-tested now Still need to complete UI for device verification and setting E2E but it works fine in existing channels

Quaternion

kitsune has been working on Quaternion:

Quaternion has gained a new way to highlight mentions - with tinted background instead of colored text. Also, the majority of work on rooms grouping is done, and the current master branch features the roomlist grouped by tag (but rooms under each tag are pretty much out of order so it's not really ready for daily use yet).

Quaternion is looking for a macOS packager - if someone has the chance to help out in this area, speak to kitsune in #QMatrixClient.

Palaver

Arne came at us with the v0.2.1 release of Palaver:

A few days back Palaver has been moved to Gitlab. And I have since released v0.2.0 and v0.2.1. A runnable jar-file of the latest release can be downloaded at https://gitlab.com/MrCustomizer/palaver/tags/v0.2.1. The biggest changes in v0.2.x are the replacement of all the web views with native JavaFX components (as I don't feel comfortable embedding a whole browser stack in a desktop application) and support for read markers. There is a short YouTube video demonstrating the read marker implementation:

Riot mobile

This week, work continued on Lazy Loading room members for Riot Android, and progress toward a release for 0.7.0 on iOS.

neo client

f0x has begun rewriting Neo using matrix-js-sdk.

Matrique

Black Hat has been working on Matrique:

Matrique now has a (basic) room management panel, a working emoji picker, an unread marker and supports playing "m.audio". It should be available at Flathub any time.

SDKs and such

Jeon, Jmsdk, Gene, Matrix-Jabber-Bridge: ma1uta-world

ma1uta is continuing to make progress on his Java-focused collection of APIs and SDKs, he's eagerly awaiting the r0 release of the spec (aren't we all!)

Jeon (https://github.com/ma1uta/jeon) (client2server and application api) has released on the Maven Central Repository with version 0.4.1. The next goal: the stable release of all apis 1.0.0 after the Matrix spec will be released (I hope it will be soon :)) Jmsdk (https://github.com/ma1uta/jmsdk) the client and bot SDK have released on the Maven Central Repository. Gene (https://github.com/ma1uta/gene) the lightweight api for android has released on Maven Central Repository with 0.2.1 version. So, everyone can use this libraries without additional settings and additional repositories. Matrix-Jabber-Bridge (https://github.com/ma1uta/matrix-jabber-java-bridge) the double-pupped bridge between the Matrix and Jabber. I am still working on it and I think it will be released in a few weeks.

Ruby SDK v0.0.3

Ananace has tagged and released the next development version of the Ruby Matrix SDK - 0.0.3. He has plans to "finish up documentation and unit testing", with the aim of making a v1.0.0 release.

mautrix-appservice-go

tulir has a new, yet-to-be-revealed project, and in support of this he's been working again on mautrix-appservice-go:

mautrix-appservice-go is a bit more active again: it now has an initial intent API similar to the one in mautrix-appservice-python (which in turn is based on the intent API from matrix-appservice-bridge).

Synapse

No release this week, but Synapse has been progressing:

• Python 3 port continues at pace, we expect to have a beta to test in monolith mode rsn (Hawkowl leads this)
• Erik has been working on some federation API refactoring to make matrix.org snappier and hopes to have the new state resolution algo ready to go if not deployed late next week
• Richvdh has been looking at performance, as well as spec PRs/ Next week will be r0 work

Many Synapse maintainers are also doing huge lifts on development of the spec.

Matrix support now in ntfy

Half-Shot took a rare 30-minute window away from bridge maintenance to add Matrix support to ntfy. ntfy describes itself as

A utility for sending notifications, on demand and when commands finish

but in fact, you also send notifications whenever you like by calling ntfy send from anywhere you'd normally execute shell commands. This will make it really easy to integrate Matrix notifications into any other application!

matrix-docker-ansible-deploy

Slavi brought updates to matrix-docker-ansible-deploy:

matrix-docker-ansible-deploy got a lot of improvements lately:

• email-sending support, so that Matrix Synapse can send email notifications (enabled by default)
• mxisd Identity Server support (enabled by default)
• more detailed docs split into various sections (used to be a single README, but with more and more features it was turning into a wall of text)
• started keeping a changelog, especially for backward-incompatible things

the playbook can now also set up and configure Maximus's matrix-synapse-rest-auth

Riot Translations

@csybr:dodsorf.as and @dandellion:dodsorf.as appeared and announced that there is now a Norwegian Nynorsk translation available for Riot Web, matrix-react-sdk and riot-android. Non-Matrix-related aside: as a lazy Brit, I was vaguely aware of the cultural weight language carries in the history of Scandinavia, but didn't realise that Norway still has some fundamental differences ongoing.

Giveth

The Giveth project have produced a bot built on matrix: giveth-bot

For some more context:

Giveth is re-engineering charitable giving, by creating an entirely free, open-source platform, built on the Ethereum Blockchain.

The current giveth-bot:

handles welcoming people, can answer questions and handle our crazy points dishing system to reward contributors!

mautrix-telegram

mautrix-telegram now has an official 0.3.0 release. Says tulir:

No major changes since the release candidates last week, except a Telethon update that might have fixed history backfill.

"The End of GSOC"

Well, it may well be the end of the Julian's GSOC 2018 experience with the Fractal project, but I think Google will continue to run the programme. Check out Julian's experience (and the large number of issues resolved!) in this blog post.

OggCamp

This weekend Neil and I will be representing Matrix at OggCamp, "an unconference celebrating Free Culture, Free and Open Source Software, hardware hacking, digital rights, and all manner of collaborative cultural activities", if you will be there, come find and chat to us. We'll be the two nerdy-looking guys, so we should be easy to spot.

We'll meet again…

But for now, you can watch Neil host Matrix Live below (using the fan-favourite format of walking around the office), and come chat to us in #twim:matrix.org