This Week in Matrix

🔗Nheko 0.6.0 released!

Get latest stable releases from GitHub.

🔗Features

• Support for sending & receiving markdown formatted messages.
• Import/Export of megolm session keys. (Incompatible with Riot)
• macOS: The native emoji picker can be used.
• Context menu option to show the raw text message of an event.
• Rooms with unread messages are marked in the room list.
• Clicking on a user pill link will open the user profile.

🔗Spec Proposals: E2E Cross-signing and bi-directional key verification

uhoreg has written up a new work-in-progress proposal for E2E cross-signing.

Also, although it wasn't this week, I don't think that we have previously announced the proposal for bi-directional key verification using QR codes: https://github.com/matrix-org/matrix-doc/pull/1544

🔗Jeon

Massive update from ma1uta about his Jeon project! This update brings Jeon into line with the most recent updates to the Client-Server, Application Service, Push and Identity APIs.

In ma1uta's words:

Jeon is a set of the java interfaces and classes which describes the Matrix API.

• client-api: r0.4.0-1 corresponds to the r0.4.0 C2S API.
• application-api: r0.1.0-1 corresponse to the r0.1.0 AS API,
• push-api: r0.1.0-1 corresponds to the r0.1.0 PUSH API,
• identity-api: r0.1.0-1 corresponds to the r0.1.0 IS API.
• All artefacts available from the Maven Central Repository.

Major changes:

• Full support for the corresponding Matrix api.
• Changed version for displaying the Matrix api version.
• Added support to the asynchronous responses.

Also updated the swagger schemas generated from the code:

• https://ma1uta.github.io/swagger/c2s.json - C2S API
• https://ma1uta.github.io/swagger/as.json - AS API
• https://ma1uta.github.io/swagger/push.json - PUSH API
• https://ma1uta.github.io/swagger/id.json - IS API

And the first hotfix: application-api r0.1.0-2 with fixed url (/transactions has been changes to the _matrix/app/v1/transactions). And this release will break all AS because synapse sends transactions to the old url.

Join #jeon:matrix.org to discuss the progress of this product more.

🔗VoIP signalling support has landed in libQMatrixClient

Exciting times for libQMatrixClient!! Thanks kitsune, developer of libQMatrixClient and Quaternion:

After some pretty long time of being in a PR/fork, VoIP signalling support has landed in libQMatrixClient! Many thanks to mariogrip (the founder of UBports) for the initial code and to delijati (a developer behind uMatriks) for getting it to work with the most recent library.
The actual VoIP stack does not come included, client developers have to take whatever WebRTC implementation is available for their platform and glue the pieces together. However, as the example of uMatriks shows, this now becomes relatively easy if your platform is on good terms with WebRTC (like UBports). I look forward to further work with UBports community on keeping this platform a first-class Matrix citizen.

Go chat in #quaternion:matrix.org to see the ways libQMatrixClient is developing.

🔗Matrique is now Spectral

After intense discussion, there is a new name for Matrique: Spectral. The repo now sites at https://gitlab.com/spectral-im/spectral, there is a new room at #spectral:encom.eu.org, and a new logo:

🔗FluffyChat featured in OpenStore

FluffyChat is getting some love from OpenStore, the official Ubuntu Touch app store: this week it was the featured app.

🔗Matrix Corporal

@slavi:devture.com, creator of Matrix Corporal (a Matrix server configuration tool - "Kubernetes for Matrix"):

Matrix Corporal has received some updates over the past few weeks since its initial release: a couple of additional HTTP APIs for retrieving/destroying user access tokens; more consistency with the Matrix Client-Server specification when it comes to error responses; faster reconciliation for user accounts that are joined to many/large rooms.

🔗matrix-docker-ansible-deploy

Another project from @slavi:devture.com, for those who prefer their DevOps ansible-flavoured:

matrix-docker-ansible-deploy now also helps you set up service discovery as per the .well-known Matrix specification.

🔗jcg ansible PR for matrix notifications

jcg has an upstream PR to have matrix notifications in ansible. Combined with Slavi's matrix-docker-ansible-deploy above, you can get Matrix notifications about issues with your Matrix deployment…

🔗Seaglass E2E + self-update

neilalexander:

Seaglass end-to-end encryption support is now complete, including device verification and blacklisting, key sharing requests, key import and export (which should be compatible with Riot) and re-requesting keys

This is really exciting news for macOS matrix users!

I'm also working on auto-updating Seaglass with Aaron Raimist's help in addition to finishing E2E support :-)

:-) is right!

🔗ma1uta jmsdk

ma1uta must have been in a work-on-Matrix mode this week, because he has also updated jmsdk:

I have released a new version of the java client (https://github.com/ma1uta/jmsdk/tree/master/client-sdk). The new client works is asynchronous mode, each method doesn't block the thread and return the CompletableFuture (promise in java). Then you can block thread to get the response or build a asynchronous promises chain.

Finally:
ma1uta is also looking forward to the release of Java 11:

with the Curve25519 key agreement (http://openjdk.java.net/jeps/324) and will try make a pure java implementation of the olm/megolm. Just for fun. :)

🔗synapse-purge

Maze, seeing that his synapse database was already at several gigabytes, decided to produce a tool to shrink it:

The synapse-purge tool allows homeserver admins to free disk space by purging old room events from the synapse database. It is an alternative for synpurge which currently does not work correctly.

Configuration is minimalistic at the moment. Meaning it purges all remote rooms on the server with a globally configured preservation period.

🔗synapse

andrewsh: 0.33.4 uploaded to Debian's stretch-backports, pending approval.

0.33.5rc1 is now available, with the big news being the inclusion of support for Python 3.5 and 3.6! Hawkowl's Py3 has merged for monoliths and is working pretty well, looking like 2-3x RAM improvement. Please help us test!

Erik's state compressor tool is pretty much finished, we've been starting to run it on things and it reduces disk usage for the state group table by at least 10x.

The only catch is that it's quite DB heavy whilst it runs, so we haven't run it on Matrix.org yet.

🔗Fractal

Alexandre Franke and the Fractal team:

refactoring of the history and other parts is going on in the master branch of Fractal. We also cleaned up build and dependency related bits.

🔗maubot and sedbot

tulir:

I made some updates to maubot and fixed most of the sedbot (S. Edbot) issues people had reported.

tulir used maubot to create a factorial bot: >

I might also make some useful bots soon

And so it was - late breaking news that maubot has been used to develop a Dictionary-definition-bot! Not available for public use yet but it proves that the project is useful!

🔗Riot Web

Lazy Loading remains the focus, we're getting closer with more bugs solved this week! To enable Lazy Loading room members and get speed and memory benefits in Riot, use the develop branch and enable "Lazy Loading" under "Labs" in the settings.

Lots of final bug hunting for lazy loading - this is taking longer than you might expect because we're doing end-to-end CI everywhere.

Lots of work on E2E, Dave has been working on:

UI for e2e key backup that's waiting for some lower level bits
and hopefully our e2e core code is moving from asm.js to webassembly making it, by current estimations, significantly faster.

Redesign work continuing as well - Bruno has been working on it this week, Jouni the designer will visit next week to continue the process.

Nad has joined us to help with design bandwidth and is working on the onboarding flows for the redesign as well as fixing all the UX issues in Communities!

🔗Riot Mobile + Mobile SDKs

Lots of work on Lazy Loading - to be released along with Riot Web.

🔗Bridges

Half-Shot is joining us to work part-time on bridges going forwards - this is great news, especially for his connection-based IRC bridging antics as well as catching up on the PR and maintenance backlog for the IRC bridge and Slack bridge.

🔗Modular

Modular (Hosted Homeservers) has first customers; if you want to give it a go please let us know!

🔗Finally

Thanks for reading, take a look at Matrix Live below!

🔗Dimension

Update (this got lost in the original post; sorry Travis!): Dimension received a security update - if you run your own Dimension instance it is strongly recommended you update right away. Telegram bridge support in Dimension is underway, with more updates expected for next week in Matrix.

🔗Clients

🔗Fluffychat

It's been some months since we checked in with FluffyChat. If you're a Ubuntu Touch user, or have a device running it, you should see the progress that has been made recently on this Matrix client. Collected changelog 0.5.0 to latest (0.5.4):

• Search chats
• Chat avatars
• Search users in chats
• Security & Privacy settings:

1. Disable typing notifications
2. Auto-accept invitations

• New message status:

1. Sending: Activity indicator
2. Sent: Little cloud
3. Received: Tick
4. Seen by someone: Usericon

• Display stickers
• Minor UI improvements
• FluffyChat now automatically opens the link to the matrix.org consens
• Updated translations

🔗Seaglass

Neil has been keeping up the pace with Seaglass development:

Seaglass has had a substantial rewrite to the room cache to help improve reliability and reduce crashes, better thumbnail behaviour on inline images, various tiny visual tweaks, in-window blending, support for encryption key sharing requests for E2E rooms.

Rendering performance has been massively increased (if you ignore the occasional bug). Resizing the window shouldn't be so slow anymore and a lot of avatar image operations are no longer repeated unnecessarily

Other than that this week has mostly featured lots and lots of bug fixes, hopefully lots of crashes fixed.

Screenshot below shows the new E2E UI:

🔗Quaternion

When not escaping typhoons, kitsune has found some time to continue work on Quaternion:

Quaternion's master branch is alive again - it's been prone to crashes in the last two weeks, now it shouldn't. Feel free to try the new room list organised by tag!

🔗SimpleMatrix

MTRNord has been working on SimpleMatrix:

SimpleMatrix now supports Basic messages sending (with Commonmark) and basic receiving of messages.

🔗miniVector

Marcus has re-packaged miniVector for F-Droid:

There's now a second matrix client available in F-Droid: https://f-droid.org/packages/com.lavadip.miniVector/

This is a fork of Riot Android done by hrjet, f-droid release done by me. It's removing mostly jitsi group call functionality and some other smaller stuff. In doing so it manages to require far less permissions and is also only 12 MB in size instead of riots 20 MB.

🔗Matrique

Black Hat:

Matrique gained alpha support for multiple accounts

This is thanks to leaning on libqmatrixclient's native multiple account support!

🔗Riot Web 0.16.4 released

This is pretty much a maintenance release - fixing the DM avatar regression that crept in with 0.16.3, adding better support for warning users when their client hasn't yet synced with the server, and the final bits of work needed before we can turn on membership Lazy Loading in the upcoming Riot 0.17.

Full changelogs as always are split over the three projects which make up Riot/Web:

• https://github.com/vector-im/riot-web/blob/master/CHANGELOG.md#changes-in-0164-2018-09-10
• https://github.com/matrix-org/matrix-react-sdk/blob/master/CHANGELOG.md#changes-in-0134-2018-09-10
• https://github.com/matrix-org/matrix-js-sdk/blob/master/CHANGELOG.md#changes-in-0110-rc1-2018-09-07

🔗SDKs and Libraries

🔗libQMatrixClient ecosystem

As you may know, Matrique, led by Black Hat, and Quaternion, led by kitsune, are both projects build using libQMatrixClient, a Qt5 library from kitsune designed for writing Matrix clients. While kitsune has been working on the library for some time, Black Hat has also now started making contributions:

libQMatrixClient now has a pkg-config file to further ease clients building on Linux systems, as well as more convenient API to track read markers if all users, not just of the local one.

🔗matrix-js-sdk v0.11.0 released

This release contains support for lazy loading room members, and also some breaking changes relating to startClient().

• Support for lazy loading members. This should improve performance for users who joined big rooms a lot. Pass to lazyLoadMembers = true option when calling startClient.
• MatrixClient::startClient now returns a Promise. No method should be called on the client before that promise resolves. Before this method didn't return anything.
• A new CATCHUP sync state, emitted by MatrixClient#"sync" and returned by MatrixClient::getSyncState(), when doing initial sync after the ERROR state. See MatrixClient documentation for details.
• RoomState::maySendEvent('m.room.message', userId) & RoomState::maySendMessage(userId) do not check the membership of the user anymore, only the power level. To check if the syncing user is allowed to write in a room, use Room::maySendMessage() as RoomState is not always aware of the syncing user's membership anymore, in case lazy loading of members is enabled.

🔗Synapse

Synapse 0.33.4 was released, with a whole host of bug fixes, some enhancements to resource usage management and a bunch of internal changes in readiness for room member state lazy loading and our ongoing port to Python 3.

Meanwhile, Python 3 support for monolithic (non-worker) Synapses has finally landed on the develop branch, thanks to massive work from hawkowl and notafile - if you want to help us test and flush out any remaining byte/utf8 style errors, please create a virtualenv for python 3.6 or 3.5 (twisted doesn't support 3.7 yet) and point the develop branch of Synapse at it, tail the logs for ERRORs and report them via Github if/when you see them.  In practice it seems pretty stable though, and noticeably reduces RAM and speeds things up thanks to improved GC and general performance work in Python.

We've also discovered that jemalloc works very well at improving RAM usage on Python 2 under Linux (we haven't tried it on Python 3 yet) by providing a more fragmentation-resistant malloc implementation; if you are having problems with your Synapse RAM spiking up we recommend giving it a go.  All of the Matrix.org server is using it now.

Also, lots of ops work this week; Erik has prototyped a new storage strategy for state groups which shrinks storage requirements by 10x, we'll be applying this shortly to Matrix.org otherwise we're going to run out of disk space.  There was also a regression on Synapse develop on federation, where outbound requests would get stuck and never retry - this impacted the matrix.org server badly over the course of the week, but as of Friday night we have a workaround in place.  We're not aware of it affecting anyone other than the matrix.org deployment (and we haven't got to the root cause yet).

🔗Construct homeserver progress

This week:

Added notification counts which show up in Riot now, and expanded support for g++-7 and 8 instead of just g++-6. Construct repository is found at: https://github.com/matrix-construct/construct.

🔗IRC Connection Tracker

Half-Shot is continuing to work on the project to split out IRC connection management from the IRC bridge, letting the bridge be restarted without interrupting IRC connections!

The project is going quite well, and is going to be used on matrix.org once production ready which will really speed up upgrades and give us near zero downtime indifferent to the size of the bridge.

At the moment the project has the ability to spin up and maintain connections, however the connections are not supporting IRC fully yet as there are bits to do on the parsing and maintaining state side. There is also work on a top-like tool to visualise and control the service outside of the bridge so we can quickly handle any oddities if they come up. Finally, it allows you to hot reload the configuration without dropping existing connections!

On the work done to support this on matrix-appservice-bridge, there is basic support for stating connections on the bridge but it's in early stages at the moment.

🔗Spec

Travis has been tidying up loose bits on the Matrix spec this week:

• experimenting with splitting out the "Room" specific bits of the spec into a dedicated section rather than being split between the CS and SS APIs;
• trying to pin down a formal grammar for room aliases
• ...and identifiers in general.

In practice, finalising the S2S API is now blocked on proving the implementation on Synapse; work on this will resume next week and then we'll document the end result and ship the r0 at last.  Timings are going to be completely determined by available manpower and what level of ops distractions we face (c.f. the Synapse section above...).  Whilst we're waiting for the final S2S details to get hashed out, Travis is going to be helping on Riot dev, to try to stop stuff like this, as there's no point in having the platonic ideal of a perfect spec if actual users are unable to benefit from it.

🔗#matrix-dev

#matrix-dev:matrix.org was reborn as a new room a couple of weeks ago to flush out old corrupted events, but maybe not everyone knows. Come join #matrix-dev:matrix.org, it's a starting point for all developers looking to build on the platform.  We're also rebuilding #test:matrix.org and #riot:matrix.org, although once we ship the new state resolution

🔗A sneak peek at Modular...

Finally, there's been a massive amount of work on the New Vector side of things to soft-launch Modular - a paid hosting platform for Matrix servers (and, in future, paid integrations).  At this point we're looking for early adopters who want a dedicated Riot+Synapse for communities or companies of 50 or more users - but don't want to have to run it themselves.  Modular takes the homeserver hosting we've already been providing for Status, TADHack and others, and turns it into a mass-market product.  The pricing for early adopters is over 5x cheaper than Slack, so if you've been dying to have a reliable, fast and expertly maintained homeserver without any of the headaches of admining one yourself, please head over to https://modular.im and give it a whirl and let us know how it goes!  This is also a great way to support Matrix development in general, as money from Modular will directly keep the core Matrix team funded to work on Matrix.  Once we're happy with the soft-launch and have incorporated any feedback we'll start yelling about it as loud as we can :)

🔗Matrix Live

We've had a bit of an accidental hiatus on Matrix Live thanks to getting submerged all the different project endgames happening atm (spec releases, lazy loading, Modular, Riot redesign etc), and for the last few Fridays we've got to midnight and beyond with too much still on the todo list to justify recording a video.  But to avoid completely falling behind, here's a slightly exhausted Saturday morning update instead (warning: Matthew is not a morning person).

---

Hi all,

Ben's away today, so this TWIM is brought to you mainly in association with Cadair's TWIMbot!

🔗Spec Activity

Since last week's sprint to get the new spec releases out, focus on the core team has shifted exclusively to the remaining stuff needed to cut the first stable release for the Server-Server API.  In practice this means fleshing out the MSCs in flight and implementing them - work has progressed on both improving auth rules, switching event IDs to be hashes and others.  Whilst implementing this in Synapse we're also doing a complete audit and overhaul of the current federation code, hence the 0.33.3.1 security release this week.

Meanwhile, in the community, ma1uta reports:

I am working on the jeon (java matrix api) to update it to the latest stable release. Also I changed versions of api to form rX.Y.Z-N where rX.Y.Z is a API version and N is a library version within API. So, I have prepared Push API (r0.1.0-1), Identity API (r0.1.0-1) and Appservice API (r0.1.0-1) for the first release and current updating the C2S API to the r0.4.0 version.

🔗XMPP Bridging

Are you in the market for a Matrix-XMPP bridge? When I say "market", I mean it because this week we have two announcements for bridging to XMPP! You can choose whether you'd prefer your bridge to be implemented as a puppet, or a bot.

Ma1uta has a new version of his Matrix-Xmpp bridge:

It is a double-puppet bridge which can connects the Matrix and XMPP ecosystems. Just invite the @_xmpp_master:ru-matrix.org and tell him: @_xmpp_master: connect [email protected] to connect current room with the specified conference.
You can ask about this bridge in the #matrix-jabber-java-bridge:ru-matrix.org room.
Currently supports only conferences and only m.text messages. 1:1 conversations and other message types will be later.

maze appeared this week and announced MxBridge, a new Matrix-XMPP bridge:

It works as a bot, so it is non-puppeting. Rooms can be mapped dynamically by the bot administrator(s). There is no support for 1-1 chats (yet). MxBridge is written as a multi-process application in Elixir and it should scale quite well (but don't tie me down on it ;)). https://github.com/djmaze/mxbridge

🔗Seaglass

Neil powers onwards with Seaglass, with updates this week including:

• Displaying stickers
• Lazy-loading room history on startup to help with performance
• Scrollback support (both forwards and backwards)
• Support for Matthew's Account (aka retries on initial sync for those of us with massive initial syncs, and general perf improvements to nicely support >2000 rooms)
• Better avatar support & cosmetics on macOS Mojave
• Encryption verification support, device blacklisting and message information
• Ability to turn encryption on in rooms
• Responding to encryption being turned on in rooms
• Paranoid mode for encryption (only send to verified devices)
• Invitation support (both in UI and /invite)

🔗Matrique

Blackhat announces that Matrique's new design is almost done, along with GNU/Linux, MacOS and Windows nightly build!

🔗

🔗Fractal

Alexandre Franke says:

Fractal 3.30 got release alongside the rest of GNOME. It includes a bunch of new and updated translations, and redacted messages are now hidden.

Meanwhile, hidden in this screenshot, uhoreg noted that E2E plans are progressing...

🔗Riot

Bruno has been hacking away on Riot/Web squashing the remaining Lazy Loading Members defects and various related optimisations and fixes. We also released Riot/Web 0.16.3 as a fairly minor point release (which unfortunately has a regression with DM avatars, which is fixed in 0.16.4, for which a first RC was cut a few hours ago and should be released on Monday).  Meanwhile the first cut of Lazy Loading also got implemented on Android as well. Both are hidden behind labs flags, but we're almost at a point where we can turn it on now!  Otherwise, the Riot team has got sucked into working on commercial Matrix stuff, for better or worse (all shall be revealed shortly though!)

🔗Construct

Jason has been working heavily on Construct, and has new test users.  Construct is able to federate with Synapse and the rest of the Matrix ecosystem.  mujx has created a docker for Construct which streamlines its deployment.

Construct development is still occurring here https://github.com/jevolk/charybdis but we are now significantly closer to pushing the first release to https://github.com/matrix-construct. Also feel free to stop by in #test:zemos.net / #zemos-test:matrix.org as well -- a room hosted by Construct, of course.

tulir has now deployed using the standalone install instructions on a very small LXC VM using ZFS. Unfortunately ZFS does not support O_DIRECT (direct disk IO) which is how Construct achieves maximum performance using concurrent reads. This is not a problem though when using an SSD or for personal deployments. I'll be posting more about how Construct hacked RocksDB to use direct IO, which can get the most out of your hardware with multiple requests in-flight (even with an SSD).

🔗Synapse

Work was split this week into spec/security work, with the critical update for 0.33.3.1 - if you haven't upgraded, please do so immediately.

Otherwise, Hawkowl has been on a mission to finish the Python 3 port, which is now almost merged.  Testers should probably wait until it fully merges to the develop branch and we'll yell about it then, but impatient adrenaline enthusiasts may want to check out the hawkowl/py3-3 branch (although it may explode in your face, mangle your DB and format your cat, and probably misses lots of recent important PRs like the 0.33.3.1 stuff).  However, i've been running a variant on some servers for the last few days without problems - and it seems (placebo effect notwithstanding) incredibly snappy...

Meanwhile, the Lazy Loaded Member implementation got sped up by 2-3x, which makes /sync roughly 2-3x faster than it would be without Lazy Loading.  This hasn't merged yet, but was the main final blocker behind Lazy Loading going live!

🔗matrix-docker-ansible-deploy

Slavi reports:

matrix-docker-ansible-deploy now supports bridging to Telegram by installing tulir's mautrix-telegram bridge. This feature is contributed by @izissise.

In addition, Matrix Synapse is now more configurable from the playbook, with support for enabling stats-reporting, event cache size configurability, password peppering.

🔗Matrix Python SDK needs a maintainer

We should say a huge Thank You to &Adam for his work leading the Python SDK over the previous months! Unfortunately due to a busy home life (best of luck for the second child!) he has decided to step down as lead maintainer. Anyone interested in this project should head to https://github.com/matrix-org/matrix-python-sdk/issues/279, and also come and chat in #matrix-python-sdk:matrix.org.

🔗MatrixToyBots!

Coffee reports that:

A new bot appears! Are you a pedantic academic who likes to correct others' misuse of Latin-derived plurals? This task can now be automated for you by means of SingularBot! Also for people who just like to have some fun. Free PongBot and SmileBot included.

🔗kitsune on Hokkaido island

I ended up being on Hokkaido island right when a major earthquake struck it; so no activity on Matrix from me in the nearest couple of days. Also, donations to GlobalGiving for the disaster relief are welcome because people are really struggling here (abusing the communication channel, sorry).

🔗Matrix Live

...has got delayed again; sorry - we're rather overloaded atm. We'll catch up as soon as we can.

🔗The Spec

As many know, we've been aiming for the end of August to cut the first ever stable releases of the remaining APIs in the spec which have up to now been marked unstable - i.e. providing a snapshot of the spec which correctly matches the reference implementations (other than implementation bugs) and which can be used in isolation to develop production grade implementations of clients, servers, etc without need to reference any other implementations. There's been a massive sprint to pull this together, and as of the time of writing there are still PRs and commits landing every few minutes.  We'll post a full update on our progress on Monday; meanwhile you can see a sneak peek over at the August 2018 r0 project board.

Spec work has completely precluded any other backend dev this week.

🔗Half-Shot, gone but not really gone

This week we say farewell to Half-Shot, who has been working fulltime on bridges over the summer. He has managed the matrix.org bridges largely single-handedly, with a big focus on the often-volatile IRC bridge(s).

🔗Bridges

🔗matrix-appservice-irc 0.11.0

It's a big deal, and it's rolling out to IRC networks this week. Half-Shot released matrix-appservice-irc 0.11.0, with the following included:

New features & improvements:

• Cache modes internally
• Replace nicks with user pill mentions
• Kick users if we fail to create an IRC client for them on join (aka ILINE kicks)
• SASL support
• Add err_nononreg so we can announce PMs that failed
• Formatting of replies

Bug Fixes:

• Fix invalidchar nick
• Don't answer any msgtypes other than text in an admin room.
• Fix provisoner leaving users on unlink
• Fixed a bug where content of events the bridge hadn't cached were not being used in replies.
• We were calling authedRequest but the request was not mocked out.
• There was a bug involving intents in m-a-b so it was bumped

Metrics:

• Metrics for MatrixHandler - Iline Kicks
• Idle connection metrics
• QueuePool.waitingItems should use it's internal queue size

Misc:

• Section out tests, linting and coverage into separate stages for Travis

🔗WhatsApp

tulir has been working on mautrix-whatsapp bridge.

now bridges a lot more stuff, such as formatting, media and replies. I'm also almost done with desegregating users so that Matrix users join the same group chat portals rather than everyone having their own portal to the same chat

🔗Zulip chat, bridged by Zulip

Matthew discovered there is a Matrix-Zulip bridge on the Zulip side. So if you're running a Zulip server (for some reason), and want to bridge with Matrix check out the integration docs here.

🔗IRC Connection Tracker

Half-Shot created a new component to enhance the reliability of IRC-Matrix bridging:

IRC Connection Tracker is a thing now. It's a project to separate out the IRC connections from the appservice so the two can be run independently, so that restarting the appservice doesn't affect the IRC connections. It's hopefully going to allow bridge stuff to run faster when it's done.

This project is still really early stage. You can take a look at the Proposal document here.

🔗Clients

🔗Nheko 0.5.4

Not technically this week, but Nheko 0.5.4 was recently released

• The settings page now includes the device id & device fingerprint (thanks @valkum )
• The Polish translation has been updated (thanks @m4sk1n )

Get the latest builds of Nheko from bintray.

🔗Fractal

Alexandre Franke and the GNOME crew have been working on Fractal, gearing up for their upcoming 3.30 release.

Fractal 3.29.92 got released and we are freezing strings to give GNOME translators a bit of time to complete translations for 3.30 next week. Latest developments include tweaks for the title bar, misc bug fixes, a new presentation for uploaded files (that are not images, those are still displayed inline) with buttons to download or open them. Development builds are now parallel installable for easier testing and CI has been improved.

🔗Seaglass

neilalexander:

Seaglass now has some early support for inline images and attachments, and supports Quick Look. Also handles emotes and notices better. It also has version numbers now, various other little fixes and Aaron Raimist has been working on auto-update support.

Version numbers! Now when you go to download the tarball from bintray, you can see what's going on!

There's also been some work on supporting dark mode on Mojave (which looks particularly sexy) and even Touch Bar support!

🔗Riot Android v0.8.15

Riot Android v0.8.15 is on it's way to the Play Store.

🔗Changes in Riot Android 0.8.15 (2018-08-30)

MatrixSdk:

• Upgrade to version 0.9.9.

Improvements:

• Improve intent to open document (#2544)
• Avoid useless dialog for permission (#2331)
• Improve wording when exporting keys (#2289)

Other changes:

• Upgrade lib libphonenumber from v8.0.1 to 8.9.12
• Upgrade Google firebase libs

Bugfix:

• Handle \\/ at the beginning of a message to send a message starting with / (#658)
• Escape nicknames starting with a forward slash / in mentions (#2146)
• Improve management of Push feature
• MatrixError mResourceLimitExceededError is now managed in MxDataHandler (vector-im/riot-android#2547 point 2)

🔗Changes in Riot Android 0.8.14 (2018-08-27)

MatrixSdk:

• Upgrade to version 0.9.8.

Features:

• Manage server quota notices (#2440)

Improvements:

• Do not ask permission to write external storage at startup (#2483)
• Update settings icon and transparent logo for notifications and navigation drawer (#2492)
• URL previews are no longer requested from the server when displaying URL previews is disabled (PR #2514)
• Fix some plural and puzzle strings, and remove other unused ones (#2444)
• Manage System Alerts in a dedicated section

Other changes:

• Upgrade olm-sdk.aar from version 2.2.2 to version 2.3.0
• move PieFractionView from the SDK to the client (#2525)

Bugfix:

• Fix media sharing (#2530)
• Fix notification sound issue in settings (#2524)
• Disable app icon badge for "listen for event" notification (#2104)

🔗Riot iOS 0.7.3

🔗Changes in 0.7.3 (2018-08-27)

Improvements:

• Upgrade MatrixKit version (v0.8.3).

Bug fix:

• Fix input toolbar reset in RoomViewController on MXSession state change (#2006 and #2008).
• Fix user interaction disabled in master view of UISplitViewController when selecting a room (#2005).

🔗Changes in 0.7.2 (2018-08-24)

Improvements:

• Upgrade MatrixKit version (v0.8.2).
• Server Quota Notices in Riot (#1937).

Bug fix:

• User defaults: the preset application language (if any) is ignored.
• Recents: Avoid to open a room twice (it crashed on room creation on quick HSes).
• Riot-bot: Do not try to create a room with it if the user homeserver is not federated.

🔗Riot Web

There's been lots of work debugging and optimising Lazy Loading, which is edging closer to being turned on by default.  We've also been working away at improving E2E UX - starting with finishing key backup, and then improved verification, and then finally cross-signing (at last!)

🔗SDKs

🔗Ruby Matrix SDK

ananace is working on the Ruby Matrix SDK "fixing issues and documenting as I go".

🔗Matrix Android SDK

🔗Changes to Matrix Android SDK in 0.9.9 (2018-08-30)

Improvements:

• Clear unreachable Url when clearing media cache (vector-im/riot-android#2479)
• "In reply to" is not clickable on Riot Android yet. Make it a plain text (vector-im/riot-android#2469)

Bugfix:

• Removing room from 'low priority' or 'favorite' does not work (vector-im/riot-android#2526)
• MatrixError mResourceLimitExceededError is now managed in MxDataHandler (vector-im/riot-android#2547)

API Change:

• MxSession constructor is now private. Please use MxSession.Builder() to create a MxSession

🔗Changes to Matrix Android SDK in 0.9.8 (2018-08-27)

Features:

• Manage server_notices tag and server quota notices (vector-im/riot-android#2440)

Bugfix:

• Room aliases including the '@' and '=' characters are now recognized as valid (vector-im/riot-android#2079, vector-im/riot-android#2542)
• Room name and topic can be now set back to empty (vector-im/riot-android#2345)

API Change:

• Remove PieFractionView class from the Matrix SDK. This class is now in Riot sources (#336)
• MXMediasCache.createTmpMediaFile() methods are renamed to createTmpDecryptedMediaFile()
• MXMediasCache.clearTmpCache() method is renamed to clearTmpDecryptedMediaCache()
• Add MXMediasCache.moveToShareFolder() to move a tmp decrypted file to another folder to prevent deletion during sharing. New API MXMediasCache.clearShareDecryptedMediaCache() can be called when the application is resumed. (vector-im/riot-android#2530)

🔗Matrix iOS SDK

🔗Changes in Matrix iOS SDK in 0.11.3 (2018-08-27)

Bug fix:

• MXJSONModel: Manage m.server_notice empty tag sent due to a bug server side (PR #556).

🔗Changes in Matrix iOS SDK in 0.11.2 (2018-08-24)

Improvements:

• MXSession: Add the supportedMatrixVersions method getting versions of the specification supported by the homeserver.
• MXRestClient: Add testUserRegistration to check earlier if a username can be registered.
• MXSession: Add MXSessionStateSyncError state and MXSession.syncError to manage homeserver resource quota on /sync requests (vector-im/riot-ios/issues/1937).
• MXError: Add kMXErrCodeStringResourceLimitExceeded to manage homeserver resource quota (vector-im/riot-ios/issues/1937).
• MXError: Define constant strings for keys and values that can be found in a Matrix JSON dictionary error.
• Tests: MXHTTPClient_Private.h: Add method to set fake delay in HTTP requests.

Bug fix:

• People tab is empty in the share extension (vector-im/riot-ios/issues/1988).
• MXError: MXError lost NSError.userInfo information.

🔗Tools

🔗matrix-to-riot

Half-Shot shared a handy Firefox extension: matrix-to-riot

This is a tiny webextension to forward matrix.to links to an open Riot tab.

Really useful if you often click matrix.to links and find yourself needing to URL-hack to get to where you need to be.

🔗That's all for now

We're going to shift this week's Matrix Live to Monday, alongside the upcoming blog post on the spec release progress. Have a good weekend!

🔗Synapse 0.33.3

Big week for Synapse: v0.33.3 was released this week. You'll find preparation for support for Lazy Loading and Room Versioning, lots of bugfixes, and a great contribution from vojeroen: support for sending SNI over federation for vhosted servers. More complete change log here, features below:

• Add support for the SNI extension to federation TLS connections. Thanks to @vojeroen! (#3439)
• Add /_media/r0/config (#3184)
• speed up /members API and add at and membership params as per MSC1227 (#3568)
• implement summary block in /sync response as per MSC688 (#3574)
• Add lazy-loading support to /messages as per MSC1227 (#3589)
• Add ability to limit number of monthly active users on the server (#3633)
• Support more federation endpoints on workers (#3653)
• Basic support for room versioning (#3654)
• Ability to disable client/server Synapse via conf toggle (#3655)
• Ability to whitelist specific threepids against monthly active user limiting (#3662)
• Add some metrics for the appservice and federation event sending loops (#3664)
• Where server is disabled, block ability for locked out users to read new messages (#3670)
• set admin uri via config, to be used in error messages where the user should contact the administrator (#3687)
• Synapse's presence functionality can now be disabled with the "use_presence" configuration option. (#3694)
• For resource limit blocked users, prevent writing into rooms (#3708)

Ananace has upgraded his K8s-optimized Synapse image to 0.33.3

🔗Spec

• A large chunk of e2e has landed in the spec, largely thanks to Zil0's work. There are still some things left, but the main bits related to sending encrypted messages is in there. Attachments and key sharing are being reviewed today
• TravisR progressed on Push, Application Services, and Client-Server APIs.
• To follow along with progress, the best place to look is the August 2018 r0 tracker.

🔗E2E

Key backups are being worked on, and we've had some new thoughts on key verification.

🔗ru-matrix.org

Well are you?

Alexey and ma1uta have a new Russian-language community wiki available at https://ru-matrix.org. This:

accumulates Russian translates of Matrix FAQ and manuals, and other info about Matrix in Russian. Also we starts a public Matrix Synapse server for Russian users with free open registration

Russian Matrix users: feel free to add and complement articles in our wiki to make Matrix more friendly for Russian users.

🔗Client News

🔗Seaglass

neilalexander has been on a roll the last few weeks working on Seaglass, the macOS-native client:

Seaglass now has support for creating, joining and leaving rooms, and accepting/rejecting room invites, redacting messages, improvements to the room settings pages, timestamps, some visual tweaks, lots of bug fixes, early support for detecting failed message sends, scrolling improvements

🔗Matrique

Black Hat continues work on Matrique, a glossy Qt client:

Typing notification and desktop notification of Matrique are working, plus other improvements

See the latest commits here.

🔗Riot Web

Lazy Loading (LL) is available on develop behind the LABS flag, and is being polished. It will probably not make it to next release, but it's progressing!

🔗Riot Mobile

New releases are imminent for both Android and iOS:

• iOS: bug fixes and Lazy Loading in settings
• Android: bug fixes, better management of permission requests at startup

🔗Shared Secret Auth and Matrix Corporal

Slavi has been working on several operations tools for Synapse. Firstly Shared Secret Auth password provider module for Matrix Synapse.

It allows external systems (which know the shared secret) to easily obtain access tokens for any user hosted on the homeserver. This is incredibly useful for all sorts of automation (no more pre-generated access tokens or plain-text passwords).

With this tool, you can pass a mxid, and get back an access_token for that user. With this, you can do whatever actions would normally be done with a token. For example, you for a new starter at a company, you could join a set of rooms known to be needed by new starters.

Support room for Shared Secret Auth: #matrix-synapse-shared-secret-auth:devture.com

Next, he has released Matrix Corporal, reconciliator and gateway for a managed Matrix server:

It's a way to take control of Matrix (Synapse) in a corporate (or other such) environment.
Based on a configuration policy (generated by an external system, like your intranet), it can auto-create and auto-disable users, keep their profile and authentication details to date, auto-join/leave users to communities and rooms, etc.
It's also meant to sit in front of the Matrix Synapse Client API, inspecting and allowing/denying requests, in accordance with the configuration policy (preventing people from leaving certain rooms/communities, preventing them from messing around with their profile details, etc.)

Support room for Matrix Corporal: #matrix-corporal:devture.com

Note that Slavi also maintains matrix-docker-ansible-deploy, which can deploy these projects along with Synapse itself.

🔗Bots

🔗twimbot

I was extremely young when I first read and understood Larry Wall's explanation about the virtue of laziness, but I did not expect it to lead here. Cadair has created (I think inspired by an idea from TravisR), a "twimbot". This bot works in conjunction with opsdroid and the opsdroid connector-matrix - it works by storing selected messages from #twim:matrix.org in room state. In this way, it either assists the author by making it easier to produce the blog post you're reading, or it coldly replaces the author, leading to his sacking and penury. To view updates currently held by the bot, join #twim_updates:cadair.com.

🔗Hashtag bot

When he's not working away on the Matrix Spec, TravisR continues to work on less [essential|sane] Matrix projects. This week we have @hashtag:t2bot.io, a bot that converts #hashtags to Matrix groups.

Why? Because why not. Source available on github: https://github.com/turt2live/matrix-hashtag-bot

🔗Bridges

🔗matrix-appservice-bridge and matrix-appservice-irc

Half-Shot will only be working full time with the Matrix Core team for another week (!!!), but until then he's going full-speed with bridge work. First matrix-appservice-irc:

Just pushed out matrix-appservice-irc 0.11.0-rc3 which contains a lot of nice things. The full changelog is there, hoping to get some of this tested and running on the matrix.org servers soon, but initial testing looks good :)

Having worked so much on IRC bridging the last few months, Half-Shot has this week managed to make improvements to matrix-appservice-bridge:

Released matrix-appservice-bridge 1.6.0 which contains mostly things I was writing for the irc bridge and factored out instead. There's a couple of goodies like automatic caching of profiles/events, and getEvent as a way to fetch events from the homeserver without fiddling with context.

Full changelog: https://github.com/matrix-org/matrix-appservice-bridge/blob/1.6.0/CHANGES.md

Some Intent operations now cache requests that would otherwise fall through to the homeserver which can be expensive. This is configurable for Intents via the opts.caching.ttl and size options. AgeCounters now allow you to set your own time period buckets. Added a new function Intent.getEvent which will fetch events from the homeserver without any context information, which should be quick. MembershipCache is now exposed to let folks read and write to the cache while also letting the bridge access it.

a release: Fix issue where roomState would fail. b release: Fix issue where we were calling this.intent inside intent like fools! c release: Fix issue where some stole js-sdk code was not checked thoroughly.

🔗WhatsApp

tulir announces that mautrix-whatsapp "is now somewhat usable for basic chatting".

It uses the go-whatsapp library to talk with the WhatsApp Web API. Using the web API means that you'll still need WhatsApp on your phone connected to the internet, but it also means you won't get banned for using the bridge.
The bridge is still very alpha, but basic message bridging and some advanced features already work. There's a features & roadmap document about that.

To come chat about the WhatsApp bridge: #whatsapp:maunium.net.

🔗mxtoot

mxtoot is a Matrix-Mastodon bot from ma1uta, creator of Jeon and the ecosystem around it.

There were two releases last week:

0.4.6 with additional commands (follow, unfollow, mute, unmute, block, unblock users, show followers and followings of the specified user) and a little bugfix.

0.4.7 is out now, which increases the length of messages to 4000 characters.

🔗GSOC is finished!

GSOC 2018 has been valuable for everyone involved!

• Thank you zil0 for all your work on the Matrix Python SDK, E2E and and spec work.
• Thank you APWhiteHat: lots of work on dendrite, including a bunch of stuff on federation, as well as typing notifications!
• and thank you to Cadair for helping to organising it!

🔗The Construct using asynchronous disk IO

Jason writes:

the Construct has advanced to fully asynchronous disk IO in a single-threaded environment.
RocksDB is a highly configurable and customizable database with a very large callback surface to support many different environments, which is good to connect it with something like AIO.
Normally RocksDB gets its performance from concurrent standard POSIX system calls (blocking IO) with multiple threads. The callback surface and its internals are not specifically suited for deviating from the pthread model. However I hacked RocksDB and tricked it into believing that my userspace contexts, which mimic the std::thread/pthread interface, are threads.
This is an example of a RocksDB callback to read from a file, it expects this callback to return immediately (i.e as soon as the kernel returns from a blocking read(2).) Except, where I call fs::read() that is actually a stack-suspension which makes a request to linux AIO where the stack resumes after AIO has called back with the data (see: https://github.com/jevolk/charybdis/blob/master/ircd/aio.cc)
No modifications to the internals of RocksDB were necessary. The Construct is being developed here: https://github.com/jevolk/charybdis.

🔗Community Guides Index

Some weeks ago I asked for suggestions for content to add to a "Community Guides Index". There was not a wave of feedback, but we have enough collected content to be able to share a first pass of this Index. Check out the content there, and please please contact me if you have articles, videos or anything else you'd like to see included. Of course, if you'd like to produce some content and have it included, that would be great too! Ideas for missing articles:

• any content in a language other than English
• DevOps, how to use k8s or ansible to install Synapse
• how to use the Application Services API to make a bot

🔗OggCamp 2018

OggCamp last weekend was great - massive thank you to Jon for organising and running the show - we love your work!

Presentations from Ben and Half-Shot seemed to go down well, we all chatted with Open Source fans and generally flew the flag for Matrix.

🔗New Rooms

• tulir has enough projects that he has created #:maunium.net, for any project of his that doesn't have it's own room
• Support room for Shared Secret Auth: #matrix-synapse-shared-secret-auth:devture.com
• Support room for Matrix Corporal: #matrix-corporal:devture.com
• If you prefer the cold output of a machine, #twim_updates:cadair.com shows you the updates as they're posted in #twim:matrix.org.
• WhatsApp bridge: #whatsapp:maunium.net

🔗So long…

Hope you had a good week, hope you enjoyed reading this post! Amandine is back after some needed vacation time, and has this week's Matrix Live below.

This feels like a bumper (extra big) post, so let's get straight into it!

🔗Spec

TravisR continues to plow through work in documenting, clarifying and confirming the spec. You can check out his project on Github: August 2018 r0, which should give an idea of both the scale of the project and the amount of work which has been done. Rather than list out individual items as I have been, please take a look at that project, and come chat in #matrix-spec:matrix.org if you'd like to contribute.

🔗Clients

🔗Riot web

Some exciting things on Riot Web this week!

Bruno has merged his work on lazy loading room members lists, which should mean we see some big performance improvements, especially in larger rooms. You can expect to see that released on /develop soon.

He has also just started to work on the riot redesign, and has begun on some of the more visible changes like resizeable panels:

More news on the redesign to come soon!

🔗nheko 0.5.3 released

nheko 0.5.3 was released this week. From the changelog:

Features

• Add option to disable desktop notifications
• Allow user to configure join rules for a room
• Add tab-completion for usernames

Improvements

• Remove the space gap taken by the typing notifications
• Remove hover event from emoji picker
• Add tooltips for the message indicators
• Fix compilation on FreeBSD
• Update Polish translation
• Small modal improvements

🔗Seaglass builds being generated

Seaglass, the MacOS-native client, now has automatically generated builds available on bintray.

Seaglass is now a very usable and stable client on macOS. neilalexander has been very active this week:

Seaglass has had various updates, including the groundwork for joining and leaving channels, LOADS of crashes fixed, some more work done on avatars and E2E encryption has been fairly well road-tested now Still need to complete UI for device verification and setting E2E but it works fine in existing channels

🔗Quaternion

kitsune has been working on Quaternion:

Quaternion has gained a new way to highlight mentions - with tinted background instead of colored text. Also, the majority of work on rooms grouping is done, and the current master branch features the roomlist grouped by tag (but rooms under each tag are pretty much out of order so it's not really ready for daily use yet).

Quaternion is looking for a macOS packager - if someone has the chance to help out in this area, speak to kitsune in #QMatrixClient.

🔗Palaver

Arne came at us with the v0.2.1 release of Palaver:

A few days back Palaver has been moved to Gitlab. And I have since released v0.2.0 and v0.2.1. A runnable jar-file of the latest release can be downloaded at https://gitlab.com/MrCustomizer/palaver/tags/v0.2.1. The biggest changes in v0.2.x are the replacement of all the web views with native JavaFX components (as I don't feel comfortable embedding a whole browser stack in a desktop application) and support for read markers. There is a short YouTube video demonstrating the read marker implementation:

🔗Riot mobile

This week, work continued on Lazy Loading room members for Riot Android, and progress toward a release for 0.7.0 on iOS.

🔗neo client

f0x has begun rewriting Neo using matrix-js-sdk.

🔗Matrique

Black Hat has been working on Matrique:

Matrique now has a (basic) room management panel, a working emoji picker, an unread marker and supports playing "m.audio". It should be available at Flathub any time.

🔗SDKs and such

🔗Jeon, Jmsdk, Gene, Matrix-Jabber-Bridge: ma1uta-world

ma1uta is continuing to make progress on his Java-focused collection of APIs and SDKs, he's eagerly awaiting the r0 release of the spec (aren't we all!)

Jeon (https://github.com/ma1uta/jeon) (client2server and application api) has released on the Maven Central Repository with version 0.4.1. The next goal: the stable release of all apis 1.0.0 after the Matrix spec will be released (I hope it will be soon :)) Jmsdk (https://github.com/ma1uta/jmsdk) the client and bot SDK have released on the Maven Central Repository. Gene (https://github.com/ma1uta/gene) the lightweight api for android has released on Maven Central Repository with 0.2.1 version. So, everyone can use this libraries without additional settings and additional repositories. Matrix-Jabber-Bridge (https://github.com/ma1uta/matrix-jabber-java-bridge) the double-pupped bridge between the Matrix and Jabber. I am still working on it and I think it will be released in a few weeks.

🔗Ruby SDK v0.0.3

Ananace has tagged and released the next development version of the Ruby Matrix SDK - 0.0.3. He has plans to "finish up documentation and unit testing", with the aim of making a v1.0.0 release.

🔗mautrix-appservice-go

tulir has a new, yet-to-be-revealed project, and in support of this he's been working again on mautrix-appservice-go:

mautrix-appservice-go is a bit more active again: it now has an initial intent API similar to the one in mautrix-appservice-python (which in turn is based on the intent API from matrix-appservice-bridge).

🔗Synapse

No release this week, but Synapse has been progressing:

• Python 3 port continues at pace, we expect to have a beta to test in monolith mode rsn (Hawkowl leads this)
• Erik has been working on some federation API refactoring to make matrix.org snappier and hopes to have the new state resolution algo ready to go if not deployed late next week
• Richvdh has been looking at performance, as well as spec PRs/ Next week will be r0 work

Many Synapse maintainers are also doing huge lifts on development of the spec.

🔗Matrix support now in ntfy

Half-Shot took a rare 30-minute window away from bridge maintenance to add Matrix support to ntfy. ntfy describes itself as

A utility for sending notifications, on demand and when commands finish

but in fact, you also send notifications whenever you like by calling ntfy send from anywhere you'd normally execute shell commands. This will make it really easy to integrate Matrix notifications into any other application!

🔗matrix-docker-ansible-deploy

Slavi brought updates to matrix-docker-ansible-deploy:

matrix-docker-ansible-deploy got a lot of improvements lately:

• email-sending support, so that Matrix Synapse can send email notifications (enabled by default)
• mxisd Identity Server support (enabled by default)
• more detailed docs split into various sections (used to be a single README, but with more and more features it was turning into a wall of text)
• started keeping a changelog, especially for backward-incompatible things

the playbook can now also set up and configure Maximus's matrix-synapse-rest-auth

🔗Riot Translations

@csybr:dodsorf.as and @dandellion:dodsorf.as appeared and announced that there is now a Norwegian Nynorsk translation available for Riot Web, matrix-react-sdk and riot-android. Non-Matrix-related aside: as a lazy Brit, I was vaguely aware of the cultural weight language carries in the history of Scandinavia, but didn't realise that Norway still has some fundamental differences ongoing.

🔗Giveth

The Giveth project have produced a bot built on matrix: giveth-bot

For some more context:

Giveth is re-engineering charitable giving, by creating an entirely free, open-source platform, built on the Ethereum Blockchain.

The current giveth-bot:

handles welcoming people, can answer questions and handle our crazy points dishing system to reward contributors!

🔗mautrix-telegram

mautrix-telegram now has an official 0.3.0 release. Says tulir:

No major changes since the release candidates last week, except a Telethon update that might have fixed history backfill.

🔗"The End of GSOC"

Well, it may well be the end of the Julian's GSOC 2018 experience with the Fractal project, but I think Google will continue to run the programme. Check out Julian's experience (and the large number of issues resolved!) in this blog post.

🔗OggCamp

This weekend Neil and I will be representing Matrix at OggCamp, "an unconference celebrating Free Culture, Free and Open Source Software, hardware hacking, digital rights, and all manner of collaborative cultural activities", if you will be there, come find and chat to us. We'll be the two nerdy-looking guys, so we should be easy to spot.

🔗We'll meet again…

But for now, you can watch Neil host Matrix Live below (using the fan-favourite format of walking around the office), and come chat to us in #twim:matrix.org

🔗Community Guides Index

As part of an overhaul of the documentation surrounding Matrix, I'm planning to launch a "Community Guides Index". This will be a page or section on Matrix.org with links to useful and informative documentation hosted elsewhere. If you have or know of informative articles, please let me know, or share them in #TWIM:matrix.org.

🔗Clients

🔗Seaglass

Seaglass now supports E2E rooms, leveraging matrix-ios-sdk! Additionally, maintainer neilalexander has had a productive week:

Seaglass has had many updates, including:

• message coalescing
• room settings (history/join rules and changing aliases/addresses)
• loading member list in rooms
• some starting work on loading avatars
• various UI and stability tweaks!

🔗Quaternion: new redacted message handling options

Many of you will have seen the incredible volumes of spam plaguing Freenode over the last few weeks. Kitsune has added new features to Quaternion to help:

In response to the recent spam deluge from Freenode Quaternion has got an option to tidy up your timeline after all the spam-cleaning: you can hide all redacted messages or just those that came from recently joined users (along with the join message). So as long as the spammer gets moderated, it can become inexistent for you. The experimental code doing that will eventually be moved to libQMatrixClient for the benefit of Matrique, uMatriks and other clients based on the library.

This is especially cool: moving functionality to the lib for the benefit of other projects makes me very happy. :D

🔗Fractal GSOC write-ups

Eisha and Julian finished their GSOC 2018 Fractal involvements this week, and Eisha has a blog post recounting her experience: My Final Report for GSOC 2018. Julian will have his ready too.

🔗Riot

Big updates are on the horizon for both iOS and Android, with 0.7.0 of iOS headed to TestFlight today with a release due once it's through beta & AppStore review.  More details once the releases are out!

Meanwhile on Riot/Web, Lazy Loading members is pretty much complete - just writing end-to-end tests to ensure that encryption still works well when lazy loading is enabled.  Next up is actually progressing the redesign at last!

🔗Spec status

• MSC1425/MSC1501 richvdh has been thinking about Room Versioning
  • A summary in his words: "room upgrades are implemented by creating a new room, shutting down the old one, and linking between the two"
• MSC1497 thorough discussion happening against Matthew's proposal for Capabilities support in the CS API
• MSC1442 though you may have seen it previously, I recommend anyone who is able to take a look at Erik's "State Resolution: Reloaded" proposal. This documents his "thoughts on the next iteration of the state resolution algorithm that aims to mitigate currently known attacks". It very clearly explains the current problem, and the proposed solution. While heavy, I found it really useful for helping to understand State Res and everything it affects.
• MSC1485 tulir proposes "Hint buttons in messages", which effectively provide pre-defined responses for users. This is especially valuable for Bots which can only accept a certain range of replies. I'd love to see this implemented, having long ago built a "choose your own adventure" engine using similar features from another messaging platform.

🔗APIs and SDKs

🔗Gene, minimalistic client-server api for Matrix

Not strictly this week, but we missed it before. ma1uta has been working away on his Java-based Matrix suite. Gene is a more lightweight API implementation than Jeon, covering all of the current C2S spec. A big motivation is to be able to use Gene on Android, which is not possible for Jeon due to it having more dependencies.

🔗Ruby Matrix SDK

The Ruby Matrix SDK now has auto-generated, proper documentation, thanks to Ananace. v0.0.2 is the current release, but it's under active development and v0.0.3 is expected presently.

🔗Telegram Puppeting on t2bot.io with mautrix-telegram

TravisR has rolled out tulir's mautrix-telegram bridge for everyone on t2bot.io's bridge.

This is a public trial to see how the bridge copes with multiple people puppeting their accounts. Of course, you don't need to puppet your account unless you want to. For more information on how to set this up, please visit https://t2bot.io/telegram

As part of this project, mautrix-telegram 0.3.0 release candidate was released.

🔗Informo project releases SaltStack states

vabd, from the Informo project, arrived to say they have produced "a set of SaltStack states that includes deploying fully working Synapse homeservers: https://github.com/Informo/salt-states"

They introduce the Informo project:

Informo's a project (still a work in progress, currently) that aims at bypassing censorship of information on the Internet, by basically having news items go through a decentralised and federated network made of Matrix homeservers. Since we need to host nodes to this network (at least for testing purposes), we've worked on a working and automated infrastructure using SaltStack, which involved hand-writing a few SaltStack states, including states handling the installation and configuration of working Synapse homeservers. In the spirit of giving back to the community, we're publishing all of those hand-written states (available at the GH link above) under the GPLv3 license. If you want to discuss with us about what we're doing, or this topic specifically, please join us in #discuss:weu.informo.network ? The project is still pretty much a WIP and the technical specifics (even the ones we already implemented in our projects) are still under discussion and subject to changes (as mentioned in the GH repo's readme). All I can add for sure to what we already say is that we'll be working in partnership with NGOs to promote the project in countries where the press is heavily censored, and we're planning on implementing a cryptographic verification chain of some sort to reduce the risk of intentional disinformation.

🔗Synapse 0.33.2

Synapse 0.33.2 is out now:

The release focuses on performance, notable highlights include reducing CPU consumption through speeding up state delta calculations and reducing I/O through lazily loading state on the master process Separately work continues on our python 3 port and we hope to have something concrete to trial very soon – we're really excited about this and expect step change improvements in CPU and memory use.

Latest synapse is now available on Debian thanks to andrewsh.

🔗Room Tagging

This is not from this week at all, other than that it was new information to me. Actually it's ancient, matrix-wise, but I think it's interesting for those who don't know! Although Riot doesn't have an interface for adding custom room tags ("Favourites" and "Low Priority" come by default), Matrix supports arbitrary strings as room tags, and will display them if you add them via another CS API call. To facilitate this TravisR created a small script, tag_room.sh, which makes it very simple to set your own room tags.

🔗FIN

That's all folks! Remember, lots of progress is happening with the Spec, if you want to be involved or follow along the best place to start is by joining #matrix-spec:matrix.org.

No Matrix Live this week due to vacation and general scheduling fun; sorry!

If you have documentation or articles related to matrix, let me know!

We'll meet again!

🔗Spec Progress

Progress on the spec has been motoring since TravisR dived (dove?) into it full time a few weeks ago - the Federation API r0 megathread bug that tracks progress on filling in the gaps on the S2S API is clearing its checkboxes at an impressive rate.

Lining up spec PRs for @matrixdotorg sure does look pretty. pic.twitter.com/YHQ00ZgzYl

— Travis Ralston (@turt2live) August 3, 2018

Some points of note regarding current proposals:

• MSC1466 Erik proposes a soft_logout field to be added to the body of 401 responses, to better help handling of encryption keys. Check the proposal notes
• MSC1452 agreement has been reached on Homeserver Warning Messages

We're going with pinned messages (option 2) and room tags (option 5) as that seems to be where the consensus is: it re-uses existing bits of the spec and room tags also help clients that don't know about this specific room tag to handle the room the right way

• MSC1426/MSC1421 Discussion about the review process itself

• MSC1425 Room Versioning It's likely that in the immediate future we'll want to change the properties of rooms in a way that will not be compatible with existing servers - for example, changing the rules for event auth or state resolution, or changing the format of an event id.

• MSC1318 Documentation describing the anticipated Open Governance of Matrix.org (aka, Matrix.org Foundation)

🔗Python SDK -> Python 3 ?

The maintainers of the Matrix Python SDK are mulling some major changes to the library. In particular, the desire to use await / async syntax means they are considering making Python 3.5 the minimum supported version. Go chat about this change and comment on the proposal issue.

🔗Clients

🔗Riot/Web 0.16

Big congratulations to the Riot/Web team on the release of 0.16. You can read all about it here, but I'll give you the headlines now:

• Replies are now available, there is UX for them and they look great
• Jitsi is now the default video conferencing provider across Web, iOS and Android, with new widget integrations for Riot Web
• New composer (text box) using Slate.js rather than Draft.js, which fixes many existing bugs and improves performance

Meanwhile, Lazy Loading implementation is approaching completion, promising several factors of improved resource utilisation!

🔗nheko 0.5.2

Also now available on flathub!

Go download nheko and check out the 0.5.2 release notes.

New features just in the last week or so:

• Mark own read messages with a double checkmark
• Add option to specify the scale factor
• Add input field to specify the device name on login.
• Add option to ignore key requests altogether.
• Show device list in user profile & add option to create 1-1 chat.

Plus lots of improvements and bug fixes.

🔗libQMatrixClient and Quaternion

kitsune has been working on resend functionality:

libQMatrixClient and Quaternion have gained ability to resend and discard unsent messages. this means if Quaternion could not, after several attempts, deliver a message, a user can click "Resend" and it will try again

On the subject of libQMatrixClient, it's exciting that Konversation, the KDE IRC client, may in future start to use libQMatrixClient for Matrix support!

🔗Matrique

Black Hat announces a Flatpak repo for Matrique:

Matrique now has a Flatpak repo. It is the nightly build of the master branch. You can add the repo by typing flatpak remote-add matrique https://b0.gitlab.io/matrique-repo/matrique.flatpakrepo and install it by flatpak install matrique org.eu.encom.matrique As it is still Alpha quality, bugs are expected. Feel free to open an issue if anything goes wrong!

🔗Fractal 3.29.6

New release of Fractal to 3.29.6. Notes from the changelog:

• Add German translation
• Message right click menu with: view source, reply, copy text and delete
• Styles for quotes in messages
• Initial sync speed up

🔗Neo

Incremental improvements to Neo from fox:

Neo now has inline youtube and image url previews, and handles room state changes such as name, avatar and topic as they occur.

🔗Riot/Mobile

• Android: a lot of bug fixes and small UI improvements
• iOS: Lazy Loading is coming to life, showing huge improvements in bandwidth usage and performance in the app

🔗Updates on IRC bridges from Half-Shot

Half-Shot has been working tirelessly on the IRC bridge lately, so I wanted to update on his recent successes:

I've recently been working on mitigating the effects of a netsplit on the IRC bridge, and optimising it to start and run faster. This week I trimmed down the heap usage (where the memory usually goes) to just under a gigabyte on my 10,000 matrix user test bridge. Previously it could spike to as much as 3.5GB. This optimisation is still in a testing phase but results are looking positive.

For reference here is the memory usage of the Freenode process during startup:

And here are the results of my local test bridge before and after the change:

Before:

After:

We also made some internal changes to the appservice-bridge to cache the joined state of all the bridge users and therefore avoiding joining rooms which saves us some time on startup.

🔗Matrix for Grafana, and more from Ananace

In his regular spot, Ananace has made progress on his Matrix sysadmin/ruby suite:

• Grafana alerting ingester supports images, plus more per-task configuration options.
• The K8s-optimized docker image for Synapse has been updated to 0.33.1
• The release tracker has gotten multiple performance improvements.
• the Ruby SDK continues to be slowly worked on.

🔗Synapse

Synapse 0.33.1 is out now as a security update release. Please update if you haven't already - it fixes two issues concerning event visibility where if you knew the event ID of an event you could read it even if you didn't have access to it; we don't believe these have been exploited in the wild, but you will definitely want to upgrade now.

Meanwhile the Python 3 port is progressing well (all sytests now pass in Python 3, i think!), and intrepid folks are starting to experiment with running it in production.

🔗Decentralised Web Summit & Matrix Live

Meanwhile, Matthew & Amandine have been in San Francisco for the 2018 Decentralised Web Summit - so this week's Matrix Live is live from SFO and gives a quick overview of the sort of things we got up to!  Some of the sessions are already online thanks to the (somewhat unreliable) live stream (e.g. here's Muneeb (Blockstack), Amandine, Danielle (Dat), and Zooko (Zcash) talking about their respective governance models & growing pains over the last 2 years: https://youtu.be/tsz3ffrJDpw?t=12133).  The summit was a massive success, with lots of discussions about decentralised reputation, UI/UX for decentralised apps, metadata-resistance, the balance of P2P versus decentralised-servers, etc.  Hopefully some of the conversations we had will result in some major improvements to Matrix in the future!

Edit: Here are the slides for our "Diving into Decentralised Communication" workshop, for those interested in a comparison between Matrix/SSB/Mastodon/Status/Vuvuzela/Briar.  They're pretty minimal, as they just formed a framework for discussion, but might still be of interest.

🔗mautrix-telegram now supports double puppeting

tulir has been working away on mautrix-telegram, and has a major breakthrough:

It means you can log in to the bridge with your Matrix account to make messages you send from other Telegram clients appear from your Matrix account.

It can also be used to enable bridging of ephemeral events, which synapse doesn't send to appservices (read receipts, typing notifications, presence).

Also, the bridge has a new HTML parser which should be much better than the old one. It might have caused some new bugs though (like m.emote bridging breaking)

🔗Coffee and the FAQBot

I've been slowly working on matrix.org the last couple of months, updating the FAQ, updating content and UI/appearance of try-matrix-now. As you may know, the FAQBot, created and maintained by Coffee, uses try-matrix-now for it's data, so Coffee updated the bot to start to use some of the new fields.

The final puzzle piece needed for FAQBot has been put into place: https://gitlab.com/Matrixcoffee/extract-web-to-org
This is the bit that grabs "Try Matrix Now!" and turns it into questions that FAQBot can use.
While I was working on that bit of code anyway, I also added support for the new 'home', 'repo', 'room' and 'language' fields. FAQBot will now give more and better information accordingly. For example, it is now possible to ask, "where is the Riot Android source code?" although it is generally better to just ask "where can I find Riot Android?" which combines several fields into a single answer.
4 of 7 items are now completed: https://gitlab.com/Matrixcoffee/FAQBot/issues/2
and the remaining 3 items before FAQBot's release are just the final review and integration tests.

🔗New Discord Bridge Maintainer

Half-Shot has been working stridently on the Discord Bridge lately, and the project has now attracted a second maintainer:

We've got a new member of the maintainer crew in discord bridge land (bringing the total count to 2.) I'd like to welcome Sorunome :). They are going to be helping review the influx of PRs, chew some issues and be a point of contact in the community. They've already filled my PR queue up and it's going to help us move even faster!

In other Bridge news, matrix-appservice-bridge 1.5.0 is out now, uses the latest matrix-js-sdk and matrix-appservice-node, and contains a lot of changes.

🔗Bruno unit tests

Bruno, the latest and greatest addition to the [riot-web] team, has been working on configuring end to end unit tests. His scripts now install synapse from the metal, install riot and then run a test suite. Take a look at the animation below to get the idea!

🔗Fractal 3.29.5

Alexandre Franke tells us about new features in Fractal 3.29.5, and how the project has benefited from GSOC 2018 contributors:

Fractal got a big new release 3.29.5, which includes Eisha's multi-line input (with markdown syntax highlighting) and Julian's room details redesign, as well as his refactor on avatar code which leads to a perf boost.

🔗Ananace: Grafana webhook for Matrix notifications

Ananace, always working on sysadmin tooling with matrix, comes back with a new tool: Grafana webhook for Matrix notifications. This release is described as "the very first 'It doesn't crash instantly' version", but this is surely a useful project.

🔗Quaternion / libQMatrixClient support local echo

kitsune had a surprising week while working on his QMatrix projects:

/me accidentally found a year-old feature branch in his Git and thought of rebasing it in order to eventually continue working on it. As a result, libQMatrixClient and Quaternion master branches support local echo from today. Too bad the same trick doesn't work for E2EE.

Too bad indeed!

🔗Riot/Web

• Cut the release candidate for 0.16-RC1
• Lots of final polishing on Jitsi; all bugs fixed now
• Likewise all Slate bugs fixed - many thanks to t3chguy for finishing them off
• Lazy Loading Members implementation progressing well
• Release due on Monday
• On crypto: uhoreg has also been progressing on hooking up incremental key backup

🔗Riot/Mobile

• Room version upgrade support!
• Replies UI implemented on iOS and working well
• Lazy Loading Members is done and working on iOS - speeds up launch by a minimum of 3x

🔗Synapse

• Hawkowl is running the first ever Python 3.6 synapse! RAM usage looks to be 2-3x less than on Python 2.7. Still work to be done to merge all the PRs though.
• Landing all the Lazy Loading patches; 3 down, 3 to go
• Gathering feedback on the state res prop - thanks to uhoreg for wading through it!
• Catching up Lots and lots of PRs
• Implementing room versioning and gathering feedback on the MSC
• Perf fixes - optimising current state res a bit & fixing sync performance
• => End-to-end message latency on matrix.org has been improved by 3-5x over the last few days(!!!)

🔗Dendrite

• Super-exciting to see E2E working thanks to fadeAce's PR: https://twitter.com/matrixdotorg/status/1022887733931520001

Super-exciting to see end-to-end encryption working with Dendrite thanks to a massive PR from first-time contributor fadeAce!! https://t.co/SLIPvgYGAm pic.twitter.com/s47SWjY46o

— Matrix (@matrixdotorg) July 27, 2018

• OpenTracing support has been dusted off and is working again cross-service
• See the community update for much more!

🔗Spec

• Travis continues to blitz through the S2S spec, PRs are flying around everywhere (and even landing!)
• Uhoreg continues to work on the E2E spec merges with Zil0

🔗We'll meet again…

See you next week, and don't forget to watch Matrix Live below!

🔗Spec Updates!

First of all: we've just released the first draft of the proposal for our next generation State Resolution algorithm as MSC1442 (State Resolution: Reloaded).  This is the result of a massive amount of work from Erik - if you are at all interested in the problems that state resets have plagued us with, then you will want to read the proposal and please comment on it!

The next step on State Resolution: Reloaded (once approved) is to be able to actually roll it out - and to this end, richvdh has proposed MSC1425 (Room Versioning proposal). This was published at the beginning of the week but we haven't had any feedback yet - again, please read & give feedback as the intention is to start working on this as soon as possible!  Matthew's also been updating the Lazy Loading proposal (MSC1227).

Meanwhile, the Matrix Spec is now awash with activity, lots of work from TravisR, Cadair and others to accelerate progress. Take a look at the proposals list to stay up to date, or straight to the matrix-doc repo and #matrix-spec:matrix.org room to get the raw activity.

Finally, discussions relating to the spec proposal process itself have attracted attention: MSC1426 and MSC1421, and we're already trialling the new process (using Github PRs rather than Google Docs) in MSC1442 (State Resolution: Reloaded) already.

🔗nheko 0.5.x

nheko 0.5.1 is out, and it's great! Get a version for your platform from bintray.com. I've been using it for the last week or so (on macOS) and found it very stable and usable. mujx commented that "highlights were encryption support & desktop notifications on all platforms", but you can get much more thorough notes from the release notes.

🔗Plasma

It's been several weeks since we mentioned Plasma, but work on the C2S parts of the spec implementation have been happening quietly behind the scenes. Project lead Nico said recently:

I've updated the CI test server with the latest snapshot version of plasma. createRoom and invite C2S endpoints are fully implemented and should work as expected.

You can find the full list of implemented endpoints in the readme, and also read the architecture overview.

🔗kitsune, live on stage

kitsune presented an introduction to Matrix (for a technical audience) at the Tokyo Linux Users Group - you'll need to skip to 1h41m16s into the video.

🔗GTAD pulls in description and summary fields from the API

kitsune continues working on GTAD, and this week has started automatically including docs in libQMatrixClient:

GTAD is now aware of description and summary fields in API descriptions so you can use them with Javadoc/Doxygen. As a result, libQMatrixClient master branch now enjoys doc-comments throughout its CS API layer - the rest of the library builds in envy.

🔗SimpleMatrix

MTRNord introduced us to SimpleMatrix:

SimpleMatrix is my approach on making an Android App that is more easy to use than the official one in Terms of UX and UI. Planned is also a Android Wear App within this App for mostly Notifications.

SimpleMatrix is planning to use matrix-java-sdk, but for now is a visual mockup, it doesn't currently connect to Matrix.

Room: #SimpleMatrix:matrix.ffslfl.net Git: https://git.nordgedanken.de/Nordgedanken/SimpleMatrix

🔗Tales of Half-Shot and his many Bridges

🔗Arrival of the Discord DMs

After lots of begging, frustration and eventually an offer of a free lunch (thanks Anoa): I give you Discord DMs.
This first cut bridges over existing regular and group DMs to Matrix. In the future we will also allow you to create DMs from Matrix so that you can contact users without even logging into Discord. With thanks to the #discord:half-shot.uk community for helping shape this feature!

Currently this is in PR form but is very likely to make it into the 0.3 release of matrix-appservice-discord.

🔗Half-Shot fixes the Foonetic IRC bridge

For those who (want to) use Foonetic IRC channels, Half-Shot is here for you! Previously this bridge had been down due to invalid certificates, but this is now resolved.

🔗Synapse

Synapse 0.33 landed on Thursday, boasting 2x speed-ups for /sync and (briefly) meaning that the matrix.org homeserver felt impressively snappy and fast!  Ironically traffic levels promptly grew again such that the synapse master process is now the bottleneck once more, but between incremental state res, python 3 and upcoming room-sharding work we should see the performance fixes continuing to come!  Meanwhile, lots of PRs in the queue for Synapse 0.34 - including Lazy Loading support as per (MSC1227), and a whole new set of stats functionality for tracking per-room and per-user stats.

🔗Riot

Lots of activity in the build up to Riot/Web 0.16 - merging the new Slate rich text editor to /develop (try it now!), upgrading Jitsi and turning it on by default (at last!! - again, try it on /develop, although you'll have to enable it in Labs).  We'd be particularly interested in how Jitsi is behaving, as so far for us it's been unrecognisably improved over the previous deployment.

Meanwhile, lots of work on Lazy Loading members on Riot/Web and Riot/iOS, and lots of stability perf work in general across mobile.

🔗State Resolution Special... Matrix Live!

And finally, all you could ever want to hear about the new State Resolution algorithm on a special edition of Matrix Live, starring Erik & Matthew!