Usage of the matrix-js-sdk

16.10.2018 00:00 — Tutorials Ben Parsons

We have a brand new, exciting guide page offering an introduction to matrix-js-sdk. This guide will live with the documentation at https://matrix.org/docs/guides/usage-of-the-matrix-js-sdk,  but you can find the text below.


Matrix allows open real-time communications over the Internet using HTTP and JSON. This makes developing clients to connect to Matrix servers really easy! Because it's open, and uses simple syntax for messages, you can connect Matrix to anything that communicates over a standard HTTP interface - later projects in this series will explore ideas such as building bots, performing machine learning on message content, and connecting IoT devices such as Philips Hue lights.

Making a Matrix Client

Let's explore how we would make a very simple Matrix client, with only the ability to perform an initial sync, and to get member lists and the timeline for rooms of our choice.

This article will explore the Matrix Client-Server API, making use of the matrix-js-sdk. Later articles may discuss making the underlying calls. Specifically we will cover:

  • login
  • simple syncing
  • listening for timeline events
  • access the `MatrixInMemoryStore`
  • sending messages to rooms
  • how to respond to specific messages, as a bot would
We'll use Node.js as our environment, though the matrix-js-sdk can also be used directly in the browser.

Before we start, make sure you have Node.js and NPM installed: follow instructions at nodejs.org for your platform. Then create a new directory to work in:

mkdir my-first-matrix-client
cd my-first-matrix-client

Let's write JavaScript

Once you're ready, the first thing to do is install the matrix-js-sdk from NPM:

npm install matrix-js-sdk

We include the SDK in our source exactly as expected:

import sdk from 'matrix-js-sdk';

Login with an access token

Instantiate a new client object and use an access token to login:

const client = sdk.createClient({
    baseUrl: "https://matrix.org",
    accessToken: "....MDAxM2lkZW50aWZpZXIga2V5CjAwMTBjaWQgZ2Vu....",
    userId: "@USERID:matrix.org"
});
// note that we use the full MXID for the userId value

(jsdoc for createClient)

If you are logged into Riot, you can find an access token for the logged-in user on the Settings page.

If the homeserver you're logging in to supports logging in with a password, you can also retrieve an access token programmatically using the API. To do this, create a new client with no authentication parameters, then call client.login() with "m.login.password":

const client = sdk.createClient("https://matrix.org");
client.login("m.login.password", {"user": "USERID", "password": "hunter2"}).then((response) => {
    console.log(response.access_token);
});

In any case, once logged in either with a password or an access token, the client can get the current access token via:

console.log(client.getAccessToken());

Note: it is essential to keep this access token safe, as it allows complete access to your Matrix account!

Sync and Listen

Next we start the client, this sets up the connection to the server and allows us to begin syncing:

client.startClient();

Perform a first sync, and listen for the response, to get the latest state from the homeserver:

client.once('sync', function(state, prevState, res) {
    console.log(state); // state will be 'PREPARED' when the client is ready to use
});

Once the sync is complete, we can add listeners for events. We could listen to ALL incoming events, but that would be a lot of traffic, and too much for our simple example. If you want to listen to all events, you can add a listen as follows:

client.on("event", function(event){
    console.log(event.getType());
    console.log(event);
})

Instead, let's just listen to events happening on the timeline of rooms for which our user is a member:

client.on("Room.timeline", function(event, room, toStartOfTimeline) {
    console.log(event.event);
});

Access the Store

When we created a new client with sdk.createClient(), an instance of the default store, MatrixInMemoryStore was created and enabled. When we sync, or instruct otherwise our client to fetch data, the data is automatically added to the store.

To access the store, we use accessor methods. For example, to get a list of rooms in which our user is joined:

// client.client.getRooms() returns an array of room objects
var rooms = client.getRooms();
rooms.forEach(room => {
    console.log(room.roomId);
});

(jsdoc for client.getRooms)

More usefully, we could get a list of members for each of these rooms:

var rooms = client.getRooms();
rooms.forEach(room => {
    var members = room.getJoinedMembers();
    members.forEach(member => {
        console.log(member.name);
    });
});

For each room, we can inspect the timeline in the store:

var rooms = client.getRooms();
rooms.forEach(room => {
    room.timeline.forEach(t => {
        console.log(JSON.stringify(t.event.content));
    });
});

Send messages to rooms

To send a message, we create a content object, and specify a room to send to. In this case, I've taken the room ID of #test:matrix.org, and used it as an example:

var testRoomId = "!jhpZBTbckszblMYjMK:matrix.org";

var content = {
    "body": "Hello World",
    "msgtype": "m.text"
};

client.sendEvent(testRoomId, "m.room.message", content, "").then((res) => {
   // message sent successfully
}).catch((err) => {
    console.log(err);
}

(jsdoc for client.sendEvent)

Knowing this, we can put together message listening and message sending, to build a bot which just echos back any message starting with a "!":

var testRoomId = "!jhpZBTbckszblMYjMK:matrix.org";

client.on("Room.timeline", function(event, room, toStartOfTimeline) {
    // we know we only want to respond to messages
    if (event.getType() !== "m.room.message") {
        return;
    }
 
    // we are only interested in messages from the test room, which start with "!"
    if (event.getRoomId() === testRoomId && event.getContent().body[0] === '!') {
        sendNotice(event.event.content.body);
    }
});

function sendNotice(body) {
    var content = {
        "body": body.substring(1),
        "msgtype": "m.notice"
    };
    client.sendEvent(testRoomId, "m.room.message", content, "", (err, res) => {
        console.log(err);
    });
}

Take a look at the msgtype in the object above. In the previous example, we used "m.text" for this field, but now we're using "m.notice". Bots will often use "m.notice" to differentiate their messages. This allows the client to render notices differently, for example Riot, the most popular client, renders notices with a more pale text colour.

Further

There is much, much more to Matrix, the Client-Server API and the matrix-js-sdk, but this guide should give some understanding of simple usage. In subsequent guides we'll cover more detail and also explore projects you can build on top, such as IoT controls and chatbot interfaces. For now you can take a look at other examples in the matrix-js-sdk itself, and also the Matrix Client-Server API which it implements.

This Week in Matrix 2018-10-12

12.10.2018 00:00 — This Week in Matrix Ben Parsons

Matrix Spec updates

  • MSC1695 Half-Shot has a newly released proposal relating to Message Edits: "The key difference to this proposal is that it's only the event schema which is based off the relates stuff. It does NOT do any kind of aggregations but simply is a format to allow bridges/clients to specify edits in the metadata."
  • MSC1693 Erik has been thinking about state res v2 in the case of rejected events

libQMatrixClient 0.4.0

We mentioned libQMatrixClient progress last week, but this week kitsune reports:

libQMatrixClient 0.4.0 has just been released: CS API 0.4.0, local echo, caching avatars to disk and more - check the release notes at https://github.com/QMatrixClient/libqmatrixclient/releases/v0.4.0!

Thanks for the work also go to Black Hat (who is using libQMatrixClient to author Spectral) and delijati.

Informo

vadb, back again with news about Informo:

The Informo specifications documentation is out! ? It now lives right here: https://specs.informo.network
As a reminder, we were working on "phase 1" of this documentation, which goal was to outline how Informo was going to work. This is basically the foundation for a more complete technical documentation, which shall come later.
The idea is to enable people to give it a look as soon as possible, and to enable them to contribute to Informo as early in the design process as possible. The spec is entirely open, with a process for contributions documented and all its source files available on GitHub here: https://github.com/Informo/specs
The online version I initially linked is a live version from the repo's master branch, which is updated each time commits are pushed to it (including on merging a PR).
If people have any question or remark regarding the specs, or Informo in a general manner, we'll gladly answer them in #discuss:weu.informo.network (or on the #informo IRC channel on Freenode, which is bridged to the Matrix room)! ?

mxisd v1.2.0-beta.1 (beta release)

Max has continued work on mxisd, an Identity Server:

mxisd got a new beta release v1.2.0-beta.1.
It adds the ability to send email notification about room invites to existing accounts in Identity store that might not have been provisioned in the homeserver yet as users never logged in, or for users' profile that are not auto-populated with 3PIDs. This is especially targeted for onboarding/enrolling times when adopoting Matrix in groups/corporations or for custom setups.
We would love to hear about your experience with it!

matrix-trello-bot

TravisR has been working on matrix-trello-bot:

matrix-trello-bot has received quite the overhaul. It's gone from a small bot that had pre-determined notifications for board activity to a much more capable bot. The bot now supports managing boards (yes, you can create a board from within Matrix!), lists, and cards including creating, moving, archiving, assigning, and querying.

On top of all that, the bot respects that not every room will want to receive every possible notification and offers commands to pick and choose which events it should notify about on a per-board basis. Feel free to give it a spin and check out the massive help menu with !trello help. Please send bugs and suggestions to GitHub :)

The help menu really is massive! Travis provided an image to highlight the scope of functionality provided (click for larger version):

Clients

Riot 0.16.6 Released

Fixes only these two issues:

  • Firefox private mode being broken > (https://github.com/matrix-org/matrix-react-sdk/pull/2195)
  • Breakage when mixing /app and /develop use (issue #7432)
As both fixes are not relevant for electron, we didn't release 0.16.6 as the electron app, but rest assured 0.17 will of course be released as electron.

Fluffychat

We missed this last week, but FluffyChat (client for Ubuntu Touch) v0.6.0 was released, with LOTS of new features:

  • User profiles
  • Design improvements
  • Audioplayer in chat
  • Videoplayer in chat (only audio at the moment)
  • Imageviewer
  • Edit chat aliases
  • Edit chat settings and permissions
  • Kick, ban and unban users
  • Edit user permissions
  • New invite page
  • Display and edit chat topics
  • Change chat avatar
  • Change user avatar
  • Edit phone numbers
  • Edit email addresses
  • Display and edit archived chats
  • New add-chat and add-contact pages
  • Display contacts and find contacts with their phone number or email address
  • Discover public chats on the user's homeserver
  • Registration (currently only working with ubports.chat and NOT with matrix.org due captchas)
  • Register and login with phone number
  • Edit identity-server
  • Add in-app viewer for the privacy policy

Construct Docker image

mujx has created a new Docker image for Construct, the C++ homeserver:

Construct has a new docker image (https://github.com/matrix-construct/docker-construct) which will ease the process of deployment

Translations for Quaternion

If it didn't seem that kitsune was busy enough, he also announced translations for Quaternion (the original client based on libQMatrixClient):

Quaternion is now localisable! The web platform for translation will be set up in the nearest days; meanwhile those who can't wait can just take the repo, look how the German translation is made and copy!

That's all we can tell you!

Safe travels to Cadair, who is off to represent Matrix at the GSOC 2018 mentors summit. Check out Matrix Live below:

The story of Giveth’s new Matrix chatbot

11.10.2018 00:00 — General Eli Etzioni

Guest post today from GivethGiveth is re-engineering charitable giving, by creating an entirely free, open-source platform, built on the Ethereum Blockchain. Our system cuts out bureaucracy and enables nonprofits to create a high level of transparency and accountability towards Givers.


Giveth's new chatbot in action!

Online or offline, joining a new community always requires some adjustment. Even the most open, inclusive communities have shared knowledge and shared practices which new members learn as they participate.

I recently joined Giveth's Riot community, where the majority of Giveth's communication occurs. Immediately upon joining, I received the message pictured above from the Giveth Bot, kindly encouraging me to download Riot mobile and change my notifications to mention-only. The bot shortened my adjustment period by giving me key tidbits of information that everyone in Giveth's community knows, but that may have taken me time to pick up on my own. This blog post will cover how the Giveth Bot came to be, what it is capable of, and where the project is headed in the future.

The Giveth Bot actually started out as an attempt to solve a completely different problem: helping Giveth efficiently distribute internal reward points. Giveth's system for rewarding people who meaningfully contribute to the project is called RewardDAO. “If someone contributes in a meaningful way, a core contributor from each of the Giveth Campaigns can dish them points to recognize the contribution”, describes Cleo in an article explaining how RewardDAO works. At the end of each month, contributors receive Ether based on how many points they have earned.

The Giveth RewardDAO motto. Photo from https://medium.com/giveth.

However, any time that a core contributor dished points to someone, they had to record who received the points, and how many, on a spreadsheet. In search of a better way, Giveth opened up the project of automating this system to the social coding hub, a community of altruistic developers looking to tackle impactful and interesting projects, offering a 2 eth bounty for a solution.

A lot of great work was submitted, and ultimately Deam's ( @deamlabs) code was chosen to power the bot and the code for the pointsbot itself was further developed and refined by Frederik Bolding. Now, by using a command of the form “!dish [number] [type] points to [contributor] for [contribution]”, Giveth core contributors can distribute points as needed, and the bot will automatically update the spreadsheet accordingly.

The Giveth Bot dishing points like a champion!

Once the bot's framework was established, chatbot features were added. In addition to the welcome message I received, the bot gives custom welcome messages in each of Giveth's different rooms, allows Matrix users to have 1-on-1 chats with it, and listens for keywords and sentences it recognizes in rooms and private chats. Riot is built on top of an open-source protocol called Matrix. Matrix has a javascript standard development kit (SDK), which the bot uses to detect events occurring in each of the Riot rooms and chats that it is a part of.

Giveth began by using Slack, but switched to Riot to support Matrix's decentralized, open-source model, which which aligns far more with Giveth's own business model and values. The Giveth Bot is a great example of how Matrix enables users to build their own solutions to problems. In the future, we hope that the Giveth Bot will be able to interact directly with the Ethereum Blockchain, and that more analytics and measurement tools can be incorporated. And of course, we welcome any and all feedback on the Giveth Bot!

Giveth is an open-source platform for building decentralized altruistic communities. Anyone interested in getting involved should head to join.giveth.io

Interested in checking out the Giveth Bot's inner workings? All code is available at https://github.com/Giveth/giveth-bot .

Interested in learning DApp development or helping out with cool projects like the Giveth Bot? Check out the social_coding Riot channel , tell us what you're interested in, and help build awesome stuff!

This Week in Matrix 2018-10-05

05.10.2018 00:00 — This Week in Matrix Ben Parsons

Bridges

Half-Shot is supposedly back at university, but he's productive enough to have THREE updates this week (delivered in ascending order of interestingness):

matrix-appservice-irc 0.11.2:

https://github.com/matrix-org/matrix-appservice-irc/releases/tag/0.11.2 is just a hotfix release containing a few things we fixed in develop so yay.

matrix-appservice-node v0.3.5:

https://github.com/matrix-org/matrix-appservice-node/blob/master/CHANGELOG.md#v035 is a long overdue update that just updates dependencies so we are no longer exposing lots of vulnerabilities
Updated body-parser, express, morgan and request packages to fix security vulnerabilities.

matrix-appservice-bridge 1.7.0 released:

matrix-appservice-bridge is out! Bundled in this version are major updates to dependencies (to stop the warnings), a logging component for quick setup of pretty winston logs and a provisoner "validator" to allow bridge admins to curate which bridges can be linked into the same room to stop the dreaded double bridge issue.
More details can be found at https://github.com/matrix-org/matrix-appservice-bridge/releases/tag/1.7.0

Informo news project

Informo is an ambitious project hoping to be a "decentralised news network, making information accessible". The project lives at https://github.com/Informo, but for now you can join #discuss:weu.informo.network to get their latest news.

vadb provided an update:

Hey there, here's a quick update to let you know that we're making great progress towards the completion of phase 1 of Informo's specifications' documentation (i.e. general outline without going to much in depth into the technical specifications)! I've also updated the board to cut the big "Basic spec" task down into smaller tasks to give a more precise insight of our current progress
Once that phase 1 is done we'll publish the whole thing and migrate our work to a public GitHub repository so people can get to know Informo a bit better and contribute towards making the doc better if they want

Riot

riot-web 16.5 released, with matrix-js-sdk 0.11.1:

Go see the latest changes and make sure to start using room member Lazy Loading with the latest matrix-js-sdk.

Libraries

Black Hat is writing an Olm wrapper for Qt:

I am writing a Qt wrapper for Olm. It has APIs similar to python-olm

kitsune and Black Hat are working on libQMatrixClient:

libQMatrixClient is nearing its release 0.4.0, with low-level support of CS API (coincidentally) 0.4.0 added this week. Another very useful addition is caching avatar images to the disk, not only in memory, relieving the network from dozens of extra requests when a client powered by the library is restarted. All thanks for the avatar caching PR go to Black Hat!

SDKs

tulir on mautrix-go and mautrix-python:

My gomatrix fork is now mautrix-go and the basic client API stuff in mautrix-python is starting to be usable enough for me to start the rewrite of maubot in Python.
For a bit of background, maubot is a dynamic plugin-based bot system. It's currently written in Go, but due to many limitations in the Go plugin system, I decided to rewrite it in Python. matrix-python-sdk doesn't have asyncio or a maintainer, so I decided to make a new framework that combines my existing mautrix-appservice-python framework with a more generic client API layer. (tulir)

Homeservers

Max introduces his new server, Gridepo:

mxhsd work has resumed via a new project, Gridepo, which will be a Matrix/Grid dual-stack server. While mxhsd focused on researching the protocol and reverse-engineering the spec, Gridepo will follow whatever is specced in its Matrix compatible mode.

Synapse 0.33.6 released: >

It's literally the previous blog post!

we've been focusing on fixing a whole host of federation bugs to improve reliability and latency. Additionally we've squashed some py3 bugs, improved lazy loading and been working hard in the background to improve our CI infrastructure. Finally, we cleaned up the Docker file, the image is now half the size of 0.33.5.1's standing at 58 MB.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI.

DevOps and Hosting

Slavi on matrix-docker-ansible-deploy:

matrix-docker-ansible-deploy has received some more improvements lately. They're mostly about the ability to tweak things affecting performance: Synapse cache factor configuration, the ability to enable/disable user presence tracking. The logging situation has also been improved for all Docker containers, so that their output would not get logged twice (once by systemd's journal, once by Docker) - something which was previously causing unpredictably-high disk usage for long-running containers.

ananace, Synapse for kubernetes:

Updated the K8s tweaked packaging for Synapse 0.33.6 (And added a preliminary Python 3 version there as well)

swedneck (literally) on IPFS:

swedneck is hosting riot on IPFS: https://riot.swedneck.xyz, see also https://riot-all.swedneck.xyz for all the versions currently hosted there.

Bots

TravisR, always working on bots:

Through use of the https://emoncms.org API, the makerspace I help run now has power monitoring statistics available from within matrix.

Rooms

Coffee, maintaining #synapse:matrix.org:

#synapse:matrix.org became an official Matrix room! It also had a small face lift, changing from "Synapse Community" to "Synapse Admins", hopefully making its purpose and intended audience clearer.

That's all for now

That's all for now. No Matrix Live this week, but we'll be back next week with more Matrix news and another episode of Matrix Live!

Synapse 0.33.6 released!

04.10.2018 00:00 — Releases Neil Johnson

Right folks, time for Synapse 0.33.6.

These past few weeks we've been focusing on fixing a whole host of federation bugs to improve reliability and latency. Additionally we've squashed some py3 bugs, improved lazy loading and been working hard in the background to improve our CI infrastructure. Finally, we cleaned up the Docker file, the image is now half the size of 0.33.5.1's standing at 58 MB.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here.

Synapse 0.33.6

Features

  • Adding the ability to change MAX_UPLOAD_SIZE for the docker container variables. (#3883)
  • Report "python_version" in the phone home stats (#3894)
  • Always LL ourselves if we're in a room (#3916)
  • Include eventid in log lines when processing incoming federation transactions (#3959)
  • Remove spurious check which made 'localhost' servers not work (#3964)

Bugfixes

  • Fix problem when playing media from Chrome using direct URL (thanks @remjey!) (#3578)
  • support registering regular users non-interactively with register_new_matrix_user script (#3836)
  • Fix broken invite email links for self hosted riots (#3868)
  • Don't ratelimit autojoins (#3879)
  • Fix 500 error when deleting unknown room alias (#3889)
  • Fix some b'abcd' noise in logs and metrics (#3892#3895)
  • When we join a room, always try the server we used for the alias lookup first, to avoid unresponsive and out-of-date servers. (#3899)
  • Fix incorrect server-name indication for outgoing federation requests (#3907)
  • Fix adding client IPs to the database failing on Python 3. (#3908)
  • Fix bug where things occasionally were not being timed out correctly. (#3910)
  • Fix bug where outbound federation would stop talking to some servers when using workers (#3914)
  • Fix some instances of ExpiringCache not expiring cache items (#3932#3980)
  • Fix out-of-bounds error when LLing yourself (#3936)
  • Sending server notices regarding user consent now works on Python 3. (#3938)
  • Fix exceptions from metrics handler (#3956)
  • Fix error message for events with m.room.create missing from auth_events (#3960)
  • Fix errors due to concurrent monthly_active_user upserts (#3961)
  • Fix exceptions when processing incoming events over federation (#3968)
  • Replaced all occurrences of e.message with str(e). Contributed by Schnuffle (#3970)
  • Fix lazy loaded sync in the presence of rejected state events (#3986)
  • Fix error when logging incomplete HTTP requests (#3990)

Internal Changes

  • Unit tests can now be run under PostgreSQL in Docker using test_postgresql.sh. (#3699)
  • Speed up calculation of typing updates for replication (#3794)
  • Remove documentation regarding installation on Cygwin, the use of WSL is recommended instead. (#3873)
  • Fix typo in README, synaspse -> synapse (#3897)
  • Increase the timeout when filling missing events in federation requests (#3903)
  • Improve the logging when handling a federation transaction (#3904#3966)
  • Improve logging of outbound federation requests (#3906#3909)
  • Fix the docker image building on python 3 (#3911)
  • Add a regression test for logging failed HTTP requests on Python 3. (#3912)
  • Comments and interface cleanup for on_receive_pdu (#3924)
  • Fix spurious exceptions when remote http client closes connection (#3925)
  • Log exceptions thrown by background tasks (#3927)
  • Add a cache to get_destination_retry_timings (#3933#3991)
  • Automate pushes to docker hub (#3946)
  • Require attrs 16.0.0 or later (#3947)
  • Fix incompatibility with python3 on alpine (#3948)
  • Run the test suite on the oldest supported versions of our dependencies in CI. (#3952)
  • CircleCI now only runs merged jobs on PRs, and commit jobs on develop, master, and release branches. (#3957)
  • Fix docstrings and add tests for state store methods (#3958)
  • fix docstring for FederationClient.get_state_for_room (#3963)
  • Run notify_app_services as a bg process (#3965)
  • Clarifications in FederationHandler (#3967)
  • Further reduce the docker image size (#3972)
  • Build py3 docker images for docker hub too (#3976)
  • Updated the installation instructions to point to the matrix-synapse package on PyPI. (#3985)
  • Disable USE_FROZEN_DICTS for unittests by default. (#3987)
  • Remove unused Jenkins and development related files from the repo. (#3988)
  • Improve stacktraces in certain exceptions in the logs (#3989)
  • Pin to prometheus_client<0.4 to avoid renaming all of our metrics (#4002)

This Week in Matrix 2018-09-28

28.09.2018 00:00 — This Week in Matrix Ben Parsons

DSN Traveller

If you've spent any time using Matrix public rooms, you've probably seen the bot DSN Traveller. This is a post-grad project from Florian Jacob, an informatics student at the Karlsruhe Institute for Technology. This week, Florian handed in his thesis on Matrix!

In summary, I could show that Matrix has few large but many small servers. Large servers reduce the overall network load, but a significant fraction of the load is concentrated in them. Introducing more small servers would further increase the load concentration. The Matrix event graph as a Conflict-Free Replicated Data Type showed to be well-suited for reliable messaging and history synchronization, and is applicable beyond Matrix.
I'm now working on a scientific paper on the results, which will boil down the more than 80 pages of the thesis to something much more digestible. ? You'll hear it in TWIM when that is finished!

Room for discussions: #dsn-traveller:matrix.org
Website with more information on the DSN Traveller: https://dsn-traveller.dsn.scc.kit.edu/

This really is exciting stuff! The thesis will be made available online in the future (we'll post it here.) Florian is also hoping to continue his work into Matrix research:

I'm currently in the process of trying to secure funding for a doctorate with Matrix as the topic, as that's where I can proof experience.

#open-science:matrix.vgorcum.com

All this talk of academia led Mathijs and Cadair to create #open-science:matrix.vgorcum.com, room for discussion of Open Science topics.

Half-Shot IRC Connection Tracker

As you may know, although he's now back studying for the final year of his Computer Science degree, Half-Shot will continue to dedicate some time to bridge maintenance. He's been working on IRC Connection Tracker, the next gen bridge for Matrix-IRC:

The IRC connection tracker has had yet more code and love applied to it. The headline changes are:

  • We now have a fully working IRC client that can connect to an IRCd, join channels and chat. These client's persist over > the lifetime of the service.
  • There is a tool included with the service ircctl which allows you to spawn and use connections en masse. It also lets > you list the state of the currently connected clients.
  • Work has just begun on a client library for connecting this up to the bridge, but should be swiftly completed thanks to…
  • A brand new spec website in the works for describing the protocol (thanks Brendan for pointing me in the right direction)

Spec docs: https://half-shot.github.io/irc-conntrack/ Repo: https://github.com/Half-Shot/irc-conntrack/ Room: #irc-conntrack:half-shot.uk

Fractal

Alexandre Franke has the handle on Fractal, the Matrix client for GNOME:

This week in Fractal, more refactoring and small bugfixes. About 50 commits by 5 people, one of which made their first contributions this week (congrats Rasmus!).

Julian Sparber, who was part of GSOC 2018 on the Fractal team, has been working on Room History:

The room history refactor I was working on for fractal is upstream, now we can start to improve how messages are displayed and make the loading of older messages better.

Julian is also pleased to still be heavily involved with Fractal outside GSOC.

Synapse

Synapse 0.33.5.1 was released with Py3 beta support:

0.33.5.1 is an interesting release. On the one hand it contains the usual bug fixes and performance improvements of a point release, but it also our first versioned release where monolith installs can be run under Python 3.5 and 3.6! Python 3 support is very much in beta, so please be cautious but if you would like to try running under a py3 environment we'd love to get your feedback.

Check out https://hub.docker.com/r/avhost/docker-matrix/tags/ for Python 3 docker images of Synapse, look for the v0.33.5.1.dev.py3 tag.
Says Mathijs:

it is functional, but much like python3 support it is still a work in progress, hence the larger size

Matrix on Hyperboria

jcg has set up hypertrix.ovh:

I spent last night setting up hypertrix.ovh, a matrix server only listening on Hyperboria, a cjdns based end-to-end encrypted IPv6 overlay mesh network. I'd be glad if someone could be found to peer and federate with me there! Registration is open, but your client needs to be connected to Hyperboria to be able to talk to the server.

If you are currently using Hyperboria, go join hypertrix.ovh, or start your matrix server listening on it, and go chat to JC!

Seaglass

Lots of discussion about this project, specifically the question of how to efficiently render Rich Text. macOS does not make this easy, so a solution being considered is to use a WebKit for room rendering:

WebKit has the advantage of being super super fast on macOS, and also very low overheads
The current approach uses Cocoa NSTableViews and it's horrible because Apple clearly couldn't decide how they wanted them to work and therefore it's not very optimised
Moving to WebKit only adds about 16mb to the RAM usage and redraws far faster than the NSTableViews can when resizing etc, and we'll save a lot on the text formatting too which currently is a bit of a mess

Cadair and Half-Shot on the Slack Bridge

Cadair has been helping Half-Shot with bridge maintenance, specifically by contributing to matrix-appservice-slack.

matrix-appservice-slack is a Node project, built on matrix-appservice-bridge, which is designed to bridge messages and metadata between Slack and Matrix. These updates:

  • Improves pills substitutions to use the new slack mentions
  • Add Gitter style edit bridging
  • Adds a config option to specify the directory the user-store.db and room-store.db files go in
  • Converts slack snippets to inline code dumps in matrix

ma1uta tests for client

ma1uta has been adding tests to jmsdk.

Spectral

Black Hat has been working on the client formerly-known-as-Matrique: Spectral:

I added elevation shadows for some components, such as message bubble, panels, etc.

Native Tor onion service enabled for matrix.org and riot.im

Cloudflare now provide Onion routing, this service has been enabled for matrix.org and riot.im. Cloudflare have a thorough explanation which is worth reading: https://blog.cloudflare.com/cloudflare-onion-service/.

We just turned on the new native Tor onion service support for https://t.co/vidAnPoIo2 & https://t.co/UIjS6gDkvf in cloudflare; feedback welcome! https://t.co/keXC4bjo5F pic.twitter.com/nTRmGHCt8P

— Matrix (@matrixdotorg) September 28, 2018

Spec

September was mainly spent cleaning up loose ends on the Spec after all the releases at the end of August, and catching up on the never-ending maintenance burden of improving Synapse.  However, in October the plan is to to go back again to working full out on the S2S r0 release. Wish us luck...

Riot

  • Lazy loading members is now on by default on riot.im/develop - reducing Riot's RAM by 3-5x.  Please give it a go and test it before we ship it in Riot 0.17 (probably next week) so we can iron out any last bugs (which will probably look like user profiles going missing)
  • Lazy loading also ships by default in Riot/iOS in Testflight 0.7.4 - if you want in on Testflight let us know in #riot-ios:matrix.org and we'll share an invite link!
  • Lazy loading in Riot/Android coming real soon now; it's behind a labs flag on the develop branch if you want to experiment.
  • Travis has started attacking the Riot/Web 'First Impressions' project (starting with unbreaking onboarding in Riot/Web when GDPR consent is enabled)
  • Lots and lots of UX work from Nad on E2E, Communities, Onboarding and the overall redesign, complete with a redesign workshop with Jouni.
  • Aiming for end of Oct for first cut of redesign to be live as an experimental branch on riot.im.
  • Lots and lots of E2E implementation work in general; backups, cross-signing, and verification.

The end is nigh!

But only for this blog post! Check out Matrix Live below, and we'll see you back here next week. :D

https://youtu.be/zo4IH4nUQ9w

Synapse 0.33.5.1 released!

24.09.2018 00:00 — Releases Neil Johnson

Folks, Synapse 0.33.5.1 is here.

0.33.5.1 is an interesting release. On the one hand it contains the usual bug fixes and performance improvements of a point release, but it also our first versioned release where monolith installs can be run under Python 3.5 and 3.6! Python 3 support is very much in beta, so please be cautious but if you would like to try running under a py3 environment we'd love to get your feedback.

We've been running it ourselves for the past few weeks, and feel pretty good about it, not least the 2-3x improvement in RAM usage.

Currently the only way to run under python 3 is to download via github, there is no deb support as yet, though this will come as soon as we are confident to recommend python 3 as the default version.

We'll be blogging about our porting project in more detail in the future, so watch this space - exciting times!

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, for the first time, Synapse is now available from PyPI, pick it up here.

Synapse 0.33.5.1

Internal Changes

  • Fix incompatibility with older Twisted version in tests. Thanks @OlegGirko! (#3940)

Synapse 0.33.5

Features

  • Python 3.5 and 3.6 support is now in beta. (#3576)
  • Implement event_format filter param in /sync (#3790)
  • Add synapse_admin_mau:registered_reserved_users metric to expose number of real reaserved users (#3846)

Bugfixes

  • Remove connection ID for replication prometheus metrics, as it creates a large number of new series. (#3788)
  • guest users should not be part of mau total (#3800)
  • Bump dependency on pyopenssl 16.x, to avoid incompatibility with recent Twisted. (#3804)
  • Fix existing room tags not coming down sync when joining a room (#3810)
  • Fix jwt import check (#3824)
  • fix VOIP crashes under Python 3 (#3821) (#3835)
  • Fix manhole so that it works with latest openssh clients (#3841)
  • Fix outbound requests occasionally wedging, which can result in federation breaking between servers. (#3845)
  • Show heroes if room name/canonical alias has been deleted (#3851)
  • Fix handling of redacted events from federation (#3859)
  • (#3874)
  • Mitigate outbound federation randomly becoming wedged (#3875)

Internal Changes

  • CircleCI tests now run on the potential merge of a PR. (#3704)
  • http/ is now ported to Python 3. (#3771)
  • Improve human readable error messages for threepid registration/account update (#3789)
  • Make /sync slightly faster by avoiding needless copies (#3795)
  • handlers/ is now ported to Python 3. (#3803)
  • Limit the number of PDUs/EDUs per federation transaction (#3805)
  • Only start postgres instance for postgres tests on Travis CI (#3806)
  • tests/ is now ported to Python 3. (#3808)
  • crypto/ is now ported to Python 3. (#3822)
  • rest/ is now ported to Python 3. (#3823)
  • add some logging for the keyring queue (#3826)
  • speed up lazy loading by 2-3x (#3827)
  • Improved Dockerfile to remove build requirements after building reducing the image size. (#3834)
  • Disable lazy loading for incremental syncs for now (#3840)
  • federation/ is now ported to Python 3. (#3847)
  • Log when we retry outbound requests (#3853)
  • Removed some excess logging messages. (#3855)
  • Speed up purge history for rooms that have been previously purged (#3856)
  • Refactor some HTTP timeout code. (#3857)
  • Fix running merged builds on CircleCI (#3858)
  • Fix typo in replication stream exception. (#3860)
  • Add in flight real time metrics for Measure blocks (#3871)
  • Disable buffering and automatic retrying in treq requests to prevent timeouts. (#3872)
  • mention jemalloc in the README (#3877)
  • Remove unmaintained "nuke-room-from-db.sh" script (#3888)

This Week in Matrix 2018-09-21

21.09.2018 00:00 — This Week in Matrix Ben Parsons

Nheko 0.6.0 released!

Get latest stable releases from GitHub.

Features

  • Support for sending & receiving markdown formatted messages.
  • Import/Export of megolm session keys. (Incompatible with Riot)
  • macOS: The native emoji picker can be used.
  • Context menu option to show the raw text message of an event.
  • Rooms with unread messages are marked in the room list.
  • Clicking on a user pill link will open the user profile.

Spec Proposals: E2E Cross-signing and bi-directional key verification

uhoreg has written up a new work-in-progress proposal for E2E cross-signing.

Also, although it wasn't this week, I don't think that we have previously announced the proposal for bi-directional key verification using QR codes: https://github.com/matrix-org/matrix-doc/pull/1544

Jeon

Massive update from ma1uta about his Jeon project! This update brings Jeon into line with the most recent updates to the Client-Server, Application Service, Push and Identity APIs.

In ma1uta's words:

Jeon is a set of the java interfaces and classes which describes the Matrix API.

  • client-api: r0.4.0-1 corresponds to the r0.4.0 C2S API.
  • application-api: r0.1.0-1 corresponse to the r0.1.0 AS API,
  • push-api: r0.1.0-1 corresponds to the r0.1.0 PUSH API,
  • identity-api: r0.1.0-1 corresponds to the r0.1.0 IS API.
  • All artefacts available from the Maven Central Repository.
Major changes:
  • Full support for the corresponding Matrix api.
  • Changed version for displaying the Matrix api version.
  • Added support to the asynchronous responses.
Also updated the swagger schemas generated from the code: And the first hotfix: application-api r0.1.0-2 with fixed url (/transactions has been changes to the _matrix/app/v1/transactions). And this release will break all AS because synapse sends transactions to the old url.

Join #jeon:matrix.org to discuss the progress of this product more.

VoIP signalling support has landed in libQMatrixClient

Exciting times for libQMatrixClient!! Thanks kitsune, developer of libQMatrixClient and Quaternion:

After some pretty long time of being in a PR/fork, VoIP signalling support has landed in libQMatrixClient! Many thanks to mariogrip (the founder of UBports) for the initial code and to delijati (a developer behind uMatriks) for getting it to work with the most recent library.
The actual VoIP stack does not come included, client developers have to take whatever WebRTC implementation is available for their platform and glue the pieces together. However, as the example of uMatriks shows, this now becomes relatively easy if your platform is on good terms with WebRTC (like UBports). I look forward to further work with UBports community on keeping this platform a first-class Matrix citizen.

Go chat in #quaternion:matrix.org to see the ways libQMatrixClient is developing.

Matrique is now Spectral

After intense discussion, there is a new name for Matrique: Spectral. The repo now sites at https://gitlab.com/spectral-im/spectral, there is a new room at #spectral:encom.eu.org, and a new logo:

FluffyChat is getting some love from OpenStore, the official Ubuntu Touch app store: this week it was the featured app.

Matrix Corporal

@slavi:devture.com, creator of Matrix Corporal (a Matrix server configuration tool - "Kubernetes for Matrix"):

Matrix Corporal has received some updates over the past few weeks since its initial release: a couple of additional HTTP APIs for retrieving/destroying user access tokens; more consistency with the Matrix Client-Server specification when it comes to error responses; faster reconciliation for user accounts that are joined to many/large rooms.

matrix-docker-ansible-deploy

Another project from @slavi:devture.com, for those who prefer their DevOps ansible-flavoured:

matrix-docker-ansible-deploy now also helps you set up service discovery as per the .well-known Matrix specification.

jcg ansible PR for matrix notifications

jcg has an upstream PR to have matrix notifications in ansible. Combined with Slavi's matrix-docker-ansible-deploy above, you can get Matrix notifications about issues with your Matrix deployment…

Seaglass E2E + self-update

neilalexander:

Seaglass end-to-end encryption support is now complete, including device verification and blacklisting, key sharing requests, key import and export (which should be compatible with Riot) and re-requesting keys

This is really exciting news for macOS matrix users!

I'm also working on auto-updating Seaglass with Aaron Raimist's help in addition to finishing E2E support :-)

:-) is right!

ma1uta jmsdk

ma1uta must have been in a work-on-Matrix mode this week, because he has also updated jmsdk:

I have released a new version of the java client (https://github.com/ma1uta/jmsdk/tree/master/client-sdk). The new client works is asynchronous mode, each method doesn't block the thread and return the CompletableFuture (promise in java). Then you can block thread to get the response or build a asynchronous promises chain.

Finally:
ma1uta is also looking forward to the release of Java 11:

with the Curve25519 key agreement (http://openjdk.java.net/jeps/324) and will try make a pure java implementation of the olm/megolm. Just for fun. :)

synapse-purge

Maze, seeing that his synapse database was already at several gigabytes, decided to produce a tool to shrink it:

The synapse-purge tool allows homeserver admins to free disk space by purging old room events from the synapse database. It is an alternative for synpurge which currently does not work correctly.

Configuration is minimalistic at the moment. Meaning it purges all remote rooms on the server with a globally configured preservation period.

synapse

andrewsh: 0.33.4 uploaded to Debian's stretch-backports, pending approval.

0.33.5rc1 is now available, with the big news being the inclusion of support for Python 3.5 and 3.6! Hawkowl's Py3 has merged for monoliths and is working pretty well, looking like 2-3x RAM improvement. Please help us test!

Erik's state compressor tool is pretty much finished, we've been starting to run it on things and it reduces disk usage for the state group table by at least 10x.

The only catch is that it's quite DB heavy whilst it runs, so we haven't run it on Matrix.org yet.

Fractal

Alexandre Franke and the Fractal team:

refactoring of the history and other parts is going on in the master branch of Fractal. We also cleaned up build and dependency related bits.

maubot and sedbot

tulir:

I made some updates to maubot and fixed most of the sedbot (S. Edbot) issues people had reported.

tulir used maubot to create a factorial bot: >

I might also make some useful bots soon

And so it was - late breaking news that maubot has been used to develop a Dictionary-definition-bot! Not available for public use yet but it proves that the project is useful!

Riot Web

Lazy Loading remains the focus, we're getting closer with more bugs solved this week! To enable Lazy Loading room members and get speed and memory benefits in Riot, use the develop branch and enable "Lazy Loading" under "Labs" in the settings.

Lots of final bug hunting for lazy loading - this is taking longer than you might expect because we're doing end-to-end CI everywhere.

Lots of work on E2E, Dave has been working on:

UI for e2e key backup that's waiting for some lower level bits
and hopefully our e2e core code is moving from asm.js to webassembly making it, by current estimations, significantly faster.

Redesign work continuing as well - Bruno has been working on it this week, Jouni the designer will visit next week to continue the process.

Nad has joined us to help with design bandwidth and is working on the onboarding flows for the redesign as well as fixing all the UX issues in Communities!

Riot Mobile + Mobile SDKs

Lots of work on Lazy Loading - to be released along with Riot Web.

Bridges

Half-Shot is joining us to work part-time on bridges going forwards - this is great news, especially for his connection-based IRC bridging antics as well as catching up on the PR and maintenance backlog for the IRC bridge and Slack bridge.

Modular

Modular (Hosted Homeservers) has first customers; if you want to give it a go please let us know!

Finally

Thanks for reading, take a look at Matrix Live below!

This Week in Matrix 2018-09-14

15.09.2018 00:00 — This Week in Matrix Ben Parsons

Dimension

Update (this got lost in the original post; sorry Travis!): Dimension received a security update - if you run your own Dimension instance it is strongly recommended you update right away. Telegram bridge support in Dimension is underway, with more updates expected for next week in Matrix.

Clients

Fluffychat

It's been some months since we checked in with FluffyChat. If you're a Ubuntu Touch user, or have a device running it, you should see the progress that has been made recently on this Matrix client. Collected changelog 0.5.0 to latest (0.5.4):

  • Search chats
  • Chat avatars
  • Search users in chats
  • Security & Privacy settings:
  1. Disable typing notifications
  2. Auto-accept invitations
  • New message status:
  1. Sending: Activity indicator
  2. Sent: Little cloud
  3. Received: Tick
  4. Seen by someone: Usericon
  • Display stickers
  • Minor UI improvements
  • FluffyChat now automatically opens the link to the matrix.org consens
  • Updated translations

Seaglass

Neil has been keeping up the pace with Seaglass development:

Seaglass has had a substantial rewrite to the room cache to help improve reliability and reduce crashes, better thumbnail behaviour on inline images, various tiny visual tweaks, in-window blending, support for encryption key sharing requests for E2E rooms.

Rendering performance has been massively increased (if you ignore the occasional bug). Resizing the window shouldn't be so slow anymore and a lot of avatar image operations are no longer repeated unnecessarily

Other than that this week has mostly featured lots and lots of bug fixes, hopefully lots of crashes fixed.

Screenshot below shows the new E2E UI:

Quaternion

When not escaping typhoons, kitsune has found some time to continue work on Quaternion:

Quaternion's master branch is alive again - it's been prone to crashes in the last two weeks, now it shouldn't. Feel free to try the new room list organised by tag!

SimpleMatrix

MTRNord has been working on SimpleMatrix:

SimpleMatrix now supports Basic messages sending (with Commonmark) and basic receiving of messages.

miniVector

Marcus has re-packaged miniVector for F-Droid:

There's now a second matrix client available in F-Droid: https://f-droid.org/packages/com.lavadip.miniVector/

This is a fork of Riot Android done by hrjet, f-droid release done by me. It's removing mostly jitsi group call functionality and some other smaller stuff. In doing so it manages to require far less permissions and is also only 12 MB in size instead of riots 20 MB.

Matrique

Black Hat:

Matrique gained alpha support for multiple accounts

This is thanks to leaning on libqmatrixclient's native multiple account support!

Riot Web 0.16.4 released

This is pretty much a maintenance release - fixing the DM avatar regression that crept in with 0.16.3, adding better support for warning users when their client hasn't yet synced with the server, and the final bits of work needed before we can turn on membership Lazy Loading in the upcoming Riot 0.17.

Full changelogs as always are split over the three projects which make up Riot/Web:

SDKs and Libraries

libQMatrixClient ecosystem

As you may know, Matrique, led by Black Hat, and Quaternion, led by kitsune, are both projects build using libQMatrixClient, a Qt5 library from kitsune designed for writing Matrix clients. While kitsune has been working on the library for some time, Black Hat has also now started making contributions:

libQMatrixClient now has a pkg-config file to further ease clients building on Linux systems, as well as more convenient API to track read markers if all users, not just of the local one.

matrix-js-sdk v0.11.0 released

This release contains support for lazy loading room members, and also some breaking changes relating to startClient().

  • Support for lazy loading members. This should improve performance for users who joined big rooms a lot. Pass to lazyLoadMembers = true option when calling startClient.
  • MatrixClient::startClient now returns a Promise. No method should be called on the client before that promise resolves. Before this method didn't return anything.
  • A new CATCHUP sync state, emitted by MatrixClient#"sync" and returned by MatrixClient::getSyncState(), when doing initial sync after the ERROR state. See MatrixClient documentation for details.
  • RoomState::maySendEvent('m.room.message', userId) & RoomState::maySendMessage(userId) do not check the membership of the user anymore, only the power level. To check if the syncing user is allowed to write in a room, use Room::maySendMessage() as RoomState is not always aware of the syncing user's membership anymore, in case lazy loading of members is enabled.

Synapse

Synapse 0.33.4 was released, with a whole host of bug fixes, some enhancements to resource usage management and a bunch of internal changes in readiness for room member state lazy loading and our ongoing port to Python 3.

Meanwhile, Python 3 support for monolithic (non-worker) Synapses has finally landed on the develop branch, thanks to massive work from hawkowl and notafile - if you want to help us test and flush out any remaining byte/utf8 style errors, please create a virtualenv for python 3.6 or 3.5 (twisted doesn't support 3.7 yet) and point the develop branch of Synapse at it, tail the logs for ERRORs and report them via Github if/when you see them.  In practice it seems pretty stable though, and noticeably reduces RAM and speeds things up thanks to improved GC and general performance work in Python.

We've also discovered that jemalloc works very well at improving RAM usage on Python 2 under Linux (we haven't tried it on Python 3 yet) by providing a more fragmentation-resistant malloc implementation; if you are having problems with your Synapse RAM spiking up we recommend giving it a go.  All of the Matrix.org server is using it now.

Also, lots of ops work this week; Erik has prototyped a new storage strategy for state groups which shrinks storage requirements by 10x, we'll be applying this shortly to Matrix.org otherwise we're going to run out of disk space.  There was also a regression on Synapse develop on federation, where outbound requests would get stuck and never retry - this impacted the matrix.org server badly over the course of the week, but as of Friday night we have a workaround in place.  We're not aware of it affecting anyone other than the matrix.org deployment (and we haven't got to the root cause yet).

Construct homeserver progress

This week:

Added notification counts which show up in Riot now, and expanded support for g++-7 and 8 instead of just g++-6. Construct repository is found at: https://github.com/matrix-construct/construct.

IRC Connection Tracker

Half-Shot is continuing to work on the project to split out IRC connection management from the IRC bridge, letting the bridge be restarted without interrupting IRC connections!

The project is going quite well, and is going to be used on matrix.org once production ready which will really speed up upgrades and give us near zero downtime indifferent to the size of the bridge.

At the moment the project has the ability to spin up and maintain connections, however the connections are not supporting IRC fully yet as there are bits to do on the parsing and maintaining state side. There is also work on a top-like tool to visualise and control the service outside of the bridge so we can quickly handle any oddities if they come up. Finally, it allows you to hot reload the configuration without dropping existing connections!

On the work done to support this on matrix-appservice-bridge, there is basic support for stating connections on the bridge but it's in early stages at the moment.

Spec

Travis has been tidying up loose bits on the Matrix spec this week:

In practice, finalising the S2S API is now blocked on proving the implementation on Synapse; work on this will resume next week and then we'll document the end result and ship the r0 at last.  Timings are going to be completely determined by available manpower and what level of ops distractions we face (c.f. the Synapse section above...).  Whilst we're waiting for the final S2S details to get hashed out, Travis is going to be helping on Riot dev, to try to stop stuff like this, as there's no point in having the platonic ideal of a perfect spec if actual users are unable to benefit from it.

#matrix-dev

#matrix-dev:matrix.org was reborn as a new room a couple of weeks ago to flush out old corrupted events, but maybe not everyone knows. Come join #matrix-dev:matrix.org, it's a starting point for all developers looking to build on the platform.  We're also rebuilding #test:matrix.org and #riot:matrix.org, although once we ship the new state resolution

A sneak peek at Modular...

Finally, there's been a massive amount of work on the New Vector side of things to soft-launch Modular - a paid hosting platform for Matrix servers (and, in future, paid integrations).  At this point we're looking for early adopters who want a dedicated Riot+Synapse for communities or companies of 50 or more users - but don't want to have to run it themselves.  Modular takes the homeserver hosting we've already been providing for Status, TADHack and others, and turns it into a mass-market product.  The pricing for early adopters is over 5x cheaper than Slack, so if you've been dying to have a reliable, fast and expertly maintained homeserver without any of the headaches of admining one yourself, please head over to https://modular.im and give it a whirl and let us know how it goes!  This is also a great way to support Matrix development in general, as money from Modular will directly keep the core Matrix team funded to work on Matrix.  Once we're happy with the soft-launch and have incorporated any feedback we'll start yelling about it as loud as we can :)

Matrix Live

We've had a bit of an accidental hiatus on Matrix Live thanks to getting submerged all the different project endgames happening atm (spec releases, lazy loading, Modular, Riot redesign etc), and for the last few Fridays we've got to midnight and beyond with too much still on the todo list to justify recording a video.  But to avoid completely falling behind, here's a slightly exhausted Saturday morning update instead (warning: Matthew is not a morning person).

Synapse 0.33.4 released!

11.09.2018 00:00 — Releases Neil Johnson

Roll up, roll up, get it while it's hot, Synapse 0.33.4 is here.

This release brings together a whole host of bug fixes, some enhancements to resource usage management and a bunch of internal changes in readiness for room member state lazy loading and our ongoing port to Python 3 (we are hoping to ship a py3 test candidate rsn!).

As ever, you can get the new update from https://github.com/matrix-org/synapse/releases/tag/v0.33.4 or any of the sources mentioned at https://github.com/matrix-org/synapse.

Features

  • Support profile API endpoints on workers (#3659)
  • Server notices for resource limit blocking (#3680)
  • Allow guests to use /rooms/:roomId/event/:eventId (#3724)
  • Add mau_trial_days config param, so that users only get counted as MAU after N days. (#3749)
  • Require twisted 17.1 or later (fixes #3741). (#3751)

Bugfixes

  • Fix error collecting prometheus metrics when run on dedicated thread due to threading concurrency issues (#3722)
  • Fix bug where we resent "limit exceeded" server notices repeatedly (#3747)
  • Fix bug where we broke sync when using limit_usage_by_mau but hadn't configured server notices (#3753)
  • Fix 'federation_domain_whitelist' such that an empty list correctly blocks all outbound federation traffic (#3754)
  • Fix tagging of server notice rooms (#3755#3756)
  • Fix 'admin_uri' config variable and error parameter to be 'admin_contact' to match the spec. (#3758)
  • Don't return non-LL-member state in incremental sync state blocks (#3760)
  • Fix bug in sending presence over federation (#3768)
  • Fix bug where preserved threepid user comes to sign up and server is mau blocked (#3777)

Internal Changes

  • Removed the link to the unmaintained matrix-synapse-auto-deploy project from the readme. (#3378)
  • Refactor state module to support multiple room versions (#3673)
  • The synapse.storage module has been ported to Python 3. (#3725)
  • Split the state_group_cache into member and non-member state events (and so speed up LL /sync) (#3726)
  • Log failure to authenticate remote servers as warnings (without stack traces) (#3727)
  • The CONTRIBUTING guidelines have been updated to mention our use of Markdown and that .misc files have content. (#3730)
  • Reference the need for an HTTP replication port when using the federation_reader worker (#3734)
  • Fix minor spelling error in federation client documentation. (#3735)
  • Remove redundant state resolution function (#3737)
  • The test suite now passes on PostgreSQL. (#3740)
  • Fix MAU cache invalidation due to missing yield (#3746)
  • Make sure that we close db connections opened during init (#3764)
  • Unignore synctl in .dockerignore to fix docker builds (#3802)