Right folks, this is our first post 1.0 release, which means that we have now officially dropped support for Python 2 and Postgres 9.4. This means that we can start making use of Python 3 specific features and you should expect lots of associated performance wins over the coming months. See the upgrade notes for more.

Synapse 1.1.0 also contains a reworked approach to the Docker image, as well lots of performance improvements with special focus on DB IO - expect more to come in this area.

Special thanks to community member Alexander Trost for rounding out our SAML support and also to Daniel Hoffend for contributing the ability to disable local password authentication.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our Synapse installation guide page

The changelog since 1.0.0 follows:

🔗Synapse 1.1.0 (2019-07-04)

As of v1.1.0, Synapse no longer supports Python 2, nor Postgres version 9.4. See the upgrade notes for more details.

This release also deprecates the use of environment variables to configure the docker image. See the docker README for more details.

No changes since 1.1.0rc2.

🔗Synapse 1.1.0rc2 (2019-07-03)

🔗Bugfixes

• Fix regression in 1.1rc1 where OPTIONS requests to the media repo would fail. (#5593)
• Removed the SYNAPSE_SMTP_* docker container environment variables. Using these environment variables prevented the docker container from starting in Synapse v1.0, even though they didn't actually allow any functionality anyway. (#5596)
• Fix a number of "Starting txn from sentinel context" warnings. (#5605)

🔗Internal Changes

• Update github templates. (#5552)

🔗Synapse 1.1.0rc1 (2019-07-02)

As of v1.1.0, Synapse no longer supports Python 2, nor Postgres version 9.4. See the upgrade notes for more details.

🔗Features

• Added possibility to disable local password authentication. Contributed by Daniel Hoffend. (#5092)
• Add monthly active users to phonehome stats. (#5252)
• Allow expired user to trigger renewal email sending manually. (#5363)
• Statistics on forward extremities per room are now exposed via Prometheus. (#5384, #5458, #5461)
• Add --no-daemonize option to run synapse in the foreground, per issue #4130. Contributed by Soham Gumaste. (#5412, #5587)
• Fully support SAML2 authentication. Contributed by Alexander Trost - thank you! (#5422)
• Allow server admins to define implementations of extra rules for allowing or denying incoming events. (#5440, #5474, #5477)
• Add support for handling pagination APIs on client reader worker. (#5505, #5513, #5531)
• Improve help and cmdline option names for --generate-config options. (#5512)
• Allow configuration of the path used for ACME account keys. (#5516, #5521, #5522)
• Add --data-dir and --open-private-ports options. (#5524)
• Split public rooms directory auth config in two settings, in order to manage client auth independently from the federation part of it. Obsoletes the "restrict_public_rooms_to_local_users" configuration setting. If "restrict_public_rooms_to_local_users" is set in the config, Synapse will act as if both new options are enabled, i.e. require authentication through the client API and deny federation requests. (#5534)
• The minimum TLS version used for outgoing federation requests can now be set with federation_client_minimum_tls_version. (#5550)
• Optimise devices changed query to not pull unnecessary rows from the database, reducing database load. (#5559)
• Add new metrics for number of forward extremities being persisted and number of state groups involved in resolution. (#5476)

🔗Bugfixes

• Fix bug processing incoming events over federation if call to /get_missing_events fails. (#5042)
• Prevent more than one room upgrade happening simultaneously on the same room. (#5051)
• Fix a bug where running synapse_port_db would cause the account validity feature to fail because it didn't set the type of the email_sent column to boolean. (#5325)
• Warn about disabling email-based password resets when a reset occurs, and remove warning when someone attempts a phone-based reset. (#5387)
• Fix email notifications for unnamed rooms with multiple people. (#5388)
• Fix exceptions in federation reader worker caused by attempting to renew attestations, which should only happen on master worker. (#5389)
• Fix handling of failures fetching remote content to not log failures as exceptions. (#5390)
• Fix a bug where deactivated users could receive renewal emails if the account validity feature is on. (#5394)
• Fix missing invite state after exchanging 3PID invites over federaton. (#5464)
• Fix intermittent exceptions on Apple hardware. Also fix bug that caused database activity times to be under-reported in log lines. (#5498)
• Fix logging error when a tampered event is detected. (#5500)
• Fix bug where clients could tight loop calling /sync for a period. (#5507)
• Fix bug with jinja2 preventing Synapse from starting. Users who had this problem should now simply need to run pip install matrix-synapse. (#5514)
• Fix a regression where homeservers on private IP addresses were incorrectly blacklisted. (#5523)
• Fixed m.login.jwt using unregistered user_id and added pyjwt>=1.6.4 as jwt conditional dependencies. Contributed by Pau Rodriguez-Estivill. (#5555, #5586)
• Fix a bug that would cause invited users to receive several emails for a single 3PID invite in case the inviter is rate limited. (#5576)

🔗Updates to the Docker image

• Add ability to change Docker containers timezone with the TZ variable. (#5383)
• Update docker image to use Python 3.7. (#5546)
• Deprecate the use of environment variables for configuration, and make the use of a static configuration the default. (#5561, #5562, #5566, #5567)
• Increase default log level for docker image to INFO. It can still be changed by editing the generated log.config file. (#5547)
• Send synapse logs to the docker logging system, by default. (#5565)
• Open the non-TLS port by default. (#5568)
• Fix failure to start under docker with SAML support enabled. (#5490)
• Use a sensible location for data files when generating a config file. (#5563)

🔗Deprecations and Removals

• Python 2.7 is no longer a supported platform. Synapse now requires Python 3.5+ to run. (#5425)
• PostgreSQL 9.4 is no longer supported. Synapse requires Postgres 9.5+ or above for Postgres support. (#5448)
• Remove support for cpu_affinity setting. (#5525)

🔗Improved Documentation

• Improve README section on performance troubleshooting. (#4276)
• Add information about how to install and run black on the codebase to code_style.rst. (#5537)
• Improve install docs on choosing server_name. (#5558)

🔗Internal Changes

• Add logging to 3pid invite signature verification. (#5015)
• Update example haproxy config to a more compatible setup. (#5313)
• Track deactivated accounts in the database. (#5378, #5465, #5493)
• Clean up code for sending federation EDUs. (#5381)
• Add a sponsor button to the repo. (#5382, #5386)
• Don't log non-200 responses from federation queries as exceptions. (#5383)
• Update Python syntax in contrib/ to Python 3. (#5446)
• Update federation_client dev script to support .well-known and work with python3. (#5447)
• SyTest has been moved to Buildkite. (#5459)
• Demo script now uses python3. (#5460)
• Synapse can now handle RestServlets that return coroutines. (#5475, #5585)
• The demo servers talk to each other again. (#5478)
• Add an EXPERIMENTAL config option to try and periodically clean up extremities by sending dummy events. (#5480)
• Synapse's codebase is now formatted by black. (#5482)
• Some cleanups and sanity-checking in the CPU and database metrics. (#5499)
• Improve email notification logging. (#5502)
• Fix "Unexpected entry in 'full_schemas'" log warning. (#5509)
• Improve logging when generating config files. (#5510)
• Refactor and clean up Config parser for maintainability. (#5511)
• Make the config clearer in that email.template_dir is relative to the Synapse's root directory, not the synapse/ folder within it. (#5543)
• Update v1.0.0 release changelog to include more information about changes to password resets. (#5545)
• Remove non-functioning check_event_hash.py dev script. (#5548)
• Synapse will now only allow TLS v1.2 connections when serving federation, if it terminates TLS. As Synapse's allowed ciphers were only able to be used in TLSv1.2 before, this does not change behaviour. (#5550)
• Logging when running GC collection on generation 0 is now at the DEBUG level, not INFO. (#5557)
• Reduce the amount of stuff we send in the docker context. (#5564)
• Point the reverse links in the Purge History contrib scripts at the intended location. (#5570)

Canonical version of this article at https://matrix.org/docs/guides/usage-of-matrix-nio

This article concerns matrix-nio, and asyncio. We'll build a simple "echo bot", meaning a bot which replies to messages with the text it has just read. Note that this article does not cover E2EE with matrix-nio.

🔗Instantiation and Login

First create a new venv, and install matrix-nio via pip. On the command line, run:

python3 -m venv env
source env/bin/activate
pip install matrix-nio

Next, create a new Python file, and open it for editing. We'll import everything we require for this tutorial:

from importlib import util
import asyncio
from nio import (AsyncClient, SyncResponse, RoomMessageText)

We're importing asyncio so we can use the AsyncClient class from matrix-nio.

Create a new instance of AsyncClient by passing the homeserver and username as arguments:

async_client = AsyncClient(
    "https://matrix.org", "%%YOUR-USERNAME-HERE%%"
)

Then login, and await the response:

response = await async_client.login("%%YOUR-PASSWORD-HERE%%")
print(response)

Of course, we are using an async client, and awaiting the response. Because of this, we must call the async_client.login() from an async method, like so:

async def main():
    response = await async_client.login("%%YOUR-PASSWORD-HERE%%")
    print(response)

asyncio.run(main())

Note that for versions of Python before 3.7 the asyncio syntax must be:

async def main():
    response = await async_client.login("%%YOUR-PASSWORD-HERE%%")
    print(response)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

The remainder of this tutorial assumes you are running everything from an async method.

The response string should look like:

Logged in as @pyconweb-bot:matrix.org, device id: ZBLAJHLKVP.

🔗Get into a /sync loop

To get updates from a Matrix homeserver to the client, the client makes a request to the /sync endpoint. In the matrix-nio AsyncClient, this is wrapped by the sync() method. We can get the latest updates:

sync_response = await async_client.sync(30000)

30000 means we will wait up to 30 seconds before returning. sync_response will now contain a Python object containing a mapping of the (JSON) response from the Matrix homeserver. We'll inspect this response in the next section.

In fact, we expect there to be updates regularly, so let's create a very simple loop:

while (True):
    sync_response = await async_client.sync(30000)
    print(sync_response) # note that this could be LARGE!
    # do some reading from sync_response

In this way, every time there is a response (i.e. new events) from the homeserver, they are made available in sync_response for processing, and we loop again.

🔗Explore the sync response object

sync_response can contain multitudes, depending on the rooms this user is part of, or has been part of. sync_response.rooms.join contains updates for the rooms which the current user is "joined to" (meaning, is a member of.)

Of these joined rooms, we are (perhaps!) most interested in the events on the timeline. These are stored in timeline.events, see below:

if len(sync_response.rooms.join) > 0:

    joins = sync_response.rooms.join
    for room_id in joins:
        for event in joins[room_id].timeline.events:
            print(event)

Message events are a specific type of event which contain an Instant Messenger message. We can check the type before proceeding:

for event in joins[room_id].timeline.events:
    if isinstance(event, RoomMessageText):
        print (event.body)

In these cases, where the event is a message to a room, the body field will contain the message text.

🔗Isolate specific message event objects

Knowing that we can get the message text from an event, we can read it to determine a response. Let's make a new variable and have it store some string we'll check for:

response_string = "!replybot"

Now let's suppose we're in our /sync loop, and just received an event. We can filter messages that are meant for our bot as follows:

if len(sync_response.rooms.join) > 0:
    joins = sync_response.rooms.join
    for room_id in joins:
        for event in joins[room_id].timeline.events:
            if hasattr(event, 'body') and event.body.startswith(response_string):
                print(event)

🔗Use room_send

To send messages, matrix-nio provides a room_send() method. There are three arguments:

• the room_id
• the message type, we will use "m.room.message"
• a JSON object representing the content of the message

Let's improve the example above, by sending back a message to echo the ones we isolated above:

joins = sync_response.rooms.join
for room_id in joins:
    for event in joins[room_id].timeline.events:
        if hasattr(event, 'body') and event.body.startswith(response_string):
            response_body = event.body.replace(response_string, "").strip()
            content = {
               "body": response_body,
               "msgtype": "m.text"
            }
            await async_client.room_send(room_id, 'm.room.message', content)

Now whenever the bot receives a message "!replybot some message" it will send back "some message".

🔗Use of /sync next_batch tokens

Finally, let's consider the importance of next_batch tokens. Whenever you receive a response from the /sync endpoint, the response will contain a "next_batch" field, which you then pass on the next request to ensure you have the latest messages. matrix-nio keeps track of this automatically, so it doesn't get repeated messages. However, when you stop the program and call the .sync() method again, how can you tell it where to start from? First let's get the latest next_batch token:

async def main():
    response = await async_client.login("%%YOUR-USERNAME-HERE%%", "")

    while (True):
        sync_response = await async_client.sync(30000)
        print(sync_response.next_batch) # this is the token

Then we'll write the token to a file:

async def main():
    response = await async_client.login("%%YOUR-USERNAME-HERE%%", "")

    while (True):
        sync_response = await async_client.sync(30000)

        # we write the token to a file here
        with open("next_batch","w") as next_batch_token:
            next_batch_token.write(sync_response.next_batch)

Once that token is written, we know we can re-use it for the first /sync/ request next time:

async def main():
    response = await async_client.login("%%YOUR-USERNAME-HERE%%", "")

    # we read the previously-written token...
    with open ("next_batch","r") as next_batch_token:
        # ... and well async_client to use it
        async_client.next_batch = next_batch_token.read()

    while (True):
        sync_response = await async_client.sync(30000)
        with open("next_batch","w") as next_batch_token:
            next_batch_token.write(sync_response.next_batch)

🔗Conclusion

With this, you can see that in very few lines, it's possible to write a working Matrix bot in Python, using matrix-nio.

Hi all,

A few weeks ago there was some discussion around the privacy of typical Matrix configurations, particularly how Riot's default config uses vector.im as an Identity Server (for discovering users on Matrix by their email address or phone number) and scalar.vector.im as an Integration Manager (i.e. the mechanism for adding hosted bots/bridges/widgets into rooms). This means that Riot, even if using a custom homeserver and running from a custom Riot deployment, will try to talk to *.vector.im (run by New Vector; the company formed by the core Matrix team in 2017) for some operations unless an alternative IS or IM has been specified in the config.

We haven't done as good a job at explaining this as we could have, and this blog post is a progress update on how we're fixing that and improving other privacy considerations in general.

Firstly, the reason Riot is configured like this is for the user's convenience: in general, we believe most users just want to discover other people on Matrix as easily as possible, and a logically-centralised server for looking up user matrix IDs by email/phone number (called third party IDs, or 3PIDs) is the only comprehensive way of doing so. Decentralising this data while protecting the privacy of the 3PIDs and their matrix IDs is a Hard Problem which we're unaware of anyone having solved yet. Alternatively, you could run a local identity server, but it will end up having to delegate to a centralised identity server anyway for IDs it has no other way to know about. Similarly, providing a default integration server that just works out of the box (rather than mandating the user configures their own) is a matter of trying to keep Riot's UX simple, especially when onboarding users, and especially given Riot's reputation for complexity at the best of times.

That said, the discussion highlighted some areas for improvement. Specifically:

1. When doing work on making Matrix GDPR compliant back in May 2018, we set up a single privacy policy for the services we run, and got users to agree to it by locking them out of the matrix.org homeserver until they did. However, we missed that users not on the matrix.org homeserver might still be using our Identity Service (IS) & Integration Manager (IM) without accepting the privacy policy. Over the last few weeks we've been working on addressing this - it turns out that it's a pain to fix, given the Identity Service doesn't have the concept of users, so tracking which users have agreed to the policy policy or not means some fairly major changes. The current proposal is to change the Identity Service to use a form of OpenID to authenticate users against their homeserver in order to check they've accepted the IS's terms of use - see MSC2140 for the gory details.

Meanwhile, Riot is being updated to prompt the user to accept the IS & IM terms of use (if different to the HS's), and thus make it crystal clear to the user that they are using an IS & IM and that they have the option not to if desired - see https://github.com/vector-im/riot-web/issues/10167 and associated issues. This includes also explicitly prompting the user as to whether they want 3PIDs they provide at registration to be discoverable, as per https://github.com/vector-im/riot-web/issues/10091.

2. Riot on iOS & Android gives the option of scanning your local addressbook to discover which of your contacts are on Matrix. The wording explaining this wasn't clear enough on Android - which we promptly fixed. Separately, the contact details sent to the server are currently not obfuscated. This is partially because we hadn't got to it, and partially because obfuscating them doesn't actually help much with privacy, given an attacker can just scan through possible obfuscated phone numbers and email addresses to deobfuscate them. However, we've been working through obfuscating the contact details anyway by hashing as per MSC2134, which has all the details. We're also adding an explicit lookup warning in Riot/Web, as per https://github.com/vector-im/riot-web/issues/10093.

3. There was a bug where Riot/Web was querying the Integration Manager every time you opened a room, even if that room had no integrations (actually, it did it 3 times in a row). This got fixed and released in Riot/Web 1.2.2 back on June 19th.

4. Matrix needs to authenticate whether events were actually sent by the server that claimed to send them. We do this by having servers sign their events when they create them, and publishing the public half of their signing keys for anyone to query. However, this poses problems if you receive an event which is signed by a server which isn't currently online. To solve this, we have the concept of trusted_key_servers (aka notary servers), which your server can query to see if they know about the missing server's keys. By default, matrix.org is configured as Synapse's trusted notary, but you can of course change this. If you choose an unreliable server as the notary (e.g. by not setting one at all) then there's a risk that you won't be able to look up signing keys, and a splitbrain will result where your server can't receive certain events, but other servers in the room can. This can then result in your server being unable to participate in the room entirely, if it's missing key events in the room's lifetime.

   Our plan here is to get rid of notaries entirely by changing how event signing works as per MSC1228, but this is going to take a while. Meanwhile we're going to check Synapse's code to ensure it doesn't talk to the notary server unnecessarily. (E.g. it should be caching the signing keys locally, and it should only use the notary server if the remote server is down.)

5. When doing VoIP in Matrix, clients need to use a TURN server to discover their network conditions and perform firewall traversal. The TURN server should be specified by your homeserver (and each homeserver deployment should ideally include a TURN server). However, for users who have not configured a TURN server, Riot (on all 3 platforms) defaulted to use Google's public STUN service (stun.l.google.com). STUN is a subset of TURN which provides firewall discovery, but not traffic relaying. This slightly increased the chances of calls working for users without a proper TURN server, but not by much - and rather than fall back to Google, we've decided to simply remove it from Riot (e.g. https://github.com/matrix-org/matrix-ios-sdk/commit/24832a2b14fb72ae6f051d5aba40262d11eef65d). This means that VoIP might get less reliable for users who were relying on this fallback, but you really should be running your own TURN server anyway if you want VoIP to work reliably on your homeserver.

6. We should make it clearer in Riot that device names are world-readable, and not just for the user's own personal reference. This is https://github.com/vector-im/riot-web/issues/10216

As you can see, much of the work on improving these issues is still in full swing, although some has already shipped. As should also be obvious, these issues are categorically not malicious: Matrix (and Riot) literally exists to give users full control and autonomy over their communication, and privacy is a key part of that. These are avoidable issues which can and will be solved. It's worth noting that we have to prioritise privacy issues alongside all the other development in Matrix however: there's no point in having excellent privacy if there are other bugs stopping the platform from being usable.

We'll do another blog post to confirm once most of the fixes here have landed - meanwhile, hopefully this post provides some useful visibility on how we're going about improving things.

🔗Matrix Live 🎙

Matthew & Amandine talk to Ross Schulman; new Matrix.org Foundation Guardian!

🔗Dept of Status of Matrix 🌡

🔗Hottest GovTech Startup in Europe at The Europas tech awards

New Vector won Hottest GovTech Startup in Europe at The Europas tech awards last night for work on rolling out Matrix for France and elsewhere!

It was mainly judge-based, but public votes were used to filter.

We're super proud to have won the hottest Gov/Reg/Civic Tech startup at @TheEuropas for creating self-sovereign secure communications on top of @matrixdotorg for the Public Sector - particularly with @_DINSIC and @tchap_dinsic! (So proud we created a twitter account at last :D) pic.twitter.com/PGqoHx007T

— New Vector (@NewVectorHQ) June 27, 2019

🔗Rust Embedded community is moving to Matrix

They held a public vote, and Matrix won very convincingly.

🔗Dept of Spec 📜

• MSC2134 Identity Hash Lookups is making steady progress
  • Readers who like crypto/hashing are encouraged to join in the fun
• MSC2140 Terms of Service for ISes and IMs is nearing a conclusion
• MSC2108 Sync over Server Sent Events hopes to add an alternative sync method to long-polling
  • Finally servers pushing events!
• MSC2010 Adding client-side spoilers is a small feature but requires a lot of thought to integrate it seamlessly
  • Sorunome thought this would be easy
• Lots of small fixes across the board.

🔗Dept of Servers 🏢

🔗Synapse

Here's Neil:

With 1.0 shipped we are now starting to take a closer look at Synapse performance more generally and this will be a theme for us over the coming months. We want to improve not only large scale deployments such as Matrix.org but also optimise for smaller instances.

You may have seen a few trial servers run by core team members in matrix.org community rooms popping up and this is a precursor for a broader effort to make synapse more manageable on less powerful infrastructure. My own instance has been sat at a pretty steady 256MB of RAM.

Other than that, based on 1.0 feedback, we have been working on improving the Synapse upgrade path and expect to put out a new release next week containing the tweaks. Specifically this means improving configuration for Docker installs, and configuration management for sending emails.

We’re also implementing open tracing into Synapse, initially to help with e2ee debugging, but it will make tracking down strange behaviour easier more generally.

Finally we’re bringing our push server Sygnal kicking and screaming into 2019 and will upgrade to Python 3, drop gevent for twisted and update our vendor specific libraries, not mention improving the monitoring and alerting. We’ll also add in open tracing which will help hunt down push failures.

Finally finally, look out for a DAG visualisation tool written by GSOCer Eisha referenced elsewhere in TWIM - we consider this to be seriously cool, and can’t wait to start using it in anger.

Note: 1.0 is the last release to support python 2 and postgres 9.4, the next Synapse release will drop support for both, see https://matrix.org/blog/2019/04/08/synapse-deprecating-postgres-9-4-and-python-2-x for more.

🔗Dendrite

Dendrite received a few updates this week. Notably:

• Cnly has been implementing typing notifications support in #718
• anoa has been fixing up the flaky sytests in #631. The team is considering moving from CircleCI to BuildKite for tests in the future.
• Cnly has added a few issues for odd bits and bobs they’ve found around the codebase. This is much appreciated!

🔗Construct

• We now support building with Clang, and also GCC-9
• Work on version 3/4/5 rooms has been underway.

🔗Ruma

This Week in Ruma: https://ruma.dev/news/this-week-in-ruma-2019-06-23/

This week was spent working on a big revamp of ruma-events, the library that defines Rust types for the "events" used in Matrix. After some discussion in #ruma:matrix.org, I decided to make a move towards treating ruma-events as a higher-level library. Previously, ruma-events has more or less offered Rust types that are exact representations of the JSON structures used by Matrix. However, by representing events this way, it would be possible for users to easily create values that, while valid JSON, would be invalid events according to the specification. The way we're approaching this problem is by separating serialization/deserialization of JSON from validation of events.

🔗Jeon/JeonServer

ma1uta:

Jeon 0.9.0 release. It is a release candidate for Jeon 1.0.0 which complies with the Matrix stable release 1.0. Not a lot changes, just added missing endpoints and events. Also I started to work on JeonServer, a Matrix server written on java.

Jeon is a set of Java interfaces to Matrix APIs, JeonServer is a proposed homeserver.

🔗matrix-media-repo

TravisR:

matrix-media-repo has received some speed improvements and is generally nicer to memory when using the s3 datastore. Give it a try and leave feedback in #media-repo:t2bot.io.

🔗GSOC project: Matrix Visualisations 🎓

Eisha appeared with this terrific GSOC update:

The GSoC project “Matrix Visualisations” has made good progress during this first period:

• The implementation of the CS API backend has been completed to properly retrieve events from a room in real time.
• Many features have been added to the UI, here are some of them:
  • The DAG is displayed vertically, every node of the same “depth” are on the same level in the graph and each node has outgoing arcs for each of its previous events (if they have already been retrieved).
  • The node at the top of the DAG allows to fetch earlier events by selecting it.
  • Each node can have two different colors whether its “origin” is the HS the application is currently talking to or not.
  • The full JSON body of an event can be displayed by double clicking on its node.
  • It is possible to select which fields of the events will be directly included in the labels of the nodes.
• A (server-side) backend has been implemented so that the application can directly talk to the PostgreSQL database of Synapse. You can find it on this repo.

Note that the support of the Federation API has been postponed so I could work on this Synapse database backend. The UI of the application isn’t very beautiful or well-organised yet, as the effort is focused on the backends and core functionalities for now, but improvements will be made once these functionalities will be completed.

Remember you can check TWIM from last week to get a reference on the other three GSOC projects.

🔗Dept of SDKs and Frameworks 🏗

🔗matrix-bot-sdk

TravisR:

matrix-bot-sdk v0.4.0-beta.1 has been published with a bunch of improvements for appservices. There's still more planned before the final v0.4.0 release, however live testing is always better than unit tests. If you use the library, try npm install [email protected] and report any issues to #matrix-bot-sdk:t2bot.io.

🔗libQuotient, now with EncryptionManager

libQuotient introduces EncryptionManager, check out the PR for more info, and also review aa13q's GSOC update from last week.

🔗Ruby SDK

Ananace:

Just cut a 1.2.0 release of the Ruby SDK, including fixes for timeout handling, some general code cleanup and documentation work, and a collection of getters and setters for most of the specced room state types

🔗Dept of Bridges 🌉

🔗freenode IRC bridge restarted

The freenode IRC bridge was restarted with the following fixes:

• Some characters in IRC usernames (such as “|”) would cause IRC user’s messages to not appear on the Matrix side.
• Room upgrades are now handled a bit more cleanly

🔗mx-puppet-bridge platform

Sorunome has been working on her all-new bridge platform, mx-puppet-bridge:

This week soru has been working more on mx-puppet-bridge, together with mx-puppet-slack and the now-new mx-puppet-tox.

🔗mx-puppet-bridge

This is work on the underlying bridge, protocol implementations still have to do their thing

• room name, icon and topic (was already implemented but untested)
• ghost name and icon (was already implemented, but untested)
• file sending in both directions (image, audio, video, file, autodetect) (was already implemented, but untested)
• have own ghosts create rooms --> appservice bot is not needed, more invisible bridging (especially nice for 1:1 rooms!)
• implemented 1:1 room bridging
• implemented provisioning interface via appservice bot
• remote -> matrix presence handling
• remote -> matrix typing notifications
• matrix -> remote puppet name tracking
• matrix -> remote puppet avatar tracking
• mx-puppet-bridge uses caching for better performance
• Added linting

🔗mx-puppet-slack

Lots of work here, smoothening things out

• room name, icon and topic bridging
• ghost name and icon bridging
• file sending in both directions
• presence and typing notifs

🔗mx-puppet-tox

This is a new one! It bridges tox over to matrix...or, well, more acts like a client (as tox doesn't have multidevice). Basic chatting was already functioning with only around 300 lines of code! The node toxcore bindings seem to only include support for 1:1 chats (and not the new group chats), so only that is implemented.

• ghost name and icon bridging
• file senidng in both directions
• presence and typing notifs
• matrix -> tox name and avatar syncing

There is already substantial documentation available, and a Matrix room at #mx-puppet-bridge:sorunome.de.

🔗Dept of Clients 📱

🔗RiotX (Android)

• We are finalizing the MVP of RiotX. Many new features, along with many bug fixes this week:
  • Notifications for version with or without Firebase Cloud Messaging
  • Reply in e2e rooms
  • Change of DI tool (We are now using dagger2)
  • New settings, split into categories
  • New set of Emojis for quick reactions
  • New application icon
  • And many other little features
  • New disclaimer screen, displayed at first startup
  • New suggestion screen (based on bug report screen)
  • Min SDK version has been set to API 19 (Kitkat), mainly for security reasons, but also because we are using MotionLayout which is available only on API 18+.
• Remaining work to do before we can release the first beta version on the PlayStore:
  • Encrypt/Decrypt attachment in e2e rooms
  • Last event display in the room list
  • Test, test, test, and fix, fix, fix :)

🔗Integration of Matrix into Delphus

Pneumaticat:

Our company, Scintillating, has integrated Riot as an end-to-end encrypted chat, video, and voice call provider for our decentralized scientific study management system Delphus. We have created a method of linking Matrix IDs with Ethereum addresses to allow scientific researchers to look up participants and create chat rooms to talk with individuals in a privacy-preserving manner.

🔗Spectral

We didn't get a Spectral update for a few weeks! Black Hat reported:

Quite a few changes in Spectral in the past few weeks. The room list filter is improved, and it only shows rooms with unread notifications by default. User can optionally hide join/leave events. Empty avatar in direct chats is fixed. Each user now has a unique message bubble background color in the timeline.

🔗Pattle: version 0.8.0

Wilko, chief Pattler:

A new version of Pattle has been pushed to F-droid!

Although this isn't the biggest release, it's still a big step: the first release of iOS will be available! The build is currently still in review by Apple.

You can download the iOS app via TestFlight soon, join #app:pattle.im to get the link immediately when it's available!

🔗Other changes

• Add .well-known support!
• Automatically update homeserver property in 'Advanced' while typing username
• Fix messages being wrongly grouped by sender
• Fix direct chats staying marked as direct when someone else joins

🔗Install this release

F-Droid:

1. Add the following repo in F-droid: https://fdroid.pattle.im/?fingerprint=E91F63CA6AE04F8E7EA53E52242EAF8779559209B8A342F152F9E7265E3EA729
2. Install 'Pattle'

APK: Download from the assets of this release

🔗Issues

If you stumble upon any issues, please report them! You can login via GitHub and Gitlab.com, so it's really easy to do!

🔗Follow Pattle

Follow development in #app:pattle.im!

🔗Support me

If you would like to support me, you can now do so via Liberapay and Patreon.

I've invested a lot of money in making Pattle happen on iOS: MacBook, Apple Developer Program, and an iPhone. Pretty costly, so any donations will be greatly appreciated!

🔗continuum updates placeholder avatars

yuforia:

continuum tweaked the appearance of placeholder avatars. To make most users appear visually distinct, continuum has always used colors based on checksums and usernames to generate placeholders if any user doesn't have an image as avatar. In previous versions, it always used two characters of the name. In the new version, if the username contains ideographic (usually east-asian) characters, a single character would be used. The reason is that the number of ideographic characters is vast and duplicates are less common, and most of them are close to a square or circle in shape so a single one would fit the GUI component better.

🔗FluffyChat edges closer to E2E

Krille:

Last weekend hummlbach (from the UBports community) visited me and we worked 18 hours on implementing end2end encryption. We are now able to send encrypted messages. Key sharing and decrypting will follow.

Not available in the released version yet, but join #fluffychat:matrix.org for more info. Also: 18 hours! Woah.

🔗Quaternion

Quaternion received several updates recently, check out #quaternion:matrix.org for more info.

🔗Fractal released 4.1 dev version

Alexandre Franke:

Fractal released a 4.1 development version, which was added to the beta channel of flathub. danigm is eager to get 4.2 out soon and is trying to fix the last few bugs we want to see gone before then. He already opened a few merge requests.

🔗riot-web

• Updated the reactions UX with a new tooltip-style interaction
• Added edit history dialogue and other small editing tweaks
• Riot Desktop config options (such as default HS, theme, etc.) can now be overridden for easy customisation

🔗riot-iOS

• New reactions in the event contextual menu
• Edits in e2e rooms
• Still refining reactions and edits

🔗matrix-enact, read-only client which performs the contents of rooms with Web Audio API

Some time ago I made a toy: https://play.matrix.org/matrix-enact/

This is a read-only Matrix client, which takes the contents of a public room and "enacts" it, that is, it performs it using the Web Audio API in your browser

The original intention was to be a demonstration of what can be done with /context endpoint, but the project scope expanded a little. Hopefully people find it fun!

Get the source code here: https://github.com/benparsons/matrix-enact

🔗Dept of Ops 🛠

🔗matrix-docker-ansible-deploy

Slavi and his many helpers have been working on matrix-docker-ansible-deploy:

matrix-docker-ansible-deploy has seen a lot of work on bridging lately. All currently existing bridges (Discord, Facebook, IRC, Telegram, Whatsapp) have been redone in a way that makes their configuration completely playbook-managed, as well as extensible. Besides this, with Synapse v1.0 already out, we've taken the opportunity to simplify the installation instructions a bit. If you haven't upgraded recently, now would be a great time. As always, be sure to take a look at the CHANGELOG before doing so.

🔗Update from the Matrix on Debian team

https://matrix-team.pages.debian.net/blogue/2019/06/26/june-2019-matrix-on-debian-update/

This is an update on the state of Matrix-related software in Debian

🔗Dept of Identity 🛂

🔗mxisd forked

Following the announcement that mxisd would not longer be worked on, ma1uta has stepped up to provide support:

I forked mxisd (https://github.com/ma1uta/mxisd) and will provide support this project. You can ask about help in a new room #ma1sd:ru-matrix.org A new temporary name will be ma1sd (thanks Dandellion ). Due to changing maintainers I start to prepare the new 2.0.0 release and should audit code and dependencies. Also I forked matrix-synapse-rest-password-provider (https://github.com/ma1uta/matrix-synapse-rest-password-provider) because it often uses with mxisd. Docker image, ansible support, debian, nixos and archlinux packages are temporary unavailable due to code auditing and changing maintainers.

🔗Link existing unix accounts with accounts on a Synapse homeserver

Cos created an article detailing how to link existing unix accounts with accounts on a Synapse homeserver.

🔗Dept of Bots 🤖

🔗poll-bot

Brendan:

I made a Matrix bot that uses reactions to do polls! The code is available at https://github.com/babolivier/matrix-poll-bot (with a screenshot of how it works) and you can invite @poll-bot:abolivier.bzh in your room if you want to try it out 🙂

Bot works by taking a list of emoji + responses from a user, then makes a new message event with those emojis each voted for once. In this way, you can quickly make a reaction-based poll.

🔗reactbot, a maubot

tulir, never enough maubots:

The reactbot I announced last week has been updated to support arbitrary response content instead of just reactions. As examples, the repo now has a stallman interjection bot (ported from this and a replacement for jesaribot. Reactions as responses are still supported and the repo still has the TWIM cookie bot example too.

🔗Final thoughts 💭

Ananace has been looking again at a release tracker they'd previously been working on.

kernel.org have set up an ActivityPub instance (see https://people.kernel.org/about.) Not strictly Matrix but interesting that they decided to move to a federated platform.

First GSOC evaluation submissions were due this week, all four Matrix projects are proceeding well. See Eisha's update above, and the other three last week.

red_sky, nheko maintainer, was seen to say: "I know there hasn't been much activity from me lately. I was on vacation last week. I'll be getting back to work on 0.7.0 today"

The Europas awards ceremony was held at a venue next to my old flat. Small world! Small flat too.

🔗That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

🔗Matrix Live 🎙

Matthew and Amandine talk about what we're hoping to ship in June

🔗Dept of Servers 🏢

🔗Dendrite

We have a whole lot of Dendrite news this week! First, from anoa, on the Matrix team:

Progress has been picking up this week with some of Cnly’s many PRs finally getting merged! Cnly is our resident Google Summer of Code student this Summer and has been making strides ever since. We also finally fully migrated to BuildKite for Continuous Integration (which is much quicker than TravisCI) and have implemented golangci-lint as the project’s opinionated linter (catching some bugs in the process).

As part of CI, we’ve set up Sytest for Dendrite, which grants us a loose method of keeping track of progress as we go along. Currently the only way to do so is to take a look at the testfile which holds all the tests we know Dendrite to be passing, and cross-referencing that with the list of all sytests (which isn’t actually available anywhere). We’re hoping to make some nice way of visualizing progress over time with this data (and possibly break it down into Federation/Client/Application Service categories), but this will take some time.

We’d also like to thank some new contributors that have shown up since Matrix 1.0’s release. serra-allgood contributed a fix for querying aliases when using at least one application service, while DrGlitchMX sent a PR in for some database fixes. SUMUKHA-PK’s PR to add room tags is still around and should see some love very shortly.

As far as the current goals of the project, we’re looking to primarily make Dendrite federate with Synapse, which will allow it to actually be usable for testing/basic day-to-day usage. We intend to have it done so in the next few months, so stay tuned!

Next, from Cnly, who is doing GSOC🎓 on Dendrite:

For Dendrite, the first phase of GSoC has primarily seen improvements for the existing code including bug fixes, refinements for testing, and some refactoring that prepares Dendrite for later feature implementation. Among the changes, two important ones will be the initial support for EDUs in /sync responses as well as a federation destination cache that allows for more effective federation.

In the next phase, more focus will be put into feature completion, mainly for the Client/Server API component. Since this part of work covers various still-under-construction areas in Dendrite, it is foreseeable that more progress will actually be made than proposed.

🔗Ruma

This Week in Ruma: https://ruma.dev/news/this-week-in-ruma-2019-06-16/

Work is underway to bring Ruma up to date with version r0.5.0 of the Matrix specification. Starting with the most foundational libraries and working up towards the higher-level ruma-client, this work should be done in the next week or two. The bulk of the work since the last update has been on ruma-events, adding all of the events that were previously missing, and doing a full pass through existing events to make sure our definitions match the specification.

I can't wait to see more updates out of Ruma!

🔗Dept of Bridges 🌉

🔗matrix-appservice-discord 0.5.1

This announcement is nearly a week old from Half-Shot, coming in right after the previous TWIM was published:

We've released v0.5.1 of the discord bridge today. It's mostly comprised of small bugfixes and features, but contains metrics support for tracking performance on the bridge. https://github.com/Half-Shot/matrix-appservice-discord/releases/tag/0.5.1

🔗mautrix-telegram

tulir:

mautrix-telegram got some bugs fixed and minor features:

• Updated to Telethon 1.8
• Option to limit maximum document size when bridging so it wouldn't download 1.5gb files and run out of ram
• Made the state cache get updated when sending state events. Not doing this was causing some problems on t2bot.io, which has disabled echoing state events (probably for performance reasons)

🔗Reliable Bridges GSOC 🎓

Kai is working on bridging in Matrix, mentored by Half-Shot:

The first phase of GSoC is nearing its end and progress has been made on the Reliable Bridges project.
The main focus of the work was on delivering a running implementation for the signaling of permanent errors. Work has been done on matrix-appservice-bridge together with matrix-appservice-discord. An internal version of the Discord bridge with the new feature is already running and the related PR for the bridge SDK is in the state of being completed.
The next phase of GSOC comes with new goals. First, the preparation of a MSC for the signaling of bridge errors, so that the feature can be discussed in the community and can become a part of The Spec. The second goal will be the modification of Riot Web, so that the user is notified when an error condition occurred.
There also has been a slight change of plan: The work on notifying clients when parts of the system are temporarily unavailable needs reconsideration as not even the core Matrix network itself has this capability. The way forward here should probably be a unified solution.

🔗New TypeScript puppeting lib: mx-puppet-bridge

Soru, who you probably know from her work on matrix-appservice-discord, has started a new library:

Soru started working on a new puppeting library, for a lack of a better name it is currently called mx-puppet-bridge.
Unlike the existing puppet bridge, the focus of this one is to handle multiple users at once, all dynamically without needing to edit the configuration file over and over. Additionally the new codebase results in quite a few things smoothened out for easier protocol implementations.
Furthermore mx-puppet-bridge is written in typescript and based on travis' matrix-bot-sdk.
As protocol implementation for testing, soru also wrote mx-puppet-slack (again, for the lack of a better name). Basic message sending in both directions is already fully functional.

Exciting stuff! And all the more so for the third-person announcement style!

🔗Dept of Clients 📱

🔗gomuks

tulir:

Gomuks had a bunch of changes:

• Added support for rendering redactions and edits (rendering reactions coming up next)
• Added commands to create rooms, start private chats and edit room tags
• Rendered images now don't take up the whole width
• Improved memory usage slightly
• Broke everyones existing caches (rm -rf ~/.cache/gomuks should help if it panics at startup)

The command for creating rooms and the image resolution limit were added by a new contributor, J. R., who is currently working on adding .well-known discovery support.

I've been having a good play with this and like it at lot!

🔗continuum, JavaFX client

yuforia:

Improved support for more types of events in continuum this week:

• invitation events are supported
• support for events that typically appear when a room is being set up is in progress
• events that are not yet supported are displayed with a fallback view. Blank rows in previous versions are now fixed. You could always right-click on any event to view the source, of course

🔗Pattle, Dart client

Wilko came in with this thorough update, Pattle development is MOTORING:

A new version of Pattle has been pushed to F-droid!

This release is mostly focused on bug fixing and bug reporting! This is why I urge all users who have been having problems before to try Pattle again! Chances are that your problem has been fixed, and if not, it will be reported with more information so I can fix it!

When an error is reported, this data is sent:

• Operating system version
• Device model, brand, manufacturer and whether it's a simulator
• A unique ID based on your device

In some errors the homeserver domain is logged, I will try to prevent this in the future.

This release also includes preparation for an iOS release next week!

Fixes and other changes:

• Handle rooms that the user has left (a notice is shown that you can't send any messages)
• Show a date header above the chat creation event (not the first known event in the list as before)
• When an error occurs during sync, show a message, including the Exception name
• Fix replies causing an error if the formattedBody does not adhere to the spec (thanks to Mathieu!)
• Fix errors not showing when logging in
• Fix loading spinner showing when checking username or logging in even if loading took less than 3 seconds
• Fix direct chats not detected when adder after the initial sync
• Some general syncing issues have been fixed (causing the dreaded infinite loading spinner).

To install this release, add the following repo in F-droid:

https://fdroid.pattle.im/?fingerprint=E91F63CA6AE04F8E7EA53E52242EAF8779559209B8A342F152F9E7265E3EA729

And install 'Pattle'.

Or download the APK.

If you stumble upon any issues, please report them!
You can login via GitHub and Gitlab.com, so it's really easy to do! (Logging in via GitHub did not seem to work previously, but it will now!)

Follow development here: #app:pattle.im!

If you would like to support me, you can now do so via Liberapay and Patreon. I actually need money for the Apple Developer program, which costs 100 euros per year to release Pattle on the App Store and Testflight.

🔗Dept of Encryption 🔐

🔗Quotient (former QMatrixClient)

aa13q is a GSOC 🎓 student, working on E2EE and mentored by Kitsune:

I'm working on my E2EE GSoC task: https://matrix.org/blog/2019/05/07/welcome-to-the-2019-g-so-c-participants One-time keys uploading part is implementing currently. Draft device keys uploading is also here. The implementation follows E2EE guide: https://matrix.org/docs/guides/end-to-end-encryption-implementation-guide
My contribution and reports are currently here: https://github.com/a-andreyev/libQuotient/wiki/GSoC-2019-reports. Also, feel free to reach me via Quotient group https://matrix.to/#/#quotient:matrix.org.
I'm trying to reuse Black Hat's qtolm project: https://gitlab.com/b0/libqtolm -- added CMake files to include to libQuotient project, special thanks to Black Hat for his work.
Thanks to my GSoC mentors and Matrix and Quotient communities for support and solid groundwork. Long term goal is to provide modern Qt-based library to support full-featured matrix clients like Quaternion, Spectral or even to revive Telepathy IM connection manager https://github.com/TelepathyIM/telepathy-tank! %)

P.S.: Maybe somebody reading this is brave enough to play with Qt-based server implementation too? :)

🔗Riot Web

• Updating reactions UX to match new designs
• Continued refinements to editing *Tweaked SAS verification dialogs to behave more nicely

Also: Riot Desktop was updated on the Snap Store.

🔗Riot iOS

On the aggregation side, we have started to work on the phase:2 sprint (https://vector-im.github.io/feature-dashboard/#/summary?repo=vector-im/riot-web&repo=vector-im/riotX-android&repo=vector-im/riot-ios&label=feature:aggregations&label=phase:2):

• Updating reactions UX to match new designs
• Support edition of event being sent

🔗Riot Android

• Sync with Weblate
• PlayStore assets has been translated to Spanish

🔗RiotX (Android)

• Implementation of notifications is progressing well
• Migration from Koin to Dagger2 is nearly finished
• Many other tasks have been done to try to close ambitious phase 1:
  • encrypt/decrypt attachments in e2e rooms
  • Display redacted events properly in e2e rooms
  • Permalink navigation
  • Display version of app and Matrix sdk in settings
  • Disambiguation of display names
  • Many other bug fixes...

🔗Dept of Ops 🛠

🔗avhost/docker-matrix

Mathijs:

My PR to the avhost/docker-matrix image (formerly the silviof) adding jemalloc support has just been merged. The image will now run synapse using jemalloc by default, which has been shown to reduce memory usage.
The first image is tagged as avhost/docker-matrix:jemalloc if all goes well the next synapse release will include jemalloc by default

🔗Dept of Services 🚀

🔗New (web) forum from Federated Networks Association

jaywink:

Feneas (short for Federated Networks Association) just launched a new discussion forum aimed at creating a collaboration space for the federated web folk. Of course there is a category for Matrix as well. Have a peek! More info here: https://feneas.org/federated-networks-forum/

🔗Diaspora mirror of this blog (meta)

jaywink:

Should you want to follow the Matrix.org blog using your Diaspora protocol compatible account, there is an unofficial mirror of the blog at [email protected]. (disclaimer: I'm not affiliated with the account, just found it and reporting)

🔗Final thoughts 💭

🤔 A company named "Slack" had an IPO, in which the share price closed up 50%. Slack are involved in "Instant Messaging" software and their product is known for the ability to connect to other services.

⏰ It's the first GSOC assessment NEXT WEEK. We'll get updates from all four projects, but notice that three of the projects have already provided updates above.

🤖 tulir created TWO new maubot plugins, one for translating things: https://github.com/maubot/translate. Check out the bot user. The other is a "react bot", https://github.com/maubot/reactbot, "that reacts to messages that match predefined rules".

⁉️ Black Hat (from Spectral and more) is "experimenting with ruma-client-api and plans to create something awesome with it", Alexandre Franke (from Fractal) got a new laptop. Fox "just did [their] last java uni exercise ever and am now free from this curse, will have time to work on Neo".

🔨 This website has had a lot of polish over the last couple of weeks. We'll be focusing on the incremental improvements still needed, and also on adding new documentation. Let us know what you think we should do next.

🔗That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

🔗Matrix Live - Prof. Jon Crowcroft, Foundation Guardian 🎙

Matthew & Amandine talk with Prof. Jon Crowcroft, one of the new Matrix.org Foundation Guardians!

🔗Matrix 1.0

The big news this week is that we declared ourselves to be out of beta. You'll want the full post for all details, but here's a taste:

We are very excited to announce the first fully stable release of the Matrix protocol and specification across all APIs - as well as the Synapse 1.0 reference implementation which implements the full Matrix 1.0 API surface.

This means that after just over 5 years since the initial work on Matrix began, we are proud to have finally exited beta!! This is the conclusion of the work which we announced at FOSDEM 2019 when we cut the first stable release of the Server-Server API and began the Synapse 0.99 release series in anticipation of releasing a 1.0.

As part of the 1.0 we also announced the Matrix.org Foundation.

For the full update on the Foundation, please check out the new website content at https://matrix.org/foundation which should tell you everything you could possibly want to know about the Foundation, the Guardians, the Foundation’s legal Articles of Association, and the day-to-day Rules which define the Open Governance process.

Thanks to everyone who has stuck with us along the way to make this possible.

🔗Synapse

We released Synapse 1.0.0 this week to coincide with Matrix 1.0. As a release Synapse 1.0.0 focuses on security and stability which in turn builds a firm foundation for the performance improvements that you can expect to see over the Summer. Read all about it here.

🔗Crypto

We have not one but two working end to end demos of device cross signing! This is going to be huge and is key step towards being able to make Matrix e2ee by default.

Cross Signing Demo

Additional Treats

But wait there's more! Pantalaimon not only provides a way for clients and bots to participate in e2ee rooms, it now provides the ability to search. Checkout this (religious themed) demo.

🔗Pattle

Pattle is going great guns at the moment. Here's Wilko:-

A new version of Pattle has been pushed to F-droid!

• Add ability to create group chats!
• Show chat creation events ('Wilko has created this group')!
• Show emote messages correctly!
• Handle display name changes! Display names of messages will now be as they were at time of sending.
• Don't show invite and join events in direct chats This is only happens for the two initial users in the direct chat. If someone invites someone else to the direct chat (trough another client), the invitation will show up in the timeline.
• Use the timeout parameter while syncing. This means that receiving new messages should be way quicker! (Thanks Mathieu!)
• Store messages retrieved remotely (thanks Mathieu!) This means that scrolling up in a chat will be faster now, because the messages are cached.
• Always show a date header above the oldest event
• Show replies correctly in chat overview
• Show sent state icon next to own message in chat overview
• Show newly joined rooms at the top in the chat overview
• Use a bit bolder font for chat names in overview To install this release, add the following repo in F-droid:

https://fdroid.pattle.im/?fingerprint=E91F63CA6AE04F8E7EA53E52242EAF8779559209B8A342F152F9E7265E3EA729

And install 'Pattle'.

Or download the APK from the link.

If you stumble upon any issues, please report them! You can now login via GitHub and Gitlab.com , so it's really easy to do!

Follow development here: #app:pattle.im!

There is now also a room for the Matrix Dart SDK (which is being developed for Pattle): #sdk:pattle.im

If you would like to support me, you can now do so via Liberapay and Patreon.

Pattle	Pattle

🔗RiotX (Android)

• Crypto has been merged to develop \o/. We are still working on the feature, for the remaining actions: delete device, export and import keys, keys backup / SAS UI polishing, cleanup keys when signing out, and also fixing bugs…
• A new screen has been added to create Rooms.
• Animation of the Floating Action Button on the catchup screen has been improved.
• Valere has started to work on notifications.
• François is working on migration to Dagger2 (instead of Koin). It should improve performance and will allow us to implement multi-account support(!)
• Copyright, Term and conditions, privacy policy and third party license screens are coming soon.
• Progress indicator on Home for initial/catchup sync is coming soon as well.

🔗Riot Android

• Fix issue on SAS, to allow cross-signing.

🔗Riot iOS

• Work continues on edits and reactions - we'll have something for you very soon ...

🔗Riot Web

• Editing now supports editing unsent messages(!!), editing emotes, and lots of polish
• Reactions now instantly cancel when you redact them
• Redactions now instantly redact when you send them
• Released v1.2.2-rc1 (up for testing at https://riot.im/staging/) - this release includes room breadcrumbs being out of labs, some fixes for inviting by email, fixes uploads in chrome canary / firefox nightly, config file validation, some fixes for the new emoji font and lots of other bug fixes.

🔗Ruma

Jimmy reports that

New versions of the ruma-api and ruma-api-macros libraries were released, and work is underway to bring ruma-events up to date with client-server spec r0.5.0

🔗Media

TravisR has this to say: >Both matrix-media-repo and Dimension have been updated to handle federation certificates. Please update, as you would for Synapse 1.0.0

🔗Fractal

Alexandre Franke has fixed the regression he found last week. The team also did a bit of housekeeping with a few dependency updates.

🔗Misc

Half-Shot reports that:

I wrote a bot to control ansible playbooks https://github.com/Half-Shot/matrix-ansible-bot. I'm informed such a thing already exists as an ansible module, but I failed to realise that and wrote a independent one in TypeScript.

tulir reports that:

I made a simple reminder maubot: https://github.com/maubot/reminder It's available at https://matrix.to/#/@reminder:maunium.net

For room admins wondering if they can upgrade their rooms to v5, I made a bash script that checks all the servers in a room and prints a nice summary of the number of servers and members on each version: https://gist.github.com/tulir/aa2df287a0d192b86e5b675687791d16

Atreatis reports that:

Currently grin is building a backend for matrixservers.net that will collect data and push statistics every hour to our website. Next to that I wanted to push forward that me and several others are looking into building a Code of Conduct that should help every new and existing home server. This code of conduct will be build and curated from scratch in-order to help and support the network or any general project.

Pneumaticat has this to say:- >My homeserver (potatofrom.space) is open for some public registration! Also on Atreatis's matrixservers.net.

They promise 'One nine of uptime guaranteed, or your money back' which sounds pretty like a bargain.

So that's it folks, until next week. Come and say hi at #twim:matrix.org

🔗Matrix 1.0

Hi all,

We are very excited to announce the first fully stable release of the Matrix protocol and specification across all APIs - as well as the Synapse 1.0 reference implementation which implements the full Matrix 1.0 API surface.

This means that after just over 5 years since the initial work on Matrix began, we are proud to have finally exited beta!! This is the conclusion of the work which we announced at FOSDEM 2019 when we cut the first stable release of the Server-Server API and began the Synapse 0.99 release series in anticipation of releasing a 1.0.

Now, before you get too excited, it’s critical to understand that Matrix 1.0 is all about providing a stable, self-consistent, self-contained and secure version of the standard which anyone should be able to use to independently implement production-grade Matrix clients, servers, bots and bridges etc. It does not mean that all planned or possible features in Matrix are now specified and implemented, but that the most important core of the protocol is a well-defined stable platform for everyone to build on.

On the Synapse side, our focus has been exclusively on ensuring that Synapse correctly implements Matrix 1.0, to provide a stable and secure basis for participating in Matrix without risk of room corruption or other nastinesses. However, we have deliberately not focused on performance or features in the 1.0 release - so I’m afraid that synapse’s RAM footprint will not have got significantly better, and your favourite long-awaited features (automatically defragmenting rooms with lots of forward extremities, configurable message retention, admin management web-interface etc) have not yet landed. In other words, this is the opposite of the Riot 1.0 release (where the entire app was redesigned and radically improved its performance and UX) - instead, we have adopted the mantra to make it work, make it work right, and then (finally) make it fast. You can read the full release notes here. It’s also worth looking at the full changelog through the Synapse 0.99 release series to see the massive amount of polishing that’s been going on here.

All this means that the main headline features which land in Matrix 1.0 are vitally important but relatively dry:

• Using X.509 certificates to trust servers rather than perspective notaries, to simplify and improve server-side trust. This is a breaking change across Matrix, and we’ve given the community several months now to ensure their homeservers run a valid TLS certificate. See MSC1711 for full details, and the 2 week warning we gave. As of ~9am UTC today, the matrix.org homeserver is running Synapse 1.0 and enforcing valid TLS certificates - the transition has begun (and so far we haven’t spotted any major breakage :). Thank you to everyone who got ready in advance!
• Using .well-known URIs to discover servers, in case you can’t get a valid TLS certificate for your server’s domain.
• Switching to room version 4 by default for creating new rooms. This fixes the most important defects that the core room algorithm has historically encountered, particularly:
  • The new State Resolution algorithm to fix the Hotel California bug and many others: State Resolution Reloaded
  • Collision resistant event IDs
• Specifying the ability to upgrade between room versions
• Full specification of lazy loading room members
• Short Authentication String (Emoji!) interactive verification of E2EE devices
• ...and lots and lots and lots of bugfixes and spec omission fixes.

That said, there is a lot of really exciting stuff in flight right now which sadly didn’t stabilise in time for Matrix 1.0, but will be landing as fast as we can finalise it now that 1.0 is at last out the door. This includes:

• Editable messages! (These are in Synapse 1.0 and Riot already, but still stabilising so not enabled by default)
• Reactions! (Similarly these are in develop)
• Threading!! (We’ve planted the seeds for this in the new ‘aggregations’ support which powers edits & reactions - but full thread support is still a bit further out).
• Cross-signed verification for end-to-end encryption (This is on a branch, but due to land any day now). We’ve also held off merging E2E backups into the Matrix 1.0 spec until cross-signing lands, given it may change the backup behaviour a bit. Once this is done, we can seriously talk about turning on E2E by default everywhere.
• Live-tracking of room statistics and state in Synapse! (This is in Synapse 1.0 already if you check out the new room_stats and room_state tables, but we need to provide a nice admin interface for it).
• Support for smaller footprint homeservers by reducing memory usage and stopping them from joining overly complex rooms.

Then stuff which we haven’t yet started, but is now unlocked by the 1.0 release:

• Fixing extremities build-up (and so massively improving performance)
• Rewriting Communities. Groups/Communities deliberately didn’t land in Matrix 1.0 as the current implementation has issues we want to fix first. MSC1772 has the details.
• Rewritten room directory using the new room stats/state tables to be super-speedy.
• Super-speedy incremental state resolution
• Removing MXIDs from events (MSC1228)

Just to give a quick taster of the shape of things to come, here’s RiotX/Android, the all-new Riot client for Android, showing off Edits & Reactions in the wild…

...and here’s a screenshot of the final test jig for cross-signing devices in end-to-end encryption, so you will never have to manually verify new devices for a trusted user ever again! We demoed a very early version of this at FOSDEM, but this here is the testing harness for real deal, after several iterations of the spec and implementation to nail down the model. + means the device/user's cross-signing key is trusted, T means it's TOFU:

So, there you have it - welcome to Matrix 1.0, and we look forward to our backlog of feature work now landing!

Massive massive thanks to everyone who has stuck with the project over the years and helped support and grow Matrix - little did we think back in May 2014 that it’d take us this long to exit beta, but hopefully you’ll agree that it’s been worth it :)

Talking of which, we were looking through the photos we took from the first ever session hacking on Matrix back in May 2014…

...suffice it to say that of the architectural options, we went with #3 in the end...

...and that nowadays we actually know how power levels work, in excruciating and (hopefully) well-specified detail :)

There has been an absolutely enormous amount of work to pull Matrix 1.0 together - both on the spec side (thanks to the Spec Core Team for corralling proposals, and everyone who's contributed proposals, and particularly to Travis for editing it all) and the implementation side (thanks to the whole Synapse team for the tedious task of cleaning up everything that was needed for 1.0). And of course, huge thanks go to everyone who has been helping test and debug the Synapse 1.0 release candidates, or just supporting the project to get to this point :)

🔗The Matrix.org Foundation

Finally, as promised, alongside Matrix 1.0, we are very happy to announce the official launch of the finalised Matrix.org Foundation!

This has been a long-running project to ensure that Matrix’s future is governed by a neutral non-profit custodian for the benefit of everyone in the Matrix ecosystem. We started the process nearly a year ago back with the initial proposal Towards Open Governance of Matrix.org, and then legally incorporated the Foundation in October, and published the final governance proposal in January.

As of today the Foundation is finalised and operational, and all the assets for Matrix.org have been transferred from New Vector (the startup we formed in 2017 to hire the core Matrix team). In fact you may already have seen Matrix.org Foundation notices popping up all over the Matrix codebase (as all of New Vector’s work on the public Matrix codebase for the foreseeable is being assigned to the Matrix.org Foundation).

Most importantly, we’re excited to introduce the Guardians of the Matrix.org Foundation. The Guardians are the legal directors of the non-profit Foundation, and are responsible for ensuring that the Foundation (and by extension the Spec Core Team) keeps on mission and neutrally protects the development of Matrix. Guardians are typically independent of the commercial Matrix ecosystem and may even not be members of today’s Matrix community, but are deeply aligned with the mission of the project. Guardians are selected to be respected and trusted by the wider community to uphold the guiding principles of the Foundation and keep the other Guardians honest.

We have started the Foundation with five Guardians - two being the original founders of the Matrix project (Matthew and Amandine) and three being entirely independent, thus ensuring the original Matrix team forms a minority which can be kept in check by the rest of the Guardians. The new Guardians are:

• Prof. Jon Crowcroft - Marconi Professor of Communications Systems in the Computer Lab at the University of Cambridge and the Turing Institute. Jon is a pioneer in the field of decentralised communication, and a fellow of the Royal Society, the ACM, the British Computer Society, the Institution of Engineering and Technology, the Royal Academy of Engineering and the Institute of Electrical and Electronics Engineers.

  Jon is a global expert in decentralisation and data privacy, and is excellently placed to help ensure Matrix stays true to its ideals.

• Ross Schulman - Ross is a senior counsel and senior policy technologist at New America’s Open Technology Institute, where he focuses on internet measurement, emerging technologies, surveillance, and decentralization. Prior to joining OTI, Ross worked for Google.

  Ross brings a unique perspective as a tech- and decentralisation-savvy lawyer to the Foundation, as well as being one of the first non-developers in the Matrix community to run his own homeserver. Ross has been known to walk around Mozfest clutching a battery-powered Synapse in a box, promoting decentralised communication for all.

• Dr. Jutta Steiner - As co-founder and CEO of Parity Technologies, Jutta is dedicated to building a better internet - Web 3.0 - where users’ privacy & control come first. Parity Technologies is a leader in the blockchain space – known to many as the creator of one of the most popular Ethereum clients, it is also the creator of two ambitious new blockchain technlogies, Polkadot and Substrate, that make it easier to experiment and innovate on scalability, encryption and governance.

  Parity has been pioneering Matrix enterprise use since the moment they decided to rely on Matrix for their internal and external communication back in 2016, and now run their own high-volume deployment, with end-to-end encryption enabled by default. Jutta represents organisations who are professionally dependent on Matrix day-to-day, as well as bringing her unique experiences around decentralisation and ensuring that Web 3.0 will be a fair web for all.

We’d like to offer a very warm welcome to the new Guardians, and thank them profusely for giving up their time to join the Foundation and help ensure Matrix stays on course for the years to come.

For the full update on the Foundation, please check out the new website content at https://matrix.org/foundation which should tell you everything you could possibly want to know about the Foundation, the Guardians, the Foundation’s legal Articles of Association, and the day-to-day Rules which define the Open Governance process.

🔗And finally…

Matrix 1.0 has been a bit of an epic to release, but puts us on a much stronger footing for the future.

However, it’s very unlikely that we’d have made it this far if most of the core dev team wasn’t able to work on Matrix as their day job. Right now we are actively looking for large-scale donations to the Matrix.org Foundation (and/or investment in New Vector) to ensure that the team can maintain as tight a focus on core Matrix work as possible, and to ensure the project realises its full potential. While Matrix is growing faster than ever, this perversely means we have more and more distractions - whether that’s keeping the Matrix.org server safe and operational, or handling support requests from the community, or helping new members of the ecosystem get up and running. If you would like Matrix to succeed, please get in touch if you’d like to sponsor work, prioritise features, get support contracts, or otherwise support the project. We’re particularly interested in sponsorship around decentralised reputation work (e.g. publishing a global room directory which users can filter based on their preferences).

Finally, huge thanks to everyone who has continued to support us through thick and thin on Patreon, Liberapay or other platforms. Every little helps here, both in terms of practically keeping the lights on, and also inspiring larger donations & financial support.

So: thank you once again for flying Matrix. We hope you enjoy 1.0, and we look forward to everything else landing on the horizon!

- Matthew, Amandine & the whole Matrix.org Team.

Well here it is: Synapse 1.0.

Synapse 1.0 is the reference implementation of the Matrix 1.0 spec. The goal of the release overall has been to focus on security and stability, such that we can officially declare Synapse (and Matrix) out of beta and recommended for production use. This means changing the default room protocol version used for new rooms to be v4, which includes the new state resolution algorithm, as well as collision-resistant event IDs, which are now formatted to be URL safe.

Synapse 1.0 also ships with support for the upcoming v5 room protocol (which enforces honouring server key validity periods), but this will not be used as the default for new rooms until a sufficient number of servers support it.

Please note that Synapse 1.0 does not include significant performance work or new features - our focus has been almost exclusively on providing a reference implementation of the Matrix 1.0 protocol. But having cleared our backlog on security/stability issues we will finally be now unblocked to pursue work around reducing RAM footprint, eliminating forward-extremity build up, and shipping new features like Edits, Reactions & E2E cross-signing support.

As part of the security work, Synapse 1.0 contains a breaking change that requires a valid TLS certificate on the federation API endpoint. Servers that do not configure their certificate will no longer be able to federate post 1.0.

It is also worth noting that Synapse 1.0.0 is the last release that will support Python 2.x and Postgres 9.4. For more information see here but the TL;DR is that you should upgrade asap.

This release has been a long time coming. Many thanks indeed to everyone who helped test the release candidates and provided feedback along the way.

Synapse 1.0 is just one component of a larger Matrix 1.0 release, which you can read all about here.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our Synapse installation guide page

The changelog since 0.99.5 follows:

🔗Synapse 1.0.0 (2019-06-11)

🔗Bugfixes

• Fix bug where attempting to send transactions with large number of EDUs can fail. (#5418)

🔗Improved Documentation

• Expand the federation guide to include relevant content from the MSC1711 FAQ (#5419)

🔗Internal Changes

• Move password reset links to /_matrix/client/unstable namespace. (#5424)

🔗Synapse 1.0.0rc3 (2019-06-10)

Security: Fix authentication bug introduced in 1.0.0rc1. Please upgrade to rc3 immediately

🔗Synapse 1.0.0rc2 (2019-06-10)

🔗Bugfixes

• Remove redundant warning about key server response validation. (#5392)
• Fix bug where old keys stored in the database with a null valid until timestamp caused all verification requests for that key to fail. (#5415)
• Fix excessive memory using with default federation_verify_certificates: true configuration. (#5417)

🔗Synapse 1.0.0rc1 (2019-06-07)

🔗Features

• Synapse now more efficiently collates room statistics. (#4338, #5260, #5324)
• Add experimental support for relations (aka reactions and edits). (#5220)
• Ability to configure default room version. (#5223, #5249)
• Allow configuring a range for the account validity startup job. (#5276)
• CAS login will now hit the r0 API, not the deprecated v1 one. (#5286)
• Validate federation server TLS certificates by default (implements MSC1711). (#5359)
• Update /_matrix/client/versions to reference support for r0.5.0. (#5360)
• Add a script to generate new signing-key files. (#5361)
• Update upgrade and installation guides ahead of 1.0. (#5371)
• Replace the perspectives configuration section with trusted_key_servers, and make validating the signatures on responses optional (since TLS will do this job for us). (#5374)
• Add ability to perform password reset via email without trusting the identity server. (#5377)
• Set default room version to v4. (#5379)

🔗Bugfixes

• Fixes client-server API not sending "m.heroes" to lazy-load /sync requests when a rooms name or its canonical alias are empty. Thanks to @dnaf for this work! (#5089)
• Prevent federation device list updates breaking when processing multiple updates at once. (#5156)
• Fix worker registration bug caused by ClientReaderSlavedStore being unable to see get_profileinfo. (#5200)
• Fix race when backfilling in rooms with worker mode. (#5221)
• Fix appservice timestamp massaging. (#5233)
• Ensure that server_keys fetched via a notary server are correctly signed. (#5251)
• Show the correct error when logging out and access token is missing. (#5256)
• Fix error code when there is an invalid parameter on /_matrix/client/r0/publicRooms (#5257)
• Fix error when downloading thumbnail with missing width/height parameter. (#5258)
• Fix schema update for account validity. (#5268)
• Fix bug where we leaked extremities when we soft failed events, leading to performance degradation. (#5274, #5278, #5291)
• Fix "db txn 'update_presence' from sentinel context" log messages. (#5275)
• Fix dropped logcontexts during high outbound traffic. (#5277)
• Fix a bug where it is not possible to get events in the federation format with the request GET /_matrix/client/r0/rooms/{roomId}/messages. (#5293)
• Fix performance problems with the rooms stats background update. (#5294)
• Fix noisy 'no key for server' logs. (#5300)
• Fix bug where a notary server would sometimes forget old keys. (#5307)
• Prevent users from setting huge displaynames and avatar URLs. (#5309)
• Fix handling of failures when processing incoming events where calling /event_auth on remote server fails. (#5317)
• Ensure that we have an up-to-date copy of the signing key when validating incoming federation requests. (#5321)
• Fix various problems which made the signing-key notary server time out for some requests. (#5333)
• Fix bug which would make certain operations (such as room joins) block for 20 minutes while attemoting to fetch verification keys. (#5334)
• Fix a bug where we could rapidly mark a server as unreachable even though it was only down for a few minutes. (#5335, #5340)
• Fix a bug where account validity renewal emails could only be sent when email notifs were enabled. (#5341)
• Fix failure when fetching batches of events during backfill, etc. (#5342)
• Add a new room version where the timestamps on events are checked against the validity periods on signing keys. (#5348, #5354)
• Fix room stats and presence background updates to correctly handle missing events. (#5352)
• Include left members in room summaries' heroes. (#5355)
• Fix federation_custom_ca_list configuration option. (#5362)
• Fix missing logcontext warnings on shutdown. (#5369)

🔗Improved Documentation

• Fix docs on resetting the user directory. (#5282)
• Fix notes about ACME in the MSC1711 faq. (#5357)

🔗Internal Changes

• Synapse will now serve the experimental "room complexity" API endpoint. (#5216)
• The base classes for the v1 and v2_alpha REST APIs have been unified. (#5226, #5328)
• Simplifications and comments in do_auth. (#5227)
• Remove urllib3 pin as requests 2.22.0 has been released supporting urllib3 1.25.2. (#5230)
• Preparatory work for key-validity features. (#5232, #5234, #5235, #5236, #5237, #5244, #5250, #5296, #5299, #5343, #5347, #5356)
• Specify the type of reCAPTCHA key to use. (#5283)
• Improve sample config for monthly active user blocking. (#5284)
• Remove spurious debug from MatrixFederationHttpClient.get_json. (#5287)
• Improve logging for logcontext leaks. (#5288)
• Clarify that the admin change password API logs the user out. (#5303)
• New installs will now use the v54 full schema, rather than the full schema v14 and applying incremental updates to v54. (#5320)
• Improve docstrings on MatrixFederationClient. (#5332)
• Clean up FederationClient.get_events for clarity. (#5344)
• Various improvements to debug logging. (#5353)
• Don't run CI build checks until sample config check has passed. (#5370)
• Automatically retry buildkite builds (max twice) when an agent is lost. (#5380)

🔗Dept of Spec 📜

We are sooo close to Matrix 1.0. So close! The PR which announces the arrival of v1.0 exists already.

TravisR said:

Work towards 1.0 is progressing nicely - still on track for a Monday release! View our progress at https://github.com/matrix-org/matrix-doc/projects/2?fullscreen=true

🔗Dept of Servers 🏢

🔗Synapse

Neil, who oversees the creation of Synapse:

Quiet week, not much happening, the only real progress has been shipping a Synapse 1.0 release candidate :) Joking aside we’d really appreciate your help in testing it ahead of the full release next week. Get the RC here https://github.com/matrix-org/synapse/releases/tag/v1.0.0rc1

Also friends don’t let friends operate a Synapse install without a valid certificate on the federation API. Post 1.0, servers that are not compliant will be locked out of the federation. While the copy needs updating to reflect 1.0’s imminent release, our handy FAQ (https://github.com/matrix-org/synapse/blob/master/docs/MSC1711_certificates_FAQ.md) has all the details.

🔗Construct

Construct, a C++ homeserver has been doing testing this week:

I'd like to give special thanks to Ddanyspin97 (Danilo Spinella) for working on building construct with the musl C library so we can enjoy extremely small VM images, as well as Black Hat for building on Alpine linux also using muslc and Thomas Lewis for starting work on the FreeBSD build.

🔗Ruma

Ruma, a homeserver written in Rust, has a new surge of activity following the Rust language syntax developments last week. This Week in Ruma received it's first edition in ~6 months!

Jimmy, project lead:

the ruma homeserver is now targeting stable rust, which means the entire project is now on stable rust

🔗New project: load balancer for synchrotron workers

Soru:

[I] wrote a proof-of-concept load balancer for synchrotron workers....well, proof-of-concept because it isn't tested much, yet!

It routes users to synchrotrons based on their user ID to maximize performance, as per worker docs and, if a synchrotron catches fire it'll also automatically start shuffling workers around.
The repo can be found here, for support just poke soru wherever she is

Contributions are welcome!

Join the newly created room at #matrix-synchrotron-balancer:sorunome.de to discuss.

🔗Dept of SDKs and Frameworks 🧰

🔗Ruby SDK

Ananace:

Just released version 1.1.0 of the Ruby SDK, which includes a few more and improved CS API endpoints, better room alias handling, lazy loading of join rules and guest access, and the forgotten - oops - handling of room avatars.

And released version 1.1.1 of the Ruby SDK too, fixing another embarrassing mistake with not including the S2S API methods correctly, which lost the server version retrieval method (The only currently implemented part of the S2S API)

🔗Dept of Clients 📱

🔗RiotX

Are you in the know about Matrix? If not, prepare to learn about RiotX, a completely re-written Matrix client for Android. This new version has a Kotlin SDK.

Benoit, from the team:

Many things to announce this week on RiotX:

• Crypto is starting to work! We can now decrypt message and send encrypted messages. The whole feature is not implemented yet, but we will merge the PR at the beginning next week before going further. Thanks François for the great work!
• On the timeline, message composed by only emoji are now displayed bigger (as in other Riot clients)
• We can now see details and actions of each message, including state event.
• We can also see the detailed list of reactions for a particular message, and reaction display rendering has been improved. More details of what has been improved by Valere can be found in this PR description: https://github.com/vector-im/riotX-android/pull/168.
• Local echo on message edition has been improved
• The 3 main themes has been implemented (Light, Dark Black). Status theme is still there but has not been tested yet. There are still some little issues which will be fixed in the next coming days.
• The new Home is on develop, composed by a catchup tab, a direct message tab and a group room tab. The drawer is now used to display the groups hierarchy.
• A debug signature has been added to the git repository, so the APK built by buildkite can now be installed and replace a previously installed APK.

There is still lots of work to do, but we are quite confident to deliver a first alpha version of RiotX to a larger audience at the end of June.

🔗famedly talk

We've heard about Famedly before, and here they are with an announcement about their new client:

After some months of work our app 'famedly talk' is now able to send and receive messages. We are now concentrating on implementing the most important functions in a stable manner.

🔗continuum

yuforia:

fix some missing avatars and names of rooms. Normally continuum saves all useful data to disk when syncing, but the previous version missed some of it, the newer version will query the server for the missing pieces

🔗Fractal

This week in Fractal: Alexandre Franke noticed nightly was broken on all three of the computers he was using it on. He bisected and found out when the regression was introduced, but still needs to figure out why it broke and how to fix it.

The symptom is syncing forever on startup and never actually syncing. The commit introducing the borkage is this one.

🔗Riot Web

• Working on polish for reactions and edits
• Improving configuration of the default homeserver
• Added Albanian and West Flemish languages thanks to translation contributors

🔗Riot iOS

• Working on reaction local echo.
• Working on message edition.
• Restrict reactions to room messages.

🔗Riot Android

Nothing special this week.

DID YOU SEE THE RIOTX UPDATE ABOVE?

🔗Dept of Bridges 🌉

🔗matrix-appservice-irc 0.12.0 released

Half-Shot:

matrix-appservice-irc got a release 0.12.0. This is a smaller release, as it's based upon 0.12.0-rc2 which was released back in March (when exams hit). In the meantime, working towards a speedy 0.13.0 release :)

Changelog for 0.12.0:

The bridge now supports upgrading rooms, and will follow room upgrades to the new room.
Added support for the RoomLinkValidator, which allows admins to manually configure rules about plumbing rooms.
A dockerfile is now included.
Add support for "feature flags", allowing users to dynamically enable/disable bridge features for their account.
Add command "!bridgeversion"

High-level plan for 0.13.0:

Fix the most serious P1 bugs like not dropping messages, handling PARTs better, dropping support for older versions of node, and generally better UX.

🔗Dept of Services 🚀

🔗matrixservers.net

Atreatis provided an update to matrixservers.net:

I've open sourced the website of https://matrixservers.net and I am open to contributions towards the website through the github repository: https://github.com/Atreatis/matrixservers
Currently I have refactored matrixservers.net from Hugo to Jekyll which was totally worth it at all times

Join #matrixservers:novaim.com for more.

🔗www.matrix-wiki.org

sanlox:

I launched https://www.matrix-wiki.org as a place to collect information, lists, guides and manuals related to matrix. I hope as a community we can work together on this to create a central knowledge base for the matrix protocol.

🔗Final thoughts 💭

I spent so much of my week migrating matrix.org to a great new design (from Nad), and am really happy with the result. I'm also really happy with Gatsby, the SSG we've chosen. All these things have flaws but overall it works well.

🔗That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

🔗Matrix Live - Half-Shot and Sorunome discuss the Discord-Matrix Bridge 🎙

🔗Dept of Spec 📜

TravisR:

Alongside Synapse’s 1.0 release, the Matrix Specification is also getting a 1.0 on June 10th. In practice this means a fresh release of the Client-Server, Server-Server, Application Service, and Identity Service specifications. The Push Specification is just so simple it hasn’t needed an update yet.

Work has been happening to get all the specs in shape for 1.0 - follow along at https://github.com/matrix-org/matrix-doc/projects/2?fullscreen=true

🔗Dept of Servers 🏢

🔗Synapse

Main news in synapse-land this week is that we released Synapse v0.99.5.2, which fixes a big performance regression which was introduced v0.99.5. Folks who are currently on v0.99.5/v0.99.5.1 are advised to upgrade asap.

Otherwise, work has been continuing apace towards a v1.0 release on 10th June.

🔗Construct

Construct is a homeserver implementation in C++:

• Construct has added support for Server Access Control Lists (m.room.server_acl). The exact behavior of the implementation is configurable by the server administrator, and is even more comprehensive than Synapse. In Construct, ACLs can be configured to cover incoming EDUs, outgoing events, and indirectly fetched events.
• The GraphicsMagick library has been utilized to implement the media thumbnailing feature called for by the specification. This is an optional dependency written in C and offers a rich API which as been extremely fun to work with. In the future, expect your Construct to be capable of things like annotating images with the Server Command Line which can be used to generate memes in your chatroom or write custom text on top of stickers!
• On the backend we have devoted significant effort to getting the most out of dynamic-linking to simplify the architecture of a large and modular free-software project. This effort makes adding features and contributions to Construct extremely straightforward. I will enumerate the basics for those familiar with developing C software: We are now loading all modules at runtime with the flags RTLD_GLOBAL|RTLD_DEEPBIND|RTLD_LAZY and we build with the flag --weak-unresolved-symbols; during execution we hook the error handling in ld.so to throw our own C++ exception directly from the PLT slot of a missing symbol. In summary this allows features to be added to Construct by simply creating an interface in a header and definitions in a loadable module, while still enjoying optional loading and unloading of the module at runtime. This is a significant feat that remains unrealized in most existing projects around the free software ecosystem!

🔗Dept of SDKs and Frameworks 🧰

🔗Dept of Clients 📱

🔗nheko 0.6.4 and 0.7.0 roadmap

red_sky updated us on nheko:

Just wanted to pop in and mention that 0.6.4 is out this week. It comes with a decent number of bug fixes.

And then:

Nheko 0.7.0 roadmap is out: https://github.com/Nheko-Reborn/nheko/milestone/2
Also, Nheko-Reborn has a new team member, with MonokelPinguin joining the team : https://github.com/MonokelPinguin. (goes by Nico in the #nheko-reborn room)

el0 pointed out that nheko is available on flathub: https://flathub.org/apps/details/io.github.NhekoReborn.Nheko

🔗continuum, JavaFX client

yuforia:

continuum modified the JSON parser to save a copy of the source. Now fields that are not standard or not yet supported are displayed when you right-click to view the source. For example, "age" and "transaction_id" are not used by the app, but are available when viewing the raw JSON

🔗Neo

f0x has been working on Neo:

Neo can now finally fetch old backlog, with a fancy css loading animation

The Neo room has moved to #neo:pixie.town, and the site+deployment are being moved to https://neo.pixie.town

🔗Fractal 🦶

Alexandre Franke:

Fractal got initial support for being kicked, implemented by first time contributor Isaac Curtis.

We also joined the new call at https://stopthemingmy.app

🔗Riot Web

Continued work on edits and reactions. Also ongoing work on sdk-level support for cross-signing. Released Riot 1.2.0 (and 1.2.1).

🔗Dept of Bridges 🌉

🔗mautrix-telegram, mautrix-facebook, mautrix-hangouts, mautrix-whatsapp

Can you believe it, tulir had a busy week working on his various bridges:

mautrix-telegram now has native Matrix edit support in both directions and basic Prometheus metrics. I'll probably start making 0.6.0 RCs after improving the metrics a bit.

mautrix-facebook and mautrix-hangouts got some bugfixes and options to configure the displaynames of the ghost users on Matrix (like adding (FB) or (Hangouts) suffixes).

mautrix-whatsapp's Matrix puppeting was finished (meaning typing notifs, read receipts and presence updates can now be bridged to WhatsApp) and unexpected disconnection handling was fixed (automatic reconnections should now usually work again). Backfilling needed some hacks to work with Matrix puppeting. The problem is that the real Matrix users have rate limits and can't send timestamp-massaged messages. I decided to just invite the default ghost user to the room while backfilling and make it leave afterwards so that private chat room names and such would stay intact. Some clients like Riot Android/iOS sometimes do weird things, such as moving the room out of "People", but that shouldn't be too bad.

🔗Dept of Ops 🛠

🔗matrix-docker-ansible-deploy makes Synapse optional

Slavi:

matrix-docker-ansible-deploy has seen some work in becoming more flexible.

Synapse is not a requirement anymore. People can now use the playbook to install any of the other services, wiring them to a Synapse instance installed in another way. This is mostly useful for installing bridges or other auxiliary services without going "all in" on the playbook. Not having Synapse as a fixed requirement also opens the door for installing alternative homeservers, when such become available in the future.

On a similar note, to ease maintainability, all bridges have been moved to separate Ansible roles (outside of the matrix-synapse role). There have been lots of improvements related to bridge configuration with even more likely to come soon.

🔗0.99.5.2 kubernetes-optimized images

Ananace had 0.99.5.2 kubernetes-optimized images ready to go very quickly.

🔗Dept of Services 🚀

🔗Matrix User Documentation launches 🎉

Aaron (formerly known as Aaron Raimist) has been a major help to the Matrix project, and has recently launched something new:

I have started another Matrix documentation project, Matrix User Documentation. I know, there are several however none of them aim to actually be a handbook for using Riot or any other clients. I would like for this to become a place where you can find "The (unofficial) Riot Handbook" or "The (unofficial) Quaternion Handbook" as well as a place where you can find answers to more of your FAQs.

It's still very early, only a few answers are written so far and none of the handbooks have been started but I would welcome any community assistance. You can view what I have so far at: http://mud.raim.ist/en/latest/

You can contribute some documentation at: https://github.com/aaronraimist/matrix-user-documentation and join our room: #matrix-user-documentation:raim.ist

🔗matrixservers.net

We welcomed Atreatis to Matrix recently. They are creating a new directory service:

Someone mentioned earlier that there is a need for a list of public servers on Github so people can find and add their public matrix servers. Which combines well with what I am building at the moment. Hopefully you guys don't mind helping me out on creating the list and spreading the word. The repository can be found here: https://github.com/Atreatis/matrix-servers

🔗Final thoughts 💭

Matrix Shell Suite is a promising looking project which could deserve more attention.

Thank You to PyConWeb and especially to Hardy for letting richvdh and myself host TWO Matrix sessions at your event last weekend - was a great time and I hope everyone enjoyed the presentations and workshops.

🔗That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!