Synapse 1.49.0 released

14.12.2021 00:00 — Releases Brendan Abolivier

Synapse 1.49.0 is out now!

Platform deprecations

Synapse 1.49.0 is the last version of Synapse to officially support Python 3.6 and PostgreSQL 9.6. This follows our platform dependency deprecation policy.

As a consequence of this, Synapse 1.49.0 is the last version of Synapse to support Ubuntu 18.04 LTS (Bionic Beaver), as it ships with Python 3.6.

On the topic of supported Ubuntu releases, please note that Ubuntu 21.04 (Hirsute Hippo) reaches its own end of life on January 20, 2022. Past this date we will stop producing new packages for Ubuntu 21.04.

Improved documentation

Up until now, a lot of very useful information was stored on the Synapse repo's wiki, which wasn't well advertised nor well reviewed.

With this release, we have migrated most of this information to Synapse's documentation website, so all the information you need to set up, maintain and troubleshoot a Synapse instance lives at the same place. Included in these new pages are the server admin FAQ and a guide to Synapse's Grafana dashboard.

The media repository documentation has also been updated with a lot of details about how Synapse stores media files.

Refresh tokens

When a Matrix client needs to authenticate a request to a homeserver, it uses what is called an access token. Sometimes server administrators might not want a user's access token to live forever (e.g. for security reasons). To address this concern, MSC2918 introduces the concept of refresh tokens to Matrix.

Initial support for refresh tokens in Synapse was introduced in version 1.38.0. Synapse 1.49.0 finalises and stabilises this implementation, allowing any client that supports this feature to use it as it is currently described in the related MSC.

Everything else

This release introduces the last changes needed to Synapse for basic threading support. It also introduces support for MSC3030, which allows clients to jump to a specific date in a room's history (expect a sneak peek of this in the next episode of Matrix Live!).

Another interesting point is the addition of a couple of admin APIs for federation. More specifically, they allow you to visualise all of the other homeservers your Synapse instance has been interacting with, as well as how successful the last attempts at communicating with them have been.

Please see the Synapse release notes for a complete list of changes in this release.

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including Dirk Klimpel, Maximilian Bosch and Tulir Asokan.

Till next year

This is the last release of Synapse of 2021! The Synapse team will take a break for the holidays, pushing the next release of Synapse (1.50.0) to January 11, 2022.

We'd like to thank everyone who has been using Synapse, contributing to it, and/or supporting us for the past year, and we hope to see you again in 2022! 🎆

Disclosure: buffer overflow in libolm and matrix-js-sdk

13.12.2021 18:35 — Security Matrix Security Team
Last update: 13.12.2021 16:11

Today we are releasing security updates to libolm, matrix-js-sdk, and several clients including Element Web / Desktop. Users are encouraged to upgrade as soon as possible. This resolves the pre-disclosure issued on December 3rd.

Fixed library versions are:

Client versions incorporating the fixes are:

These releases mitigate a buffer overflow in olm_session_describe, a libolm debugging function used by matrix-js-sdk in its end-to-end encryption (E2EE) implementation. If you rely on matrix-js-sdk for E2EE, you are affected. This vulnerability has been assigned CVE-2021-44538.

Clients which do not use matrix-js-sdk for E2EE, like FluffyChat or Element Android / iOS, are not affected.

This issue has been present since the introduction of the olm_session_describe function in October 2019 (commits: libolm, matrix-js-sdk).

We do not believe it is practical to successfully exploit this issue. However, upgrading remains important as the overflow can be triggered remotely.

Separately from the above vulnerability, we noticed during an internal audit that the libolm bindings in matrix-js-sdk were not zeroing out certain arrays containing entropy for cryptographic operations. This causes the entropy to remain resident in memory longer than necessary. As a defense-in-depth measure, this release of libolm now proactively overwrites those arrays when it is safe to do so.

Lastly, we are also taking this opportunity to update the version of Electron bundled with Element Desktop, pulling in the latest backported security fixes there.

The buffer overflow was found and reported by GitHub user @brevilo in the course of developing jOlm, a library of Java bindings to libolm; thank you. If you believe you've discovered a security vulnerability in Matrix or its implementations, please see our Security Disclosure Policy for how to get in touch.

This Week in Matrix 2021-12-10

10.12.2021 19:24 — This Week in Matrix Thib
Last update: 10.12.2021 19:09

The Adventures of TWIM bot

Last week the Matrix Scientists had turned this room into a portal to TWIM bot's ship tank so we could all fuel it with news about our work. The spec and hookshot news were reported late when the tank was already full, sending TWIM bot's ship into hyperspeed mode 🚀

We lost contact with the bot for a few days but managed to restore a connection with it. It appeared to have crashed on the green planet of Fuj'ehr. Its ship shattered in pieces at impact! 💥

The bot managed to find most of them, but some critical pieces of the engine were missing. We needed to find those pieces quickly before the mushy ground of Fuj'ehr swallows them forever! 😱

The Editor has asked the Matrix Scientists to reconfigure the room and bridge it to TWIM bot's navigation tools. Each news report highlighted the position of a piece of engine on its map. 🗺️

Matrix Live 🎙

This week my guests are the FluffyChat and MinesTRIX maintainers, following the v1 release of the simple and beautiful FluffyChat… and they want to work together!

Dept of Status of Matrix 🌡️

Austin Huang reports

Version numbers of each homeserver, updated daily using a GitHub Action, are now added onto my public homeserver list, in hopes that it will further aid those choosing a homeserver to use Matrix on.

Very handy to quickly know if a server is well maintained or not before making it your new home!

Dept of Spec 📜

anoa says

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://spec.matrix.org/unstable/proposals.

MSC Status

New MSCs:

MSCs with proposed Final Comment Period:

  • No MSCs entered proposed FCP state this week.

MSCs in Final Comment Period:

  • No MSCs are in FCP.

Merged MSCs:

Spec Updates

Extensible events are coming! With lots of potential usecases to be built on top of the concept (such as threading, polls, and any type of data one would like to through on top of existing room events), extensible events are finally getting some love. New MSCs are available above, which detail some of these usecases. Exciting times!

A small correction from last week's issue: the next aggregations MSC to be focused on (after MSC2675 (serverside aggregations) is MSC2677 (annotations and reactions), as it's a more pressing blocker for usecases such as threading and polls.

Otherwise, the Spec Core Team is continuing to wind down in preparation for the holidays ☃️

Random MSC of the Week

The random spec of the week is... MSC3015: Room state personal overrides.

Quite a novel concept, and one that would enable many usecases, such as the ones described in the MSC itself. Check it out if that's something that interests you!

Dept of Servers 🏢

Synapse

Synapse is the reference homeserver for Matrix

dmr reports

This week we cut Synapse release candidate 1.49.0rc1 It includes a bunch of work to supporta plethora of MSCs (MSC 2675, MSC3030, MSC2918, MSC2946), a crop of bugfixes and improvements to our documentation which incorporate Synapse's old wiki. And as ever, there's a bunch of internal type hinting to keep the Synapse team's blood pressure at healthy levels.

The formal release of Synapse 1.49.0 is scheduled for the coming Tuesday. This will be the last Synapse release of 2021 as the Synapse team prepare for a break over the Christmas period. Releases will continue at the usual pace in the new year, with 1.50.0rc1 slated for 2022/01/04 and 1.50.0 for 2022/01/11.

Please note: Synapse 1.49 will be the last version to support Python 3.6, PostrgreSQL 9.6, and Ubuntu 18.04 LTS (Bionic): by our next release, these will have reached their upstream end-of-life. If you're reliant on any of these platforms, please ensure you have plans to upgrade.

In other news, we're preparing to release a new version of Sygnal with a series of fixes for common errors. This should make Sygnal administrators much happier by removing an awful lot of error spam from logs!

Sydent

Sydent is the reference Matrix Identity server. It provides a lookup service, so that you can find a Matrix user via their email address or phone number (if they have chosen to share it).

dmr says

The second blog post on improving Sydent's type coverage should be published now. See the first post here from last week.

Homeserver Deployment 📥️

Helm Chart

Matrix Kubernetes applications packaged into helm charts

Ananace reports

This week has seen yet another set of updates to my Helm Charts, with element-web being bumped to 1.9.6 and matrix-synapse seeing fixes to non-standard port configurations and better support for modern ingressClass handling.

Dept of Clients 📱

Nheko

Desktop client for Matrix using Qt and C++17.

Nico reports

If you are using Nheko on a mobile device like the PinePhone, you should now be able to swipe between the room list and the spaces list. Since I don't use a PinePhone, feedback will be appreciated!

Neochat

A client for matrix, the decentralized communication protocol

Tobias Fella announces

NeoChat version 21.12 is out! You may have noticed that this version number is roughly twenty times higher than the previous one. This means that NeoChat is now twenty times as good as the last version. Or it means that version numbers are utterly meaningless and we switched to a date-based version number system since NeoChat is now released together with many other plasma-mobile related apps. This also means that new versions will arrive monthly from now on. New features and fixes in this version include - but are not limited to:

  • Spell checking while writing a message
  • Improved markdown to html conversion when sending a message
  • Built-in theme switching
  • Various fixes to login, logout and account switching
  • Support for custom emojis
  • Support for Spoilers
  • Support for Blurhashes

Element

Everything related to Element but not strictly bound to a client

Danielle Kirkwood announces

Threads

  • Threads is making excellent progress; This week we held 2 internal testing sessions, both of which went swimmingly.
  • We’re continuing our hard-work on Notifications to fix those up as best we can.
  • Also, we started work on the Threads Filter. The filter will allow you to choose between all the threads in a room and threads you’ve actively participated in.
  • If you’re using the Labs version of Threads, let us know what you think so far!

Polls

  • Exciting news on Polls; All development on Polls MVP is nearly complete, and will soon be making their way to a production environment near you!
  • Polls will be available behind Labs flags at first. We're excited to see people using it and we’re looking forward to hearing any feedback/comments.

Community Testing

  • Closed 20 out of 36 issues in encryption and verification (E2EE) this week.
  • We are planning three testing sessions for next week:
    • Tuesday 17:00-18:00 UTC - first time user experience on iOS, Android and Web
    • Wednesday: 16:00-17:30 UTC - information architecture changes on Web (with Michael and Nique joining us from the Delight team)
    • Thursday 16:30-18:00 UTC - bug squash edition on encryption, can we get the issue count to an all new low?
  • Join us! We’re at #element-community-testing:matrix.org

Element Web/Desktop

Secure and independent communication, connected via Matrix. Come talk with us in #element-web:matrix.org!

Danielle Kirkwood announces

  • We are monitoring and triaging feedback, which is submitted through the new feedback interface in the app.
  • In Labs:
    • Work continues on Information Architecture: this week we’ve made a spotlight search labs feature that we’ll be testing to replace the current filter.
    • We’re also starting to test preferences per space, so keep your eyes peeled for those.

madlittlemods (Eric Eastwood) reports:

Jump to date headers soon in Element

From the experimental MSC3030 implementation merge to Synapse update in TWIM last week, we now also have the start of some client usage in Element to make featureful jump to date headers!

If you've ever tried to find a message back in the past, you've experienced the burdensome task of having to scroll back manually for days, even months! With the jump to date headers, that will be a thing of the past 😌. Clicking any date separator in the room timeline, will give shortcuts to jump to last week, last month, jump to any date using the date picker, or even the beginning of the room to follow a room upgrade chain.

This is currently still in a draft pull request state but will give another update when it lands in Element Labs for everyone to use.

Element iOS

Secure and independent communication for iOS, connected via Matrix. Come talk with us in #element-ios:matrix.org!

Danielle Kirkwood reports

  • On iOS it’s been a week of completing things!
  • We’ve made some final changes to PostHog analytics and MatrixKit has been integrated in element-ios.
  • There have also been lots of bug fixes - especially around an app crash.
  • Don’t forget! As per our update from last week, our release candidates are now prepared on Tuesdays (not Wednesdays).

Element Android

Secure and independent communication for Android, connected via Matrix. Come talk with us in #element-android:matrix.org!

Danielle Kirkwood reports

  • Element Android 1.3.9 has been released on the PlayStore and is available for the beta testers:
    • This version adds support for draft voice messages and a new design for URL previews.
  • Opt-in PostHog analytics will land soon, and will be included in the next release.
  • A new "Legals” screen has been added to Settings in order for users to see all legal info pages for Element, the user's homeserver and the user's identity server (if any).
  • The next release candidate will be prepared next Tuesday.

Commune

Commune is a communications suite built on top of matrix. Commune aims to bring together chat, discussions, email and other interactive apps into a single matrix client.

erlend_sh says

So here’s the thing ahq (dev) and I (product) have been working on: https://github.com/commune-org/commune It’s a chat/forum hybrid. Still in pre-alpha, proof-of-concept stage.

Dept of Non Clients 🎛️

matrix-streamchat

Matrix powered stream overlay for OBS, to integrate live chat in your favorite (selfhosted) streaming setups.

f0x announces

TWIM I wrote a Matrix powered stream overlay for OBS, to integrate live chat in your favorite (selfhosted) streaming setups. Was a great little 2 evening project to develop while livestreaming it's development :)

You can find the code and instructions at https://git.pixie.town/f0x/matrix-streamchat and a hosted instance at https://streamchat.pixie.town

Dept of SDKs and Frameworks 🧰

jOlm

Olm bindings for Java

brevilo announces

jOlm has seen two releases since the previous update, v1.0.7 and v1.0.8. jOlm now supports (and requires at least) the latest libolm version 3.2.7. Please note that we deprecated a number of methods in favor of renamed siblings. The majority of the old ones will be removed in the upcoming jOlm 1.1 release, likely published soon after libolm's announced security release on Dec. 13th. Please follow suit and update your implementations accordingly.

Summary:

  • 🧰 Maintenance and upstream update releases
  • ⚠️ Renamed methods for improved coherence and due to upstream changes (old ones deprecated)
  • ✅ Up to date with Olm 3.2.7 (new minimum requirement)

Changelog:

  • Deprecated methods (see the individual release notes for full details):
    • Account.markOneTimeKeysAsPublished()
    • Account.fallbackKey()
    • InboundGroupSession.export()
    • InboundGroupSession.importer()
    • Utility.ed25519_verify()
  • Refined unit tests
  • Updated dependencies

Cheers!

Dept of Videos 📹

Matthew reports

We previewed yet more native Matrix VoIP conferencing at CommCon: https://www.youtube.com/watch?v=A4k7DVIK5TE&list=PLvNS4EBAxmJJbvGW-PfXdXOSy9AjHjCLV

Half-Shot says

If you've got room for more video content, I also said more things about bridges. In this one, we do a live code session for a twilio bridge and watch it fly! https://www.youtube.com/watch?v=S5q3FLLvRn4

Dept of Interesting Projects 🛰️

Henri says

Wily Messenger Matrix client

Wily has launched an iOS Matrix client to enable messaging in restricted- or poor networks. mText and Room events are transferred as DNS payload, thus bypassing most captive portals, while message headers are minimized to enable messaging in very low bandwidth/high latency networks.

Wily Messenger is in POC stage, missing i.e. encryption at the moment, among others. We are committed to develop it further and invite a Kotlin developer to join our journey. DM @hp:hq.wily.im

Download: https://apps.apple.com/lv/app/id1576476396

Room of the Week 📆

TravisR says

We've set up a new Element Space for the Element family of clients and projects, finally. Feel free to join it at #community:element.io and be sure to check out #community:matrix.org while you're there for everything Matrix related.

There's also #element-translators:matrix.org for the Element Translators community out there.

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server.

#ping:maunium.net

Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1envs.net494.5
2nicoll.xyz548
3matrix.markshorten.co.uk857.5
4helderferreira.io1373
5almum.de1886.5
6diasp.in2958.5
7matrix.liamgooch.com2982
8mailstation.de3030.5
9kde.org3234
10matrix.org3415

#ping-no-synapse:maunium.net

Join #ping-no-synapse:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1envs.net138.5
2construct.supercable.onl165
3weasy-is-my.name317
4grin.hu403
5dendrite.supercable.onl531
6spacedn.com534.5
7matrix.org828
8matrix.awesomesheep48.me1084
9dendrite.s3cr3t.me1103.5
100x1a8510f2.space1332

The Adventures of TWIM bot continued

The members of the Federation have been very active and helped TWIM bot to find all the pieces of its ship! TWIM bot assembled everything together and put the engine back in its place... but at the last moment, as it was ready to take off, another bright spot appeared on the map following madlittlemods late report!

How could this happen? All the pieces of the engine were already there! Unsure of what to do, the bot asked Earth for directions. Earth confirmed: it was worth going to that new bright spot on the map.

The bot welded back the plate of the engine casing, made sure nothing could get into the ship in its absence, and started heading to the mysterious spot. The signal of our communication tools with TWIM bot weakens as it enters in the thick forest of Fuj'ehr…

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Type coverage for Sydent: annotation

10.12.2021 00:00 — Tech David Robertson

This is the second in a series of three posts which discuss recent work to improve type annotations in Sydent, the reference Matrix Identity server. Last time we discussed the motivation for doing this work in the first place: the why. Now I want to talk about the how. How did we add annotations to individual files, and across the project as whole? What common idioms did we learn on the way?

The process of improving coverage

From experience adding typing to Synapse, we decided to annotate one module at a time. We configured a list of files in pyproject.toml which we knew passed mypy's checks. We could run mypy in CI to check that new PRs didn't introduce type problems in modules already covered.

From there, the workflow was

  1. Choose a new module to annotate. Add it to the list of files in mypy's configuration.
  2. Run mypy. See how many errors you get.
  3. Choose an error. Fix it. Re-run mypy.
  4. Repeat until no errors remaining.
  5. Submit for review.

There are two parts in there which involve a choice. Being honest, I made those choices unscientifically: I tried to choose the easy tasks to do first. My first target was actually the entire sydent.util subpackage. Probably a bit too large for a first bite! My thinking was that util sounded like something with few dependencies that would have impact across the whole source tree.

Within a given module, I'd try to fix easier errors first: partly for confidence, partly to build up momentum, and partly to get myself familiar with that piece of source code. For example, I'd often start by telling mypy that mylist = [] was actually was a List[str] (rather than the generic List[Any] which it would use otherwise).

Picking and choosing easy targets works fairly well, but sometimes that means you end up fixing an error that's really a symptom of an earlier one. Other times fixing one error, e.g. by giving a return type annotation to a function—would solve a series of other errors throughout the file. Watching the total number of errors mypy reports bob up and down was intriguing!

In retrospect, I think it would be smoother to generate some kind of dependency graph for the package. I'm imagining a DAG where whose vertices are modules, and there's an edge A -> B if A imports from B. The sinks of this DAG (i.e. modules which don't depend on any others in the package) are the ideal place to start: you can get something strictly typechecked there without having to annotate a long chain of dependencies across other files. Another strategy would be to see which modules were the least precise according to mypy's reports—but more on those next time.

You're at the mercy of your dependencies

I think this is my single biggest takeaway from the process of adding annotations to Sydent. I'll admit the phrasing is melodramatic, but I think it rings true.

Improving coverage boils down to giving the typechecker more information about your program. The more information it has, the more it can check—and the more errors it can spot. (Hopefully this doesn't make typing come across like a pyramid scheme.) If your dependencies aren't typed, mypy can't validate you're correctly providing inputs and correctly consuming outputs. You might have a bigger impact on overall typing coverage by annotating a dependency (directly or via stubs). I have a hunch that bugs are more likely in code that uses an external dependency: we're much more familiar with the details of our own source code compared to that of a third party we trust.

It's worth looking to see if your dependencies have a newer version including type annotations. Failing that, they may have a stub package added to typeshed and published on PyPI. I saw example of both cases when choosing how to configure mypy for Sydent. If some of your dependencies are under your control, consider annotating them—you'll feel the benefits across multiple projects pretty quickly.

Annotations when working with twisted

Twisted is Sydent's biggest dependency, and I certainly felt at its mercy! In particular, it has a few quirks which make it trickier to annotate applications using it. Here's a summary of the work we had to do to get those annotations working.

Partially typed modules and stubs

Early into the process, mypy reported that calling the function twisted.python.log.err was an error. It did so because I was running mypy in --strict mode. We'll talk more about why I did so and what this means next time; for now, it's enough to know that calling a function that isn't fully annotated from within a function that is constitutes an error under strict mode. twisted is partially annotated: many key modules and functions have type annotations, but others don't. I was reluctant to give up on --strict. Instead, I decided to stub the err function myself.

A stub is a cut-down version of a python function, class or module which lives in a .pyi file. All implementation details are removed; only type annotations remain. Stubs are useful when you want to write annotations for code you don't control. The typeshed library is probably the best example: a collection of third party stubs for the standard library, plus some popular third party packages. Microsoft's python-type-stubs is another example. They'd also solve the problem I mentioned at the end of the first part: I could use a stub to teach mypy that IResponse.headers was a Headers object.

Writing a stub for log.err was straightforward, thanks mainly to Twisted's thorough documentation. I chose to write it by hand, rather than use stubgen. I'd heard of the latter, but was reluctant to use it for a few reasons.

  • Twisted is a big project with many big files. I didn't want to commit huge stub files for me and my colleagues to maintain.
  • The more our stubs cover, the larger the risk of a stub becoming out-of-date with twisted itself. Upstream twisted is the best place for these annotations.
  • We only need annotations for the bits of twisted that we're using.
  • And, being honest: I wanted the low-level experience of writing stubs, cross-referencing between the source and working how to best capture its type semantics.

I think this decision to write a targeted stub made sense at the time. After all, twisted.python.logging.err is just one simple function! But for the project as a whole, I regret not using stubgen to generate module-level stubs. The reason for this is that a .pyi stub file accounts for an entire module: no more and no less. This means that the stubs I was writing for parts of twisted only covered the functions and classes I'd stubbed. Any existing annotations in the twisted source code would be ignored, along with any types that mypy was able to infer for itself.

I think it would have been more efficient to use stubgen to generate stubs and patch them up, rather than writing them. That would have helped avoid a few cases I encountered where stubbing one function would cause additional typechecking failures (because mypy was no longer examining the twisted source for that file). It would also have meant that I could just faithfully stub Twisted as it is; in practice, I would sometimes hesitate to stub to avoid having to cover another module. sydent.http was the most painful part of the source tree for this: that's where we make the most use of Twisted.

defer.inlineCallbacks

This is a decorator which allows us to write code in the style of async/await without actually using that syntax. (Twisted predates asyncio and the async/await syntax, introduced in Python 3.4 and 3.5 respectively. It was originally released in 2002, back when Python had released version 2.2.) We only use it in one place in Sydent nowadays, but I've seen used across Synapse too. I mention it here because it was a bit fiddly to annotate. Here's its use in Sydent:

    @defer.inlineCallbacks
    def request(
        self,
        method: bytes,
        uri: bytes,
        headers: Optional["Headers"] = None,
        bodyProducer: Optional["IBodyProducer"] = None,
    ) -> Generator["defer.Deferred[Any]", Any, IResponse]:

Here, request is a generator function because its body uses the yield keyword. Yielding allows the function to relinquish control back to twisted's reactor, only for its execution to be resumed asynchronously in the future. The Generator type takes three parameters:

  • a YieldType, Deferred[Any];
  • a SendType, Any; and
  • a ReturnType, IResponse.

Why have I opted to use Any here, when we've seen (and will see) that this limits mypy's ability to run checks? The answer is that we yield two different types within the function. Firstly a routing result:

        routing: _RoutingResult
        routing = yield defer.ensureDeferred(self._route_matrix_uri(parsed_uri))

and later, the IResponse we go on to return:

        res: IResponse
        res = yield agent.request(method, uri, headers, bodyProducer)
  • In this example:

  • We yield a value y: Deferred[Any].

  • That will later be resolved by twisted to an x: Any value. Twisted will send that value to request, and the execution continues.

  • This repeats, until we eventually return an IResponse.

I could more correctly annotate the YieldType as Deferred[Union[_RoutingResult, IResponse]] so that the SendType was Union[_RoutingResult, IResponse]. But this would mean we end up having some kind of type check at each yield point: a cast, or a type: ignore, or a runtime isinstance check. It didn't feel like it was worth the boilerplate, especially since I could narrow e.g. res: Any to res: IResponse with minimal effort.

This problem doesn't arise with using the async/await syntax:

async def foo() -> int:
    return 1

async def bar() -> None:
    x = await foo()
    reveal_type(foo())
    reveal_type(x)
$ mypy example.py
example.py:6: note: Revealed type is "typing.Coroutine[Any, Any, builtins.int]"
example.py:7: note: Revealed type is "builtins.int*"

Behind the scenes, I think that x = await foo() is really using the same mechanism as the inlineCallbacks approach.

  • An async def function is really a generator function behind the scenes.
  • When we await foo(), we yield the expression foo()
  • Then the machinery running our coroutine c will call c.send(x) to resume execution, where x is the value produced by waiting for foo().

With the await form, mypy knows two things:

  • the value foo() which was yielded should be Awaitable[T], and
  • the value x send to the coroutine should come from awaiting foo(), and therefore be of type T.

Mypy can't assume or enforce these rules for the yield form, which can yield and send whatever it likes. There's no reason why the send type should be related to the yield type. Here's a toy example:

from typing import Any, Dict, Generator


def generator_function() -> Generator[int, str, Dict[str, Any]]:
  y: int = 10
  x: str = yield y
  print("Coroutine was sent", x)  # -> Coroutine was sent hello
  return {"got": x, "done": True}

coroutine = generator_function()
y = next(coroutine)

try:
  coroutine.send("hello")
except StopIteration as e:
  return_value = e.value
  print(return_value)  # -> {'got': 'hello', 'done': True}

All in all, the handling of inlineCallbacks is a situation specific to working with (older?) twisted code. It's still nice to understand what's going on behind the scenes though!

zope.interface.Interface

Twisted makes use of zope's Interface to define a number of abstract interface classes. Speaking personally, I've not seen it used outside twisted, and I think that means it's not supported by much of the typechecking tooling. For example, I've definitely seen PyCharm struggle to realise that it's okay to pass a Response to a function which expects an IResponse! Here's a more complicated example where PyCharm isn't happy with me widening the type LoggingHostnameEndpoint to IStreamClientEndpoint, even though the latter implements the former.

Screenshot from pycharm showing a false positive warning

Mypy out of the box doesn't play well with a zope Interface (nor does any other typechecker I tried). Fortunately, the excellent mypy-zope plugin helps here: it teaches mypy that any class like Response which @implements(IResponse) can be passed in place of an IResponse.

Tricks of the trade

At this point I'd like to share a few generic lessons about typing I'd picked up. Nothing ground-breaking here: I think these are all fairly well-known. Hopefully they're the start of a good cheat sheet for annotating—though it pales in comparison to the Mypy cheat sheet.

Annotating decorators

Annotating decorators is fiddly, but it's also vitally important. An unannotated decorator will mask or throw away your decoratee's annotations! The way to write the annotation is best explained by the mypy docs, but briefly: it involves a TypeVar and sometimes a cast too. Here's an example.

from typing import TypeVar, Callable, Any, cast

F = TypeVar("F", bound=Callable[..., Any])

def decorator(input: F) -> F:
    def wrapped(*args: Any, **kwargs: Any) -> Any:
        print(f"Calling {f.__name__}")
        return input_func(*args, **kwargs)
    return cast(F, wrapped)

The idea here is

  1. Use Callable[..., Any] to describe a generic function with no particular signature.
  2. Use that as a bound on a type variable F. At each usage of @decorator, mypy will deduce a more specific version of F, e.g. Callable[[int], str].
  3. Within that usage of @decorator, F is fixed to that specific type. We use -> F to express that "we return a function with the same signature as the decoratee".
  4. Unfortunately, we don't have a good way to tell mypy that wrapped also has that signature F. We resort to a cast to force mypy to accept this without proof.

This might change in the future, e.g. when ParamSpec is fully understood by mypy.

Prefer object over Any

Both of these are general types for expressing "I don't know anything about this expression". But only the former will undergo static type checks. We want those type checks to guard against bugs accidentally introduced in the future. For instance, imagine a class with an Any attribute.

from typing import Any
import dataclasses

@dataclasses.dataclass
class C:
    label: Any

Imagine in the future we add a new method which assumes that label is a string:

    def greeting(self) -> str:
        return "My name is " + self.label

Mypy will consider this valid, because no type-checking is done on an Any value. It will complain that it can't prove that greeting returns a str, if --warn-return-any is enabled; but putting that aside, it can't identify the call site as a bug. The bug slips through to runtime.

C(123).greeting()  # TypeError: can only concatenate str (not "int") to str

Replacing the Any with object does allow mypy to spot the problem.

error: Unsupported operand types for + ("str" and "object")  [operator]

# type: ignore and cast sparingly

There's a good chance that mypy knows better than we do, so we should only overrule it if there's no better option. There are two techniques for this. One option is to tell mypy to just silence the error, by appending a # type: ignore comment to the erroneous line. The other is to force it to accept that a certain expression has a given type: that's what cast is for.

I never like using either of these, but sometimes they're the most practical choice. I'd recommend two best practices for their use, however:

  1. Only ignore a specific error code, e.g. # type: ignore[assignment]. Those codes can be displayed by passing --show-error-codes to mypy.
  2. Leave a comment before every #type: ignore[...] and every cast to justify their correctness. I don't think this is a widely-held practice. I picked it up from the Rust world, where it's encouraged as a way to justify unsafe source code.

There's good stuff in typing

It's worth a read through the module documentation—plenty of things in there I wish I'd known about sooner. There's lots in the toolkit to chose from. Some bits I use fairly often include:

  • Protocol: lets you formalise duck typing. To use it, define a class that inherits from Protocol. Its methods and attributes are all stubs which describe what you require of objects belonging to this type. It's like an abstract base class or interface, but purely at typecheck time.
  • Generic: define your own generic types. A bit of a slippery slope to type mania!
  • [Optional](https://docs.python.org/3/library/typing.html?highlight=typing%20generic#typing.Optional): I use this all the time. It's always good to make your None`s explicit!
  • NewType: I haven't had a chance to use it much. As I understand it, it's a way to define a "strong typedef". For example, we can use it to distinguish lengths from durations, even if they're both represented by a float at runtime.

I want to call out two parts of typing in particular:

overload

overload is a way to provide extra information about a function depending on how it's called. For instance, consider this function which takes a str or bytes as input and returns its uppercase version.

def upper(x: Union[bytes, str]) -> Union[bytes, str]:
    return x.upper()

The problem with this annotation is that we'll have to check at every call site to see if the return value was a bytes or a str object. But we know that uppercasing a str gives us a str, and uppercasing a bytes gives us a bytes. We can use overload to express this.

from typing import overload

@overload
def upper(x: str) -> str: ...

@overload
def upper(x: bytes) -> bytes: ...

def upper(x: Union[bytes, str]) -> Union[bytes, str]:
    return x.upper()

The first two @overload definitions are like stubs: they're purely used for their annotations. The actual runtime implementation is specified at the end, undecorated. (NB: this specific example is better expressed without overloads, by using AnyStr.)

I was really impressed to see how mypy could use this overloading information. Here's an example:

from typing import overload, Literal

@overload
def f(x: int) -> Literal[1]: ...

@overload
def f(x: str) -> Literal[2]: ...

def f(x: object) -> int:
    if isinstance(x, int):
        return 1
    elif isinstance(x, str):
        return 2
    return 3

if f(10) == 2:
    print("potato")

We can see that f(10) == 1 and so the equality is always False: we'll never print the word "potato". Mypy can reason through this too, if we ask it nicely.

$ mypy example.py
Success: no issues found in 1 source file

$ mypy --strict-equality --warn-unreachable example.py
example.py:16: error: Non-overlapping equality check (left operand type: "Literal[1]", right operand type: "Literal[2]")
example.py:17: error: Statement is unreachable
Found 2 errors in 1 file (checked 1 source file)

TypedDict

I mentioned this in last week's post) when talking about the missing await bug. Subclassing from TypedDict allows us to define a new type whose values are dictionaries with

  • a fixed set of keys (some mandatory, some not), and
  • a fixed type for each key's value.

PEP 589 can best describe the motivation, but in short: there's a lot of source code out there that passes dictionaries around. TypedDict is a way to gradually add typechecking to that code, without having to refactor it to use e.g. a dataclass or a NamedTuple. Let's have a quick example.

from typing import List, TypedDict

class Person(TypedDict):
    full_name: str
    nicknames: List[str]
    age: int

p: Person = {
    "full_name": "David Matthew Robertson",
    "nicknames": ["dmr"],
    "age": 29,
}

Mypy will detect if you delete or omit a required key, or if you insert a key that's not part of the type.

# error: Key "age" of TypedDict "Person" cannot be deleted
del p["age"]

# error: Missing keys ("full_name", "nicknames", "age") for TypedDict "Person"
p2: Person = {}

# error: TypedDict "Person" has no key "eye_colour"
p["eye_colour"] = "brown"

TypedDict is a useful tool, but there are a few important gotchas to be aware of. In my view, these are all minor and worth putting up with, to benefit from the extra type information that a TypedDict provides. Let's take a look.

TypedDict is incompatible with Dict

This one surprised me at first. Why can't I pass my TypedDict to a function that accepts a generic dictionary?

import json
from typing import Dict, TypedDict

class Foo(TypedDict):
    bar: str

def print_size(d: Dict[object, object]) -> None:
    print(len(d))

f: Foo = {"bar": "baz"}
# error: Argument 1 to "print_size" has incompatible type "Foo"; expected "Dict[object, object]"
print_size(f)

The answer is that it wouldn't be type-safe! For all we know, the function print_size might mutate its argument d. It might add a key, remove a key, or change the type of a value. Each of those would break the contract that TypedDict is supposed to enforce, so mypy is correct to flag this as an error. This GitHub issue has more discussion.

There are few workarounds for this kind of problem.

  • The function might be able to accept a TypedDict directly.
  • The function could accept a Mapping instead of a Dict. This is type-safe because the Mapping type does not offer mutating methods such as pop, __setitem__, __del__; but it only works if the function does not mutate the dictionary.
  • A more drastic approach would discard the TypedDict information with a cast to Dict[str, object] or similar. If so, I'd strongly recommend a comment explaining why the cast is correct and safe.
Mixing mandatory and required fields

Secondly, defining a TypedDict with a mixture of optional and required keys is a little fiddly. All keys can be made optional by passing total=False in the class definition. To have some keys optional and others mandatory, we have to make two TypedDicts. Say for instance we wanted to allow a potentially-missing favourite_colour field to Person. We can't add an annotation favourite_colour: Optional[str] to the first class body. That's optional in a different sense: it would mean that favourite_colour is a mandatory field which is allowed to be None. Instead, we apply total=False to a subclass:

from typing import List, TypedDict

class _PersonRequired(TypedDict):
    full_name: str
    nicknames: List[str]
    age: int

class Person(_PersonRequired, total=False):
    favourite_colour: str

This means that a valid Person dictionary may have no favourite_colour key. If it is present, it must be a str.

The syntax is unfortunately a little clunky. PEP 655 acknowledges this and proposes an alternative.

TypedDict is typecheck-time only

One final note: a TypedDict does no validation or conversion whatsoever. If mypy can't know the keys and values a dictionary will have a typecheck-time, you'll need to validate it by hand at runtime. We see this a lot because when deserialising json objects into a dictionary. Here's an example.

import json
from typing import TypedDict

class Foo(TypedDict):
    bar: str

# note: Revealed type is "Any"
reveal_type(json.loads("{}"))
# no error. mypy can't do analysis on Any.
f: Foo = json.loads("{}")

Next time

This covers a lot of the machinery and the day-to-day process of annotating Sydent. The last part of this series will give us a chance to quantify our efforts and reflect on the wider typing ecosystem.


Many thanks for reading! If you've got any corrections, comments or queries, I'm available on Matrix at @dmrobertson:matrix.org.

Pre-disclosure: upcoming security release of libolm and matrix-js-sdk

03.12.2021 00:00 — Security Matrix Security Team

On Monday, 13th December we plan to publish a security release of libolm at 15:00 UTC to address a single high severity issue. To the best of our knowledge, only matrix-js-sdk and clients relying on it for E2EE are affected by this issue. This includes Element Web/Desktop and their forks (like SchildiChat). The release of libolm will be immediately followed by a security release of matrix-js-sdk and the affected clients. Users of these clients are encouraged to upgrade as soon as the patched versions are released.

We will be reaching out to downstream packagers to ensure they can prepare patched versions of the affected packages at the time of the release. The details of the vulnerability will be disclosed in a blog post on the day of the release. There is so far no evidence of the vulnerability being exploited in the wild.

The patched version numbers will be as follows:

  • libolm 3.2.8
  • matrix-js-sdk 15.2.1
  • Element Web/Desktop 1.9.7

Thank you for your patience while we work to resolve this issue.

Edit, 2021-12-13: Added patched release numbers.

This Week in Matrix 2021-12-03

03.12.2021 00:00 — This Week in Matrix Thib

The Adventures of TWIM bot

One thing you might not know is that TWIM bot is a space traveler, sent by the Matrix scientists to explore that zone called "The Possibilities". The #twim:matrix.org room is a portal to its energy tank, and we had received a distress signal!

To help the TWIM explorer fulfil its mission, we asked the Matrix community to fuel it with news before it crashed into space debris made of aggregated ignorance!

This week again, the community has been very active and explored many possibilities of the Matrix universe!

Matrix Live 🎙

For this week's Matrix Live my guest is Amandine and we're discussing how Element and 50 other organisations are trying to shape the future of EU's law for more interoperability. Bonus point: we have a double bridge demo with Matrix, Slack and Telegram!

Dept of Status of Matrix 🌡️

FOSDEM!

This year, the Matrix.org Foundation is excited to host the first ever Matrix.org Foundation and Community devroom at FOSDEM. A full day of talks, demos and workshops around Matrix itself and projects built on top of Matrix. Read (and answer to) our Call for Partipactions!

Finnish Admins to the Rescue

cos says

A group of Finnish Matrix admins have set up a free homeserver for Finnish public called pikaviestin.fi (literally instant messenger dot fi). It offers a bunch of bridges and registration requires an e-mail address in one of Finnish e-mail providers or organizations. We welcome all Finns to register there and help decentralize Matrix. Support room can be found at #aula:pikaviestin.fi

That's a fantastic initiative! Kudos to all the sysadmins involved!

Dept of Spec 📜

anoa reports

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://spec.matrix.org/unstable/proposals.

MSC Status

New MSCs:

MSCs with proposed Final Comment Period:

  • No MSCs entered proposed FCP state this week.

MSCs in Final Comment Period:

Merged MSCs:

  • No MSCs were merged this week.

Spec Updates

The end of the year is drawing to a close. Thus many of the Spec Core Team members are focusing on implementation in order to meet deadlines. Review is still occurring though! As above, we have MSC3419 (allow guests to send more event types). This was born out of next-generation VoIP work, but it should have a positive impact on improving the guest experience in Matrix on the whole.

Otherwise work is still ongoing by Bruno and others on untangling the aggregations MSCs, specifically MSC2675 and MSC2676.

And finally, Alexandre Franke has PR'd some work to allow for matrix.org's OpenAPI spec to be widely available, meaning anyone with a Swagger (or other OpenAPI viewer) client can easily pull it and start sending requests against a Matrix homeserver. Fun times!

Random MSC of the Week

The random spec of the week is... MSC3419: Allow guests to send more event types.

Random numbers, ladies and gentleman.

Dept of Servers 🏢

Synapse

Synapse is the reference homeserver for Matrix

callahad says

Goooood evening TWIM readers!

I want to start by drawing attention to a blog post which we published today: Type coverage for Sydent: motivation. This the first in a series of three articles discussing what we've learned from making Sydent pass the mypy type checker in strict mode. Improving type coverage across Synapse, Sygnal, and Sydent has been a major focus of the backend team at Element for the past few months, and we think we've learned a few useful things in the process.

This week we also released Synapse 1.48 with loads of internal improvements, new Admin APIs, better alignment with the Matrix 1.1 spec, and more. We're planning one more release for the year, 1.49 on December 14th, and then we're taking a break until Synapse 1.50 on January 11th.

Importantly: Synapse 1.49 will be the last release to support Python 3.6, PostrgreSQL 9.6, and Ubuntu 18.04 LTS (Bionic) — if you're reliant on any of these platforms, please ensure you have plans to upgrade.

Let us know what you think of the article (and the Synapse release!), and we'll see you next week!

Sydent

Sydent is the reference Matrix Identity server. It provides a lookup service, so that you can find a Matrix user via their email address or phone number (if they have chosen to share it).

dmr reports

I've just published a blog post (part one of three) about our efforts to improve Sydent's type coverage. It should hopefully be of interest to anyone who works with Python or is interested in static analysis more generally.

Gitter

madlittlemods (Eric Eastwood) reports

In the vein of Gitter feature parity on Matrix, we've made the first steps towards a better public static archive. We merged an experimental implementation of MSC3030 into Synapse which lets you use the unstable /timestamp_to_event client API endpoint go from a given timestamp to the closest event ID. This will allow us to implement a calendar jump to date interface to be able to navigate to any day in the rooms history. Our first target to add the jump to date UI in is Hydrogen since we plan to server-side render Hydrogen for the actual public static archive as well.

To enable the MSC3030 unstable API endpoints in Synapse, add experimental_features -> msc3030_enabled: true to your homeserver.yaml:

GET /_matrix/client/unstable/org.matrix.msc3030/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>
{
    "event_id": ...
    "origin_server_ts": ...
}

Also as part of MSC3030, when you use the client API endpoint, if your homeserver sees that the closest event it has locally in the database is next to a gap in the history, it will go out and ask other federated homeservers what they have as the closest event instead.

GET /_matrix/federation/unstable/org.matrix.msc3030/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>
{
    "event_id": ...
    "origin_server_ts": ...
}

*--

MSC2716 to import batches of historical messages is still marching along getting some polishing passes and strengthening the assertions in the Complement tests to make sure things are going absolutely correctly. It's also good to see Beeper utilizing it and catching a few bugs along the way 💪.

Dept of Bridges 🌉

Hookshot

Half-Shot reports very late, to the great despair of TWIM's editor:

Hookshot gets provisioning!

Stop the press. This is a last minute TWIM. We've been beavering away on matrix-hookshot. It's gained many features in the last week, but the big thing is that hookshot has gained the ability to provision connections over a provisioning API, which means it should hook nicely into Dimension (and other integration managers, in the future)!

Other notable features are:

  • Support for multiple webhooks per room
  • Support for the username/text fields on an incoming webhook (slack style)
  • Named webhooks, so each hook now has a sensible displayname
  • The ability to spawn GitHub actions from rooms using the !gh workflow run command
  • Lots of new supported events from GitLab, such as reviews and tag pushes
  • Hosted documentation (so all of the above is easy to setup), it's a bit in progress atm.

We're aiming for a release very very soon, hopefully in the next week or so!

Homeserver Deployment 📥️

Helm Chart

Matrix Kubernetes applications packaged into helm charts

Ananace says

And this week, as a complete and utter surprise, my Helm Charts got updates; with matrix-synapse updated to 1.48.0

Dept of Clients 📱

SchildiChat

SchildiChat is a fork of Element that focuses on UI changes such as message bubbles and a unified chat list for both direct messages and groups, which is a more familiar approach to users of other popular instant messengers.

qg announces

In a new release being published just now we added the possibility to mark rooms as unread also on Web/Desktop (using MSC2867, huge thanks to @alangecker for his PR on Element Web!). This has already been implemented in SchildiChat-Android and is now enabled on both by default.

Nheko

Desktop client for Matrix using Qt and C++17.

Nico says

We finally figured out what caused the issues with the flatpak on GNOME, especially on Arm. It should now work properly, if you use Flathub. On the Pinephone (and other systems, that don't set a locale/use the C locale), timestamps should now not be needlessly long anymore. Redactions got a face-lift to distinguish them better from normal messages. We added a workaround for Synapse not allowing you to leave a banned room. We now delete the room from the room list permanently if Synapse returns "unknown room" when trying to leave it. Spaces can now show the entire hierarchy in the sidebar (if you pull it out) and you can navigate to subspaces by clicking on them in the roomlist, even if you collapsed the space hierarchy in the sidebar.

That's all, now let me bake some cookies! 🍪

FluffyChat

Krille Fear says

Today we have released FluffyChat 1.0.0 with a whole new design, a lot of bug fixes and huge performance improvements.

New design

The new design has bigger message bubbles with fancy shadows and bigger fonts. The contrast has been improved and some elements, like the time on every single message bubble, are now hidden by default. But they are not gone! Detailed message information are now accessible in the new message info page, where we not also can see the message type and the timestamp, but also the whole JSON source code of each timeline event.

Spaces

Spaces have got a lot improvements and bug fixes. They have moved to the bottom bar of the chat list (while this bottom bar is still hidden if you have not joined any space yet). The multi account switcher have instead been moved to a top left drop-down menu. So we finally got rid of the drawer, which seems to be a deprecated material design feature anyway. This new UX makes spaces much easier to use. You can long press on them to go to the space settings and long press on any chat in the chat list, to add or remove a chat to (or from) a space. We still have no support for the spaces summary API though so we don't have yet the ability to discover new rooms inside of a space but this feature might land soon in the Matrix Dart SDK.

Multi Account

FluffyChats multi account is still in beta but got a lot of bug fixes as well. You are now able to sort your accounts in "bundles" which can be very handy. The new account switcher button gives you a much better overview over your connected Matrix accounts now.

Performance

We did a lot refactoring under the hood in our Matrix Dart SDK and have improved our in-app database a lot. On the web it now uses IndexedDB natively while it tunes all database transactions on all platforms. This leads to the fastest FluffyChat experience we ever had and makes the app finally kinda usable with bigger accounts on all platforms. The room list is now lazy loaded which speeds up the app start (especially with multi account enabled) a lot. Choose your own primary color This was a long requested feature. You can now choose your favorite color to style your FluffyChat for your needs:

What will you choose? Let me know in the comments. I mostly like blue on my Ubuntu desktop.

New major version?

Ahhh by the way... What does it mean that we now have FluffyChat 1.0.0? It does NOT mean that the previous versions were not yet stable or ready for daily use. It just means that we make so many changes at once that we thought, bumping the first digit of our pseudo-semver version string might make sense. We totally messed up our versioning and are now going to do it better. Promised!! What's next? We are often asked: What is the roadmap of FluffyChat? Well... we still don't have a clear roadmap and might never have. FluffyChat is completely driven by volunteers. But what I can say that we would like to do in the next months is:

  • Better QA -> We would like to write some integration tests, push release candidates before new releases and involve everyone in testing them to offer the best stability possible.
  • Native video calls -> Yes! There will soon land support for native video calls in the Matrix Dart SDK and we are going to implement this in FluffyChat.
  • Stories -> Like you might know from SnapChat, WhatsApp or Instagram, stories are little messages you can send to all of your contacts and which will disappear after 24 hours. I would really like to implement this in FluffyChat!
  • Better notifications for iOS
  • Deeper support for spaces
  • Knocking feature
  • Drag&Drop for web

But as I said this is what we would like to do. We can't give any warranties on anything. We can only do our best. But you can help us if you like (You don't have to).

  • Join the FluffyChat community: https://matrix.to/#/#fluffychat:matrix.org
  • Report bugs at our issue tracker: https://gitlab.com/famedly/fluffychat/-/issues
  • Help with the translations and join our translators team: https://matrix.to/#/#fluffychat-translation:matrix.org
  • Help with development directly in GitLab <3
  • ... or support us on Liberapay so we can organize more FluffyChat developer meetings: https://matrix.to/#/#fluffychat-translation:matrix.org

The complete changelog for FluffyChat 1.0.0:

  • design: Chat backup dialog as a banner
  • design: Encrypted by design, all users valid is normal not green
  • design: Move video call button to menu
  • design: Display edit marker in new bubbles
  • design: Floating input bar
  • design: Minor color changes
  • design: Move device ID to menu
  • design: Place share button under qr code
  • design: Redesign and simplify bootstrap
  • design: Remove cupertino icons
  • feat: Display typing indicators with gif
  • feat: Fancy chat list loading animation
  • feat: New database backend with FluffyBox
  • feat: Make the main color editable for users
  • feat: Move styles one settings level up
  • feat: Multiple mute, pin and mark unread
  • feat: New chat design
  • feat: New chat details design
  • feat: New Public room bottom sheet
  • feat: New settings design
  • feat: Nicer images, stickers and videos
  • feat: nicer loading bar
  • feat: Open im.fluffychat uris
  • feat: Redesign multiaccounts and spaces
  • feat: Redesign start page
  • feat: Send reactions to multiple events
  • feat: Speed up app start
  • feat: Use SalomonBottomBar
  • feat: Drag&Drop to send multiple files on desktop and web
  • fix: Adjust color
  • fix: Automatic key requests
  • fix: Bootstrap loop
  • fix: Chat background
  • fix: Chat list flickering
  • fix: Contrast in dark mode
  • fix: Crash when there is no prev message
  • fix: Do display error image widget
  • fix: Do not display bottombar in selectmode
  • fix: Dont enable encryption with bots
  • fix: Dont loose selected events
  • fix: Dont rerun server checks
  • fix: download path for saving files
  • fix: Hide FAB in new chat page if textfield has focus
  • fix: Let bottom space bar scroll
  • fix: Load spaces on app start
  • fix: Only mark unread if actually marked
  • fix: Public room design
  • fix: Remove avatar from room
  • fix: Remove broken docker job
  • fix: Report sync status error
  • fix: Self sign while bootstrap
  • fix: Sender name prefix in DM rooms
  • fix: Set room avatar
  • fix: Various multiaccount fixes
  • fix: Wrong version in snap packages

What a massive update! Little birds told me we will hear about FluffyChat very soon!

Element

Everything related to Element but not strictly bound to a client

kittykat says

Threads

  • On Web, work continues on notifications and integration with homeserver APIs to improve user experience.
  • On Mobile, link sharing has been added and work is about to start on notifications.

Polls

  • Polls are nearly ready! If you enable this feature in labs, you can create a poll with several options, and people can vote on it.
  • We’re working on the finishing touches, and the first version of polls will be available in a release (on Element Desktop, Web, Android and iOS) within a few weeks.

Community testing

  • We closed 34 encryption bugs which had been resolved by improvements to the workflows and user interfaces.
  • Due to the overwhelming success with bug squash sessions in the last few weeks, we are making these a regular feature. Our next session will be on Thursday 9 December at 17:00 UTC.
  • For information about upcoming sessions and to join in, join #element-community-testing:matrix.org

Element Web/Desktop

Secure and independent communication, connected via Matrix. Come talk with us in #element-web:matrix.org!

kittykat announces

  • In labs, work continues on Information Architecture: new history interaction to replace breadcrumbs
  • We are monitoring and triaging feedback which is submitted through the new feedback UI in the app.
  • Fixed long standing bug around link formatting - links are not formatted as markdown any more.

Element iOS

Secure and independent communication for iOS, connected via Matrix. Come talk with us in #element-ios:matrix.org!

kittykat says

  • Analytics: final changes to allow opt-in analytics reporting with PostHog
  • MatrixKit has been integrated into element-ios in preparation for moving to the SwiftUI framework
  • Release Candidates are now scheduled on Tuesdays (previously on Wednesdays) which will bring them in line with Web releases.

Element Android

Secure and independent communication for Android, connected via Matrix. Come talk with us in #element-android:matrix.org!

kittykat announces

  • Element 1.3.9 has been submitted to Google: it adds support for voice message drafts and many bug fixes.
  • Starting work on new login flow: the user will be asked if they have an account or want to create one on the very first screen.
  • Analytics (PostHog): implementing the opt-in screen. Should be included in the 1.3.10 release.
  • Release Candidates are now scheduled on Tuesdays (previously on Wednesdays) which will bring them in line with Web releases.

Dept of SDKs and Frameworks 🧰

Trixnity

Multiplatform Kotlin SDK for Matrix

Benedict says

Trixnity, a multiplatform Matrix SDK written in Kotlin, has grown up since the last release 6 month ago! It has it first release candidate for v1.0.0!

If you don't heard about Trixnity: Trixnity aims to be strongly typed, customizable and easy to use. You can register custom events and Trixnity will take care, that you can send and receive that type.

The most exciting thing is the new trixnity-client module. It provides a high level client implementation and allows you to easily implement clients for Desktop, Mobile and Web. You just need to render data from and passing user interactions to Trixnity. The key features are:

  • exchangeable database
  • fast cache on top of the database
  • E2E (olm, megolm)
  • verification
  • room list
  • timelines
  • user and room display name calculation
  • asynchronous message sending without caring about E2E stuff or online status
  • media support (thumbnail generation, offline "upload", etc.)
  • redactions

At the moment, Trixnity only supports JVM in all modules, but JS and Native will follow soon (to be exact: when Kotlin 1.6.10 and ktor 2.0.0 is released). I also implemented the module trixnity-olm, which implements the wrappers of libolm for Kotlin JVM/JS/Native.

Cross signing is one of the next big features, I want to implement.

simplematrixbotlib

simplematrixbotlib is an easy to use bot library for the Matrix ecosystem written in Python and based on matrix-nio.

krazykirby99999 reports

Version 2.4.1 Released!

Docs Changes:

  • Added missing await statements to several examples
  • Added additional clarification on using the "m.notice" msgtype
  • Used Markdown instead of HTML to display a specific link

Example usage is shown below:

import simplematrixbotlib as botlib

creds = botlib.Creds("https://home.server", "user", "pass")
bot = botlib.Bot(creds)
PREFIX = '!'

@bot.listener.on_message_event
async def echo(room, message):
    match = botlib.MessageMatch(room, message, bot, PREFIX)
    if match.is_not_from_this_bot() and match.prefix() and match.command( "echo"):
        response = " ".join(arg for arg in match.args())
        await bot.api.send_text_message(room.room_id, response)

bot.run()

A thank you to HarHarLinks for their contributions to version 2.4.1!

Request additional features here.

View source on Github View package on PyPi View docs on readthedocs.io https://matrix.to/#/#simplematrixbotlib:matrix.org

Dept of Videos 📹

andybalaam announces

I'm exploring the matrix-rust-sdk on my live stream every week. I'm working on a simple Rust bot for Matrix. Come watch me struggle with the compiler on PeerTube or Twitch every wednesday at 14:00 UTC!

Room of the Week 📆

Timo ⚡️ says

Hi everyone! Did you ever feel lost in the Matrix world? The room directory is big, but it's still hard to find something you like. Or are you a room moderator, but there is not much activity in your room because it doesn't have enough users?

This is why I want to share rooms (or spaces) I find interesting.


This week's room is: #audiophiles:matrix.org

"Headphones, Speakers, IEM and any audio related equipment. Music recommendations as well."


If you want to suggest a room for this section, tell me in #roomoftheweek:fachschaften.org

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server.

#ping:maunium.net

Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1matrix.markshorten.co.uk1013
2helderferreira.io1099
3envs.net1438
4thomcat.rocks2552.5
5matrix.sp-codes.de3732.5
6jauriarts.org3756.5
7trygve.me3829
8grimneko.de5095.5
9kittenface.studio5606
10jeroenhd.nl6970

#ping-no-synapse:maunium.net

Join #ping-no-synapse:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1matrix.awesomesheep48.me1042
20x1a8510f2.space1076
3dendrite.s3cr3t.me4405

The Adventures of TWIM bot continued

Following the late reports of the spec and hookshot updates, TWIM bot's ship went into hyperspeed. Our dear bot lost control of the ship and we lost its signal. We're doing our best to contact it and hope it's safe!

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Type coverage for Sydent: motivation

03.12.2021 00:00 — Tech David Robertson

This is the first of three posts on improving type coverage in Sydent. Join us next Friday for the second part!

Sydent is the reference Matrix Identity server. It provides a lookup service, so that you can find a Matrix user via their email address or phone number (if they've chosen to share it).

We recently worked on improving Sydent's type coverage: the proportion of its source code with explicit annotations denoting the kind of data that each variable, expression and return value is expected to hold. These annotations are optional, but if present, they allow tools like mypy to analyze your programs and spot entire classes of bugs before you ship them! In this instance, we aimed to make Sydent pass mypy --strict, which it now does. Here's what the process looked like:

Coverage as measured by mypy. Precision and the number of typed expressions increase over the latter half of 2021.

Two lines show two different measures of how well-typed the project is. The grey region covers our two-week sprint towards improving coverage; the earliest data point is from just before previous efforts to improve typing earlier in the year.

In a series of posts, I'd like to reflect on this sprint and share what we've learned. In particular, I aim to:

  • explain why we wanted to improve type coverage now;
  • work through examples to see how (if?) mypy could have spotted bugs;
  • describe the annotation process;
  • illustrate common patterns I learned along the way;
  • discuss the checks that mypy provides; and finally
  • reflect on the state of Python's typing ecosystem.

In this first part, we'll concentrate on the first two topics; the second covers the middle two; and the third the last two.

Why do this now?

It took us a long time (too long) to notice that the Sydent instance serving matrix.org was failing to send SMS messages for verification. We suspected that something was going wrong with our API call to OpenMarket. Our first step was to improve logging, so we could start to deduce what was going wrong and why. Whilst trawling through logs, we spotted one problem which meant we weren't actually sending off the API request in the first place. Further investigation revealed a strings-versus-bytes confusion which meant that we would always (incorrectly) interpret the API response as having failed.

All in all, phone number verification was unknowingly broken in the 2.4.0 release, to be fixed in 2.4.6 a month later. How could we do better? Better test coverage is (as ever) one answer. But it struck me that the two bugs we'd encountered might be ripe for automatic detection:

  • we created an Awaitable but didn't await it or use it in any way, and
  • we tried to look up a str key in a dictionary which mapped bytes to bytes.

Could a static analysis tool like mypy detect these? How much work would it take to do so? Are there other bugs and problems we could spot with it? I was curious to answer these questions and learn more about the tools that Python's typing ecosystem provides.

Could typing have spotted these problems?

Let's start with the first of question: what can mypy detect?

The missing await

Instead of writing x = await foo(), we simply had x = foo() and didn't then go on to await x. Mypy doesn't have means to detect this at present. There was interest in this issue on such a feature, with related threads here and here.

Are there other opportunities to spot the error? Here's the relevant bit of source code from before the fix.

            sid = self.sydent.validators.msisdn.requestToken(
                phone_number_object, clientSecret, sendAttempt, brand
            )
            resp = {
                "success": True,
                "sid": str(sid),
                "msisdn": msisdn,
                "intl_fmt": intl_fmt,
            }

The call to requestToken produces a value of type Awaitable[int]. If we tried to assign that to an expression of type int we'd get an error that mypy can spot.

$ cat example.py
async def foo() -> int:
    return 1

async def bar():
    x = foo()      # no error
    y: int = foo() # error: rhs is Awaitable[int], but lhs expects int

$ mypy --check-untyped-defs example.py
example.py:6: error: Incompatible types in assignment (expression has type "Coroutine[Any, Any, int]", variable has type "int")
Found 1 error in 1 file (checked 1 source file)

Note that we have to specifically ask mypy to typecheck the body of bar by passing --check-untyped-defs; by default, mypy will only typecheck annotated code.

We might also have been able to detect the error by looking at how we used sid. Unfortunately, the only use of was in a conversion str(sid), which is a perfectly type-safe call for both sid: int and sid: Awaitable[int]. But let's put that aside for a second—suppose we added "sid": sid directly into the resp dictionary. Could mypy have spotted there was a problem with that?

Unfortunately not. Because resp has no annotation, we have to rely on how it's used to spot any type inconsistencies. There's only one use of resp: as the return value from its enclosing function, render_POST. Mypy's only chance to spot a type problem would be to compare the mypy's inferred type for resp to the return type of render_POST. What are those types? We can use reveal_type to see the former is Dict[str, object]. For the latter:

    @jsonwrap
    def render_POST(self, request: Request) -> JsonDict:

The return type is JsonDict, which is an alias for Dict[str, Any]. This is bad news, because Dict[str, object] and Dict[str, Any] are compatible. Digging a level deeper, this is because sid: Any holds true for both sid: int and sid: Awaitable[int]—so there's no error to spot here. The Any type is compatible with any other type whatsoever! Mypy uses Any as a way to defer all type checking to runtime; mypy won't (and can't!) statically analyse the usage of an expression of type Any. Indeed, mypy's reports will tell you how many Anys you're working with, and offer a variety of options to warn or error on usages of Any.

If we were inserting sid directly into a dictionary, we could do better by annotating the dictionary (or the function's return type) as a TypedDict. This is a way to specify a dictionary with a fixed set of keys, each with a fixed type. It comes in really handy for Sydent, Sygnal and Synapse—all of the Matrix APIs exchange JSON dictionaries, so anything we can do to teach mypy about their shape and types is gold dust.

In short, there were options for detecting this with some code changes, but no magic wand that would have spotted the error in the code as written.

The strings/bytes confusion

Our error was here:

        headers = dict(resp.headers.getAllRawHeaders())
        request_id = None
        if "X-Request-Id" in headers:
            request_id = headers["X-Request-Id"][0]

In this sample, resp.headers is a twisted.web.http_headers.Headers instance. getAllRawHeaders is documented as returning an iterable of (bytes, Sequence[bytes]) pairs. Even better, mypy can see this because getAllRawHeaders is annotated (many thanks to the twisted authors for this). Mypy should be able to deduce that we build a dictionary headers: Dict[bytes, Sequence[bytes]. We can check this using reveal_type:

        headers = dict(resp.headers.getAllRawHeaders())
        reveal_type(headers)
$ mypy
sydent/sms/openmarket.py:110: note: Revealed type is "builtins.dict[builtins.bytes*, typing.Sequence*[builtins.bytes]]"

(The * in builtins.bytes* here means mypy has inferred that the dictionary's keys are bytes, rather than being told explicitly that they must be bytes.)

That's all fine and dandy. But why didn't we spot this before if the annotations were all in place in twisted? Let's put aside the fact that, erm, we weren't running mypy in Sydent's CI until the recent sprint, unlike our other projects. Checking out the problematic version, we can run mypy on the file we know to contain the bug.

$ git checkout v2.4.0
$ mypy --strict sydent/sms/openmarket.py
sydent/sms/openmarket.py:82: error: Dict entry 0 has incompatible type "str": "int"; expected "str": "str"  [dict-item]

Huh. Mypy spots something, but not the error we were hoping for. What's going on here? We can ask mypy to show its working with reveal_type again.

        resp = await self.http_cli.post_json_get_nothing(
            API_BASE_URL, send_body, {"headers": req_headers}
        )
        reveal_type(resp)
        headers = dict(resp.headers.getAllRawHeaders())
        reveal_type(resp.headers)
        reveal_type(resp.headers.getAllwRawHeaders())
        reveal_type(headers)

This yields:

$ mypy sydent/sms/openmarket.py
sydent/sms/openmarket.py:82: error: Dict entry 0 has incompatible type "str": "int"; expected "str": "str"  [dict-item]
sydent/sms/openmarket.py:102: note: Revealed type is "twisted.web.iweb.IResponse*"
sydent/sms/openmarket.py:104: note: Revealed type is "Any"
sydent/sms/openmarket.py:105: note: Revealed type is "Any"
sydent/sms/openmarket.py:106: note: Revealed type is "builtins.dict[Any, Any]"
Found 1 error in 1 file (checked 1 source file)

Ahh, the Any type. As mentioned above, this represents a value whose type can't be statically determined. We're left to runtime checks to detect the problem. But we won't detect it at runtime, because dictionaries don't enforce any kind of type requirements on their keys and values.

The problem here is that mypy can't see that resp.headers is a twisted Headers object. If we could inform it of this, mypy would spot our bug:

        import twisted.web.http_headers
        raw_headers: twisted.web.http_headers.Headers = resp.headers
        reveal_type(resp)
        headers = dict(raw_headers.getAllRawHeaders())
        reveal_type(raw_headers)
        reveal_type(raw_headers.getAllRawHeaders())
        reveal_type(headers)
$ mypy sydent/sms/openmarket.py
sydent/sms/openmarket.py:82: error: Dict entry 0 has incompatible type "str": "int"; expected "str": "str"  [dict-item]
sydent/sms/openmarket.py:104: note: Revealed type is "twisted.web.iweb.IResponse*"
sydent/sms/openmarket.py:106: note: Revealed type is "twisted.web.http_headers.Headers"
sydent/sms/openmarket.py:107: note: Revealed type is "typing.Iterator[Tuple[builtins.bytes, typing.Sequence[builtins.bytes]]]"
sydent/sms/openmarket.py:108: note: Revealed type is "builtins.dict[builtins.bytes*, typing.Sequence*[builtins.bytes]]"
sydent/sms/openmarket.py:114: error: Invalid index type "str" for "Dict[bytes, Sequence[bytes]]"; expected type "bytes"  [index]
sydent/sms/openmarket.py:114: error: Argument 1 to "split" of "bytes" has incompatible type "str"; expected "Optional[bytes]"  [arg-type]
Found 3 errors in 1 file (checked 1 source file)

There it is, on line 114: Invalid index type "str" for "Dict[bytes, Sequence[bytes]]"; expected type "bytes".

Unfortunately it'd be a pain to annotate our application code to mark every use of IResponse.headers as a Headers object. We'll see a better way to do things in this the next post, which discusses the nitty-gritty details of adding annotations file-by-file.


Many thanks for reading! If you've got any corrections, comments or queries, I'm available on Matrix at @dmrobertson:matrix.org.

Call for Participation for the FOSDEM 2022 Matrix Dev Room!

02.12.2021 00:00 — General Thib

A full day of Matrix talks

This year, the Matrix.org Foundation is excited to host the first ever Matrix.org Foundation and Community devroom at FOSDEM. A full day of talks, demos and workshops around Matrix itself and projects built on top of Matrix.

Matrix is the open source project that publishes the Matrix open standard for secure, decentralised, real-time communication, and its Apache licensed reference implementations.

We encourage people working on the Matrix protocol or building on it in an open source project to submit a proposal! Note that companies are welcome to talk about the Matrix details of their open source projects, but marketing talks are not welcome.

We want this devroom to be a space where the Matrix community can show its work, where developers can talk about the challenges they faced and how they overcame them, and where people can get a glimpse of the future of the Matrix protocol and ecosystem.

Talk Details

The talks will be pre-recorded in January. They will be played during FOSDEM, followed by a session of live Q&A depending on the format. During the playback of the talk, people will be able to comment and ask questions in the chat (via Matrix!).

The talks can follow one of three formats:

  • 5 min lightning talk, ideal to showcase your project and make people want to have a look at it
  • 20 min talk + 10 min Q&A, for topics that can be covered briefly
  • 50 min talk + 10 min Q&A for more complex subjects which need more focus

We strongly encourage you to prepare a demo when it makes sense, so people can actually see what your work looks like in practice!

Of course, the proposal must respect the FOSDEM terms as well:

The conference language is English. All content must relate to Free and Open Source Software. By participating in the event you agree to the publication of your recordings, slides and other content provided under the same licence as all FOSDEM content (CC-BY).

Submitting a Proposal

Proposals must be submitted on FOSDEM's conference management system Pentabarf before December 17th 2021. If you are not used to Pentabarf, you can follow this beginners guide to Pentabarf.

We expect to receive more requests than we have slots available. The devroom organisers (two community members and one core team rep) will be reviewing the proposals and accepting them based on the potential positive impact the project has on Matrix (as defined in by the Mission section of https://matrix.org/foundation).

If a project proposal has been turned down, it doesn't mean we don't believe it has good potential. Maintainers are invited to join the #twim:matrix.org Matrix room to give it some visibility.

Synapse 1.48.0 released

30.11.2021 00:00 — Releases Brendan Abolivier

Synapse 1.48.0 is out now!

NOTE: Synapse 1.49, due out on December 14th, will be the last release of Synapse to support Python 3.6 or PostgreSQL 9.6 per our platform dependency deprecation policy. Accordingly, we will remove support for Ubuntu 18.04 LTS (Bionic) at the same date, as it ships with Python 3.6.

Password resets and identity servers

This release removes the long-deprecated trust_identity_server_for_password_resets configuration option. This option was initially deprecated in Synapse 1.4.0 back in October 2019.

Admins of servers still using this configuration option will need to update their Synapse configuration to send password resets through an SMTP server directly rather than relying on identity servers to send them on their behalf.

New admin APIs and improved alignment with Matrix 1.1

This release also introduces a handful of new admin APIs, allowing administrators to un-shadow-ban users, block a room, and run specific background updates (but we'll talk about this last one a bit later on). The delete room API has also been updated to be able to run in the background or to block a room pre-emptively, even if the server doesn't know about it yet.

This release also brings Synapse into greater alignment with version 1.1 of the Matrix specification by adding support for API paths beginning /_matrix/client/v3 and /_matrix/media/v3.

Background updates

When Synapse updates from one version to another, it might need to run large scale updates on its database. In order to avoid blocking startup for too long while waiting for these updates to run, Synapse runs them in the background after starting.

Lately the Synapse team has been doing some work to improve the performance of these background updates. More specifically, this release includes a performance fix for a background update introduced in Synapse 1.47.0, as well as a new admin API to let admins rerun specific updates.

Future Synapse updates will also include module capabilities and more configuration options for controlling background updates.

Everything else

This release also includes some improved support of MSC3440 to help threading. It also adds support for the stable identifiers from MSC2778, bringing Synapse closer to supporting end-to-end (or end-to-bridge) encryption support for application services.

We also now publish a Docker image, matrixdotorg/synapse:develop, which tracks the development head of Synapse.

Please see the Synapse Release Notes for a complete list of changes in this release.

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including Dirk Klimpel, Stanislav Motylkov, Tulir Asokan and Neeeflix.

This Week in Matrix 2021-11-26

26.11.2021 00:00 — This Week in Matrix Thib

Matrix Live 🎙

Dept of Spec 📜

anoa says

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://spec.matrix.org/unstable/proposals.

MSC Status

New MSCs:

MSCs with proposed Final Comment Period:

  • No MSCs entered proposed FCP state this week.

MSCs in Final Comment Period:

  • No MSCs are in FCP.

Merged MSCs:

Closed MSCs:

Spec Updates

MSC2675 (serverside aggregations) is getting lots of updates from Bruno in order to align the proposal with what is currently implemented in the wild (as it's easier to iterate on incremental improvements from a starting point grounded in reality). This MSC is a bit of a special case though, as it was implemented with stable prefixes before the MSC landed (in the before times...).

Regardless, thank you very much to Bruno for going through and finally untangling and help land aggregations in the spec! This MSC is one of four which describe how aggregations should work in Matrix, and it's great to see them finally being properly spec'd, especially as further features start to be built on top of them (such as threading!).

And finally, a further thank you to community members @ankur12-1610, @Dominaezzz for an OpenAPI schema fix!

Random Spec of the Week

The random spec of the week is... MSC3395: Synthetic appservice events!

This proposal aims to allow appservices to get ever greater visibility into what is happening on the homeserver, while still maintaining full process separation.

A lot of the time solutions to complex problems require knowing when a user has registered or logged in, with what name/3pids etc, when users change their emails, etc. Hooking into these non-room-based actions can help with developing useful features. Maybe you want an appservice that plays a sound effect whenever a user signs up to your homeserver!

These days, that is often achieved by homeserver implementation-specific solutions, such as modules in Synapse. Being able to notify of these events using a standard API shape would be hugely beneficial to generalisation of projects.

So give the proposal a look over and review if that interests you!

Some exciting MSCs this week, I've been waiting for server-side aggregations for long!

Dept of Servers 🏢

Synapse

Synapse is the reference homeserver for Matrix

callahad announces

As predicted last week, we released Synapse 1.47.1 on Tuesday. This is a security release which fixes an issue with Synapse's built-in media repository. Admins are strongly encouraged to upgrade.

Otherwise, relatively slow week: a handful of of the team have been away (Happy Thanksgiving, Americans!), but we did release Synapse 1.48.0rc1. Most importantly, this release candidate includes changes to improve the efficiency of large background updates from past releases, which should significantly reduce database load when upgrading. It also adds support for the /v3 APIs defined in version 1.1 of the Matrix specification.

We'll talk more about 1.48 when it's formally released next week, but as always, we appreciate folks trying out the release candidates and letting us know how they behave.

Administrators, keep your users safe: update as soon as you can!

Homeserver Deployment 📥️

Helm Chart

Matrix Kubernetes applications packaged into helm charts

Ananace announces

A bit of an earlier update this week, but I wanted to make sure to note that my Hem Charts have been updated to matrix-synapse 1.47.1 for the security fix - and element-web has also been bumped to 1.9.5

Dept of Bridges 🌉

mautrix-googlechat

tulir reports

mautrix-googlechat has seen lots of improvements over the past few days. New features include:

  • Bridging edits, deletions, reactions, formatting and read receipts in both directions (even /rainbow somewhat works from Matrix)
  • Bridging typing notifications and any types of files from Matrix to Google Chat
  • Bridging Google Meet links from Google Chat to Matrix
  • Syncing group members from Google Chat

(edits and deletions are only available on Google Workspace accounts, not normal accounts. I have no idea why they did that, but that's just how Google Chat works 🤷)

There's still a bug where it sometimes silently stops receiving messages, which I'm currently trying to solve (or work around). After that I'll make a v0.3.0 release. Backfilling history may also happen in the near future

Matrix Webhook Receiver

An add-on for the matrix-appservice-webhooks bridge. Webhooks are essentially web interfaces for applications to "push" data to. The bridge can receive messages in a certain format, which is nice if the notifying app can be configured. Often it cannot.

kim announces

Matrix Webhook Receiver

Do you like to receive notifications in matrix? Matrix Webhook Receiver (MWR) is an add-on for the matrix-appservice-webhooks bridge. Webhooks are essentially web interfaces for applications to "push" data to. The bridge can receive messages in a certain format, which is nice if the notifying app can be configured. Often it cannot.

This is where MWR comes in: It can receive any (JSON) content, optionally reformat it nicely (customizable!), and forward it to the webhooks bridge which will post it to a room for you. If you are running any software service, there is a good chance it can notify you via webhooks!

Right now, several example configurations exist, ready for you to use:

  • GitHub
  • GitLab (including Community Edition/self hosted)
  • Ansible Tower/AWX
  • Grafana Alerts

For example, here is the GitHub webhook as seen in #matrix-webhook-receiver:matrix.org:

More examples of apps currently in development: Prometheus Alertmanager, Jellyfin

It is also easy to use to send messages from the commandline with standard tools (curl), e.g. for your cron jobs! See the README for an example.

Some other webhook matrix things exist, but often require

  • ➖ admin access to server (appservices)
  • ➖ installing and running and maintaining a specific bot per service
  • ➖ create and set up accounts and passwords for each bot to use

In contrast, MWR requires:

  • ➖ matrix-appservice-webhooks set up on your server by your admin
  • ➖ some knowledge of how to send POST requests for one time setup. I want to improve that, suggestions are welcome!
  • ➕ MWR can be installed and run by anyone who can access the bridge
  • ➕ one single MWR supports any amount of notifying apps!
  • ➕ you don't need admin access to the server to add more apps
  • ➕ no need to develop a bot/plugin to support new apps, just write a quick jinja template
  • ➕ multiple people can use a single MWR instance by sharing HTTP basic auth credentials

Other features:

  • can post into encrypted rooms (see readme)
  • automatically generated API docs
  • message formatting presets including html markup or m.notice
  • arbitrarily customizable webhook URLs
  • list and manage currently installed webhooks in your browser (screenshot)

Links:

What a massive update! At this rate I won't need to leave my Matrix client for anything!

matrix-hookshot

A multi purpose multi platform bridge, formerly known as matrix-github

Half-Shot says

Hey folks! Some exciting new news on the bridge front: I've renamed matrix-github to matrix-hookshot to better reflect it's not-just-GitHub-ness. That's not all though, as there are new features too:

  • The bridge now supports Rust as a companion language (we're aiming to rewrite critical sections in rust). Some parts of the formatting code have already been rewritten.
  • The bridge now supports JIRA (full puppeting!)
  • The bridge now supports generic webhooks too, with the ability to write custom handling code inside the state event to process these hooks into pretty messages.
  • Basic support for GitHub discussions.

In the works:

  • A provisioning API to hook into integration managers
  • More GitLab support
  • Better GitHub discussions support

We're not quite ready for a 0.2.0 release, but please check us out at https://github.com/Half-Shot/matrix-hookshot.

An interesting update, and Half-Shot even demoes it in today's Matrix Live!

Dept of Clients 📱

Nheko

Desktop client for Matrix using Qt and C++17.

Nico says

After the 0.9.0 release last week, we have of course been busy fixing all the bugs different people reported. Messing around with the sticker pack editor and then leaving room should not make Nheko crash anymore. The problems where the flatpak has issues starting on Gnome systems are still under investigation. We thought we had a solution, but that seems to have broken other stuff! 💥

Apart from that we have been doing some after release party cleanup. Apart from some refactorings, you can now filter your rooms on whether they are a direct chat or not in the sidebar. This is in addition to the filters we already had for favourites, spaces and your other personal tags. User colors should also now be much less biased towards blue and jdenticons should have more variance. Expect the next release to be a much more colorful experience!

Speaking of colors, Twily made this awesome ZX Spectrum inspired logo after we changed our Gitlab bot to be more colorful! Check it out:

Gorgeous, I love it!

Hydrogen

Hydrogen is a lightweight matrix client with legacy and mobile browser support

Bruno says

We're still distracted with SDK work and other things less visible for users, but this week we've also released 0.2.22 that fixes login on Element One (and other servers using SSO login and not yet supporting the experimental dehydrated devices).

Element

Everything related to Element but not strictly bound to a client

kittykat says

Threads

  • On Web we’ve been working on Notifications and Badges, making sure no message goes unread.
  • On Mobile we’re building out the new Threads Panel so you can easily see all the Threads in a room.

Polls

  • The Polls team is making great progress, focusing their efforts on creating and voting on all platforms.
  • User testing sessions are coming up!

Community Testing

Element Web/Desktop

Secure and independent communication, connected via Matrix. Come talk with us in #element-web:matrix.org!

kittykat announces

  • Work continues on Information Architecture: head over to Sidebar settings on develop.element.io to see what the team are currently working on.

Element iOS

Secure and independent communication for iOS, connected via Matrix. Come talk with us in #element-ios:matrix.org!

kittykat announces

  • We’re still working hard on replacing Matomo with PostHog to improve how we collect analytics data and making the MatrixKit obsolete.
  • Adding more features to Spaces and improvements to Spaces performance is also happening.

Element Android

Secure and independent communication for Android, connected via Matrix. Come talk with us in #element-android:matrix.org!

kittykat announces

  • Voice message drafts and other improvements to this feature are underway.
  • We’re also looking to integrate PostHog on Android.

Dept of SDKs and Frameworks 🧰

simplematrixbotlib

simplematrixbotlib is an easy to use bot library for the Matrix ecosystem written in Python and based on matrix-nio.

krazykirby99999 says

simplematrixbotlib is an easy to use bot library for the Matrix ecosystem written in Python and based on matrix-nio. Version 2.4.0 provides several new features and a fix.

New Features:

  • Newlines are now supported when sending markdown messages.
  • The msgtype of text and markdown messages can now be specified. Text and markdown messages can now optionally be sent as "m.notice" to avoid alerting everybody of the new message. The default msgtype will continue to be "m.text".

New Fixes:

  • Fixed issue where the homeserver was hardcoded in an http request.

Example usage is shown below:

import simplematrixbotlib as botlib

creds = botlib.Creds("https://home.server", "user", "pass")
bot = botlib.Bot(creds)
PREFIX = '!'


@bot.listener.on_message_event
async def echo(room, message):
    match = botlib.MessageMatch(room, message, bot, PREFIX)

    if match.is_not_from_this_bot() and match.prefix() and match.command(
            "echo"):
            
        response = " ".join(arg for arg in match.args())
        await bot.api.send_text_message(room.room_id, response, "m.notice") ## Uses the msgtype of m.notice instead of m.text

bot.run()

A thank you to HarHarLinks for their contributions to version 2.4.0!

Request additional features here.

View source on Github View package on PyPi View docs on readthedocs.io https://matrix.to/#/#simplematrixbotlib:matrix.org

jOlm

Olm bindings for Java

brevilo reports

This week saw three releases of jOlm which fix a native memory management issue, an Olm API (buffer) issue and add a few other improvements. Everyone is strongly encouraged to update to the latest release.

Notes:

  • ✅ Bugfix and maintenance releases
  • ✅ Up to date with Olm 3.2.6

Changelog:

  • Fixed the backing store retention for all Olm instances
  • Fixed a buffer issue in InboundGroupSession.decrypt()
  • Ensured conversions of variable native strings are trimmed
  • Centralized conversions to canonical JSON
  • Completed initial set of unit tests (effectively full coverage now)

Cheers!

The only Java bindings of Olm to my knowledge, that's some very valuable work here!

Dept of Ops 🛠

synadm

Command line admin tool for Synapse (Matrix reference homeserver)

jojo reports

synadm v0.32 is out!

My personal favorites of the new features are:

  • Dates and times in several subcommands are translate the admin API's UNIX epoch timestamps to a human readable format.
  • synadm room list now displays room aliases (#room:your.homeserver)

Read the full release notes here: https://github.com/JOJ0/synadm/releases/tag/v0.32

synadm is very useful as a homeserver administrator. Thanks JOJ0

Dept of Bots 🤖

maubot

A plugin-based Matrix bot system.

tulir reports

maubot v0.2.0 was released last weekend. Highlights:

  • Enabling encryption should be much easier: the device ID can be entered in the web UI or you can just do mbc auth --update-client to automatically log in and store the access token and device ID in maubot.
  • mbc auth can now log in with SSO.
  • The standalone mode for running a single plugin with a static config is now mostly functional and somewhat documented.

Also, I finally took a day to figure out Sphinx/autodoc and made some decent-looking autogenerated docs for mautrix-python. I'll probably extend that to generate maubot-specific API references too eventually.

Dept of Interesting Projects 🛰️

MinesTRIX

A privacy focused social media based on MATRIX

Henri Carnot announces

Quick update on MinesTRIX (a privacy focused social media based on MATRIX). This week was focused on performance and stability.

  • Changed database to use Fluffybox, this should greatly improve performances on web (thanks Famedly !)
  • Scrolling through the posts of a profile now properly request history.
  • Friend suggestions are now sorted according to the sum of user appearance in all rooms. Naïve, but it's the first step.
  • Chat page has also been redesigned. Now support replies and reactions. Chat settings now display room avatar and fetch user list from server.
  • Bug affecting MinesTRIX profile creation has been fixed. Login process should be way more stable now.
  • Minestrix rooms sync has been rewritten to take into account sync events to rebuild the list.
  • Debug page now allow forcing sorting rooms.
  • Various post display enhancement (links are now clickable, thanks kellya!)

Come chat with us : #minestrix:carnot.cc

That's one exciting project, I can't wait to see how far it's going!

Sign in with Matrix

Mish says

Federated sign-in component for your web app (using Matrix)

This week's update:

  • Gained 180 stars on GitHub since release (thanks!)
  • Added login states, accessible from the API
  • Sign out
  • Added CSS styling via variables
  • Updated demo

more on https://github.com/mishushakov/signin-with-matrix

As last week, a note to keep in mind that this is a community project and that there is a MSC to make Matrix more OAuth2 friendly. More on that very soon!

Dept of Built on Matrix 🏗️

Matrix Forms

Mish reports

First release of "Matrix Forms", a project which redirects form submissions to designated Matrix rooms

Features:

  • Server-side, no additional JavaScript
  • Rich formatting
  • Many forms on same instance
  • File uploads
  • Templates
  • CORS
  • Metadata accessible for bots

Can be installed using NodeJS or Docker

  • demo: https://mishushakov.github.io/matrix-forms
  • code: https://github.com/mishushakov/matrix-forms

Final Thoughts 💭

Server_Stats

MTRNord says

Today is a good day for those calling me a spy, someone not wanting to care about privacy and for those who did publicly harass me for server_stats.

People using the API likely already noticed it wasn't reachable for a while. Effective immediately I am currently leaving all rooms the bot is part of. This will take days or even months considering this are 6397 rooms at the time of writing. I am not going into the motives of why I am shutting it down. It comes down to personal reasons.

There won't be any dump of the data. The source will be kept public. Note though if anyone ever tries to run it that you need about 600GB of space for synapse, a lot of CPU, a lot of RAM and plenty of workers as this can easily crash synapse.

Server_Stats was an incredibly useful project. It pains me a lot to see it go, but it pains me even further that its author got harassed. This is not an acceptable behaviour, and we are better than that as a community. Thanks for this incredible project MTRNord, it's been both exciting and useful.

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server.

#ping:maunium.net

Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1boba.best413
2kapsi.fi508.5
3maescool.be528
4envs.net540.5
5converser.eu547
6matrix.org577
7thesilentlink.org838
8matrix.markshorten.co.uk1068
9aria-net.org1217
10trygve.me2137

#ping-no-synapse:maunium.net

Join #ping-no-synapse:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1dendrite.neilalexander.dev430.5
2grin.hu555.5
3matrix.org1027
4dendrite.matrix.org1056
5matrix.awesomesheep48.me1191
6sspaeth.de1251.5
70x1a8510f2.space3350.5
8spooks.cyou10963.5

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!