Matrix logo

Security release of matrix-appservice-irc 0.35.0 (High severity)

13.09.2022 16:56 — Releases — Denis Kasak

We've released a new version of matrix.org's node-irc 1.3.0 and matrix-appservice-irc 0.35.0, to patch several security issues:

The details of the final vulnerability will be released at a later date, pending an audit of the codebase to ensure it's not affected by other similar vulnerabilities.

The vulnerabilities have been patched in node-irc version 1.3.0 and matrix-appservice-irc 0.35.0. You can get the release on Github.

The bridges running on the Libera Chat, OFTC and other networks bridged by the Matrix.org Foundation have been patched.

Please upgrade your IRC bridge as soon as possible.

The above vulnerabilities were reported by Val Lorentz. Thank you!

Security releases: matrix-js-sdk 19.4.0 and matrix-react-sdk 3.53.0

31.08.2022 18:13 — Releases — Denis Kasak

Today we are issuing security releases of matrix-js-sdk and matrix-react-sdk to patch a couple of High severity vulnerabilities (reserved as CVE-2022-36059 for the matrix-js-sdk and CVE-2022-36060 for the matrix-react-sdk).

Affected clients include those which depend on the affected libraries, such as Element Web/Desktop and Cinny. Releases of the affected clients will follow shortly. We advise users of those clients to upgrade at their earliest convenience.

The vulnerabilities give an adversary who you share a room with the ability to carry out a denial-of-service attack against the affected clients, making it not show all of a user's rooms or spaces and/or causing minor temporary corruption.

The full vulnerability details will be disclosed at a later date, to give people time to upgrade and us to perform a more thorough audit of the codebase.

Note that while the vulnerability was to our knowledge never exploited maliciously, some unintentional public testing has left some people affected by the bug. We made a best effort to sanitize this to stop the breakage. If you are affected, you may still need to clear the cache and reload your Matrix client for it to take effect.

We thank Val Lorentz who discovered and reported the vulnerability over the weekend.

This Week in Matrix 2022-08-26

26.08.2022 19:03 — This Week in Matrix — Brendan Abolivier
Last update: 26.08.2022 18:43

Happy TWIMday everyone! Thib is away this week again, so I'm covering for him as your host in this edition of This Week In Matrix.

Matrix Live 🎙

Following up on last week's tutorial about using Docker Compose to install Synapse, this week Thib explains how to use Ansible to deploy your own Matrix homeserver.

Dept of Status of Matrix 🌡️

TravisR reports

Earlier in the year, t2bot.io passed 1 Million known rooms and now it's passed 10 Million bridged users (10,039,915 users to be exact, at time of writing). Most of these users will be people who have participated in a channel/chat on Discord or Telegram that was bridged to Matrix through t2bot.io's free service, with about 500 thousand being active each month.

Approximately 8 Million of the users are from Telegram, covering about 11% of all Telegram users (previously 15% based on information available at the time). The remaining 2 Million are Discord users, roughly 0.5% of Discord's user base. For perspective, t2bot.io has just over 683 Million events in the database and is bridging between 30 and 40 thousand people a day.

Like last time, this is just a milestone update, though it's also a good reminder to host your own server if you can. Element's own hosting platform is a great option if you'd like to have a server without running it yourself, and Beeper offers a richer bridging experience than t2bot.io can feasibly provide. If you'd like to go down the self-hosting route, check out Thib's video guide on hosting synapse or last week's Matrix Live for a better understanding of what hosting Synapse actually means.

As for an interesting statistic: despite not having much functionality that deals with Spaces, t2bot.io can see 2687 Spaces from the wider world. The plan in the coming months is to support a way to bridge a whole Discord server to a Matrix Space, making this statistic hopefully more interesting as time goes on.

Dept of Spec 📜

uhoreg reports

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.

MSC Status

Merged MSCs:

  • No MSCs were merged this week.

MSCs in Final Comment Period:

  • No MSCs are in FCP.

New MSCs:

Spec Core Team

The Spec Core Team has been continuing to push forward on the spec. Several new MSCs have been opened recently. The Spec Core Team is available in #sct-office:matrix.org when MSC authors think that they are ready for primetime.

Random MSC of the Week

The random MSC of the week is... MSC2162: Signaling Errors at Bridges!

Bridges sometimes are unable to relay messages to the remote service for one reason or another. This MSC proposes a way to allow bridges to indicate that a message failed to be delivered, and allow users to tell the bridge to retry.

Dept of Outreachy 🎓️

andybalaam announces

Usman's internship, working on Favourite Messages, is coming to an end! Check out Usman's blog post and Andy's blog post! To follow progress on Favourite Messages (which is still very much a prototype), check out the tracking issue: Tracking issue for Favourite Messages. Thanks to Usman for being an awesome mentee!

Dept of Servers 🏢

Synapse (website)

Synapse is a Matrix homeserver implementation developed by the matrix.org core team

Brendan Abolivier says

This week we've released Synapse 1.66.0rc1! This upcoming release deprecates delegating email validation to an identity server (more info here) and includes improved validation around user-interactive authentication, support for a couple of experimental features, as well as the usual batch of bug fixes and performance improvements 🙂

As always, any help with testing and feedback on this RC is appreciated! Feel free to drop any feedback or bug report in #synapse:matrix.org and the Synapse repo respectively.

Dendrite (website)

Second generation Matrix homeserver

neilalexander announces

This week we released Dendrite 0.9.5 which includes a number of fixes, particularly for federation:

  • The roomserver will now correctly unreject previously rejected events if necessary when reprocessing
  • The handling of event soft-failure has been improved on the roomserver input by no longer applying rejection rules and still calculating state before the event if possible
  • The federation /state and /state_ids endpoints should now return the correct error code when the state isn't known instead of returning a HTTP 500
  • The federation /event should now return outlier events correctly instead of returning a HTTP 500
  • A bug in the federation backoff allowing zero intervals has been corrected
  • The create-account utility will no longer error if the homeserver URL ends in a trailing slash
  • A regression in /sync introduced in 0.9.4 should be fixed

As always, please feel free to join us in #dendrite:matrix.org for more Dendrite-related discussion.

Dept of Ops 🛠

matrix-docker-ansible-deploy (website)

Matrix server setup using Ansible and Docker

Slavi reports

Thanks to Aine of etke.cc, matrix-docker-ansible-deploy can now set up the new Postmoogle email bridge/bot. Postmoogle is like the email2matrix bridge (also already supported by the playbook), but more capable and with the intention to soon support sending emails, not just receiving.

See our Setting up Postmoogle email bridging documentation to get started.

Dept of Bridges 🌉

Aine reports

follow-up to Slavi's announcement: Postmoogle is here!

Actually, he explained it pretty good, so here are some additional links

Source code and Roadmap with implemented and planned features and as usual, say hi in the #postmoogle:etke.cc

Dept of Clients 📱

Quadrix (website)

A Minimal, simple, multi-platform chat client for the Matrix protocol.

JFA says

Quadrix v1.2.5 has been released! The update is already available for Linux, MacOS and iOS. The Windows and Android updates are awaiting approval from the respective stores. This release has mostly "under the hood" improvements (upgrade to React Native 0.69, React 18 and other key dependencies), but also fixes a few bugs and brings minor UI improvements.

Great news: Quadrix finally made it to https://matrix.org/clients/ :-) Many thanks to @madlittlemods:matrix.org!!!

Please leave feedback/comments at #quadrix:matrix.org or in the issues at https://github.com/alariej/quadrix (stars welcome :-))

Element Web/Desktop (website)

Secure and independent communication, connected via Matrix. Come talk with us in #element-web:matrix.org!

kittykat says

In labs (you can enable labs features in settings on develop.element.io or on Nightly):

  • We’re working hard on updating Threads, squashing bugs and improving performance. We have several MSCs open introducing new functionality to read receipts so that notifications work better than ever.

Element iOS (website)

Secure and independent communication for iOS, connected via Matrix. Come talk with us in #element-ios:matrix.org!

Ștefan reports

  • We’re working hard on making the new layout ready for general use, squashing bugs and taking names until everything is in tip top shape. We have a test flight build out: we’ve delayed the release to next week while we iron out the last creases.
  • In ElementX land we have started on adding analytics and Xcode Cloud support and have updated our logging strategy. We will also start adopting sliding sync and using the new Rust Timeline providers

Dept of VoIP 🤙

Element Call (website)

Native Decentralised End-to-end Encrypted Group Calls in Matrix, as a standalone web app

Robin announces

Element Call v0.2.7 and v0.2.8 have been released this past week, adding local volume control, full screen mode, audio in screen sharing and, ahem, fixing an embarrassing bug where we broke walkie-talkie mode... 🐑 Oh, and it's also all in TypeScript now. 🚀 https://github.com/vector-im/element-call/releases/tag/v0.2.7

Dept of SDKs and Frameworks 🧰

simplematrixbotlib (website)

simplematrixbotlib is an easy to use bot library for the Matrix ecosystem written in Python and based on matrix-nio.

HarHarLinks says

simplematrixbotlib has reached version 2.7.0, adding support for end-to-end encryption! 🎉 Come chat over at #simplematrixbotlib:matrix.org!

Here is a summary of things that have happened since we last announced v2.6.3 on this channel:

  • 🌐 The repo canonically moved to https://github.com/i10b/simplematrixbotlib, but the PyPI package remains available in the usual place.
  • 🔒️ E2EE support! To enable it, simply install the optional e2ee dependencies. Find out how in the manual.
  • 😄 Emoji verification support! Enable the option and you'll be able to interactively verify between the bot and your devices. (But mind that for now, in-room verification is not supported, but only to-device).
  • ☝️ Fingerprint verification support! As an additional method, the bot will print it's encryption fingerprint so you can "manually verify".
  • 🗄️ Extensible config file! It is now easier than ever to add your own configuration options to the built-in TOML config file.
  • 🧹 The usual housekeeping, bumping matrix-nio to 0.19.0.
  • 🗨️ I (HarHarLinks) will be presenting the library this weekend at #matrix-summit-berlin-2022:c-base.org! If you are in the Berlin 🇩🇪 area, come visit c-base!

The easiest to use bot library for Matrix. Get started in just 10 lines of code:

import simplematrixbotlib as botlib

config = botlib.Config()
config.emoji_verify = True
creds = botlib.Creds("https://home.server", "user", "pass")
bot = botlib.Bot(creds, config)

@bot.listener.on_message_event
async def echo(room, message):
    if botlib.MessageMatch(room, message, bot, PREFIX).is_not_from_this_bot():
        await bot.api.send_text_message(room.room_id, message.body)

bot.run()

matrix-rust-sdk (website)

Matrix Client-Server SDK for Rust

ben announces

With a few people out of office, this weeks has been one of the more quiet ones, but progress has been made non-the-less. Again a lot happens in draft PRs and the background, like with the upcoming Timeline API but also the path forward for integrating the crypto bindings into the js-sdk. There are a few notable PRs merged this week still improving the API (#972 and #973, #961), upgrading to latest ruma, removing dependencies (parking_lot) to improve compile times as well as merging the release infrastructure for crypto-js.

👉 Wanna hack on matrix rust? Go check out our help wanted tagged issues and join our matrix channel at Matrix Rust SDK.

Dept of Bots 🤖

Alertbot (website)

moanos [he/him] announces

This new bot allows users to use webhooks to forward monitoring alerts (e.g from prometheus) to matrix rooms. This means that you no longer have to use E-Mail or Slack to receive alerts. To set it up visit Github Alertmanager or join #alertbot:hyteck.de

Opsdroid (website)

An open source chat-ops bot framework

Oleg says

Though this release doesn't include Matrix-related changes. Still there are new feature and fixes worth mentioning:

Thanks for all the contributions! 🙌 See the full changelog for details.

Dept of Events and Talks 🗣️

HarHarLinks says

Greetings to the world from #matrix-summit-berlin-2022:c-base.org!

Dept of Interesting Projects 🛰️

Array in a Matrix reports

Matrix AI that generates messages based off other users' messages using a neural network. The bot trains its GPT-2 model using the CPU and is written in JavaScript (Node.JS) and Python. The project's code can be found here.

MinesTRIX (website)

A privacy focused social media based on MATRIX

Henri Carnot reports

Hi all, quite a lot happened since the last twim post a few months ago.

In a nutshell, we refactored the feed page and user page for a better viewing experience. We also now allow displaying and commenting post images in a dedicated view. Also, you can now send follow request using knocking, thanks to profile as space support. (Yes, MSC is coming)

Finally, we have now multi account support, better stories display and refactored login and settings page.

Well... we almost modified everything :D

See more at https://minestrix.henri2h.fr/posts/

Stay tuned, event organization is coming soon (you can see the first implementation in the blog post.

PS: For those at the Matrix summit, I will be presenting it tomorrow

Dept of Guides 🧭

Nate Covington reports

I recently made a blog post / video walk through of Matrix, hopefully it will be helpful to someone: https://www.covingtoncreations.com/blog/what-can-matrix-do-for-your-organization

Room of the Week 📆

ssorbom ⚡️ says

Have you ever felt lost in the Matrix world? Too many rooms and spaces to manage? Well, back by popular demand (with Timo's blessing), I present, The Room of the Week! Every week we strive to highlight a room or a space that we believe deserves attention for discussing interesting going on across the Matrix Network.

This week on room of the week:

We Are All Tech enthusiasts on The Matrix Network, but do you ever experience Tech burnout? Do you ever wish you could find discussions in The Matrix Universe about things other than Tech? Well, this week we bring you a very technical solution!

Because we are highlighting:

#non-technology:matrix.org

A space where you will find information about everything except technology. Groups are helpfully categorized by Subspace, and feature discussions about everything from musical instruments to beverages. If it isn't about computing, it's there.

If you have a room you wish to see highlighted, join us at: https://matrix.to/#/!bIyiUUnriVoHtYzuPS:fachschaften.org?via=chat.shawnsorbom.net&via=matrix.org&via=fachschaften.org To get your favorite room of the week highlighted.

Dept of Ping 🏓

No ping stats while Thib is away, but you can always join the fun at #ping:maunium.net and #ping-no-synapse:maunium.net!

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

This Week in Matrix 2022-08-19

19.08.2022 21:42 — This Week in Matrix — Brendan Abolivier
Last update: 19.08.2022 18:52

Hey folks, welcome to a new edition of This Week In Matrix! Thib is offline this week and next so I'll be taking over while he's away.

Matrix Live 🎙

This week Thib (is he ever really away?) shows us how to host Synapse with Docker Compose.

Dept of Status of Matrix 🌡️

Matthew shared with us the Matrix Summer Special 2022! Come read all about what's happened in Matrix-land so far this year, and what's coming up next, right here: https://matrix.org/blog/2022/08/15/the-matrix-summer-special-2022

Dept of Spec 📜

Andrew Morgan (anoa) announces

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.

MSC Status

New MSCs:

MSCs in Final Comment Period:

  • No MSCs are in FCP.

Merged MSCs:

  • No MSCs were merged this week.

Closed MSCs:

Spec Updates

This week the Spec Core Team focused on improvements to the spec source itself. richvdh opened a PR for edit events, and yours truly did a small PR to clarify the required state of the response to /_matrix/client/v3/login/.

There's a lot more open issues available for people to tackle however, so feel free to get involved and help out if you have some spare time!

Finally other than the usual rounds of review by the team, I've been working on a spec process document that aims to explain the practical portions of the text found at https://spec.matrix.org/proposals/, but in a easily scannable manner. Look out for a PR with that in the near future.

Random MSC of the Week

The random MSC of the week is... MSC2871: Give widgets an indication of which capabilities they were approved for! What a mouthful!

Historically, widgets have laid outside of the spec and have only been implemented in a small subset of clients - mainly Element. As of recent weeks though, there's now a team at Element backing the feature. So exciting times ahead!

Regardless, let's highlight this MSC! It solves a crucial problem with a simple solution. Widgets can ask the client they're embedded in to do certain things (if granted certain capabilities), and the client, potentially after asking the user for permission, can allow or deny those actions. This MSC adds the machinery for a further step: the client will tell the widget what capabilities they requested were allowed, and which were denied.

I believe this spec is non-contentious, but is blocked on widgets entering the spec as a whole. Regardless, if this particular piece of the puzzle interests you, or you'd like to read all about widgets in general, the see either the MSC above or this widget spec tracking PR: https://github.com/matrix-org/matrix.org/pull/825.

Dept of Servers 🏢

Synapse (website)

Synapse is a Matrix homeserver implementation developed by the matrix.org core team

Shay announces

It's the last days of summer here and we are toiling away at making Synapse faster and leaner! In addition to continued work on faster joins, we released Synapse v1.65.0, with new features such as support for stable prefixes for private read receipts and a new module API method for creating a room (plus some other features!), as well as a host of bugfixes and internal changes to make Synapse faster and more stable. Make sure to check it out!

Dendrite (website)

Second generation Matrix homeserver

neilalexander announces

This week we released Dendrite 0.9.4, containing primarily bug fixes:

  • A bug in the roomserver around handling rejected outliers has been fixed
  • Backfilled events will now use the correct history visibility where possible
  • The device list updater backoff has been fixed, which should reduce the number of outbound HTTP requests and Failed to query device keys for some users log entries for dead servers
  • The /sync endpoint will no longer incorrectly return room entries for retired invites which could cause some rooms to show up in the client "Historical" section
  • The /createRoom endpoint will now correctly populate is_direct in invite membership events, which may help clients to classify direct messages correctly
  • The create-account tool will now log an error if the shared secret is not set in the Dendrite config
  • A couple of minor bugs have been fixed in the membership lazy-loading
  • Queued EDUs in the federation API are now cached properly

As always, please feel free to join us in #dendrite:matrix.org for more Dendrite-related discussion.

Homeserver Deployment 📥️

Helm Chart (website)

Matrix Kubernetes applications packaged into helm charts

Ananace says

This week has seen a quite a few Helm Chart updates; element-web got updated to 1.11.3, matrix-media-repo got a Redis usage fix, and matrix-synapse got updated to 1.65.0

Dept of Clients 📱

Nheko (website)

Desktop client for Matrix using Qt and C++17.

Nico reports

Nheko now has some very dirty hack to render spoilers on desktop platforms. This does not show the reason and not work in mobile mode, doesn't hide it from notifications or from the sidebar. But it is at least something. Similarly we tightened what tags we allow when validating the incoming html again. Also, as a small fix, DMs should now also properly start with encryption enabled when started from a profile and there were a few crash fixes when searching for direct chat partners and closing the window too quickly or when a user uploads a device with invalid keys.

Element Web/Desktop (website)

Secure and independent communication, connected via Matrix. Come talk with us in #element-web:matrix.org!

Danielle announces

  • Bug fixes and final polishing has been taking place for our “Start DM on first message” project. This is where the user receiving a new room invite as a DM will not get the notification until you’ve sent a message.
  • The team is testing embedding Element Call in Element Web, as well as working on other improvements to Video rooms.
  • The new user’s checklist is live in product. It’s our first version so let us know what you think!

In labs (you can enable labs features in settings on develop.element.io or on Nightly):

  • Notifications improvements to Threads are underway. The team has been testing the new MSC and related Proof of Concept (POC) which we think will solve most of the issues with Threads right now.

Element iOS (website)

Secure and independent communication for iOS, connected via Matrix. Come talk with us in #element-ios:matrix.org!

Ștefan announces

  • We did the AppStore review dance and version 1.8.27 is now available. We even got better usage strings out of it.
  • We now have UI integration and performance tests in ElementX. Even more, they’re joined by some really nice Screenshot UI tests. Test all the things!
  • We’ve fixed some small bugs and some not so small ones, coming to an Element close to you early next week. The way things are laid out, you might even see a new feature land 😉
  • The new app layout testing session went well and we are looking for more iOS testers for future sessions. If you’d like to help out in future sessions, please join #element-community-testing:matrix.org

Element Android (website)

Secure and independent communication for Android, connected via Matrix. Come talk with us in #element-android:matrix.org!

Danielle says

  • We had a very successful testing session with the all new app layout. If you’d like to take part in a future session, join us over at #element-community-testing:matrix.org
  • This week we’ve been working on fixing some FTUE crashes and covering some edge cases. Thanks to the community members submitting bugs and more info - keep it coming!
  • We’re investigating reports of missing messages, as well as a bug with the Threads beta where not all 'threaded messages' are showing up in the right place…

Dept of SDKs and Frameworks 🧰

Trixnity (website)

Multiplatform Kotlin SDK for Matrix

Benedict reports

Trixnity got some minor updates with bugfixes and iOS ARM64 Simulator support.

ruby-matrix-sdk (website)

Ruby SDK for the Matrix communication protocol

Ananace reports

Just pushed another version (2.8.0) of the Ruby Matrix SDK, which drops support for the EoL Ruby 2.6 (and drops a bunch of workarounds for it) in order to support much improved caching of room state data, along with more helper methods and a fix for a floating accessor that didn't get hooked up correctly.

matrix-rust-sdk (website)

Matrix Client-Server SDK for Rust

ben says

The Matrix Rust SDK team has been busy this week, too. Progress has been made on Siding Sync in particular, the types have been finalised and merged into mainline ruma, and more API has been made accessible via FFI. The new reactive Timeline API has been progressed, it, too, has some FFI definitions now, allowing mobile client to start playing with it. Crypto-JS, too, has progressed, adding support for de/encrypting attachments in an memory efficient fashion. On the crypto-side, the longer on-going refactor and improvements have yielded another few PRs, too, that have successfully merged, while others are still pending reviews. Other than that, we've seen quite a few smaller fixes and improvements, around logging, docs and the examples.

👉️ Wanna hack on matrix rust? Go check out our help wanted tagged issues and join our matrix channel at Matrix Rust SDK.

Matrix Dart SDK (website)

Matrix SDK written in pure Dart.

Nico announces

This week in the dart SDK we mostly fixed bugs. Many of those are related to call negotiations where streams were closed in the wrong order, not closed at all or in group calls the call never learned about new members or the call_ids would not match. There were also a few fixes to the background thumbnailing support added in 0.11.1, helper methods were added to easily send the right message corresponding to the mimetype of some media and fetching a timeline for some event id should work properly again.

We have some support to mark a room as either a dm or a group using slashcommands now, you have more flexibility when implementing the SSSS Bootstrap now (using the extra Bootstrap parameter in onUpdate() and to round it all off, we now have nice coverage numbers as well as coverage display on merge request diffs.

For more info, check the release notes for 0.11.2, 0.12.0, 0.12.1 and 0.12.2: https://pub.dev/packages/matrix/changelog ;-)

Dept of Events and Talks 🗣️

ChristianP says

Six days until Matrix Community Summit Berlin 2022

In less than a week the Matrix Community Summit Berlin is taking place at c-base. Join us early on Thursday (25th August) for a Barcamp where we will brainstorm, draft and prototype new ideas. The main conference days are Friday and Saturday (26th and 27th August). We have a schedule all about Matrix hosting, clients and development. With three simultaneous tracks there sure is something for you to listen to. It's also the perfect place to get to know other community members. Look forward to talks, workshops, a signing party, a Matrix P2P live test, dinner and barbecue!

We're not planing to stream or record the event. Our focus lies on providing a great in-person activities. If attendees want to blog or toot/tweet about it, please use the hashtag #MatrixCommunitySummit. Also, German-speaking folks can look forward to more coverage from the event on my podcast. https://chrpaul.de/podcast/

Because we've heard about some confusion: The event is NOT organised by the Matrix Foundation. To minimize the misconception, we've renamed it to the Matrix Community Summit Berlin. This is an event initiated by Yan, jaller94 and other Matrix enthusiasts. We also organise the Matrix Meetup Berlin.

Matrix Space: #matrix-summit-berlin-2022:c-base.org

Schedule: https://cfp.summit2022.matrixmeetup.de/matrix-summit-conference-2022/schedule/

cel says

One week until DWeb Camp

Next week (August 24-28), people will meet outdoors in California to learn and share about decentralized web technologies.

Some attendees and organizers are using Matrix internally for camp chat.

Public chat: https://matrix.to/#/#decentralizedweb-general:matrix.org

Site: https://dwebcamp.org/

Schedule (in progress): https://dwebcamp2022.sched.com

More info: https://gitlab.com/getdweb/dweb-camp-2022/#dweb-camp-2022

Previously announced: https://matrix.org/blog/2022/04/14/this-week-in-matrix-2022-04-14#dweb-camp

Room of the Week 📆

ssorbom ⚡️ announces

Have you ever felt lost in the Matrix world? Too many rooms and spaces to manage? Well, back by popular demand (with Timo's blessing), I present, The Room of the Week! Every week we strive to highlight a room or a space that we believe deserves attention for discussing interesting going on across the Matrix Network.

Last week, we were serving coffee, this week, it's tea! Specifically, tea at

#tea:matrix.org

Do you prefer sheng or shu puerh? Maybe bit more into lapsang, liuan or perhaps cliff oolongs? Greens or whites? Sencha? Having mood to discuss impact of varios clays and pot shapes? Is it better warm up water in bofura, tetsubin or electric kettle? Simply everything about the tea.

Come join us while the kettle is whistling.

If you have a room you wish to see highlighted, join us at: https://matrix.to/#/!bIyiUUnriVoHtYzuPS:fachschaften.org?via=chat.shawnsorbom.net&via=matrix.org&via=fachschaften.org To get your favorite room of the week highlighted.

Dept of Ping 🏓

No ping stats while Thib is away, but you can always join the fun at #ping:maunium.net and #ping-no-synapse:maunium.net!

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Synapse 1.65 released

17.08.2022 15:44 — Releases — Brendan Abolivier
Last update: 17.08.2022 15:25

Hey everyone! We've just released Synapse 1.65! Let's have a peek at what's inside.

Private read receipts

A feature that the more privacy-focused users of Matrix have been missing was the ability to hide read receipts from other users. Read receipts in rooms can tell a user which messages another user has read in a room. However, they can also be an unwelcome indicator that a user is currently reading a certain room, thus giving away the user's activity on Matrix at a given time.

Hiding one's read receipts from other Matrix users is unfortunately not as straightforward as simply preventing a client from sharing read receipts with the server. This is because read receipts are also used by Matrix homeservers to calculate how much of a room a user has read, and generate notification counts for rooms accordingly.

Synapse 1.65 introduces stable support for private read receipts. This feature, described by MSC2285, allows clients to send a different type of read receipt to the server. This then tells the homeserver to use this piece of information to update the user's notification counts, but not to share it with other users.

Improved room management APIs for modules

This version of Synapse includes two new module API methods to help Synapse modules interact and manage rooms. The first one, lookup_room_alias, allows modules to retrieve the room ID corresponding to a given room alias. This works both for local and remote aliases. The second one, create_room, allows modules to create new rooms on behalf of an existing user.

The update_room_membership method has also been updated in this release of Synapse to allow modules to join a room the server is not already in via federation. This can be done by using the new remote_room_hosts argument, which takes a list of homeservers to try to join via.

Everything else

Synapse 1.65 stabilises the implementation of MSC3827, which allows filtering public room searches on room types. This means it is now possible to search specifically for public spaces. For more information on this feature, see the Synapse 1.63 announcement.

Additionally, Synapse 1.65 implements the new experimental error codes documented by MSC3848. Once stabilised, these error codes will allow clients to show more specific errors to their users about why an event could not be sent.

See the full changelog for a complete list of changes in this release.

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including (in no particular order) Beeper, andrewdoh, Julian-Samuel Gebühr and Dirk Klimpel, as well as anyone helping us make Synapse better by sharing their feedback and reporting issues.

The Matrix Summer Special 2022

15.08.2022 00:00 — General — Matthew Hodgson

Hi all,

At the end of each year it’s been traditional to do a big review of everything that the Matrix core team got up to that year, and announcing our predictions for the next. You can see the last edition in 2021 here - and if you’re feeling nostalgic you can head down memory lane with the 2020, 2019, 2018 ones etc too.

This year is turning out to be slightly different, however. Our plans for 2022 are particularly ambitious: to force a step change in improving Matrix’s performance and usability so that we firmly transition from our historical “make it work” and “make it work right” phases into “making it fast”. Specifically: to succeed, Matrix has to succeed in powering apps which punch their weight in terms of performance and usability against the proprietary centralised alternatives of WhatsApp, Discord, Slack and friends.

We’ve seen an absolute tonne of work happening on this so far this year… and somehow the end results all seem to be taking concrete shape at roughly the same time, despite summer traditionally being the quietest point of the year. The progress is super exciting and we don’t want to wait until things are ready to enthuse about them, and so we thought it’d be fun to do a spontaneous Summer Special gala blog post so that everyone can follow along and see how things are going!

Making it fast

We have always focused on first making Matrix “work right” before we make it “work fast” - sometimes to a fault. After all: the longer you build on a given architecture the harder it becomes to swap it out down the line, and the core architecture of Matrix has remained essentially the same since we began in 2014 - frankly it’s amazing that the initial design has lasted for as long as it has.

Over the years we’ve done a lot of optimisation work on the core team implementations of that original architecture - whether that’s Synapse or matrix-{js,react,ios,android}-sdk and friends: for instance Synapse uses 5-10x less RAM than it used to (my personal federated server is only using 145MB of RAM atm! 🤯) and it continues to speed up in pretty much every new release (this PR looks to give a 1000x speedup on calculating push notification actions, for instance!). However, there are some places where Matrix’s architecture itself ends up being an embarrassingly slow bottleneck: most notably when rapidly syncing data to clients, and when joining rooms for the first time over federation. We’re addressing these as follows…

Sliding Sync (aka Sync v3)

Historically, /sync always assumed that the client would typically want to know about all the conversations its user is in - much as an IRC client or XMPP client is aware of all your current conversations. This provided some nice properties - such as automatically enabling perfect offline support, simplifying client and server development, and making features like “jump to room” and “tab complete” work instantly given the data is all client-side. In the early days of Matrix, when nobody was yet a power user, this wasn’t really a problem - but as users join more conversations and join bigger rooms, it’s become one of Matrix’s biggest performance bottlenecks. In practice, logging into a large account (~4000 rooms) can take ~10 minutes and hundreds of megabytes of network traffic, which is clearly ridiculous. Worse: if you go offline for a day or so, the incremental sync to catch back up can take minutes to calculate (and can even end up being worse than an initial sync).

To fix this, we started work on Sliding Sync (MSC3575) in 2021: a complete reimagining of the /sync API used by Matrix clients to receive data from their homeserver. In Sliding Sync, we only send the client the data it needs to render its UI. Most importantly, we only tell it about the subset of rooms which it is visible in the scroll window of its room list (or that it needs to display notifications about). As the user scrolls around the room list, they slide the window up and down - hence the name “sliding sync”. Sliding Sync was originally called Sync v3, given it’s our 3rd iteration of the sync API - it got renamed Sliding Sync given the current sync API confusingly ended up with a prefix of /v3.

Back in December our work on Sliding Sync was still pretty early: we had the initial MSC, an experimental proxy that converted the existing sync v2 API into Sliding Sync, and a simple proof-of-concept web client to exercise it. Since then, however, there has been spectacular progress:

  • MSC3575 has undergone some big iterations as we converge on the optimal API shape.
  • The sliding-sync proxy has matured to be something which we’re now running in stealth against matrix.org for those dogfooding the API
  • We added the concept of extensions to split out how to sync particular classes of data (to avoid the API becoming a monolithic monster) - specifically:
    • Account Data
    • End-to-end Encryption
    • To-device messages
    • Ephemeral events (to be done)
    • Presence (to be done)
  • We added support for spaces!
  • We implemented it in matrix-js-sdk (which merged a few weeks ago!)
  • …and have a WIP implementation in matrix-rust-sdk too.

But most importantly, we’ve also been busy implementing Sliding Sync in Element Web itself so we can start using it for real. Now, this is still a work in progress, but as of today it’s just getting to the point where one can experiment with it as a daily driver (although it’s definitely alpha and we’re still squishing bugs like crazy!) - and we can see just how well it really performs in real life.

For instance, here’s a video of my account (4055 rooms, redacted for privacy) logging in on an entirely fresh browser via Sliding Sync - the actual sync takes roughly 1 second (at 00:18 in the video). And if we’d started the sync operation while the user is setting up E2E encryption, it would have completed in the background before they even got to the main screen, giving instant login(!). Given my account typically takes ~10 minutes to initial sync (plus even more time for encryption to sync up), this is at least a real-life 600x improvement. Moreover, the sync response is only 20KB (a ~5000x improvement) - a huge win for low-bandwidth Matrix situations.

Then, having logged in, the client subsequently launches pretty much instantly, no matter how long you’ve been offline. Total launch time is roughly 4 seconds, most of which is loading the app’s assets - which in turn could well be improved by progressively loading the app. It could also be sped up even more if we cached state locally - currently the implementation simply reloads from the server every time the app launches rather than maintaining a local cache.

As you can see, this is palpably coming together, but there’s still a bunch of work to be done before we can encourage folks to try it, including:

  • Switching the RoomList to be fully backed by sliding sync (currently the v2 roomlist is jury-rigged up to the sliding sync API, causing some flakey bugs such as duplicate rooms)
  • Spec and hook up typing / receipts / presence extensions
  • Hook up favourites, low_priority, knocked and historical rooms
  • Adding back in loading room members
  • Apply quality-of-service to to-device messages so we prioritise ones relevant to the current sliding window
  • Sync encrypted rooms in the background to search for notifications (and for indexing).
  • More local caching to speed up operations which now require checking the server (e.g. Ctrl/Cmd-K room switching)

We also need to determine whether it’s viable to run the sliding-sync proxy against matrix.org for general production use, or whether we’ll need native support in Synapse before we can turn it on by default for everyone. But these are good problems to have!!

matrix-rust-sdk and Element X

Meanwhile, over in the land of Rust, we’ve been making huge progress in maturing and stabilising matrix-rust-sdk and exercising it in Element X: the codename for the next generation of native Element mobile apps. Most excitingly, we literally just got the first (very alpha) cut of Sliding Sync working in matrix-rust-sdk and hooked up to Element X on iOS - you can see Ștefan’s demo from last week here:

matrix-rust-sdk itself is now getting a steady stream of releases - including long-awaited official node bindings, providing excellent and performant encryption support via the newly audited vodozemac Rust implementation of Olm. It’s also great to see loads of major contributions to matrix-rust-sdk from across the wider Matrix community - particularly from Ruma, Fractal, Famedly and others - thank you!! As a result the SDK is shaping up to be much more healthy and heterogeneous than the original matrix-{js,ios,android}-sdk projects.

On Element X itself: matrix-rust-sdk is being used first on iOS in Element X iOS - aiming first for launching a stable “barbecue” feature set (i.e. personal messaging) asap, followed by adding on “banquet” features (i.e. team collaboration) such as spaces and threads afterwards. We’ve shamelessly misappropriated the barbecue & banquet terminology from Tobias Bernard’s excellent blog post “Banquets and Barbecues” - although, ironically, unlike the post, our plan is still to have a single app which incrementally discloses the banquet functionality as the user’s barbecue starts to sprawl. We’ve just published the brand new development roadmap for Element X from the rust-sdk perspective on GitHub. Above all else, the goal of Element X is to be the fastest mobile messenger out there in terms of launch and sync time, thanks to Sliding Sync. Not just for Matrix - but the fastest messenger, full stop :D Watch this space to see how we do!

Finally: Element is getting a major redesign of the core UI on both iOS and Android - both for today’s Element and Element X. I’m not going to spoil the final result (which is looking amazing) given it’ll have a proper glossy launch in a few weeks, but you can get a rough idea based on the earlier design previewed by Amsha back in June:

In addition to the upcoming overall redesign, Element also landed a complete rework of the login and registration flows last week on iOS and Android - you can see all about it over on the Element blog.

Fast Remote Joins

In terms of performance, the other area that we’re reworking at the protocol level is room joins.

One of the most glaring shortcomings of Matrix happens when a new server admin excitedly decides to join the network, installs a homeserver, tries to join a large room like #matrix:matrix.org, and then looks on in horror as it takes 10+ minutes to join the room, promptly despairs of Matrix being slow and complains bitterly about it all over HN and Reddit :)

The reason for the current behaviour is that the Matrix rooms are replicated between the servers who participate in them - and in the initial design of the protocol we made that replication atomic. In other words, a new server joining a room picks a server from which to acquire the room (typically the one in the room’s alias), and gets sent a copy of all the state events (i.e. structural data) about the room, as well as the last 20 or so messages. For a big room like Matrix HQ, this can be massive - right now, there are 79,969 state events in the room - and 126,510 auth_chain events (i.e. the events used to justify the existence of the state events). The reason there are so many is typically that the act of a user joining or parting the room is described by a state event - and in the naive implementation, the server needs to know all current state events in the room (e.g. including parted users), in order to keep in sync with the other servers in the room and faithfully authorise each new event which they receive for that room.

However, each event is typically around 500 bytes in size, and so the act of joining a big room could require generating, transmitting, receiving, authenticating and storing up to 100MB of JSON 😱. This is why joining big rooms for the first time is so painfully slow.

Happily, there is an answer: much as Sliding Sync lets clients synchronise with the bare minimum of data required to render their UI, we’ve created MSC3706 (and its precursor MSC2775) in order to rework the protocol to let servers receive the bare minimum of state needed to join a room in order to participate. Practically speaking, we only really care about events relevant to the users who are currently participating in the room; the 40,000 other lurkers can be incrementally synced in the background so that our membership list is accurate - but it shouldn’t block us from being able to join or read (or peek) the room. We already have membership lazyloading in the client-server API to support incrementally loaded membership data, after all.

The problem with this change is that Synapse was written from the outset to assume that each room’s state should be atomically complete: in other words, room state shouldn’t incrementally load in the background. So the work for Faster Joins has been an exercise in auditing the entirety of Synapse for this assumption, and generally reviewing and hardening the whole state management system. This has been loads of work that has been going on since the end of last year - but the end is in sight: you can see the remaining issues here.

As of right now, faster joins work (although aren’t enabled by default) - with the main proviso that you can’t speak in the room yet until the background sync has completed, and the new implementation has not yet been optimised. However, thanks to all the preparation work, this should be relatively straightforward, so the end is in sight on this one too.

In terms of performance: right now, joining Matrix HQ via the unoptimised implementation of faster joins completes on a fresh server in roughly 30 seconds - so a ~25x improvement over the ~12 minutes we’ve seen previously. However, the really exciting news is that this only requires synchronising 45 state events and 107 auth_chain events to the new server - a ~1400x improvement! So there should be significant scope for further optimising the calculation of these 152 events, given 30 seconds to come up with 152 events is definitely on the high side. In our ideal world, we’d be getting joins down to sub-second performance, no matter how big the room is - once again, watch this space to see how we do.

Finally, alongside faster remote joins, we’re also working on faster local joins. This work overlaps a bit with the optimisation needed to speed up the faster remote join logic - given we are seeing relatively simple operations unexpectedly taking tens of seconds in both instances. Some of this is needing to batch database activity more intelligently, but we also have some unknown pauses which we’re currently tracking down. Profiling is afoot, as well as copious Jaeger and OpenTracing instrumentation - the hunt is on!

Ratcheting up testing

All the work above describes some pretty bold changes to speed up Matrix and improve usability - but in order to land these changes with confidence, avoiding regressions both now and in future, we have really levelled up our testing this year.

Looking at matrix-react-sdk as used by Element Web/Desktop: all PRs made to matrix-js-sdk must now pass 80% unit test coverage for new code (measured using Sonarqube, enforced as a GitHub PR check). All matrix-react-sdk PRs must be accompanied by a mix of unit tests, end-to-end tests (via Cypress) and screenshot tests (via percy.io). All regressions (in both nightly and stable) are retro’d to ensure fixed things stay fixed (usually via writing new tests), and we have converted fully to typescript for full type safety.

Concretely, since May, we’ve increased js-sdk unit test coverage by ~10% globally, increased react-sdk coverage by ~17%, and added ever more Cypress integration tests to cover the broad strokes. Cypress now completely replaces our old Puppeteer-based end-to-end tests, and Sliding Sync work in matrix-react-sdk is being extensively tested by Cypress from the outset (the Sliding Sync PR literally comes with a Cypress test suite).

In mobile land, the situation is more complex given our long-term strategy is to deprecate matrix-ios-sdk and matrix-android-sdk2 in favour of matrix-rust-sdk. matrix-rust-sdk has always had excellent coverage, and in particular, adopting the crypto module in the current matrix-{js,ios,android}-sdk will represent a night and day improvement for quality (not to mention perf!). We’ll also be adopting PR checks, and screenshot testing for the mobile SDKs.

On the backend, we continue to build out test cases for our new integration tester Complement (in Golang), alongside the original sytest integration test suite (in Perl). In particular, we can now test Synapse in worker mode. The intention with Complement is that it should be homeserver agnostic so that any homeserver implementation can benefit. Indeed the project was initiated by Kegan wearing his Dendrite hat.

Finally, we’ve had a huge breakthrough with true multi-client end-to-end testing in the form of Michael Kaye’s brand new Traffic Light project. For the first time, we can fully test things like cross signing and verification and VoIP calls end-to-end across completely different platforms and different clients. It’s early days yet, but this really will be a game changer, especially for crypto and VoIP.

Next up, we will turn our attention to a performance testing framework so that we can reliably track performance improvements and regressions in an automated fashion - heavily inspired by Safari’s Page Load Test approach. This will be essential as we build out new clients like Element X.

A whole new world

All the stuff above is focused on improving the core performance and usability of Matrix - but in parallel we have also been making enormous progress on entirely new features and capabilities. The following isn’t a comprehensive list, but we wanted to highlight a few of the areas where new development is progressing at a terrifying rate…

Native VoIP Conferencing

2022 is turning out to be the year that Matrix finally gets fully native voice/video conferencing. After speccing MSC3401 at the end of last year, Element Call Beta 1 launched as a reference implementation back in March, followed by enabling E2EE, spatial audio and walkie-talkie mode in Element Call Beta 2 in June.

However, the catch was that Element Call beta 1 and 2 only ever implemented “full mesh” conferencing - where every participant calls every other participant simultaneously, limiting the size of the conference to ~7 participants on typical hardware, and wasting lots of bandwidth (given you end up sending the same upstream multiple times over for all the other participants). Element Call has been working surprisingly well in spite of this, but the design of MSC3401 was always to have “foci” (the plural of ‘focus’ - i.e. conference servers) to optionally sit alongside homeservers in order to aggregate the participating calls, a bit like this:

MSC3401 Architecture

With foci, clients only need to send their upstream to their local focus, rather than replicating it across all the other participants - and the focus can then fan it out to other foci or clients as required. In fact, if no other clients are even watching your upstream, then your client can skip sending an upstream to its focus entirely!

Most importantly, foci are decentralised, just like Matrix: there is no single conference server as a single point of control or failure responsible for powering the group call - users connect to whichever focus is closest to them, and so you automatically get a standards-based heterogeneous network-split-resilient geographically distributed cascading conferencing system with end-to-end-encryption, powered by a potentially infinite number of different implementations. To the best of our knowledge, this is the first time someone’s proposed an approach like this for decentralised group calling (watch out, Zoom, we’re coming for you!)

Now, the VoIP team have been busy polishing Element Call (e.g. chasing down end-to-end encryption edge cases and reliability), and also figuring out how to embed it into Element and other Matrix clients as a quick way to get excellent group VoIP (more on that later). As a result, work on building out foci for scalable conferencing had to be pushed down the line.

But in the last few months this completely changed, thanks to an amazing open source contribution from Sean DuBois, project lead over at Pion - the excellent Golang WebRTC implementation. Inspired by our initial talk about MSC3401 at CommCon, Sean independently decided to see how hard it’d be to build a Selective Forwarding Unit (SFU) focus that implemented MSC3401 semantics using Pion - and published it at https://github.com/sean-der/sfu-to-sfu (subsequently donated to github.com/matrix-org). In many ways this was a flag day for Matrix: it’s the first time that a core MSC from the core team has been first implemented from outside the core team (let alone outside the Matrix community!). It’s the VoIP equivalent of Synapse starting off life as a community contribution rather than being written by the core team.

Either way: Sean’s SFU work has opened the floodgates to making native Matrix conferencing actually scale, with Šimon Brandner and I jumping in to implement SFU support in matrix-js-sdk… and as of a few weeks ago we did the first ever SFU-powered Matrix call - which worked impressively well for 12 participants!

12 person Element Call

Now, this isn’t released yet, and there is still work to be done, including:

  • We actually need to select the subset of streams we care about from the focus
  • We need to support thumbnail streams as well as high-res streams
  • We need rate control to ensure clients on bad connections don’t get swamped
  • We need to hook up cascading between foci (although the SFU already supports it!)
  • We need E2EE via insertable streams
  • Faster signalling for switching between streams

You can see the full todo list for basic and future features over on GitHub. However, we’re making good progress thanks to Šimon’s work and Sean’s help - but with any luck beta 3 of Element Call might showcase SFU support!

Meanwhile it’s worth noting that Element Call is not the only MSC3401 implementation out there - the Hydrogen team has added native support to Hydrogen SDK too (skipping over the old 1:1 calling), so expect to see Element <-> Hydrogen calling in the near future. The Hydrogen implementation is also what powers Third Room (see below…)

Matryoshka VoIP Embedding

Elsewhere on VoIP, we’ve also been hard at work figuring out how to embed Element Call into Matrix clients in general, starting with Element Web, iOS & Android. Given MSC3401 is effectively a superset of native 1:1 Matrix VoIP calling, we’d ideally like to replace the current 1:1-only VoIP implementation in Element with an embedded instance of Element Call (not least so we don’t have to maintain it in triplicate over Web/iOS/Android, and because WebRTC-in-a-webview really isn’t very different to native WebRTC). To do this efficiently however, the embedded Element Call needs to share the same underlying Matrix client as the parent Element client (otherwise you end up wasting resources and devices and E2EE overhead between the two). Effectively Element Call ends up needing to parasite off the parent’s client. We call this approach “matryoshka embedding”, given it resembles nested Russian dolls. 🪆

In practice, we do this by extending the Widget API to let Matrix clients within the widget share the parent’s Matrix client for operations such as sending and receiving to-device messages and accessing TURN servers (c.f. MSC3819 and MSC3846). This in turn has been implemented in the matrix-widget-api helper library for widget implementers - and then a few days ago Robin demonstrated the world’s first ever matryoshka embedded Element Call call, looking like this:

Matryoshka embedded Element Call

Note that the MSC3401 events are happening in the actual room where the widget has been added, sent by the right users from Element Web rather than from Element Call, and so are subject to all the normal Matrix access control and encryption semantics. This is a huge step forwards from embedding Jitsi widgets, where the subsequent call membership and signalling happens in an entirely separate system (XMPP via Prosody, ironically) - instead: this is proper native Matrix calling at last.

Moreover, the same trick could be used to efficiently embed other exotic Matrix clients such as Third Room or TheBoard - giving the user the choice either to use the app standalone or within the context of their existing Matrix client. Another approach could be to use OIDC scopes to transparently log the embedded client in using the parent’s identity; this has the advantage of no code changes being needed on the embedded client - but has the disadvantage that you needlessly end up running two Matrix clients for the same account side by side, and adding another device to your account, which isn’t ideal for a performance sensitive app like Element Call or Third Room.

Matryoshka embedding isn’t live yet, but between scalable calls via SFU and native Element Call in Element Web/iOS/Android, the future is looking incredibly exciting for native Matrix VoIP. We hope to finish embedding Element Call in Element Web/iOS/Android in Sept/Oct - and if we get lucky perhaps the SFU will be ready too and then Element Call can exit beta!

Finally, we also added Video Rooms to Element Web - adding the user interface for an “always on” video room that you can hop into whenever you want. You can read about it over on the Element blog - the initial implementation uses Jitsi, but once Element Call and Matryoshka embedding is ready, we’ll switch over to using Element Call instead (and add Voice Rooms too!)

Video Rooms

Third Room

Just as MSC3401 and Element Call natively adds decentralised voice/video conferences to boring old textual Matrix chatrooms, MSC3815 and Third Room go the whole enchilada and adds a full decentralised 3D spatial collaboration environment into your Matrix room - letting you turn your Matrix rooms into a full blown interconnected virtual world.

I can’t overstate how exciting this is: one of the key origins of Matrix was back in Oct 2013 when Amandine and myself found ourselves in Berlin after TechCrunch Disrupt, debating why Second Life hadn’t been more successful - and wondering what you’d have to do to build an immersive 3D social environment which would be as positive and successful as a wildly popular chat network. Our conclusion was that the first key ingredient you’d need would be a kick-ass open decentralised communication protocol to build it on - providing truly open communication primitives that anyone could build on, much like the open web… and that was what got us thinking about how to build Matrix.

Fast forward 9 years, and Third Room is making spectacular progress in building out this dream, thanks to the incredibly hard work of Robert, Nate and Ajay. The goal of Third Room is to be an open platform layered directly on Matrix for spatial collaboration of any kind: effectively a blank canvas to let folks create freeform collaborative 3D (and in future 2D, VR or AR) experiences, either by using existing assets or building their own user-generated content and functionality. Just like the open web itself, this unlocks a literally infinite range of possibilities, but some of the obvious uses include: spatial telepresence, social VR, 3D visualisation of GIS or weather data, 3D simulated environments, search and rescue and disaster response operations (imagine streaming LIDAR from a drone surveying hurricane devastation into Third Room, where you can then overlay and collaborate on GIS data in realtime), and of course 3D gaming in all its various forms.

Now, we’re hoping to give Third Room a proper launch in a few weeks, so I’m not going to spoil too much right now - but the final pieces which are currently coming together include:

  • Finalising the initial version of Manifold, the multi-threaded game engine which powers Third Room (built on Three.JS, bitECS and Rapier), using SharedArrayBuffers as triple-buffers to synchronise between the various threads. See this update for a bit more detail on how the engine works.
  • Finalising the Matrix client interface itself, powered by Hydrogen SDK in order to be as lightweight as possible
  • Adding in full spatial audio and game networking via MSC3401 and Hydrogen SDK (currently full mesh, but will go SFU as soon as SFUs land!)
  • Adding in animated avatars (currently using Mixamo animations)
  • Adding in name tags and object labels
  • Adding in 3D Tile support in order to incrementally load 3D map tiles à la Google Earth
  • Building an asset pipeline from Unity and Blender through to the glTF assets which Third Room uses.
  • Initial framework for an in-world direct-manipulation editor
  • Lightmap support for beautiful high-performance static lighting and shadows
  • Full post-processing pipeline (bloom, depth-of-field, anti-aliasing etc)
  • Integrating with OIDC for login, registration, and account management (see OIDC below)

As a quick teaser - here’s an example of a Unity asset exported into Third Room, showing off lightmaps (check out the light and shadows cast by the strip lighting inside, or the shadow on the ground outside). Ignore the blurry HDR environment map of Venice in the background, which is just there to give the metals something to reflect. Check out the stats on the right-hand side: on Robert’s M1 Macbook Pro we’re getting a solid 60fps at 2000x1244px, with 13.12ms of unused gametime available for every 16.67ms frame, despite already showing a relatively complicated asset!

Meanwhile, here are some shots of Robert and Nate chasing each other around the UK City demo environment (also exported from Unity), showing off blended Mixamo animations and throwing around crates thanks to the Rapier physics engine.


And don't forget, it's just a Matrix client - with no infrastructure required other than a normal Matrix server:

Third Room Overlay

As you can see, we are rapidly approaching the point where we’ll need support from technical artists to help create beautiful scenes and avatars and assets in order to make it all come to life - especially once the Blender and Unity pipelines, and/or the Third Room editor are finished. If you’re interested in getting involved come chat at #thirdroom-dev:matrix.org!

WYSIWYG

Back in the real world, a recent new project that we haven’t spoken about much yet is adding consistent WYSIWYG (What You See Is What You Get) editing to the message composer in matrix-{react,ios,android}-sdk as used by Element Web/iOS/Android - as well as publishing the resulting WYSIWYG editor for the greater glory of the wider ecosystem.

This is a bit of a contentious area, because we’ve tried several times over the years to add a rich text editor to matrix-react-sdk - firstly with the Draft.js implementation by Aviral (which we abandoned after Facebook de-staffed Draft), and then later with a Slate implementation by me (which we abandoned thanks to the maintenance burden of keeping up with Slate’s API changes). Finally, burnt by the experience with third party solutions, Bruno wrote his own editor called CIDER, which was a great success and is what Element Web uses today to author messages including ‘pills’ for structured rooms/users etc… but this deliberately didn’t provide full WYSIWYG functionality. Meanwhile, Slack added WYSIWYG, forced it on, and screwed it up - and apps like WhatsApp and Discord seem to get by fine without WYSIWYG.

However, given that users are now used to WYSIWYG in Teams and Slack, we’ve now decided to have another go at it, inspired by CIDER’s success - and with the novel twist that the heavy lifting of modelling and versioning the document and handling Unicode + CJK voodoo will be provided by a cross-platform native library written in Rust, ensuring that matrix-{react,ios,android}-sdk (and in future matrix-rust-sdk-based apps like Element X) all have precisely the same consistent semantics, and we don’t spend our lives fixing per-platform WYSIWYG bugs unless it really is a platform-specific issue with the user interface provided on that particular platform.

The project is fairly young but developing fast, and lives over at https://github.com/matrix-org/matrix-wysiwyg (better name suggestions welcome ;) - we’re aiming to get it into clients by the end of October. The editor itself is not Matrix specific at all, so it’ll be interesting to see if other projects pick it up at all - and meanwhile, if we’ve done a good job, it’ll be interesting to see if this can be used to power Matrix-backed collaborative-editing solutions in future…

Update: we should have mentioned that the WYSIWYG editor project is being built out by staff at Element, who very kindly have been sponsored to work on it by one of Element's Big Public Sector Customers in order to get to parity with Teams. Thank you!!

Are We OIDC Yet?

On the other hand, a project we recently yelled about a lot is Matrix’s transition to Open ID Connect for standards-based authentication and account management. We announced this at the end of the year and the project has built up huge momentum subsequently, culminating with the release of https://areweoidcyet.com last week to track the progress and remaining work.

Our plan is to use native OIDC in production for the first time to provide all the login, registration and account management for Third Room when it launches in a few weeks (using a branded Keycloak instance as the identity provider, for convenience). After all, the last thing we wanted to do was to waste time building fiddly Matrix-specific login/registration UI in Third Room when we’re about to move to OIDC! This will be an excellent case study to see how it works, and how it feels, and inform the rest of the great OIDC experiment and proposed migration.

Dendrite + P2P

Meanwhile, the Next Generation team has continued to focus on their mission to make Dendrite as efficient and usable as possible. Within recent months, Dendrite has matured dramatically, with a considerable list of bugs fixed, performance significantly improved and new features added - push notifications, history visibility and presence to name a few notable additions.

Neil Alexander, Kegan and Till have continued to streamline the Dendrite architecture and to refactor areas of the codebase which have long needed attention, as well as moving from Kafka to NATS JetStream, an all-new caching model and some other fairly major architectural changes. We’ve also seen an increase of code contributions from the community and outside organisations, which is exciting, and the gomatrixserverlib library which underpins much of Dendrite is also seeing more active development and attention thanks to its use in the Complement integration testing suite.

With the most recent 0.9.3 release, we are proud to announce that Dendrite now passes 90% of Client-Server API tests and 95% of Server-Server API tests and has support for all specced room versions in use today. We have a growing community of users who are (quite successfully) trialling using Dendrite homeservers day-to-day, as well as our own public dendrite.matrix.org homeserver, which is open to the public for registration for anyone who wants to experiment with Dendrite without running their own deployment.

Dendrite plays an important role in our future strategy as it is also the homeserver implementation used for embedded homeservers, P2P development and experimentation. In addition to being able to scale up, we have also successfully scaled down, with the Element P2P demos proving that an embedded Dendrite homeserver can run comfortably on an iOS or Android device.

Research on the Pinecone overlay network for P2P Matrix has also continued, with Devon and Neil having experimented with a number of protocol iterations and spent considerable time bringing the Pinecone Simulator up to scratch to help us to test our designs more rapidly. Our work in this area is helping us to form a better direction and strategy for P2P Matrix as a whole, which is moving more towards a hybridised model with the current Matrix federation — a little different to our original vision, but will hopefully result in a much smoother transition path for existing users whilst solving some potential scaling problems. The arewep2pyet.com site is a living page which contains a high level overview of our goals and all the progress being made.

What’s left?

Comparing all of the above with the predictions for 2022 section of the end-of-year blog post, we’re making very strong progress in a tonne of areas - and the list above isn’t comprehensive. For instance, we haven’t called out all the work that the Trust & Safety team are doing to roll out advanced moderation features by default to all communities - or the work that Eric has been doing to close the remaining gap between Gitter and Matrix by creating new static archives throughout Matrix. Hydrogen has also been beavering away to provide a tiny but perfectly formed web client suitable for embedding, including the new embeddable Hydrogen SDK. We haven’t spoken about the work that the Cryptography team have been doing to adopt vodozemac and matrix-rust-sdk-crypto throughout matrix-{js,ios,android}-sdk, or improve encryption stability and security throughout. We’ve also not spoken about the new initiative to fix long-term chronic bugs (outside of the work above) in general - or all the work being done around Digital Markets Act interoperability…

Other things left on the menu for this year include getting Threads out of beta: we’ve had a bit of an adventure here figuring out how to get the right semantics for notification badges and unread state in rooms with threads (especially if you use a mix of clients which support and don’t support threads), and once that’s done we’ll be returning to Spaces (performance, group permissions etc).

Matrix 2.0?

Looking through this post (and congratulations if you’re still reading it at this point ;P), it really feels that Matrix is on the verge of shifting into a new phase. Much as MacOS X started off as a promising but distinctly unoptimised operating system, and then subsequently got unrecognisably faster year by year (even on the same hardware!) as Apple diligently worked away optimising the kernel… similarly: we are now landing the architectural changes to completely transform how Matrix performs.

Between protocol changes like Sliding Sync, Faster Joins, native OIDC and native VoIP conferencing all landing at roughly the same time - and alongside new implementations like matrix-rust-sdk and vodozemac, let alone Third Room - it feels unquestionably like we have an unrecognisable step change on the horizon. Our aim is to land as much of this as possible by the end of the year, and if we pull it off, I’m tempted to suggest we call the end result Matrix 2.0.

To the future! 🚀

Matthew, Amandine & the whole core team.