Dendrite is entering Beta!

08.10.2020 00:00 — Releases — Matthew Hodgson

Hi all,

We’re very excited to announce that Dendrite, the next-generation Matrix homeserver from the core Matrix team, is at last exiting alpha development and entering beta testing!

The path we’ve taken to get here has been quite a curious one, and it’s worth recapping to give context on why it’s taken reality a little while to catch up with the dream. :)

The Dendrite project has its roots in 2016 as Dendron: an attempt to write a next-generation homeserver in Golang rather than Python, in order to benefit from Go’s stronger typing, ease of profiling (no twisted stack-shredding via deferredInlineCallbacks), multithreading and faster GC performance. The idea for Dendron was to do a strangler pattern rewrite of Synapse - where we’d insert Dendron in front of Synapse as a load balancer, and incrementally replace Synapse’s API endpoints with ones implemented by Dendron.

However, as the project started to progress, it became clear that this was going to end up with many of Synapse’s architectural choices being baked into the project - particularly the DB schema and data flow architecture, such that the new endpoints could interoperate with the existing Python ones. We got as far as putting Dendron live on matrix.org and moving some of the login/registration APIs over to it… but then work fizzled out due to Synapse demanding more urgent attention as traffic grew on Matrix.org, combined with concerns about whether Dendron was the right approach in general.

So, towards the end of 2016 (after the rush to launch ~~Vector~~ ~~Riot~~ Element that summer), we went back to the drawing board to devise Dendrite—“Dendron done right!”—as opposed to Dendron, which in retrospect was Dendrite done wrong. ;) The new vision was:

Build a massively horizontally scalable architecture, such that large Matrix deployments like matrix.org and big government deployments could run smoothly without the constant scalability headaches we were seeing at the time with Synapse
Do so by splitting the server into well-defined microservice components, each of which could independently horizontally scale, each with its own DB (if desired)
Connect the components together with a set of append-only logs via Kafka or similar, easily letting components shard and maintain their databases from the logs, allowing rolling upgrades, possibly schema upgrades, and all sorts of other niceties. The logs effectively become a primary source of truth rather than putting all the onus on a massive monolithic ever-growing database

Rather than Dendron’s top-down approach, instead Dendrite started bottom-up with the very hardest bit: gomatrixserverlib, a standalone Go library implementing the state resolution algorithms and performing federation requests (such that it might also someday be used as a general purpose way to add Matrix federation support to an existing Go codebase).

Then we started building out the various components to implement the various services, starting with the roomserver (the service which models the history and state of one or more rooms in the server), then the syncserver (the service which implements the /sync API to let clients receive messages), etc. We even implemented a simplified in-memory version of Kafka named naffka—useful for glueing together the microservice components when running them all within a single binary.

Things were looking pretty positive by the summer of 2017: we had the server sending/receiving messages, federating with Synapse, and looking tantalisingly close:

We just sent the first ever synapse->dendrite federated traffic, including full dendrite media API (thumbnailing, fed, etc)!!! :D :D :D pic.twitter.com/sBcM2jMAr6
— Matrix (@matrixdotorg) June 8, 2017

However, we then hit three fairly major obstacles:

Matrix lost its funding
In the ensuing uncertainty, the two lead developers (Mjark & Kegan) went to work elsewhere
Meanwhile, Matrix uptake was starting to explode and Synapse was failing to scale to handle the traffic on matrix.org (and elsewhere)

At first, having formed what would become New Vector (now Element) to keep the rest of the core team hired, we pushed to see if we could get Dendrite finished fast enough to replace Synapse, with Erik & richvdh jumping over from Synapse to pick up the remaining work. However, it became clear that we urgently needed a quicker solution to address all the overloaded Synapses out there, and so they swung back to focus on improving Synapse (taking inspiration from some of the design of Dendrite - e.g. offloading endpoints onto worker processes connected via replication streams, and using OpenTracing to debug traffic as it flows over the various services).

At this point, Dendrite maintenance was in effect valiantly taken over by the community, with Brendan and later Anoa keeping the ball going in 2017, joined by APWhitehat in GSoC 2018 and cnly in GSoC 2019. The fact that Dendrite is now here today is thanks in no small part to their work to keep the project alive in its “wilderness years” between Sept 2017 and Dec 2019.

Meanwhile, it became clear that we were overdue getting Matrix itself out of beta - and the last thing we wanted to do was to split and dilute the implementation work of Matrix 1.0 over both Synapse and Dendrite - so we consciously made the decision to focus all our effort on Synapse for solving the remaining bugs and challenges.

Then, in July 2019, Matrix and Synapse exited beta, and we finally started to see light at the end of the tunnel. In October we started dusting off Dendrite again - looking to use it as a relatively simple and flexible codebase for experimenting with Peer-to-Peer Matrix, not least because being Go it can compile to WebAssembly and run clientside, and because even though Dendrite was originally built with massive deployments in mind, it turns out the elastic scaling means it can also scale down pretty small too—as a part of the iOS P2P demo, we’ve even ran full Dendrite homeservers on iPhones embedded into Element iOS! :)

In Dec 2019, we finally got to the point where Element could fund full-time dedicated development on Dendrite once again, with Neil Alexander joining the project and focusing fulltime on getting Dendrite out of alpha and getting it working for P2P and embedded usage (adding libp2p as a federation transport, and adding SQLite support) - and in Jan 2020 we got Dendrite successfully running clientside in a WASM service worker (just in time for FOSDEM!). Then, in Feb 2020, Kegan returned to the project to work fulltime on Dendrite - and the race began in earnest to get Dendrite ready for beta!

Here’s a pretty picture courtesy of GitHub to visualise the progress:

Throughout 2020 there’s been a huge amount of stabilisation work and polish:

Refactoring much of Dendrite’s foundation to make the codebase more maintainable
Created all-new user server, key server, signing key server microservices
Moving some work from existing microservices (ultimately superseding the former currentstateserver, publicroomsapi and typingserver microservices altogether)
Developing new testing infrastructure:
- Complement - our brand new Golang Matrix integration test harness
- Are We Synapse Yet - an aggregator which parses sytest/complement output to compare how close Dendrite is to passing
All the Matrix 1.0 work - particularly state res v2 & room version support
Making it work with more P2P transports for all the exciting P2P experiments
Supporting backfill and fetching missing events
Fixing up SQLite support to make it work as a first class citizen (with shared storage code where we can!)
Supporting both sending and rejecting invites (even over federation)
E2E encryption support (one-time keys, device lists, send-to-device support)
Improved federation sender logic (resend retries, backoffs, blacklisting, metrics, resetting backoffs when receiving transactions)
Handling both inbound and outbound redactions
User interactive authentication (and implemented on various ‘sudo’ endpoints e.g. deleting devices and changing passwords)
Respecting server ACLs
Rejecting / soft-failing events properly
Support for database schema upgrades

... which brings us at last to the present day (Oct 2020), as we declare Dendrite sufficiently stable that we consider it ready for beta testing!

In practice, this means **Dendrite is now ready for experimentation by adventurous Matrix sysadmins. It is NOT ready for production usage yet, but we need folks to test it and help us iron out the remaining bugs! **Please do not trust it with sensitive data yet, and we don’t recommend trying to run it at scale yet as we haven’t done any serious optimisation work yet.

That said, we do provide the following guarantees:

We’re providing versioned releases from here on in, beginning with 0.1.0
We don’t expect any major breaking changes to the config or architecture before 1.0
Ready for early adopters to try running Dendrite without experiencing ~daily breaking churn
The database schema is now stable and will upgrade itself going forwards - your database should now be here to stay! (assuming we don’t hit any nasty data loss bugs during beta)

In terms of comparison with Synapse, the main things you should get excited about are:

Dendrite aims to provide an efficient, reliable and scalable alternative to Synapse:
- Efficient: A small memory footprint with better baseline performance than an out-of-the-box Synapse
- Reliable: Implements the Matrix specification as written, using the same test suite as Synapse as well as a brand new Go test suite
- Scalable: can run on multiple machines and eventually scale to massive homeserver deployments
This means significantly less memory usage than Synapse (depends on joined rooms, often between 50MB - 400MB resident memory) - although we haven’t tuned this at all yet!
All-new database model, where every microservice instance has its own database tables, letting them scale arbitrarily wide
The ability to efficiently use all your available CPU cores without needing to split into separate processes, thanks to Go and our extensive use of goroutines. No more Python global interpreter lock! :)
Future experimental MSCs are likely to land in Dendrite before Synapse (e.g MSC2753 Peeking via /sync and MSC2444 Peeking over Federation are already being prototyped (#1370 and #1391) in Dendrite rather than Synapse!)

The provisos you should know about however are:

We’re not feature complete yet: sytest reports 56% CS API coverage and 77% Federation coverage. NB: these are always going to be underestimates of how much Dendrite actually performs due to how the tests are spread out, in actuality it’s likely more 70% CS, 95% Fed.
No read receipts, membership lazy-loading, presence, push notifications, search, event context, key backups, cross-signing. See changelog for full limitations.
Not battle-tested in the wild by many people (there are probably only ~10 dendrites on the open network today!) - so there’s likely to be a broad spectrum of bugs at first.
Clients that require more exotic features, like lazy loading, may not behave properly yet
Please use Postgres rather than SQLite wherever possible—it’s faster and has fewer issues regarding concurrency (some requests on SQLite Dendrites may 500 with ‘database is locked’ - though we’ve worked hard to eliminate most of these)
Dendrite can run in either “monolith” or “polylith” mode. In monolith, all the microservices are linked into a single binary - and we recommend running in this configuration wherever possible for now. Monolith mode is extremely capable as it is and has fewer moving parts for things to go wrong and will be the right choice for the majority of beta deployments!
Whilst Dendrite is nearly 100% federation compatible, there may still be situations where it will split-brain and disagree with the current room state that Synapse has calculated. We expect these issues to resolve as we get more user feedback.

Architecture-wise, this is what Dendrite looks like under the hood today:

To get up and running, please install Go and head on over to the Get Started guide at https://github.com/matrix-org/dendrite#get-started to join the fun :)

In terms of where we’re going next:

Read receipts. It’s a major missing feature and impacts UX significantly.
100% Federation coverage (according to sytest). It’s crucial that Dendrite instances play nicely with other servers. This will be the best metric we have for asserting that we are just as capable as Synapse at the fed level.
Optimisation—Dendrite has not been optimised yet for speed or resource utilisation!
- We plan to add benchmarks which will stress test different microservices in the presence of many different scaling factors (number of users, number of rooms, size of room, number of devices per user, number of sync requests, etc). This will hopefully allow us to identify early on bottlenecks and slow algorithms
- Good old fashioned pprof with known slow scenarios to see what’s consuming CPU/memory and fixing issues ad-hoc (which we’ve already done a bit of pre-beta). This may involve adding additional in-memory caches, with a healthy respect for the complexities it may introduce (which Synapse has been bitten by)
We plan to add first class feature flag support for experimental MSCs—experimentation is one thing which makes Dendrite notably different from Synapse, and supporting it more thoroughly going forwards will be important. This may mean adding additional hooks; potentially a dedicated microservice to cleanly separate experiments, we don’t know yet
P2P work will continue with vigour now we have a working, featureful, and relatively stable HS to embed and play with

Longer term, it’s pretty hard to say right now when we expect to exit beta (it took Synapse 5 years to exit beta, after all ;) - but obviously we’ll need Dendrite to have parity with Synapse and have no known serious bugs.

Finally: you’re probably wondering what this means for Synapse. Synapse is here to stay - with tens of thousands of deployments around the world serving tens of millions of users. The majority of the core team is still focused on improving and optimising Synapse, and we’ll be keeping improving it for the foreseeable.

However, we’ll certainly be experimenting with new stuff on Dendrite first - whether that’s P2P, portable accounts, new-style communities, peeking etc. We expect Synapse to be the stable long-term-supported solution, while Dendrite (particularly while in beta) will be the more unstable and experimental platform. In the longer term we’ll provide ways of migrating from Synapse to Dendrite however (probably via portable accounts), and perhaps in future new deployments may choose to use Dendrite - a bit like you might choose to use nginx rather than Apache for a new web server these days. But this will be a long transition—meanwhile we expect to see more and more next-generation homeservers like Conduit, Mascarene or Construct coming of age too.

So, there you have it. If you’re an intrepid sysadmin please spin up a Dendrite and start filing bugs! :)

— Matthew, Neil Alexander, Kegan and the whole Matrix team.

Here’s the official changelog:

Client-Server API Features

Account registration and management

Registration: By password only.
Login: By password only. No fallback.
Logout: Yes.
Change password: Yes.
Link email/msisdn to account: No.
Deactivate account: Yes.
Check if username is available: Yes.
Account data: Yes.
OpenID: No.

Rooms

Room creation: Yes, including presets.
Joining rooms: Yes, including by alias or ?server_name=.
Event sending: Yes, including transaction IDs.
Aliases: Yes.
Published room directory: Yes.
Kicking users: Yes.
Banning users: Yes.
Inviting users: Yes, but not third-party invites.
Forgetting rooms: No.
Room versions: All (v1 - v6)
Tagging: Yes.

User management

User directory: Basic support.
Ignoring users: No.
Groups/Communities: No.

Device management

Creating devices: Yes.
Deleting devices: Yes.
Send-to-device messaging: Yes.

Sync

Filters: Timeline limit only. Rest unimplemented.
Deprecated /events and /initialSync: No.

Room events

Typing: Yes.
Receipts: No.
Read Markers: No.
Presence: No.
Content repository (attachments): Yes.
History visibility: No, defaults to joined.
Push notifications: No.
Event context: No.
Reporting content: No.

End-to-End Encryption

Uploading device keys: Yes.
Downloading device keys: Yes.
Claiming one-time keys: Yes.
Querying key changes: Yes.
Cross-Signing: No.

Misc

Server-side search: No.
Guest access: Partial.
Room previews: No, partial support for Peeking via MSC2753.
Third-Party networks: No.
Server notices: No.
Policy lists: No.

Federation Features

Querying keys (incl. notary): Yes.
Server ACLs: Yes.
Sending transactions: Yes.
Joining rooms: Yes.
Inviting to rooms: Yes, but not third-party invites.
Leaving rooms: Yes.
Content repository: Yes.
Backfilling / get_missing_events: Yes.
Retrieving state of the room (/state and /state_ids): Yes.
Public rooms: Yes.
Querying profile data: Yes.
Device management: Yes.
Send-to-Device messaging: Yes.
Querying/Claiming E2E Keys: Yes.
Typing: Yes.
Presence: No.
Receipts: No.
OpenID: No.

This Week in Matrix 2020-10-02

02.10.2020 00:00 — This Week in Matrix — Andrew Morgan

Matrix Live 🎙

Don't forget that Matrix Live is also available in podcast form. Search for "Matrix Live" wherever you get your podcasts.

Dept of Status of Matrix 🌡️

Gitter Enters the Matrix

Half-Shot reported:

Gitter is joining the Matrix ecosystem!

It's true! Element has acquired Gitter from GitLab and will be implementing Gitter's current chat features in Matrix, before rebuilding Gitter as a Matrix client! Before all that however, the first step is to build a proper, modern bridge between the two networks, replacing the old one that's been around for years.

The full spectrum of news is:

And if you'd like to chat with your fellow Matrix and Gitter users, there's a bridged room already set up at #gitter:matrix.org!

Welcome to the community, Gitter! 🎉

Dept of Spec 📜

anoa reported:

Spec

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.

MSC Status

Merged MSCs:

No MSCs were merged this week.

MSCs in Final Comment Period:

MSC2788: Room version 6 as the default room version (merge)

MSC2414: Make reason and score parameters optional for reporting content (merge)

New MSCs:

MSC2801: Make it explicit that event bodies are untrusted data

Spec Core Team

In terms of Spec Core Team MSC focus for this week, after dropping off MSC2414 in the FCP bucket, we're heading back for another big swing at widgets. MSC2774 (widget ID URL parameter), MSC2765 (widget avatars) and MSC2790 (modal widgets) are the focus for this week 🙂

Matrix URIs

kitsune told us:

being totally shook up about the Gitter announcement for a good part of the week, I completed the work on MSC2312 (it's about Matrix URIs, ICYMI) and it hopefully won't be long before it becomes the actual standard to share Matrix coordinates in popular web browsers. Ok, who am I kidding - before Matrix clients get on with adoption.

This will allow people to post links to Matrix rooms/messages/users and when clicked will open right in your favourite Matrix client. Super convenient and great for adoption!

Dept of P2P 👥

iOS P2P Demo

Dendrite is a next-generation homeserver written in Go. It is currently serving as the basis for peer-to-peer Matrix experiments

Neil Alexander announced:

Build 38 of the iOS P2P Demo, as built using Element iOS and Dendrite, has been submitted to TestFlight and will hopefully be available for testers shortly (pending Apple approval)! It features lots of updates in the Dendrite backend which should hopefully make it more reliable.

If you have an iPhone or iPad and enjoy things that sometimes work, join the TestFlight here!

Dept of Servers 🏢

Synapse

Neil reported:

This week we put out a new release candidate -1.21.0rc2

Highlights include

Add experimental support for sharding event persister. (#8294, #8387, #8396, #8419)

Add experimental prometheus metric to track numbers of "large" rooms for state resolutiom. (#8425)

Add prometheus metrics to track federation delays. (#8430)

Fix messages not being sent over federation until an event is sent into the same room. (#8230, #8247, #8258, #8272, #8322)

Fix a regression in v1.21.0rc1 which broke thumbnails of remote media. #8438

Aside from that we are working on moving background processes away from the main process, actually getting the event persister sharding onto matrix.org and trying to improve Synapse stability generally.

Dendrite / gomatrixserverlib

Dendrite is a next-generation homeserver written in Go

Neil Alexander said:

Dendrite is nearing beta! Today we will be cutting a candidate 0.1.0rc1 version after a week of hunting and fixing bugs. We are on track to release version 0.1.0 next week, at which point we will be inviting people to try installing and using Dendrite!

Changes this week include:

Initial sync is fixed after a bug caused us to fall back on incremental sync

The Content-Type HTTP header is now handled properly when MIME-formatted

Internal API calls over HTTP in polylith mode now use their own HTTP client with higher timeouts

Dendrite now tries harder to find missing auth events, using fetcher workers

Dendrite no longer falls back on /state unnecessarily, which was the cause of a major memory leak and high CPU usage

Federation HTTP calls now include the User-Agent header

The event depth field is now ignored in the federation API

We now report the password change capability properly (thanks bn4t!)

Registration flows now include the completed field after a failure (thanks Lesterpig!)

A bug in the previous event updater in the roomserver has been fixed

A bug where we didn't check our own old verify keys when verifying event signatures is fixed

A bug where we didn't handle event ID domains being different to the event origin in earlier room versions has been fixed

TLS fingerprints have been removed

Spec compliance is unchanged:

Client-Server APIs: 56%, same as last week

Server-Server APIs: 77%, same as last week

Recently, we updated some of our documentation and created a whole new set of easy first issues and medium to hard issues for contributors.

If contributing to Dendrite sounds like something you would be interested in, please take a look at these issues and join us in #dendrite-dev:matrix.org! There's also #dendrite:matrix.org for general Dendrite chat and updates and #dendrite-alerts:matrix.org for release notifications and important alerts.

Conduit

Conduit is a Matrix homeserver written in Rust https://conduit.rs

timo reported:

Fix bug leading to many requests at the same time due to error in waiting code

Implement /query/profile over federation

Start work on /invite over federation

Thanks to everyone who supports me on Liberapay or Bitcoin!

The Construct

Jason reported:

This week in Matrix Construct supports aarch64 (ARMv8) architectures. The experience has been phenomenal, with performance exceeding all expectations. Last week I wrote about all new vector code inside Construct using SSE through AVX-512 hardware acceleration in servers; that same code is now accelerated by SVE (Scalable Vector Extensions) on ARM architectures when you compile with clang.

This result is important: ARM virtual machines are offered at a significantly lower price compared to x86 from the same vendor. The systems are cheaper, require less power, but generally perform worse. The trick is to optimize the software for the weaker hardware. The benefit allows, for example, Matrix server hosts running Construct to make th eir profit margin from the lower TCO, getting more out of every computing cycle.

Construct is available at https://github.com/matrix-construct/construct and don't forget to idle and idle #construct:zemos.net.

Synapse Deployment 📥️

YunoHost

Pierre announced:

YunoHost is an operating system aiming for the simplest administration of a server, and therefore democratize self-hosting.

Synapse integration had been updated to 1.19.3 (1.20.1 available in branch testing)

Element Web integration had been updated to 1.7.7 (1.7.8 available in branch testing)

Dept of Bridges 🌉

mautrix-signal

Tulir said:

I made a Signal bridge using signald.

Currently it supports bridging messages, reactions and signal read receipts. End-to-bridge encryption also exists (mautrix-python does all the work there). The bridge can be linked as a secondary device and possibly even registered as the main device, but I didn't actually test registering yet. Setup instructions are currently somewhat non-existent, but it's mostly the same as my other bridges plus signald as a separate daemon.

The repo is https://github.com/tulir/mautrix-signal and the room is #signal:maunium.net

He then popped up again a few days later to say:

After the initial announcement earlier this week, I implemented a few of the missing features like media bridging, and even added setup instructions.

The speed at which Tulir writes bridges scares me sometimes.

mx-puppet-discord

mx-puppet-discord is a (double)puppeting and relay bridge for discord, based on mx-puppet-bridge

sorunome said:

mx-puppet-discord got updated to the newest discord.js version, meaning you have to update if you want to continue to operate it, due to discord having changed their gateway url!

mx-puppet-discord now also supports the intent stuff, so be sure to update by 7th oct

If you run mx-puppet-discord (like I do), make sure to update by October 7th or it will stop working!

Bridges do Hackertoberfest

Half-Shot reported:

Hey folks, I wanted to give another shoutout to say that we are still accepting PRs as part of hacktoberfest. Contributing 4 PRs will get you a T-Shirt (sadly not a Matrix one). Obviously, please ensure your PRs are meaningful (no copyright adjustments, typo fixes).

You can work on any issue, but we've highlighted some issues that would be perfect for newcomers over at https://github.com/matrix-org/matrix-appservice-bridge/issues?q=is%3Aopen+is%3Aissue+label%3Ahacktoberfest and https://github.com/matrix-org/matrix-appservice-slack/issues?q=is%3Aopen+is%3Aissue+label%3Ahacktoberfest

Cadair reported:

In the hope of expanding the number of people contributing to the matrix-appservice-slack repo I have spent a chunk of my morning improving the issue descriptions and labelling up issues. If you are interested in fixing a little annoyance with the slack bridge or just fancy writing some typescript see the good first issue label on the repo.

matrix-appservice-slack 1.6.0 rocks the block

Half-Shot reported:

The matrix.org team are delighted to bring you the latest in Slack bridging technology. Do not let the

minor version bump fool you, this release is packed with the good stuff. The headline feature is that our phase 1 encryption feature has landed and is free for users to experiment with. Head over to the

docs to see how to set this up.

There have been other notable changes, such as:

New configuration options to allow or deny some channels from being bridged.

Support removing reactions from Slack and Matrix messages.

Add onboarding message for new users when puppeting is enabled, to encourage them to puppet.

Improved feature documentation

(and many many bugfixes)

You can read the release information over at https://github.com/matrix-org/matrix-appservice-slack/releases

Looking forward to seeing all the new Hacktoberfest contributors! 🎃

Dept of Clients 📱

fluffychat

sorunome offered:

Fluffychat 0.19.1 has been released!

Features

Implemented ignore list

Jump to events in timeline: When tapping on a reply and when tapping a matrix.to link

Display messages with up to 10 emotes or emoji bigger

New design for the chat list and message bubbles

Implement reactions

Implement password change

Implement deactivate user account

Fixes

Timeline randomly resorting while more history is being fetched

Automatically request history if the "load more" button is on the screen

Hydrogen

Bruno said:

More browser compatibility work this week, making Hydrogen run IE11 on Windows 7, and on Safari on macOS and iOS (still with some caveats). Also fixed several bugs:

fix for unable to open session after a synapse bug manifested itself

prevent the app locking up when you start the app with previously unsent messages

fix sync errors being reported as "null" in the banner

handle timeout during initial sync (important for large accounts) (although I have seen 1 report that this still isn't fixed, please report if you can't login with a large account)

SchildiChat for Android

SpiritCroc said:

SchildiChat's codebase has been updated to Element 1.0.8!

Furthermore, there have been a few design updates:

Media items (pictures, videos, stickers) are no longer displayed in message bubbles

Bigger stickers

Avatars are now hidden in direct chats (when using dual-side message bubbles)

Finally, Schildi doesn't crash anymore if somebody sends an empty message.

SpiritCroc also mentioned some relevant links!

website: https://spiritcroc.de/schildichat-android

repo: https://github.com/SpiritCroc/SchildiChat-android

room: #schildichat-android:matrix.org

Element-iOS

Manu announced:

1.0.14 is in on the release path. It has:

Room: Differentiate wordings for DMs

Room: New room details screen

Add Estonian support

Polishment in several areas and many bug fixes

Full changelog:

https://github.com/vector-im/element-ios/releases/tag/v1.0.14 https://github.com/vector-im/element-ios/releases/tag/v1.0.13

Element Web

Neil announced:

This week released v1.7.8

Highlights include

Secure Backup has been moved out of the registration flow to a toast when you first encounter an E2EE room, which simplifies the new user experience

Added options to hide various UI features when hosting a custom Element

Aside from that we continue to improve on widget support for resizable widgets, modal widgets and generally making widgets better.

We are also continuing to work on instrumenting the app, improving mobile support for matrix.to and making jitsi calling more reliable.

Element Android

benoit told us:

Element Android: Version 1.0.8 is now available on the stores, it fixes issues with verification and PIN code among other issues (see https://github.com/vector-im/element-android/releases/tag/v1.0.8 for more details). Now we are working on improving performance when sending messages to rooms, and also improving global UX, especially of the home (rooms list). Search messages (in clear rooms for the moment) is coming soon, and it will be also possible to filter the room members list.

We will also spend some time on the new Android SDK, https://github.com/matrix-org/matrix-android-sdk2, which is for the moment a quick extract of what we have in Element Android. We have to take care of it as a real product now: document it properly, set up CI, export Javadoc, develop a sample app, etc.

Element for Nextcloud

Gary Kim reported:

Element for Nextcloud v0.6.11 has been released this week. The new version comes with various bug fixes, dependency upgrades, and an upgrade to Element Web v1.7.8. The version is also compatible with Nextcloud 20 which is being released soon.

Dept of SDKs and Frameworks 🧰

Ruma

iinuwa told us:

Over the past couple of weeks, we've received PRs for all of the remaining federation endpoints, and all but one have already been merged!

Now that the end is in sight, we're turning our focus elsewhere. We're working on cleaning up and fixing a few bugs in our event signing code and soon will create tracking issues for filling out the Identity Service API.

Dept of Services 🚀

t2bot.io

TravisR reported:

t2bot.io has crossed 1M monthly active users

All of these users are Telegram/Discord users that have been brought into Matrix over the last 30 days. This doesn't appear to be a temporary spike either: over the last 8 weeks t2bot.io has been hovering at 900-950 thousand monthly active users, up from 600-700 thousand. Record-setting traffic levels have also been achieved, with matrix.org being able to keep up for the first time in a long while.

Overall it's a good sign to see so many communities making the jump to Matrix and sticking around ❤️

Dept of Bots 🤖

zabbix-matrix

progserega told us:

I added zabbix-bot to my zabbix-matrix repo, which can get information about current problems from zabbix-server and send it to matrix user. https://github.com/progserega/matrix_zabbix

In our company we use it for get current situation and https://github.com/progserega/im_sender_service for sending events from zabbix.

I've found that having your systems reporting in a room while you chat around it can be really productive. Props to supporting yet another monitoring platform!

Dept of Interesting Projects 🛰️

Jitsi E2EE Calls using Olm

While 1-1 calls benefit from end-to-end encryption due to WebRTC, Jitsi group calls have always only benefitted from transport encryption.

A while ago Jitsi announced that they were adding E2EE to Jitsi. But did you know that it's using Matrix's Olm encryption under the hood? It's currently available as an experimental feature on https://meet.jit.si!

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

Rank	Hostname	Median MS
1	fairydust.space	368
2	blob.cat	543
3	nuclearlimes.co.uk	580
4	nuclearlemons.uk	889
5	conduit.rs	912.5
6	mailstation.de	1075
7	matrix.org	1092
8	test.zemos.net	1123
9	shortestpath.dev	1251.5
10	chatcloud.net	1338.5

We also now have a room where the ping bots are only hosted on non-Synapse servers! See the scoreboard below.

Rank	Hostname	Median MS
1	conduit.rs	104
2	settgast.org	154
3	construct.grin.hu	182
4	blob.cat	209
5	grin.hu	324.5
6	dendrite.neilalexander.dev	367
7	test.zemos.net	379
8	maunium.net	448.5
9	conduit.nordgedanken.dev	476
10	inferiorlattice.com	592

Final Thoughts 💭

XKCD 2365

Alexandre Franke told us:

XKCD made another strip about messaging systems and I’m outraged to see that Matrix has been left out of it.

Not to worry though, we've fixed it up!

TWIM authorship

I hope you enjoyed this week and last's TWIM editions! Benpa will return next week for more of your regularly scheduled programming. Ciao!

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Welcoming Gitter to Matrix!

30.09.2020 16:28 — General — Matthew Hodgson
Last update: 30.09.2020 14:58

Gitter ♥️ Matrix

Hi all,

We are ridiculously excited to announce that Gitter is joining the Matrix ecosystem and will become the first major existing chat platform to switch to natively speaking Matrix!

If you’re reading this from the Gitter community and have no idea what Matrix is: we’re an open source project that provides an open protocol for secure, decentralised communication - effectively the missing real-time communication layer of the open Web. The open Matrix network has more than 20M users on it and is growing fast (adding another 1.7M or so with the arrival of Gitter!)

Gitter is easily one of the best developer community chat systems out there, used by the communities of some massive projects (Node, TypeScript, Angular, Scala etc) and is a custodian of huge archives of knowledge via their chat logs. Gitter is unique in specifically focusing on developers: their tagline is literally “Where developers come to talk” (unlike Slack, which has barely any community features - or Discord, with its ban on unofficial clients, where developers are a bit of an afterthought relative to the gamers). With Gitter natively joining Matrix, we’re super excited to see the global developer community converging on the open Matrix network - and Gitter’s community rooms should see a huge new lease of life as they’re properly made natively available to the wider network as first class citizens :)

We’ve always had a bit of a crush on Gitter ever since we ended up opposite each other in the exhibition hall at TechCrunch Disrupt Europe 2014 - particularly when they demoed us not only their sexy webapp but also their official IRC server bridge at irc.gitter.im :D Over the years we’ve been gently nudging them to consider fully embracing Matrix, but perhaps understandably they’ve been busy focusing on their own stuff. However, earlier this year, our friends at GitLab (who acquired Gitter in 2017) reached out to explore the opportunity of Gitter becoming a core part of Matrix rather than a non-core project at GitLab… and we’ve jumped on that opportunity to bring Gitter fully into Matrix.

In practice, the way this is happening is that Element (the company founded by the Matrix core team to fund Matrix development) is acquiring Gitter from GitLab, with a combined Gitter and Element dev team focusing on giving Gitter a new life in Matrix! You can read about it from the Element angle over on the Element blog.

Practically speaking, we have a pretty interesting plan here, which we’d like to be very transparent about given it’s a little unusual:

At first, Gitter will keep running as it always has - needless to say, we will be doing everything we can to delight the Gitter community and keep the service in good shape.

Then we’re going to build out native Matrix connectivity - running a dedicated Matrix homeserver on gitter.im with a new bridge direct into the heart of Gitter; letting all Gitter rooms be available to Matrix directly as (say) #angular_angular:gitter.im, and bridging all the historical conversations into Matrix via MSC2716 or similar. We will of course do this entirely as open source, just as Gitter itself is open source thanks to GitLab releasing it under the MIT license in 2017. The plan is to comprehensively document our progress as the flagship worked example case study of “how do you make an existing chat system talk Matrix.”

This will of course replace the old and creaky matrix-appservice-gitter bridge we’ve been running since 2016. Gitter users will also be able to talk to other users elsewhere in the open Matrix network - e.g. DMing them, and (possibly) joining arbitrary Matrix rooms. Effectively, Gitter will have become a Matrix client.

Now we come to the interesting bit. Gitter has some really nice features which are sorely lacking in Element today:

Instant live room peeking (less than a second to load the webapp into a live-view of a massive room with 20K users!!)
Seamless onboarding thanks to using GitLab & GitHub for accounts
Curated hierarchical room directory
Magical creation of rooms on demand for every GitLab and GitHub project ever
GitLab/GitHub activity as a first-class citizen in a room’s side-panel
Excellent search-engine-friendly static content and archives
KaTeX support for Maths communities
Threads!

...and we promise to do everything in our power to preserve and honour these features at all costs and continue to give the Gitter community the experience they’ve come to know and love.

However: in the medium/long term, it’s simply not going to be efficient for the combined Element/Gitter team to split our efforts maintaining two high-profile Matrix clients. Our plan is instead to merge Gitter’s features into Element (or next generations of Element) itself and then - if and only if Element has achieved parity with Gitter based on the above list - we expect to upgrade the deployment on gitter.im to a Gitter-customised version of Element. The inevitable side-effect is that we’ll be adding new features to Element rather than Gitter going forwards.

In practice, the main outcome in the end should be Element having benefited massively from levelling up with Gitter - and Gitter benefiting massively from all the goodies which Element and Matrix brings, including:

E2E Encryption
Reactions
Constantly improving native iOS & Android clients (which should be a welcome alternative to Gitter’s natives ones, which are already being deprecated)
VoIP and conferencing
All the alternative clients, bots, bridges and servers in Matrix
The full open standard Matrix API
Widgets (embedding webapps into rooms!)
...and of course participation in the wider decentralised Matrix network.

So, there you have it. It’s a new era for Gitter - and we look forward to reinvigorating Gitter’s communities over the coming months. We hope Gitter users will be blown away by the features arriving from Matrix… and we hope that Element users will be ecstatic with the performance and polish work that Gitter-parity will drive us towards. Imagine having guest access in Element that can launch and load a massive room in less than a second!

Finally, we would like to explicitly reassure the Gitter community again that we love and understand Gitter (it was one of the very first ever bridges we wrote for Matrix, for instance) - and we will be doing everything we can to not screw up our responsibility in looking after it. Please, please let us know if you have any concerns or if we ever fall short on this.

Any questions, come talk to us on #gitter:matrix.org - which is bridged with https://gitter.im/matrix-org/gitter. Exciting times ahead!

- Matthew, Amandine, and the whole Matrix, Element and Gitter teams.

Matthew and Amandine model 2014-vintage Matrix & Gitter swag in celebration :D

Bonus update - The Changelog Interview!

Sid Sijbrandij (CEO at GitLab) and Matthew had a chance to sit down with The Changelog to talk about Gitter's Big Adventure - so tune in to hear the story first hand! Warning: contains non-ironic use of the word "synergy" :D

The Changelog 414: Gitter's Big Adventure - Listen on Changelog.com

This Week in Matrix 2020-09-25

26.09.2020 00:38 — This Week in Matrix — Andrew Morgan
Last update: 25.09.2020 22:21

Hello all, and welcome to this week's addition of This Week in Matrix! Matrix is an open network protocol for secure, decentralized communication on the web.

My name is Andrew (aka anoa), and I'm a Synapse developer at Element. Thanks to Ben for letting me take over the reins of TWIM for this week! I'll actually be doing the same for next week as well, so please adjust your clocks to Anoa Nonstandard Time accordingly.

With that out of the way, let's jump right in!

Matrix Live 🎙

It's demos week again. This time around we've got the following lineup:

Michael shows off all the new widget goodies coming to Element Web!
Bruno shows off Hydrogen's new encrypted session backup support!
Ismail details the new room creation flow on Element iOS!
Jorik presents his work on revamping the UX of https://matrix.to that he completed for his second summer internship at Element!
Hubert fixes another class of failure-to-decrypt messages edge case with Device Dehydration!
Half-Shot closes off with the tale of Encrypted Bridges!

Don't forget that Matrix Live is also available in podcast form, if you're into that sort of thing. Search for "Matrix Live" wherever you get your podcasts.

Dept of Spec 📜

anoa (hey that's me!) told us:

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.

MSC Status

Merged MSCs:

MSC1960: OpenID information exchange with widgets

MSCs in Final Comment Period:

No MSCs are in FCP.

New MSCs:

MSC2790: Modal widgets (acquiring user input from a widget)

MSC2788: Room version 6 as the default room version

MSC2787: Portable Identities

MSC2785: Event notification attributes and actions

[WIP] MSC2783: Homeserver Migration Data Format

MSC2782: Pushers with the full event content

MSC2781: Deprecate the (reply) fallbacks in the Matrix specification

Spec Core Team

In terms of Spec Core Team MSC focus for this week, we've been rather busy with implementation, so we'll be continuing on with the same focus as last week. As a reminder, that's MSC2414 (making reason and score optional on reports).

Dept of Servers 🏢