The Matrix Holiday Special 2020

25.12.2020 00:00 — General Matthew Hodgson

Hi all,

Over the years it’s become a tradition to write an end-of-year wrap-up on Christmas Eve, reviewing all the things the core Matrix team has been up over the year, and looking forwards to the next (e.g. here’s last year’s edition). These days there’s so much going on in Matrix it’s impossible to cover it all (and besides, we now have This Week In Matrix and better blogging in general to cover events as they happen). So here’s a quick overview of the highlights:

Looking back at our plans for 2020 in last year’s wrap-up, amazingly it seems we pretty much achieved what we set out to do. Going through the bulletpoints in order:

  • We turned on End-to-end Encryption by default.
  • We have a dedicated team making major improvements to First-Time User Experience in Element (as of the last few months; hopefully you’ve been noticing the improvements!)
  • RiotX became Element Android and shipped.
  • Communities have been completely reinvented as Spaces (MSC1772) and while in alpha currently, they should ship in Jan.
  • Synapse scalability is fixed: we now shard horizontally by event - and Synapse is now pretty much entirely async/await!
  • Dendrite Beta shipped, as did the initial P2P Matrix experiments, which have subsequently continued to evolve significantly (although we haven’t implemented MSC1228 or MSC2787 portable accounts yet). Check out the Dendrite end-of-year update for more.
  • MLS experiments are in full swing - we got the first MLS messages passing over Matrix a few days ago, and Decentralised MLS work is back on the menu after an initial sprint in May.
  • There’s been a valiant mission to improve Bridge UX in the form of MSC2346 and its implementations in Element Web, although this has ended up failing to get to the top of the todo list (sorry Half-Shot! :/)
  • Spec progress has improved somewhat, and we are very excited to have welcomed Will Bamberg (formerly MDN) to support the spec from a professional tech writer perspective, with the all-new engine landing any day now! We’re still experimenting with ways to ensure the spec gets enough time allocated to keep up with the backlog, however - particularly community contributions.
  • ...and in terms of Abuse/Reputation - we properly kicked off our anti-abuse work and launched a first PoC implementation in the depths of Cerulean last week.

Perhaps more interesting is the stuff we didn’t predict (or at least didn’t want to pre-announce ;) for 2020:

  • Riot, Modular and New Vector got unified at last behind a single name: Element; hopefully the shock has worn off by now :)
  • Mozilla joined Matrix in force, turning off Moznet IRC in favour of going full Matrix.
  • We welcomed Gitter into the heart of the Matrix ecosystem (with Element acquiring Gitter from Gitlab in order to ensure Gitter’s Matrix integration acts as a reference for integrating future chat silos into Matrix) - with native Matrix support in Gitter going live shortly afterwards.
  • Automattic launched itself into the Matrix ecosystem with an investment in Element, and since then we’ve been working on getting Matrix better integrated and available to them (although all of Element’s Matrix-for-governments activity has ended up delaying this a bit). If you want to work for Automattic on integrating Matrix, they’re hiring!
  • We previewed Cerulean as a super-exciting proof-of-concept client, demonstrating how social media could work on Matrix, with native threading, profiles-as-rooms, decentralised reputation, and (shortly) peeking-over-federation.
  • We completely rewrote matrix.to and relaunched it as a much more capable and friendly permalink redirection service; a precursor to finally getting matrix:// URLs everywhere!
  • We certainly didn’t predict that the “how to install Synapse” video tutorial published at the beginning of the COVID-19 pandemic would end up with 25.5K views (and counting…)

Then, there’s whole new waves of exciting stuff going on. The most obvious has to be the amount of Government uptake we’ve seen with Matrix this year, following on from France embracing Matrix across the public sector last year. Firstly the German armed forces announced their transition to Matrix, and then the German states of Schleswig-Holstein and Hamburg announced a mammoth 500K user Matrix deployment for education and public administration. Meanwhile, North Rhine Westphalia (the biggest state in Germany) launched their own Matrix-powered messager for education; loads of different universities have rolled out Matrix for collaboration - and we hear Famedly is making good progress with Matrix-powered healthcare messaging solutions. Finally, outside of Germany, we’re seeing the first official deployments in the UK government and US federal government - we’ll share details where possible (but sometimes big deployments of encrypted communication systems want to remain discreet). It’s incredibly exciting to see Matrix spreading across the public sector and education, and we’re hoping this will follow a similar pattern to how the Internet, email or indeed the Web first developed: a mix of high profile public sector deployments, complemented by a passionate grass-roots technical community, eventually spreading to span the rest of society :).

Another exciting thing which emerged this year is the amazing academic work that Karlsruhe Institute of Technology’s Decentralized Systems and Network Services Research Group has been conducting on Matrix. This really came on the radar back in June when their Matrix Decomposition: Analysis of an Access Control Approach on Transaction-based DAGs without Finality paper was published - a truly fascinating analysis of how state resolution works in Matrix, and how we manage to preserve access control within rooms without using blockchain-style ‘sealed blocks’ (and has helped fix a few nasty bugs!). I’m not sure any of us realised that Matrix’s state resolution counts as a new field of research, but it’s been great to follow along with their independent work. Most recently, and even more excitingly, they’re circulating a preview of their Analysis of the Matrix Event Graph Replicated Data Type paper - a deep analysis of the properties of Matrix DAGs themselves. We highly recommend reading the papers (what better way to spend the holiday break!). To give a taste, the final paragraph of the paper concludes:

MEG summary

2020 has also seen the arrival and maturation of a whole new generation of Matrix clients - Hydrogen is really impressive as an experimental next-generation Web (and Mobile Web) client; an account with 3000 rooms that uses 1.4GB of RAM on Element Web uses 14MB of RAM on Hydrogen and launches instantly, complete with excellent E2EE implementation. It even works on MSIE! The whole app, including dependencies, is about 70KB of code (200KB including Olm). Meanwhile, matrix-rust-sdk is coming along well, providing a general purpose native library for writing excellent native Matrix clients. Fractal merged initial matrix-rust-sdk a few weeks ago, and we’ll be experimenting with switching to it in Element iOS and Element Android (for its e2ee) in the coming year. It’s not inconceivable to think of a world where matrix-rust-sdk ends up being the no-brainer official SDK for native apps, and Hydrogen’s SDK becomes the no-brainer official SDK for JS apps.

Meanwhile, in the community, there’s been so much activity it’s untrue. But on the subject of maturing apps, it’s been incredibly exciting to see NeoChat emerge as an official KDE Matrix client (built on libQuotient and Kirigami, forked from Spectral), FluffyChat going from strength to strength; Nheko continuing to mature impressively; Mirage appearing out of nowhere as a fully featured desktop client; Fractal merging matrix-rust-sdk etc. On the serverside, Conduit was the big community story of the year - with an incredibly fast Rust + Sled server appearing out of the blue, with viable federation coming up on the horizon. The best bet for an overview of all things community is to checkout the TWIM backlogs however - there’s simply way too much stuff to mention it all here.

Obviously, no 2020 wrap-up post would be complete without acknowledging the COVID-19 pandemic - which increased focus on Matrix and other remote collaboration technology more than anyone could have predicted (especially given all the privacy missteps from Zoom, Teams and others). One of the highlights of the year was seeing the HOPE (Hackers On Planet Earth) conference shift their entire proceedings over to Matrix - turning the conference into a 10 day television station of premium hacking content, with Matrix successfully providing the social glue to preserve a sense of community despite going virtual. Similarly, we’re incredibly excited that FOSDEM 2021 is highly likely to run primarily via Matrix (with bridges to IRC and XMPP, of course) - our work is going to be cut out for us in January to ensure the amazing atmosphere of FOSDEM is preserved online for the >8,500 participants and ~800 talks. And if any other event organisers are reading this - please do reach out if you’re interested in going online via Matrix: we want Matrix to be the best possible ecosystem for online communities, including virtual events, and we’ll be happy to help :)

Talking of FOSDEM, a really fun bit of work which landed in Element this year was to (finally!) polish Widgets: the ability to embed arbitrary webapps into Matrix chatrooms. This includes being able to embed widgets in the RightPanel on Element Web, the LeftPanel too, add as many as you like to a room, resize them(!), and generally build much more sophisticated dashboards of additional content. Modal and fullscreen widgets are coming too, as are ways to simplify and unify access control. It turns out that these have arrived in the nick of time for events like FOSDEM, where we’re expecting to very heavily use widgets to embed video streams, video conferences, schedules, and generally automate the workflow of the conference via adding in web UIs as widgets wherever necessary. The work for this has been driven by the various German education deployments, where the same tricks are invaluable for automating online learning experiences. We originally wrote Widgets back in 2017 as a proof-of-concept to try to illustrate how chatrooms could be used to host proper custom UIs, and it's fantastic to see that dream finally come of age.

Finally, it’s been really exciting to see major progress in recent months on what’s essentially a whole new evolution of Matrix. Two years ago, a quiet patch during the Christmas holidays gave birth to a whole bunch of wild science fiction Matrix Spec Changes: MSC1772: Spaces (groups as rooms), MSC1769: Profiles as rooms, MSC1767: Extensible events, MSC1776: Peeking over /sync, MSC1777: Peeking over federation, etc. This was in part trying to ensure that we had something to look forward to when we emerged from the tunnel of launching Matrix 1.0, and in part trying to draw a coherent high-level sketch of what the next big wave of Matrix features could look like. Inevitably the MSCs got stuck in limbo for ages while we exited beta, launched Matrix 1.0, turned on E2EE by default etc - but in the latter half of this year they’ve hit the top of the todo list and it’s been incredibly exciting to see entirely new features landing once again. Implementation for Spaces is in in full swing and looking great; Profiles-as-rooms are effectively being trialled in Cerulean; Peeking over /sync has landed in Dendrite and peeking over federation is in PR (and unlocks all sorts of desirable features for using rooms more generically than we have today, including Spaces). Only Extensible events remains in limbo for now (we have enough to handle getting the others landed!)

Of these, Spaces has turned out to be exciting in wholly unexpected ways. While prototyping the UX for how best to navigate hierarchies of spaces, we had a genuine epiphany: the ability for anyone to define and share arbitrary hierarchies of rooms makes Matrix effectively a global decentralised hierarchical file system (where the ‘files’ are streams of realtime data, but can obviously store static content too). The decentralised access controls that KIT DSN wrote about could literally be file-system style access controls; enforcing access on a global decentralised hierarchy. We obviously have shared hierarchical filesystems today thanks to Dropbox and Google Drive, but these of course are centralised and effectively only store files - whereas Spaces could potentially scale to the whole web. In fact, you could even think of Spaces as flipping Matrix entirely on its head: the most defining building block going forwards could be the Spaces themselves rather than the rooms and events - just as directories are intrinsic to how you navigate a conventional filesystem. How has Matrix got this far without the concept of folders/directories?!

Right now these thoughts are just overexcited science fiction, but the potential really is mindblowing. It could give us a global read/write web for organising any arbitrary realtime data - with the social controls via ACLs to delegate and crowdsource curation of hierarchies however folks choose. The Matrix.org Foundation could seed a ‘root’ hierarchy, go curate all the rooms we know about into some Linnean-style taxonomy, delegate curation of the various subspaces to moderators from the community, and hey presto we’ve reinvented USENET… but with modern semantics, and without the rigid governance models. Hell, we could just mount (i.e. bridge) USENET straight into it. And any other hierarchical namespace of conversations you can think of - Google Groups, Stackoverflow, Discourse, IMAP trees…

Of course, the initial Spaces implementation is going to be focused of on letting communities publish their existing rooms, and users organise their own rooms, rather than managing an infinite ever-expanding global space hierarchy - but given we’ve been designing Spaces to support government (and inter-government) scales of Spaces, it’s not inconceivable to think we could use it to navigate gigantic public shared Spaces in the longer term.

Anyway, enough Space scifi - what’s coming up in 2021?

2021

Our current hit list is:

  • Spaces - see above :)
  • Social Login - we’re going to be making Single Sign On (SSO) a proper first-class citizen in Matrix (and Synapse and Element) in the coming weeks, and enabling it on the matrix.org homeserver, so users can do single-click logins via Github/Gitlab/Google and other SSO providers. Obviously this means your Matrix identity will be beholden to your identity provider (IdP), but this may well be preferable for many users who just want a single-click way to enter Matrix and don’t care about being tied to a given IdP.
  • VoIP - we have a lot of work in flight at the moment to make 1:1 VoIP super robust. Some of it has already landed in Element, but the rest will land in the coming weeks - and then we’re hoping to revisit Matrix-native group voice/video.
  • Voice messaging - we’re hoping to finally add voice messaging to Element (and Matrix)
  • Location sharing - ...and this too.
  • **P2P **- Lots of P2P work on the horizon, now Dendrite is increasingly stable. First of all we need to iterate more on Pinecone, our pre-alpha next-generation P2P overlay network - and then sort out account portability, and privacy-preserving store-and-forward. We’re hoping to see the live P2P Matrix network turn on this year, however, and ideally see homeservers (probably Dendrite) multihoming by default on both today’s Matrix as well as the P2P network, acting as gateways between the two.
  • Threads - Cerulean is excellent proof for how threading could work in Matrix; we just need to get it implemented in Element!
  • Peeking - Peeking is going to become so much more important for participating in non-chat rooms, such as Spaces, Profiles, Reputation feeds, etc. We’ll finish it in Dendrite, and then implement it in Synapse too.
  • **Decentralised Reputation **- Cerulean has the first implementation of decentralised reputation for experimentation purposes, and we’ll be working solidly on it over the coming year to empower users to counter abuse by applying their own subjective reputation feeds to their content.
  • **Incremental Signup **- Once upon a time, Element (Riot) had the ability to gradually sign-up without the user even really realising they’d signed up. We want to bring it back - perhaps this will be the year?
  • DMLS - with the first MLS messages flowing over Matrix, we want to at least provide MLS as an option alongside Megolm for encryption. It should be radically more performant in larger rooms (logarithmic rather than linear complexity), but lacks deniability (the assurance that you cannot prove a user said something in retrospect, in order to blackmail them or similar), and is still unproven technology. We’ll aim to prove it in 2021.
  • E2EE improvements - We improved E2EE immeasurably in 2020; turning it on by default, adding cross-signing, QR code verification etc. But usability and reliability can still be improved. We’ll be looking at further simplifying the UX, and potentially combining together your login password and recovery/security passphrase so you only have one password to remember going forwards.
  • Hydrogen - We’ll keep polishing Hydrogen, bringing it towards feature parity with Element, ensure its SDK is available for other clients, and start seeing how we can use it in Element itself. For instance, the Spaces-aware RoomList in Element may well end up stealing alien technology from Hydrogen.
  • matrix-rust-sdk - Similarly, we’ll keep polishing matrix-rust-sdk; stealing inspiration from Hydrogen’s state model, and start migrating bits of the native mobile Element apps to use it.
  • The Spec - get Will’s new spec website live, and get improving all the surrounding material too.

I’m sure I’m missing lots here, but these are the ones which pop immediately to mind. You can also check Element's public roadmap, which covers all the core Matrix work donated by Element (as well as everything else Element is getting up to).

As always, huge huge thanks goes to the whole Matrix community for flying Matrix and keeping the dream alive and growing faster than ever. It’s been a rough year, and we hope that you’ve survived it intact (and you have our sincere sympathies if you haven’t). Let’s hope that 2021 will be a massive improvement, and that the whole Matrix ecosystem will continue to prosper in the new year.

-- Matthew, Amandine, and the whole Matrix team.

Introducing Cerulean

18.12.2020 00:00 — General Matthew Hodgson

Hi all,

We have a bit of an unexpected early Christmas present for you today…

Alongside all the normal business-as-usual Matrix stuff, we’ve found some time to do a mad science experiment over the last few weeks - to test the question: “Is it possible to build a serious Twitter-style decentralised microblogging app using Matrix?”

It turns out the answer is a firm “yes” - and as a result we’d like to present a very early sneak preview of Cerulean: a highly experimental new microblogging app for Matrix, complete with first-class support for arbitrarily nested threading, with both Twitter-style (“vertical”) and HN/Reddit-style (“horizontal”) layout… and mobile web support!

Cerulean screenie

Cerulean is unusual in many ways:

  • It’s (currently) a very minimal javascript app - only 2,500 lines of code.
  • It has zero dependencies (other than React).
    • This is to show just how simple a fairly sophisticated Matrix client can be...
    • ...and so the code can be easily understood by folks unfamiliar with Matrix...
    • ...and so we can iterate fast while figuring out threading...
    • ...and because none of the SDKs support threading yet :D
  • It relies on MSC2836: Threading - our highly experimental Matrix Spec Change to extend relationships (as used by reaction & edit aggregations) to support free-form arbitrary depth threading.
  • As such, it only works on Dendrite, as that’s where we’ve been experimenting with implementing MSC2836. (We’re now running an official public Dendrite server instance at dendrite.matrix.org though, which makes it easy to test - and our test Cerulean instance https://cerulean.matrix.org points at it by default).

This is **very much a proof of concept. **We’re releasing it today as a sneak preview so that intrepid Matrix experimenters can play with it, and to open up the project for contributions! (PRs welcome - it should be dead easy to hack on!). Also, we give no guarantees about data durability: both Cerulean and dendrite.matrix.org are highly experimental; do not trust them yet with important data; we reserve the right to delete it all while we iterate on the design.

What can it do?

So for the first cut, we’ve implemented the minimal features to make this something you can just about use and play with for real :)

  • Home view (showing recent posts from folks you follow)
  • Timeline view (showing the recent posts or replies from a given user)
  • Thread view (showing a post and its replies as a thread)
  • Live updating (It’s Matrix, after all! We’ve disabled it for guests though.)
  • Posting plain text and images
  • Fully decentralised thanks to Matrix (assuming you’re on Dendrite)
  • Twitter-style “Vertical” threading (replies form a column; you indent when someone forks the conversation)
  • HN/Reddit/Email-style “Horizontal” threading (each reply is indented; forks have the same indentation)
  • Basic Registration & Login
  • Guest support (slightly faked with non-guest users, as Dendrite’s guest support isn’t finished yet)
  • Super-experimental proof-of-concept support for decentralised reputation filtering(!)

Obviously, there’s a huge amount of stuff needed for parity with a proper Twitter-style system:

  • Configurable follows. Currently the act of viewing someone’s timeline automatically follows them. This is because Dendrite doesn’t peek over federation yet (but it’s close), so you have to join a room to view its contents - and the act of viewing someone’s timeline room is how you follow them in Cerulean.
  • Likes (i.e. plain old Matrix reactions, although we might need to finally sort out federating them as aggregations rather than individually, if people use them like they use them on Twitter!)
  • Retweets (dead easy)
  • Pagination / infinite scrolling (just need to hook it up)
  • Protect your posts (dead easy; you just switch your timeline room to invite-only!)
  • Show (some) replies to messages in the Home view
  • Show parent and sibling context as well as child context in the Thread view
  • Mentions (we need to decide how to notify folks when they’re mentioned - perhaps Matrix’s push notifications should be extended to let you subscribe to keywords for public rooms you’re not actually in?)
  • Notifications (although this is just because Dendrite doesn’t do notifs yet)
  • Search (again, just needs to be implemented in Dendrite - although how do you search beyond the data in your current homeserver? Folks are used to global search)
  • Hashtags (it’s just search, basically)
  • Symlinks (see below)
  • Figure out how to handle lost unthreaded messages (see below)
  • Offline support? (if we were using a proper Matrix SDK, we’d hopefully get this for free, but currently Cerulean doesn’t store any state locally at all).

How does it work?

Every message you send using Cerulean goes into two Matrix rooms, dubbed the "timeline" room and the "thread" room. The "timeline" room (with an alias of #@matthew:dendrite.matrix.org or whatever your matrix id is) is a room with all of your posts and no one else's. The "thread" room is a normal Matrix room which represents the message thread itself. Creating a new "Post" will create a new "thread" room. Replying to a post will join the existing "thread" room and send a message into that room. MSC2836 is used to handle threading of messages in the "thread” room - the replies refer to their parent via an m.relationship field in the event.

These semantics play nicely with existing Matrix clients, who will see one room per thread and a flattened chronological view of the thread itself (unless the client natively supports MSC2836, but none do yet apart from Cerulean). However, as Cerulean only navigates threaded messages with an m.reference relationship (eg it only ever uses the new /event_relationships API rather than /messages to pull in history), normal messages sent by Matrix into a thread or timeline room will not yet show up in Cerulean.

In this initial version, Cerulean literally posts the message twice into both rooms - but we’re also experimenting with the idea of adding “symlinks” to Matrix, letting the canonical version of the event be in the timeline room, and then the instance of the event in the thread room be a ‘symlink’ to the one in the timeline. This means that the threading metadata could be structured in the thread room, and let the user do things like turn their timeline private (or vice versa) without impacting the threading metadata. We could also add an API to both post to timeline and symlink into a thread in one fell swoop, rather than manually sending two events. It’d look something like this:

Cerulean diagram

We also experimented with cross-room threading (letting Bob’s timeline messages directly respond to Alice’s timeline messages and vice versa), but it posed some nasty problems - for instance, to find out what cross-room replies a message has, you’d need to store forward references somehow which the replier would need permission to create. Also, if you didn’t have access to view the remote room, the thread would break. So we’ve punted cross-room threading to a later MSC for now.

Needless to say, once we’re happy with how threading works at the protocol level, we’ll be looking at getting it into the UX of Element and mainstream Matrix chat clients too!

What’s with the decentralised reputation button?

Cerulean is very much a test jig for new ideas (e.g. threading, timeline rooms, peeking), and we’re taking the opportunity to also use it as an experiment for our first forays into publishing and subscribing to reputation greylists; giving users the option to filter out content by default they might not want to see… but doing so on their own terms by subscribing to whatever reputation feed they prefer, while clearly visualising the filtering being applied. In other words, this is the first concrete experimental implementation of the work proposed in the second half of Combating Abuse in Matrix without Backdoors. This is super early days, and we haven’t even published a proto-MSC for the event format being used, but if you’re particularly interested in this domain it’s easy enough to figure out - just head over to #nsfw:dendrite.matrix.org (warning: not actually NSFW, yet) and look in /devtools to see what’s going on.

So, there you have it - further evidence that Matrix is not just for Chat, and a hopefully intriguing taste of the shape of things to come! Please check out the demo at https://cerulean.matrix.org or try playing with your own from https://github.com/matrix-org/cerulean, and then head over to #cerulean:matrix.org and let us know what you think! :)

This Week in Matrix 2020-12-18

18.12.2020 00:00 — This Week in Matrix Ben Parsons

Matrix Live 🎙

This week we're all about video content! So you get Matrix Live AND Demos - you are so lucky! 🥰

Demos

uhoreg added a late improvement to his MLS demo:

This is how my demo would have looked if I had managed to finish debugging in time.

2020-12-18-mls.png

So, now you need to watch the video to make sense of this image!

Dept of Status of Matrix 🌡️

Introducing Cerulean

Matthew reported:

Introducing Cerulean - highly experimental threaded microblogging for Matrix. Read all about it at https://matrix.org/blog/2020/12/18/introducing-cerulean

Cerulean is being discussed on the front page of Hacker News.

Community Demos

The last set of community demos were well received, so if you have an idea for a demo, get it ready for 2021-01-15, which will be the next demo session!

Dept of Spec 📜

New spec platform

wbamberg said:

The spec core team has been trying out the new spec this week to decide if it's ready to ship. And we've been applying Dean's design updates (currently in a separate branch at https://friendly-yonath-3de225.netlify.app/).

Spec

anoa told us:

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.

MSC Status

Closed MSCs:

Spec Core Team

In terms of Spec Core Team MSC focus for this week, we made a healthy amount of progress on the MSCs in focus. A few members have also been iterating quite a bit on MSC1772 for Spaces. Look for these MSCs to start landing come the first few weeks of the New Year as implementation work starts to relax a bit. The focus for next week is... Christmas! 🎄

But first week of January will be MSC2844, MSC2366 (ready and done in key verification) and MSC2790 (modal widgets).

2020-12-18-7E797-stacked_area_chart.png

Dept of Servers 🏢

Dendrite / gomatrixserverlib

Dendrite is a next-generation homeserver written in Go

neilalexander announced:

If you haven't already seen it, we wrote a Dendrite 2020 Progress Update blog post detailing our year in Dendrite. It also includes an announcement for our new public dendrite.matrix.org homeserver, which is open for registration!

Today we released Dendrite 0.3.4, which is a performance and bug-fix release, including the following changes from the last two weeks:

  • The stream tokens for /sync have been refactored, giving PDUs, typing notifications, read receipts, invites and send-to-device messages their own respective stream positions, greatly improving the correctness of sync

  • A new roominfo cache has been added, which results in less database hits in the roomserver

  • Prometheus metrics have been added for sync requests, destination queues and client API event send perceived latency

  • Event IDs are no longer recalculated so often in /sync, which reduces CPU usage

  • Sync requests are now woken up correctly for our own device list updates

  • The device list stream position is no longer lost, so unnecessary device updates no longer appear in every other sync

  • A crash on concurrent map read/writes has been fixed in the stream token code

  • The roomserver input API no longer starts more worker goroutines than needed

  • The roomserver no longer uses the request context for queued tasks which could lead to send requests failing to be processed

  • A new index has been added to the sync API current state table, which improves lookup performance significantly

  • The client API /joined_rooms endpoint no longer incorrectly returns null if there are 0 rooms joined

  • The roomserver will now query appservices when looking up a local room alias that isn't known

  • The check on registration for appservice-exclusive namespaces has been fixed

If you are running a Dendrite server, it is highly recommended that you upgrade to the latest version!

Spec compliance is unchanged:

  • Client-server APIs: 58%, same as last week

  • Server-server APIs: 83%, same as last week

As always, feel free to join us in #dendrite:matrix.org for general Dendrite chat or #dendrite-dev:matrix.org for development discussion.

Dendrite also powers Cerulean, mentioned above!

Synapse

Synapse is a popular homeserver written in Python.

callahad announced:

Hello again from Synapse land! This is our last TWiM of 2020, but we're hard at work landing our last patches before going on holiday. Expect 1.25.0rc1 in early January with loads of fixes, performance improvements, and a few new features. In particular:

You can look forward to all of this and more next year, but until then, happy holidays from all of us over here at Element and in the broader Synapse community!

Conduit

Conduit is a Matrix homeserver written in Rust https://conduit.rs

timokoesters said:

Hey, I'm back! I was busy with University. I also forgot to TWIM this last week, so here you go:

  • Conduit is beginning to support appservices
  • Devin R and I are currently working on a better state store and improving federation

Homeserver Deployment 📥️

YunoHost

Pierre said:

YunoHost is an operating system aiming for the simplest administration of a server, and therefore democratize self-hosting.

Synapse integration had been updated to 1.23.1

Element Web integration had been updated to 1.7.15

Dept of Bridges 🌉

GitHub / Gitlab bridge

Half-Shot told us:

Hello everyone! It's been ages since I've talked about a personal project on here. I've been working on the matrix-github bridge, which is an all singing all dancing service to connect you to Github AND Gitlab. (Yes, it's now poorly named).

I've landed the initial bits to make GitLab work, so that you can join issues and MRs as rooms in the UI but there is more work to do to make that solid and more interactive. The thing I am most looking forward to is notifications for GitLab so I can get away from emails.

ALSO, off the back of TravisR's fantastic widget work, I've landed initial support for widgets in admin rooms and soon issue/PR rooms. The idea here being that widgets give you a nicer UI to control the bridge or render details like commits going past or diffs. All the groundwork for authenticating widgets against the bridge and rendering them is done, now it just needs features.

That's the highlights this week. I'm hoping to really tighten the documentation to get more people spun up on it as the notifications features have turned out to be a killer feature for me :)

2020-12-18-x7YaI-image.png

I've been using this (the bridge, rather than the upcoming widgets) for a couple of months now and find it really really powerful. I'm not in GitHub most of the day, so getting notifs piped direct into Matrix is great!

mautrix-instagram

Tulir said:

I added bridging of typing notifications and read receipts from Matrix to Instagram and fixed a bunch of bugs. Only thing that's still missing is presence and Matrix->Instagram non-image files.

This is now almost feature complete! Nice one Tulir.

Gitter

Eric Eastwood said:

A few updates to the Gitter bridge itself this week handling Matrix edits outside of the 5 minute edit window on Gitter and fixing up @mentions being mangled by : colons on the Gitter side when someone mentions you from Element. We also strip the redundant Matrix reply quote when it goes into a Gitter threaded conversation because it's already easily viewable in the thread.

We're also getting started on some of the Element feature parity with Gitter issues. First directly tackling backfilling existing message history.

Other teams are already underway working on Spaces, Social login for Github which will ship in the new year!

Dept of Clients 📱

Element Clients

Compiled by the team.

  • Spaces
    • We’ve been progressing spaces on Web, focusing on managing spaces, members and rooms. Meanwhile, we’ve also been iterating on designs for all platforms.
  • Social login
    • Synapse support for picking a username should be landing in develop today focusing next on supporting multiple identity providers. On iOS, we’ve implemented SSO redirect authentication mechanisms which also should land soon.
  • VoIP
    • We’ve been making progress all round on all platforms, with the next web release including improvements to holding & resuming calls and multiple line support!

Web

  • 1.7.16-rc.1 available on staging
    • Added ability to put a VoIP call on hold and answer another
    • Added a special visual effect for confetti
    • Improved the login flow
  • Investigating changelog options in the background

Android

iOS

  • We’re prepping 1.1.3 after ironing out final kinks relating to the new background service, hoping to release before Christmas.

Nheko

Nico (@deepbluev7:neko.dev) offered:

In celebration of the cross-signing MSC being finally merged, Nheko master now supports cross signing other users and your own devices. 🎉 The UX is currently not final, you have to manually fetch the cross signing keys in the settings, if you don't have them after a verification. You are also not yet prompted to verify new logins and can't bootstrap cross-signing from Nheko, but the functional bits and bobs are there now. We are also still investigating, if prompting for your recovery key/passphrase is less confusing by using just one input field or if we should prompt for them separately. Currently Nheko just asks you to unlook and will accept whatever you input (if it can decrypt the secrets with it). FluffyChat does that and I think it is really nice!

There are also a few smaller features added recently:

  • Nheko now uses the icon from your theme on the login page

  • You can react instead of reply with a (very) short text using /react

  • A few improvements(?) regarding whitespace and code rendering.

SchildiChat Web/Desktop

qg reported:

Today, we can announce the web/desktop variant of SchildiChat is out of beta!

SchildiChat is a fork of Element, which focuses on UI changes such as message bubbles and a unified chat list for both direct messages and groups, which is a more familiar approach to users of other popular instant messengers.

Since the last announcement on TWIM, we have not only fixed most of the layout bugs caused by the message bubbles, but can also report the following:

  • Settings for layouts work, so you can also use SchildiChat if you only want the unified overview and no message bubbles

  • All information about desktop is now on our new website: https://schildi.chat/desktop

  • The hosted instance is now here: https://app.schildi.chat

  • For Linux, SchildiChat is now available from AUR, apt repo for debian, flathub and as AppImage

  • There's a portable Windows version now too

Code on GitHub: https://github.com/SchildiChat/schildichat-desktop

Matrix room: #schildichat-web:matrix.org

2020-12-18-bIasy-2.png

FluffyChat

FluffyChat is a cute cross-platform matrix client. It is available for Android, iOS, Web and Desktop.

krille reported:

FluffyChat 0.24.0 is released with new features and fixes.

What is new:

  • A new public room discovery page, where you can find and join public rooms

  • Experimental bootstrap support for cross-signing and keybackup and something really weird that is named SSSS

  • Enhanced power level managing for rooms

  • Minor design changes like a new icon set

Beside of that we have of course a lot of bug fixes and stability improvements. All languages should now also work on iOS.

Also we are looking forward to publish the app in the AppStore for iOS but we will do this under the hood of Famedly. Btw. FluffyChat will soon move to the Famedly GitLab project and we will move the app in the PlayStore to the Famedly account too. This doesn't mean that anything will change there. It is just that Famedly will now handle the legal stuff for us and may offer paid customer support for businesses, which have interest in FluffyChat. But the app will still be available under the same AGPL3 license. This will just make it easier for us to publish the app in the official stores as we don't need to pay so much for the developer accounts.

2020-12-18-JruOC-screenshot_20201218-144554.png

Dept of Bots 🤖

Timetracking Bot by famedly

MTRNord reported:

Find it at: https://gitlab.com/famedly/bots/timetracking or join us at #timetracking-bot:famedly.de

Numerous new features and fixes arrived this week. Getting closer to production readiness. Famedly also started daily internal testing for the bot to find more issues and fixing them.

v0.1.5

New Features

  • Implement !setTimezoneDefault

  • Add timezone conversion to !in and !out

  • Write admin status to table in Database

  • Add default_timezone field to Config which defines the bot wide default. If not set UTC is assumed.

  • Add !stats command

  • Use Unix timestamps in database (migrated automatically)

Fixes

  • Make sure Version sha actually gets generated in CI

  • Make tests work again in the CI

  • Fix that !help and !version work without the database in case of issues

!record command polish

  • Only accept positive durations greater than 1

  • Only accept durations less than 24h

  • Check if duration is numeric

Refactoring

  • Move message templates where needed to templates folder

  • Use color-eyre instead of anyhow

v0.2.0

Refactoring

  • All new command definition. Allowing for much easy adding of commands and helptext

  • New !help response

Fixes

  • Add the missing default for !stats

  • Check for required amount of arguments in !delete

Chore

  • Fix typos in responses

  • Remove some dead code

GDQBot

daenney told us:

It's the most wonderful time of the year! The ⛄️ winter edition of Games Done Quick is almost upon us. This recurring speedrunning event helps raise money for charity each year. As usual, the proceeds of the week long AGDQ are going to the Prevent Cancer Foundation. With some luck we might break last year's $3.13 million record.

To help you keep track of the event I've created GDQBot. Right now you can use it to query for the schedule. Next week should see the code land to add the announcer routine so the bot will announce events as they happen to channels its invited to.

Also a big thank you to tulir for patiently answering my questions as I was trying to find my way around mautrix-go.

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1neko.dev322
2bridges.matrix.linux.pizza420
3almum.de474
4envs.net598.5
5saces.de627
6kb1rd.net722
7matrix.sp-codes.de905
8casavant.org943
9aria-net.org1034
10flobob.ovh1050

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Matrix.to: Reloaded

17.12.2020 15:08 — General Nad Chishtie
Last update: 17.12.2020 14:11

One of the goals on the core team is to reduce friction for new users joining Matrix. A challenge we regularly face is all the factors which make Matrix flexible and powerful as an open, secure decentralised protocol also increase the difficulty of getting started.

As one example— to create a healthy, vibrant ecosystem and put power back into the hands of end users, it’s critical multiple clients exist, and that users ultimately have control over which one to use. However, needing to choose a client before getting going is counter-intuitive, and adds cognitive load which proprietary services simply don’t have.

Striking this balance is tricky, and one we’re aiming to improve with the latest version of Matrix.to.

What's Matrix.to?

Matrix.to is a simple URL redirection service which lets users share links to rooms, users and communities. It isn’t tied to any client, and for end users it serves as a landing page for many as their first touch to Matrix. And until the matrix:// URI scheme is deployed commonplace (we finally registered it with IANA a few months ago!) it is the only client-agnostic way to link to content in Matrix.

It’s privacy preserving, with no information stored server side, with all calculation done in client-side JavaScript. It started life years ago as a temporary minimum viable hack by Matthew which then hung around untouched for years being… minimally viable. But we’ve recently given it a huge upgrade in usability and functionality at last!

What's new?

The matrix.to room, on the new matrix.to!

We’ve re-written matrix.to from scratch, giving it a visual upgrade and refocused the experience around:

  1. Helping new users find the best client for them to easily get going. By default, links will prioritise showing clients which are compatible with the platform the link is being viewed from, including mobile platforms.

  2. Optionally, remembering your preferred client for future visits. This also includes deeplinking into native apps for Element Desktop & Element Mobile (and in future, other clients too, of course).

  3. Fetching useful previews. One of our main observations when testing with new users is a lack of confidence trying out a new service without personalised, contextual information on the rooms or people they’re heading to. Matrix.to now displays room metadata like avatars, names and topics directly (fetched via matrix.org by default, asking for permission).

What's next?

We’ve baked in the ability to specify clients and deployments within links, allowing sharing to give the option of a specific destination to guarantee a predictable experience. For instance, Mozilla might share matrix.to links which recommend using chat.mozilla.org, if you don’t already have a preferred Matrix client configured. We’ve yet to implement this feature in Element, but we’ll be researching and experimenting with different implementations soon and will recommend best practises when we have them.

We plan to evolve matrix.to over time, including eventually evolving it to better support the Matrix URI scheme. As before, you can find the source code on GitHub and please go ahead and submit pull requests to get your Matrix client listed.

Let us know what you think over in the matrix.to room, linked to via matrix.to!

Dendrite 2020 Progress Update

15.12.2020 00:00 — General Neil Alexander

It's been a year since Dendrite development picked up again and it's certainly been a busy one at that! We started off 2020 by sprinting to complete the FOSDEM P2P demo and, since then, we have continued to develop Dendrite into a more featureful and stable homeserver.

In October, we moved Dendrite into beta, and have since released a number of releases. We've also seen quite a lot of interest from the community, so I'm here today to write about some of the fun things that have been going on in Dendrite-land.

Announcing dendrite.matrix.org

I'm happy to announce that we've finally deployed our own public Dendrite instance at dendrite.matrix.org! It's running the latest Dendrite code and is open for registration, so if you have been looking for an opportunity to play with Dendrite without hosting your own deployment, here's your chance!

There are still bugs and missing features, but overall the server is quite usable, so please feel free to register and try it and let us know how you get on.

This is the first deployment that we've built for any kind of scale, so we are cautious of the fact that there may be performance bottlenecks still. That said, over the last few weeks, a number of performance-improving changes have been merged, including:

  • Around 20x performance improvement on the State Resolution v2 algorithm, which is used in room versions 2 and above
  • Significantly reducing the amount of time spent recalculating event reference hashes and event IDs in the roomserver and sync API, reducing CPU usage
  • Optimised memory usage and reduced database hits in the federation sender, which helps particularly in large rooms with lots of resident servers

We're optimistic that running this deployment will help us to identify scaling pain points and to make Dendrite leaner in the long run. Feel free to sign up and give it a spin! :-)

Features

Since the beginning of the year, we've added a number of new features, including but not limited to:

  • Room versions support for all currently specced versions (v1-v6), including support for State Resolution v2
  • SQLite storage support in addition to PostgreSQL, largely useful for the P2P demos
  • Support for database schema upgrades, making updating Dendrite significantly easier
  • Early end-to-end encryption support, including device list syncing and send-to-device messages, although with key backup and cross-signing still to come
  • A number of federation features, including invites, retries and backing off unreachable homeservers
  • User-interactive authentication (UIA), which is used in password changes and deleting devices from your device list
  • Support for local pagination, backfilling over federation and fetching missing events
  • Redaction of events both locally and over federation
  • Entirely new microservices for managing server signing keys, E2E keys, user and device management
  • Lots of great contributions from the community - including all of Read Receipts (thanks to S7evinK) and Read Markers (thanks to Lesterpig)!

...and of course entered Beta in October!

Spec compliance

Of course, Dendrite also needs to be able to fulfill the promise of being a usable next-generation Matrix homeserver at the same time as being a sci-fi development platform. We have spent much of the last year working specifically on this. Today, Dendrite's Sytest compliance sits at:

  • 59% compliance for Client-Server APIs, up from 33% in May 2020
  • 83% compliance for Server-Server APIs, up from 26% in May 2020

As you can see, these are significant leaps in the numbers of tests passing against Dendrite.

Experimental MSCs

We have been increasingly trying to use Dendrite for the development and testing of some new Matrix feature proposals. Recently we've seen early support added for Peeking (MSC2753) and there is work in progress on Peeking over Federation (MSC2444).

Peeking enables temporarily subscribing to a room for real-time events without joining the room. This will only be possible with rooms that are world-readable, but it reduces the overhead of looking into a room significantly as there is no need to update the room state for each peeking user/device.

In addition to that, we've also been working on Threading (MSC2836) support, which is the gateway to building some pretty new and interesting Matrix experiences. Twitter-like or Reddit-like social prototypes like this have traditionally been difficult to build on top of Matrix as the m.reference relation type from MSC1849 had never really been fleshed out.

Threading adds m.relationship fields for embedding these relationships, and also specifies an additional /event_relationships API endpoint for finding other events related to a given event in either direction. This makes it possible to build threads.

P2P Matrix

Dendrite has also been our primary development platform for P2P Matrix. This year we have released multiple P2P Matrix demos, including:

  • p2p.riot.im, which uses libp2p rendezvous and embeds a full Dendrite homeserver into an in-browser Service Worker
  • Element iOS P2P, available on TestFlight, which embeds a full Dendrite homeserver into the Element iOS app, initially using QUIC over Yggdrasil as a transport for federation traffic, but with more recent versions using QUIC over the experimental Pinecone protocol
  • dendrite-demo-libp2p, a standalone binary which formed the basis of the FOSDEM 2020 P2P demo, using libp2p and local multicast discovery as a transport for federation traffic
  • dendrite-demo-yggdrasil, another standalone binary like above, but using QUIC over Yggdrasil connectivity instead of libp2p as the transport for federation traffic

Each experiment teaches us more about potential issues that need to be resolved in order to bring P2P Matrix closer to being reality, and we are continuing to use Dendrite for this work. We'll be announcing more information in the New Year about our latest efforts and the Pinecone routing scheme that we are developing.

It's also worth highlighting that all of the other experimental work taking place right now, including Threading and Peeking, also work over P2P!

What's next

We'll be taking a short break for Christmas, but will then be continuing work on Dendrite in 2021, with the main aims being to add new features, improve spec compliance further, fix bugs and eventually exit beta. We'll also be continuing further experimental work in the P2P and Threading areas, as well as supporting the development of new MSCs such as Portable Identities (MSC2787).

We'd like to say thank you for the community support and interest, and also to send out a special thanks to our community contributors who have contributed a number of fixes and features in recent months! We always welcome code contributions via GitHub if you are an interested developer.

As always, stay tuned for more Dendrite updates either by joining us in #dendrite:matrix.org or by getting your regular dose of This Week in Matrix!

— Neil Alexander and Kegan

This Week in Matrix 2020-12-11

11.12.2020 00:00 — This Week in Matrix Ben Parsons

Matrix Live 🎙 (Demos week!)

Dept of Status of Matrix 🌡️

Lots of things to talk about! Firstly I'm very happy to say that spaces are happening, you can keep track of them here: https://github.com/vector-im/element-web/issues/15930.

Next Open Tech Will Save Us, was great this month, and you can also get a great introduction to Matrix Spaces there too.

Then, Marcus Schopen shared his love for Matrix by showing us the Matrix-branded mugs he has had printed:

2020-12-11-8_-UX-IMG_20201209_160020.jpg

If you'd like to buy some OFFICIAL Matrix merch (we don't have mugs just yet), then please go to The Matrix Shop! Marcus also added:

People in Bielefeld can come over to my office at university and get one.

Dept of Spec 📜

wbamberg told us:

New spec platform: we've asked the spec core team to spend some proper time with the new spec so we can decide if it's ready to ship.
You can also file bugs at https://github.com/matrix-org/matrix-doc/labels/spec-redesign

It's looking really good!

Spec

anoa offered:

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.

MSC Status

New MSCs:

Spec Core Team

In terms of Spec Core Team MSC focus for this week, we've continued to be quite busy with implementation during holiday crunch time. There were some constructive ideas for the team to carve out more time for spec work during this week's Spec Core Team retro, which will hopefully lead to an improved cadence this week and into the new year.

As such, the focus remains at MSC2844 (global versioning), MSC2366 (ready and done in key verification), and MSC2790 (modal widgets).

2020-12-11-9fxga-stacked_area_chart.png

Dept of Servers 🏢

Synapse

Synapse is a popular homeserver written in Python.

callahad offered:

Hello, friends! On Wednesday we released Synapse 1.24.0 and 1.23.1, which include security fixes. Please upgrade! (Also, if you've installed Synapse from source, be sure to pip install 'cryptography>=3.3' in your virtualenv to address the recent OpenSSL CVE).

This is our last full release for the year, as we don't want to do anything that might risk the stability Santa's federated command and control centre at the North Pole, but we're looking forward to lots of new features and improvements in the new year — more details on that next week.

We'd also like to remind you that we're planning on removing support for Python 3.5 early next year (a formal announcement and timeline will accompany the 1.25.0 release), and we'd appreciate and comments or concerns in this GitHub issue.

Strikes me that Santa has a more centralised operation, but who can say?

Homeserver Deployment 📥️

YunoHost

Pierre told us:

YunoHost is an operating system aiming for the simplest administration of a server, and therefore democratize self-hosting.

Synapse integration had been updated to 1.23.0 (1.23.1 available in branch testing)

Element Web integration had been updated to 1.7.14 (1.7.15 available in branch testing)

Kubernetes

Ananace offered:

Just updated my Synapse K8s-optimized image and Chart to 1.24.0, planning on deprecating the image at some point soon though, instead pointing people to upstream - or platform-optimized - images as I should be able to move all the Kubernetes-specific things into the chart itself to make it more compatible.

The image in question started off as a plain docker image, before there were official ones. I've since rebased it onto the official Matrix image with just the modified scripts and volume points.

My plan is to move all the Kubernetes-specific scripts into configmaps (which are a K8s way of storing text files/environment variables/etc), and then build my own launch arguments through the chart to get it to run on any image that has a Python interpreter and the Synapse modules available in the Python search path.

With all that done, it means I won't be a bottleneck in pushing updated image tags for people to be able to update, and I won't have to finagle multi-arch images 😃

Docker ARM images for Dendrite

TR_SLimey reported:

As the PR from last TWIM is still awaiting review, I have once again updated the Docker ARM images for Dendrite for version 0.3.3. As always, they can be found in my docker repo at https://hub.docker.com/u/trslimey

Direct links:

Monolith: https://hub.docker.com/r/trslimey/dendrite-monolith

Polylith: https://hub.docker.com/r/trslimey/dendrite-polylith

Dept of Bridges 🌉

Gitter

Eric Eastwood announced:

If you didn't catch it already, we did a full announcement this week for Gitter now speaking Matrix 🎉! If you're curious about how we made it and some more technical details, the blog post is a great read: https://matrix.org/blog/2020/12/07/gitter-now-speaks-matrix

In terms of bridge progress, we added support for m.emote status messages which means your /me says hi messages will bridge back and forth properly. On the Gitter side, we fixed the browser notifications so they show properly from the virtualUser instead of the generic matrixbot.

mautrix-instagram

Tulir offered:

I made an Instagram DM bridge: https://github.com/tulir/mautrix-instagram / #instagram:maunium.net

It's a bit barebones still, but text bridging and backfilling works. The main advantage over mx-puppet-instagram is that it uses the realtime MQTT API instead of polling.

There was a second update from tulir, he added more features as of today:

After the initial announcement on Monday, I added bridging of reactions, redactions and photos in both directions and also typing notifications, read receipts and all types of messages from Instagram to Matrix.

Dept of Clients 📱

Element Client Work

Compiled by the team

  • Spaces

    • We’re making more progress on implementing Spaces.. On Web, we’ve made progress on creating spaces and switching between them. On mobile, we’ve laid down the foundations in the Android SDK. If you missed it, we gave a preview in this weeks Open Tech Will Save Us.
  • Social login

    • We’ve been working on server-side support in synapse for picking an MXID during login with SSO, and also implementing support in iOS to gain parity with Android & Web, to launch together in future.
  • VoIP

    • We’ve been working on implementing hold & resume (line 1 / 2) support on Web, iOS & Android. Web has something sneak previewable on develop.element.io!

Web

  • Element Web 1.7.15 released
    • Improved voice / video call UI
    • Notifications are now ignored from rooms you've been active in recently
    • Login / registration flows redirect inside the app if you are already logged in
  • JS SDK browser build steps now only run on release (instead of every install) to save time

Android

  • URL Previews are coming in the next release! There will also be a brand new Emoji keyboard accessible from the timeline composer.
iOS
  • The new background service has been merged. Messages from notifications will be displayed faster within the app. A TestFlight build for testing will be available tomorrow.

Dept of Ops 🛠

axon

kaiyou announced:

This is a first update regarding a very small project named "axon", aimed at operating the synapse admin API from Python, either as a library or a CLI tool. It is currently in use everyday on our synapse instance for managing users and looking for suspicious rooms, and is being integrated into a prototype Web interface for managing synapse.

Admin API coverage is currently about 50%, mostly focusing on users and rooms management (get, list, update, delete). It is still enough to delete a user, edit an offensive avatar, or run as a cron to periodically purge remote media.

Our main repository: https://forge.tedomum.net/tedomum/axon

Github project mirror for contributing: https://github.com/kaiyou/axon

I asked if there was a GUI:

Not currently, but we are integrating it with another of our tools named "hiboo", which is a general indie hoster user manager (provides SAML & OIDC authentication with config templates for many projects including synapse, and integrates the account workflows and moderation features). Currently it provides room navigation and purging, plus deleting accounts either on moderator decision or when the user deletes her SSO account.

Sounds promising! Synapse Admin would benefit from this kind of tooling...

... so it was a surprise to find another project designed to help in this area!

Announcing synadm

jojo told us:

Well, maybe it's about time for "synadm - a CLI frontend to Matrix-Synapse admin APIs" to be announced more officially. As the name implies it's a CLI tool that eases the usage of admin API commands for Synapse admins. It currently covers everything the room and user API's can do: https://github.com/JOJ0/synadm#implementation-status--commands-list

It can be configured interactively and/or using a config file in yaml format: https://github.com/JOJ0/synadm#configuration

It's configurable whether the responds of the API are shown in pretty printed JSON or in a table format (using Python tabulate).

It's well documented via it's online help and supposed to be quite self-explanatory to the admin.

Any thoughts, ideas, feature-requests are very welcome in #synadm:peek-a-boo.at or as usual by filing PR's or issues in the project repo https://github.com/JOJ0/synadm.

Dept of Bots 🤖

Advent of Code 2020 bot

Asbjørn told us:

This one might be interesting to those participating in Advent of Code 2020:

I made a (quick and dirty) bot that posts whenever somebody on a private leaderboard solves a challenge. https://gitlab.com/AsbjornOlling/aocbot

I imagine that we're not the only private AoC Matrix room around, so I figured I'd share - even though it's not anything particularly remarkable 👨‍💻

Dept of Events and Talks 🗣️

Matrix talk at Open Source Summit EU

Oleg reported:

The recording of the Matrix talk from the annual Open Source Summit EU is online! 🎉

I talked about Matrix 101 and showed different bridging capabilities with Slack and IRC and, of cause, showed some bots. 😉

Thanks benpa and Half-Shot for your support! ❤️

▶️ Check it out: https://www.youtube.com/watch?v=cD8xbci4wAY

NLUUG talk from Matthew

Matthew offered:

I gave a talk to NLUUG the other week: https://www.youtube.com/watch?v=gY2PCKEaKlM

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1neko.dev298
2comsys.rwth-aachen.de341
3pc.koesters.xyz:59003444
4maescool.be446
5bordum.dk465
6jauriarts.org505
7dendrite.matrix.org554
8casavant.org717
9matrix.sp-codes.de758.5
10envs.net913

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Synapse 1.24.0 and 1.23.1 released

09.12.2020 23:51 — Releases Dan Callahan

Synapse 1.24.0 is now available!

This release fixes a denial of service vulnerability (GHSA-hxmp-pqch-c8mm / CVE-2020-26257) in which a malicious homeserver could send malformed events into a room which would then break federation of that room.

This follows the disclosure of a denial of service vulnerability in OpenSSL (CVE-2020-1971). If you have installed Synapse from source, please ensure your host is up to date and then execute pip install 'cryptography>=3.3' inside your Synapse virtualenv.

We've also released Synapse 1.23.1 which includes that security fix and a small patch to maintain Python 3.5 compatibility. It is otherwise identical to 1.23.0. Note that Synapse 1.24.0 includes backwards incompatible changes which may affect a small number of users. See the notes on upgrading for more information.

Synapse 1.24.0 brings a pair of new Admin APIs, including a way to log in as users and to forcibly purge rooms when deleting them. We've also made numerous bug fixes and improvements to SSO support, especially around OpenID Connect and SAML providers.

This release includes an optional change to push notification badges: currently, the number in the badge is based on the count of rooms with unread messages. However, in some specialized cases you may want the badge to show the count of all unread messages, even if there are multiple unread messages in the same room. This behavior can now be toggled with a new configuration setting.

Additionally, for server admins, the deprecated /_matrix/client/*/admin Admin API endpoints have been removed. If you have tools which target these endpoints, please update them to use the /_synapse/admin URL prefix instead.

See the full changelog for more.

Installation instructions are available on GitHub, as is the v1.24.0 release tag.

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including @angdraug, @chagai95, @daenney, @dklimpel, @jordanbancino, @localguru, @nchamo, @ShadowJonathan, @TeFiLeDo, @tulir, and @waylon531.

Gitter now speaks Matrix!

07.12.2020 00:00 — General Matthew Hodgson

Hi all,

It’s been just over 2 months since we revealed that Gitter was going to join Matrix - and we are incredibly proud to announce that Gitter has officially turned on true native Matrix connectivity: all public Gitter rooms are now available natively via Matrix, and all Gitter users now natively exist on Matrix. So, if you wanted to join the official Node.js language support room at https://gitter.im/nodejs/node from Matrix, just head over to #nodejs_node:gitter.im and *boom*, you’re in!

This means Gitter is now running a Matrix homeserver at gitter.im which exposes all the active public rooms - so if you go to the the room directory in Element (for instance) and select gitter.im as a homeserver, you can jump straight in:

Gitter room directory

Once you’re in, you can chat back and forth transparently between users on the Gitter side and the Matrix side, and you no longer have the ugly “Matrixbot” user faking the messages back and forth - these are ‘real’ users talking directly to one another, and every public msg in every public room is now automatically exposed into Matrix.

Gitter and Matrix going native!

So, suddenly all the developer communities previously living only in Gitter (Scala, Node, Webpack, Angular, Rails and thousands of others) are now available to anyone anywhere on Matrix - alongside communities bridged from Freenode and Slack; the native Matrix communities for Mozilla, KDE, GNOME communities etc. We’re hopeful that glueing everything together via Matrix will usher in a new age of open and defragmented dev collaboration, a bit like we used to have on IRC, back in the day.

This is also great news for mobile Gitter users - as the original mobile Gitter clients have been in a holding pattern for over a year, and native Matrix support for Gitter means they are now officially deprecated in favour of Element (or indeed any other mobile Matrix client).

What features are ready?

Now, this is the first cut of native Matrix support in Gitter: much of the time since Gitter joined Element has been spent migrating stuff over from Gitlab to Element, and it’s only really been a month of work so far in hooking up Matrix. As a result: all the important features work, but there’s also stuff that’s yet to land:

Features ready today:

  • Ability to join rooms from Matrix via #org_repo:gitter.im
  • Bridging Edits, Replies (mapped to Threads on Gitter), Deletes, File transfer
  • Bridging Markdown & Emoji

What remains:

  • Ability to send/receive Direct Messages
  • Ability to plumb existing Matrix rooms into Gitter natively
  • Ability to view past Gitter history from Matrix. This is planned thanks to https://github.com/matrix-org/matrix-doc/pull/2716
  • Synchronising the full Gitter membership list to Matrix. Currently the membership syncs incrementally as people speak
  • Turning off the old Gitter bridge
  • Bridging emotes (/me support) (almost landed!)
  • Bridging read receipts
  • Synchronising room avatars
  • Bridge LaTeX

Stuff we’re not planning to support:

  • Ability to join arbitrary rooms on Matrix from Gitter. This could consume huge resources on Gitter, and we’re not in a rush to mirror all of Matrix into Gitter. This will get addressed when Gitter merges with Element into a pure Matrix client.
  • Bridging Reactions. Gitter doesn’t have these natively today, and rather than adding them to Gitter, we’d rather work on merging Gitter & Element together.

For more details, we strongly recommend checking out the native Matrix epic on Gitlab for the unvarnished truth straight from the coal-face!

How do you make an existing chat system talk Matrix?

In terms of the work which has gone into this - Gitter has been an excellent case study of how you can easily plug an existing large established chat system into Matrix.

At high level, the core work needed was as simple as:

  • Add ‘virtual users’, so remote Matrix users can be modelled and represented in Gitter correctly: https://gitlab.com/gitterHQ/webapp/-/merge_requests/2027/diffs.
    • This can be accomplished by simply adding a virtualUser property to your chat message/post/tweet schema which holds the mxid, displayName, and avatar as an alternative to your author field. Then display the virtualUser whenever available over the author.
  • Add an application service to Gitter to bridge traffic in & out of Matrix: https://gitlab.com/gitterHQ/webapp/-/merge_requests/2041/diffs
    • This "application service" comes pre-packaged for you in many cases, so for example you can simply drop in a library like matrix-appservice-bridge in a Node.js application, and all of the Matrix talking complexity is handled for you.
  • Polish it!

In practice, Eric (lead Gitter dev) laid out the waypoints of the full journey:

  1. First big step was to add the concept of virtual users to Gitter. We could also have created a new Gitter user for every new matrix ID that appears, but tagging them as virtual users is a bit cleaner.
  2. Figuring out how to balance the Matrix traffic coming into/out of Gitter.
    1. Spreading the inbound load comes for free via our existing load-balancer setup (ELB) where we already have 8 webapp servers running the various services of gitter.im. We just run the Matrix bridge on those servers alongside each web and api process, and then the load-balancer’s matrix.gitter.im spreads out to the servers.
    2. Events from Matrix then hit the load balancer and reach one of the servers (no duplication when processing events).
    3. If something on Gitter happens, the action occurs on one server and we just propagate it over to Matrix (no duplication or locking needed).
  3. We have realtime websockets and Faye subscriptions already in the app which are backed by Mongoose database hooks whenever something changes. We just tapped into the same thing to be able to bridge across new information to Matrix as we receive it on Gitter.
  4. Hooking up the official Matrix bridging matrix-appservice-bridge library to use Gitter’s existing MongoDB for storage instead of nedb.
  5. Figuring out how to namespace the mxids of the gitter users:
    1. It’s nice to have the mxid as human readable as possible instead of just the numerical userId in your service.
    2. But if people can change their username in your service, you can’t change your mxid on Matrix. In the future, we’ll have portable accounts in Matrix to support this (MSC2787) but sadly these are still vapourware at this point.
    3. If you naively just switch the user’s mxid when they rename their username, then you could end up leaking conversation history between mxids(!)
    4. So we went with @username-userid:gitter.im for the Matrix ID to make it a bit more human readable but also unique so any renames can happen without affecting anything.
  6. For room aliases, we decided to change our community/room URI syntax to underscores for the room aliases, #community_room:gitter.im
  7. Figuring out how to bridge features correctly;
    1. Emoji - mapping between :shortcode: and unicode emojis
    2. Mapping between Gitter threaded conversations <-> Matrix replies
    3. Mapping between Matrix mentions and Gitter mentions
  8. Keeping users and room data in sync
    1. We haven’t gotten there yet, but the data comes through the same Mongoose hook and we can update the bridged data as they change on the Gitter end.

Meanwhile, the Matrix side of gitter.im is hosted by Element Matrix Services and is a plain old Synapse, talking through to Gitter via the Application Service API. An alternative architecture would be to have got Gitter directly federating with Matrix by embedding a “homeserver library” into it (e.g. embedding Dendrite). However, given Dendrite is still beta and assumes it is storing its data itself (rather than persisting in an existing backend such as Gitter’s mongodb), we went for the simpler option to start with.

It’s been really interesting to see how this has played out week by week in the Gitter updates in This Week in Matrix: you can literally track the progress and see how the integration came to life between Oct 9, Oct 23, Nov 6, Nov 27 and finally Dec 4.

Huge thanks go to Eric Eastwood, the lead dev of Gitter and mastermind behind the project - and also to Half-Shot and Christian who’ve been providing all the support and review from the Matrix bridging team.

What’s next?

First and foremost we’re going to be working through the “What remains” section of the list above: killing off the old bridge, sorting out plumbed rooms, hooking up DMs, importing old Gitter history into Matrix, etc. This should then give us an exceptionally low impedance link between Gitter & Matrix.

Then, as per our original announcement, the plan is:

In the medium/long term, it’s simply not going to be efficient for the combined Element/Gitter team to split our efforts maintaining two high-profile Matrix clients. Our plan is instead to merge Gitter’s features into Element (or next generations of Element) itself and then - if and only if Element has achieved parity with Gitter - we expect to upgrade the deployment on gitter.im to a Gitter-customised version of Element. The inevitable side-effect is that we’ll be adding new features to Element rather than Gitter going forwards.

Now, that means implementing some features in Matrix/Element to match...

  • Instant live room peeking (less than a second to load the webapp into a live-view of a massive room with 20K users!!)
  • Seamless onboarding thanks to using GitLab & GitHub for accounts
  • Curated hierarchical room directory
  • Magical creation of rooms on demand for every GitLab and GitHub project ever
  • GitLab/GitHub activity as a first-class citizen in a room’s side-panel
  • Excellent search-engine-friendly static content and archives
  • KaTeX support for Maths communities
  • Threads!

...and this work is in full swing:

The only bits which aren’t already progressing yet are tighter GL/GH integration, and better search engine optimised static archives.

So, the plan is to get cracking on the rest of the feature parity, then merge Gitter & Element together - and meanwhile continue getting the rest of the world into Matrix :)

We live in exciting times: open standards-based interoperable communication is on the rise again, and we hope Gitter’s new life in Matrix is the beginning of a new age of cross-project developer collaboration, at last escaping the fragmentation we’ve suffered over the last few years.

Finally, please do give feedback via Gitter or Matrix (or mail!) on the integration and where you’d like to see it go next!

Thanks for flying Matrix and Gitter,

-- The Matrix Team

This Week in Matrix 2020-12-04

04.12.2020 00:00 — This Week in Matrix Ben Parsons

Matrix Live 🎙

In which Nad & Matthew go through the new consolidated Element roadmap at https://github.com/vector-im/roadmap/projects/1

Dept of Status of Matrix 🌡️

Open Tech Will Save Us 9

Taking place NEXT WEDNESDAY, the ninth edition of my favourite monthly Open Tech-themed live video broadcast! Subscribe to the calendar on https://matrix.org/open-tech-will-save-us/ and check out the lineup:

  • Angie Gaudion, Coordinatrice at CHATONS, presenting Mobilizon: a new libre events management tool https://joinmobilizon.org/en/.
  • Ben Francis, Founder at Krellian, presenting the relaunch of WebThings (previously from Mozilla)
  • Nad Chishtie, Product Lead at Element, presents Matrix Spaces: a new vision for room groups in Matrix

Dept of Spec 📜

Spec

anoa offered:

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.

MSC Status

Merged MSCs:

  • No MSCs were merged this week.

MSCs in Final Comment Period:

  • No MSCs are in FCP.

New MSCs:

Spec Core Team

In terms of Spec Core Team MSC focus for this week, we've mostly been focusing on implementation. Thus the same 3 MSCs are in focus: MSC2844 (global versioning), MSC2366 (ready and done in key verification), and MSC2790 (modal widgets).

2020-12-04-3rO5x-stacked_area_chart.png

Anoa also provided a revised, more festive version of the chart, but it's too jolly to render inline, so needed to be linked instead.

New spec platform

wbamberg offered:

Still at https://adoring-einstein-5ea514.netlify.app/

  • Dean has started work on design updates for the spec

  • We've written replacements for all the templates and updated all template calls

  • We've implemented a mechanism for representing different spec versions and changelogs

Seriously progressing! Much more readable than what we have.

Dept of Servers 🏢

Dendrite / gomatrixserverlib

Dendrite is a next-generation homeserver written in Go

Neil Alexander's Dendrite mxid announced:

This week we released version 0.3.2, which contains some bug fixes. Changes this week include:

  • Checking if a server is allowed to see an event has been optimised somewhat

  • The ability to disable federation has been added

  • Application services are now sent correctly formatted client events

  • create-account has been simplified so that you can just point it to your Dendrite config

  • /unpeek support has been added (for MSC2753)

  • Event IDs are now calculated far less often in the roomserver, since this is an expensive operation

  • The federation sender now caches events in memory to reduce load on the database

  • The "since" parameter is now checked more thoroughly in the sync API, which led to a bug that could cause forgotten rooms to reappear (thanks to kaniini)

  • The polylith now proxies signing key requests through the federation sender correctly

Spec compliance is unchanged:

  • Client-server APIs: 58%, same as last week

  • Server-server APIs: 83%, same as last week

As always, feel free to join us in #dendrite:matrix.org for general Dendrite chat or #dendrite-dev:matrix.org for development discussion.

Synapse

Synapse is a popular homeserver written in Python.

callahad offered:

⚠️ The OpenSSL project has pre-announced a high severity issue that will be disclosed on Tuesday. We will be regenerating Docker images and releasing Synapse 1.24.0 and 1.23.1 on Tuesday following that announcement. Note that these releases contain security fixes of their own, and you are encouraged to upgrade.

In other news, Synapse 1.24.0rc2 is out and includes numerous bug fixes and internal improvements, especially around our OpenID Connect (OIDC) and SAML providers. Of note, the OIDC provider can now re-try generating MXIDs when there's a collision, reaching parity with the SAML provider.

We've been talking about MSC2403: Add "knock" feature for a few weeks, but the end is in sight: the PR is reviewed and the tests pass, so we're on track for including it in 1.25.0! We're also on the cusp of a finishing a new algorithm for calculating the auth chain difference in State Resolution v2, which should significantly reduce worst case performance.

Lastly, the deprecated /_matrix/client/*/admin Admin API endpoints have been removed. If you have tools which target these endpoints, please update them to use the /_synapse/admin URL prefix instead.

Homeserver Deployment 📥️

Docker ARM images for Dendrite

TR_SLimey offered:

The Docker ARM images for Dendrite have been updated for version 0.3.2. As always, they can be found in my docker repo at https://hub.docker.com/u/trslimey

Direct links:

I have also submitted a PR for automated multiarch docker builds for Dendrite, so if that gets accepted, no more unofficial images for me :P

YunoHost

Pierre told us:

YunoHost is an operating system aiming for the simplest administration of a server, and therefore democratize self-hosting.

Synapse integration had been updated to 1.21.2 (1.23.0 available in branch testing)

Element Web integration had been updated to 1.7.13 (1.7.14 available in branch testing)

Dept of Bridges 🌉

Gitter

Eric Eastwood reported:

We've soft-launched the new bridge to all public Gitter rooms! You can now find any public room on Gitter checking out the room directory for gitter.im or directly by replacing the forward slash in the URI part of the Gitter room with an underscore and joining #*:gitter.im(replace the *). If you come across any bugs, feel free check the list of known problems and create an issue. There will be a full announcement next week so stay tuned!

We've also added a few more features this week and now transform :emoji: syntax from Gitter to Unicode so they're compatible in Matrix, propagate the room topic/description, and some bug fixes around threaded conversations and replies. You can track our full progress from the GitLab epic for the native Gitter-Matrix bridge.

Massive milestone! Great work Eric, it's been such fast progress!

Dept of Clients 📱

Fractal

Alexandre Franke announced:

After I ran out of ways to break it, we finally merged Our Hugestest Merge Request Ever. This concludes the “Port to matrix-rust-sdk” saga (see these three links for previous episodes).

2020-12-04-2yOTb-01-celebration.png

Just kidding.

This was actually just the beginning, the ground work needed to lay the foundations for A Better Future. We already have a bunch of other merge requests to continue down that path. Hopefully there shouldn’t be such large ones in the (near) future. So what’s next? The work that landed this week was a switch to the SDK for almost all endpoint queries. Now need to handle the syncs with it. Later on, we will use it to handle storage, and then enable E2EE as well, but that will take quite some time.

Big progress from Fractal here! Will be good to hear more from the Fractal club to see what the ETA is for storage and E2EE.

Nheko

Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE (with the notable exception being device verification for now) and intends to be full featured and nice to look at

Nico (@deepbluev7:neko.dev) reported:

While I've actually not had much time to work on Nheko, I did at least fix most of the UTD (Unable to decrypt) issues, when sending encrypted messages. This has been a long standing annoyance, but it should now finally not be an issue anymore (well... at least it should not be caused by Nheko as often anymore)! With this there are only a few tickets left to finally bring E2EE out of beta in Nheko (SSSS and bootstrapping cross-signing, as well as a general review for potential issues).

And while I have actually not been doing much work, we actually had more Pull Requests (and merged them too) this week than I have ever seen before in Nheko! Thank you very much to everyone who contributed. A short summary:

  • Nheko now uses the Nheko icon from your system theme, if available

  • You can finally look at avatars in their full resolution!

  • Megolm sessions are now lazily loaded from disk.

  • The registration page now also has nice inline error messages, hinting at why a specific input is unacceptable.

  • Updates to the russian translation

  • The README now lists all required GStreamer plugins for voice and video calls.

It feels super great, when you have so many people contributing, I love it! <3

Nheko really seems to be doing something right when it comes to attracting PRs!

Element Update

As compiled by the team:

  • Public Roadmap
    • We now have a public roadmap! We’ll be covering more details on it in this week’s Matrix Live, as well as a blog post on element.io soon.
  • Spaces
    • Work has started in earnest on the web and android implementations.
      • On the web side T3chguy completed the Spaces creation flows and is now looking at the Space landing page and Space post-creation prompts.
        • We have noticed that prefixing any word with ‘Space’ makes it sounds 20% cooler.
      • The mobile MVP will be read only and Valere will start with the ability to respond to invites and select Spaces.
    • So far it is looking very promising with lots of feedback from user testing informing the design.
  • Social login
    • We plan to ship this as a cross client feature this month. This week we completed the implementations for web and android with iOS and Synapse ongoing.
  • VoIP
    • Line 1 / 2 support is almost ready on web: ironing out some bugs and cosmetic issues, then PR will be ready for review. VoIP V1 support and hold/resume is getting close on Android & iOS, along with stability fixes.

Web

  • Element Web 1.7.15-rc.1 is now available at https://staging.element.io, including
    • Improved voice / video call UI
    • Notifications are now ignored from rooms you've been active in recently
    • Login / registration flows redirect inside the app if you are already logged in
  • Supporting tools like CI scripts are now using the correct branding
  • React SDK PRs now have experimental preview sites, which makes it easy to try out changes during review

iOS

  • We can now use XCode 12.2 to build all projects
  • We are still working on 2 big changes:
    • Introducing a background sync service to display notifications quicker in the timeline
    • Introducing XCodeGen to stop having bad merge conflicts in pbxproj files

Android

  • Release 1.0.11 in production (roll out to 10% for now)
  • Feature Parity / URL preview
  • Coming next Social logins
  • Bug fixes

Aleks Kissinger added:

LaTeX support will be available under Labs in the next release of Element: https://github.com/vector-im/element-web/blob/develop/docs/labs.md

cryptovoxels client

Matthew offered:

cryptovoxels released a cute little preact matrix client for all your VR needs: https://github.com/cryptovoxels/chat

We don't have much information about this one yet, other than that cryptovoxels have been interested in developing a Matrix client for their world for a while.

Dept of Bots 🤖

Hemppa

Hemppa the Bot is a multipurpose bot for writing modules super easily in Python.

Cos offered:

Hemppa the Bot is a multipurpose bot for writing modules super easily in Python. Last week Hemppa received a useful feature of printing files! You can set up a room for a specific printer and invite anyone you wish to it to allow printing. Any attachments uploaded to the room will be automatically printed. It's actually one of the easiest way to print from a phone and very useful if you have a printer that does not have drivers for modern Windows or MacOS versions. https://github.com/vranki/hemppa#printing

Matrix-Alertmanager

A bot to receive Alertmanager webhook events and forward them to chosen rooms.

jaywink announced:

Matrix-Alertmanager has a new release v0.4.0 with improved formatting of alert messages and an update of Node to v14. Find more information here.

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1tum.de451
2nerdsin.space568.5
3envs.net572
4matrix.sp-codes.de785
5mgp.ax.lt888
6selea.se896
7casavant.org918
8nobelium.no1089.5
9chatserver.ca1189
10mailstation.de1254

Final Thoughts 💭

This is a pretty cool, Matrix-ish video! Doug shows Element-iOS and Nio running side-by-side, natively (!) on an M1 Mac Mini.

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

This Week in Matrix 2020-11-27

27.11.2020 00:00 — This Week in Matrix Ben Parsons

Matrix Live 🎙

Dept of Spec 📜

Spec

anoa reported:

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.

MSC Status

Closed MSCs

Merged MSCs:

MSCs in Final Comment Period:

  • No MSCs are in FCP.

New MSCs:

Spec Core Team

In terms of Spec Core Team MSC focus for this week, we're switching from QR Code verification to key verification additions. Otherwise things remain the same: MSC2844 (global versioning), MSC2366 (ready and done in key verification) and MSC2790 (modal widgets).

2020-11-27-4wL5N-stacked_area_chart.png

Dept of Servers 🏢

Synapse

callahad offered:

On behalf of the Americans on the Synapse team: Happy Thanksgiving! This past week saw us land some improvements to the OpenID Connect SSO provider, bringing it closer to parity with the SAML provider. In particular, #8801 made it possible to gracefully handle username collisions when creating a new account over SSO.

We've also merged a pull request (#8785) from @dklimpel which removes deprecated paths for the Synapse Admin APIs. As of the next Synapse release, all Admin APIs will be accessibly only under the /_synapse/admin/v1 path prefix.

Otherwise, much of our time has been spent on getting pull requests like #6739, implementing MSC2403: Add "knock" feature, ready to merge. It's coming to a Synapse release soon!

Speaking of releases, we're looking forward to cutting a first release candidate of 1.24.0 early next week; keep your eyes peeled for that!

I'd be remiss in not mentioning that we started the week by disclosing CVE-2020-26890, which is a denial of service affecting Synapse versions prior to 1.20.0. If you've not upgraded, please do.

Dept of Bridges 🌉

Gitter

Eric Eastwood offered:

Lots of features added this week including Gitter threaded conversations and Matrix reply bridging, image/file uploads, transforming mentions so notifications flow, and better looking flairs!

Not everything is available on production yet but you can try out the bridge for yourself at https://gitter.im/gitter/testing-matrix-bridge: #5faa0809d73408ce4ff3ad8e:gitter.im

We'll be working through some of the bridge oddities and hoping to open up the bridge to all public rooms on Gitter soon. You can track our full progress from the GitLab epic for the native Gitter-Matrix bridge.

2020-11-27-sCDuB-ScreenShot2020-11-27at1.37.40AM.png

MX-Puppet-VK (Na Zdorovie!)

Alexey offered:

The MX-Puppet-VK bridge for VK (Vkontakte) social network via this commit have added initial support for User tokens. This means that we now can use it for puppeting the real user account, to automatically bridge all "VK Messenger" private dialogs and groups into Matrix. Previously bridge worked only with VK community tokens as bot, to bridge only conversation with specific communities.

To get the VK User token without registering new app (and waiting approve from admins), users can reuse the "Kate Mobile" app token (or other app, that have access to Messages API) and receive it via https://vkhost.github.io/ service.

Support room: #mx-puppet-vk:inex.rocks

matrix-sms-bridge

Benedict offered:

The matrix-sms-bridge can finally be used with an Android smartphone as SMS-Gateway instead of gammu and a modem. @rebekka:imbitbu.de developed the app android-sms-gateway-server, that the bridge uses to send and receive sms messages. The new bridge version is used in combination with the app in production (medical care centre) for a week now. We have a throughput of 50-100 SMS message a day. The update is also integrated in matrix-docker-ansible-deploy.

Dept of Clients 📱

Nheko

Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE (with the notable exception being device verification for now) and intends to be full featured and nice to look at

Nico (@deepbluev7:neko.dev) offered:

The new completer is now on master. Please try it out and report any issues you may find. It now actually inserts a user link, keeps a per room message history and should look quite a bit nicer and be faster, when completing emojis or user names.

There were also quite a few cleanups, most of them done by Kirillpt! Some spacing here and there, saturation changes, the login page should now be less confusing, when you don't enter a valid mxid and toggles should now not get stuck in confusing intermediate states. There is also a russian translation now and some VOIP fixes. Pretty busy week, but I like it!

2020-11-27-LCo9S-Screenshot_20201127_162000.png

FluffyChat

FluffyChat is a cute cross-platform matrix client. It is available for Android, iOS, Web and Desktop.

krille reported:

FluffyChat 0.23.0 is out now with password recovery options and a lot of minor fixes and UX improvements. Now also available on Flathub!! 🙂 Learn more at: https://fluffychat.im

2020-11-27-nO6wZ-fluffychat-screenshot.png

Element

Nad offered:

In development

Improving usability

On mobile, we’re busy adding better suggestions to empty/starting views of screens, improving searching for contacts, improving adding other users by QR code scanning, and improving deep link integrations on Android. Expect releases for iOS & Android soon!

Spaces

Spaces UI designs & user research testing are underway, with initial development for the Element Web client kicked off too. We've been doing research & testing in private to avoid advance bias from the people we're testing with, but hope to share things publicly soon.

Social login

Social login is in review for Element Web, with synapse & mobile support hot on the tails for next week prior to releasing.

VoIP

We’ve been focusing on implementing UI for call holding, and general UI polish across the VoIP experience, to be released soon!

Releases

Element Web

Element Web 1.7.14 was released this week, which includes:

  • Several tweaks and improvements to the room list filter

  • Improved registration based on user feedback

  • Improved invite / create DM flow.

  • Future releases will no longer publish element-web to NPM, and the package is marked as deprecated

  • Renamed build artifacts from riot-.tar.gz to element-.tar.gz

  • Experimental LaTeX maths rendering support is available via a labs flag on develop. Thanks to akissinger, thosgood, uhoreg, and others who have worked on various pieces of this puzzle!

Element iOS

Element iOS 1.1.1 has been released:

  • It contains contains several major bug fixes, including end-to-end encryption bug fixes.

  • Full history available on GitHub viewing the 1.1.1 & 1.1.0 tags.

Element Android

The Element Android 1.0.11 release is pending:

  • We're working through final bug fixes.

  • Separately, we’re also working on adding missing features to the client like; managing room aliases, room history visibility, join rules, etc.

  • In the SDK, the migration of the services API from MatrixCallback to coroutines is going well, tracked here.

Vtx

CraftedCart offered:

In addition to boring stuff like being able to persist events (hopefully) right, so we don't have to initial-sync every launch, I've been making a start towards multi-account support!

You can sign in as multiple users and switch between the "active" user you want to send messages as - iit's a bit flaky for now, esp. when scrolling up to read past messages, but hey it's a start I guess

Asked if they were using an SDK:

Preeetty much doing it from scratch, though I am loosely basing the matrix-api side of it on Ruma

Source is at https://gitlab.com/CraftedCart/vtx

Dept of SDKs and Frameworks 🧰

Ruma

Ruma is a set of Rust library crates around Matrix.

jplatte announced:

We gained a new contributor, kiwidu38, who added the server-default push rules, paving the way to Conduit supporting push notifications.

I improved compatibility with some commonly-broken DAG events and added rudimentary support for message edits to help Fractal finish their move onto matrix-rust-sdk.

Dept of Guides 🧭

Matrix introduction series (German)

Samuel offered:

I published the second part of the Matrix introduction series on my blog (in German). It deals with end-to-end encryption, contact verification and cross signing. https://blog.sp-codes.de/sichere-chats-mit-matrix-matrix-teil-2/.

I have not read this series so I must trust that Samuel is saying nice things! Anyway it seems like a useful resource for telling kameraden about getting started with Matrix.

Matrix in the News 📰

Niklas Zender offered:

Today there is a German article about Famedly on Gründerszene -> https://www.gruenderszene.de/health/famedly-kommunikation-app-gesundheitswesen. Famedly is developing a solution based on Matrix to enable healthcare providers to communicate securely with each other and to facilitate intersectoral processes such as patient transfers or the integration of IomT devices. We have also almost completed a successful financing round and hope to support the Matrix ecosystem more and more. It would be so cool to use FluffyChat, Element, or a Matrix Messenger of your choice to start a video consultation with the doctor of your choice, or to be integrated into existing clinical communication flows concerning you!

Congratulations to Famedly on the progress they're making! If you're ready to graduate from TLAs (Three-Letter-Abbreviations) to the next stage, you'll be please to learn "IomT", which means "Internet of medical Things".

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1maescool.be357
2heitkoetter.net367
3matrix.47q.de445
4matrix.thedisco.zone455
5kif.rocks477.5
6matrix.sp-codes.de605
7matrix.rubberboat.xyz713
8nuclearlemons.uk732.5
9casavant.org754
10dodsorf.as764

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!