Category – General
106 posts tagged with "General" (See all categories)

Synapse 1.11.0 released

2020-02-24 — General — Matthew Hodgson

Hi all,

Synapse 1.11 landed on Friday (sorry for running late on blogging the release notes!)

The main change is to introduce an experimental API MSC2432 for managing aliases for rooms on your local server. In Synapse 1.10 we removed support for m.room.aliases events, which were a way to try to track which aliases a room had on the room itself (but were vulnerable to abuse). In this release we've re-added the ability to query which aliases a given server has for the room, giving visibility for managing aliases, without having them spray all over the room state itself. Riot/Web 1.5.10 supports the new API, giving a way to manage aliases on your local server while we finish off the remaining work to improve alias safety & maintenance.

We've also changed the default power levels for new rooms so that room upgrades and ACLs require you to be an Admin (PL100), and invites in public rooms now require you to be a moderator (PL50).

Get the new release from github or any of the sources mentioned at https://github.com/matrix-org/synapse/blob/master/INSTALL.md.

Synapse 1.11.0 (2020-02-21)

Improved Documentation

  • Small grammatical fixes to the ACME v1 deprecation notice. (#6944)

Synapse 1.11.0rc1 (2020-02-19)

Features

  • Admin API to add or modify threepids of user accounts. (#6769)
  • Limit the number of events that can be requested by the backfill federation API to 100. (#6864)
  • Add ability to run some group APIs on workers. (#6866)
  • Reject device display names over 100 characters in length to prevent abuse. (#6882)
  • Add ability to route federation user device queries to workers. (#6873)
  • The result of a user directory search can now be filtered via the spam checker. (#6888)
  • Implement new GET /_matrix/client/unstable/org.matrix.msc2432/rooms/{roomId}/aliases endpoint as per MSC2432. (#6939, #6948, #6949)
  • Stop sending m.room.aliases events when adding / removing aliases. Check alt_aliases in the latest m.room.canonical_alias event when deleting an alias. (#6904)
  • Change the default power levels of invites, tombstones and server ACLs for new rooms. (#6834)

Bugfixes

  • Fixed third party event rules function on_create_room's return value being ignored. (#6781)
  • Allow URL-encoded User IDs on /_synapse/admin/v2/users/<user_id>[/admin] endpoints. Thanks to @NHAS for reporting. (#6825)
  • Fix Synapse refusing to start if federation_certificate_verification_whitelist option is blank. (#6849)
  • Fix errors from logging in the purge jobs related to the message retention policies support. (#6945)
  • Return a 404 instead of 200 for querying information of a non-existant user through the admin API. (#6901)

Updates to the Docker image

  • The deprecated "generate-config-on-the-fly" mode is no longer supported. (#6918)

Improved Documentation

  • Add details of PR merge strategy to contributing docs. (#6846)
  • Spell out that the last event sent to a room won't be deleted by a purge. (#6891)
  • Update Synapse's documentation to warn about the deprecation of ACME v1. (#6905, #6907, #6909)
  • Add documentation for the spam checker. (#6906)
  • Fix worker docs to point /publicised_groups API correctly. (#6938)
  • Clean up and update docs on setting up federation. (#6940)
  • Add a warning about indentation to generated configuration files. (#6920)
  • Databases created using the compose file in contrib/docker will now always have correct encoding and locale settings. Contributed by Fridtjof Mund. (#6921)
  • Update pip install directions in readme to avoid error when using zsh. (#6855)

Deprecations and Removals

  • Remove m.lazy_load_members from unstable_features since lazy loading is in the stable Client-Server API version r0.5.0. (#6877)

Internal Changes

  • Add type hints to SyncHandler. (#6821)
  • Refactoring work in preparation for changing the event redaction algorithm. (#6823, #6827, #6854, #6856, #6857, #6858)
  • Fix stacktraces when using ObservableDeferred and async/await. (#6836)
  • Port much of synapse.handlers.federation to async/await. (#6837, #6840)
  • Populate rooms.room_version database column at startup, rather than in a background update. (#6847)
  • Reduce amount we log at INFO level. (#6833, #6862)
  • Remove unused get_room_stats_state method. (#6869)
  • Add typing to synapse.federation.sender and port to async/await. (#6871)
  • Refactor _EventInternalMetadata object to improve type safety. (#6872)
  • Add an additional entry to the SyTest blacklist for worker mode. (#6883)
  • Fix the use of sed in the linting scripts when using BSD sed. (#6887)
  • Add type hints to the spam checker module. (#6915)
  • Convert the directory handler tests to use HomeserverTestCase. (#6919)
  • Increase DB/CPU perf of _is_server_still_joined check. (#6936)
  • Tiny optimisation for incoming HTTP request dispatch. (#6950)

Synapse 1.10.0 released

2020-02-12 — General — Neil Johnson

The whole Matrix project is racing towards enabling e2ee by default. Synapse is no different and v1.10.0 contains multiple e2ee UX improvements, as well as a bug fix that prevented cross signing requests over federation to work reliably.

If any of your users are on the bleeding edge and have already started using cross signing (by enabling labs flags in Riot), then it will be necessary for them to force Synapse to re-send device updates by renaming all of their devices.

We've also included a temporary fix to address alias abuse. The idea is that until #6898 lands, servers will refrain from sharing events of type m.room.aliases with clients. Most admins will not be affected, but if you are present in rooms subject to alias abuse, then upgrading provides a pragmatic short term solution.

Finally, as of this release Synapse validates client_secret parameters in the Client-Server API as per the spec. See #6766 for details.

Get the new release from github or any of the sources mentioned at https://github.com/matrix-org/synapse/blob/master/INSTALL.md.

Changelog since Synapse 1.9.0

Synapse 1.10.0 (2020-02-12)

WARNING to client developers: As of this release Synapse validates client_secret parameters in the Client-Server API as per the spec. See #6766 for details.

Updates to the Docker image

  • Update the docker images to Alpine Linux 3.11. (#6897)

Synapse 1.10.0rc5 (2020-02-11)

Bugfixes

  • Fix the filtering introduced in 1.10.0rc3 to also apply to the state blocks returned by /sync. (#6884)

Synapse 1.10.0rc4 (2020-02-11)

This release candidate was built incorrectly and is superceded by 1.10.0rc5.

Synapse 1.10.0rc3 (2020-02-10)

Features

  • Filter out m.room.aliases from the CS API to mitigate abuse while a better solution is specced. (#6878)

Internal Changes

  • Fix continuous integration failures with old versions of pip, which were introduced by a release of the zipp library. (#6880)

Synapse 1.10.0rc2 (2020-02-06)

Bugfixes

  • Fix an issue with cross-signing where device signatures were not sent to remote servers. (#6844)
  • Fix to the unknown remote device detection which was introduced in 1.10.rc1. (#6848)

Internal Changes

  • Detect unexpected sender keys on remote encrypted events and resync device lists. (#6850)

Synapse 1.10.0rc1 (2020-01-31)

Features

  • Add experimental support for updated authorization rules for aliases events, from MSC2260. (#6787, #6790, #6794)

Bugfixes

  • Warn if postgres database has a non-C locale, as that can cause issues when upgrading locales (e.g. due to upgrading OS). (#6734)
  • Minor fixes to PUT /_synapse/admin/v2/users admin api. (#6761)
  • Validate client_secret parameter using the regex provided by the Client-Server API, temporarily allowing : characters for older clients. The : character will be removed in a future release. (#6767)
  • Fix persisting redaction events that have been redacted (or otherwise don't have a redacts key). (#6771)
  • Fix outbound federation request metrics. (#6795)
  • Fix bug where querying a remote user's device keys that weren't cached resulted in only returning a single device. (#6796)
  • Fix race in federation sender worker that delayed sending of device updates. (#6799, #6800)
  • Fix bug where Synapse didn't invalidate cache of remote users' devices when Synapse left a room. (#6801)
  • Fix waking up other workers when remote server is detected to have come back online. (#6811)

Improved Documentation

  • Clarify documentation related to user_dir and federation_reader workers. (#6775)

Internal Changes

  • Record room versions in the rooms table. (#6729, #6788, #6810)
  • Propagate cache invalidates from workers to other workers. (#6748)
  • Remove some unnecessary admin handler abstraction methods. (#6751)
  • Add some debugging for media storage providers. (#6757)
  • Detect unknown remote devices and mark cache as stale. (#6776, #6819)
  • Attempt to resync remote users' devices when detected as stale. (#6786)
  • Delete current state from the database when server leaves a room. (#6792)
  • When a client asks for a remote user's device keys check if the local cache for that user has been marked as potentially stale. (#6797)
  • Add background update to clean out left rooms from current state. (#6802, #6816)
  • Refactoring work in preparation for changing the event redaction algorithm. (#6803, #6805, #6806, #6807, #6820)

The 2019 Matrix Holiday Update!

2019-12-24 — General — Matthew Hodgson

Hi all,

Every year we do an annual wrap-up and retrospective of all the things happening in the Matrix core team - if you’re feeling particularly curious or bored you can check out the 2015, 2016, 2017 and 2018 editions for context. The idea is to look at the bigger picture trends in Matrix outside of the weekly TWIM posts to get an idea of the stuff which we made progress on, and the stuff which still remains.

That said, it’s hard to know where to start - Matrix accelerated more than ever before in 2019, and there’s been progress on pretty much all battlefronts. So as a different format, let’s take the stuff we said we had planned for 2019 from the end of last year’s update and see what we actually achieved...

2019: the immediate priorities

So, our immediate priorities for 2019 were:

  • r0 spec releases across the board (aka Matrix 1.0)
  • Implementing them in Synapse

✅ Well, unless you’ve been floating in a sensory deprivation tank for the last year, hopefully you spotted that Matrix (as a protocol) finally exited beta - starting off with the announcement at FOSDEM in February of the first stable release of the Server-Server API, alongside the Synapse 0.99.x series as we began the process of migrating to the 1.0 APIs.

Specifically this meant killing off self-signed certificates, adding .well-known server discovery and implementing room version semantics so we could upgrade the underlying room version algorithm to fix the residual flaws. This culminated in June with the official release of Matrix 1.0 - now including the remaining APIs and a stable release of Synapse 1.0. The emphasis was on addressing all the main pre-1.0 design flaws rather than adding features or performance, but with 1.0 out the door at last we’ve been able to make progress there too.

  • Landing the Riot redesign

✅ The full redesign of Riot’s UI on Web/Desktop landed shortly after FOSDEM in Feb with The Big 1.0. Cosmetically we got most of the new look & feel in place, and have had very positive feedback overall - although some of the UX thinkos of the old app remain and coming up on the radar for fixing.

  • Finalising the Matrix.org Foundation

✅ This happened too, coincident with releasing Matrix 1.0 in June - read all about it at https://matrix.org/foundation. So far the governance and legal infrastructure the Foundation provides has helped the project significantly, and while it was a mammoth task to organise, we’re very glad it’s here! Huge thanks go out to Jon, Ross and Jutta for agreeing to join the foundation as Guardians - they have been excellent in patiently listening to the various dramas of the year and ensuring Matrix’s neutrality and that we keep an even keel.

  • Landing all the new E2E encryption UX and features

The good news on E2E encryption is that we’ve been making solid progress throughout the year - the bad news is that we are still yet to turn it on by default. Progress updates for the various pieces of the puzzle are as follows:

  • ✅ The final UX is pretty much locked down (after several iterations as we try to tread the balance between trustworthiness and security) - here’s a sneak preview of what we’re aiming at.

  • 🏗 Cross-signing is the single biggest remaining piece of work in progress - letting users attest to the trustworthiness of their own devices, so you only ever have to trust a given user once rather than trusting all their devices individually. We gave a very early demo of an experimental implementation back at FOSDEM in Feb, inspired by some of the initial spec proposal at MSC1680 (MSC = Matrix Spec Change, our process for evolving Matrix).

    However, having played with it a bit, MSC1680 turned out to be too generic and complicated (it worked by the user signing a device with any other of their devices, building a twisted maze of which device vouched for which) - and we replaced it with MSC1756, which shifts the model to be the simpler “the user has a key, which they use to sign their devices”. However, this in turn requires more infrastructure - you need somewhere secure to store your signing key, which prompted MSC1946 - Secure Secret Storage & Sharing (SSSS): the ability to sync your signing key between devices by storing it (encrypted, of course) on the server.

    Meanwhile, it also became obvious that the primitives for key verification needed to be improved too: introducing verification by emoji comparison (MSC1267) and QR codes (MSC1543), and switching key verification to be performed in the context of a DM (MSC2241) so that you can see your verification history, find verifications, and easily dip in and out of verifying users as needed.

    Whilst everyone else was panicking about Matrix 1.0 and associated baggage, Uhoreg was off in the wilderness plugging his way through all of this - iterating on the design, speccing it and implementing it in synapse and matrix-js-sdk, complete with a test jig to demonstrate it all working (part 1 and part 2). Over the last few months the rest of the team has joined him though, and we’ve been frantically working away implementing it all on both Riot/Web, iOS & RiotX/Android. For instance - here’s verification happening in DM between Riot/Web & RiotX a few weeks ago, and here’s a very early (unskinned) cut of verification happening in Riot/Web’s RightPanel a few days ago.

    We were hoping to get cross-signing ready for the end of 2019, but in practice we’re now sprinting to get it done by FOSDEM 2020 in Feb - not least because we have a main-stage talk proposed to tell everyone how we landed it and turned on E2E by default... ;)

  • ✅ Support for non-E2E clients. The last thing we want is to make it impossible to write a simple Matrix client, or to suddenly excommunicate (hah) all the existing Matrix bots & bridges which haven’t implemented E2E. To this end, poljar created pantalaimon - our very own Matrix daemon, which can sit in the background and offload all your E2EE from your Matrix client by acting as a transparent Matrix proxy which magically encrypts everything. Built on matrix-nio and asyncio python3, We use it in production today for running various bots and it works excellently.

  • ✅ Support for search in E2E rooms. Hot off the heels of pantalaimon’s success, poljar also created seshat - a native library for clientside indexing encrypted Matrix events written in Rust, powered by the tantivy full-text search engine. (pantalaimon also has support for indexing via tantivy, which involved contributing python bindings for tantivy, but we ended up going with Rust so we could embed it natively in as many Matrix clients as possible). Seshat is particularly cool in that the indexes themselves are encrypted in on disk - and in future could even be synced between clients using SSSS so you don’t have to reindex your messages every time you log in on a new device. Seshat is implemented behind a labs flag on Riot/Desktop and it will ship as soon Riot/Desktop’s build pipeline is fully updated to support native modules (which will also unlock other goodies, such as using faster/safer native E2E primitives, safer key storage, and Discord-style keyboard-shortcuts for VoIP).

  • 🏗 Fixing “unable to decrypt” errors. We’ve done big sprint over the last month or so to track down the final straggling causes of unable to decrypt errors. Some of these are legitimate bugs (e.g. https://github.com/matrix-org/synapse/issues/6399) - but many are artefacts of the current architecture: for instance, if the sender has no way to know your device was in the room when it encrypted a message, you won’t be able to decrypt. We’re addressing this by improving better error messages and feedback so the user isn’t surprised by what’s going on (aiming for Jan) - and in future we’ll have to revisit E2E’s fundamentals to ensure that it’s impossible to receive a message without also receiving the key to decrypt it.

  • ✅ Support for push notifications in E2E rooms. This is kinda solved right now by having all clients get (silently) pushed whenever they receive a message in an E2E room with push enabled, and relying on the client to be woken up by the push in order to decrypt the message in order to display the push notification. However, this is battery intensive, and we could probably do better - but this isn’t a blocker for going live.

  • 🏗 Support for FilePanel and NotifPanel in E2E rooms. Seshat should fix this by indexing all your messages (and so tracking whether they contain pushes or files, and populating up your local view of your file & notif panels respectively) - just need to ensure it’s hooked up.

...and that’s where things stand right now on E2E by default. We’ll start turning it on by default for private rooms as soon as the UX has landed (probably starting first with new DMs and private rooms, prompting the user in case they want to opt out - and then migrating existing ones). It’s worth noting that we have poured a lot of work into E2E encryption now - often to the detriment of the rest of Matrix; our rich featureset and decentralisation has combined to make this a tough nut to crack, but the end is in sight. Thanks to all for your patience and support while we’ve been working through this.

That takes us to the end of the stuff we planned to prioritise in 2019 - but what about the more speculative medium-term stuff which was on the menu this time last year?

2019: the medium-term priorities

  • Reworking and improving Communities/Groups.

We have some really promising UX work and a fairly early spec proposal (MSC1772), but work in earnest hasn’t kicked off yet. It’s going to be one of the next big projects though.

  • Reactions.

✅ Riot now has Reactions! 🎉🎉🎉 The only remaining work is to finish the remaining rough edges of the spec proposal (MSC1849) and actually land them in the Matrix spec proper.

  • E2E-encrypted Search

Seshat exists! (see above)

  • Filtering. (empowering users to filter out rooms & content they're not interested in).

✅ We’ve ended up thinking lots in 2019 about empowering users to filter content. The main impetus has been to ensure that users and communities can filter out abuse (on their own terms), and also to start building infrastructure which can be used for folks to share their own filters. Over the last few months, this has started to take concrete form - with the arrival of MSC2313 “Moderation policies as rooms”, and Mjolnir - a bot you can run to enforce moderation policies on your rooms. It’s all quite early, but we expect a lot more work in this space over the coming year (and it’s wryly amusing that Twitter has also woken up to it being an interesting problem needing to be solved.)

  • Extensible events

Sorry folks; no progress here since a flurry of spec work (MSC1767) back in Jan 2019. The good news is that the spec proposal seems to be relatively well received. The bad news is that we haven’t had bandwidth to finish reviewing it, implementing it and migrating it anywhere. It blocks a bunch of really useful stuff in Matrix, and there are users willing to pay for it (via New Vector) - we’ll get to it as soon as we can.

  • Editable messages.

✅ These landed too and are a thing of joy! Just need to merge MSC1849.

  • Extensible Profiles (we've actually been experimenting with this already).

Similar to Extensible Events, there was a flurry of spec work (MSC1769) back in Jan, but little progress since. This will also unlock a lot of really useful features - e.g. custom status, custom profile data, social timeline rooms etc. We’ll likely get to it shortly after communities work.

  • Threading.

🏗 So we actually landed label-based threading (MSC2326) in Synapse 1.6, but it’s not exposed in Riot yet (or elsewhere). It doesn’t have quite the same semantics as Slack-style threading; the idea is to filter down your room based on which messages are tagged as part of a given topic. However, it’s very powerful, and it’ll be fun to add it to Riot at some point in 2020. Meanwhile, better-than-label-based-threading is also on the cards, although slightly lower priority than some of the other stuff in this section.

  • Landing the Riot/Android rewrite

🏗 As you probably know, RiotX is a full rewrite of Riot/Android in Kotlin using modern AndroidX and Jetpack idioms - and it entered beta back in June. Since then we’ve been frantically working away on both playing catch-up with the old app… as well as implementing all the new stuff (reactions, edits, new E2E verification, cross-signing etc) which makes no sense to waste time adding in Riot/Android, but also pushes out the timeline on RiotX itself.

We’re currently sprinting to try to get RiotX ready for FOSDEM in February - hopefully users will have felt the app starting to really stabilise over the last few months (it even supports breadcrumbs now!)

  • Considering whether to do a similar overhaul of Riot/iOS

🏗 It’s cheating a bit, but Manu (the lead developer on Riot/iOS and delivery manager of Riot/Mobile in general) has been hacking on an entirely new client called Messagerie in his spare time, using SwiftUI. The idea of throwing away the whole UI layer and replacing it with the latest best practices sounds suspiciously like RiotX - it’ll be interesting to see how RiotX/iOS takes shape next year!

  • Scaling synapse via sharding the master process

We ended up bottlenecked on IO rather than CPU in 2019, and as a result we worked on splitting synapse’s database across multiple database instances on a per-table granularity. However, the master process itself doesn’t shard yet; so we’re now bottlenecked on CPU and need to get on and do this asap to unlock further Synapse scalability for mega-monolithic-deployments like the Matrix.org homeserver.

  • Bridge UI for discovery of users/rooms and bridge status

🏗 There’s been a bit of movement in the last few weeks on this, but nothing concrete yet.

  • Bandwidth-efficient transports

✅ We finished the 100bps CoAP transport proof-of-concept for Matrix, demoed it at FOSDEM and shipped it in March. However, we haven’t progressed it much further; it really needs a corporate sponsor who wants to fund work to finish it off and bake it properly into Matrix. If you’re interested, please get in touch.

  • Bandwidth-efficient routing

🏗 We also did a bunch of related work on bandwidth-efficient routing, which sadly hasn’t been released yet. However, it’s interesting to note that the Decentralized Systems and Network Services Research Group at Karlsruhe Institute of Technology’s Institute of Telematics has been looking into this space too - c.f. their A Glimpse of the Matrix paper, which ponders very similar problems.

  • Getting Dendrite to production.

🏗 Dendrite work has been bubbling away in the background thanks to Anoa, Brendan, cnly (our GSoC dendrite contributor) and others. Inevitably most of our bandwidth has gone into getting Synapse to 1.0 and making sure it’s fit for purpose, but we want and need to keep Dendrite alive for next-generation purposes - and in fact New Vector is hiring new people to work on it in 2020.

  • Inline widgets (polls etc)

🏗 We have an MSC (MSC2192), but not an implementation.

  • Improving VoIP over Matrix.

Very little progress here, frustratingly. Jitsi has been upgraded and conference calls should kick ass these days (let us know if they don’t), but 1:1 needs a lot of love. Hopefully we’ll get to it in 2020.

  • Adding more bridges, and improving the current ones.

Lots of bridging progress in 2020 - all new puppeting Slack support; huge fixes to the IRC bridge (including shifting to Postgres at last); Bifrost (the XMPP bridge) progressed too, and there’s been loads of community bridging work around WhatsApp, Discord and others.

  • Account portability
  • Replacing MXIDs with public keys

We’ve just started looking at implementing these seriously via MSC1228 (as of last week) - expect progress in 2020.

So that sums up progress on the medium term menu - as you can see, a bunch actually happened; a bunch made progress; a few didn’t happen at all.

2019: the longer-term priorities

Finally, on the longer term radar:

  • Shared-code cross-platform client SDKs (e.g. sharing a native core library between matrix-{js,ios,android}-sdk)

No progress here. Instead, all three main platforms have continued to write and maintain their own platform-specific SDKs for now. Seshat however will be the first piece of native rust code shared across all 3 platforms - let’s see how that goes first...

  • Matrix daemons (e.g. running an always-on client as a background process in your OS which apps can connect to via a lightweight CS API)

Pantalaimon lives!

  • Push notifications via Matrix (using a daemon-style architecture)

No progress here, unless you count the CoAP low-bandwidth work. However, Bubu (also Riot/Android Fdroid maintainer) has been working on a project called OpenPush which looks to help in this space (albeit not built on Matrix, but could be used by Matrix). There are a few other related projects. If someone wants to build this on top of Matrix + CoAP please get in touch asap!

  • Clientside homeservers (i.e. p2p matrix) - e.g. compiling Dendrite to WASM and running it in a service worker.

🏗 Work is actually happening on this currently. Dendrite has successfully compiled to WASM and runs, and we’ve had it (almost) talking HTTP tunnelled over libp2p as part of P2P Matrix experiments. In 2020 we’re going to be investing a lot in P2P Matrix - to give users full control of their communication without even having to run a server, and also to simplify onboarding and account portability enormously. We have a talk about this accepted for FOSDEM 2020 (The Path to P2P Matrix) and we’re actively (frantically) hacking on Dendrite to make it happen - keep an eye out for how things develop!

  • Experimenting with MLS for E2E Encryption

🏗 Now that E2E-by-default has entered the “it works! let’s land it in Riot asap” phase, Uhoreg has had some time to start thinking about the longer term future of encryption in Matrix. MLS (Messaging Layer Security) is the IETF’s initiative to define a standard mechanism for end-to-end-encrypted group chats, which has some major algorithmic improvements over Olm/Megolm and the Double Ratchet Algorithm as used by Signal. The catch is that it doesn’t work at all well with decentralisation - however, we’ve been working with them to try to ensure MLS can work in a decentralised world. More recently, uhoreg has had a chance to think a lot more about this and we’re working on a proposal for Decentralised MLS which builds on plain MLS while also giving the semantics needed for Matrix. It’s all very experimental at this point (and the proof-of-concept implementation is written in Julia!) - but looks promising. We’ll share more asap, and will certainly be investing more time in this in 2020..

  • Storing and querying more generic data structures in Matrix (e.g. object trees; scene graphs)

Sadly no progress here :(

  • Alternate use cases for VR, IoT, etc.

...and none here either.

So, of all the myriad things on our radar for 2019 (as of Dec 2018), hopefully this gives some idea of where we hit the mark.

2019: the unpredictable bits

However, there’s also a tonne of other stuff which happened which wasn’t explicitly on the radar. On the synapse side, we finished fully migrating from Python 2 to Python 3, and started using asyncio and all the latest Python 3 goodies! We finally implemented configurable history retention for servers and rooms! We even implemented self-destructing messages in Synapse (not that Riot exposes them yet). And there has been loads of optimisation and performance work since 1.0 landed in June.

On the ops side, we overhauled all our ops processes and security after the Matrix.org datacenter breach in April, throwing away our legacy infrastructure and rebuilding it properly - and subsequently have been expanding our ops team from one dedicated ops person to four. We also found ourselves having to do another emergency datacenter migration back in November when the old one was unable to reliably service IO for our database cluster.

We also spent a bunch time after shipping Matrix 1.0 working on tightening up Matrix’s privacy model - particularly around third party identity servers, integration managers, and making sure that folks self-hosting Matrix don’t accidentally depend on use 3rd party services without realising it. If you missed out on the fun at the time, you can read all about it here and here. This ended up being way more work than we expected, but we’re very glad to have sorted it out now.

Meanwhile, mainstream uptake of Matrix has properly taken off, with the French Government launching Tchap (their fork of Riot), now with hundreds of thousands of daily active users. The German Government revealed today that they are also formally trialling Matrix, starting with the Bundeswehr (Ministry of Defense); we’ve been helping them out with the deployment too. It is not an exaggeration to suggest that we could end up with an official cross-government Matrix network, publicly federated with the wider Internet, for self-hosted encrypted decentralised instant messaging. In fact Ulrich Kelber, the Bundesdatenschutzbeauftragte (Federal Data Protection Commissioner) for Germany pointed out: “You could even set up a privacy-friendly messenger service in cooperation with France, which in the medium term could represent a real alternative to existing products on the market as a pan-European solution”.

Alongside all this, Mozilla announced they are replacing the Moznet IRC network with Matrix; KDE joined Matrix in Feb, Wikimedia is getting set up on their server, and more and more massive players (including the largest in the world) keep getting in touch to find out how they can best get onboard Matrix - it’s incredibly exciting. It also means that we were able to raise capital to keep folks employed to work on Matrix fulltime via New Vector and scale up Modular.im as a paid hosting platform - which massively helps support core Matrix development.

2020

All that remains now is to make some predictions for 2020. Our main priorities are:

  • Get E2E enabled for private rooms by default (see above).

  • Riot First-time User Experience (FTUE). While we redesigned Riot’s UI in 2019, there are still far too many weird gotchas which trip over new users. Starting in October we began a shift to completely change how Riot development works - transitioning the project to being led by the UX design team rather than the dev team, and ensuring that the design team considers the app holistically across all 3 platforms. Above all else, our priority is to make it kick ass for normal non-technical mainstream users - not just for opensourcey wizards. This is a tough ask, but we believe it’s literally make-or-break for the project in the long term if Matrix is ever to become as prevalent as Slack or WhatsApp, and we are throwing everything we have at it. The second that E2E is on by default, the entirety of the Riot teams will be focusing on the mission to clear our FTUE backlog.

  • RiotX. We’re shipping RiotX on Android as fast as we can - currently users on Riot/Android are left high and dry and we need to do everything we can to finish RiotX and get them upgraded as rapidly as possible.

  • Communities. Off the back of FTUE comes the importance of grouping rooms & users together into communities in a much better way than we have today. This will be up next.

  • Synapse: shard the master by user/room to avoid being it being bottlenecked on CPU. We also need to apply smarter queue management on federation traffic to better reduce the memory footprint (and so eliminate complexity limits on small-footprint hosted servers!) - and we also desperately need to speed up joins.

  • Dendrite & P2P Matrix: the plan currently is to use Dendrite as the basis for our P2P Matrix experiments. In practice this means making it federate using MSC1228-semantics (no point in wasting time implementing the ‘legacy’ key management), and then experiment with hooking it up to various P2P transports (e.g. our low-bandwidth CoAP transport) and discovery systems (e.g. mDNS; libp2p; etc). How we go about actually getting it into production depends entirely on how well the experiment goes; we could evolve Matrix to be hybrid CS/P2P; we could treat it as a new protocol and bridge to it; who knows. Watch this space...

  • MLS: figure out our plan for next-generation E2E - for better scaling, and better reliability, and what (if anything) we should do with MLS.

  • Bridges: loads of work on the horizon to put a better UX on Bridging. Bridge stability has improved enormously over the last year (thanks Half-Shot!) but we need to transition from being robust but ugly to being robust and polished...

  • Spec: we need to work out how to go faster on reviewing MSCs (both our own and from the wider community). While the governance process in general feels healthier than it’s ever been, empirically we’re not exactly burning through the MSC backlog - and this is in part that MSC work is squeezed in alongside the other dayjob stuff everyone’s working on. Finding the right balance between sculpting spec and sculpting code is tough, but we’re going to try to improve it in 2020.

  • Abuse / Reputation: we want to empower users to make their own minds up about what content they want to see and not see on Matrix (or what they want to host or not host on their servers / communities / rooms). Mjolnir is a good start, but we’ll be continuing to work on this throughout the year.

Meanwhile, all the things listed above that we didn’t get to in 2019 are of course still options on the menu too.

So there you have it. I’ve not even tried to talk about the amazing stuff that the wider Matrix community has been up to - whether that’s amazing new clients like Ditto (React Native!) or Nio! (SwiftUI), or new bridges like mautrix-facebook and mautrix-hangouts, or even poljar’s secret rewrite of weechat-matrix in Rust; your best bet there is to skim through TWIM. Huge undying thanks go out though to everyone who builds on Matrix and keeps the ecosystem maturing and growing (especially while we’re scurrying around shoring up the foundations) - there’s simply no point in Matrix as a protocol without the vibrant community building on top.

All told, it’s been a bit of an epic year (both in terms of wins and fails), and all that remains is to thank everyone who continues to use Matrix (particularly our Patreon supporters) for their ongoing support and for helping the project accelerate forwards. More than ever before, the world needs free and open communication open to all; the age of proprietary communication silos may be coming to an end - consigned to live alongside AOL CDs and Compuserve IDs in the history books. With your support, Matrix can provide a decent mainstream yet decentralised alternative - and we’ll do everything we can to make that happen in 2020.

Happy holidays!

Matthew, Amandine & the whole Matrix.org team.

Avoiding unwelcome visitors on private Matrix servers

2019-11-09 — Privacy, Security, General — Matthew Hodgson

Hi all,

Over the course of today we've been made aware of folks port-scanning the general internet to discover private Matrix servers, looking for publicly visible room directories, and then trying to join rooms listed in them.

If you are running a Matrix server that is intended to be private, you must correctly configure your server to not expose its public room list to the general public - and also ensure that any sensitive rooms are invite-only (especially if the server is federated with the public Matrix network).

In Synapse, this means ensuring that the following options are set correctly in your homeserver.yaml:

# If set to 'false', requires authentication to access the server's public rooms
# directory through the client API. Defaults to 'true'.
#
#allow_public_rooms_without_auth: false

# If set to 'false', forbids any other homeserver to fetch the server's public
# rooms directory via federation. Defaults to 'true'.
#
#allow_public_rooms_over_federation: false

For private servers, you will almost certainly want to explicitly set these to false, meaning that the server's "public" room directory is hidden from the general internet and wider Matrix network.

You can test whether your room directory is visible to arbitrary Matrix clients on the general internet by viewing a URL like https://sandbox.modular.im/_matrix/client/r0/publicRooms (but for your server). If it gives a "Missing access token" error, you are okay.

You can test whether your room directory is visible to arbitrary Matrix servers on the general internet by loading Riot (or similar) on another server, and entering the target server's domain name into the room directory's server selection box. If you can't see any rooms, then are okay.

Relatedly, please ensure that any sensitive rooms are set to be "invite only" and room history is not world visible - particularly if your server is federated, or if it has public registration enabled. This stops random members of the public peeking into them (let alone joining them).

Relying on security-by-obscurity is a very bad idea: all it takes is for someone to scan the whole internet for Matrix servers, and then trying to join (say) #finance on each discovered domain (either by signing up on that server or by trying to join over federation) to cause problems.

Finally, if you don't want the general public reading your room directory, please also remember to turn off public registration on your homeserver. Otherwise even with the changes above, if randoms can sign up on your server to view & join rooms then all bets are off.

We'll be rethinking the security model of room directories in future (e.g. whether to default them to being only visible to registered users on the local server, or whether to replace per-server directories with per-community directories with finer grained access control, etc) - but until this is sorted, please heed this advice.

If you have concerns about randoms having managed to discover or join rooms which should have been private, please contact [email protected]

matrix-appservice-slack bridge 1.0 is here!

2019-10-03 — General, Bridges, Releases — Half-Shot

Hello Matrix enthusiasts! Yesterday we released matrix-appservice-slack 1.0. This marks a major milestone in bridge development for the Matrix.org team, being our first bridge to ever reach 1.0. The decision to release this version came after we decided that the bridge had gained enough features and reached a point of stability where it could be deployed in the wild with minimal risk.

For those not in the know, the Slack bridge is Node.JS based, and bridges slack channels & users into Matrix seamlessly. And for those wondering, yes it works with Mattermost too (since their API is compatible with Slack)! In previous versions only a limited subset of features were supported, making heavy usage of Slack’s webhook API. As of 1.0, the bridge now makes use of the newer Slack Events/RTM API which gives us all we need for a richer bridging experience. Everything from edits and reactions to typing notifications is supported in the 1.0 release.

Finally for those who are self hosting, we are pleased to offer the ability to "puppet" your Slack account using the bridge. Puppeting is the process in which the bridge will send messages as if you were sending them from the Slack client directly, when you talk using your Matrix account. This opens the door to seamless bridging and direct messaging support.

For those wishing to bridge their whole workspace across, picard exists as a tool to manage large scale Slack bridge deployments. This tool is provided by Cadair and SolarDrew

Slack Screenshot Threading & Reactions!

The bridge has undergone some pretty serious code surgery as well. The whole codebase has been rewritten in TypeScript to take advantage of type checking and generic types. The bridge is currently based upon the matrix-appservice-bridge library. The datastore interface now supports PostgreSQL, wich allows for administrators to inspect and edit the database while the bridge is running, as well as offering helpful performance boost over the NeDB datastore format that was used previously. Finally, the codebase has proper Unit and Integration Tests to ensure new changes will not cause any regressions in behaviour. In short, now is an excellent time to get involved and hack on the bridge. There is already a crafted list of easy issues for new and experienced bridge devs.

Grafana memory usage graph Memory usage of the bridge comparison

Grafana CPU usage graph CPU usage of the bridge comparison

In terms of how many users matrix.org is currently serving at the moment, we present to you some figures:

  • 2562 bridged rooms
  • 764 teams connected to the bridge
  • 103711 events have passed through the bridge since the launch of 0.3.2

Of course, our work doesn’t stop at 1.0. The plan for the immediate future of the bridge is to continue adding support for other event types coming from Matrix and Slack to create an ever richer experience. Obvious features are things like topic changes, and syncing membership across the bridge. In the long term future we would also like to add community support to the bridge, so whole Slack workspaces can be bridged across with a single click.

That’s all from me, and I would like to say a massive thank you to Cadair and Ben for both their code and review work on the project and as always, thank you to the community for using the bridge and reporting issues. 🙂

Useful links:

5-user Matrix homeserver hosting now available from Modular

2019-07-17 — General, In the News — Modular.im

Hi all,

If you’ve been looking for a way to have you own Matrix homeserver without having to run it yourself, you may be interested to hear that Modular (the Matrix hosting provider run by New Vector, the startup which hires many of the Matrix core team) is now offering a personal-sized small homeserver hosting service, supporting a minimum size of 5 user servers.

A lot of recent performance work on Synapse has been driven by the need to make smaller dedicated servers more efficient to run - and so if you run your own homeserver you’ll be benefiting from all this work too :) Meanwhile, if you choose to outsource your server hosting to Modular, you’ll be indirectly supporting core Matrix and Synapse development, given most of the core Matrix team work for New Vector - it’s through buying services like this which lets us keep folks able to hack on Matrix as their day job.

See more details over at the Modular blog post!

PreviousPage 1
NextPage 3