Synapse 1.54 is
out! Let's see what goodness is coming your way with this new release.
Improving URL previews
As part of the Matrix specification, Matrix clients have the option to rely on
the homeserver to generate URL previews. This means the homeserver (in this case
Synapse) needs to have a look at the content at that URL and extract data to
send back to the client.
In the past, Synapse has had issues with generating complete URL previews, and
some metadata (e.g. image, description) would be missing from the data that
Synapse sends to clients. Synapse 1.54 includes improvements to the generation
of URL previews, and while testing it we've been able to observe improvements to
previews generated for Twitter and Reddit.
New module callbacks
Synapse modules allow third-party developers to write extra features for
Synapse, that wouldn't necessarily be generic enough to fit within the Matrix
specification. This includes custom behaviours such as smarter event filters,
bespoke media storage providers, or spam checkers. Since we rewrote Synapse's
module system in Synapse
1.37,
we have been improving it by adding new callback functions that module can
implement, allowing them to interface better with Synapse.
Synapse 1.54 includes three new module callbacks. One of them,
get_displayname_for_registration,
allows modules to define the display name for newly registered users. Together
with the existing
get_username_for_registration
introduced in Synapse
1.52, they allow
modules a better control over the user registration process.
The two other module callbacks introduced in Synapse 1.54,
on_profile_update
and
on_user_deactivation_status_changed,
allow modules to react to profile changes, as well as the deactivation (or
reactivation) of users.
Everything else
This release of Synapse also includes more work to make joining large Matrix
rooms faster (I was telling you about that in the Synapse 1.53 release
announcement). While
it's still very experimental and not yet ready for show time, it's still very
exciting to see this work happening!
Synapse 1.54 also includes a change to the client-side versions
endpoint
to advertise Matrix 1.1 and 1.2. See the Matrix
1.2 release
announcement to read all about the changes this latest version of the
specification brings to the ecosystem.
See the full
changelog for a
complete list of changes in this release. Also please have a look at the
upgrade
notes
for this version.
Synapse is a Free and Open Source Software project, and we'd like to extend our
thanks to everyone who contributed to this release, including (in no particular
order) Dirk Klimpel, Andrew
Ryan, and
lukasdenk.
t2bot.io - a public integrations network for Matrix - has passed 1 Million known rooms and 8.3 Million bridged users. 10,000 of these rooms are attributed to Voyager (a bot which actively goes out to find rooms to map Matrix, with the map currently down for maintenance), leaving the remaining ones either bridged, previously bridged, or using a different integration offered by t2bot.io for free.
The 8.3 Million users are mainly Discord and Telegram users which have been brought over to Matrix through bridges. The stats say "excluding Twitter-bridged" because there's 424,832 old accounts from back in the day when t2bot.io had a free Twitter bridge available. To further break this down, about 6.8M are Telegram users (12% of Telegram) and 1.3M are Discord (<1% of Discord).
For perspective, t2bot.io has about 569 Million events stored in its database and sees approximately 30 thousand people bridged daily from the wider world into Matrix through its bridges.
This post is just a milestone update, but it also serves as a reminder that running your own server/bridges is also possible. In fact, it's even recommended to have better control over your own data and avoid latency issues that large providers, like t2bot.io, can unintentionally introduce. Synapse is relatively easy to set up with minimal sysadmin knowledge (guide), and there's always paid offerings like Element Matrix Services (for home and also for fun) and Beeper for a richer bridging experience than t2bot.io can feasibly provide.
It is to be noted that eQualitie has set-up public Matrix and Element instances for Ukrainian people who are struggling right now. If you are in Ukraine or are in touch with Ukrainian people who need secure communications, you can show them https://kyiv.dcomm.net.ua, https://odessa.dcomm.net.ua, or https://kharkiv.dcomm.net.ua depending on their location so they get instructions on how to create an account on Matrix and stay in touch
MSC3575 Sliding Sync work is progressing, and a number of new features have been added to the MSC with implementations in the built-in web client in the proxy. A list of changes include:
spec/proxy: the ability to filter the room list by room name.
spec/proxy: Aninitial flag on the room to distinguish between updates and the initial sync for a given room. This can be used as a hint to clients to determine if they should update/replace their local data for that room.
proxy: more efficient algorithm for determining overlapping sets of ranges, resulting in fewer bytes over the wire when scrolling.
proxy: the client implementation now supports sending basic text messages as well as issuing/join,/invite,/leave commands.
proxy: the client implementation now has a developer HUD which tracks the Tx/Rx bytes as well as a visual representation of the sliding window.
The net result is a basic but incredibly low bandwidth syncing client: 60.79KB to download an entire initial sync on matrix.org. Further improvements will be done on the client to make sure it doesn't scale with the number of rooms on the user's account (it currently does because it naively adds a placeholder for each room in the list) to ensure it remains extremely snappy and a vision for what Sliding Sync can do in practice, right now. Please continue giving feedback on MSC3575 or in #sliding-sync:matrix.org as the API is still in development and will change depending on what clients require.
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
Heads up! You may have noticed some changes to the matrix-org/matrix-doc github repository this week. The first major one being that it no longer exists!
We have separated it out into two repositories:
matrix-org/matrix-spec - the source text of the spec. Issues and pull requests filed here should relate to the source of the spec.
matrix-org/matrix-spec-proposals - all Matrix Spec Change proposals. New MSCs should be filed against this repository as pull requests. Issues should not be filed against this repository (and no, the issue tracker cannot be disabled, as it would cause transferred issue link redirects to 404).
The primary motivation for this was to separate concerns of interacting with matrix-org/matrix-doc. Those interacting with the spec's source would have to wade through long-lived spec proposals, while those only interested in spec proposals would need to filter out spec source files and other issues and PRs related to them. This also helps writing tooling against the repo either as it no longer needs to filter MSCs via issue labels.
Existing links to matrix-doc should be automatically redirected by github to the right place. MSC authors and developers should update their git remote settings to point to the new repos.
What happened in detail to achieve this was:
matrix-doc was renamed tomatrix-spec-proposals
matrix-spec was created
All files other than the "proposals" folder were transferred frommatrix-spec-proposals tomatrix-spec
All open issues frommatrix-spec-proposals were transferred tomatrix-spec
Further details are available here: https://github.com/matrix-org/matrix-spec/issues/927
This MSC asks for the simple change of adding a "destination" field to the Authorization header of federation requests. This would have the benefit of allowing reverse proxies - potentially handling traffic for multiple homeservers - to know where requests should be sent. Or for forward proxies, which could validate outgoing requests from a homeserver before they're sent.
There is currently some discussion on the fundamental idea of the MSCs, which will need to be resolved first before it can move forwards.
Quite a varied bag this week! Firstly we cut a release candidate for Synapse 1.54.0, which we hope to release next week. As ever, testing and feedback is much appreciated.
Our switch to Poetry for dependency management has come a few steps closer, by removing tox from our Continuous Integration, and converting some of our ad-hoc scripts to proper Python entry points.
Meanwhile we've been continuing to knock out blockers to improving room join speeds, including better exception handling and logging, and some exciting work to reduce wasted work when a client disconnects while we're still processing a request.
We've also been taking a wider look at performance problems, such as slow /sync and /login times.
I have been asked by numerous people over months if there are any kind of graphs comparing Synapse and Dendrite. Up until now, my answer has been "as far as I know, no". However, this got me thinking whether I can use Complement to engineer some kind of performance testing infrastructure. After hacking on it for a day or two, I can now share some of the work I've done in this area. There now exists two new binaries in the Complement repo: perftest and perfgraph. The former will run some tests and output a .json file. The latter will consume these files to generate .svg graphs. The tests performed include:
Creating a bunch of rooms with different users. (currently 10 users and 50 rooms)
Joining a bunch of rooms with different users.
Sending messages into rooms with different users. (currently 100 messages)
Doing an initial sync for all users.
Updating the display name of all users.
Doing an incremental sync for all users.
The metrics are collected using docker stats. This means the entire container is measured, not just the homeserver binary (e.g any daemons or assistive processes are included in the metrics) and includes:
CPU time consumed
Memory consumed
Bytes transmitted/received
Disk read/writes
Time taken
Here are some results:
This was a surprising find and probably indicates that Dendrite has a memory leak somewhere when creating events. However, the fun doesn't stop there. We can also use this tool to compare different versions of the same homeserver:
This allows us to quantify any performance improvements we've been working on, and you can clearly see how we're being more efficient with CPU in newer versions. But the fun doesn't stop there either. We can also compare different flavours of the same version of a homeserver:
You can quantify the cost of running sqlite or postgres for your particular installation.
Care must be taken when interpreting results. Lower doesn't always mean better. For example, a low readout on "create_users" may be a cause for concern as it might indicate that the server is not hashing passwords correctly (e.g using bcrypt/scrypt which are designed to consume CPU/memory). Servers which don't implement all of the specification will also be abnormally low on CPU/memory (e.g not having to calculate unread counts or handle push notifications naturally means less work to do so less resources consumed). Furthermore, these tests do not yet test federation, so expensive remote joins are not measured (remote joins require the joining server to check signatures, hashes, etc of a lot of events which doesn't happen for local servers). That being said, I have hopefully illustrated how useful this tool can be for both server developers trying to improve their software and server admins who want to use the right server for their hardware. The graphs are still a work-in-progress and there's a lot more that can be done in this area beyond a few days work, but it's a start.
Today we've released Dendrite 0.6.5 which contains early push notification support as well as a number of fixes and improvements. This release includes the following changes:
Early support for push notifications has been added, with support for push rules, pushers, HTTP push gateways and the /notifications endpoint (contributions by danpe, PiotrKozimor and tommie)
Spaces Summary (MSC2946) is now correctly supported (when msc2946 is enabled in the config)
All media API endpoints are now available under the /v3 namespace
Profile updates (display name and avatar) are now sent asynchronously so they shouldn't block the client for a very long time
State resolution v2 has been optimised further to considerably reduce the number of memory allocations
State resolution v2 will no longer duplicate events unnecessarily when calculating the auth difference
The create-account tool now has a -reset-password option for resetting the passwords of existing accounts
The /sync endpoint now calculates device list changes much more quickly with less RAM used
The /messages endpoint now lazy-loads members correctly
Read receipts now work correctly by correcting bugs in the stream positions and receipt coalescing
Topological sorting of state and join responses has been corrected, which should help to reduce the number of auth problems when joining new federated rooms
Media thumbnails should now work properly after having unnecessarily strict rate limiting removed
The roomserver no longer holds transactions for as long when processing input events
Uploading device keys and cross-signing keys will now correctly no-op if there were no changes
Parameters are now remembered correctly during registration
Devices can now only be deleted within the appropriate UIA flow
The /context endpoint now returns 404 instead of 500 if the event was not found
SQLite mode will no longer leak memory as a result of not closing prepared statements
Spec compliance, as measured by Sytest, currently sits at:
Client-server APIs: 76%, up from 65% last time
Server-server APIs: 95%, up from 94% last time
As always, you can join us in #dendrite:matrix.org for Dendrite discussion and announcements.
Hey folks, I'm back from holidays and I'm proud to say that hookshot is kicking out release after release. This release is especially large and contains a bounty of new features and fixes. Of note are:
JIRA Datacenter (On Premise) is now supported.
You can now configure fine grained permissions for users on the bridge by userId, homeserver domain or room membership (think spaces).
Generic webhooks has sprouted a versioned API, with v2 allowing for finer control over the output.
GitHub connections now include the closing comment when issues and PRs are closed.
GitHub connections will also notify a room when an existing issue has been relabled to one filtered by that room.
Figma now uses MSC3440 for comment threads.
But seriously go check out the release, there is way more there than I can include in this TWIM post. Happy webhookin, matrix gang!
In case you did not know, Hookshot can be installed via spantaleev's Ansible playbook. In addition to updates for this new release, the role has recently gotten some improvements and fixes by the community, so in case you had issues before, now is the time to try again!
P.S oh and I almost forgot, we rehomed to https://github.com/matrix-org/matrix-hookshot. This doesn't change much, other than some redirects in your browsers cache.
Allows anyone to run github commands in rooms. User part of the room listed can actually go and create connections to anywhere (we use it internally at $work), and then there is a final random user who gets those permissions because they refuse to join that room :p
Released v1.10.5 and also v1.10.6 as a hotfix for a crash
Surveyed users and collated feedback from search result ordering for improvements to the new search experience (enable it in Settings -> Labs in production)
In labs (you can enable labs in settings on develop.element.io or on Nightly):
Pin drop location sharing has been added
Fixed message ordering issues in threads
Fixed threads discovery when scrolling back in a room
Added a couple of new ways to access pinned messages, in the room header and info panel
Improved the reliability of pinned messages with edits
Making progress on the registration flow by setting name and avatar
In development:
We will be trying out Spaces on iOS at 16:00 UTC / 17:00 CET on Tuesday, 8th of March. Head over to #element-community-testing:matrix.org to hear the latest on all testing sessions!
Live location sharing has started
In labs:
An option to use the last member avatar and name in the timeline
a very simple invite-accepting bot as part of matrix-streamchat to more easily facilitate peeking into rooms over federation.
You run one on your guest-access enabled homeserver, invite it to your remote room, and now all guest accounts from your server know they are in fact allowed to publicly read the room over federation without joining themselves. https://git.pixie.town/f0x/matrix-streamchat/src/branch/main/autojoin-bot
Work on having a setup page for the bot was added which allows to easily add the bot to your room (sets up permissions and invites the bot as required)
Added support for a state event which tells the widget about "who is the bot?" and "which lists does this bot monitor?" Which also allows you to use the widget without being in the banlist room yourself. (See https://github.com/MTRNord/matrix-moderation-widget#use-a-state-event-to-allow-showing-relevant-lists-only-in-the-dropdown for more information)
Element Call has entered beta! Head over to https://call.element.io (formerly matrixvoip.dev) to play with in-browser native Matrix voice/video calling powered by MSC3401, supporting all webrtc-capable mobile & desktop browsers. See all the details over at https://element.io/blog/introducing-native-matrix-voip-with-element-call/
Offers high quality video calls with around 6 or 7 participants as a small & simple, standalone app that allows anyone to drop into a video conference easily.
This is beta - please file bugs at https://github.com/vector-im/element-call/issues and be aware that currently all participants need good bandwidth
This is the first implementation of MSC3401: a spec for secure, federated voice and video conferences.
Integrations into other clients are coming soon, Hydrogen is working on it right now
We’ll also be developing a server to mix media streams so calls can scale up to much larger number of participants. The server will only ever see your encrypted media, so calls will stay secure and confidential.
A set of Rust library crates for working with the Matrix protocol. Ruma’s approach to Matrix emphasizes correctness, security, stability and performance.
We're back (to TWIM)! Here's some highlights from the past few months:
Stronger types for some things:
Base64 which ensures you always decode / encode with the appropriate base64 dictionary [@jplatte]
TransactionId which is simply a little more explicit than String and easier to generate a random instance of [@jplatte]
Stable support for lots of previously-unstable functionality that got stabilized upstream as part of the Matrix specification releases v1.1 and v1.2 [@zecakeh]
More fine-grained feature flags for MSC implementations (before: everything behind unstable-pre-spec, after: most things behind dedicated unstable-mscXXXX features) [@zecakeh]
All of these changes are available in version 0.5.0 which was released about two weeks ago.
Additionally, @zecakeh recently started implementing extensible events and doing some pretty heavy internal refactoring that will make our release process faster and hopefully make contributing to Ruma easier as well.
With all these changes we're very close to supporting all of Matrix v1.1 (and Matrix v1.2 too!), with the only major omission being Secure Secret Storage and Sharing (SSSS). Some work on that was done as far back as last year's GSoC but it was blocked on a medium-sized refactoring of how we handle account data. I'm planning to get that over the finish line over the next weeks.
I'm working on a little hobby project, The Matrix Scribe.
The Matrix Scribe helps us re-post or transcribe messages into matrix that we received from somewhere else, posing as different ghost users to represent the original authors.
Scribe will post this:
Ann I'm Ann.
Bob Hello! I'm Bob.
when I write this to @scribe-bot:
@julian /scribe as Ann
@julian I'm Ann.
@julian /scribe as Bob
@julian Hello! I'm Bob.
That is an illustration of the goal. The output part works already; the bot part isn't implemented yet.
Why?
I want to encourage people to experiment and play with extending Matrix in creative ways.
As a building block for myself towards building more usual matrix bridges. Scribe is the input half of the matrix half of a bridge, so literally one quarter of a typical matrix bridge.
I hope that I myself and maybe some other people will find Scribe fun or useful in itself.
a simple bot https://github.com/borisrunakov/maubot_azuracast built with maubot, that you can use to request data from your azuracast self hosted instance. It's not much but ok...
I'll be speaking at the new HYTRADBOI conference (29 Apr) about building collaborative, open software on Matrix. All of the talks sound super interesting, highly recommended! 😄 The talks will be public after the conference as well.
Dept of Ping 🏓
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server.
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
MSC3440 continues to be the focus on much review as the proposal inches towards entering final comment period.
Otherwise both MSC3589 (room version 9 as the default room version) and MSC3582 (remove m.room.message.feedback) merged this week. The former brings with it updated versions for rooms as they continue to be created across the federation, whereas the latter is simply a nice clean up to the existing spec :)
This MSC aims to introduce a set of generic administrative APIs for Matrix homeservers, starting with those that could potentially be useful for user moderation.
There has long been talk of adding administration APIs akin to Synapse's Admin API to the spec, in part to reduce the number of tools that are specifically created for it, and thus can only be used with Synapse.
This week we released Synapse 1.53.0! This release adds support for sending to-device messages to application services, adds a background database update to purge account data for deactivated users, and adds more features to improve performance and stability, as well as bugfixes and improved documentation. Check out the release notes here.
In addition to the release, work continues on improving the performance of room joins-progress is being made! Finally, we began the process of switching over Synapse to use Poetry for dependency management-keep your eyes peeled for more information on that project as it develops.
This week we released Dendrite v0.6.4 which contains a significant number of improvements and fixes. It includes the following:
All Client-Server API endpoints are now available under the /v3 namespace
The /whoami response format now matches the latest Matrix spec version
Support added for the /context endpoint, which should help clients to render quote-replies correctly
Accounts now have an optional account type field, allowing admin accounts to be created
Server notices are now supported
Refactored the user API storage to deduplicate a significant amount of code, as well as merging both user API databases into a single database
Guest registration can now be separately disabled with the new client_api.guests_disabled configuration option
Outbound connections now obey proxy settings from the environment, deprecating the federation_api.proxy_outbound configuration options
The roomserver input API will now strictly consume only one database transaction per room, which should prevent situations where the roomserver can deadlock waiting for database connections to become available
Room joins will now fall back to federation if the local room state is insufficient to create a membership event
Create events are now correctly filtered from federation /send transactions
Excessive logging when federation is disabled should now be fixed
Dendrite will no longer panic if trying to retire an invite event that has not been seen yet
The device list updater will now wait for longer after a connection issue, rather than flooding the logs with errors
The device list updater will no longer produce unnecessary output events for federated key updates with no changes, which should help to reduce CPU usage
Local device name changes will now generate key change events correctly
The sync API will now try to share device list update notifications even if all state key NIDs cannot be fetched
An off-by-one error in the sync stream token handling which could result in a crash has been fixed
State events will no longer be re-sent unnecessary by the roomserver to other components if they have already been sent, which should help to reduce the NATS message sizes on the roomserver output topic in some cases
The roomserver input API now uses the process context and should handle graceful shutdowns better
Guest registration is now correctly disabled when the client_api.registration_disabled configuration option is set
One-time encryption keys are now cleaned up correctly when a device is logged out or removed
Invalid state snapshots in the state storage refactoring migration are now reset rather than causing a panic at startup
Sytest compliance is up slightly:
Client-server APIs: 67%, up from 65%
Server-server APIs: 95%, same as before
As always, please feel free to join us in #dendrite:matrix.org for Dendrite-related chat!
Here are the first steps for a bridge to KakaoTalk! The bridge is based on mautrix-python (having used mautrix-facebook as a starting point--there's still plenty of Facebook-specific code in there), with the backend handled by node-kakao (connected via RPC, as there seems to be no Python API for KakaoTalk!).
The bridge doesn't do much yet; all it can do is log in & sync your list of chats (if that). But it's under rapid development & decent momentum, so hopefully it will be usable soon!
For anyone brave enough to try it out, its setup steps are very similar to that of any of the Python-based mautrix bridges (though Docker is currently unsupported).
We made a small release that just is compiled against the new mtxclient version to fix an issue with servers announcing support for Matrix v1.1 or higher. We strongly recommend you update before the next Synapse stable release is out.
Apart from that Nheko now has support for hidden read receipts (thanks to symphorien, see MSC2285). ZenWalker updated our usage of deprecated gstreamer APIs. Malte has been spending a lot of effort on improving the scrolling experience on the PinePhone as well as allowing to search on mobile. Forwarding should now work properly again as well as calling on mobile and we fixed a small memory leak when opening some dialogs.
“Twosday” wasn’t the only exciting thing happening this week. Take a peek at everything else we had going on…
Coming to a Poll near you…
From next week’s releases, you’ll discover two new updates on polls! First off, you’ll be able to edit a poll as long as no one has yet voted on it - which is great if you create a poll and realise you’ve made a small mistake. Even better, there’s now a new type of poll: ‘closed polls’ don’t show any results until the poll has ended, to keep the surprise.
Location Sharing
Location Sharing is now available by default for users on all platforms, except desktop (where you can receive but not send locations). Check it out!
The next stage is live location sharing and ‘pin dropping’, expect more soon.
Threaded Messaging
Designed to make catching up on rooms easier, and to keep the main timeline as clutter free as possible, Threads are nearly here.
You can try Threads out on all platforms - you’ll find them in Labs. This feature is experimental; let us know your feedback, and report any bugs as we continue to improve.
Community testing
We will be looking at search result ordering on Web as part of the new search experience at 17:30 UTC / 18:30 CET on Tuesday, 1st of March
We’re also hoping to test Threads on mobile devices towards the end of the week, join the testing room to get involved!
The EleWeb team has been working on Spaces, Threads, and defects this week.
We are starting to look at batched updates, which could be bringing performance improvements to us.
On the process improvement side, we are looking at test coverage and process improvements around PR submission. Don’t be surprised if our developers start a conversation around tests when you submit your next PR 🙂
V1.10.5 release candidate is available and release is expected to go out on Monday, 28th February.
And don’t forget; the new and improved search experience is available. It’s in Beta so turn it on, try it out, and send us your feedback!
We will be talking to the community about planned improvements in the next Community Testing Session on Tuesday over in#element-community-testing:matrix.org
In labs (you can enable labs in settings on develop.element.io or on Nightly):
Improvements to Threads reliability are happening everyday. We’re also making some tweaks to the user experience details, like dragging and dropping files into the Thread panel.
Next week’s release (1.8.3) includes changes and improvements we’ve made to our overall app experience by closing some pesky UI defects.
This week we’ve also been working on improving the reliability of our Labs features. If you’ve turned on Threads or Bubbles in Labs you may have experienced app slow downs or crashes. In the next version, these will be minimised.
Spaces on iOS are also getting some attention at the moment and we’re hoping to improve the user experience of Spaces on Mobile.
The next release of Android (1.4.2) includes support for “@ room” and other usability defects you might have seen before… Fixes include steadying the notification badge in the room list, adding the correct interactions on bottom sheets, and opening a DM from the Space member list.
We’ve also been working on upgrading the voice message experience, adding improvements like scrubbing! Keep sending voice notes and let us know what other improvements we should make.
Our onboarding flow is also getting a new lick of paint. We want new users to our platform to have a simple and straight-forward experience when they’re creating an account.
Over at Populus-Viewer, we're continuing to refine the UX, for maximum focus, efficiency and enjoyment. Since last time we've:
Reworked the mobile view controls into a sidebar design.
Improved the generation of highlight rectangles.
Made sure that LaTeX and code listings are always displayed nicely
Made it possible to modify text selections within a PDF using the keyboard
Added "one-click" links for onboarding new users into a particular server, SSO flow and PDF collection.
We've also had some bug fixes related to federation, and had some of our first ever (maybe the first on matrix - first in the history of the universe?) federated social annotation sessions.
Populus-Philarchive
Populus-Philarchive, our proof-of-concept discussion overlay for preprint archives, now incorporates an OAI-PMH harvester, so it can aggregate OAI bibliographic metadata, and use that data for room creation and discovery. The implementation is pretty general, so it should be easy to tweak for any archive that supports OAI-PMH.
MSC3574
MSC3574 - marking up resources got some love this week, as we added a proposal for serializing annotations on matrix that ought to be compatible with the w3c web annotation data model. This paves the way for interoperability between the matrix annotation ecosystem and services like hypothes.is, and hopefully will make matrix a compelling option even for institutions where compliance with existing web standards is a must.
As always, if you'd like to chat about any of these developments, come visit us at #opentower:matrix.org !
The Circles beta on iOS continues inching toward a public release later this Spring.
This week I added support for infinite scrolling on timelines. (Previously, scrolling the timelines was very clunky -- the user had to manually tap a button to "Load More" every 10-15 posts.)
Also added a confirmation dialog when the user attempts to leave a group.
On Android, the prototype is coming along nicely, thanks to the efforts of our new developer Taras:
The login screen works
Currently working on implementing the timeline of social posts for groups
In the last 2 weeks, I increasingly had to learn how to moderate rooms properly, which brought up a lack of nice Mjölnir gui for me.
Due to that, I just started to write one.
It is at the time of writing still fairly young.
The current features are:
An overview of the ban list data the user is in (Not for the specific Mjölnir currently. Also requires a user to have joined the list room)
A quick form to ban a person
A form to redact someone or a message
Planned features are:
Support for showing MSC1929 information if available
Writing a patch for Mjölnir, so the widget can know which banlist the bot watches, so only relevant lists show up.
Editing the banlist (aka unbanning)
Adding support for more advanced features like deactivation of users and removal of media on the matrix-media-repo.
Covering most of Mjölnir's commands
Redact on ban and similar utilities you might want while banning.
Small getting started (it is simple :D)
To use it, you simply can add it to your Mjölnir Admin room by putting /addwidget https://moderation_widget.nordgedanken.dev?room_id=$matrix_room_id (the variable will get replaced automatically) in the message bar and pressing enter. The widget runs entirely client side, so this is not sending any events to my server. If you still are concerned due to the big amount of permissions asked, you can just build it yourself and host it.
You know what would be embarrassing? If changing the version number of something broke Nheko... Well, completely unrelated, mtxclient 0.6.2 is out now which fixes an issue where it would aggressively validate that version numbers started with an 'r'. Otherwise that release is API and ABI compatible, so if packagers could pick that up as a bugfix release into stable releases, that would be great!
A small update was released, merging 2 month old PRs.
The changes are mainly features being now deactivated because we did not actually use them and fixing the example in one case. No updates to dependencies.
0.4.0
Dependencies have been updated to the newest versions.
The distribution-provided Debian packages for Synapse will only be provided for Bookworm (in testing/unstable) and Bullseye (in bullseye-backports). If you’re still using Buster (through buster-backports-sloppy), consider switching to Bullseye or, alternatively, to packages provided by the Synapse upstream. 1.52.0 is the last version to be provided for Buster through the backports repository.
I saw an interesting (to me) reMarkable telegram bot somewhere. But I prefer matrix and node.js was more difficult to deploy on embedded. So I wrote a reMarkable matrix bot in Go. https://gitlab.com/ptman/remarkable-matrix
Looking for a bot to manage events and feedback from your community?
MCM an information bot. It manages the flow of information between community leaders and their community.
It aggregates messages from community members in several ways.
A @ mention. You can mention the bot with a message.
A Direct Message. Members of your community can message this bot privately.
hash tags. Using hash tags members of your community can send messages tagged to go to a specific back room.
You as an administrator of the bot can send timed announcements to any room using the built in matrix administration interface.
You can also manage tags, add and remove admin of the bot, add automatic replies and more. All from the comforts of your Matrix client.
Rejoice everyone, Synapse 1.53 is out! Let's have a look at what's new with this release.
Stabilisation of registration tokens
Registration tokens is a feature introduced in Synapse 1.42.0, which allows homeserver administrators to force their users to use specific tokens when registering. This is similar to Synapse's registration shared secret support, but with added features, such as the possibility to limit how users can be registered with the same token, or to make a token expire. See the admin API documentation for more information on how to manage registration tokens on your homeserver.
Registration tokens were initially proposed to the Matrix specification in MSC3231 by Callum Brown during their Google Summer of Code internship last summer. The MSC has since been accepted, and released in the stable Matrix specification as of Matrix v1.2. As a result, its Synapse implementation has been updated to remove support for unstable identifiers. Administrators of homeservers on which the reverse proxy rules explicitly allow the unstable route for this feature need to update their configuration. Same goes for developers of Matrix clients that support this feature. See the upgrade notes for more information.
Time-based cache expiry now enabled by default
To avoid being overly intensive on resources by making too many queries to the database, Synapse maintains several in-memory caches to store data it needs to use frequently. However, this comes with the inconvenience that, if Synapse needs to store too much data, these caches can become fairly big and occupy too much space in the host's memory.
Historically, Synapse has dealt with this issue by having set sizes for each cache, either hardcoded or set in the configuration, and evicting the oldest items when exceeding this size. Synapse 1.38 introduced the possibility for homeserver administrators to configure Synapse to evict cache entries based on the time they were last accessed on. This mechanism acts on top of the aforementioned eviction policy, and allows automatically evicting entries that haven't been accessed for some time, leaving more room in the caches to store data that needs to be accessed more often.
Synapse 1.53 enables this behaviour by default. Without specific configuration, Synapse will automatically evict cache entries that haven't been accessed for more than 30 minutes. Server administrators that were already using this feature might need to update their configuration, as this change deprecates the expiry_time configuration setting, which will be removed in a future version of Synapse. See the upgrade notes for more information.
Everything else
You might have heard that we're working on improving the time it takes to join big Matrix rooms with Synapse. If not, then you definitely want to have a look at the demos Matrix live that was published earlier this month and includes more details and a demo of the work we've been doing in this area.
This release of Synapse includes an implementation of MSC3706, which is part of this work. It's still very experimental and definitely not production-ready, but it's a huge stepping stone towards making room joins snappier than ever.
We've also been continuing our work towards enabling end-to-end encryption for application services (see the Synapse 1.50 release blogpost for more context on that). Synapse 1.53 includes support for sending to-device messages to application services. This is also still very experimental, watch this space for future updates.
See the full changelog for a complete list of changes in this release. Also please have a look at the upgrade notes for this version.
Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including (in no particular order) Dirk Klimpel, Brad Jones, and Alexander Mnich.
Hello everyone! It's me, not-anoa, here with your weekly spec update. I finally got the scripts to run which means you get a proper update once again (yay).
The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
In terms of Spec Core Team MSC focus for this week, we've been looking at unblocking MSC3440: Threads so it can make a smooth journey into acceptance. This involves quite a lot of work to ensure that the features we're concerned about are addressed, but so far the FCP progress on it is looking good.
We're also spending a bunch of time working out what room version 10 looks like to try and fix some usability issues with join rules and general maintenance of things like power levels. The hope is that in time for Matrix 1.3 we'll have v10 out (but not made default) for folks to experiment with.
Random MSC of the week
Your random MSC is MSC2974: Widgets: Capabilities re-exchange. I swear it was actually random and only after running the script exactly once. The MSC is interesting because it allows widgets (a feature working its way into the general spec, slowly) to ask for more permissions when they need them. This ensures the widget doesn't need to ask for everything at startup (which is more likely to mean that it gets rejected), and that it can maintain a clean security state during normal operation.
One day it'll probably make it into the spec as part of the larger widget movement, but first we need to get widgets into the spec properly.
The Graph
Numbers only mean so much, and graphs show progress. Here's the breakdown of MSCs over time:
Of note
If you look carefully at the OpenAPI viewer, you might notice a dropdown for not-client-server APIs 👀. There's still work to be done to bring on the missing APIs, but this is in a place where it can be experimented with. Thanks to Alexandre Franke for getting this over the line :)
This week we've released Synapse 1.53.0rc1, which includes a bunch of new features, improvements, and other niceties... But more on that next week when we release Synapse 1.53.0 🙂 As usual, we're super grateful for anyone that helps us test release candidates by running them with their homeservers! Please report any breakage or feedback in #synapse:matrix.org
Apart from this we have been continuing our experimentations with Poetry to better manage our dependencies in our Python projects after we switched Sydent to it last week. We're already starting to see improvements off the back of this work, such as automated security PRs and an opportunity to centralise our automated workflows to better reuse them in our projects (see https://github.com/matrix-org/backend-meta). We look forward to bringing all this goodness to Synapse soon!
And my Helm Chart updates are still happening, also now listed on Artifact Hub for easier discovery. This week saw several element-web updates, finally ending on 1.10.4
Is using Nheko a bit of a pain on your PinePhone? Do you just want bubbles in your chat app, not raw lines of text? Does Nheko waste too much space on timestamps and other metadata for messages?
Well, I am assuming Malte was annoyed by that or similar reasons. But in the end, they did spend a lot of effort reorganizing the way Nheko layouts messages so that people can have bubbles. This should make Nheko feel much more at home on the PinePhone and it seems like they are doing even more to make Nheko a great experience on mobile devices! But just look at it yourself:
If you are scared now and you think: "This is not the Nheko I came to love!", don't worry, all of this is optional. You can play around with the different avatar sizes as well as the bubbles themselves. You can even make Nheko look like it always looked!
There will be some regressions though. If you want to contribute, it would help a lot if you test those changes and report issues that you find! <3
Apart from that, there were also a lot of other bugfixes and cleanups by various contributors. You can also hide events by type now. If you don't want to see stickers or when someone joins a room, just disable that in the room settings! Tastytea implemented that. Long usernames should now also no longer overflow the profile pages and you can reset the state for a single room using the /reset-state /command. This is helpful when updating Nheko to use widgets, but Nheko just threw those events away (or pinned messages or other state events).
The new search experience came out this week, you can enable it in Beta to try it out! We are collecting feedback from users to inform upcoming improvements.
We’re working hard on smashing bugs and reducing the number of defects.
Along with closing bugs, we’re working hard on adding finesse to our app and removing some of those “papercut” issues that users experience.
Keep your eyes peeled for updates and let us know what you think!
Our first time user experience is being updated also. We’ve introduced new splash screens to help introduce Element. Don’t panic! You can skip straight to “sign-in” if this isn’t for you.
A new spinner is here… While we work on the speed and performance of the app we’ve introduced a new spinner that does not get in your way while you work.
Bugs and Papercuts… Our app is getting some love from our developer team as we try to reduce confusion and simplify flows throughout the app.
Keep an eye out for any small changes and let us know what you think!
Creating a new account in Element can be intimidating for new users, especially those who aren’t familiar with Matrix. We’re introducing new screens and simplifying our questions so that users can sign up with confidence!
If you have any feedback, or want to share your thoughts on our first time user experience, get in touch.
A new layout option hit our app this week: Message Bubbles! If you’re used to seeing inbound messages on the right, and outgoing messages on the left this might be for you. You can access the new appearance option from Settings.
I wrote a small web app thing that can download, decrypt and display Matrix media in a browser. The goal is to use it in the Android SMS bridge for sending attachments that are too big for MMS, but it might be useful for other things too.
Currently it consists of the web frontend, a maubot plugin to generate links, and a small server that stores file metadata (so the URLs just contain the decryption key and a short ID to find the metadata on the server). I'll probably continue working on the exact URL format to make it shorter and to encrypt the metadata, and maybe also add an alternative mode where all info is included in the URL to make the server component optional.
I worked on adapting the NixOS module for Synapse to support worker deployments. It's a great reproducible deployment method, and super easy to configure. My module is perhaps less quality than the rest of the chain, but it's pretty neat to define your whole Matrix deployment like this: https://git.pixie.town/f0x/nixos/src/branch/synapse-workers/nodes/cosmos/containers/synapse-workers.nix#L60 and have Nix figure out all the systemd units and nginx routes that have to be added
Full Nix code for the module is at https://git.pixie.town/f0x/nixos/src/branch/synapse-workers/common/modules/synapse
pictured is the ping stat for a little test deployment with 8 federation senders :P
matrix-docker-ansible-deploy now supports installing the matrix_encryption_disabler Synapse module (details here), which homeserver admins can use to prevent End-to-End-Encryption from being enabled by users on their homeserver. The popular opinion is that this is dangerous and shouldn't be done, but there are valid use cases for disabling encryption discussed in this Synapse issue.
Pushed 234 updates and enhancements to the automation framework used as the service core
Integrated 17 additional components to the matrix stack
Developed 5 bots and tools to extend matrix capabilities
Installed 92 new matrix servers
Helped 172 people and organizations to achieve their goals in the matrix
Posted 74 updates in the announcements room
Some history:
the project (not service yet) started on February 12, 2021
the first installed server was etke.cc itself (yes, etke.cc homeserver is a customer of etke.cc service from the day 1)
the second installed server was a chatbot, that uses matrix as a platform to interact with users across different chat networks (Telegram, WhatsApp, Signal, etc.) - one API to rule them all.
After a few years, matrix-room-directory-server has finally gotten an upgrade to modern times. Intended to eventually be a standalone directory server (making vanity aliases possible, but otherwise not functional as a homeserver), it currently only supports overriding the federation /publicRooms endpoint.
The changes made today are to replace the old, broken, appservice-backed approach with a space-backed approach. You can see this in action on t2bot.io: querying the room directory over federation will hit the room directory server, which is watching #directory:t2bot.io in the background to determine which rooms to serve.
Hay y'all. I created a blog post about how to host element and matrix .well-known files using cloudflare pages that I thought might be worth sharing. Have a few kinks to work out regarding CORS (might need to clean my browsers cache) though.
https://minecraftchest1.wordpress.com/2022/02/17/hosting-element-and-matrix-well-known-files-with-cloudflare-pages/
https://matrix.to/#/#minecraftchest1-blog-matrix-elemet-cloudflare:matrix.org
Dept of Ping 🏓
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server.
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
MSC3440 (threading) continues to move forward with a proposed final comment period this week. As a really useful feature in Matrix, it's great to see it getting closer to landing in the spec.
In a similar vein, MSC3613 (combinatorial join rules) had a proposed final comment period. The intention of this is to be a simple mechanism for resolving the "it's not possible to have both a restricted and knock room" situation - as both introduced mutually exclusive join rules. This MSC will likely serve as the basis of a new room version whenever it lands.
This MSC makes the case for allowing widgets to be created in private rooms by default. The actual spec for it is blocked behind having widgets in the spec (see this tracking PR if you're interested).
As noted in the PR though, this MSC could build on the existing im.vector.modular.widgets unstable prefix for widgets in order to create an implementation for this feature. It does not appear to have happened yet though.
Hello, TWIM! This week we released Synapse 1.52, which includes new Admin APIs for monitoring federation status and immediately retrying failed connections, as well as for reporting on rooms shared between two homeservers. We've also fixed / improved media previews for sites like Reddit. Plus lots more fixes / refactors / improvements under the hood.
This is a smaller release as it's coming off of our FOSDEM sprint, but we look forward to getting back to more regularly-sized and featureful Synapses this month.
Notably, we've been experimenting with Poetry to better manage Python dependencies in our applications (Synapse, Sygnal, and Sydent). We just switched Sydent over this week and are working out the kinks in our CI and build pipelines. We're looking forward to bringing this consistency to Synapse next month!
Yesterday we released Dendrite 0.6.3 which is a primarily bug fix release. It contains the following changes:
Initial support for m.login.token
A number of regressions from earlier v0.6.x versions should now be corrected
Missing state is now correctly retrieved in cases where a gap in the timeline was closed but some of those events were missing state snapshots, which should help to unstick slow or broken rooms
Fixed a transaction issue where inserting events into the database could deadlock, which should stop rooms from getting stuck
Fixed a problem where rejected events could result in rolled back database transactions
Avoided a potential race condition on fetching latest events by using the room updater instead
Processing events from /get_missing_events will no longer result in potential recursion
Federation events are now correctly generated for updated self-signing keys and signed devices
Rejected events can now be un-rejected if they are reprocessed and all of the correct conditions are met
Fetching missing auth events will no longer error as long as all needed events for auth were satisfied
Users can now correctly forget rooms if they were not a member of the room
As always, please feel free to join us in #dendrite:matrix.org for more Dendrite-related discussions!
NeoChat 22.02 is out with the possibility of sharing files directly from NeoChat to other apps and services (Nextcloud, Imgur, email, ...). We also added support for minimizing the application to the system tray on startup and you can now label accounts to better organize them if you are using the multi-account feature. Aside from that, we spend some time fixing many small bugs and paper cuts reducing the total amount of open bug reports by 20%.
We've release 0.2.26. You can now create rooms and direct messages with Hydrogen. There's also a bug fix in this release for replies not loading in e2ee rooms under certain conditions.
Welcome to another week of TWIM at Element! Here’s our updates:
Polls and Location Sharing
Polls is now out of labs, and available in the composer for all users with the latest app versions.
Location sharing is available on the mobile apps. For now you will need to enable it in settings in order to see the composer icon and send your location. As of next week’s releases it will be enabled by default on all platforms.
Threads
Threads aims to reduce cross-talk on the timeline by moving side-convo’s to the right panel. If you want to try it out, it's available in Labs on Web today and in Labs on Mobile from next week!
We’ve been working on improving the stability and speed of threads across platforms.
Community Testing
Join#element-community-testing:matrix.org to join us for future testing sessions!
Metaspaces have landed! Giving users a new way to display favourites, DMs and rooms outside of other spaces. Switch these on in Quick Settings at the bottom left of your app.
New and improved Search! We’re pleased to move the new search experience into Beta. Head to Settings > Labs to access it.
Those of you using Nightly or Develop will see the new experience by default.
Provide feedback on your experience directly from the Search window.
On Web we’ve also been chasing away some bugs, specifically desynced memberlist and stale display names.
In Labs (Enable labs features in settings on develop.element.io or Nightly)
Thanks to everyone who tested the maximised widgets feature which was recently added to Element Web. It received a sudden burst of attention as part of the FOSDEM conference. 😄 This week we've been fine tuning the feature to ensure it works for more workflows, and we've also cleaned up various associated widget bugs. In particular, in the develop version of Element Web, the pin and maximise actions are more consistent accessible, so you can toggle between pinned and maximised if e.g. you're in a conference room and temporarily want the stream widget to be larger while still having access to the chat timeline.
Next week we'll be focusing more on our Matrix-native collaborative document plans.
The next update of iOS (1.8.0) will have improved emoji reactions and location sharing will be on by default!
We will also be releasing fixes to the incorrect scrolling on the timeline and home screen sorting.
Work on a Rust prototype is also underway. We’re excited to learn about the opportunities and advantages of this approach as we start to learn and experiment.
In Labs
From next week Threads will be available in Labs on Mobile. Switch it on and try it out!
Beeper is a universal chat app built on top of Matrix. We've created 12+ open source Matrix bridges and integrated them into an easy to use all-in-one service which does not require setting up your own homeserver.
We’re excited to announce that as of today we are now out of Beta! We have been hard at work and we’re now ready to be sharing Beeper with a lot more people, starting first with the people we’ve built up on our waitlist.
For more details and demos on the current state of Beeper, including our recently rebuilt and relaunched iOS app, check out the update on our blog: https://blog.beeper.com/p/beeper-update-4-out-of-beta or watch the video below.
The new version contains two new features around the Synapse Admin API, making someone a room admin and listing all unencrypted media URLs.
I've had some good ideas on how to make Matrix Wrench cover more use cases while staying simple for both users and me as a developer. New components allow me to add new API endpoints in as little as 7 lines. So far this only applies to simple endpoints without parameters, like joining and leaving a room, but it can be expanded to more endpoints because of how similar all Matrix endpoints are. Props to the Spec! 🙌
Trixnity is now on version 1.1.7. Since 1.1.0 we have fixed many bugs and added some improvements and features. Our reference client runs amazingly reliable and fast thanks to Trixnity 🚀
Version 1.1.1 released
This was a quick bugfix release for a finding by jeffcasavant, in which joining a room with a predecessor, the bot could crash. Thanks Jeff!
If you missed it last week, I published a template / demo stock reporting bot you can find here: https://github.com/WesR/Halcyon-stock-bot . Short and sweet, this little bot should be a great starting point if your looking into making a bot. Make sure to also checkout the library itself for more examples, and usage information https://github.com/WesR/Halcyon.
I'm always looking for more feedback, and love to see what people are working on. Come hangout with us in the Halcyon room: https://matrix.to/#/#halcyon:blackline.xyz
We have been using Matrix as a chat solution for the attendees of our biannual conference (KIF) and for new students during the introduction week, as well as Pretix to register attendants. Up until now the process to invite attendees to the corresponding Matrix Rooms and Spaces involved some curl and jq magic as well as manual intervention to connect both.
We have now replaced said manual process with a Pretix plugin that invites registered attendees to a configurable Matrix Room or Space. This plugin is now available on the Pretix marketplace or directly from its repository.
I'm excited to share a side-project I've been working on for the last ~3 months: Gatho (http://gatho.party) - a web app for hosting small gatherings.
Gatho is perfect as a replacement for Facebook Events for small social gatherings. I'm a 24 year old Australian who recently deleted Facebook - I've used Gatho for a few parties now and the guests have loved it!
Gatho includes a Matrix bot which, when added to a room, can synchronise RSVP emoji reactions to a linked Gatho event! See gatho.party/getting-started to hear how the Matrix synchronization works.
You can also create one-click RSVP links to send to your friends - no matter which chat/social app they use!
It's fully GDPR compliant and multi-region with an EU server and database, and uses NextAuth.js for passwordless authentication.
The Gatho website and bot is open source (AGPL-3.0) on Github, PRs and Github issues are very welcome! It's built using Next.js in Typescript.
The ping room has been upgraded to room v9. During the 452 days the previous (v6) ping room was alive, it received a total of 735997 org.matrix.dummy_events and 604459 m.room.message events.
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server.
Synapse 1.52 is out! Here's what's new with this week's release.
Twisted security release
The team behind Twisted, which is the main framework Synapse uses under the hood, recently released Twisted 22.1. This version fixes a security vulnerability within the Twisted library.
While preparing the release of Synapse 1.52, we have investigated the impact of this vulnerability on Synapse. We came to the conclusion that it does not affect Synapse. We however advise server administrators to ensure they use an up-to-date version of the library as a matter of good practice.
For instances installed with pip, the library can be updated with pip install --upgrade Twisted treq. For instances installed with the matrixdotorg/synapse Docker image or Debian packages from packages.matrix.org, updating to Synapse 1.52.0 is sufficient, as these images and packages include up-to-date versions of all dependencies.
It is also worth noting that a release candidate for Twisted 22.2 has been published, with a fix for a potential denial of service vulnerability with SSH. Administrators of Synapse homeservers that have the manhole feature enabled (which is the only feature of Synapse using SSH) are encouraged to ensure access to the manhole is correctly restricted (e.g. by preventing access from external locations).
Federation admin APIs
This release of Synapse introduces a few admin APIs to help server administrators monitor and handle how their Synapse homeserver interacts with other federated homeservers. One of these APIs offers server administrators a way to visualise which rooms are shared between the local homeserver and a given remote one.
Another API allows server administrators to reset federation timeouts. If Synapse fails to connect to a remote homeserver, it will make note of the failure and will not retry the connection after a certain amount of time. This can happen if the remote homeserver goes offline or experiences connectivity issues. Synapse has a few ways of figuring out whether a remote homeserver has come back online, but this new admin API adds a way for administrators to manually tell Synapse a destination should be available.
Everything else
This release also improves Synapse's deactivation behaviour by deleting account data when deactivating a user. "Account data" refers to private arbitrary data that is specific to an account. It is used among other things for secure server-side storage (SSSS) which allows securely backing up end-to-end encryption keys.
Please see the Synapse release notes for a complete list of changes in this release.
Last year was the first time FOSDEM was hosted on Matrix, and it was generally a huge success - and so the FOSDEM team trusted us again this year and we’re happy to say that it seems to have gone really well! This year’s FOSDEM was massive once again, featuring 654 speakers, 731 events, and 103 tracks.
This year hosting the event went smoother than last year, the only significant issue was some of the Q&A Jitsis not being broadcast to the devrooms on Saturday before 10:15 UTC, for which we offer our apologies to the speakers impacted. This turned out to be a problem with the Matrix<->Jitsi access control sync system which hadn’t showed up during earlier testing, but we patched around it rapidly on the day.
The most notable difference between this year and the previous year has been the usage of a “attendees.fosdem.org” instance in addition to the original “fosdem.org” one, specifically for attendees. The graphs speak for themselves: Synapse could handle the load of the 23K users (13K joined users and 10K lurkers) spread across a total number of 941 rooms. The real eye-opener however is that of the 13K joined users, only 4K came came from the FOSDEM attendee server, and 1K from Libera Chat, meaning that ~70% of the Matrix participants were already on Matrix and came in from existing servers! 🤯 That means the vast majority of people attended over federation. Decentralisation at work, people! It works! We didn’t host the conference… you did!!
But not only did the backend handle the load smoothly: the general user experience felt tightly integrated. People were welcomed by a tailor-made home page in Element to help them navigate through all the tracks and stands:.
One of the great things is it doesn’t require heavy modifications to Element: anyone who installs their own instance of Element can use a simple html file to display relevant information to their audience.
New this year, we also generated a space hierarchy for the whole conference at #fosdem2022:fosdem.org to help navigate the maze of rooms, making it even easier for users on their own servers to jump in:
Another greatly appreciated feature was the famous “maximised widgets” I (Thib) keep telling you about in Matrix Live episodes. Attendees and speakers could give the conference the central attention it deserved while simultaneously keeping an eye on what was happening in the chat.
From the speaker's perspective, we tried to streamline the user journey as much as possible: a bot invited them to a backstage room, in which they joined a Jitsi widget while their talk was being played in the track or devroom. They could see the most upvoted questions by the audience in a dedicated widget. A few minutes before their pre-recorded talk was over, a countdown (new this year!) could be displayed to tell them and the host they were about to go live. At the end of the countdown, the backstage Jitsi was broadcasted to the track so the speaker could answer the questions.
If you want to have an in-depth look at the backend’s architecture, it didn’t change much from last year. You can have a look at last year’s blog post for the details on the setup. Most of the heavy lifting was around the conference bot used to set rooms up, create the spaces, populate them with widgets, arrange layouts and trigger countdowns before going live…
Huge thanks to the FOSDEM team for trusting us, massive shout-out to Element Matrix Services and Element’s Ops and infrastructure team for their fantastic job in setting everything up and making sure everything was ready in time, a sincere thank you to all the fantastic speakers who shared awesome content, and finally to all the attendees. What a weekend!
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://spec.matrix.org/unstable/proposals.
I'd also like to point out that v1.2 released on 2/2/2022 :)
Otherwise, thank you to aaronraimist and devonh from the community for their fixes to the spec's text that merged this week. They are highly appreciated!
This MSC proposes to pave a way to allowing relating an event to multiple other events, which unlocks some additional use cases. Go check it out if you're interested in extending Matrix's capabilities even further!
Synapse 1.52.0rc1 is out, but honestly, we're just all hands on deck for FOSDEM tomorrow! As a reminder, members of the Synapse team will be giving a couple of talks during at the event:
Today we've released Dendrite 0.6.1 which contains a number of fixes and improvements. This release includes the following changes:
Roomserver inputs now take place with full transactional isolation in PostgreSQL deployments
Pull consumers are now used instead of push consumers when retrieving messages from NATS to better guarantee ordering and to reduce redelivery of duplicate messages
Further logging tweaks, particularly when joining rooms
Improved calculation of servers in the room, when checking for missing auth/prev events or state
Dendrite will now skip dead servers more quickly when federating by reducing the TCP dial timeout
The key change consumers have now been converted to use native NATS code rather than a wrapper
Go 1.16 is now the minimum supported version for Dendrite
Local clients should now be notified correctly of invites
The roomserver input API now has more time to process events, particularly when fetching missing events or state, which should fix a number of errors from expired contexts
Fixed a panic that could happen due to a closed channel in the roomserver input API
Logging in with uppercase usernames from old installations is now supported again (contributed by hoernschen)
Federated room joins now have more time to complete and should not fail due to expired contexts
Events that were sent to the roomserver along with a complete state snapshot are now persisted with the correct state, even if they were rejected or soft-failed
Spec compliance, as measured by Sytest, currently sits at:
Client-server APIs: 65%
Server-server APIs: 94%
As always, you can join us in #dendrite:matrix.org for Dendrite discussion and announcements.
It's FOSDEM! Don't forget to checkout my 5 min ramble about custom stickers and emotes in Matrx: https://fosdem.org/2022/schedule/event/matrix_custom_stickers/
In preparation for FOSDEM, I've also been working on some goodies so that you can participate using Nheko in a limited manner:
Previews for rooms are now fetched for spaces using the hierarchy API
Widgets for the talks should be shown below the topic as links
While this isn't grand, it should be good enough for some to participate using Nheko. You will need the latest master (some commits aren't even pushed at the time of writing!), but if you are just watching, this should give you an easy way to fetch the talk link for each room. You will still have to watch the actual talk in your browser though.
LorenDB also added a big, fat, red offline indicator. If your server dies because of FOSDEM, now Nheko will tell you.
I hope you all will have a great time and I hope I see you around!
Very customizable, with ability to adjust any keybinding/functionality
A multi-account client
Adjusts to any screen size
After 6 months of inactivity on the Mirage project, Moment brings it back to life. We have fixed some important issues and will continue to maintain Moment!
🎺 We've released Hydrogen 0.2.24 and 0.2.25 with session backup writing and some bug fixes! From now on you shouldn't need to have another client running along hydrogen for keys to be written to the key backup!
Welcome to another week of TWIM at Element! Here’s our updates:
Polls and Location Sharing
Polls is now out of labs, and available in the composer for all users with the latest app versions.
Location sharing is also now available on the mobile apps. For now you will need to enable it in settings in order to see the composer icon and send your location.
Threads
Threaded Messaging is making forward progress on all 3 platforms; we’re aiming to help clear up cross-talk on the timeline by moving side-convo’s to the right panel. If you want to try it out, it's available in Labs on Web. We’ll be pushing Threads into Labs on Mobile in the next release!
Bubbles are now available! Keeping your inbound messages on the left, and your outbound messages on the right your timeline should now be easier to skim read. This layout is off by default but to see it in action, update your Message Layout appearance preferences from Settings.
Metaspaces have landed! Giving users a new way to display favourites, DMs and rooms outside of other spaces. Switch these on in Quick Settings at the bottom left of your app.
In Labs (Enable labs features in settings on develop.element.io or Nightly)
New and improved Search!
Provide feedback on your experience directly from the Search window.
Threads, including design updates. The MSC is available hereMSC3440
We’ve been working hard on improving the startup and resume times, you should start to see these in your app in 1.7.0.
Work on a Rust prototype is underway. We’re excited to learn about the opportunities and advantages of this approach as we start to learn and experiment.
Also, Xcode13 & iOS15 compatibility has been added for developers
Coming soon in 1.8.0! Bubbles and an improved Onboarding experience
Threads will also be hitting Labs on Mobile soon so let us know what you think.
A hotfix (1.3.18) on Android will fix some bugs we found in this week’s release.
Bubbles will also be landing soon, you can find this new feature in Settings > Preferences > Timeline. The feature has been merged to develop if you want to give it a try!
Threads have also been merged on develop.
We’ve been making improvements to the Onboarding experience for new users too.
In Labs
Threads will soon be in Labs on Mobile. Switch it on and try it out!
Here's a big populus viewer update! Since last time, I've been mostly working on improving the user experience and onboarding for non-experts, as well as making my teaching-assistant bot a little smarter - this work is ongoing. But I have had time for a few new features 😁
I've reworked the sidebar for the PDF view, improving the aesthetics and allowing for a "fullscreen" view of PDF content.
I've added user-directory search and improved the usability of the invite modal.
The reason that MSC3574 is not a final draft is that I'm increasingly looking at the w3c's web annotation data model as a compelling foundation for annotations on Matrix. Stay tuned for a upcoming MSC, or come over to #opentower:matrix.org to talk about the future of annotation on Matrix!
Populus-Philarchive
I've also been working on a proof-of-concept for one Matrix use-case that I'm hoping Populus can help fill. Social annotation can be a good tool for getting feedback on works in progress, or for discussing new research with your team or lab. Wouldn't it be nice if you could just pop open a paper on a preprint archive and start commenting, or say "hey friends, I'm reading this paper, what do you think?" And wouldn't it be nicer if you could do that and host the discussion on your university or team's Matrix server?
Here's a first pass at that idea: https://opentower.github.io/populus-philarchive/
At the moment works by just pasting in paper codes from philarchive.org - it'll preview bibliographic information and give you a list of discussions taking place around a given paper, with the option to create a new one. Sessions are shared with populus-viewer, and pdfs continue to be hosted by the original archive. There are a number of clear upgrade paths, by integrating with a preprint archive that has an open search API, or even by adding an OAI-PMH metadata harvester to the backend, to combine the metadata from a bunch of open paper archives. I'm really excited to see where this work goes.
I have been working on a little project over the last week: nheko-krunner. nheko-krunner is a KRunner plugin that loads rooms from nheko, displays them to the user, and allows the user to activate said rooms. How does it do this? Well, I've also been creating a D-Bus API for nheko! This code has not been merged yet, but once it is, you will be able to create your own plugins that access nheko via D-Bus!
The current functionality of nheko-krunner is simply (1) search and switch to rooms that you are in (not unlike the Ctrl-K switcher), and (2) join rooms from their aliases.
If you want to try out nheko-krunner, you will need to build from the dbus branch in my personal fork of nheko and then install nheko-krunner from the above repo.
I hope that somebody finds this useful and I am excited to see what other D-Bus plugins may show up in the future!
The versions 0.4.x brought improvements to the network log, letting you spot and investigate HTTP errors, bad JSON and network errors.
Ideally, I want to focus as much on the Matrix Spec as possible, but for the v0.5.0 I might venture into the territory of the Synapse Admin API, e.g. for listing and deleting media in a room. Please contact me, if you have use cases around media moderation you'd like me to consider.
No new release this week. I've been working on the video sending functionality, and I am looking forward to seeing what that enables ya'll to do!
In the meantime, I've published a template / demo bot you can find here: https://github.com/WesR/Halcyon-stock-bot . In just 37 lines of (unminified) code, this little bot:
Sets a status message
Automatically joins rooms via invite
Searches messages for $stocks mentioned, then formats and replies the current price and daily percent change for all the tickers in the message
Screenshot below
I'm always looking for more feedback, and love to see what people are working on. Come hangout in the Halcyon room: https://matrix.to/#/#halcyon:blackline.xyz
Lastly, for more info at on the bot library visit https://github.com/WesR/Halcyon
Happy Hacking!
The Polyjuice Project has a new component: Polyjuice Client Test, a tool for testing Matrix clients. Each test has its own preconfigured homeserver environment, implemented using the Polyjuice Server library, and can be customized according to the needs of the test. Only a few tests are implemented so far, but many more are planned, initially focusing on testing functionality related to end-to-end encryption. You can see a demo of it in the Matrix Live video.
matrix-corporal (as of version 2.2.3) is now published to Docker Hub (see devture/matrix-corporal) as a multi-arch container image with support for all these platforms: linux/amd64, linux/arm64/v8 and linux/arm/v7. Users on these ARM architectures no longer need to build matrix-corporal manually.
New version comes with in-memory caching for thread relations. It significantly decreases amount of requests to homeserver during thread relations solving, both for MSC3440 threads and reply-to chain fallback
This week the focus was on fixing quirks and making the profile (at least partially) editable and adding translations.
Changes
Fonts now get hosted on the page itself due to the legal issues in Germany with Google Fonts hosted fonts
Profile page has a rough UI to edit the profile page if logged in
Licence gets now correctly displayed according to the Creative Commons Licence requirements
Initial work on translations has started. A Weblate instance has been set up at https://trans.nordgedanken.dev for this.
German translation was added
Progress on the HS side to be able to use it as a public registration server for anyone who wants to post to Matrix Art. (yes, this really has a public registration HS)
Mjölnir instance is set up which also is used to moderate Matrix Art in complete (Aka it is joining on creation.) This is used for being able to moderate the website.
ChatStat is an R package for making reports on Matrix rooms.
Ahead of my lightning talk on Sunday, ChatStat has been polished up and given it's first release. You can get it here. Huge thanks to @johrpan:johrpan.de for their contributions too.
The main highlight since my last TWIM is that we actually have report generation now! You can see an example here, or check the project wiki for an example of using the raw data with ggplot2 to make custom plots of your own.
The season 2 of Raising Dion (a Netflix TV show) featured shortly a video call with Riot/Element and a few other open source software (e.g. Karbon). Oh and you might recognize a few usernames in the sidebar 😅
Circles has made it into a small German speaking Apple News site/App called ifun.de.
„Circles-App: Neue alte Ideen für private soziale Netzwerke“ https://www.iphone-ticker.de/circles-app-neue-ideen-fuer-private-soziale-netzwerke-185764/
There wasn’t a mention of Matrix - which is kind of exciting really. This means it can just be the transparent layer of great apps :)
Dept of Ping 🏓
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server.
Today the European Parliament, the European Council and the European Commission will meet again for a discussion about the Digital Markets Act (DMA). This is the second of three of these meetings, appropriately called trilogues, where each party exposes their stance on a proposed law and the group tries to agree on the final version.
The DMA is a groundbreaking step forward in shaking the hold a few gatekeepers have on users and the market, in particular because it looks to (among others):
Require gatekeepers to allow other services to interoperate with their services
Prevent them to treat their own services and products more favourably (for example by ranking)
Require them to allow users to uninstall any pre-installed software or app
The interoperability obligation is obviously the one on which we’ve kept a particularly close eye, as if it lands well it could take the success of Matrix to the next level completely overnight.
However, whilst in our mind interoperability automatically implies “open standard”, there are actually different ways of implementing it, depending on how far one wants to go. Typical debates here have been between whether to force gatekeepers to maintain open and well documented APIs, or whether to go full swing and mandate an open standard, and every shade in between.
We’ve been lucky to have had the opportunity to talk to policy advisors from different European member states, and it has been pretty fascinating to realise that it was always the same arguments which were being presented back at us, straight from the gatekeepers partyline.
We’ve ended up just listing them in a quick, high level, Myth Debunking exercise and thought it would be useful to actually publish them for everyone to access, so here they are!
MYTH #1 - "It is impossible to have a standard that is open, decentralized and secure at the same time"
⇒ false: HTTPS did it, Matrix did it.
MYTH #3 - "Interoperability is incompatible with end-to-end encryption"
⇒ false: services just have to speak the same language, email has proved this with S/MIME and PGP - where different vendors can and do interoperate with E2EE. It’s even better when the protocol is E2EE by default.
MYTH #4 - "It may work for messaging, but less so for social networks"
⇒ false: it's still about managing content and users. Even though social networks have more varied content, it is already well modelled for their own APIs, ready to be expressed in a common language. The key is in the fallback option on unsupported features, as well as the ability to have moderation tools (more on that later).
MYTH #5 - “Interoperability is not compatible with data privacy”
⇒ false: Interoperability gives the ability to users to choose who is hosting their data and as such choose providers they trust. Besides, the DMA doesn’t live in a vacuum: it will exist alongside horizontal regulations like the GDPR and the Data Act, which give people sufficient control over their data to rectify their choices if they are not happy. Because the possibility of interoperability is there, it does not mean it will become mandatory for users to use it: they will still have their own threat models and will make decisions accordingly, just as they do today. But enshrining interoperability in law will at least ensure gatekeepers need to provide recourse for people to have further control over their data, which will be an improvement from the landscape today.
MYTH #6 - "There is no user need"
⇒ false: most haven't had a taste of interoperable chat/social media (but they know email), others are demanding bridges between services: 25% users of 2 communication apps lose contact with friends because they are using too many apps. And this figure doubles for people using more than 5 apps. There was no demand for cars when they were created: people only wanted faster horses.
MYTH #7 - "There is no demand from European companies"
⇒ false: The fact it is so hard for European companies to remain competitive enough to stay alive means there are few of them to complain about what is killing them! However these companies are gathering to push for interoperability (like the Coalition for Competitive Digital Markets). It will enable them to be more innovative in the product they develop by benefiting from an existing open network rather than being slowed down by having to build one from scratch. Companies will compete on the value they add rather than the size of their network. An open standard also gives an open field for innovation from a business model perspective. The Web is an excellent example of how much an open network fuels innovation and growth.
MYTH #8 - "It is better to require providers to have open and stable APIs than define a single open standard"
⇒ false: this is the best way to leave gatekeepers at the center of the ecosystem as it means that each player has to multiply its effort to interface with every single other player, but every player will only have the resources to interface with a few of its counterparts and will logically default to the bigger ones, effectively not solving the problem. In addition, if providers are not aligned on which encryption to use it will just break end-to-end encryption and create risk for the user in every bridge. In practice the DMA is about forcing the gatekeepers to interoperate only, but we strongly believe that everyone should be interoperating if we are about improving the user’s experience and control, and giving more space to companies to innovate. Limiting it to the gatekeepers is a first step, but only a defensive one.
MYTH #9 - “An open standard limits innovation if it defines a lowest common denominator”
⇒ false: the lowest common denominator should match what users consider as table stakes in a messaging or social media app. Providers can innovate on top by providing different features which go beyond table stakes, for example by targeting niche use cases, like messaging services focused on elderly and disabled users, or focused on healthcare, warehouse workers, or integrated in a CRM for call centers, or creatives… Providers also can implement a profile of the standard which is a subset of its full scope, ensuring the standard remains a highest common denominator..
MYTH #10 - “It will be impossible to moderate social networks built on an open standard”
⇒ false: decentralised networks actually have driven the adoption of much more sophisticated moderation techniques than the coarse approaches of centralised silos. Appropriate moderation means have to be part of the open standard definition, and some are already used in Matrix. It would also empower victims who today have no choice but get in touch with providers one by one. Each provider will also have control over their own users, and users can select providers whose T&Cs are aligned with their ethics. The world is not black and white, unlike what Silicon Valley tries to make us believe.
MYTH #11 - “It will take years before being able to define an open standard”
⇒ you don’t have to: You could leverage existing technologies which are being used by the industry. Matrix, XMPP and ActivityPub exist today. For instance, Matrix has been managed by its own standard body (The Matrix Foundation) and could be ratified by a more established one like IETF, ETSI or W3C if needed.
Obviously the devil will be in the details of the way the final text is formulated, as well as the limits, obligations and controls put in place, but overall it should be an improvement for all European users and companies and we’re looking forward to seeing how today’s trilogue goes!