We've done quite a bit of work this release on Synapse's new interface for extension modules:
Synapse's account_validity option has been extracted into a built-in extension module. If your configuration previously enabled account_validity, it will continue working as before, but it is now easier to override and customize.
The third party event rules callbacks have also been experimentally ported to the new module interface.
We've also improved the Space Summary API to list all rooms which a user could join (#10298). This is especially important with the pending standardization of Room Version 8, which includes MSC3083: Restricting room membership based on membership in other rooms.
MadLittleMods landed several pull requests on the path to implementing MSC2716: Incrementally importing history into existing rooms (#10250, #10276).
More than anything else, this release contains an absolutely enormous load of internal cleanup. For example:
ShadowJonathan ran pyupgrade on the entire codebase, bringing it up to modern Py3.6+ conventions, then separately used com2ann to move us from legacy # type: foo type hints to more modern inline annotations.
We finished converting several integer columns in our database to bigint as a precautionary measure.
We've made innumerable improvements to our CI configuration; including moving completely to GitHub Actions.
Countless small improvements to performance, reliability, and error logging.
But that's not all! Synapse 1.39 also allows for setting credentials for HTTP proxy connections thanks to work by dklimpel; previously, Synapse was only able to provide authentication for HTTPS proxies.
Big news today: Element, the startup founded by the team who created Matrix,
just raised $30M of Series B funding in order to further accelerate Matrix
development and improve Element, the flagship Matrix app. The round is led by
our friends at Protocol Labs and Metaplanet,
the fund established by Jaan Tallinn (co-founder of
Skype and Kazaa). Both Protocol Labs and Metaplanet are spectacularly on
board our decentralised communication quest, and you couldn't really ask for
a better source of funding to help take Matrix to the next level. Thank you
for believing in Matrix and leading Element's latest funding!
You can read all about it from the Element perspective over at the
Element Blog,
but suffice it to say that this is enormous news for the Matrix ecosystem as a
whole. In addition to transforming the Element app, on the Matrix side this
means that there is now concrete funding secured to:
finish building out P2P Matrix
and get it live (including finishing Dendrite, given our P2P work builds on Dendrite!)
adding Threading to Element (yes, it's finally happening!)
speeding up room joins over federation
creating 'sync v3' to lazy-load all content and make the API super-snappy
lots of little long-overdue fun bits and pieces (yes, custom emoji, we're looking at you).
If you're wondering whether Protocol Labs' investment means that we'll be
seeing more overlap between IPFS and Matrix, then yes -
where it makes tech sense to do so, we're hoping to work more closely
together; for instance collaborating with the libp2p team on our P2P work
(we still need to experiment properly with gossipsub!), or perhaps giving
MSC2706
some attention. However, there are no plans to use cryptocurrency incentives
in Matrix or Element any time soon.
So, exciting times ahead! We'd like to inordinately thank everyone who has
supported Matrix over the years - especially our Patreon supporters, whose
donations pay for all the matrix.org infrastructure while inspiring others to
open their cheque books; the existing investors at Element (especially Notion
and Automattic, who have come in again on this round); all the large scale
Matrix deployments out there which are effectively turning Matrix into an
industry (hello gematik!) -
and everyone who has ever run a Matrix server, contributed code, used the
spec to make their own Matrix-powered creation, or simply chatted on Matrix.
Needless to say, Matrix wouldn't exist without you: the protocol and network
would have fizzled out long ago were it not for all the people supporting it
(the matrix.org server can now see over 35.5M addressable users on the
network!) - and meanwhile the ever-increasing energy of the community and the
core team combines to keep the protocol advancing forwards faster than ever.
We will do everything we possibly can to succeed in creating the long-awaited
secure communication layer of the open Web, and we look forward to large
amounts of Element's new funding being directed directly into core Matrix
development :)
It's only been two weeks since Dendrite 0.4 landed, but there's already a
significant new release with Dendrite 0.4.1 (it's amazing how much work we
can do on Dendrite when not off chasing low-bandwidth and P2P Matrix!)
This release further improves memory performance and radically improves
state resolution performance (rumour has it that it's a 10x speed-up).
Meanwhile, SS API sytest coverage is up to 91%(!!) and CS API is now at 63%.
We're going to try to keep the pressure up over the coming weeks - and once
sytest is at 100% coverage (and we're not missing any big features which sytest
doesn't cover yet) we'll be declaring a 1.0 :)
If you're running Dendrite, please upgrade. If not, perhaps this would be a
good version to give it a try? You can get it, as always from,
https://github.com/matrix-org/dendrite/releases/tag/v0.4.1. The changelog
follows:
Key notary support is now more complete, allowing Dendrite to be used as a notary server for looking up signing keys
State resolution v2 performance has been optimised further by caching the create event, power levels and join rules in memory instead of parsing them repeatedly
The media API now handles cases where the maximum file size is configured to be less than 0 for unlimited size
The initial_state in a /createRoom request is now respected when creating a room
Code paths for checking if servers are joined to rooms have been optimised significantly
A bug resulting in cannot xref null state block with snapshot during the new state storage migration has been fixed
Invites are now retired correctly when rejecting an invite from a remote server which is no longer reachable
The DNS cache cache_lifetime option is now handled correctly (contributed by S7evinK)
Invalid events in a room join response are now dropped correctly, rather than failing the entire join
The prev_state of an event will no longer be populated incorrectly to the state of the current event
Receiving an invite to an unsupported room version will now correctly return the M_UNSUPPORTED_ROOM_VERSION error code instead of M_BAD_JSON (contributed by meenal06)
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
Room Version 8 has been proposed as part of MSC3289! This intends to include the new "restricted" room version from MSC3083, which allows access to a room based on one's membership in another room, as is most useful for features related to Spaces.
Sorus Image Packs MSC2545 is taking shape more and more! While barely anything happened on the actual MSC itself, nheko recently implemented a sticker picker based on it, and there is a PR for fluffychat for a sticker picker, too, finally making stickers interoperable between clients of different vendors!
Here's a video of fluffychats sticker picker in action: https://gitlab.com/famedly/fluffychat/uploads/26382317e2424d7566eb18e102ddd6cb/fluffy-sticker-picker.webm
🚨 Yesterday we published Synapse 1.38.1 which mitigates a client bug triggered by changes to /sync responses in 1.38.0. Specifically, new Element Android sessions were unable to decrypt messages in end-to-end encrypted conversations.
⚙️ Otherwise, we've been doing quite a lot of internal / infrastructure work. Notably, we've completed the move from three separate CI systems to just one: GitHub Actions. We're also, for the first time, publishing Debian packages for our Release Candidates. We hope these changes move us towards a more reliable and seamless release process.
We'll have a lot to talk about re: Synapse 1.39 next week, but until then, enjoy the weekend! 🏖
A bridge for LINE Messenger based on running LINE's Chrome extension in Puppeteer.
This week brings plenty of usability improvements:
There is now the list-contacts bot command to--you guessed it!--list the Matrix puppets of all of your contacts.
It's now possible to sync a chat with a LINE user by inviting their Matrix puppet to a DM, instead of having to wait for someone to send you a message in LINE for the bridge to auto-sync it. This is especially useful with list-contacts.
LINE group chats that haven't been used recently are now included in the set of chats that get bridged on a sync operation.
The number of chats to sync at startup time & on-demand is now configurable, as a bridge config setting & a parameter to the sync command, respectively.
These changes make it possible to sync LINE chats/groups that previously may not have gotten synced, due to the intricacies of how LINE on Chrome handles chat history (especially on first use of the bridge).
LorenDB has rewritten the invite dialog as well as the memberlist in Qml. This means they work now much more reliably and show the users in appropriate colors and a few other small improvements.
Apart from that, Nheko now has support for sending stickers! This is based on Sorunome's awesome MSC2545, so no integration manager is needed and it is trivially easy to share your sticker packs (just invite someone into the same room!). You can use stickers from your account data or upload stickers into a room so every room member can use it. If you found a sticker pack you like, you can enable a pack from a room globally by going into Nheko's room settings. You can check my current preferred set of stickers here: matrix:r/nico's-stickers:neko.dev
Currently you can't edit a sticker pack yet from Nheko. Support for that will come in the near future. Until then you can use either FluffyChat, Element's DevTools or my hack to import Telegram sticker packs into a room. Note that Nheko's sticker pack support is incompatible with integration server based sticker packs like the one Element uses. On the other hand, you don't need an integration manager to use them! Custom emote support based in the image packs MSC is coming in the future to Nheko too.
Furthermore we fixed some bugs to move E2EE in Nheko out of Beta. We are close now and we have a plan!
As I write this, our interns Alejandro and Kai are at GUADEC, the annual GNOME conference, presenting during the interns lightning talk. The former has submitted a couple of merge requests (!793 and !794) that were necessary for the UI to accommodate his upcoming multi-account work. The latter has provided a view for when no room is selected. As for their mentor, Julian, he changed member loading so it only happens once a room is opened, to improve performances. He also changed Matrix event handling to use SyncRoomEvent, which paves the way for display of various message types in the timeline of a room.
Giuseppe offered two nice additions to the login view:
We’ve been shepherding MSC3083 into FCP to improve private spaces
We’ve also been working on restricted room support in Synapse and Element Web & Android
We’ve been improving onboarding experiences for Spaces, improving 3pid invites on Android, improving email invites and making matrix.to aware of spaces
On iOS, we’ve finished the first implementation of Space navigation which we can share soon!
We’re also figuring out what’s missing to mature Spaces enough to exit beta
VoIP
Call event tiles for 1:1 calls merged, for better summaries of call history.
1:1 screen sharing ready to merge!
Looking at bug where jitsi calls stay always-on-screen even when hung up.
Web
Released v1.7.33
Message bubbles PR merged!
Updated desktop to electron 13, complete with new n-api build of seshat (thanks, Poljar!)
More work on changelog tooling
iOS
We cancelled the release of 1.4.6 because of the discovery of high profile bugs or crashes . We will release 1.4.7 on the App Store on Monday. Those releases contain mainly bug fixes and papercuts
Voice messages are almost there. They are on develop under a LABS settings. They should be part of the next release candidate next Wednesday.
There are several papercuts in progress: URL preview, decryption of notifications content by default, removal of local contacts from the DM room list tab, removal of the room directory from the group room list, etc to list the major ones.
Android
Release 1.1.14 which fixes an important issue regarding message decryption is live on the PlayStore, on the beta channel, and will be in production quite soon. A patch has been applied to matrix.org to fix the problem.
Beside that, we are still polishing the spaces and voice messages
We've integrated the LinkedIn bridge into Beeper. The bridge is in beta and is generally ready for use. Since the last update, we moved it to GitLab, added a Docker image, and implemented reaction and redaction handling in both directions! We also fixed a bunch of bugs across the board to improve reliability.
The Android SMS bridge mentioned in the last update is progressing. It should be ready for self-hosted use in a few weeks, so our next update will probably include that.
Related to the SMS bridge, I've made mautrix-syncproxy, which is a microservice to do the /syncing required for end-to-bridge encryption (to-device events and such) in the cloud and forwarding the data to the normal appservice transaction endpoint. When combined with mautrix-wsproxy, it allows the SMS bridge to only have a websocket connection, which uses significantly less battery than a HTTP request every 30 seconds. The sync proxy partially implements MSC3202 and the to-device part of MSC2409 (partially because it only does individual users rather than all appservice-owned users).
We're hiring React, iOS, Android and SRE/Devops engineers. If you're interested, check out https://jobs.ashbyhq.com/beeper or DM Eric Migicovsky.
Just tagged and pushed version 2.4.0 of the Ruby SDK, along with an updated image for the example bot. This version adds support for the Matrix URIs as well as some basic spaces functionality, it also fixes some errors due to the removed empty /sync fields in Synapse 1.38.0
Several GNOME contributors and I have been following TWIM and really enjoyed the concept. Since open source is about stealing sharing ideas, we wanted to deploy something similar for GNOME.
I gave this room's beloved twim-o-matic a try and fiddled a bit with the code. But I soon found that it is too specialised for matrix.org. That's when I decided to write a project agnostic TWIx bot based on matrix-rust-sdk called Hebbot!
I released a first version on July 17, but I’m already working on making it smarter for a next release. In the meantime you can already use it for your project!
Love it, though I wish twim-o-matic had more of my attention! Maybe we can retire it and use Hebbot..?
This week I have developed a new plugin for OctoPrint which is a tool for controlling and monitoring 3D printers, which sends notifications about your prints and status of your 3D printer to matrix. The existing OctoSlack plugin also has Matrix support but fell foul of the /sync response changes in Synapse 1.38, and also sent the images from the camera to Matrix as inline images rather than m.image events, so they didn't render on some clients.
My new plugin can send messages on most events supported by OctoPrint, but the core events which are configurable through the OctoPrint web UI are print started, paused, completed and failed as well as print progress notifications. If you want Matrix notifications for your 3D prints, you can install the plugin through the official OctoPrint plugin repository or by installing from the GitHub repository. If you have any feature requests or pull requests, contributions to the plugin are very welcome.
Here is a preview of a print progress notification from something I was printing last night:
Following on from my post about how I'd like to convince Ansible to move to Matrix, I looked into the relative sizes of the networks, took a look at the active users in our channels, and even took a shot quantifying bridge restarts and netsplits to assess relative stability. Check it out here
A new Beyond Chat room (#beyond-chat:matrix.org) has been created! This room is meant for discussing various projects that build on Matrix for use cases beyond chat, such as collaborative documents, blogs, generic key-value databases, tools for thought, etc. Please join if these topics interest you. 😄
The Beyond Chat chat room is where Ryan chats about uses of Matrix beyond chat
Hey everyone, my name's Robert Long. I've just joined Element last week and I'm coming in to work on the VoIP team. This week I started working on full mesh video conferencing and I'm super excited to share more as the work progresses!
Welcome Robert! Can't wait to see progress here. We'll even bring back the Dept of VoIP :D
Hi everyone! Did you ever feel lost in the Matrix world? The room directory is big, but it's still hard to find something you like. Or are you a room moderator, but there is not much activity in your room because it doesn't have enough users?
This is why I want to share rooms (or spaces) I find interesting.
"A very helpful and friendly place for general discussion or questions pertaining to the FOSS Godot game engine and using it to make games, whether one is just beginning to learn or wish to discuss more advanced aspects of game (or Godot) development."
Amazing to see @Navy_Innovation showing off low-bandwidth @matrixdotorg decentralised comms to 1SL (Head of the @RoyalNavy) using Titanium: our CivTAK Matrix plugin. At this rate Matrix is on track to become the common language across the entire public sector!⚓️📻🇬🇧 https://t.co/UDIA2GBVm9
We’re incredibly excited to officially announce that the national agency for
the digitalisation of the healthcare system in Germany (gematik)
has selected Matrix as the open standard on which to base all its
interoperable instant messaging standard - the TI-Messenger.
gematik has released a concept paper
that explains the initiative in full.
With the TI-Messenger, gematik is creating a nationwide decentralised private
communication network - based on Matrix - to support potentially more than
150,000 healthcare organisations within Germany’s national healthcare system.
It will provide end-to-end encrypted VoIP/Video and messaging for the whole
healthcare system, as well as the ability to share healthcare based data,
images and files.
Initially every healthcare provider (HCP) with an HBA (HPC ID card) will be
able to choose their own TI-Messenger provider. The homesever for HCP
accounts will be hosted by the provider’s datacentre. The homeserver for
institutions can be hosted by TI-Messenger providers, or on-premise.
Each organisation and individual will therefore retain complete ownership and
control of their communication data - while being able to share it securely
within the healthcare system with end-to-end encryption by default. All
servers in the Matrix-based private federation will be hosted within
Germany.
Needless to say, security is key when underpinning the entire nation’s
healthcare infrastructure and safeguarding sensitive patient data. As such,
the entire implementation will be accredited by BSI
(Federal Office for Information Security) and BfDI
(Federal Commissioner for Data Protection and Freedom of Information).
Germany’s digital care modernisation law (“Digitale Versorgung und Pflege
Modernisierungs Gesetz” or DVPMG), which came into force in June 2021, spells
out the need for an instant messaging solution.
The urgency has increased by a significant rise in the use of instant
messaging and video conferencing within the healthcare system - for instance,
the amount of medical practices using messenger services doubled in 2020
compared to 2018 (much of this using insecure messaging solutions).
gematik, majority-owned by Germany’s
Federal Ministry of Health,
is responsible for the standardised digital transformation of Germany’s
healthcare sector. It focuses on improving efficiency and introducing new
ways of working by setting, testing and certifying healthcare technology
including electronic health cards, electronic patient records and
e-prescriptions.
TI-Messenger is gematik’s
technical specification for an interoperable secure instant messaging
standard. The healthcare industry will be able to build a wide range of apps
based on TI-Messenger specifications knowing that, being built on Matrix, all
those apps will interoperate.
More than 150,000 organisations - ranging from local doctors to clinics,
hospitals, and insurance companies - can potentially standardise on instant
messaging thanks to gematik’s TI-Messenger initiative.
By 1 October 2021, TI-Messenger will initially specify how communication
should work in practice between healthcare professionals (HCPs). Physicians
will be able to find and communicate with each other via TI-Messenger
approved apps - specifications include secure authentication mechanisms with
electronic health professional cards (eHBAs), electronic institution cards
(SMC-B) and a central FHIR directory. The first
compliant apps for HCPs are expected to be licensed by Q2 2022.
Eric Grey (product manager for TI-Messenger at gematik), reckons there will
initially be around 10-15 TI-Messenger compliant Matrix-based apps for HCP
communications available from different vendors.
Healthcare professionals will be able to choose a TI-Messenger provider, who
will be hosting their personal accounts and provide the messenger-client.
Healthcare organisations will choose a TI-Messenger provider to build the
dedicated homeserver infrastructure (on prem or in a data center), provide
the client and ongoing support.
Matrix is already integral to huge parts of the public sector; from the French
government’s Tchap platform, to Bundeswehr’s use of BwMessenger and adoption
by universities and schools across Europe.
Germany’s healthcare system standardising on Matrix takes this to entirely the
next level - and we can’t wait to see the rest of Europe (and the world!)
converge on Matrix for healthcare!
We'll have more info about TI-Messenger on this week's Matrix Live, out on
Friday - stay tuned!
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://spec.matrix.org/unstable/proposals.
Several members of the Spec Core Team reviewed MSC2674 (Event relationships) this week in order to help push along the efforts to finally ship aggregations in the spec. Otherwise MSC3245 (voice messages via extensible events) is moving along with final comment period proposed this week. The implementation in Element Web in reportedly working well, helping to prove the spec in practice. And finally, MSC3277 (scheduled messages) which appeared over the weekend to try and allow for scheduling events to send later in Matrix (and all the fun edge cases that come with it).
Thanks to everyone who submitted, read and reviewed MSCs this week. It takes people to move this stuff forward!
🔗Google Summer of Code 2021: first evaluations complete!
We heard from Callum last week, and will have more reports in future, but for now just know that all seven GSoC projects are progressing well. To remind yourself of this year's projects, see the list provided by Google or our welcome blog post.
In case you missed it, we released Dendrite 0.4.0 on Monday and wrote a blog post about it! It's taken us a little while to get to this release, but it includes a number of quality-of-life improvements and changes that will significantly reduce the amount of resources needed to run a Dendrite server. The full changelog is available on GitHub and many juicy details in the aforementioned blog post, but at a high level this release includes:
All-new state storage, designed to reduce the amount of disk space that the roomserver takes up to store room state by aggressively deduplicating state blocks and snapshots
Improved appservice support, with a number of bridges now working with Dendrite
Shared secret registration (using the same API shape as Synapse)
Optimisations in the federation API /send and /get_missing_events endpoints to reduce memory usage
Improved state resolution v2 performance when dealing with power level events
Per-room queuing to reduce head-of-line blocking on the roomserver input API
Lots of bug fixes around invites, registration, sync and media, and 5 panics fixed
Since the release, we've been working on:
Completing key notary support
Fixing state_default for power levels in gomatrixserverlib
Resolving some issues around rejecting invites, particularly when the remote server is not available
Reducing the cost of checking if the local server is in a given room
Since our last update, our Sytest compliance numbers have been on the rise again, taking us ever closer to our goals:
The big news of the week is the release of Synapse 1.38, which converts several integer columns to bigint, allowing Synapse to process more than 2 billion (231) events. Which, incidentally, matrix.org did last week 📈:
But that's not the only thing new in Synapse 1.38. We also landed the ability to set an expiry time on cache entries, allowing you to reclaim memory from infrequently accessed caches. Configuring this to "1h" on matrix.org has already yielded a noticeable reduction in overall memory use 📉:
We'd encourage you to read the full announcement for more — See you next week 👋!
🔗Registration management for Token Authenticated Registration
Another short update about my GSoC project, this time about the Synapse admin API for managing tokens. https://calcuode.com/matrix-gsoc/2021-07-16_admin-api.html
A bridge for LINE Messenger based on running LINE's Chrome extension in Puppeteer.
Docker is now supported, via Dockerfiles that actually work now! But for the time being, Docker images must be built manually, as I am yet to deploy a Docker registry for prebuilt images.
Also, sample systemd service unit configuration files are now available, courtesy of @lecris:lecris.me 🙂
What I'm working on next is a bot command to list all of your LINE contacts & groups (similar to mautrix-whatsapp's list <contacts|groups>), and the ability to sync a LINE DM by inviting a contact to a Matrix DM. This will allow messaging LINE contacts that the bridge didn't yet create a portal for.
And please, feel free to try out this bridge! It should be serviceable for day-to-day usage now. I still won't be able to host a public instance of it for a while, though (as it is fairly hefty due to having to run Chrome), so you'll have to self-host if you want to try it.
This week we fixed bugs. Switching room should now be quite a bit faster again and once Qt 6.3 is released with some important bugfixes, scrolling in Nheko should be super smooth. (We can't enable that flag yet, because of 3 bugs in the item pooling code in Qt.)
Other bugfixes:
Inline images sometimes wouldn't show, but now inline emotes and images should render once they are loaded! (This took me a year to figure out)
You can now send edits in encryted rooms again, if they are a reply to an event.
No more reply fallback in the room list.
At some point timed out verification requests started showing up on startup. That regression is fixed now.
Fix rooms not showing up after login because we were off by one.
Fix some cases where the loading spinner wouldn't stop animating and as such consistently use CPU, when a room is open.
Cache db transactions to reduce allocations and memory zeroing when loading a room or scrolling.
Fix some edge cases in the blurhash decoding, that could lead to brownish image previews.
Fix accepting an invite not placing you in the joined room.
We’ve been shepherding through MSCs to improve private spaces, namely MSC3083 (Restricting room membership based on space membership)
Meanwhile, we’re also implementing outstanding polish, planning steps for Spaces to exit beta
Web
v1.7.33-rc.1 now up for testing on https://staging.element.io/ with support for blurhash, draggable picture-in-picture view for calls
Contributing to element web? There are new labels and magic keywords for pull requests to show better information in the changelogs. See the contributor guide for more detail.
Do you use the master branch of element web or any of the web projects? Please let us know - it may go away soon.
Android
Still polishing the voice message feature: add support for Android 5, improve timeline rendering, improve animation in the composer, support for RTL language
First news about Python bindings for libQuotient, C++ Qt-based Matrix SDK
After a month of hard work, we(I as GSoC student and kitsune as mentor) have bindings with tests for almost all core classes and also initial version of the demo client, in which you can log in with a password or SSO, the server you enter is resolved automatically and also after successful login data is being synchronized. There is also a possibility to log out. So this part is on the same level as in Quotient.
Over the first weeks of July, Teckids e.V. held their annual summer camp for kids between 9 and 15 years. This year, after we started introducing Matrix and Element as a chat platform from September 2020 onwards, the camp was heavily relying on the platform for various parts.
Before the camp started, we invited all 90 participants to a chat room to get together, share a bit about what they expect, already did with coding and technical stuff, talk about what food they'd like to have for the barbecue, and stuff like that. Many of them engaged in the discussion, and started exploring Element (before you ask, yes, we hat a lot of snow and party poppers 😛!). Some got really excited that they could even change or add features to Element, or ask for such changes – we had to promise to hold a session where we find out how to add one new animation to Element. Unfortunately, Element is developed on GitHub, so the potential young contributors are locked out by the exclusive Terms of Use there. We are trying to reach out to Element HQ to find a solution.
During the camp, verifying crypto sessions using emojis again made for a good party game to get to know each other (like, find the kid a nickname belongs to on the camp site, start verification, and compare emojis – a lot of fun that we, again, did not even have to start, because someone always finds out about it and asks what it is about).
Now that everyone got to know Matrix for chatting, in one of our workshops, the participants discovered that not only people, but also devices can send messages, and react to replies – in that workshop, the kids built a chat-ops IoT door beel (for their tent on site, or room at home). They soldered a circuit board to fit an ESP (MicroPython) micro controller on, and coded a small program (using templates with differing complexity levels), defining what the door bell should send when a button is pressed, on what messages to react, and the like. We produced a fun video about the project (German audio, English subtitles): https://eduvid.org/videos/watch/20a50c25-ecb4-48c0-9b13-de2548f290d4?subtitle=en . The (minimal and somewhat buggy) MicroPython client library is published as µtrix.
Now, sadly, the event is over, and we slowly see (as expected, only a part of the) participants moving over to our long-term project chatrooms; we will start clearing the virtual camp site chatroom during the weekend to make room for a new group.
Asked about the regularity of these events, Nik replied:
We are still experimenting with our new camp formats. As bad as it all is, COVID caused a lot of innovation here because we were forced to leave the known roads we normally travelled, and now we are starting to integrate all that new stuff (like really embracing Matrix) into outdoor and presence events. I think we are on a really good way with it, and surely I will keep posting updates that might be of interest for the greater community.
Reidel Law Firm remains committed to providing top notch legal services in Franchise Law, International Trade Law, and Business Law while maintaining accessibility, (one of our Firm’s core values) to our clients, colleagues, and friends of the firm. Utilizing our own secured chat platform allows us to be in communication with our clients around the world while maintaining the utmost in data security and client privacy.
I believe in Matrix+Element and encourage my clients and law firms to get ahead of the curve or get left behind and become irrelevant. Email has to die, its just the worst. Also, I forgot to note earlier that the guide is copyright free. I hope other business owners will use and revise it for their own uses.
Hi everyone! Did you ever feel lost in the Matrix world? The room directory is big, but it's still hard to find something you like. Or are you a room moderator, but there is not much activity in your room because it doesn't have enough users?
This is why I want to share rooms (or spaces) I find interesting.
"The pinnacle of motorsport! We're in an exciting time in Formula 1 with a close championship and exciting battles. Come hang out and chat about the upcoming race in Silverstone on July 18th!"
NOTE: We released Synapse 1.38.1 on Thursday, July 22nd. It mitigates a client bug with Synapse 1.38.0's smaller sync responses which prevented new Element Android sessions from successfully participating in encrypted conversations. Server administrators are strongly encouraged to upgrade.
Synapse's database schema used integer columns in a few places where values could potentially overflow a maximum value of 231. One such column is events.stream_ordering, which surpassed 231 on matrix.org last week.
To prevent overflows, Synapse 1.38 will automatically convert several integer columns to bigint as a background update. While homeservers will function normally during this task, it could result in increased disk I/O for several hours or days. Note that homeservers may need several gigabytes of free space to successfully rebuild associated database indexes and complete the upgrade.
Synapse has a new configuration option, caches.expiry_time, which can be set to enable evicting items from caches if they go too long without being accessed. This helps servers reclaim memory used by large yet infrequently used caches.
The response to /sync now omits optional keys when they would otherwise be empty. This can significantly reduce the size of incremental syncs, as demonstrated in #6579. Thanks to deepbluev7 for initially submitting this in #9919, which made it into this release via #10214.
This release includes an experimental implementation of MSC2918: Refresh tokens, which adds initial support for complementary access / refresh tokens in line with OAuth best practices (#9450).
Synapse now ships a script to review recently registered accounts, which can be useful in cleaning up servers in the wake of malicious, automated registrations like we witnessed during last month's spam attack.
We've also fixed a few rough edges (#10263, #10303, #10336) in the spam mitigations from 1.37.1, and would encourage you to update.
These are just the highlights; please see the Upgrade Notes and Release Notes for a complete list of changes in this release.
Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including deepbluev7, dklimpel, fkr, and sideshowbarker
After quite a significant gap between releases — version 0.3.11 was released at the beginning of March — we're happy to finally announce the release of Dendrite 0.4.0 today!
The full changelog for the release is available on GitHub, but we wanted to take the opportunity to talk a little about some of the changes that have gone into this release.
Recently our release cadence for Dendrite has slowed as we have spent more time within the team working on Pinecone and Low Bandwidth Matrix. These are major areas of research for us which we hope will unlock a number of new opportunities within the Matrix ecosystem, allowing us to build on Matrix anywhere and to reduce the protocol-level footprint. However, Dendrite has not been forgotten amidst the excitement and we will be spending more time working on Dendrite again in the coming months.
One of the major features in v0.4.0 is that we've introduced newly-refactored state storage in the roomserver database. The goal here is to make state storage significantly more efficient by ensuring that we deduplicate state blocks and snapshots wherever we can. By ensuring that all state blocks and snapshots are ordered strictly, and by enforcing uniqueness constraints on the hashes of the blocks/snapshots, we've been able to achieve this.
This was largely spurred on by watching dendrite.matrix.org consuming a rather alarming amount of disk space on a daily basis. In this particular instance, moving to the new state storage resulted in a 15x improvement on disk utilisation for state blocks and a further 2x improvement for state snapshot references immediately after the migration, and the growth rate of the database has slowed substantially since.
Ensuring that we don't waste disk space is one of the most important factors in ensuring that Dendrite operates well at any scale — future datacentre deployments supporting many users will find storage overheads decreased and small/embedded single-user deployments (such as P2P, on mobile devices or in the browser) will fit much more effectively onto resource-constrained targets.
After upgrading to v0.4.0, Dendrite will run an automatic migration to update your homeserver to the new state storage format. This might take a while on larger databases so please expect some downtime.
We've continued to squeeze further optimisations into the federation and state resolution code, aiming to reduce the amount of CPU burn and memory utilisation. Some of the feedback that we receive most often from those that have been experimenting with the Dendrite betas is around the sudden spikes in resource usage, especially when joined to large rooms.
The bulk of this resource usage comes either from attempting to reconcile missing events or running state resolution in rooms with lots of members, as potentially large state sets of events need to be brought into memory in order to do so. We've introduced some transaction-level caches for dealing with missing auth/prev events to reduce the memory pressure and we've also tweaked the caching around around /get_missing_events to ensure we don't duplicate any state events in memory.
Resource spikes aren't completely eliminated but this should smooth out CPU and memory utilisation significantly. In the case of dendrite.matrix.org, which is joined to some 6500 rooms at present, memory utilisation of the Dendrite process typically sits around 1.5GB at present.
State Resolution v2 has also seen further optimisations in the power-level checking, which should reduce CPU usage even more.
Thanks to Half-Shot's perseverance and contributions, we've merged a couple PRs and worked on some further fixes for getting Application Services working correctly in Dendrite. Whilst not entirely feature-complete and with a number of features still to go, enough support is now present to support basic bridging functionality.
We've done quite a bit of preliminary testing with matrix-appservice-irc and have also heard a number of success stories from the community with mautrix-whatsapp and mautrix-telegram. Others may work too — let us know what you find!
A number of bugs in various places (including the roomserver, federation API and media API) which could cause Dendrite to crash have also been fixed. Some of these have been contributed by the community in pull requests, so we extend our thanks to anyone who has submitted a fix to the project.
A special mention also goes to Jakob Varmose Bentzen for reporting a security issue to us around the legacy /v1/register endpoint, where a flaw in the legacy shared secret registration allowed malicious users to create accounts. We've since removed this legacy endpoint and the vulnerability is now fixed.
There are still a number of missing user-facing features which we will be working on over the coming months, as well as some architectural issues that we will look to address.
A notable area of work involves attempting to remove the dependency on Kafka for polylith deployments. Kafka is very resource-heavy in operation and somewhat limits us to the types of interactions that we can perform between components. It's also very difficult to manage retention correctly, in the interests of not endlessly consuming disk space here either.
As usual, Dendrite is still considered beta so you may not want to rely on it for production systems, although it should be stable enough to experiment with. If you find any bugs or anything that doesn't look right, please let us know:
I'm the Principal Data Scientist for the Ansible Community. We're hoping to switch to Matrix as our primary platform in the near future, and I've just written up my thoughts on why that's a good idea, what the consequences might be, and where we go from here. Find it at https://ansible.github.io/community/posts/matrix_and_ansible.html
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://spec.matrix.org/unstable/proposals.
A concrete plan has been drafted for publishing the new spec release, and is currently undergoing execution. This release will include many changes that have built up since the last release (back before the new spec redesign even), as well as the new Matrix Global Version Number scheme. Look forward to it dropping soon!
Otherwise Bruno has been hard at work continuing to push forward the various aggregation MSCs (1234). MSC3083 (restricted room memberships) is being updated as part of finalising the new Spaces feature as well as MSC2716 (history import).
As well as lots of new MSCs as listed above. Busy times!
The last two weeks I worked on a few very big optimizations. We also almost finished sqlite support for Conduit, which is slower than sled in benchmarks, but has much better RAM usage characteristics.
Batch up and cache /sync responses for when clients time out
LRU cache for deserialized PDUs
More efficient state res by only fetching events it needs
Big(int) news! This week Matrix.org processed its 2^31st event, exceeding the range of a PostgreSQL integer column for the first time. This caused a bit of a scramble in the aftermath of last week's spam attack, as we had a few integer columns in our schema which we needed to convert to bigint. Fortunately, we were able to complete the change sufficiently in advance (#8255), and also took the opportunity to audit other columns and sequences in the database which could conceivably overflow. Synapse 1.38, due out next week, will automatically migrate homeservers when they upgrade. We run the migration as a background task, so homeservers should continue functioning as normal throughout, though they may use a bit more disk and memory, especially when rebuilding indexes for the new bigint column.
We're also starting to hone in on our team's goals for this quarter, and it's looking like our primary focus will be on improving room join speeds. Wish us luck!
Lastly, we're overjoyed to announce that @reivilibre, a former intern on the backend team, joined Element this week! We can't wait to see where he helps us take Synapse!
matrix-puppeteer-line: A bridge for LINE Messenger based on running LINE's Chrome extension in Puppeteer.
This week was spent on adding proper support for LINE user joins/leaves (though invites/kicks are still a TODO), bug fixes, and ease of deployment. Docker and systemd setups will be ready shortly.
And this bridge should soon be listed on https://matrix.org/bridges/, if it isn't already 🙂 Thanks madlittlemods (Eric Eastwood) for accepting the PR!
SchildiChat is a fork of Element that focuses on UI changes such as message bubbles and a unified chat list for both direct messages and groups, which is a more familiar approach to users of other popular instant messengers.
There are two announcements that we can share with you this week:
SchildiChat for Android is back in the Google Play Store! Users who have previously installed the release using our own F-Droid repo will be able to update without the need to re-install. All previous ways to install the app will remain available as well.
You can now help us translate SchildiChat using Weblate! Note that this only contains SchildiChat-specific translations, we continue to use Element's translations where possible.
Apart from that, we have mainly been focusing on smaller improvements and fixes, while staying up-to-date with new Element releases.
For more information about SchildiChat, feel free to visit our website or check out our source code!
Also, feel free to join our Matrix rooms, which you can find in the new SchildiChat space: #schildichat:matrix.org
Spaces work is making progress. Some rooms can now be previewed. To improve that situation, I wrote an MSC to preview specific rooms. Alternatively we will try to get the previews for the few rooms you aren't joined to from the space summary API, currently we are just fetching the existing state. You can also now join previewed rooms and the design of joining invites was adapted to match it.
red_sky☄️ went through the pain of fixing the Windows builds after we changed our http backend last week. So if you want to try it out, you can test it on Windows. We also replaced the old, boring spinner with an animated Nheko logo. If you see that a lot and think it is Nheko's fault, don't hesitate to open an issue! But in most cases it will probably be your server. Sadly no screenshot of how the spinner looks like, my server is too fast and taking a proper screenshot is too much effort because of that. ;p
We also fixed an issue with updating device lists in the develop version of Nheko. If you were using the nightlies, now is a good time to update! In more E2EE news, symmetric megolm backup fixes the issues I had with the online key backup, so looking forward to implementing that.
The latest release of opsdroid is out with various fixes which can be seen in the changelog. The main point to note for matrix users is that older versions of matrix-nio (the matrix client library used by opsdroid) did not support the synapse change to omit optional fields from sync. Therefore if you are using our docker images you will need to update to 0.23 to get a container with the newest matrix-nio included.
The other change which is relevant to matrix users is that Oleg has added support for version 2 of the Rasa NLU framework, so you can once again do open source, self hosted natural language bots.
Thanks to Gwmngilfen I touched RStudio and toyed a little with some data as well.
You can find some graphs over at https://github.com/MTRNord/server_stats_r_statistics/blob/main/scripts/rooms_members.md
For the first graph the credit fully goes to Gwmngilfen :)
The second one is in log scale for both axis but essentially the same :)
This is obviously currently very spare but I hope to add more statistics when I understand R lang :) This is in fact my first time doing something with R so my skillset is limited :)
Run Matrix Python bots inside of Docker Containers with Simple-Matrix-Bot-Lib and Docker!
This is a guide for isolating and running your Matrix bot within a Docker container. It is also applicable to bots written using other libraries and languages.
Hi everyone! Did you ever feel lost in the Matrix world? The room directory is big, but it's still hard to find something you like. Or are you a room moderator, but there is not much activity in your room because it doesn't have enough users?
This is why I want to share rooms (or spaces) I find interesting.
In meta twim news, the twim updates bot (which posts in #twim_updates:cadair.com) has been upgraded to opsdroid 0.23 and now correctly keeps the formatted body when an event is edited.
This is a long read, but a detailed argument. Also please note that this is was produced for the purpose of an MSc course of study, it was not commissioned for any other purpose, as the preamble makes clear.
Next, a high billing for Beeper this week, who have been working hard and getting product out!
It's been a month since our last update. A lot of the work since then has been on making everything more reliable, but we've also added new features to our clients and started making some new bridges.
Desktop
Added thread UI for Slack-bridged rooms. Internally they're just replies like before, but the client will intelligently collapse replies in Slack rooms into threads.
Merged upstream Element additions like voice messages.
Also merged some features that aren't in Element yet, like Giphy integration.
iOS
Released Beeper iOS to Testflight.
Android
Added grouping rooms by chat network based on the m.bridge state event. The UI is similar to spaces, but they're not actual Matrix spaces (yet).
Added support for Android 11's "conversation" notifications.
Bridges
Android Messages is turning out difficult to reverse-engineer to a sufficiently reliable level, so we're building a new SMS bridge into our Android app. It'll also be available as a standalone open-source app, which already exists at https://gitlab.com/beeper/android-sms (but doesn't have any setup instructions yet).
We've funded development of a LinkedIn bridge. sumner will post a more detailed update about that.
We're hiring React, iOS, Android and SRE/Devops engineers. If you're interested, check out https://angel.co/company/beeperhq or DM Eric Migicovsky.
Bruno has been working on aggregations as part of his work for Hydrogen. He reported:
I've been cleaning up the relations MSCs, finding a balance between documenting the current state and not losing track of community concerns. I've started with MSC 2674 which is the very basic format of relations, and will move on to annotations/reactions (MSC 2677) next week.
We're pleased to announce the release of Synapse 1.37.1 this week, which includes mitigations for the recent distributed spam attack across the public Matrix network. We advise upgrading as soon as possible.
Got another week of Helm Chart updates, with the Synapse chart getting a bunch of worker improvements and additional configurability, as well as being updated to first 1.37.0 and then 1.37.1
I'm excited to announce that I started working on a new bridge for bringing LinkedIn messages to Matrix! It's currently in the early stages of development and not production-ready. The current feature set includes: backfill from LinkedIn, user name and profile picture sync, message sending from Matrix -> LinkedIn, and real-time message puppetting from LinkedIn -> Matrix. There's much more to come, and you can join #linkedin-matrix:nevarro.space for updates. Development is being funded by Beeper, and is being designed with integration into Beeper as it's primary goal. However, the bridge is open source (Apache 2.0) and will be available to self-host. The source code is here: https://github.com/sumnerevans/linkedin-matrix.
Great work from Sumner! Glad to see people have the option to bridge their LinkedIn messages!
matrix-puppeteer-line: A bridge for LINE Messenger based on running LINE's Chrome extension in Puppeteer.
Better LINE->Matrix read receipt bridging is now supported in the testing branch! The bridge now checks all LINE chats (not just the most recently-used one) to see if messages you sent have been read (in LINE). This works by cycling through all LINE chats where the final message is posted by you and doesn't have a "Read" marker on it yet (or for multi-user chats, if your last message hasn't been read by everyone in the room).
With that, I'll consider the bridge to be in Early Beta! 🎉 I'm now testing the bridge for myself to iron out a few kinks, and am preparing a PR to the matrix.org webpage to have this listed on https://matrix.org/bridges/.
This addon for the WebThings gateway lets you send Matrix messages when your IoT fridge is empty – or whatever you have connected to your gateway.
The update fixes predefined messages getting sent to the default room and is the first to be tested against gateway version 1.0.0.
https://gitlab.com/webthings/matrix-adapter or in the addon list of your WebThings gateway
This week, NeoChat gained support for a Global Menu on Plasma and macOS. Aside from that, we fixed a few crashes.
But the biggest news of the week is that we will get funding from NLNet to implement E2EE support in Quotient and NeoChat as part of their grants to improve the internet. We will report on our progress on that front here!
This is terrific news, big thanks to NLNet for making this choice!
We merged the Spaces branch, which means Nheko master can now show some spaces. Peeking unjoined rooms, nesting spaces and creating them should be coming soon. We are also looking into how to fit knocking into the UI (we already rendered incoming knocks in the timeline for a while).
You can also now edit still pending messages, which should help if your server is slow and you notice a typo. The edit will then get queued and be sent as soon as the server acknowledges they received the original message. Apart from that there have been some improvements to the readability of the room list and some other UI elements.
Last but not least, we switched out our entire http backend from Boost to Curl. For that I wrote a simple wrapper around Curl. This fixes about 10 issues around connection shutdown, brings proxy support, http/2 and http/3 support and in general makes Nheko crash less and reduces latency a LOT! This will obviously cause some pain for packagers, but I hope it isn't too bad. Some of the issues this fixes only had 2 digits in our bugtracker and one was even filed by benpa!
Chris tweaked the UI in various places. It’s a lot of small details that together make for a smoother experience. I encourage you to read the details in the description of !782. This is the only MR that landed since last week, but our people have been hard at work nonetheless. Kai blogged about his journey working on the search bar of doom and Alejandro shared his own struggle. In the meantime, Julian’s work has mostly happened upstream in matrix-rust-sdk.
Research: We’ve been reaching out to people to walk us through how they use Spaces now and what they’d like to see different to help us learn and iterate;
Restricted room access: Some good progress towards shipping improved team spaces
Some major progress on conversion to TypeScript, finding some bugs along the way. The main source of the element-desktop project is now fully converted to TypeScript!
A styled player component for the audio messages feature, available in the labs section.
v0.5.19 of the matrix-bot-sdk is out now with fixed power level checking (with an added utility function), improved default error logging, and a typo fix in reply creation. Check it out, and visit #matrix-bot-sdk:t2bot.io for help & support.
Exciting new update, we can now wireguard an on-premises server from just about anywhere and make it work with the AWX system. This is useful when your server doesn't have a static or public IP address, or when some other networking issue prevents you from running a Matrix service on it.
Follow of on GitLab: https://gitlab.com/GoMatrixHosting
* Add '00 - Create Wireguard Server' template for AWX admin to provision Wireguard servers that on-premises servers can use to connect.
* Subscription involved can view an additional '0 - {{ subscription_id }} - Provision Wireguard Server' template.
* Add /docs/Setup_Wireguard_Server.md guide.
* Add onboarding script for Windows 10 users.
* Raise maximum download size to 200MB.
Mjolnir is a moderation bot for communities on Matrix. It helps with a lot of the actions covered by the moderation guide, including capabilities to apply bans from other trusted communities. It's still a bit terse in its documentation, but if you're looking for a featureful moderation bot then it's worth a go.
In related news, Mjolnir v0.1.18 is out with a couple quality of life fixes - if you've been bothered by the log spam, it's now fixed :)
I started a tiny repo to collect various limits and related factoids about the Matrix specification and implementations. I hope that distilling and summarising such things at glance will make it easier to see what is and is not possible.
If you know of more that should be listed, please contribute! 🙂
🔗Self hosting your own Matrix server on a Raspberry Pi
@ed:selfhostingblog.com of theselfhostingblog.com has written a guide on getting started with Synapse on a Raspberry Pi using Docker Compose. You can read it here.
Together with J. Ryan Stinnett, I created the 🔖 #matrix-science-reading-group:dsn.tm.kit.edu for exchange of and on scientific papers, books and related resources on all things Matrix: Topics ranging from peer-to-peer broadcast overlay networks over conflict-free replicated data types to end-to-end encryption. Investigating security, performance, deployability, or whatever else is interesting, by methods from observation over simulation to formal verification. 🎓️ Please join if you want to read about papers that might not be Matrix-related enough to make it into TWIM, or want to engage in the discussion. 😊 The resulting papers are collected at: https://github.com/jryans/awesome-matrix#research
Hi everyone! Did you ever feel lost in the Matrix world? The room directory is big, but it's still hard to find something you like. Or are you a room moderator, but there is not much activity in your room because it doesn't have enough users?
This is why I want to share rooms (or spaces) I find interesting.
"A public space for Free/Open Source Software maintainers to swap notes and discuss their craft. Inspired by https://github.com/github/maintainerweek, all maintainers welcome!"
