Matrix Live

We continue the new format for Matrix Live season 3 by chatting with Francois from New Vector, to get a sneak preview of the work he's doing on a new version of matrix-android-sdk, and how that will impact Riot:

Dendrite and Brendan

Welcome to Brendan, who has started working on Dendrite as his new day job at New Vector:

There's been some progress on Dendrite (?), with a couple of bug getting fixed along with some progress in the implementation of Matrix endpoints, such as the /backfill federation one (documented here), which has already been merged, and the /messages one from the client-server API (documented here) for which a pull request has been opened and is currently under review.

Python Matrix Client async API

Cadair:

I have pushed my asyncio wrapper of the Python matrix client api class to pypi, so I can use it in a few different projects while the PR to the Python SDK is in an unmerged state. It exposes all the methods on the MatrixHttpApi class as awaitables. https://github.com/Cadair/matrix_api_async and https://pypi.org/project/matrix-api-async/

These improvements will, in time, be merged back into the main Python SDK.

Discord bridge 0.3.0 released

Half-Shot is working on so many bridges that these days he just casually mentions a huge release like Discord bridge v0.3.0. Some of the many many new features:

• #251 Support for Postgresql and a newer SQLite3 backend!
• #182 Replace npmlog with winston, for logging to files and better logging overall.
• #221 Add support for m.sticker.
• #210 Discord-side moderation of matrix users.
• #259 Show Matrix replies as Discord embeds.
• #164 Bot will now mention name, topic and membership changes on Discord.
• #175 Add special discord keys onto m.room.member for ghosts
• #179 Add rooms to the directory when bridged.
• #220 Messages are now deleted by a users puppet rather than the bot.

Shoutout to our new member of the team, Sorunome who did a lot of the review work behind the scenes for this release. Also, thank you to everyone who submitted a PR or an issue!

matrix-puppet-slack v1.8.0 released

tom came out of the wild to tell us about matrix-puppet-slack:

TWIM (but actually over the last few weeks), thanks to the efforts of a new contributor, matrix-puppet-slack v1.7.0 and v1.8.0 have been released, fixing a number of old and new issues and adding support for new types of Slack events, including bidirectional @-mentions, Slack-to-Matrix typing notifications, fixing Slack-to-Matrix image/file uploads with comments, and more! The matrix-puppet-bridge projects have gotten relatively quiet over the past year or so, but there's still plenty of bridges with plenty of features to implement and plenty of bugs to fix (and, even more importantly, bugs to report!) for any would-be-contributors who'd like to use a trusted Matrix homeserver as their double-puppetting Slack/iMessage/Facebook/GroupMe/iChat/Skype/Hangouts/Signal/Tox client from which to brag to their friends on the other platforms about how great Matrix is.

The room for collaborating on matrix-puppet-bridge bridges is: #matrix-puppet-bridge:matrix.org

maubot management UI progress

tulir has been making big progress on his maubot management UI:

The maubot management UI has progressed well, but isn't quite ready yet. I think it should be usable by Next Week in Matrix. It can be used to set up and configure a maubot instance and plugins. Once it's ready, it should be possible to do everything except installing maubot itself through the UI: installing and updating plugins, adding matrix clients, configuring plugin instances, viewing logs, etc.

HK Bot is an anti-abuse bot

Coffee continues his streak of TWIM mentions by bringing a new bot to combat abuse:

HK Bot is an anti-abuse bot for public Matrix chatrooms. This is a bot that really shouldn't exist, but since some people just like to make others' lives more difficult, here we are. The purpose of this bot is twofold. It can automatically oversee rooms and respond to abuse, based on programmable pattern rules, providing a stop-gap measure in case no human moderators are nearby. It can also automate some tedious tasks via its command interface, the primary one being the complete redaction of all of an abuser's messages. HK Bot is still under construction and contains dangerous features. Use with caution.

X-SAMPA to IPA bot

Dandellion "made an X-SAMPA to IPA bot based on matrix-bot-sdk and discord's conniebot". This bot essentially lets you use ASCII characters to get an output in IPA. The advantage of this is that you can much more easily type and transmit pronunciation (because you don't need to find the characters).

I'll just have to make it a bit more configurable, but then I'll throw the source up When making constructed languages and talking about linguistics, it's nice to get an easily readable IPA representation of a word, but it's really hard to write IPA, which is why x-sampa exists as a way to input IPA with a normal keyboard!

Informo

vabd, the unknown organiser of Informo provided an update about the spec:

A handful of SCS (Specs Changes Submissions) to the Informo specifications have been happening over the last week and a half, with some of them still open to public review for at least a few days before being merged into the specs. The list of SCS open to review can be found here, and people can track new SCS and status changes to existing ones through our specs bot that's living in #discuss:weu.informo.network ?

Fractal

Julian Sparber has been working on message rendering in Fractal:

I increased message rendering performance in the room history. Also the history doesn't move it's position anymore when older messages are loaded, which results in a much better experience. All the changes are in master, but we didn't make a new release.

Video intro to Matrix and Riot in French

Nouts:

I have made a video about Riot which could fit in the Guides on matrix.org. It's for beginners, in French. It's 6 months old, but I just uploaded it on youtube. Also it's available on peertube.

More Things This Week

ma1uta introduces a new client built on JavaFX: no code yet, but it's built on the previous Java Matrix work he's been doing. "It will be cross-platform (linux,windows,osx I hope) client with supports of the multi accounts."

Half-Shot also continued work on bridging via libpurple: "I've nearly got group chats working, with invites (and hopefully people joining and leaving them showing up properly) as well as user's having profiles."

Work continues on Koma:

Since the last TWIM update, koma is updated to kotlin 1.3, experimental coroutines are replaced with stable ones. The changes are being tested and should be merged soon. A new contributor has joined, so expect development to speed up a bit.

Spectral, a Matrix client built using libqmatrixclient, runs great on macOS. I'm excited because I get to play with another client! Find .dmgs on bintray.

FIN

You made it, right to the end! Nice going! Come back here next week to find out what's been happening, or even better, come join us in #twim:matrix.org and tell us what YOU'VE been working on!

The Matrix.org Foundation

This week, the first steps were taken in the creation of the Matrix Foundation! Read our blog post from earlier this week for more:

in preparation for the upcoming Matrix 1.0 release, we've been moving ahead with the rest of the open governance plan – and we're happy to announce that as of a few hours ago, the initial incarnation of The Matrix.org Foundation exists!
Watch this space over the coming weeks as we announce the Guardians and finish bootstrapping the Foundation into its final long-term form! Meanwhile, any questions: come ask in #matrix-spec-process:matrix.org

Hello Matrix

Alex has managed the invaluable https://www.hello-matrix.net/public_servers.php for some years, but in two weeks time will be making some changes to the page:

Just wanted to let everyone know that changes are coming to the server list: I've put up a notice on the site that starting Nov 16th I will only show a curated list of servers I would recommend to join.
This reduces the workload for me quite a bit and avoids me becoming some kind of arbiter on what encompasses the Matrix universe… I think it is also more useful for users who are looking for a server to join.
And there is always matrixstats.org for those who are looking for a more complete-ish list of known homeservers.

However, if you have ideas on how to continue the project, or would like to step up and get involved in maintaining a list using data and tools from Hello Matrix, please contact me. Alex told me:

if you find someone willing to take up the project of a more automated, self-service and complete list at a later date, I am more than willing to hand over all the stuff I currently have and might also lend a helping hand myself (if I have time then).

maubot management API

tulir has continued work on the revamped, now-Pythonic maubot, and has added a management API:

The maubot management API I mentioned last week is now mostly ready. It should be possible to use it to set up a maubot instance and plugins without filling the database manually. There's no UI yet though, so it still means curling manually.
The management API also supports the fancy plugin reloading stuff which was the > reason I rewrote maubot in python: You can POST an updated plugin to the API and it'll install it without having to restart.
I also made a bunch of plugins while working on the API that I used to test the API: a dice rolling/calculator bot, a bot that replies with the MXC URI of images you send it, a simple echo/ping bot and an xkcd bot

Next steps are making the management UI, a few more plugins and making setup and development instructions so that other people could run it and make plugins

Announcing the Fractal Hackfest in Seville

Tobias Bernard officially announced the second ever Fractal hackfest:

there's going to be a Fractal hackfest in Seville in December https://blogs.gnome.org/tbernard/2018/10/26/fractal-hackfest-in-seville

matrix-client-core

Coffee continues his streak of weekly updates by introducing us to matrix-client-core:

My lightweight bot/client framework, matrix-client-core, received its first tag ever. Version 0.0.1 is relatively stable, and lies on the doorstep of some refactoring work (ongoing) which should keep the master branch backwards compatible for now, but could make things less stable as I add new commits.

It turns out this isn't strictly a new project:

[it has] been there all along, quietly powering FAQBot and all of my bots. ? Maybe I have failed to explicitly indicate it as such up until now. (oops)

Always good news to see more bot-creation tooling!

Synapse 0.33.8

Synapse 0.33.8 was released, and andrewsh made sure it was updated on Debian. Docker packages are also updated.

mxisd v1.2.0-rc.1

Max has updates on mxisd, Identity Server for Matrix:

mxisd v1.2.0-rc.1 is out with support of all features for the Exec Identity Store, allowing connectivity to totally custom/arbitrary backends. Feedback is extremely welcome!

Half-Shot interfaces with libpurple

It's a job that someone needed to do, and that someone was Half-Shot (who else?)

I've been reviving node-purple (a library for communicating with libpurple) and making a brand new bridge service to make use of it called matrix-appservice-purple. Today I got it to the point where you could link your XMPP account to your matrix user and have it bridge PMs over.
Work is ongoing to make it bridge group chats, profiles, contacts lists and support other protocols better in the coming weeks

NSFW image detection API (on Matrix)

Black Hat is often found working on Spectral (previously 'Matrique'.) This week, he has been building @nsfw:encom.eu.org, which is a bot designed to give scores for how likely an image should be classified as NSFW. It's a simple mechanism, you give it an image, it gives you a JSON object with the result. For example:

My avatar returned less than 1% probability of being NSFW, which I was actually a little offended by.

To talk more about the bot and it's development, chat in #nsfw:encom.eu.org.

Gaming Discord bridging

swedneck has been dutifully bridging channels from the Linux Gaming discord.

All but 3 channels on the linux gaming discord have now been bridged to matrix.

To find these channels, join the community at +linux-gaming:matrix.org and get chatting.

Other thoughts

• kitsune is forced by his employers to use Viber, so is thinking of creating a Matrix bridge.
• I spent some time this week repeatedly installing Synapse, then working with Stefan to create a new, hopefully definitive installation guide (available soon). I can also personally recommend Slavi's matrix-docker-ansible-deploy project, this is a great way to get a Synapse installation (and more!) running.
• lately in #twim:matrix.org posters have been providing much clearer and atomic updates, which I like a lot

Matrix Live, season 3 episode 2

We continue the new season of Matrix Live. This re-booted season has a slightly different format to previous: in each outing, there will be a single deep-dive topic. This week, Matthew and recent-Matrix-arrival Nad discuss UX for E2E encryption key handling. This is an unsurprisingly complex design question, both in terms of how it should behave and how it should look. Nad shared his latest thinking in a blog post earlier today, and you can watch the video below.

Farewell

Enjoy your week. We'll see you back here next week, but if you are working on something using Matrix, come chat with us in #twim:matrix.org!

It's been a long-standing goal to enable end-to-end encryption by default for private communication in Matrix. The technical effort so far has included our libolm library, an independent cryptographic review and a massive backlog of feature development and bug fixes. Today, instead I'd like to focus on some of the User Experience challenges and goals we're facing.

I should also introduce myself—I'm Nad Chishtie ( @nadonomy:matrix.org) and I recently joined the Matrix core team (at New Vector) as Lead Designer, most recently focusing on end-to-end encryption.

When using encrypted messages, most existing services fall short in one or all of the following:

• They don't allow you to use multiple devices independently. For example, a web session might be locally tethered to a mobile device.
• They don't support a way to restore or temporarily access message history. For example, if you don't have physical access to your main device because it's broken or has been stolen.
• They don't allow you to verify that devices are controlled by their owners rather than eavesdroppers, and persist that trust across multiple devices, sessions or rooms.

(For some users, restrictions such as limiting the number of places encryption keys reside, and not having a synchronised message history may be desirable security features. We'll support these cases, but just not as the default behaviour.)

Let's dive in to some of the fundamental concepts we'll be putting forward to deliver a default end-to-end encryption experience that makes sense for most modern users. In this post we'll look at an overview of work-in-progress wireframes, in the spirit of designing in the open and gathering feedback from the wider Matrix community. Please note that these don't represent the actual interface design.

Cross-signing personal devices

When logging in to a new device, you'll be able to use an existing device to verify your new one. Verification is done by scanning a QR code on whichever device has the most convenient camera to use, or by comparing a short text string. You only have to complete this process once to mutually verify both devices.

Verifying your new device by cross-signing transfers encryption keys, giving it access to your encrypted messages, and also signals to other users that the new device is trustworthy.

Secure Message Recovery

To the end user, Secure Message Recovery works a lot like setting up disk encryption or a password manager. A user can optionally secure their message history using a recovery passphrase and/or key. If logged out, or using another device, the user can use the recovery passphrase or key to access their encrypted message history.

In practise, this incrementally encrypts and backs up encryption keys to a user's homeserver, kept secure by the homeserver never having access to the passphrase or key. Like cross-signing, using a recovery passphrase or key will also signal to other users that a device is trustworthy.

We think that in most cases users will cross-sign personal devices, but as a safety net (for example, if a user's devices are broken or lost) Secure Message Recovery is an invaluable tool for users to minimise the chance of them losing their encrypted message history.

People should trust people

With both cross-signing and Secure Message Recovery in place, we think that people should trust people, instead of individual devices. Now, when you verify a device, it'll mark all of that users trusted devices as trusted.

Gone are the days of every person you talk to having to independently verify your new device upgrade. Like cross-signing, you can verify a device by scanning a QR code or comparing a short text string.

Sensible and extensible

In Riot, we're implementing these features with a sensible default experience that strikes a balance between usability and security. We think most people would prefer to trust cross-signed devices, and that user trust shouldn't block encryption. However, if you aren't most people, you'll be free to configure whatever level of security you need.

In Summary

With all of the above in place, and after resolving any remaining technical issues, users will be able to:

• Use end-to-end encryption by default in private rooms.
• Use an existing device or Secure Message Recovery to access their encrypted message history on multiple devices, and to signal device trust to other users.
• Access their encrypted message history using Secure Message Recovery, by storing encrypted message keys on their homeserver.
• Mark a user as trusted by verifying one of their devices, persisting across all rooms and devices.
• Keep their encrypted messages out of the hands of eavesdroppers.
• Opt out, or further configure if they have more specific security requirements.

Over the coming days we're polishing wireframes, nomenclature, iconography and microcopy as we dig deeper into development and implementation, as well as designing these features for the upcoming Riot redesign. Cryptography needn't be intimidating, and we're excited to iterate on these advanced features to make them work for everyone.

We'd love to hear your feedback! Let us know your thoughts here or in #e2e-dev:matrix.org.

Wowzers - our 8th dot release for v0.33!

This time we have a bunch of bug fixes and db performance improvements as well as better support for auto-join rooms and the ability for admins to limit who can create rooms aliases.

v0.33.8 also contains more python 3 fixes: we are running most of matrix.org on python 3 as of right now and seeing some pretty impressive performance improvements. Look out for Hawkowl's write up coming soon.

For those interested in what we are working on right now, take a look at our task board.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here.

Synapse 0.33.8 changelog

Features

• Servers with auto-join rooms will now automatically create those rooms when the first user registers (#3975)
• Add config option to control alias creation (#4051)
• The register_new_matrix_user script is now ported to Python 3. (#4085)
• Configure Docker image to listen on both ipv4 and ipv6. (#4089)

Bugfixes

• Fix HTTP error response codes for federated group requests. (#3969)
• Fix issue where Python 3 users couldn't paginate /publicRooms (#4046)
• Fix URL previewing to work in Python 3.7 (#4050)
• synctl will use the right python executable to run worker processes (#4057)
• Manhole now works again on Python 3, instead of failing with a "couldn't match all kex parts" when connecting. (#4060, #4067)
• Fix some metrics being racy and causing exceptions when polled by Prometheus. (#4061)
• Fix bug which prevented email notifications from being sent unless an absolute path was given for email_templates. (#4068)
• Correctly account for cpu usage by background threads (#4074)
• Fix race condition where config defined reserved users were not being added to the monthly active user list prior to the homeserver reactor firing up (#4081)
• Fix bug which prevented backslashes being used in event field filters (#4083)

Internal Changes

• Add information about the matrix-docker-ansible-deploy playbook (#3698)
• Add initial implementation of new state resolution algorithm (#3786)
• Reduce database load when fetching state groups (#4011)
• Various cleanups in the federation client code (#4031)
• Run the CircleCI builds in docker containers (#4041)
• Only colourise synctl output when attached to tty (#4049)
• Refactor room alias creation code (#4063)
• Make the Python scripts in the top-level scripts folders meet pep8 and pass flake8. (#4068)
• The README now contains example for the Caddy web server. Contributed by steamp0rt. (#4072)
• Add psutil as an explicit dependency (#4073)
• Clean up threading and logcontexts in pushers (#4075)
• Correctly manage logcontexts during startup to fix some "Unexpected logging context" warnings (#4076)
• Give some more things logcontexts (#4077)
• Clean up some bits of code which were flagged by the linter (#4082)

Hi all,

Back in June we blogged about the plan of action to establish a formal open governance system for the Matrix protocol: introducing both the idea of the Spec Core Team to act as the neutral technical custodian of the Matrix Spec, as well as confirming the plan to incorporate the Matrix.org Foundation to act as a neutral non-profit legal entity which can act as the legal Guardian for Matrix's intellectual property, gather donations to fund Matrix work, and be legally responsible for maintaining and evolving the spec in a manner which benefits the whole ecosystem without privileging any individual commercial concerns.  We published a draft proposal for the new governance model at MSC1318: a proposal for open governance of the Matrix.org Spec to gather feedback and to trial during the day-to-day development of the spec. Otherwise, we refocused on getting a 1.0 release of the Spec out the door, given there's not much point in having a fancy legal governance process to safeguard the evolution of the Spec when we don't even have a stable initial release!

We were originally aiming for end of August to publish a stable release of all Matrix APIs (and thus a so-called 1.0 of the overall standard) - and in the end we managed to publish stable releases of 4 of the 5 APIs (Client-Server, Application Service, Identity Service and Push APIs) as well as a major overhaul of the Server-Server (SS) API.  However, the SS API work has run on much longer than expected, as we've ended up both redesigning and needing to implement many major changes to to the protocol: the new State Resolution algorithm (State Resolution Reloaded) to fix state resets; versioned rooms (in order to upgrade to the new algorithm); changing event IDs to be hashes; and fixing a myriad federation bugs in Synapse.  Now, the remaining work is progressing steadily (you can see the progress over at https://github.com/orgs/matrix-org/projects/2 - although some of the cards are redacted because they refer to non-spec consulting work) - and the end is in sight!

So, in preparation for the upcoming Matrix 1.0 release, we've been moving ahead with the rest of the open governance plan - and we're happy to announce that as of a few hours ago, the initial incarnation of The Matrix.org Foundation exists!

Now, it's important to understand that this process is not finished - what we've done is to set up a solid initial basis for the Foundation in order to finish refining MSC1318 and turning it into the full Articles of Association of the Foundation (i.e. the legal framework which governs the remit of the Foundation), which we'll be working on over the coming weeks.

In practice, what this means is that in the first phase, today's Foundation gives us:

• A UK non-profit company - technically incorporated as a private company, limited by guarantee.
• Guardians, whose role is to be legally responsible for ensuring that the Foundation (and by extension the Spec Core Team) keeps on mission and neutrally protects the development of Matrix.  Matrix's Guardians form the Board of Directors of the Foundation, and will provide a 'checks and balances' mechanism between each other to ensure that all Guardians act in the best interests of the protocol and ecosystem.
  
  For the purposes of initially setting up the Foundation, the initial Guardians are Matthew & Amandine - but in the coming weeks we're expecting to appoint at least three independent Guardians in order to ensure that the current team form a minority on the board and ensure the neutrality of the Foundation relative to Matthew & Amandine's day jobs at New Vector.The profile we're looking for in Guardians are: folks who are independent of the commercial Matrix ecosystem (and especially independent from New Vector), and may even not be members of today's Matrix community, but who are deeply aligned with the mission of the project, and who are respected and trusted by the wider community to uphold the guiding principles of the Foundation and keep the other Guardians honest.
• An immutable asset lock, to protect the intellectual property of the Foundation and prevent it from ever being sold or transferred elsewhere.
• An immutable mission lock, which defines the Foundation's mission as a non-profit neutral guardian of the Matrix standard, with an initial formal goal of finalising the open governance process.  To quote article 4 from the initial Articles of Association:
  • 4. The objects of the Foundation are for the benefit of the community as a whole to:

    4.1.1  empower users to control their communication data and have freedom over their communications infrastructure by creating, maintaining and promoting Matrix as an openly standardised secure decentralised communication protocol and network, open to all, and available to the public for no charge;

    4.1.2  build and develop an appropriate governance model for Matrix through the Foundation, in order to drive the adoption of Matrix as a single global federation, an open standard unencumbered from any proprietary intellectual property and/or software patents, minimising fragmentation (whilst encouraging experimentation), maximising speed of development, and prioritising the long-term success and growth of the overall network over the commercial concerns of an individual person or persons.
• You can read the initial Articles of Association here (although all the rest of it is fairly generic legal boilerplate for a non-profit company at this point which hasn't yet been tuned; the Matrix-specific stuff is Article 4 as quoted above).  You can also see the initial details of the Foundation straight from the horse's mouth over at https://beta.companieshouse.gov.uk/company/11648710.

• Appoint 3+ more Guardians (see above).
• Finalise MSC1318 and incorporate the appropriate bits into the Articles of Associations (AoA).  (We might literally edit MSC1318 directly into the final AoA, to incorporate as much input as possible from the full community)
• Tune the boilerplate bits of the AoA to incorporate the conclusions of MSC1318.
• Register the Foundation as a Community Interest Company, to further anchor the Foundation as being for the benefit of the wider community.
• Perform an Asset Transfer of any and all Matrix.org property from New Vector to the Foundation (especially the Matrix.org domain and branding, and donations directed to Matrix.org).

thanks,

Matthew, Amandine, and the forthcoming Guardians of [the] Matrix!

We have a LOT of updates this week, so let's dive straight in!

matrix-docker-ansible-deploy

We've covered the growth of this project several times in TWIM, but I wanted to give a little more attention to the work Slavi has been doing with matrix-docker-ansible-deploy. Synapse is a large Python project with many configurable options, and many optional components, so installing it has sometimes been a challenge. I have seen many reports that using Ansible and Docker, and in particular using these playbooks from Slavi, is a more sane way to install Synapse. The tools get a lot of attention and updates. This week, Slavi reports:

matrix-docker-ansible-deploy now has a self-check command that can help diagnose various configuration problems with the setup (DNS records being misconfigured, firewall ports not being open, etc).

For support and more info, come join the associated room: #matrix-docker-ansible-deploy:devture.com >

Late breaking news from Slavi:

One more matrix-docker-ansible-deploy update: the playbook can now set up Whatsapp-bridging via mautrix-whatsapp. Thanks to @izissise!

Dimension

Dimension is an integration manager for Matrix. It's written and maintained by TravisR, and allows you to an a pre-defined selection of widgets, bots and bridges to improve your self-hosted Matrix experience. Check out: https://dimension.t2bot.io/. This week, TravisR reports:

Dimension has received quite a lot of updates over the last week. Here's what's hot off the press:

• 4 new bridges can be self-hosted and managed in Dimension: Telegram, Webhooks, Slack, and Gitter.
• 3 new widgets are available: Grafana, TradingView, and Spotify.
• Add your own custom bots for people to add to their rooms.
• A dark theme has been added and is automatically applied if you use the dark theme in Riot.
• The overall UI has been updated to be slightly more modern and less bright orange.
• Various bug fixes and improvements (is it even possible to have a changelog without this?)

As per usual, if you find any bugs or have ideas for things to add to Dimension feel free to come by #dimension:t2bot.io

mautrix-telegram

tulir has been working on mautrix-telegram, and has made some massive performance improvements:

mautrix-telegram now uses the non-ORM part of SQLAlchemy for database tables that are used often. That change made the CPU usage on the t2bot.io instance drop from ~100% to ~7%

We have graphs to illustrate the improvements:

TravisR, who hosts the bridge on t2bot.io, reports that the bridge is now effectively instantaneous!

The bottleneck has returned to being synapse instead of the bridge

Discord Bridge 0.3.0-rc1

A tonne of work has happened on the Discord Bridge, and it has all been brought together in 0.3.0-rc1.

From the release notes, this is just a subset of the features:

• #251 Support for Postgresql and a newer SQLite3 backend!
• #182 Replace npmlog with winston, for logging to files and better logging overall.
• #221 Add support for m.sticker.
• #210 Discord-side moderation of matrix users.
• #259 Show Matrix replies as Discord embeds.
• #164 Bot will now mention name, topic and membership changes on Discord.
• #175 Add special discord keys onto m.room.member for ghosts

Shoutout to our new member of the team, @Sorunome who did a lot of the review work behind the scenes for this release. Also, thank you to everyone who submitted a PR or an issue!

Slack Bridge 0.2.0 released!

We covered progress on the Slack Bridge previously, but Half-Shot has now declared it ready for 0.2.0 final! The bridge is reportedly running and very stable - go try it out now!

https://github.com/matrix-org/matrix-appservice-slack/releases/tag/0.2.0

Spectral

We just missed out on this update from Spectral last week. Black Hat says:

Spectral now provides an AppImage along with Flatpak build. Also, thanks to the notification codes from nheko, Spectral can show icons in notifications, and now enters corresponding room when clicking on the notification. It also gains several UI/UX improvements.
P.s. I have resubmitted Spectral to Flathub.

FAQBot

Coffee featuring two weeks in a row in TWIM:

FAQBot has been set free at last! Find its code at https://gitlab.com/Matrixcoffee/FAQBot, and its room at #faqbot:matrix.org.

FAQBot sits in various public rooms and answers common Matrix questions. Very useful for demonstrating the product to newcomers!

Also, if anyone wants to help write answers for FAQBot, https://gitlab.com/Matrixcoffee/matrix-knowledge-base is the place.

Strongly encourage people to go take a look at the knowledge base and see what can be improved.

New communities

swedneck has created a new gaming community on Matrix:

we just bridged the linux-gaming community from discord to matrix, with a matrix community and individually bridged rooms/channels and all main room is #general_linuxgaming:matrix.org community is +linux-gaming:matrix.org

i've set up an instance of matrix-appservice-discord, which is bridging some select rooms from the linux-gaming discord server to +linux-gaming:matrix.org

The Linux Gaming community has gotten a proper matrix community (+linux-gaming:matrix.org) with a fair few rooms in it, all of which are bridged to a discord channel via my matrix-appservice-discord instance.

#prismo:matrix.org and #anfora:matrix.org have also been bridged to discord.

Matrix for Ansible notifications

If you are using ansible, jcgruenhage has a useful addition that will allow you to get notifications from matrix:

After over a month of waiting, the matrix notification module has been merged on to ansible devel which will be released as ansible 2.8 early next year. Src: https://github.com/ansible/ansible/pull/45823

mxisd v1.2.0-beta.3

Max:

mxisd v1.2.0-beta.3 is out with the start of a brand new Identity store based on arbitrary executable, to connect to anything and everything. Authentication is implemented at the moment (see doc). Feedback is very welcome to improve as much as possible for v1.2.0

MSC1695 Message Edits

Discussion of message editing, in particular for how message edits from Bridges are handled has progressed. Nothing is final yet so check out https://github.com/matrix-org/matrix-doc/pull/1695 for the latest.

Quaternion translations: German and Polish now available

Last week we had an update from kitsune to say that there was a new Lokalise project to allow Quaternion translations. This week, we learn that the first translations are now available:

First couple of translations - German and Polish - have made it to the released Quaternion 0.0.9.3 - thanks to krombel and krkk for their contributions! Swedish and Russian translations are in the works.

Fluffychat

We anticipated it last week, but here it is:

The first stable release of the #matrix messenger #fluffychat is out now. ? Get it from: https://www.fluffy.chat
Thanks to all who have helped me. Thanks to regionetz for hosting the ubports.chat homeserver, thanks to @matrix for the awesome work, thanks to @Ubports for this awesome platform and to fabiyamada, advocatux, wayneoutthere, lionelb, Diogo, mithgarthsormr, mark, and all the awesome people from the community!!
With your help, Ubuntu Touch is still alive and has got a new stable messenger!

Informo: new bot for spec changes

Informo is an ambitious project hoping to be a "decentralised news network, making information accessible". The project lives at https://github.com/Informo, but for now you can join #discuss:weu.informo.network to get their latest news.

This week, vabd reports:

We made a Matrix bot that shouts about updates to change submissions to Informo's specifications. It basically processes all changes made to the list of labels for each issue and PR of a GitHub repository's, and generates a notice message that it sends to the configured room(s). We made it because we wanted the people that are interested in Informo to know in real time about any change made to the state of proposals, along with the calls for public reviews. We just set it up in #discuss:weu.informo.network, and published its source code along with a built binary here: https://github.com/Informo/specs-bot
It might also be worth noticing that, although we designed it to shout about updates to Informo's specifications proposals, we also made it compatible with other projects, e.g. the Matrix specs

Push-to-talk functionality with Jitsi on Riot

anoa has been making improvements to Video Calling on Riot:

I've been working on global push-to-talk functionality with Jitsi on Riot. I've got toggle on/off functionality working, but still trying for walkie-talkie mode. To do so, I need to get this library working with Riot: https://github.com/WilixLead/iohook
If anyone has experience with native Node modules and/or Electron, please hit me up! @andrewm:amorgan.xyz

Greetings from Mozfest

Mozfest is a tech-focused event happening this weekend in London. Neil and I have been along tonight and we've been chatting to a lot of people about Matrix, decentralisation and all those things we love! Check out our very short and sweet video below!

This is the start of Season Three of Matrix Live. Matrix Live seasons are variable in length, based on the data available so far. From this season, Matrix Live with change the format slightly, based on feedback. The videos will try to be a bit more interesting, varied, and deep. With this video being the start of a new season, it was meant to be more substantive with us talking to Mozfest-ites, but I lost track of time, so this shorter but still energetic video will hopefully convey the idea.

Olm 3.0.0 has been released, which features several big changes. It can be downloaded from https://git.matrix.org/git/olm/. The npm package for JavaScript can be downloaded from https://matrix.org/packages/npm/olm/olm-3.0.0.tgz

Python

The biggest change is the merge of poljar's improved Python bindings. These bindings should be much easier to use for Python programmers, and are used by Zil0's E2E support in the Matrix Python SDK.

Since the binding API has changed, existing Python code will need to be rewritten in order to work with this release.

poljar has also included comprehensive documentation for the new API.

CMake

mujx contributed support for building olm using CMake. This should allow for easier building on different platforms. Currently the library can be built using either make or CMake. In the future, make support may be removed.

JavaScript

The JavaScript bindings now use WebAssembly by default. WebAssembly is much faster than the previous asm.js build, and is supported by recent versions of the most popular browsers. For compatibility with browsers that do not support WebAssembly, the asm.js version is still provided.

Due to adding support for WebAssembly, the API had to be changed slightly. There is now an init function that must be called before the library can be used. This function will return a promise that will resolve once the library is ready to be used. The matrix-js-sdk has not yet been updated to do this, so users of matrix-js-sdk should continue using olm 2.x until it has been updated.

Key backups

The public key API has been updated to support the proposal for server-side key backups. More details on how to use these functions will be published in the future.

Hi folks,

Today is one of those pivotal days for the Matrix ecosystem: we're incredibly excited to announce that the world's first ever dedicated homeserver hosting service is now fully available over at https://modular.im!  This really is a massive step for Matrix towards being a mature ecosystem, and we look forward to Modular being the first of many hosting providers in the years to come :D

Modular lets anyone spin up a dedicated homeserver and Riot via a super-simple web interface, rather than having to run and admin their own server.  It's built by New Vector (the startup who makes Riot and hires many of the Matrix core team), and comes from taking the various custom homeserver deployments for people like Status and TADHack and turning them into a paid service available to everyone.  You can even point your own DNS at it to get a fully branded dedicated homeserver for your own domain!

Anyway, for full details, check out the announcement over at the Riot blog.  We're particularly excited that Modular helps increase Matrix's decentralisation, and is really forcing us to ensure that the Federation API is getting the attention it deserves.  Hopefully it'll also reduce some load from the Matrix.org homeserver! Modular will also help Matrix by directly funding Matrix development by the folks working at New Vector, which should in turn of course benefit the whole ecosystem.

Many people reading this likely already run their own servers, and obviously they aren't the target audience for Modular.  But for organisations who don't have a sysadmin or don't want to spend the time to run their own server, hopefully Modular gives a very cost-effective way of running your own dedicated reliable Matrix server without having to pay for a sysadmin :)

We're looking forward to see more of these kind of services popping up in the future from everywhere in the ecosystem, and have started a Matrix Hosting page on the Matrix website so that everyone can advertise their own: don't hesitate to get in touch if you have a service to be featured!

If you're interested, please swing by #modular:matrix.org or feel free to shoot questions to [email protected].

Bridges and SDKs

Slack Bridge 0.2.0-rc1

Half-Shot, master of the bridge, has been working with Cadair to produce a new RC for bridging Matrix and Slack.

We've got a Slack RC out at long last! Rejoice! It's absolutely massive and full of features.

Half-Shot even provided this explanation of how it works:

Integrations manager > sign into slack > click your channel > ??? > channel bridged :)

See release notes:

This is the first release of the Slack bridge. 0.1.0 has been the version number for previous efforts but was never an official release.

To test this integration, use Riot on https://riot.im/develop and select the "Event Bridging" when adding the integration.

maubot Python

tulir's maubot bot system:

The maubot Python rewrite I twimmed two weeks ago is now complete. The plugin system seems to work well and I'm pretty sure I'll be able to implement proper plugin reloading now. Next I'll implement plugin config storage and some way to manage maubot and plugins (maybe a plugin to manage plugins?)

Clients

Riot Web 0.17, Riot Android 0.8.18: Lazy Loading

This week saw the launch of Riot Web 0.17, and two bug-release updates (0.17.1 and 0.17.2) to fix issues on the Windows desktop app. This version is substantially faster due to its use of Lazy Loading room members. Read more here.   Meanwhile, Travis continues his foray into 'first impressions' bugs - including an initial implementation of .well-known URI support!

Riot Android 0.8.18 is also available from the Play Store and F-Droid, with Lazy Loading option available in the Labs menu (but still has a few bugs left).

Riot iOS meanwhile is busy implementing incremental server-side E2E key backups, and there's generally been a huge amount of work on E2E encryption UX across the board in preparation for all-new E2E on Web and iOS.  More details will be coming soon!

With this done, Riot is now getting a lot more attention on the impending redesign, with Bruno starting to merge code to the experimental branch.

Quaternion 0.0.9.3

After being in RC for a week, Quaternion 0.0.9.3 is ready and will be released over the weekend. Most importantly, you can now translate it into your language! Just head over to https://lokalise.co/public/730769035bbc328c31e863.62506391/, register (you can optionally reuse your GitHub account), ask in #qmatrixclient:matrix.org to add your language to the list (if it's not there yet) and start translating!

Coffee on Matrix Console for Android

Coffee rolled a natural twenty for bravery this week.

I have decided to take up maintainership of the Matrix Console for Android client. This is still the only multi-account capable Android client, but it is unmaintained and growing long in tooth. In particular, the API endpoints it uses may be removed from Synapse soon. I will not be developing new features for it, but I will integrate reasonable patches if others want to take that on. My own goals are to remove GCM and analytics, and get it added to F-Droid. And of course to switch to the latest API. As part of this work, I have been fighting Gradle and its bugs to get matrix-android-sdk to build together with matrix-android-console as a git submodule, so it's no longer necessary to inject the precompiled sdk into the source code. I did not win yet.

FluffyChat stable release creeps closer

Ubuntu Touch fans will know that FluffyChat has been progressing rapidly, and the project is approaching a first stable release. You can find current features being tested as part of the release here.

nheko

mujx has decided to stop maintaining nheko. Since many people are using it, we hope that someone will step in to continue his work. Thanks to mujx for his work so far.

Featured on Mozilla Hacks blog

If you like blog posts about Matrix (and if you're reading this, you may well do), then you'll be interested to see that Matrix was featured on the Mozilla Hacks blog: Decentralised, Real-Time, Interoperable Communication with Matrix. The article was also picked up by Hacker News.

Raiden Network uses Matrix for transport - this article explains it

The Raiden Network enables fast payments for Ethereum. From their website:

The Raiden Network is an off-chain scaling solution, enabling near-instant, low-fee and scalable payments. It's complementary to the Ethereum blockchain and works with any ERC20 compatible token.

To help explain their use of Matrix in their solution, they have prepared an article: Raiden Transport Explained.

Synapse 0.33.7

Soon, so soon, there will be a full Python 3 release for those running Synapse in worker configuration, but this one is not it. Check out Synapse 0.33.7 changes here.

Synapse on kubernetes update

Ananace:

tracking the latest synapse release on my K8s optimized image. Got no real time to work on things due to deadlines at work, but that should end come November, so expect more odd Ruby stuff after that point.

"Odd Ruby stuff" will hopefully include a return to the Matrix Ruby SDK!

mxisd v1.2.0-beta.2

Max continues work on mxisd:

mxisd v1.2.0-beta.2 is out, fixing bugs found during beta.1

Cadair at GSOC

Cadair was at the GSOC 2018 Mentors Summit in Mountain View last week:

I attended the GSOC mentor summit. I had some great conversations with people who are using matrix and with people about bridging in different chat services. A lot of matrix stickers all vanished off the overloaded sticker table. I have lots of ideas for GSOC next year, and plan to try and get many more community projects involved. Finally, I dont need to eat chocolate for a month.

GSOC chocolate: GCHOC

Until next week

Next Friday /me is going to be at MozFest 2018, promoting Matrix, Open Source, decentralisation etc. (all the stuff we know and love), so I may change the schedule a little next week.  We're also going to reboot Matrix Live, so consider this a hiatus before Season 3 begins next week!!

Hey ho, let's go. Synapse 0.33.7 has arrived.

Regular readers will know how close we are to a full python 3 release. We are not quite there yet but 0.33.7 has support for Synapse under worker mode and we've running it on matrix.org this week. We need more time to conclusively gauge performance improvements but the Synchrotron workers are running with 33% less RAM. Thanks to everyone who has been running their servers under py3, if you do spot anything unusual just let us know. Once we've been running it a bit longer on matrix.org, we'll cut a 0.34.0 release with a recommendation that one and all upgrade to python 3.

Aside from that this release contains support for server side end to end key backups, paving the way for client side support in Riot and Rich continues his long running federation bug squash-a-thon which should help with a whole host of federation snafus.

Up next on the horizon is returning in earnest to getting the server to server r0 spec out starting with shipping our brand new super shiny state resolution algorithm.

As a final point, for those of you that deploy from git checkout or a snapshot url and have email notifications enabled please take a look warning in the change log.

As a final final point #synapse:matrix.org is now an officially supported room, aimed at Synapse admins. If you've not done so already please do drop by and say Hi.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here.

Onwards!

Synapse 0.33.7 Change Log

Warning: This release removes the example email notification templates from res/templates (they are now internal to the python package). This should only affect you if you (a) deploy your Synapse instance from a git checkout or a github snapshot URL, and (b) have email notifications enabled.

If you have email notifications enabled, you should ensure that email.template_dir is either configured to point at a directory where you have installed customised templates, or leave it unset to use the default templates.

The configuration parser will try to detect the situation where email.template_dir is incorrectly set to res/templates and do the right thing, but will warn about this.

Features

• Ship the example email templates as part of the package (#4052)
• Add support for end-to-end key backup (MSC1687) (#4019)

Bugfixes

• Fix bug which made get_missing_events return too few events (#4045)
• Fix bug in event persistence logic which caused 'NoneType is not iterable' (#3995)
• Fix exception in background metrics collection (#3996)
• Fix exception handling in fetching remote profiles (#3997)
• Fix handling of rejected threepid invites (#3999)
• Workers now start on Python 3. (#4027)
• Synapse now starts on Python 3.7. (#4033)

Internal Changes

• Log exceptions in looping calls (#4008)
• Optimisation for serving federation requests (#4017)
• Add metric to count number of non-empty sync responses (#4022)