🔗Matrix Live S03E06

In which Matthew & Amandine discuss The Matrix.org Foundation, go-live for the French Government deployment for Matrix, and some exciting random diversions into post-1.0 Matrix features which should land once once 1.0 is out the door!

🔗Purism sponsoring Fractal development

We are proudly sponsoring the work of Julian Sparber on Fractal, the @matrixdotorg client for GNOME. Read his latest update - https://t.co/q4ubjzETxC #LibreDesktop #DemandFreedom #gnome

— Purism (@Puri_sm) December 5, 2018

Purism announced they are sponsoring Fractal development, starting with Julian Sparber working last month to improve the message view. You can read about his progress here. Highlights:

Before, there was a jarring cut when new messages were loaded, but now you can just scroll upward and older messages are loaded continuously.
The part I'm most excited about is the new "new message divider". When the user opens a room they can directly start reading the conversation from the last seen message and they don't need to search for the new message divider. Not only has the UX gotten a lot better, but also the underlying code is much cleaner now.
I also spent some time on making message rendering faster. I replaced the RegExp with more efficient code, this made the rendering much faster (from ~10ms to ~ 1ms) for every single message.
In summary, all of these things improve Fractal's UX a lot and make it feel more like a modern messaging app.

🔗Synapse 0.34.0rc1

0.34.0rc1 is out - please test it! 0.34.0 will be the first to officially support python 3. We're still working on the debian packaging but 0.34.0 proper should be out next week.

🔗Dendrite

Brendan has been getting on with Dendrite development:

Started auditing Dendrite's codebase to identify what is left to implement, along with what hasn't been implemented correctly. Still a lot of work left to do on this, but it's looking promising so far.

🔗Modular

Modular is a Hosted Homeservers product, check out modular.im. This week:

Matrix -> Matrix account migration tool available at https://www.modular.im/tools/matrix-migration. It's still beta, so please test it and let us know if you experience issues.

🔗New Homeserver: linuxgaming.life

swedneck has set up a new public homeserver at linuxgaming.life! There is also a Riot installation at https://riot.linuxgaming.life, served via IPFS.

Currently working on the website (based on t2bot.io, thank you travis for making that available on github!) and making things nicer in general.
I will be adding more bots shortly. Please let me know of any issues with the HS or anything surrounding it.

🔗Informo

Informo is a project to create a specification and implementation for distributing information and news. This week they have been working on merging changes to the specification:

Work on the Informo specifications has slowed down a bit in the last couple of weeks, though since the last update we did manage to get some relatively big SCSs merged into the specs, including SCS #9 (rendered here) which specifies how information sources must publish their information through the federation, and SCS #11 (rendered here) which describes how sources must register themselves in order to be picked up by clients, and handle localisation. ?
There's still a couple of big items to take care of before we can cut a 1.0 release of the Informo specs but this is definitely a huge step towards this goal.

🔗Minecraft Bridge

Dandellion:

I started working on a Minecraft bridge pretty heavily based on Travis's old project, but using Minecraft-protocol instead of mineflayer.
You need a bot Minecraft account that can join and idle in the server, it then uses /tellraw to post messages.

🔗libQMatrixClient 0.4.1 released

libQMatrixClient, which powers Spectral and Quaternion was released by kitsune, Lazy Loading coming soon:

libQMatrixClient 0.4.1 has been released today, with small fixes in the stable branch. Meanwhile, active work is ongoing on lazy-loading support in the library, with ETA for the feature landing in master being in about a week or so.

🔗Riot Web

• Redesign progress: all resizeable sections are now persisted, (slow) progress on improving read markers
• Communities and community invitations should be more resilient to errors (on /develop only so far)

🔗Riot iOS

• Reskin still ongoing
• A new release is coming
• Less activity because of POSS

🔗Riot Android

• A new developer joined us to help maintaining Riot: Valere Fedronic
• Keys backup PR in review
• Privacy Terms acceptance in login flow in review
• FCM issue investigation/troubleshooting by Valere
• François Off (sick :()
• New theme is coming soon

🔗Koma

druig has been working on Koma, a JavaFX Matrix client:

This week in koma: implementation of json library switched from runtime reflection to compile-time code generation
The matrix api is implemented manually in the project, retrofit and moshi are used to interact with the rest api.

🔗SimpleMatrix

As we mentioned in TWIM last week, MTRNord has been working on designs for SimpleMatrix, a Matrix client for Android in development. This week he has made a video showcasing the new design.

🔗Redecentralize meetup in London

Last night Half-Shot & Neil & Brendan went to see the Redecentralize folk at their meetup:

Beer! And pizza! And also Redecentralize
We chatted to a nice bunch of folks both demoing their decentralized projects and talked to the likes of scuttlebutt, BBC, IPFS and more. Was a interesting experience having around 8 minutes to quickly explain to newcomers what Matrix is and why they should use it in a speed dating format. We chatted (and demoed) bridges, new-riot and generally how it all fits together. There were a lot of very interesting people with different profiles and backgrounds, and diversified questions which lead to a lot of interesting discussions.

Note for the confused: the format of the meetup was comparable to speed dating. There is no suggestion that Matrix should be used at regular speed dating. If you do find such a use for Matrix, come tell us in #twim:matrix.org.

🔗matrix-wug, X-SAMPA to IPA bot

Last month Dandellion introduced a bot designed to convert between formats for pronunciation notation, this week he released the source code: https://github.com/dali99/matrix-wug

🔗The end of the post, and nearly the end of 2018

This week it has been winter-in-Europe kind of weather, but that's ok, it happens every year. TWIM on the other hand happens every week, so if you have something to share, and would like to share what you've been working on, come chat to us in #twim:matrix.org.

🔗Porting to Python 3 - Matrix Live S3E5

Longer than usual episode of Matrix Live this week. Amber talks to Neil about the works she's been doing over the last few months to port Synapse from Python 2 to Python 3. Recommended for anyone who's been following along the progress of Synapse, or who wants a good intro on the benefits of Python 3 over Python 2.

🔗ZeroPhone

This week we learned about ZeroPhone, a fairly mature project aiming to build a working cellphone on top of a Raspberry Pi Zero.

an open-source smartphone that can be assembled for 50$ in parts. It is Linux-powered, with UI software written in Python. Currently, ZeroPhone is based on Raspberry Pi Zero, SIM800L GSM modem and 1.3" OLED screen and button interface.

&Adam was the first to bring it to our attention:

I stumbled across this blog post which mentions a matrix app in a project to create an open, hackable phone out of a raspberry pi. It's old news at this point I guess, but I don't think I've seen it mentioned here, so I thought it might be a neat thing worth featuring: https://zerophone.github.io/newsletter/ZeroPhone-Weekly-No.-16/

The creator of the Matrix app, derivmug also arrived to tell us about his work:

Hey everyone, I have been suggested to post about the Zerophone project here. I have written a basic matrix client for it. It's basically a simple phone based on a raspberry pi zero, focused on privacy, security and hackability.

The client app is built using the matrix-python-sdk, I'm also hoping to chat to derivmug as part of a future Matrix Live. Take a look at the code here.

Just to prove it's running on real hardware!

🔗Spec / Foundation

• Work continues on MSC1730 - Mechanism for redirecting to an alternative server during login https://github.com/matrix-org/matrix-doc/pull/1730
• Foundation: Working with the Foundation lawyers on the CIC37 to register the Foundation as a Community Interest Company proper

🔗The Federation (the-federation.info)

Jason Robinson is passionate about Federated technology, and has been working on his site at https://the-federation.info/ for some time. He recently added Matrix to the list of protocols he scrapes and lists:

Matrix (or more specifically, Synapse) servers are now visible on the The Federation info website at https://the-federation.info/matrix%7Csynapse. The website collects lists of servers for various federation protocols (like Matrix, ActivityPub, Diaspora, etc). If a server outputs usage information, it also shows some historical information for servers and projects. Currently the information available from Synapse on the site is just version, availability of registrations and (obviously) domain name. If you want to register your server, check https://the-federation.info/info for more details. Tips also welcome on how to register other Matrix server projects and how to get more data out of servers.

🔗maubot

tulir has continued work on maubot:

• I made a rss plugin for maubot and added a fancy log viewer in the maubot management UI.
• mautrix-telegram 0.4.0 was released, though it has no changes since the release candidate. I also copied mautrix-telegram's HTML parser into mautrix-python. Hopefully it's now generic enough so that when mautrix-telegram switches to mautrix-python, it can drop most of the built-in HTML parser.

🔗matrix-appservice-discord

Bridge-of-the-week this week is matrix-appservice-discord. Anyone in the dev room (#discord:half-shot.uk) will be very aware of this as Half-Shot and Sorunome have spent the day testing @room notifications.

Sorunome:

matrix-appservice-discord received a lot of work these past days. Not only the usual small bugfixes, but also some rather big changes: The parsers for both Discord->Matrix sending and Matrix->Discord sending were re-written completely to properly handle things (and thus, at the same time, crunch a bunch of edge-case bugs!). In addition, the long-awaited highlighting issues with @ everyone and @ here (on discord) and @ room (on matrix) were addressed. Some of these things are still in PR stage, but expected to be merged soon!

Half-Shot also provided a great screenshot of everything working well:

is just showing off having IRC+Matrix+Discord+XMPP plugged together. XMPP being the one purple is bridging

🔗matrix-docker-ansible-deploy

Slavi reports that:

matrix-docker-ansible-deploy now supports configuring the matrix-synapse-ldap3 LDAP auth password provider. Thanks to @tvo6 for contributing this!

🔗Riot iOS

• User Agreement (GDPR) in now part of the registration flow
• Reskin has started
• New fixes for unexpected “Empty room”
• We have PR templates

🔗Riot Android

• Realm store for e2e keys has landed
• Riot reboot: support timelines merge in DB to avoid to trash data as less as possible when navigating through permalinks
• New settings to choose ringtone for incoming call
• We have PR templates also

🔗SimpleMatrix

MTRNord, is working on the design for SimpleMatrix:

As I currently have less time for actual coding I am planning out the SimpleMatrix Design. You can find the Images of the Design (and a PDF file) at https://gitlab.com/Nordgedanken/simplematrixredesignimages Feel free to also join #SimpleMatrix:matrix.ffslfl.net to give feedback.

🔗Synapse

• This week our focus has been to work through some final bugs blocking 0.34.0 which will be our first official python 3 release. Expect a RC rsn. For a sneak peak see this week's Matrix Live video at the top!

🔗Auf Wiedersehen…

That's it for this week. If you have anything to share with us, and you'd like to be included on the blog, please come talk in #twim:matrix.org!

In one month from now, many Matrix-acolytes, including myself, will be at 35c3. If you'll be there, come chat to us in #matrix-35c3:matrix.org!

A little delay to the release of this week's TWIM, expect us to resume normal schedule on Friday.

🔗Spec & Foundation

• Two workshops happened with the Foundation lawyers to work on MSC1318 - basically fleshing out a proposal for the details of the foundation governance in terms of how the Guardians (i.e. directors of the foundation board) should interact with the Core Spec Team to act as a back-stop - and working out how to port the MSC (once finished) into formal Articles of Association & adjunct documents. We'll get the workshop notes written up as a rewrite of MSC1318 very shortly.
• New MSCs from vdH on:
  • MSC1721 Rename m.login.cas to m.login.sso
  • MSC1730 Mechanism for redirecting to an alternative server during login
• …and an alternate to MSC1730: MSC1731 Mechanism for redirecting to an alternative server during SSO login.
• These three SSO-related MSCs are derived from work being done for a large-scale Matrix deployment for a New Vector customer which we should be able to talk about in future. Improving SSO and other auth/identity work in Matrix is vital, and needless to say we're trying to ensure all the resulting work ends up in the spec to the benefit of the whole ecosystem :)

🔗matrix-media-repo

TravisR reports that:

matrix-media-repo has received a lot of updates in the last few weeks. Although mostly bug fixes and improvements, there's added support for streaming downloads from remote servers. This is mostly useful when someone posts a 100mb video in a room: previously the media repo would download the entire thing then send it over to you. Now it'll send you the video while it also caches it.

🔗matrix-docker-ansible-deploy

Big update from Slavi this week:

matrix-docker-ansible-deploy has received some updates in the past few weeks - fixing minor bugs, improving documentation, adding support for additional Synapse/Riot features, etc.

Most notably (in oldest to most-recent order):

• Cadair heard you like playbooks, so he made it possible to run the playbook from another playbook. Thanks Cadair!
• some configuration renaming to make things more consistent. Most people would likely be unaffected.
• support has been added for Postgres 11. Upgrading is possible and easy (as a related success story, Aaron Raimist shared that he recently upgraded his ~40GB database)
• more flexible and more secure-by-default TLS configuration. Thanks Aaron Raimist!
• the ability to customize more parts of riot-web. Thanks Aaron Raimist!
• better control over Synapse logging. Thanks Aaron Raimist!
• the ability to set up a public server and the ability to make use of Synapse's \\"auto-join\\" feature. Until now, the playbook was only suited for setting up a private server, but with this, setting up public servers is now a possibility. Thanks @anadahz

I'm quite happy that the playbook is getting more contributions from others now and is improving in such diverse ways!

Additionally, the #matrix-docker-ansible-deploy:devture.com support room in Matrix is now bridged to the #matrix-docker-ansible-deploy IRC Freenode channel, making it easier for people to get support when they don't have a working homeserver.

🔗libQMatrixClient

kitsune has been working on libQMatrixClient:

the work on lazy-loading in libQMatrixClient has advanced to the point when client authors may start trying to integrate it in their applications (welcome to #qmatrixclient:matrix.org for details - it's a bit rough around the edges yet).

🔗Spectral

This week Black Hat has been working to give Spectral a more Material design oriented UI.

🔗SMSMatrix

swedneck reports that:

SMSMatrix now has a room (#smsmatrix:matrix.org). SMSMatrix is a simple bridge between Matrix and SMS, developed by tijder. You install an android app and register a bot account which will invite you to direct chats under the name of anyone who sends you an SMS, which will then act as a normal SMS conversation.

I had a very quick play with this tool and it works great - a good way to quickly bridge SMS to Matrix.

🔗Riot

News from the Riot teams:

• Android rewrite is continuing in Kotlin - lots of infrastructure work over the last few weeks
• E2E key backups landed in Riot/iOS develop (but no UI yet, as we finalise the implementation first on Riot/Web)
• E2E key backups also landed in Riot/Web develop (but with placeholder UI)
• E2E key backups in dev on Riot/Android too (using Realm for storage!)
• Lots of work from uhoreg on E2E cross-signing
• Work on the Riot/Web redesign paused for a week due to other commitments but as of Nov 26th we're back on the case again, making up for lost time.

🔗Synapse

Most of the focus from the Synapse team this week has been on the new version using Python 3:

• Py3 is finished, other than Debian packages - ongoing debate on whether they should use a virtualenv or depend on debian dependencies, and who should maintain them in a Py3 world…
• Lots of running around on final Py3 bugs

🔗matrix-appservice-purple

Half-Shot has continued work on matrix-appservice-purple.

🔗Modular

Modular.im provides Hosted Homeservers for Matrix.

• Modular now has full support for Custom DNS (letting you point your own SRV records at it)
• Lots of work on a migration tool to allow folks to easily shift Matrix accounts between servers, as well as (in future) migrations from HipChat, Slack & others.

🔗That's it!

Apologies for the delayed release this week - please expect an episode of Matrix Live in the week, and back to a normal schedule for TWIM as of Friday. As always, come talk to us in #thisweekinmatrix:matrix.org and tell us what you've been doing.

Well here you are then, the 9th episode in the Synapse 0.33.x series.

Features wise, 0.33.9 contains a change to the way that GDPR consent works under the hood. It is now plumbed in to the login flow (rather than following immediately afterwards) such that it does not inadvertently break on-boarding. This is part of a broader set of changes that span Synapse and Riot to improve initial first impressions of using matrix.

Separately we now have support for room version upgrades which is pre-requisite for rolling out the new state resolution algorithm, come and join us in #teststateresv2:jki.re if you would like to help us test.

Finally we've spent a bunch of time further improving perf especially in and around reducing device ids federation traffic.

I know I say it every time, but full python 3 support is really really close now, matrix.org is now running entirely on py3 and seeing some amazing perf improvements - the remaining blocker is getting py3 deb packages ready and then we'll ship an official python 3 release. There will also be a blog post to explain what we've been up to and what to expect perf wise.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.

🔗Synapse 0.33.9 changelog

🔗Features

• Include flags to optionally add m.login.terms to the registration flow when consent tracking is enabled. (#4004, #4133, #4142, #4184)
• Support for replacing rooms with new ones (#4091, #4099, #4100, #4101)

🔗Bugfixes

• Fix exceptions when using the email mailer on Python 3. (#4095)
• Fix e2e key backup with more than 9 backup versions (#4113)
• Searches that request profile info now no longer fail with a 500. (#4122)
• fix return code of empty key backups (#4123)
• If the typing stream ID goes backwards (as on a worker when the master restarts), the worker's typing handler will no longer erroneously report rooms containing new typing events. (#4127)
• Fix table lock of device_lists_remote_cache which could freeze the application (#4132)
• Fix exception when using state res v2 algorithm (#4135)
• Generating the user consent URI no longer fails on Python 3. (#4140, #4163)
• Loading URL previews from the DB cache on Postgres will no longer cause Unicode type errors when responding to the request, and URL previews will no longer fail if the remote server returns a Content-Type header with the chartype in quotes. (#4157)
• The hash_password script now works on Python 3. (#4161)
• Fix noop checks when updating device keys, reducing spurious device list update notifications. (#4164)

🔗Deprecations and Removals

• The disused and un-specced identicon generator has been removed. (#4106)
• The obsolete and non-functional /pull federation endpoint has been removed. (#4118)
• The deprecated v1 key exchange endpoints have been removed. (#4119)
• Synapse will no longer fetch keys using the fallback deprecated v1 key exchange method and will now always use v2. (#4120)

🔗Internal Changes

• Fix build of Docker image with docker-compose (#3778)
• Delete unreferenced state groups during history purge (#4006)
• The "Received rdata" log messages on workers is now logged at DEBUG, not INFO. (#4108)
• Reduce replication traffic for device lists (#4109)
• Fix synapse_replication_tcp_protocol_*_commands metric label to be full command name, rather than just the first character (#4110)
• Log some bits about room creation (#4121)
• Fix tox failure on old systems (#4124)
• Add STATE_V2_TEST room version (#4128)
• Clean up event accesses and tests (#4137)
• The default logging config will now set an explicit log file encoding of UTF-8. (#4138)
• Add helpers functions for getting prev and auth events of an event (#4139)
• Add some tests for the HTTP pusher. (#4149)
• add purge_history.sh and purge_remote_media.sh scripts to contrib/ (#4155)
• HTTP tests have been refactored to contain less boilerplate. (#4156)
• Drop incoming events from federation for unknown rooms (#4165)

🔗Matrix Live S3E04

This week we continue Matrix Live Season 3 by talking to community member axx, Axel Simon from La Quadrature du Net, a French advocacy group that promotes digital rights and freedoms of citizens. We talk about the work La Quadrature du Net do, with a focus on the importance of decentralisation and how Matrix helps support this.

🔗mxisd v1.2.0 released

Max has been giving updates on the road to mxisd v1.2.0, which was released this week:

A new stable version of mxisd is out: v1.2.0. It comes with:

• The ultimate identity store that lets you run any command on the system to fetch info, making it the most generic yet. The sky is now your limit!
• The ability to send email notifications about room invites when done using a Matrix ID regardless if their users is already provisioned on synapse, using emails found in an Identity store. Targeted at onboarding/migration to Matrix for any org/corp.

🔗Spec

uhoreg:

TWIM: five(!) new MSCs so far this week, including a long-awaited MSC for displaying math(s) in messages: https://github.com/matrix-org/matrix-doc/pull/1722

🔗matrix-appservice-purple

Half-Shot has been working on important work for his final undergrad year. Just kidding! He's been working on bridges as you'd expect:

I've been working on matrix-appservice-purple, and the community immediately rallied around and helped me get it into shape. We've got automated builds for both the bridge and the libpurple binding modules. In features land, group chats are now working at a basic level and I will be working on supporting profiles next.
#purple-bridge:half-shot.uk is now a room where you can tell me why your favorite protocol doesn't work with the bridge. (It's also used for updates.)

🔗Informo

vabd:

A couple of SCS (Specs Changes Submissions) have been merged into the Informo specs, notably SCS #2 which introduces a complete technical description of the network's nodes and their expected behaviour, and SCS #4 which changes the duration of the call for public review period, shrinking it from 14 days to 7 days, in order to speed specs work up while letting a decent amount of time for people to give a look at new SCSs and voice their concerns.

🔗Synapse 0.33.9rc1

Synapse has a new release candidate - 0.33.9rc1 - If you'd like to help us test it, you can get it here.

🔗Koma

druig:

This week in Koma, I have been working on a correct implementation of the user registration process. Currently waiting for a small issue with synapse to get fixed
For kotlin programmer who might be interested https://github.com/koma-im/koma/pull/6

🔗f-droid.org has set up a new matrix (synapse) server

Mathijs:

f-droid.org has set up a new matrix (synapse) server for internal conversation and to chat in #fdroid:matrix.org , which will obviously also get a :f-droid.org alias.

For now the server is private, only allowing core team members to get an account in order to keep the performance manageable.

🔗mautrix-telegram v0.4.0

tulir released an RC for mautrix-telegram:

mautrix-telegram 0.4.0 release candidate was released. It's mostly a lot of bugfixes.

…and TravisR hosted it on t2bot.io:

as of a few minutes ago, the t2bot.io Telegram bridge is updated to 0.4.0rc2

🔗Push-to-Talk for Jitsi calls in Riot

anoa:

An implementation for Push-to-Talk for Jitsi calls in Riot has now been completed and is in the review phase. This was a result of multiple weeks of work, with code changes across many different repositories. Will hopefully make a difference with background noise or many participants. Look forward to seeing it land in /develop sometime shortly! Works with both Scalar and Dimension setups.

• https://github.com/matrix-org/matrix-react-sdk/pull/2280
• https://github.com/vector-im/riot-web/pull/7709

🔗gomuks progress

tulir:

gomuks development is slowly continuing. It now uses mautrix-go and I also added proper reply rendering.

🔗More Things

Last week we mentioned that Julian Sparber has been working on Fractal UI. He has since published a blog post about the work, and also notes that Purism are beginning to sponsor Fractal development: https://blogs.gnome.org/jsparber/2018/11/10/purism-fractal-sponsorship/

Half-Shot: "The events API update for Slack was released on riot.im/develop a little while ago, which let you do more than webhooks could let you do. The UI scalar/integration manager bits were left on /develop for testing but got rolled out to /app this week."

If you have feedback about the matrix.org website (which you may be on right now!), we have a new room to collect and organise it: #matrix.org-website:matrix.org. Thank you to Aaron Raimist for much-needed chasing (of me) on this work…

This week I attended TADSummit in Lisbon to tell them about the excellent progress Matrix has been making this year. You can see more details of the conference (plus video) here: http://blog.tadsummit.com/2018/11/16/tadsummit-2018-web3/

🔗That's all I got!

If you need more, come back here next week, for all the latest from This Week in Matrix! Also, join us in #twim:matrix.org to tell us what you've been doing.

🔗Matrix Live

We continue the new format for Matrix Live season 3 by chatting with Francois from New Vector, to get a sneak preview of the work he's doing on a new version of matrix-android-sdk, and how that will impact Riot:

🔗Dendrite and Brendan

Welcome to Brendan, who has started working on Dendrite as his new day job at New Vector:

There's been some progress on Dendrite (?), with a couple of bug getting fixed along with some progress in the implementation of Matrix endpoints, such as the /backfill federation one (documented here), which has already been merged, and the /messages one from the client-server API (documented here) for which a pull request has been opened and is currently under review.

🔗Python Matrix Client async API

Cadair:

I have pushed my asyncio wrapper of the Python matrix client api class to pypi, so I can use it in a few different projects while the PR to the Python SDK is in an unmerged state. It exposes all the methods on the MatrixHttpApi class as awaitables. https://github.com/Cadair/matrix_api_async and https://pypi.org/project/matrix-api-async/

These improvements will, in time, be merged back into the main Python SDK.

🔗Discord bridge 0.3.0 released

Half-Shot is working on so many bridges that these days he just casually mentions a huge release like Discord bridge v0.3.0. Some of the many many new features:

• #251 Support for Postgresql and a newer SQLite3 backend!
• #182 Replace npmlog with winston, for logging to files and better logging overall.
• #221 Add support for m.sticker.
• #210 Discord-side moderation of matrix users.
• #259 Show Matrix replies as Discord embeds.
• #164 Bot will now mention name, topic and membership changes on Discord.
• #175 Add special discord keys onto m.room.member for ghosts
• #179 Add rooms to the directory when bridged.
• #220 Messages are now deleted by a users puppet rather than the bot.

Shoutout to our new member of the team, Sorunome who did a lot of the review work behind the scenes for this release. Also, thank you to everyone who submitted a PR or an issue!

🔗matrix-puppet-slack v1.8.0 released

tom came out of the wild to tell us about matrix-puppet-slack:

TWIM (but actually over the last few weeks), thanks to the efforts of a new contributor, matrix-puppet-slack v1.7.0 and v1.8.0 have been released, fixing a number of old and new issues and adding support for new types of Slack events, including bidirectional @-mentions, Slack-to-Matrix typing notifications, fixing Slack-to-Matrix image/file uploads with comments, and more! The matrix-puppet-bridge projects have gotten relatively quiet over the past year or so, but there's still plenty of bridges with plenty of features to implement and plenty of bugs to fix (and, even more importantly, bugs to report!) for any would-be-contributors who'd like to use a trusted Matrix homeserver as their double-puppetting Slack/iMessage/Facebook/GroupMe/iChat/Skype/Hangouts/Signal/Tox client from which to brag to their friends on the other platforms about how great Matrix is.

The room for collaborating on matrix-puppet-bridge bridges is: #matrix-puppet-bridge:matrix.org

🔗maubot management UI progress

tulir has been making big progress on his maubot management UI:

The maubot management UI has progressed well, but isn't quite ready yet. I think it should be usable by Next Week in Matrix. It can be used to set up and configure a maubot instance and plugins. Once it's ready, it should be possible to do everything except installing maubot itself through the UI: installing and updating plugins, adding matrix clients, configuring plugin instances, viewing logs, etc.

🔗HK Bot is an anti-abuse bot

Coffee continues his streak of TWIM mentions by bringing a new bot to combat abuse:

HK Bot is an anti-abuse bot for public Matrix chatrooms. This is a bot that really shouldn't exist, but since some people just like to make others' lives more difficult, here we are. The purpose of this bot is twofold. It can automatically oversee rooms and respond to abuse, based on programmable pattern rules, providing a stop-gap measure in case no human moderators are nearby. It can also automate some tedious tasks via its command interface, the primary one being the complete redaction of all of an abuser's messages. HK Bot is still under construction and contains dangerous features. Use with caution.

🔗X-SAMPA to IPA bot

Dandellion "made an X-SAMPA to IPA bot based on matrix-bot-sdk and discord's conniebot". This bot essentially lets you use ASCII characters to get an output in IPA. The advantage of this is that you can much more easily type and transmit pronunciation (because you don't need to find the characters).

I'll just have to make it a bit more configurable, but then I'll throw the source up When making constructed languages and talking about linguistics, it's nice to get an easily readable IPA representation of a word, but it's really hard to write IPA, which is why x-sampa exists as a way to input IPA with a normal keyboard!

🔗Informo

vabd, the unknown organiser of Informo provided an update about the spec:

A handful of SCS (Specs Changes Submissions) to the Informo specifications have been happening over the last week and a half, with some of them still open to public review for at least a few days before being merged into the specs. The list of SCS open to review can be found here, and people can track new SCS and status changes to existing ones through our specs bot that's living in #discuss:weu.informo.network ?

🔗Fractal

Julian Sparber has been working on message rendering in Fractal:

I increased message rendering performance in the room history. Also the history doesn't move it's position anymore when older messages are loaded, which results in a much better experience. All the changes are in master, but we didn't make a new release.

🔗Video intro to Matrix and Riot in French

Nouts:

I have made a video about Riot which could fit in the Guides on matrix.org. It's for beginners, in French. It's 6 months old, but I just uploaded it on youtube. Also it's available on peertube.

🔗More Things This Week

ma1uta introduces a new client built on JavaFX: no code yet, but it's built on the previous Java Matrix work he's been doing. "It will be cross-platform (linux,windows,osx I hope) client with supports of the multi accounts."

Half-Shot also continued work on bridging via libpurple: "I've nearly got group chats working, with invites (and hopefully people joining and leaving them showing up properly) as well as user's having profiles."

Work continues on Koma:

Since the last TWIM update, koma is updated to kotlin 1.3, experimental coroutines are replaced with stable ones. The changes are being tested and should be merged soon. A new contributor has joined, so expect development to speed up a bit.

Spectral, a Matrix client built using libqmatrixclient, runs great on macOS. I'm excited because I get to play with another client! Find .dmgs on bintray.

🔗FIN

You made it, right to the end! Nice going! Come back here next week to find out what's been happening, or even better, come join us in #twim:matrix.org and tell us what YOU'VE been working on!

🔗The Matrix.org Foundation

This week, the first steps were taken in the creation of the Matrix Foundation! Read our blog post from earlier this week for more:

in preparation for the upcoming Matrix 1.0 release, we've been moving ahead with the rest of the open governance plan – and we're happy to announce that as of a few hours ago, the initial incarnation of The Matrix.org Foundation exists!
Watch this space over the coming weeks as we announce the Guardians and finish bootstrapping the Foundation into its final long-term form! Meanwhile, any questions: come ask in #matrix-spec-process:matrix.org

🔗Hello Matrix

Alex has managed the invaluable https://www.hello-matrix.net/public_servers.php for some years, but in two weeks time will be making some changes to the page:

Just wanted to let everyone know that changes are coming to the server list: I've put up a notice on the site that starting Nov 16th I will only show a curated list of servers I would recommend to join.
This reduces the workload for me quite a bit and avoids me becoming some kind of arbiter on what encompasses the Matrix universe… I think it is also more useful for users who are looking for a server to join.
And there is always matrixstats.org for those who are looking for a more complete-ish list of known homeservers.

However, if you have ideas on how to continue the project, or would like to step up and get involved in maintaining a list using data and tools from Hello Matrix, please contact me. Alex told me:

if you find someone willing to take up the project of a more automated, self-service and complete list at a later date, I am more than willing to hand over all the stuff I currently have and might also lend a helping hand myself (if I have time then).

🔗maubot management API

tulir has continued work on the revamped, now-Pythonic maubot, and has added a management API:

The maubot management API I mentioned last week is now mostly ready. It should be possible to use it to set up a maubot instance and plugins without filling the database manually. There's no UI yet though, so it still means curling manually.
The management API also supports the fancy plugin reloading stuff which was the > reason I rewrote maubot in python: You can POST an updated plugin to the API and it'll install it without having to restart.
I also made a bunch of plugins while working on the API that I used to test the API: a dice rolling/calculator bot, a bot that replies with the MXC URI of images you send it, a simple echo/ping bot and an xkcd bot

Next steps are making the management UI, a few more plugins and making setup and development instructions so that other people could run it and make plugins

🔗Announcing the Fractal Hackfest in Seville

Tobias Bernard officially announced the second ever Fractal hackfest:

there's going to be a Fractal hackfest in Seville in December https://blogs.gnome.org/tbernard/2018/10/26/fractal-hackfest-in-seville

🔗matrix-client-core

Coffee continues his streak of weekly updates by introducing us to matrix-client-core:

My lightweight bot/client framework, matrix-client-core, received its first tag ever. Version 0.0.1 is relatively stable, and lies on the doorstep of some refactoring work (ongoing) which should keep the master branch backwards compatible for now, but could make things less stable as I add new commits.

It turns out this isn't strictly a new project:

[it has] been there all along, quietly powering FAQBot and all of my bots. ? Maybe I have failed to explicitly indicate it as such up until now. (oops)

Always good news to see more bot-creation tooling!

🔗Synapse 0.33.8

Synapse 0.33.8 was released, and andrewsh made sure it was updated on Debian. Docker packages are also updated.

🔗mxisd v1.2.0-rc.1

Max has updates on mxisd, Identity Server for Matrix:

mxisd v1.2.0-rc.1 is out with support of all features for the Exec Identity Store, allowing connectivity to totally custom/arbitrary backends. Feedback is extremely welcome!

🔗Half-Shot interfaces with libpurple

It's a job that someone needed to do, and that someone was Half-Shot (who else?)

I've been reviving node-purple (a library for communicating with libpurple) and making a brand new bridge service to make use of it called matrix-appservice-purple. Today I got it to the point where you could link your XMPP account to your matrix user and have it bridge PMs over.
Work is ongoing to make it bridge group chats, profiles, contacts lists and support other protocols better in the coming weeks

🔗NSFW image detection API (on Matrix)

Black Hat is often found working on Spectral (previously 'Matrique'.) This week, he has been building @nsfw:encom.eu.org, which is a bot designed to give scores for how likely an image should be classified as NSFW. It's a simple mechanism, you give it an image, it gives you a JSON object with the result. For example:

My avatar returned less than 1% probability of being NSFW, which I was actually a little offended by.

To talk more about the bot and it's development, chat in #nsfw:encom.eu.org.

🔗Gaming Discord bridging

swedneck has been dutifully bridging channels from the Linux Gaming discord.

All but 3 channels on the linux gaming discord have now been bridged to matrix.

To find these channels, join the community at +linux-gaming:matrix.org and get chatting.

🔗Other thoughts

• kitsune is forced by his employers to use Viber, so is thinking of creating a Matrix bridge.
• I spent some time this week repeatedly installing Synapse, then working with Stefan to create a new, hopefully definitive installation guide (available soon). I can also personally recommend Slavi's matrix-docker-ansible-deploy project, this is a great way to get a Synapse installation (and more!) running.
• lately in #twim:matrix.org posters have been providing much clearer and atomic updates, which I like a lot

🔗Matrix Live, season 3 episode 2

We continue the new season of Matrix Live. This re-booted season has a slightly different format to previous: in each outing, there will be a single deep-dive topic. This week, Matthew and recent-Matrix-arrival Nad discuss UX for E2E encryption key handling. This is an unsurprisingly complex design question, both in terms of how it should behave and how it should look. Nad shared his latest thinking in a blog post earlier today, and you can watch the video below.

🔗Farewell

Enjoy your week. We'll see you back here next week, but if you are working on something using Matrix, come chat with us in #twim:matrix.org!

It's been a long-standing goal to enable end-to-end encryption by default for private communication in Matrix. The technical effort so far has included our libolm library, an independent cryptographic review and a massive backlog of feature development and bug fixes. Today, instead I'd like to focus on some of the User Experience challenges and goals we're facing.

I should also introduce myself—I'm Nad Chishtie ( @nadonomy:matrix.org) and I recently joined the Matrix core team (at New Vector) as Lead Designer, most recently focusing on end-to-end encryption.

When using encrypted messages, most existing services fall short in one or all of the following:

• They don't allow you to use multiple devices independently. For example, a web session might be locally tethered to a mobile device.
• They don't support a way to restore or temporarily access message history. For example, if you don't have physical access to your main device because it's broken or has been stolen.
• They don't allow you to verify that devices are controlled by their owners rather than eavesdroppers, and persist that trust across multiple devices, sessions or rooms.

(For some users, restrictions such as limiting the number of places encryption keys reside, and not having a synchronised message history may be desirable security features. We'll support these cases, but just not as the default behaviour.)

Let's dive in to some of the fundamental concepts we'll be putting forward to deliver a default end-to-end encryption experience that makes sense for most modern users. In this post we'll look at an overview of work-in-progress wireframes, in the spirit of designing in the open and gathering feedback from the wider Matrix community. Please note that these don't represent the actual interface design.

🔗Cross-signing personal devices

When logging in to a new device, you'll be able to use an existing device to verify your new one. Verification is done by scanning a QR code on whichever device has the most convenient camera to use, or by comparing a short text string. You only have to complete this process once to mutually verify both devices.

Verifying your new device by cross-signing transfers encryption keys, giving it access to your encrypted messages, and also signals to other users that the new device is trustworthy.

🔗Secure Message Recovery

To the end user, Secure Message Recovery works a lot like setting up disk encryption or a password manager. A user can optionally secure their message history using a recovery passphrase and/or key. If logged out, or using another device, the user can use the recovery passphrase or key to access their encrypted message history.

In practise, this incrementally encrypts and backs up encryption keys to a user's homeserver, kept secure by the homeserver never having access to the passphrase or key. Like cross-signing, using a recovery passphrase or key will also signal to other users that a device is trustworthy.

We think that in most cases users will cross-sign personal devices, but as a safety net (for example, if a user's devices are broken or lost) Secure Message Recovery is an invaluable tool for users to minimise the chance of them losing their encrypted message history.

🔗People should trust people

With both cross-signing and Secure Message Recovery in place, we think that people should trust people, instead of individual devices. Now, when you verify a device, it'll mark all of that users trusted devices as trusted.

Gone are the days of every person you talk to having to independently verify your new device upgrade. Like cross-signing, you can verify a device by scanning a QR code or comparing a short text string.

🔗Sensible and extensible

In Riot, we're implementing these features with a sensible default experience that strikes a balance between usability and security. We think most people would prefer to trust cross-signed devices, and that user trust shouldn't block encryption. However, if you aren't most people, you'll be free to configure whatever level of security you need.

🔗In Summary

With all of the above in place, and after resolving any remaining technical issues, users will be able to:

• Use end-to-end encryption by default in private rooms.
• Use an existing device or Secure Message Recovery to access their encrypted message history on multiple devices, and to signal device trust to other users.
• Access their encrypted message history using Secure Message Recovery, by storing encrypted message keys on their homeserver.
• Mark a user as trusted by verifying one of their devices, persisting across all rooms and devices.
• Keep their encrypted messages out of the hands of eavesdroppers.
• Opt out, or further configure if they have more specific security requirements.

Over the coming days we're polishing wireframes, nomenclature, iconography and microcopy as we dig deeper into development and implementation, as well as designing these features for the upcoming Riot redesign. Cryptography needn't be intimidating, and we're excited to iterate on these advanced features to make them work for everyone.

We'd love to hear your feedback! Let us know your thoughts here or in #e2e-dev:matrix.org.

Wowzers - our 8th dot release for v0.33!

This time we have a bunch of bug fixes and db performance improvements as well as better support for auto-join rooms and the ability for admins to limit who can create rooms aliases.

v0.33.8 also contains more python 3 fixes: we are running most of matrix.org on python 3 as of right now and seeing some pretty impressive performance improvements. Look out for Hawkowl's write up coming soon.

For those interested in what we are working on right now, take a look at our task board.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here.

🔗Synapse 0.33.8 changelog

🔗Features

• Servers with auto-join rooms will now automatically create those rooms when the first user registers (#3975)
• Add config option to control alias creation (#4051)
• The register_new_matrix_user script is now ported to Python 3. (#4085)
• Configure Docker image to listen on both ipv4 and ipv6. (#4089)

🔗Bugfixes

• Fix HTTP error response codes for federated group requests. (#3969)
• Fix issue where Python 3 users couldn't paginate /publicRooms (#4046)
• Fix URL previewing to work in Python 3.7 (#4050)
• synctl will use the right python executable to run worker processes (#4057)
• Manhole now works again on Python 3, instead of failing with a "couldn't match all kex parts" when connecting. (#4060, #4067)
• Fix some metrics being racy and causing exceptions when polled by Prometheus. (#4061)
• Fix bug which prevented email notifications from being sent unless an absolute path was given for email_templates. (#4068)
• Correctly account for cpu usage by background threads (#4074)
• Fix race condition where config defined reserved users were not being added to the monthly active user list prior to the homeserver reactor firing up (#4081)
• Fix bug which prevented backslashes being used in event field filters (#4083)

🔗Internal Changes

• Add information about the matrix-docker-ansible-deploy playbook (#3698)
• Add initial implementation of new state resolution algorithm (#3786)
• Reduce database load when fetching state groups (#4011)
• Various cleanups in the federation client code (#4031)
• Run the CircleCI builds in docker containers (#4041)
• Only colourise synctl output when attached to tty (#4049)
• Refactor room alias creation code (#4063)
• Make the Python scripts in the top-level scripts folders meet pep8 and pass flake8. (#4068)
• The README now contains example for the Caddy web server. Contributed by steamp0rt. (#4072)
• Add psutil as an explicit dependency (#4073)
• Clean up threading and logcontexts in pushers (#4075)
• Correctly manage logcontexts during startup to fix some "Unexpected logging context" warnings (#4076)
• Give some more things logcontexts (#4077)
• Clean up some bits of code which were flagged by the linter (#4082)

Hi all,

Back in June we blogged about the plan of action to establish a formal open governance system for the Matrix protocol: introducing both the idea of the Spec Core Team to act as the neutral technical custodian of the Matrix Spec, as well as confirming the plan to incorporate the Matrix.org Foundation to act as a neutral non-profit legal entity which can act as the legal Guardian for Matrix's intellectual property, gather donations to fund Matrix work, and be legally responsible for maintaining and evolving the spec in a manner which benefits the whole ecosystem without privileging any individual commercial concerns.  We published a draft proposal for the new governance model at MSC1318: a proposal for open governance of the Matrix.org Spec to gather feedback and to trial during the day-to-day development of the spec. Otherwise, we refocused on getting a 1.0 release of the Spec out the door, given there's not much point in having a fancy legal governance process to safeguard the evolution of the Spec when we don't even have a stable initial release!

We were originally aiming for end of August to publish a stable release of all Matrix APIs (and thus a so-called 1.0 of the overall standard) - and in the end we managed to publish stable releases of 4 of the 5 APIs (Client-Server, Application Service, Identity Service and Push APIs) as well as a major overhaul of the Server-Server (SS) API.  However, the SS API work has run on much longer than expected, as we've ended up both redesigning and needing to implement many major changes to to the protocol: the new State Resolution algorithm (State Resolution Reloaded) to fix state resets; versioned rooms (in order to upgrade to the new algorithm); changing event IDs to be hashes; and fixing a myriad federation bugs in Synapse.  Now, the remaining work is progressing steadily (you can see the progress over at https://github.com/orgs/matrix-org/projects/2 - although some of the cards are redacted because they refer to non-spec consulting work) - and the end is in sight!

So, in preparation for the upcoming Matrix 1.0 release, we've been moving ahead with the rest of the open governance plan - and we're happy to announce that as of a few hours ago, the initial incarnation of The Matrix.org Foundation exists!

Now, it's important to understand that this process is not finished - what we've done is to set up a solid initial basis for the Foundation in order to finish refining MSC1318 and turning it into the full Articles of Association of the Foundation (i.e. the legal framework which governs the remit of the Foundation), which we'll be working on over the coming weeks.

In practice, what this means is that in the first phase, today's Foundation gives us:

• A UK non-profit company - technically incorporated as a private company, limited by guarantee.
• Guardians, whose role is to be legally responsible for ensuring that the Foundation (and by extension the Spec Core Team) keeps on mission and neutrally protects the development of Matrix.  Matrix's Guardians form the Board of Directors of the Foundation, and will provide a 'checks and balances' mechanism between each other to ensure that all Guardians act in the best interests of the protocol and ecosystem.
  
  For the purposes of initially setting up the Foundation, the initial Guardians are Matthew & Amandine - but in the coming weeks we're expecting to appoint at least three independent Guardians in order to ensure that the current team form a minority on the board and ensure the neutrality of the Foundation relative to Matthew & Amandine's day jobs at New Vector.The profile we're looking for in Guardians are: folks who are independent of the commercial Matrix ecosystem (and especially independent from New Vector), and may even not be members of today's Matrix community, but who are deeply aligned with the mission of the project, and who are respected and trusted by the wider community to uphold the guiding principles of the Foundation and keep the other Guardians honest.
• An immutable asset lock, to protect the intellectual property of the Foundation and prevent it from ever being sold or transferred elsewhere.
• An immutable mission lock, which defines the Foundation's mission as a non-profit neutral guardian of the Matrix standard, with an initial formal goal of finalising the open governance process.  To quote article 4 from the initial Articles of Association:
  • 4. The objects of the Foundation are for the benefit of the community as a whole to:

    4.1.1  empower users to control their communication data and have freedom over their communications infrastructure by creating, maintaining and promoting Matrix as an openly standardised secure decentralised communication protocol and network, open to all, and available to the public for no charge;

    4.1.2  build and develop an appropriate governance model for Matrix through the Foundation, in order to drive the adoption of Matrix as a single global federation, an open standard unencumbered from any proprietary intellectual property and/or software patents, minimising fragmentation (whilst encouraging experimentation), maximising speed of development, and prioritising the long-term success and growth of the overall network over the commercial concerns of an individual person or persons.
• You can read the initial Articles of Association here (although all the rest of it is fairly generic legal boilerplate for a non-profit company at this point which hasn't yet been tuned; the Matrix-specific stuff is Article 4 as quoted above).  You can also see the initial details of the Foundation straight from the horse's mouth over at https://beta.companieshouse.gov.uk/company/11648710.

• Appoint 3+ more Guardians (see above).
• Finalise MSC1318 and incorporate the appropriate bits into the Articles of Associations (AoA).  (We might literally edit MSC1318 directly into the final AoA, to incorporate as much input as possible from the full community)
• Tune the boilerplate bits of the AoA to incorporate the conclusions of MSC1318.
• Register the Foundation as a Community Interest Company, to further anchor the Foundation as being for the benefit of the wider community.
• Perform an Asset Transfer of any and all Matrix.org property from New Vector to the Foundation (especially the Matrix.org domain and branding, and donations directed to Matrix.org).

thanks,

Matthew, Amandine, and the forthcoming Guardians of [the] Matrix!