Matrix 1.0 - Are We Ready Yet?

15.03.2019 00:00 — GeneralNeil Johnson

TL;DR

  • If you run a Synapse ensure that your federation certificates are valid here.
  • If they are not valid check out the FAQ.
  • Follow along with progress at https://arewereadyyet.com
  • Tell all your admin friends.
Folks, as you know we are now very close to achieving Matrix 1.0 and finally being in a position to shed our ‘beta' tag. It has been a long time coming and speaks to the huge effort from hundreds of people over the past 5 years.

A critical step towards this goal is the release of Synapse 1.0. We want to ship Synapse 1.0 as soon as possible but can't do so without your help!

We'd like to introduce AreWeReadyYet.com - a quick and easy way for everyone to track the progress and check if their federation is ready for Matrix 1.0!!

Are we ready yet?

Synapse 1.0 is good news for anyone running a Synapse installation - it contains critical bug fixes, security patches, a new room algorithm version and dramatically improved user and room search. However, as part of the security work it also contains a breaking change from previous Synapse versions. From 1.0 onwards it will necessary to ensure a valid TLS certificate on the federation API. Self signed certificates will no longer be accepted. Why would we do such a thing?

In anticipation for this, everyone currently running a homeserver must ensure that they have checked their federation certificate (check yours here). Failure to do so will mean being unable to federate with any Matrix 1.0 compliant server. If your server fails the check, our FAQ has all the details on what you need to do.

This post is a call to arms to try and get as many admins to upgrade their certificates as possible. We are tracking adoption at https://arewereadyyet.com - currently this sits at about 55% - we need this figure to be higher before we can pull the lever.  

So what are you waiting for? Check that your server has valid certs - then tell all your admin pals to do the same. Friends don't let friends miss out on Synapse 1.0, send them to arewereadyyet.com (or tweet here to remind them!) We really need the community to help us here because at some point soon, we will need to pull the lever and release.

Once we make more progress on adoption, we will announce an official release date and finally get Synapse out of beta!

This Week in Matrix 2019-03-15

15.03.2019 00:00 — This Week in MatrixBen Parsons

🔗Matrix Live S3E17: Hello Ryan

This week you're stuck with me, but I'm chatting to Ryan, who works on Riot web. Having previously worked at Mozilla, Ryan has a LOT of interesting things to say about Firefox, the browser market, the importance of decentralisation, Matrix being GREAT, and more.

🔗arewereadyyet.com: Matrix 1.0-ready Federation

https://arewereadyyet.com is not just an amazing domain we somehow managed to buy, it's also the first place to look for progress towards full Matrix 1.0 readiness. Read more details at https://matrix.org/blog/2019/03/15/matrix-1-0-https-arewereadyyet-com/.

🔗Spec

Lots of activity but most notably MSC 1915 (a blocker for Matrix 1.0) is making progress. https://github.com/matrix-org/matrix-doc/pull/1915

🔗Synapse . . . 1.0?

It's all about 1.0 for the Synapse gang this week. This means performance improvements across the board in the form of read receipt batching, user directory (room directory coming soon!) and rate limiting on log in and registration APIs.

Brendan shipped the low bandwidth CoAP proxy we demo'd at FOSDEM

As well as a bunch of spec implementation projects to ensure that Synapse (and Sydent) are ready for Synapse 1.0.

🔗Synapse workers projects

Turn-of-phrase of the week from Half-Shot ("make it more performant and less crashy"):

synapse-netcore-workers is progressing as strongly as ever. This week has mainly been supporting a couple of users trying to use the fed sender, and also trying to make it more performant and less crashy. I've been using it solidly for two weeks now, and by and large it's been working nicely :)

Black Hat has been working on Cortex, a similar project in Rust:

Support for replication protocol in Cortex is mostly complete. A federation sender worker implementation is being worked on. If anyone is interested or wants to contribute, please go to #cortex:encom.eu.org

🔗Construct

Lots of progress this week. I'd like to give a special thanks to Yan Minari for another great week of testing, bugfixes and feature requests.

  • We implemented features related to room directory lists, reporting content, ignoring users, VoIP turnServer, prev_content in state events.
  • Improving SSL: allowing configurable lists of ciphers, and sending/receiving SNI.
  • At the lower level: adding support for Linux AIO features that are present in newer kernels, giving a nice performance boost.

For more Construct news watch the repo or join #zemos-test:matrix.org

🔗New Client for Android: Pattle

Wilko announced Pattle, a Matrix client for Android:

Hey all, I've been working on my Matrix Android app, Pattle! The goal of Pattle is to be an easy to use app for Matrix, with it's design inspired by popular apps such as WhatsApp and Telegram. Development happens here, and contributions are encouraged! The app is not currently suited for daily use, but some functionality is there, such as registering, logging in and viewing chats. Currently it's an Android only app using the official Matrix Android SDK, but the plan is to support iOS and web too, in the future. There's also a sort of design document available, stating how Pattle differs from standard Matrix apps and what it's goals are. The intent of the design document is to make development easier later on for other platforms

🔗Quaternion 0.0.9.4 beta 2 available

kitsune:

Quaternion 0.0.9.4 beta 2 (too many numbers? That too shall pass) is out, with bugfixes and translation updates. Notably, Quaternion won't crash on upgraded rooms in some cases, and won't cry in #gsoc:matrix.org and other v3 rooms. Translators are still strongly encouraged to push forward - due to all the features and fixes, there are many untranslated strings across the board! Also, some bugfixes are still in order before we can call the release RC, and some of them are really easy - so if you'd like to contribute, it's a great time to start!

🔗Riot Web

  • Preparing for 1.0.4 release with lots of small polish fixes
  • Planning our roadmap for the next few quarters

🔗Riot iOS

iOS released 2 times. Last release was to fix an issue with invalid scalar token. Review of one PR from the community for iOS10 notifications. Started implementing device verification with emoji.

🔗Riot Android

We've released v0.8.25 on Thursday, containing refresh of invalid scalar token, and some bugfixes. Links on m.notice messages are now clickable again.
Started implementing device verification with emoji.

🔗RiotX (Android)

  • We started to setup build tools and CI configuration.
  • The timeline scroll issues are addressed.

🔗FluffyChat 11 released

Ubuntu Touch fans rejoiced this week as a new version of FluffyChat was made available: https://open-store.io/app/fluffychat.christianpauly

  • Improved stability and performance
  • New translations - thanks to all translators
  • Design improvements to make FluffyChat look more like a common Ubuntu Touch app
  • Better tablet support
  • Lots of Bugfixes which caused a critical error
  • Fixed the automatic scrolling up to the last seen message
  • Fixed disabling account

🔗miniVector v0.8.25

miniVector is a minified Android Matrix client derived from Riot Android. v0.8.25 keeps it up to date with Riot Android.

🔗Matrix Ruby SDK

Ananace is returning his attention to the Matrix Ruby SDK:

I just realized that I haven't had lazy loading activated by default in the Ruby SDK, despite having had lazy loading code in place since ages back, so now that's going to be the default value going forward.

Further:

I started hacking - just a couple of hours or so ago - on a notification sender for The Foreman, a server orchestration system. So it can forward notifications to Matrix. https://github.com/ananace/foreman_notification_send

If you're using, or considering using, The Foreman for orchestration this may be very useful.

🔗maubot

tulir:

maubot plugins can now add their own endpoints to the main maubot webserver. The pingstat plugin I announced last week uses this feature for the leaderboard widget: https://maubot.xyz/_matrix/maubot/plugin/pingstat/stats?room_id=!MbRaSiMIRhhxDtJENL:maunium.net

🔗matrix-appservice-bridge 1.8.0, matrix-appservice-irc 0.12.0-rc1: room upgrades and more

Half-Shot:

matrix-appservice-bridge got a 1.8.0 release last night, featuring automatic handling of room upgrades for all your room upgrade needs. Providing your bridge uses the RoomStore as designed, it's literally a few lines to enable :). Changelog here

What is this? A matrix-appservice-irc release? No, it's a release candidate. Announcing that 0.12.0-rc1 is now out and about for folks to play with. More IRC updates to come in the future :)

🔗mautrix-whatsapp

tulir:

mautrix-whatsapp now has support for postgres databases thanks to a pull request by @Rennerdo30

🔗mautrix-telegram

tulir:

mautrix-telegram got a pull request with a pytest setup and a few unit tests by @V02460. I haven't merged that one yet, but will soon.

🔗matrix-docker-ansible-deploy supports Dimension

This is really neat, and a shout-out that these playbooks are my preferred way to install Synapse and other components. Slavi:

Thanks to NullIsNot0, matrix-docker-ansible-deploy can now (optionally) install the Dimension Integration Manager.
The playbook can now also assist with serving the base domain from the Matrix server, thus making it easier for some folks to set up well-known Server-Delegation.

🔗Self-hosting guide for NixOS

Florian:

I wrote a Matrix self-hosting guide for the NixOS operating system: Check its progress and give it a test if you're a fellow NixOS user!

🔗banhammer: new bot for applying a ban policy

jcgruenhage reports that:

I wrote a small bot that takes a kick/ban policy from room state from all rooms it's a member of and tries to enact that policy. In practice that means it applies a regex to all MXIDs and tries to kick/ban them based on that. It's been a request of TravisR , source code is available at https://gitlab.com/jcgruenhage/banhammer, documentation is still lacking but will hopefully soon be added

🔗matrix-wug support for Navajo script

Continuing his theme of Native American languages, Dandellion added Navajo script to matrix-wug.

🔗FreedomBox Tutorial: Setting Up a Chat Server with Matrix and Riot

It's been possible for some time to set up a Matrix server with FreedomBox, and they just recently released a video tutorial showing the process.

🔗Around the Internets

There were a whole lot of outages for the big names this week - good time to think about self-hosting?

🔗That's all I know

So that's all I have to say! I hope you enjoyed this edition of This Week in Matrix, and whether you did or you didn't, I'd love too hear from you in #twim:matrix.org. If you have Matrix news to share, that's the place to come and do so!

Breaking the 100bps barrier with Matrix, meshsim & coap-proxy

12.03.2019 00:00 — In the NewsMatthew Hodgson

Hi all,

Last month at FOSDEM 2019 we gave a talk about a new experimental ultra-low-bandwidth transport for Matrix which swaps our baseline HTTPS+JSON transport for a custom one built on CoAP+CBOR+Noise+Flate+UDP.  (CoAP is the RPC protocol; CBOR is the encoding; Noise powers the transport layer encryption; Flate compresses everything uses predefined compression maps).

The challenge here was to see if we could demonstrate Matrix working usably over networks running at around 100 bits per second of throughput (where it'd take 2 minutes to send a typical 1500 byte ethernet packet!!) and very high latencies.  You can see the original FOSDEM talk below, or check out the slides here.

Now, it's taken us a little while to find time to tidy up the stuff we demo'd in the talk to be (relatively) suitable for public consumption, but we're happy to finally release the four projects which powered the demo:

In order to get up and running, the meshsim README has all the details.

It's important to understand that this is very much a proof of concept, and shouldn't be used in production yet, and almost certainly has some glaring bugs.  In fact, it currently assumes you are running on a trusted private network rather than the public Matrix network in order to get away with some of the bandwidth optimisations performed - see coap-proxy's Limitations section for details.  Particularly, please note that the encryption is homemade and not audited or fully reviewed or tested yet.  Also, we've released the code for the low-bandwidth transport, but we haven't released the "fan-out routing" implementation for Synapse as it needs a rethink to be applicable to the public Matrix network.  You'll also want to run Riot/Web in low-bandwidth mode if you really wind down the bandwidth (suppressing avatars, read receipts, typing notifs and presence to avoid wasting precious bandwidth).

We also don't have an MSC for the CoAP-based transport yet, mainly due to lack of time whilst wanting to ensure the limitations are addressed first before we propose it as a formal alternative Matrix transport.  (We also first need to define negotiation mechanisms for entirely alternative CS & SS transports!).  However, the quick overview is:

  • JSON is converted directly into CBOR (with a few substitutions made to shrink common patterns down)
  • HTTP is converted directly into CoAP (mapping the verbose API endpoints down to single-byte endpoints)
  • TLS is swapped out for Noise Pipes (XX + IK noise handshakes).  This gives us 1RTT setup (XX) for the first connection to a host, and 0RTT (IK) for all subsequent connections, and provides trust-on-first-use semantics when connecting to a server.  You can see the Noise state machine we maintain in go-coap's noise.go.
  • The CoAP headers are hoisted up above the Noise payload, letting us use them for framing the noise pipes without having duplicated framing headers at the CoAP & Noise layers.  We also frame the Noise handshake packets as CoAP with custom message types (250, 251 and 252).  We might be better off using OSCORE for this, however, rather than hand-wrapping a custom encrypted transport...
  • The CoAP payload is compressed via Flate using preshared compression tables derived from compressing large chunks of representative Matrix traffic. This could be significantly improved in future with streaming compression and dynamic tables (albeit seeded from a common set of tables).
The end result is that you end up taking about 90 bytes (including ethernet headers!) to send a typical Matrix message (and about 70 bytes to receive the acknowledgement).  This breaks down as as:
  • 14 bytes of Ethernet headers
  • 20 bytes of IP headers
  • 8 bytes of UDP headers
  • 16 bytes of Noise AEAD
  • 6 bytes of CoAP headers
  • ~26 bytes of compressed and encrypted CBOR
The Noise handshake on connection setup would take an additional 128 bytes (4x 32 byte Curve25519 DH values), either spread over 1RTT for initial setup or 0RTT for subsequent setups.

At 100bps, 90 bytes takes 90*8/100 = 7.2s to send... which is just about usable in an extreme life and death situation where you can only get 100bps of connectivity (e.g. someone at the bottom of a ravine trying to trickle data over one bar of GPRS to the emergency services).  In practice, on a custom network, you could ditch the Ethernet and UDP/IP headers if on a point-to-point link for CS API, and ditch the encryption if the network physical layer was trusted - at which point we're talking ~32 bytes per request (2.5s to send at 100bps).  Then, there's still a whole wave of additional work that could be investigated, including...

  • Smarter streaming compression (so that if a user says 'Hello?' three times in a row, the 2nd and 3rd messages are just references to the first pattern)
  • Hoisting Matrix transaction IDs up to the CoAP layer (reusing the CoAP msgId+token rather than passing around new Matrix transaction IDs, at the expense of requiring one Matrix txn per request)
  • Switching to CoAP OBSERVE for receiving data from the server (currently we long-poll /sync to receive data)
  • Switching access_tokens for PSKs or similar
...all of which could shrink the payload down even further.  That said, even in its current state, it's a massive improvement - roughly ~65x better than the equivalent HTTPS+JSON traffic.

In practice, further work on low-bandwidth Matrix is dependent on finding a sponsor who's willing to fund the team to focus on this, as otherwise it's hard to justify spending time here in addition to all the less exotic business-as-usual Matrix work that we need to keep the core of Matrix evolving (finishing 1.0, finishing E2E encryption, speeding up Synapse, finishing Dendrite, rewriting Riot/Android etc).  However, the benefits here should be pretty obvious: massively reduced bandwidth and battery-life; resilience to catastrophic network conditions; faster sync times; and even a protocol suitable for push notifications (Matrix as e2e encrypted, decentralised, push!).  If you're interested in supporting this work, please contact support at matrix.org.

This Week in Matrix 2019-03-08

08.03.2019 00:00 — This Week in MatrixBen Parsons

🔗AreWeReadyYet.com?

From Neil of the Synapse-team

Folks, in the run up to Synapse 1.0, if you are running your own homeserver now would be an excellent time to check that your TLS certificates are up to date. Point your server name at https://matrix.org/federationtester/ and if there are errors check our handy FAQ on how to fix it. If you do not have valid TLS certificates Synapse 1.0 will refuse to federate with you.

benpa has put together a federation checker to quantify how many homeservers are 1.0 ready - https://www.arewereadyyet.com/ - It currently stands at 50.5% let's try and get that to 60% over the weekend.

Aside from all that, the team have been working on preparing for Synapse 1.0, you can track our progress here. We promise not to just land 1.0 out of the blue - we'll give everyone a 2 week warning to give stragglers a chance to get their certificates in order.

And this week we have Neil and Erik talking about this in more detail on Matrix Live

ma1uta has been working on Jeon - Java interfaces to the various Matrix APIs - and is now getting ready to start work on a homeserver. He was previously asking for a name for this project, but might now have settled on "JeonServer".

First Release Candidate of the Jeon Project with upcoming Client-Server API 0.5, Server-Server API 0.1.1, Application API 0.1, Identity API 0.2 and Push API 0.1. Also the RC of the jmsdk has been prepared with Java Matrix Client for Client-Server API 0.5. Changes in the C2S: Added the m.push_rules event, removed presence list methods and other minor fixes. Added S2S API of the 0.1.1 version. I prepared a very simple page https://ma1uta.github.io/ with links to the swagger schemas (json and yaml) for all Matrix API which generated from the Jeon code.

Chat more in #jeon:matrix.org or #jeon:ru-matrix.org.

🔗Construct updates

There's a lot of progress, a few endpoints and features have been implemented this week such as Room Tags and all of the spec features for /createRoom. Most of the progress has been with testing and bugfixes thanks to Yan Minari, and tulir and mujx. We've fixed several interactions with synapse such as invite accept/deny and synapse's ability to join and leave construct created rooms without any issues. Lastly and most important, we've generated an official issues list thanks again to our star tester Yan Minari available here https://github.com/matrix-construct/construct/issues

For more new Construct news watch the repo or join #zemos-test:matrix.org

🔗OpenSAPS: Open Slack API Server

pztrn has created a new mechanism for relaying apps that use Slack webhooks into Matrix:

To everyone who wonders how to connect his application to Matrix (at least for notifications of some kind) - use OpenSAPS! It just reached v0.1.0. OpenSAPS stands for Open Slack API Server and able to retransmit messages from applications (like Gitlab or everything that can send data to Slack) to somewhere else. Right now these "somewhere else" is a Telegram (with HTTP proxy support) and Matrix! Written in Golang to ensure minimal memory footprint. Take a look at https://gitlab.com/pztrn/opensaps Tested with Gitlab and Gitea but should work with almost any service. Join #opensaps:pztrn.name to talk with developers or get help. BTW, there is OpenSAPS instance in our room that transmits everything from gitlab.com into room! (almost) immediately after 0.1.0 comes 0.1.1, with fixed URLs parsing and fixed inability to login into servers which use .well-known for delegation. It should work [with other webhooks]. If something strange happens there is also a possibility to write own parser to make everything work :)‚ Tested with Gitlab and Gitea ATM. Share application names that work, I'll start to make a list of them. :D

🔗matrix-puppet-bridge: matrix-puppet-slack and matrix-puppet-hangouts updates

tom reports from #matrix-puppet-bridge:matrix.org:

🔗MiniVector update

bubu let us know about MiniVector updates

Minivector, a minimalistic fork of riot-android had a new release last week, getting rid of a few more unused dependencies. This brings the final apk size down to 13mb vs riot android's 25mb. This work was done by @hrjet:matrix.org. The project room is here: #miniVectorAndroid:matrix.org

🔗matrix-docker-ansible-deploy, now with Discord and email templates

The volume of discussion about installing/configuring Synapse and other Matrix-related components is like a subculture in itself. Standing tall within this is Slavi's matrix-docker-ansible-deploy collection of Ansible playbooks. They're a great way to quickly and reliably get a Synapse instance running.

Thanks to Lionstiger, matrix-docker-ansible-deploy now supports Discord bridging via matrix-appservice-discord.
Those wishing to give it a try can refer to the Setting up Appservice Discord bridging documentation page.

Thanks to Sylvia van Os, matrix-docker-ansible-deploy now has the Ability to customize mxisd's email templates

🔗Quaternion 0.0.9.4 beta

kitsune:

between chasing bugs in Quaternion 0.0.9.4 beta (translators, your help is hugely needed to catch up with new and updated strings) there happens almost literal bikeshedding in #qmatrixclient:matrix.org, under an excuse of discussing The Universal Algorithm to Colour Usernames. Join the fun!

🔗Bifrost bridge gatewaying

Half-Shot:

Bifrost is now starting to comfortably support gatewaying. For those that don't know, gateways allow a remote user to participate in the matrix network without prior bridging, it's very fancy. The latest changes are that XMPP clients can now ask for the public room list by querying the bridge component. There is a video on this using the Yaxim XMPP client on Android (Credit to Ge0rG). Come chat with us in #bifrost:half-shot.uk

You can check out the video of the bridge in operation here:

🔗Spectral development restart

Black Hat:

Spectral development now continues after a short break. The new "material design" has been merged into master, along with a few bug fixes. I plan to work on Kirigami port in the following two weeks.

🔗Fractal

Alexandre Franke reports on Fractal:

Since our last update about Fractal we landed typing notifications, found out we actually weren't logging anything, recovered the ability to log, and restored spellcheck support.
We are also now tagging uploaded audio files as such and adding thumbnails to image files, courtesy of our Outreachy intern, and we fixed a bunch of other things.

🔗#ping:maunium.net

tulir making crazy maubots this week:

I made a room for !pinging echo maubots: #ping:maunium.net. The room has a bunch of echo bots, currently on maunium.net, c.mau.dev and matrix.vgorcum.com. It also has a new maubot plugin called pingstat, which collects the pong data and makes a leaderboard website. The website is linked to the room as a widget.

Check out instructions to get your own maubots setup here: https://github.com/maubot/maubot/wiki/Setup

🔗synapse-netcore-workers

Half-Shot:

Not much news to report on the netcore-workers other than it's maturing and we have nice things now like logging, metrics and a docker image you can run at home. I'm running the federation sender fulltime on half-shot.uk to dogfood it and will announce when I think it's good for general consumption :)

Related: Black Hat is investigating a similar project in Rust. Anyone interested in that please do go chat, and take a look at the repo they've created.

🔗matrix-wug, IPA rendering bot, gets support for cherokee

Dandellion on matrix-wug:

This is "script" or written language has a very interesting story behind it, where the creator of it actually couldn't read and write. Regardless, he wanted to write down his language, and developed his own writing system.
The characters look a lot like latin characters because he tried to imitate the characters of a bible. Just a fun history lesson!

🔗weechat-matrix

Paul created a dockerfile to install weechat-matrix:

since weechat-matrix is a bit involved to install, I create a dockerfile that takes care of it: https://github.com/poljar/weechat-matrix/pull/60 . And people allergic to docker can look inside to check the steps (just reading the readme doesn't give a full picture)

🔗modular.im

Modular.im is a Matrix-as-a-SaaS offering. It's suitable for users who want the benefits of having their own Matrix homeserver, but don't want to host and run one.

There are some new announcements this week:

Today we are pleased to announce that you can now customise your Modular hosted Riot at the touch of a button, through the Modular host admin interface. Better yet, this is available to all Modular customers at no additional cost.
Today we are also launching a proof of concept for purchasing additional content via the Modular integration manager.
For the initial experiment we're offering a set of "snazzy", limited edition Matrix stickers for the princely sum of $0.50. These are digital versions of the Matrix and Riot hexagons that some of you may have seen in real life.

🔗Still more!

decred.org are an organisation concerned with blockchain technology, and they also use Matrix for their communications! They now have a stickerpack available in Riot, so if you'd like to use their stickers you can add the pack and get going with blockchain-related memes.

If you have a stickerpack you'd like to see included, please let me know.

Celebrating the creation of a new homeserver instance at kde.org, Jonathan Riddell has written an article.

sortebill created #drums:matrix.org:

I've made a room for anyone interested in drums and percussion. Acoustic, Electronic, Played or Programmed, anything goes as long as it's rhytm related. The room is focused on both playing/programming drums and equipment.

There was a somewhat longstanding bug that affected a number of users: often manifesting as unexpected and frustrating "Unable to decrypt" messages - this has now been fixed, take a look at https://github.com/vector-im/riot-web/issues/6838

🔗They think it's all over...

It is now! If you'd like to have your project included in this post, come chat to us in #twim:matrix.org!

Experiments with payments over Matrix

07.03.2019 00:00 — In the NewsMatthew Hodgson

Hi all,

Heads up that Modular.im (the paid hosting Matrix service provided by New Vector, the company who employs much of the Matrix core team) launched a pilot today for paid Matrix integrations in the form of paid sticker packs.  Yes kids, it's true - for only $0.50 you can slap Matrix and Riot hex stickers all over your chatrooms. It's a toy example to test the payments infrastructure and demonstrate the concept - the proceeds go towards funding development work on Matrix.org :)   You can read more about over on Modular's blog.

We wanted to elaborate on this a bit from the Matrix.org perspective, specifically:

  • We are categorically not baking payments or financial incentives as a first class citizen into Matrix, and we're not going to start moving stuff behind paywalls or similar.
  • This demo is a proof-of-concept to illustrate how folks could do this sort of thing in general in Matrix - it's not a serious product in and of itself.
  • What it shows is that an Integration Manager like Modular can be used as a way to charge for services in Matrix - whether that's digital content within an integration, or bots/bridges/etc.
  • While Modular today gathers payments via credit-card (Stripe), it could certainly support other mechanisms (e.g. cryptocurrencies) in future.
  • The idea in future is for Modular to provide this as a mechanism that anyone can use to charge for content on Matrix - e.g. if you have your own sticker pack and want to sell it to people, you'll be able to upload it and charge people for it.
Meanwhile, there's a lot of interesting stuff on the horizon with integration managers in general - see MSC1236 and an upcoming MSC from TravisR (based around https://github.com/matrix-org/matrix-doc/issues/1286) proposing new integration capabilities.  We're also hoping to implement inline widgets soon (e.g. chatbot buttons for voting and other semantic behaviour) which should make widgets even more interesting!

So, feel free to go stick some hex stickers on your rooms if you like and help test this out.  In future there should be more useful things available :)

Synapse 0.99.2 released!

04.03.2019 00:00 — ReleasesNeil Johnson

Well now, what have we here? Synapse 0.99.2 is the latest in the 0.99.x series as we step ever closer to 1.0.

0.99.2 is an incremental release including a bunch of performance improvements, enhancements to room upgrades and generally a plethora of bug fixes.

The most important thing that admins should know is that prior to 1.0 landing later this month, it is essential that the federation API has a valid TLS certificate - self signed certificates will no longer be accepted. For more details see our handy guide. Failure to do this will result in being unable to federate with other 1.0 servers.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.

🔗Synapse 0.99.2 changelog

🔗Features

  • Added an HAProxy example in the reverse proxy documentation. Contributed by Benoît S. (“Benpro”). (#4541)
  • Add basic optional sentry integration. (#4632#4694)
  • Transfer bans on room upgrade. (#4642)
  • Add configurable room list publishing rules. (#4647)
  • Support .well-known delegation when issuing certificates through ACME. (#4652)
  • Allow registration and login to be handled by a worker instance. (#4666#4670#4682)
  • Reduce the overhead of creating outbound federation connections over TLS by caching the TLS client options. (#4674)
  • Add prometheus metrics for number of outgoing EDUs, by type. (#4695)
  • Return correct error code when inviting a remote user to a room whose homeserver does not support the room version. (#4721)
  • Prevent showing rooms to other servers that were set to not federate. (#4746)

🔗Bugfixes

  • Fix possible exception when paginating. (#4263)
  • The dependency checker now correctly reports a version mismatch for optional dependencies, instead of reporting the dependency missing. (#4450)
  • Set CORS headers on .well-known requests. (#4651)
  • Fix kicking guest users on guest access revocation in worker mode. (#4667)
  • Fix an issue in the database migration script where the e2e_room_keys.is_verified column wasn't considered as a boolean. (#4680)
  • Fix TaskStopped exceptions in logs when outbound requests time out. (#4690)
  • Fix ACME config for python 2. (#4717)
  • Fix paginating over federation persisting incorrect state. (#4718)

🔗Internal Changes

  • Run black to reformat user directory code. (#4635)
  • Reduce number of exceptions we log. (#4643#4668)
  • Introduce upsert batching functionality in the database layer. (#4644)
  • Fix various spelling mistakes. (#4657)
  • Cleanup request exception logging. (#4669#4737#4738)
  • Improve replication performance by reducing cache invalidation traffic. (#4671#4715#4748)
  • Test against Postgres 9.5 as well as 9.4. (#4676)
  • Run unit tests against python 3.7. (#4677)
  • Attempt to clarify installation instructions/config. (#4681)
  • Clean up gitignores. (#4688)
  • Minor tweaks to acme docs. (#4689)
  • Improve the logging in the pusher process. (#4691)
  • Better checks on newsfragments. (#4698#4750)
  • Avoid some redundant work when processing read receipts. (#4706)
  • Run push_receipts_to_remotes as background job. (#4707)
  • Add prometheus metrics for number of badge update pushes. (#4709)
  • Reduce pusher logging on startup (#4716)
  • Don't log exceptions when failing to fetch remote server keys. (#4722)
  • Correctly proxy exception in frontend_proxy worker. (#4723)
  • Add database version to phonehome stats. (#4753)

This Week in Matrix 2019-03-01

01.03.2019 00:00 — This Week in MatrixNeil Johnson

No Ben this week, apparently he is allowed to go on holiday from time to time. Also no Matrix Live because we are terrible people - bring back Ben, that's what I say.

🔗Quaternion

kitsune:

Quaternion (master branch, and upcoming v0.0.9.4) can now open rooms by their aliases or ids upon pressing Ctrl+O, as long as those rooms are already in your room list (opening arbitrary public rooms will come in later versions). You can even paste matrix.to URIs for users (will open direct chat) and rooms in the same dialog. Navigation to known rooms inside Quaternion also works.

libQMatrixClient 0.5 is out - with members lazy-loading, room versions and upgrades and much more: https://github.com/QMatrixClient/libqmatrixclient/releases/tag/0.5.0

Which is, err, pretty amazing actually. As far as I know this is the only independent SDK supporting lazy loading and room upgrades. Congratulations.

🔗weechat-matrix

Poljar has been hard at work on weechat-matrix

Matthew reports:-

weechat-matrix's e2e support is really impressive (via matrix-nio and python-olm). It can only read rather than send right now, but otherwise looks to be massively on the right track. It even does fingerprint-based verification!

🔗Synapse

We shipped 0.99.2 this week, it's a point release containing all the usual bug fixes and perf improvements. We have also been taking a look at our docs and trying to improve where we can.

Hawkowl has spent some time improving CI so that we don't get queued up for hours waiting for builds (woo).

Admins - your weekly reminder that if you've not already done so, you must ensure the TLS certificate on your federation endpoints is no longer self signed - see our handy guide for all the details.

🔗matrix-puppet-bridge

tom:

matrix-puppet-bridge v1.17.0 has been released! This release:

Matrix rooms representing remote rooms being joinable by anyone who knows the room ID (which is generated, at least in part, from the remote room ID in all the matrix-puppet-bridge applications) was a big deficiency, and it's finally resolved. It wasn't known whether or not we could do this, and have our ghost users still be able to join the rooms (they need to be invited instead of just joining themselves), until it was attempted and tested in a few of the bridge applications.
Because it is a big deal, new minor versions of matrix-puppet-slackmatrix-puppet-facebookmatrix-puppet-hangoutsmatrix-puppet-signalmatrix-puppet-imessage and matrix-puppet-groupme, bumping the matrix-puppet-bridge version to 1.17.0, have been released.

🔗Riot iOS

  • Support .well-known data from login response
  • Create right permalinks for room v3
  • Fix builder to Xcode 10.1 in order to still be able to release
  • Update swift version. Add swiftLint

🔗matrix-media-repo

TravisR reports that:

matrix-media-repo has alpha-quality support for s3 (and s3-like services) on the travis/s3 branch. Intrepid testers are encouraged to give it a shot, and report bugs. Caution: may upload your cat.

🔗Scylla

It's been a little while since we saw any news about Scylla, a web-based client written in Elm. VaNilLa said:

Scylla, the Elm-based Matrix client, has received a few improvements. Included in this update:

  • Files that are uploaded now keep their original name (as opposed to being called File)
  • Sent messages are displayed right away, instead of when they are delivered to the server
  • Usernames are only shown once per several consecutive messages, to avoid clutter Some CSS changes have been made to keep the interface looking clean
Almost forgot! You can find Scylla's source code here and a live instance here!

🔗Riot Android

  • Implementation of .well-known support (SDK and Riot)
  • Minor change on some colors of the themes (link, home badges)
  • Many issue will be fixed regarding linkification
  • KeysBackup: improvement on recovery process: importing keys step is 8 times faster, and user get more feedback during the process which can take several seconds
  • We will prepare a new release for the beginning of next week.
  • PlayStore new descriptions have been updated for the following languages: Bulgarian, German, English (US), French, Hungarian, Russian and Chinese (Taiwan).

🔗RiotX (Android)

  • Timeline:
    • HTML formatting
    • Implementation of pills (need optimization)
  • Many Github issues have been created to track parity with the Riot Android

🔗Riotic: new fork with updates

Aaron Raimist has made some updates to Riotic, which was a good chance for me to revisit it. It works nicely and is a great alternative to the Electron version of Riot. I also like being able to use https://riot.im/develop as an app.

I've slightly tweaked Joakim Ahlen's Riot wrapper for macOS, riotic, which uses the native WKWebView instead of Electron. I updated the app to be sandboxed so it has very limited access to your system. I also updated the interface to follow macOS conventions and updated it to use the latest version of Swift.

riotic does have some limitations though. Riot doesn't support VoIP on Safari so riotic can't support VoIP either and WKWebView doesn't support notifications as far as I can tell. Right now it uses a really old Riot icon, maybe I'll ask about using one of these community made icons https://github.com/vector-im/riot-web/pull/4474.

It does have some advantages over the official Riot Electron app though. The app is only ~12 MB compared to Riot which is ~180 MB, it also uses significantly less RAM. riotic also allows you to pick what Riot URL to use so you can run /develop as a desktop app.

You can download riotic from https://github.com/aaronraimist/riotic/releases/tag/v1.1.

 

🔗synapse-netcore-workers

Half-Shot has been working on synapse-netcore-workers:

The synapse-netcore-worker project has continued to evolve. You can now federate with other servers using the federation sender implementation. It supports everything except device lists at the moment, so it supports PDUs/EDUs and can just be connected up to one of your existing synapse instances. It's not been battletested enough yet to be put in production (hence no dockerfile), but it's very fast.
Oh and for those of you who don't know, "synapse-netcore-worker" is Travis's .NET implementation of synapse workers, the room can be found at #synapse-netcore-workers:t2bot.io.

If you're as uninitiated as I was three days ago, this project is a replaceable worker component for Synapse, which just happens to be written in .NET.

But wait, there's more! Half-Shot continues with

To give an update to the earlier exploits of synapse-netcore-worker's federation sender, we've still got a few more things to iron out before we can suggest people use it actively. The hit list of remaining things to fix is in https://github.com/turt2live/synapse-netcore-workers/pull/5

🔗Mautrix Whatsapp

That Ben guy eh? When he's not swanning off on holiday and having fun, he's writing super groovy guides to getting whatsapp bridging up and running. You don't even need a real device. Check it out.

🔗Ruby SDK

Ananace reports that:

Just merged the protocol split branch I've been working on for the Ruby SDK, including a first PoC for an application service base. Not tested in any actual use as of yet, but expect Things TM in the next release.

🔗Koma

yuforia reports that:

in project koma, a new bot picsay is created. It like the classic easter-egg program cowsay, but it uses actual photos instead of ASCII art. It configured to use any image just by editing a json file. So you can run your own version for fun.

🔗TChap

Willem reports that:

I took some time to hack on Tchap again. This time I disabled virus scanning of thumbnails and downloads in Tchap to be able to see avatars. I also wrote a non-scanning virus scanner API implementation to be able to see files, pictures and videos that are sent. The updated Tchap can be found at https://github.com/14mRh4X0r/tchap-android, the virus scanner API implementation at https://git.snt.utwente.nl/14mRh4X0r/tchap-media-scanner.

🔗Debian

andrewsh reports that:

Unless things get seriously on fire in the coming months, Debian buster will have Synapse 0.99.2 plus something newer in backports

🔗Matrix WUG

Dandellion's Wug now supports Inuktitut Syllabics and Iñupiatun Orthography. In the bot's own words.

Hi I can help you translate X-SAMPA, Z-SAMPA to IPA, and transcribe into proto-indo european notation! Use (x/z/p) together with either / or [] as delimiters x/"hEloU/ z[or` 5aIk DIz] p/mreghnom/

I also can transcribe to Inuktitut Syllabics like this: i[tusaumaqattautijjutinik aulattijiit]. Find my source at https://github.com/Dali99/matrix-wug

🔗New Rooms

Aaron Raimist has created a new room for Formula 1 fans:

Now that the Formula 1 season is getting underway it's probably a good time to announce @CIA:matrix.org's new(ish) room: #f1:matrix.org When this blog post is released there will be 16 days left before the first race of the 2019 season
There was previously a Formula 1 room but it was merged with Snoonet's IRC channel which tends to be extremely busy. This is a matrix only room.

That's it folks, your normal Ben orientated programming will continue next week. Bring back Ben, bring back Ben.

Bridging Matrix with WhatsApp running on a VM

26.02.2019 00:00 — TutorialsBen Parsons

This guide will live with the documentation at https://matrix.org/docs/guides/whatsapp-bridging-mautrix-whatsapp, but you can find the text below.


Matrix is:

an open standard for interoperable, decentralised, real-time communication.

In this article we'll benefit from all three of these attributes:

  • interoperable: we'll see how Matrix can be made to interact with WhatsApp
  • decentralised: you can perform this on your own server while still enjoying the benefits of being connected to the rest of the Matrix federation
  • real-time communication: we'll see how to send and receive messages in real-time

🔗Install your homeserver and install mautrix-whatsapp, the WhatsApp bridge

Firstly, you need to have a Matrix homeserver installed. If you don't currently have one, take a look at the instructions at Installing Synapse, and also in the Synapse README.

Next, install mautrix-whatsapp by following the instructions at mautrix-whatsapp/wiki.

If you are starting from scratch, I suggest you take a look at matrix-docker-ansible-deploy, as this project will enable you to deploy Synapse, mautrix-whatsapp and other components easily.

For example, if you have an existing deployment using matrix-docker-ansible-deploy, you can add mautrix-whatsapp simply by adding the following line to your configuration file:

matrix_mautrix_whatsapp_enabled: true

... and re-running the setup:

ansible-playbook -i inventory/hosts setup.yml --tags=setup-all

Either way, you will soon have a functioning Matrix Synapse homeserver and mautrix-whatsapp installed with it. Next, we will set up an Android VM.

🔗Set up an Android VM

The best way to run an Android Virtual Machine is to use the Android Studio tools from Google. First, install Android Studio, making sure to follow the post-install steps, as they will install additional tools we need, including AVD Manager.

Once installed, run AVD manager by choosing Tools -> AVD Manager from the menu.

Follow the steps to create a new virtual machine, in this example I have a Nexus 5X running Android 9, but almost any configuration is fine here. Make sure that you give the device access to the Play Store.

🔗Install WhatsApp and sign-in

Launch the Virtual Device, the open the Play Store and sign in. Now use the Play Store to install WhatsApp on the Virtual Device.

You will be asked to verify your phone number, use your number on another device to complete this step.

🔗Setup mautrix-whatsapp bridge

Now that you have WhatsApp working in a VM, and Matrix working on your server, it's time to bridge them together!

Per the instructions at mautrix-whatsapp/wiki, you must start a new chat with @whatsappbot:<yourdomain>. Type login to begin the authentication process.

mautrix-whatsapp operates by using the WhatsApp Web feature of WhatsApp - which means it uses a QR code that you must now scan on the device running WhatsApp - which in your case is the AVD. In order to scan the presented QR code, set your AVD camera to passthrough the camera device on your host machine - see the images below.

Once this is complete, you can type sync, to start bridging contacts, and sync --create to automatically create room invites.

And that's it! You may need to take a little time to watch the sync happen, particularly if you have a very large number of chats on the WhatsApp side, but there is no further configuration needed.

🔗Demo!

This Week in Matrix 2019-02-22

22.02.2019 00:00 — This Week in MatrixBen Parsons

🔗Matrix Live S03E15 - Push To Talk and XMPP Gatewaying!

(sorry that the audio from Nad & Half-Shot went missing from the recording... :( )

🔗nheko_reborn roadmap

red_sky has continued to work on nheko, and shared a general roadmap in #nheko-reborn:matrix.org:

The remainder of 0.6.x is expected to be bug fixes. This includes bug fixes in mtxclient as well as nheko.
0.7.0 will be the next feature release. The end goal here is to add some of the capabilities that people have been requesting, and other things that people haven't been requesting.
AFTER 0.7.0 releases, I will then focus on updating the encryption to include more than just text messages.

🔗libQMatrixClient project, Quaternion

kitsune has been writing the codes this week, Quaternion now supports room upgrades and more:

Master branch of Quaternion can now store access token in your secure storage/keychain, better integrating into frameworks of GNOME and KDE, as well as macOS and Windows.
Also, you can see the room version and upgrade rooms in Quaternion (just like Matthew said on FOSDEM ;) ) - this feature is enabled by libQMatrixClient, so other clients are welcome to support room versions/upgrades too!

All this is within the work on the upcoming release of libQMatrixClient 0.5 and Quaternion 0.0.9.4 - stay tuned!

Oh we will. Spectral support for these new features when?

🔗Matrix Ruby SDK

Ananace has continued his work on Ruby Matrix SDK:

Just tagged and pushed a new version (0.0.4) of the Ruby SDK, since I've completely forgotten to do that for a while now.

What's new?

A whole bunch of small stuff I did, mostly just additional exposed methods and parameters, and some fixed issues as well. I did add support for HS URL discovery using both SRV and .well-known though, which I think is probably the largest new feature.

As if all this wasn't enough, a room on Matrix!

Created a discussion room for the Ruby SDK - since it seems like there are now people who have actually used it apart from me; #ruby-matrix-sdk:kittenface.studio

🔗synapse-netcore-workers

A project rarely seen has received a lot of attention this week!

Half-Shot has been working on federation workers:

I've been working on a federation worker for synapse written in .NET to improve outgoing federation performance. It's still pretty barebones, but progress can be followed at https://github.com/turt2live/synapse-netcore-workers/pull/5

TravisR too has been working on the project:

I'm also working on my own netcore workers project by setting up an appservice sender. It's also very barebones (doesn't even send a transaction), but is being tracked here: https://github.com/turt2live/synapse-netcore-workers/pull/6
Also, if you'd like to follow along with the project from the comfort of Matrix, head on over to #synapse-netcore-workers:t2bot.io

🔗koma project

yuforia is working on koma:

Fix txnId generation that made sending messages unreliable. Work around a bug in OkHttp that crashed the dispatcher thread when the server certificate was not received in time.

Good thing to fix! Here's a screenshot of Continuum-desktop, the client from koma: >

yuforia continues:

Fix text processing in the bot avecho, feel free to try it out in #koma-im:matrix.org

🔗mxisd v1.3.1 released

Max:

mxisd v1.3.1 is out. It is a maintenance release that fixes a set of regressions following the changes in v1.3.0 to massively improve performance. If you haven't updated to the v1.3 branch, now is the time!

🔗Construct progresses toward Matrix 1.0 Spec

Over the past three weeks Construct has made rapid progress with covering the Matrix 1.0 specification!
Last week we implemented room version 1 and room version 2 authentication rules.
This week we implemented device support and management, and have nearly wrapped up successful End-To-End Encryption testing.
The Construct is a community-driven server implementation written in C++ for maximum performance. To all experienced C++ developers out there: we need your contributions to accelerate and test the 1.0 release!
Please check out https://github.com/matrix-construct/construct and stop by #test:zemos.net / #zemos-test:matrix.org today!

🔗uMatriks planning

mariogrip announced his plans for going ahead with uMatriks:

  • Clean up c++ code
  • Make the c++ code less heavy, we should focus on speed
  • Offload js code to direct qml binding, with this i mean instead of having ablock if-else code that modifies qml object that runs on component load, we do this directly in qml: see 0a800fc and https://github.com/uMatriks/uMatriksblob/0a800fcdc0af4fa2e08526dbff88e06bcb591779/uMatriks/componentsChatItem.qml as an example for a cleaner code
  • Make all ui depended calls async, we should NEVER block ui
  • Offload heavy js logic to c++
  • Cleaner UI, with less crust
  • Async everything
  • Merge our WIP call support
  • Finish impl of call support (maybe move to a standalone webrtc module)
  • Slowly start moving to QQC2
  • Implement reply
  • Implement sending of media

🔗Riot Android

  • Finalising version and delivery of Riot 0.8.23 on the Play Store.
  • RiotX: testing library to render events in the timeline.

🔗Debian packaging

andrewsh:

Since Debian has entered soft freeze, updates to Synapse have been slow lately, but nevertheless we've got 0.99.1.1 in

🔗matrix-archive-keyring

Linda is continuing to prepare matrix-archive-keyring:

matrix-archive-keyring (from last week's TWIM) is ready to be uploaded to Debian. Attempting to get it uploaded to unstable right now as I speak.

🔗Riot breadcrumbs

We linked to the announcement last week, and have been using it for even longer, but somehow I missed a great labs feature that shipped with Riot 1.0: "room breadcrumbs", which you can enable from the Labs page of Riot settings. This gives you a series of quick links with the history of rooms you were most recently in. So if you're like me, clicking between rooms all day, you can get where you're going a little faster.

🔗mautrix-whatsapp

There is no big news about mautrix-whatsapp this week, other than that I installed it on my own server and found that it works really really well. I'd love to switch to this as a main interface for my less-decentralised friends, but like others, I've been stung by this issue in an upstream project. If there are go-fans reading (and I know there are), I wonder if it's an issue which can be fixed?

🔗Around the Web

A tidy little article on "non-centralised" applications. Article by Jontheniceguy helps pin down some of the terminology around decentralisation and the like. https://jon.sprig.gs/blog/post/1041

joenas wrote an article on configuring Synapse with Traefik: Matrix homeserver (Synapse v0.99.1.1) with Traefik

🔗That's it

All good things must come to an end - and here we are!

Matrix is getting a lot of attention recently (has it been on the front page of HN every day this week? I remember that I used to be excited seeing that happen), and seeing a lot of growth as well.

If you have something to say here, something to add, come chat to us in #twim:matrix.org - I love that this is a supportive and engaged project ecosystem, so come share what you have!

Welcome to Matrix, KDE!

20.02.2019 00:00 — GeneralMatthew Hodgson

Hi all,

We're very excited to officially welcome the KDE Community on to Matrix as they announce that KDE Community is officially adopting Matrix as a chat platform, and kde.org now has an official Matrix homeserver!

You can see the full announcement and explanation over at https://dot.kde.org/2019/02/20/kde-adding-matrix-its-im-framework.  It is fantastic to see one of the largest Free Software communities out there proactively adopting Matrix as an open protocol, open network and FOSS project, rather than drifting into a proprietary centralised chat system.  It's also really fun to see Riot 1.0 finally holding its own as a chat app against the proprietary alternatives!

This doesn't change the KDE rooms which exist in Matrix today or indeed the KDE Freenode IRC channels - so many of the KDE community were already using Matrix, all the rooms already exist and are already bridged to the right places.  All it means is that there's now a shiny new homeserver (powered by Modular.im) on which KDE folk are welcome to grab an account if they want, rather than sharing the rather overloaded public matrix.org homeserver.  The rooms have been set up on the server to match their equivalent IRC channels - for instance, #kde:kde.org is the same as #kde on Freenode; #kde-devel:kde.org is the same as #kde-devel etc.  The rooms continue to retain their other aliases (#kde:matrix.org, #freenode_#kde:matrix.org etc) as before.

There's also a dedicated Riot/Web install up at https://webchat.kde.org, and instructions on connecting via other Matrix clients up at https://community.kde.org/Matrix.

This is great news for the Matrix ecosystem in general - and should be particularly welcome for Qt client projects like Quaternion, Spectral and Nheko-Reborn, who may feel a certain affinity to KDE!

So: welcome, KDE!  Hope you have a great time, and please let us know how you get on, so we can make sure Matrix kicks ass for you :)

  • the Matrix Core Team