Client-server r0.6.0 and Identity Service r0.3.0 releases

08.11.2019 00:00 — Releases, SpecTravis Ralston

Hey all,

For the last several months the team has been working on tightening up privacy in Matrix, and with the 1.4 release of Synapse and Riot quite a lot has been done in the area. One of the remaining pieces was to release all the specification changes to help other client/server implementations achieve the same goals, and now we've done that.

The Client-Server r0.6.0 and Identity Service r0.3.0 spec releases both cover the privacy improvements added through a number of MSCs in the last few months. Of particular note is that identity servers are now expected to support terms of service endpoints, which requires authentication that clients might need to worry about - check the spec changelogs for details.

The full changelog for the Client-Server r0.6.0 release is:

  • Breaking Changes

    • Add id_access_token as a required request parameter to a few endpoints which require an id_server parameter as part of MSC2140. (#2255)
  • New Endpoints

    • Add POST /account/3pid/unbind for removing a 3PID from an identity server. (#2282)
  • Backwards Compatible Changes

    • Add M_USER_DEACTIVATED error code. (#2234)
    • Remove bind_msisdn and bind_email from /register now that the identity server's bind endpoint requires authentication. (#2279)
    • Add m.identity_server account data for tracking the user's preferred identity server. (#2281)
    • Deprecate id_server and make it optional in several places. (#2310)
  • Spec Clarifications

    • Add missing format fields to m.room.message$m.notice schema. (#2125)
    • Remove "required" designation from the url field of certain m.room.message msgtypes. (#2129)
    • Fix various typos throughout the specification. (#2131, #2136, #2148, #2215)
    • Clarify the distinction between m.key.verification.start and its m.sas.v1 variant. (#2132)
    • Fix link to Olm signing specification. (#2133)
    • Clarify the conditions for the .m.rule.room_one_to_one push rule. (#2152)
    • Clarify the encryption algorithms supported by the device of the device keys example. (#2157)
    • Clarify that /rooms/:roomId/event/:eventId returns a Matrix error. (#2204)
    • Add a missing state_key check on .m.rule.tombstone. (#2223)
    • Fix the m.room_key_request action value, setting it from cancel_request to request_cancellation. (#2247)
    • Clarify that the submit_url field is without authentication. (#2341)
    • Clarify the expected phone number format. (#2342)
    • Clarify that clients should consider not requesting URL previews in encrypted rooms. (#2343)
    • Add missing information on how filters are meant to work with /context. (#2344)
    • Clarify what the keys are for rooms in /sync. (#2345)

The full changelog for the Identity Service r0.3.0 release is:

  • New Endpoints

    • Add /account, /account/register, and /account/logout to authenticate with the identity server. (#2255)
    • Add endpoints for accepting and handling terms of service. (#2258)
    • Add /hash_details and a new /lookup endpoint for performing hashed association lookups. (#2287)
  • Backwards Compatible Changes

    • Deprecate the v1 API in favour of an authenticated v2 API. (#2254)

Fun and games with certificate transparency logs

06.11.2019 00:00 — SecurityMatthew Hodgson

Hi all,

This morning (06:11 UTC) it became apparent through mails to [email protected] that a security researcher was working through the TLS Certificate Transparency logs for *.matrix.org,*.riot.im and *.modular.im to identify and try to access non-public services run by New Vector (the company formed by the original Matrix team, which hosts *.matrix.org on behalf of the Matrix.org Foundation, and develops Riot and runs the https://modular.im hosting service).

Certificate Transparency (CT) is a feature of the TLS ecosystem which lets you see which public certificates have been created and signed by given authorities - intended to help identify and mitigate against malicious certificates. This means that the DNS name of any host with a dedicated public TLS certificate (i.e. not using a wildcard certificate) is visible to the general public.

In practice, this revealed a handful of internal-facing services using dedicated public TLS certificates which were accessible to the general internet - some of which should have been locked to be accessible only from our internal network.

Specifically:

  • kibana.ap-southeast-1.k8s.i.modular.im - a Kibana deployment for a new experimental Modular cluster which is being set up in SE Asia. The Kibana is in the middle of being deployed, and was exposed without authentication during deployment due to a firewall & config error. However, it is not a production system and carries no production traffic or user data (it was just being used for experimentation for hypothetical geography-specific Modular deployments). We firewalled this off at 07:53 UTC, and are doing analysis to confirm there was no further compromise, and will then rebuild the cluster (having fixed the firewall config error from repeating).
  • AWX deployments used by our internal Modular platform, which were behind authentication but should not be exposed to the public net.
  • Various semi-internal dev and testing services which should be IP-locked to our internal network (but are all locked behind authentication too).

Additionally, certain historical Modular homeservers & Riots (from before we switched to using wildcard certs, or where we’ve created a custom LetsEncrypt certificate for the server) are named in the CT logs - thus leaking the server’s name (which is typically public anyway in that server’s matrix IDs if the server is federated).

We’re working through the services whose names were exposed checking for any other issues, but other than the non-production SE Asia Kibana instance we are not aware of problems resulting from this activity.

Meanwhile, we’ll be ensuring that semi-internal services are only exposed on our internal network in future, and that Modular server names are not exposed by CT logs where possible.

TL;DR: You can list all the public non-wildcard TLS certs for a given domain by looking somewhere like https://crt.sh/?q=%25.matrix.org. This lets you find internal-sounding services to try to attack. In practice no production services were compromised, and most of our internal services are correctly firewalled from the public internet. However, we’re reviewing the IP locking for ones in the grey zone (and preventing the bug which caused an experimental Kibana to be exposed without auth).

We’d like to thank Linda Lapinlampi for notifying us about this. We’d also like to remind everyone that we operate a Security Disclosure Policy (SDP) and Hall of Fame at https://matrix.org/security-disclosure-policy/ which is designed to protect innocent users from being hurt by security issues - everyone: please consider disclosing issues responsibly to us as per the SDP.

This Week in Matrix 2019-11-01

01.11.2019 00:00 — This Week in MatrixBen Parsons

🔗Dept of Spec 📜

🔗Spec Updates

anoa reported:

Merged MSCs

Final Comment Period

New MSCs

No other new MSCs appeared this week.

MSCs the Spec Core Team are focusing on next week are: MSC2244 (mass redactions), MSC1946 (SSSS), and MSC2313 (ban lists).

🔗Dept of Servers 🏢

🔗Synapse 1.5.0 released

Various updates including a security fix, check the announcement.

Several packaging projects have been updated to deploy the new version:

Like always the mvgorcum/docker-matrix repo also includes the release candidates, as they were released.

🔗Install Party

Brendan announced:

A couple of weeks ago I shouted here about a project I've been working on named Install Party, which provides tools for provisioning and managing servers for Matrix homeserver install workshops/parties.

Since then, I've been working on improving it, and today it's finally reached v1.0! This version includes configurable DNS and infrastructure providers, the ability to create multiple server in one run, user-defined post-install scripts, as well as codebase cleanups and a better documentation.

You can find more details about the project and this release at https://brendan.abolivier.bzh/install-party-1.0/, and in #install-party:abolivier.bzh 🙂

🔗Dept of Bridges 🌉

🔗mautrix-telegram

Tulir told us:

  • Parallelized file transfer: The bridge now has an option to use multiple telegram connections and a streaming connection to the Matrix media repo when copying files. This should make it much faster and use less ram for big files.
  • Matrix doesn't have native captions, so !tg caption <text> now exists to send the next image or file to telegram with <text> as the caption.
  • Animated sticker bridging and helm charts were merged into master.

🔗matrix-appservice-node 0.4.1 released

Half-Shot reported:

Bridges-in-nodejs-fans, today we have released 0.4.1 of the matrix-appservice-node library. For those not aware (presumably most), this library is a barebones piece of kit that helps you to listen over the AS api for transactions, in a more barebones manner than matrix-appservice-bridge. The changes in this release are a total transformation of the library into Typescript, and updating dependency packages which had gotten out of date.

EDIT: I cocked up the release, so have a 0.4.1

🔗matrix-appservice-bridge 1.11.1 released

Half-Shot offered:

Also, https://github.com/matrix-org/matrix-appservice-bridge/releases/tag/1.11.1

🔗Dept of Clients 📱

🔗Ditto Chat project announcement

Annie said:

Officially announcing a new mobile client project: Ditto Chat

  • Project Status

    • Minimum Viable Product is on TestFlight
    • Can: login, send / receive messages
  • Tech Stack

    • Bare React Native - no Expo
    • matrix-js-sdk
  • Vision

  • I want Ditto to be a chat app that is user-friendly and performant for the average user, rivaling apps like GroupMe, Messenger, and WhatsApp.

  • Later on, I want Ditto to have a desktop / browser version and be stable enough for regular use.

CONTRIBUTORS: I would love to have some help on Ditto if anybody is looking for a side project! Join #ditto:elequin.io for updates.

More photos: http://dittochat.org

Check out the room at #ditto:elequin.io.

Ditto image

🔗miitrix

sorunome offered:

Miitrix has received some updates!

  • Save state and resume when starting up - no need for an initial sync each time anymore!
  • Send read receipts
  • Send typing notifications
  • Remove HTTPC and only use CURL to add support for servers that don't have TLS1.1 anymore! And way faster!

Support room: #miitrix:sorunome.de Donate: https://liberapay.com/Sorunome

Miitrix was featured as a star attraction at MozFest last weekend, thanks Soru!

Good morning #mozfest! We're here for the final day, come and see us on the 6th floor and see Matrix running on a Nintendo 3DS 🎮 #matrix #mozfest19 pic.twitter.com/cZqcAn2tcZ

— Matrix (@matrixdotorg) October 27, 2019

🔗Continuum 0.9.27

yuforia reported:

Continuum, desktop client in Kotlin, version 0.9.27:

Added notification viewer.

https://matrix.org/_matrix/media/r0/download/matrix.org/ziPiTnybBwEEcSMPZRjhNlsV

🔗Fractal

Alexandre Franke told us:

We fixed redaction, it’s working again. We still don’t remove the messages from view but there is a WIP merge request to do just that.

🔗#matrix-client-developers:matrix.org

f0x announced:

#matrix-client-developers:matrix.org has been revived, intended for discussion between developers of different Matrix clients.

This is the room to head to for Matrix Client developer chat.

🔗Matrix Notepad

KB1RD announced:

Version 0.1.2 update brings a major overhaul of the user interface. The main features are:

  • A new sign in dialog (with password authentication. No access tokens!)
    • Those who are already signed in will stay signed in
    • Also created a Matrix Notepad logo
  • Technically, a single Matrix client is used instead of re-creating it each time a new document is opened
    • This makes document loading faster
  • Added a document list and an add button
  • Added the document room ID to the URL
  • Fixed various bugs that have been encountered. Maybe I can finally say that it's bug-free!

🔗Riot v1.5.1-rc.1

Riot v1.5.1-rc.1 is up on https://riot.im/staging with a collection of bugfixes, some further a11y fixes and... a new reaction picker from tulir!

🔗Dept of Bots 🤖

🔗matrix-episode-bot

anoa told us:

I made a bot that can give you links to TV show episodes that get mentioned in rooms: https://github.com/anoadragon453/matrix-episode-bot

It's not as cool as it sounds. Basically you put all the titles and links in a config file and whenever someone says e.g "I really like S05E09", it'll give you the name of that episode and a link to it. You can also just mention an episode title and it'll give you the link.

Made with nio-template.

🔗Dept of Ping 🏓

RankHostnameMedian MS
1matrix.tetraodon.nl336
2room409.xyz490
3nora.codes491
4freitrix.de491
5matrix.dapor.net492
6stratum0.org500
7midov.pl503
8bau-ha.us537
9linuxgl.ch539
10maunium.net577

🔗That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Synapse 1.5.0 released

29.10.2019 00:00 — ReleasesNeil Johnson

That's right folks Synapse 1.5.0 is here and ready to make your life just a little bit better.

First things first, this release includes a security fix (#6262, below). Administrators are encouraged to upgrade as soon as possible.

Aside from that, the main thing you'll notice in 1.5.0 is a massive performance improvement to the room directory, which means that servers with large directories to scan will return much more quickly. This is especially true for matrix.org but all servers will benefit.

Another key win is finally fixing some bugs in the sqlite -> postgres migrator script. Sqlite mode is there strictly for testing purposes and should never be used in a production setting let alone a federating homeserver. So if you are currently using Sqlite now is the time to migrate. What's more the script is now in CI so it can't easily break in the future (with apologies to anyone bitten by the old script...).

A final final point, we have some preparatory work for e2ee cross signing, the overall feature is not ready for release just yet but we are getting really close. Watch this space.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Also, check out our Synapse installation guide page

The changelog since 1.4.1 follows:

🔗Synapse 1.5.0 (2019-10-29)

🔗Security updates

This release includes a security fix (#6262, below). Administrators are encouraged to upgrade as soon as possible.

🔗Bugfixes

  • Fix bug where room directory search was case sensitive. (#6268)

🔗Synapse 1.5.0rc2 (2019-10-28)

🔗Bugfixes

  • Update list of boolean columns in synapse_port_db. (#6247)
  • Fix /keys/query API on workers. (#6256)
  • Improve signature checking on some federation APIs. (#6262)

🔗Internal Changes

  • Move schema delta files to the correct data store. (#6248)
  • Small performance improvement by removing repeated config lookups in room stats calculation. (#6255)

🔗Synapse 1.5.0rc1 (2019-10-24)

🔗Features

  • Improve quality of thumbnails for 1-bit/8-bit color palette images. (#2142)
  • Add ability to upload cross-signing signatures. (#5726)
  • Allow uploading of cross-signing keys. (#5769)
  • CAS login now provides a default display name for users if a displayname_attribute is set in the configuration file. (#6114)
  • Reject all pending invites for a user during deactivation. (#6125)
  • Add config option to suppress client side resource limit alerting. (#6173)

🔗Bugfixes

  • Return an HTTP 404 instead of 400 when requesting a filter by ID that is unknown to the server. Thanks to @krombel for contributing this! (#2380)
  • Fix a bug where users could be invited twice to the same group. (#3436)
  • Fix /createRoom failing with badly-formatted MXIDs in the invitee list. Thanks to @wener291! (#4088)
  • Make the synapse_port_db script create the right indexes on a new PostgreSQL database. (#6102, #6178, #6243)
  • Fix bug when uploading a large file: Synapse responds with M_UNKNOWN while it should be M_TOO_LARGE according to spec. Contributed by Anshul Angaria. (#6109)
  • Fix user push rules being deleted from a room when it is upgraded. (#6144)
  • Don't 500 when trying to exchange a revoked 3PID invite. (#6147)
  • Fix transferring notifications and tags when joining an upgraded room that is new to your server. (#6155)
  • Fix bug where guest account registration can wedge after restart. (#6161)
  • Fix monthly active user reaping when reserved users are specified. (#6168)
  • Fix /federation/v1/state endpoint not supporting newer room versions. (#6170)
  • Fix bug where we were updating censored events as bytes rather than text, occasionally causing invalid JSON being inserted breaking APIs that attempted to fetch such events. (#6186)
  • Fix occasional missed updates in the room and user directories. (#6187)
  • Fix tracing of non-JSON APIs, /media, /key etc. (#6195)
  • Fix bug where presence would not get timed out correctly if a synchrotron worker is used and restarted. (#6212)
  • synapse_port_db: Add 2 additional BOOLEAN_COLUMNS to be able to convert from database schema v56. (#6216)
  • Fix a bug where the Synapse demo script blacklisted ::1 (ipv6 localhost) from receiving federation traffic. (#6229)

🔗Updates to the Docker image

  • Fix logging getting lost for the docker image. (#6197)

🔗Internal Changes

  • Update user_filters table to have a unique index, and non-null columns. Thanks to @pik for contributing this. (#1172, #6175, #6184)
  • Allow devices to be marked as hidden, for use by features such as cross-signing. This adds a new field with a default value to the devices field in the database, and so the database upgrade may take a long time depending on how many devices are in the database. (#5759)
  • Move lookup-related functions from RoomMemberHandler to IdentityHandler. (#5978)
  • Improve performance of the public room list directory. (#6019, #6152, #6153, #6154)
  • Edit header dicts docstrings in SimpleHttpClient to note that str or bytes can be passed as header keys. (#6077)
  • Add snapcraft packaging information. Contributed by @devec0. (#6084, #6191)
  • Kill off half-implemented password-reset via sms. (#6101)
  • Remove get_user_by_req opentracing span and add some tags. (#6108)
  • Drop some unused database tables. (#6115)
  • Add env var to turn on tracking of log context changes. (#6127)
  • Refactor configuration loading to allow better typechecking. (#6137)
  • Log responder when responding to media request. (#6139)
  • Improve performance of find_next_generated_user_id DB query. (#6148)
  • Expand type-checking on modules imported by synapse.config. (#6150)
  • Use Postgres ANY for selecting many values. (#6156)
  • Add more caching to _get_joined_users_from_context DB query. (#6159)
  • Add some metrics on the federation sender. (#6160)
  • Add some logging to the rooms stats updates, to try to track down a flaky test. (#6167)
  • Remove unused timeout parameter from _get_public_room_list. (#6179)
  • Reject (accidental) attempts to insert bytes into postgres tables. (#6186)
  • Make version optional in body of PUT /room_keys/version/{version}, since it's redundant. (#6189)
  • Make storage layer responsible for adding device names to key, rather than the handler. (#6193)
  • Port synapse.rest.admin module to use async/await. (#6196)
  • Enforce that all boolean configuration values are lowercase in CI. (#6203)
  • Remove some unused event-auth code. (#6214)
  • Remove Auth.check method. (#6217)
  • Remove format_tap.py script in favour of a perl reimplementation in Sytest's repo. (#6219)
  • Refactor storage layer in preparation to support having multiple databases. (#6231)
  • Remove some extra quotation marks across the codebase. (#6236)

This Week in Matrix 2019-10-26

26.10.2019 00:00 — This Week in MatrixBen Parsons

🔗Dept of Status of Matrix 🌡

🔗Matrix.org

Public service notice from vdh:

It won't have escaped many peoples' notice that the matrix.org homeserver has been struggling over the last few weeks. The main problem is poor disk I/O performance on our database server: for various reasons, it seems that the server can no longer keep up with the demands we're putting on it.

The good news is that we have a plan to sort it out, and we're working on setting up alternative hosting which will be able to handle our traffic for the foreseeable future! Please bear with us over the next couple of weeks while we get new servers set up.

🔗Matrix Notepad

@kb1rd:kb1rd.net told us:

Just released the first version of the Matrix Notepad, a small webapp designed to allow people to collaborate on text files over Matrix. There are a bunch of bugs that I'm still working on fixing, but if you'd like to check it out it's here! I will be adding incremental fixes for various issues, but feel free to add to the issues. :) If you'd like to chat, I'll be checking #matrix-collaboration:kb1rd.net.

🔗Dept of Servers 🏢

🔗Synapse

@richvdh:sw1v.org told us:

This week much of the team's focus has been on performance, both in terms of dealing with matrix.org's immediate woes and some more strategic work on helping Synapse to scale. On the latter front, Erik has been making great progress in supporting multiple Postgres databases.

We've also put out a release candidate for Synapse 1.5.0 with lots of bugfixes and changes under the hood. As ever, help with testing the RC is much appreciated!

🔗Dept of Bridges 🌉

🔗matrix-appservice-discord v0.5.2 released

Half-Shot reported:

matrix-appservice-discord has had a new release (v0.5.2) to fix some out of date packages.

🔗matrix-appservice-bridge v0.11.0

Half-Shot announced:

matrix-appservice-bridge has had a new release (v0.11.0) which allows developers to disable usage of the stores entirely. This should solve problems facing some bridges which no longer use them.

🔗mautrix-hangouts image bridging

Tulir offered:

Cadair's pull request to add Hangouts->Matrix image bridging to mautrix-hangouts was merged.

🔗Dept of Clients 📱

🔗miitrix

From sorunome, this crazy 3DS project!

Miitrix is a client for the Nintendo 3DS, using the matrix-3ds-sdk! Here is an early demonstration video. The current featureset includes:

  • Sorting room list by last messages
  • Room name fallbacks (canonical alias, joined members)
  • sending messages
  • receiving messages
  • receiving edits & redactions
  • receiving join, leave, ban events
  • lazy-loading extra information in the background

Support room: #miitrix:sorunome.de Donate: https://liberapay.com/Sorunome

miitrix client running on 3DS

🔗Riot web

Tulir announced:

Two of the Riot pull requests that I twimmed last week, the reaction emoji picker and edit html parser improvements, have been merged into develop. The remaining one (reply rendering improvements) still needs some code and design work.

🔗Ruby Matrix SDK version 1.5.0 released

Ananace said:

The Ruby SDK is out with another new version, 1.5.0 this time. Adding an event and a check to let applications track and handle errors that occur in the background listener thread of the client abstraction. This version also exposes a setter for the open_timeout value on the lower-level API client (and the client abstraction through it), for users that are plagued by slow-to-open TCP connections to their relevant HSes.

As always, feel free to drop by #ruby-matrix-sdk:kittenface.studio if you have questions, comments, corrections, or just want to discuss the project - or use cases - in general.

🔗Fractal

Alexandre Franke reported:

Since the 4.2.1 release a month ago, there’s been a slow but steady stream of commits including code cleanups, translation updates and minor bugfixes. The highlights are issues solved around settings not being reloaded after verification dialog was closed, account settings not loading, connections to http only servers failing, and a regression that prevented users from sending messages to newly created rooms.

🔗Riot iOS

Manu reported:

We are still working on 2 fronts:

  • fixing most important issues in our stabilisation sprint
  • privacy project: Adding an email or phone number is now protected by a password (if your homeserver requests this authentication)

🔗Riot Android

Manu offered:

  • privacy project: Adding an email or phone number is now protected by a password (if your homeserver requires it)

🔗RiotX Android

Manu offered:

  • We released 0.7.0 this week. You now have read markers and can share content to Riot. You should also find the app even faster.

🔗Dept of SDKs and Frameworks 🧰

🔗matrix-3ds-sdk

sorunome offered:

A lot has happened on the side of matrix-3ds-sdk. So much so, that it is possible now to write clients out of it! The changelog includes:

  • fetching member info
  • fetching room info
  • proper sync loop
  • aggressive filters to reduce data (initial sync would crash on large accs otherwise)
  • room leave and invite callbacks
  • room info callback

Support room: #matrix-3ds-sdk:sorunome.de

🔗Dept of Ops 🛠

🔗Helm charts for mautrix-telegram, mautrix-twilio and maubot

Tulir offered:

mautrix-telegram and mautrix-twilio now have experimental Helm charts: https://github.com/tulir/mautrix-telegram/tree/helm/helm/mautrix-telegram. There's no replication/horizontal scaling stuff, but they should be useful if you're already using Helm for other stuff.

To make managing maubot plugins possible with Helm, I added a simple standalone plugin executor to maubot. It's basically just a simple script that sets up all the things plugins want (matrix client, config, database, webserver) and passes them to the plugin.

The standalone executor can be used to run any maubot plugin without any changes to the plugin itself, but building good docker images needs to be done separately for each plugin. The supportportal plugin has a standalone docker image and a helm chart using that image.

🔗Dept of Ping 🏓

RankHostnameMedian MS
1matrix.tetraodon.nl298
2linuxgl.ch319.5
3flip.earth402
4matrix.allmende.io546
5midov.pl601
6tout.im622
7matrix.kimani.dev648.5
8matrix.vgorcum.com660
9uraziel.de675
10kif.rocks711

🔗That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

This Week in Matrix 2019-10-18

18.10.2019 21:46 — This Week in MatrixBen Parsons
Last update: 18.10.2019 19:52

🔗Matrix Live 🎙

🔗Dept of Status of Matrix 🌡

Last week Matrix had a presence at UbuCon Europe and PyCon Ireland. We gave workshops on using Matrix to create bots, and also a session on installing Synapse (see also: Brendan's entry below!)

Hey #PyConIE! We hope you enjoyed the Matrix workshop, find the slides and code samples at https://t.co/aT1dDuEcm1 and make sure to ask any more questions! pic.twitter.com/C2qwV5R3gl

— Matrix (@matrixdotorg) October 12, 2019

Tomorrow I'm off to sunny Manchester with Michael from the Ops team. We'll visit OggCamp, where we'll show off Matrix with a fun demo I previewed on Matrix Live a few weeks ago.

Matrix will ALSO be at:

🔗Dept of Spec 📜

anoa:

Last week we set MSC1219 (key backups), MSC2241 (verification over DMs), and MSC2313 (ban lists) for the spec core team to focus on. Those 3 are rolling on into this week as we didn't get a lot of work done last week :)

In other Spec news, Matthew uncovered a stone tablet describing what would in future be known as "MSC2324":

msc2324

🔗Dept of Servers 🏢

🔗install-party, nice result from UbuCon Europe

Brendan said:

For a workshop anoa and I did at Ubucon Europe last Sunday on how to install Synapse, I worked on a side project that creates a server, attaches a domain name to it and installs Riot and Caddy on it. Attendees can then SSH into it, follow the instructions to install a Matrix homeserver, and use Riot to register an account on it, log in, and join a federated room with all of the other attendees' homeservers in it. We tried it out last week-end for the actual workshop and it worked quite well 🙂 The project is called Install Party, and lives at https://github.com/babolivier/install-party, and if you want to chat about it I've just created #install-party:abolivier.bzh 😉

🔗Synapse

Neil reported:

This week we've been thinking about the future and brainstorming on ideas to improve perf for small instances and sparing some cycles for MSC1228. Next week we'll return to improving IO usage on matrix.org.

Synapse 1.4.1 was released, which fixes a small regression in 1.4.0.

🔗Dept of Bridges 🌉

🔗Matrix-Slack Bridge on NixOS

@christian:kampka.net said:

Just FYI, the matrix slack bridge is now available via NixOS: https://github.com/NixOS/nixpkgs/pull/70753

🔗Dept of Clients 📱

🔗riots.im available via IPFS

Remember https://riots.im from last week? Now available on IPFS!

@swedneck:permaweb.io said:

toml's riots.im is also available via IPFS: ipfs get riots.im or visit localhost:8080/ipns/riots.im with ipfs installed and running.

🔗Riot-iOS 0.10.0

Manu offered:

On Riot-iOS land, Riot-iOS 0.10.0 has been released on the App Store. We have started a stabilisation sprint. In parallel, we are still polishing the privacy work

🔗RiotX release next week

benoit announced:

RiotX: we are merging waiting PRs and have postponed the release to ensure a maximum of stability and polishing. The release will be done at the beginning of next week and will contains: read marker, camera picker and improved file picker, share to RiotX capability and many bugfixes.

🔗Dept of SDKs and Frameworks 🧰

🔗matrix-3ds-sdk

sorunome said:

Heya, matrix-3ds-sdk is a new matrix sdk - for the Nintendo 3DS! It is still deep in development, so expect bugs, some things not working properly etc. Current features include:

  • logging in
  • sending text messages to rooms
  • redacting events from rooms
  • resolving aliases to room ids
  • basic framework to get a /sync loop working

If you are into 3DS homebrew development or are interested in helping out making a full client based on this, please contact soru (@sorunome:sorunome.de)! 3ds homebrew dev is pretty new to her, so there are plenty of open questions / debugging help would be great!

Support room: #matrix-3ds-sdk:sorunome.de Donate: https://liberapay.com/Sorunome

This is really impressive, I'm looking forward to playing with it. (I'm sure someone at the office must own a 3DS!)

🔗koma

yuforia offered:

koma, Kotlin library for building clients: Added API for retrieving notifications.

🔗Dept of Ops 🛠

🔗mvgorcum/docker-matrix

Mathijs announced:

I uploaded the synapse 1.4.1 image to mvgorcum/docker-matrix:v1.4.1 within about half an hour after the release

🔗docker image of synapse-develop

Mathijs announced:

I've made a docker image of the develop branch of synapse, and automated it to build daily. It's on my docker hub as mvgorcum/docker-matrix:develop, based on the avhost dockerfile. Note that there are no checks, the image is simply built from the develop branch of the synapse git repo every night.

🔗Matrix Synapse for Kubernetes

Ananace offered:

just pushed Version 1.4.1 of the Matrix Synapse for Kubernetes packaging. A bit fiddly to do on a phone 🙂

🔗Synapse 1.4.1 multi-arch docker image

Black Hat told us:

I'm building the Synapse 1.4.1 multi-arch docker image. It will be pushed to Docker Hub in a couple of hours.

🔗Dept of Tulir 🇫🇮

A special section this week from the guy with one editor open for everything, tulir:

I haven't done anything on my own projects this week, but I did contribute to a bunch of other projects:

  • Updated SmsMatrix to the latest matrix-android-sdk to fix outgoing sms duplication bug (https://github.com/tijder/SmsMatrix/pull/60)
  • Fixed Riot web sending reply fallbacks in edited message content (https://github.com/matrix-org/matrix-react-sdk/pull/3551)
  • Fixed some things in the Riot web edit html -> markdown parsing (https://github.com/matrix-org/matrix-react-sdk/pull/3552)
  • Made Riot web reply rendering much nicer and more compact (https://github.com/matrix-org/matrix-react-sdk/pull/3553)
  • Added full emoji picker for reactions to Riot web (https://github.com/matrix-org/matrix-react-sdk/pull/3554)

The first two are already merged (and SmsMatrix even got a new release on f-droid), the html parsing is waiting for code review and the emoji picker and reply rendering are waiting for design review.

Also, I made a read-only status.matrix.org rss feed room since some people wanted one: #matrix.org-status:maunium.net. I don't remember if I TWIMed these before, but #xkcd:maunium.net and #commitstrip:maunium.net are similar read-only rooms, new xkcds and commitstrips are posted there whenever they come out.

🔗Dept of Ping 🏓

RankHostnameMedian MS
1matrix.tetraodon.nl340
2flip.earth353.5
3matrix.bn4t.me397.5
4midov.pl435
5aragon.sh443
6finallycoffee.eu493
7dodsorf.as564
8matrix.vgorcum.com575.5
9maunium.net595
10room409.xyz598

🔗That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Synapse 1.4.1 released

18.10.2019 00:00 — ReleasesMatthew Hodgson

Hi all,

We've released Synapse 1.4.1 as a small but important bugfix to 1.4.0.

This fixes a regression which crept in with our newly implemented "erase redacted data after N days" feature where some APIs would fail when hitting erased redactions - anyone on Synapse 1.4.0 will want to upgrade asap.

As always, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Also, check out our Synapse installation guide page

The changelog since 1.4.0 follows:

🔗Synapse 1.4.1 (2019-10-18)

No changes since 1.4.1rc1.

🔗Synapse 1.4.1rc1 (2019-10-17)

🔗Bugfixes

  • Fix bug where redacted events were sometimes incorrectly censored in the database, breaking APIs that attempted to fetch such events. (#6185, 5b0e9948)

This Week in Matrix 2019-10-11

11.10.2019 17:36 — This Week in MatrixBen Parsons
Last update: 11.10.2019 17:31

🔗Dept of Status of Matrix 🌡

As you may have heard, New Vector, a major entity in the Matrix world, recently completed a series A funding round.

Further reading:

🔗Mozilla IM trial ended

Mozilla have been trialing different IM solutions to replace IRC, including Matrix. This trial ended this week, and we hope to hear the results this month. (C'mon, Matrix!)

🔗Dept of Servers 🏢

🔗Synapse

Neil told us:

Matrix.org hit some IO problems earlier this week, while largely a problem with our hosting provider, we’re spending a bit of time to make Synapse more resilient if the same thing were to happen again. This will mean the ability to shard the DB (by table) and spread the load so we are not so dependent on high performance from a given db box. Outside of that we’ve been working on the final polishing of the room directory and getting the sqlite -> Postgres port script into better shape.

🔗multi-arch docker image of Synapse

Black Hat said:

My multi-arch docker image of Synapse has been updated to v1.4.0.

🔗Dendrite

anoa offered:

Not much for Dendrite this week as anoa is off at Ubucon 2019. But we had a few valuable bugs reported by the community, and a pressing reminder to get Dendrite's Monolith mode in as part of its CI.

🔗Dept of Bridges 🌉

🔗slack bridge 1.0.1

Half-Shot told us:

Hey folks, The slack bridge 1.0.1 release is out https://github.com/matrix-org/matrix-appservice-slack/releases/tag/1.0.1 containing a few bug fixes found since the original release.

🔗IRC bridge

Half-Shot offered:

I've not got much for this week, but the IRC bridge has been undergoing some serious refactors and changes for a larger release. Should be quite a big one when it lands :)

🔗Dept of Encryption 🔐

🔗Search inside E2EE rooms

Update from poljar:

A PR for riot-web has emerged that adds support for search in E2E encrypted rooms. The PR is utilizing Seshat to perform event indexing and search on riot-desktop. While the PR is missing any sort of UI, it is in a usable state.

🔗Dept of Clients 📱

🔗Riot iOS

Manu told us:

phase:1 of privacy work is done. Riot -iOS 0.10.0 will be available soon

🔗every version of Riot Web released on GitHub

toml offered:

Take a trip down memory lane with the Riots of yesteryear at https://riots.im (note the 's'). Hosting every version of Riot Web released on GitHub 😁

Comes with free Wikipedia hole.

🔗Riot Android

benoit announced:

Valere has done a release and is doing some maintenance. He has started to work on integration manager

🔗RiotX

benoit offered:

RiotX: We have fixed quite a lot of issues during the stabilization sprint. We are now working on Sprint 4: read marker, report content, mark all room read, etc. François is changing the media/file picker and we will also be able to share elements from other apps to RiotX. We will schedule a release soon (tm) (should have happen this week, but has been delayed due to stabilization)

🔗Dept of SDKs and Frameworks 🧰

🔗Elixir projects from uhoreg, Polyjuice and Igor

uhoreg told us:

Polyjuice Client, a Matrix client library for Elixir, has a new release. There is now a short tutorial that will teach you how to make a simple echo bot with it.

Then:

Igor, a bot framework for Elixir, has had its first release.

🔗Dept of Ping 🏓

RankHostnameMedian MS
1matrix.tetraodon.nl323
2aime.lesmatric.es361
3nerdsin.space370
4ru-matrix.org371.5
5c-base.org377
6fachschaften.org430
7linuxgl.ch435
8secureim.de479
9kif.rocks484
10aragon.sh511.5

🔗That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

New Vector raises $8.5M to accelerate Matrix/Riot/Modular

10.10.2019 00:00 — In the NewsMatthew Hodgson

Hi all,

Massive news for the Matrix ecosystem today: New Vector (the startup which the Matrix core team formed to fund development in 2017) has raised an additional $8.5M of funding in order to speed up Riot/Matrix development and expand Matrix hosting via Modular.im!

The new funding comes in the form of a Series-A equity investment in New Vector from three of the top venture capital funds in London. The round is led by Notion - a fund set up by the founders of MessageLabs, who many will know as one of the leaders in secure hosted email services. Notion's long history with email means they immediately clocked the potential of Matrix's mission to build a new open global communication network - after all, Matrix aims to provide a worthy replacement to email (and the phone network, for that matter!). Joining Notion in the round is First Minute - a fund set up by the founders of Lastminute.com (arguably the UK's most famous original dotcom), and Dawn - one of the largest SaaS tech specialist funds in Europe (famous for backing iZettle, Mimecast, Neo4J and many more).

The last funding round in Jan 2017 from Status was instrumental in stabilising the big 1.0 release of Matrix and exiting beta in June; creating the Matrix.org Foundation as a neutral custodian for the standard; stabilising and optimising Synapse; redesigning Riot’s user interface; bringing in a full-time professional UI/UX designer to the team; supporting the huge amount of encryption work required to turn on E2EE by default (cross-signing, key backups, device verification, e2e search, the pantalaimon e2e daemon etc); creating RiotX/Android; and launching the Modular.im hosting platform.

With today’s new funding, the priorities for Matrix will be:

  • Turning on end-to-end encryption by default for DMs

  • Much better support for grouping rooms into Communities

  • More anti-abuse/anti-spam mechanisms

  • Shrinking Synapse (and/or finishing Dendrite)

  • Canonical DMs (having one DM per user, and have them feel clearly distinct from ‘rooms’)

  • Extensible Profiles

  • Decentralised accounts

  • Threading

  • ...and furthering development on P2P Matrix, so users can have full control of their communications without having to run or trust a server.

On the New Vector side, this funding will support:

  • A whole new wave of UX improvements to Riot (particularly around onboarding and first time user experience).

  • Making Modular hosting as polished and powerful as possible.

  • Creating a whole new set of next-generation Modular integrations.

While New Vector’s contributions to the Matrix ecosystem can’t be ignored, it’s important to remember that the Matrix protocol and specification itself is governed and controlled by the independent and neutral Matrix.org Foundation and its extensive governance processes. We set up the Foundation very deliberately to enforce the protocol's neutrality, formalise the project's mission, goals and values and hold true to them no matter what - specifically to protect the project from conflicts of interest with commercial Matrix endeavours, including New Vector.

That said, New Vector would not be taking money from any investors if they did not believe their goals are aligned with Matrix's. To clarify:

  • Matrix exists to create an open secure decentralised communication network and protocol for the benefit of all.
  • New Vector exists to help grow Matrix and be one of many successful companies in the Matrix ecosystem.
  • Tech VCs exist to invest their money in growing companies in order to get a return when the company IPOs or gets bought.

It turns out that these goals are not incompatible if one understands that the potential of the Matrix ecosystem is directly linked to its openness and size (hint: funding sources who didn’t understand this self-selected out ;). By funding Matrix development and helping the open ecosystem and public network grow, New Vector can go provide more Matrix hosting via Modular.im and more Government & Enterprise deployments via Vector.im. Critically, other companies can and do build on top of Matrix too - and frankly the more players there are, the more valuable the network, and the more value to be shared for everyone (including New Vector). This model worked relatively well for the Web, and we believe it'll work for Matrix too.

Update: the best way to gauge the investment in New Vector is to hear it first hand from the investors. Jos from Notion is leading the round, and has a fascinating blog post (written with zero input from Matrix or New Vector folks) to explain where the investors are coming from.

Update 2: More excellend first-hand analysis from Dan at Dawn Capital, who does a really deep dive into how they see Matrix and New Vector. Another must read.

In this case, all of New Vector's new investors have a background as respected tech entrepreneurs, and everyone involved categorically understands that Matrix itself is a neutral open source project, and the mission is to help build up the whole network to be as successful as possible rather than sabotage it by constraining it in any way.

All in all, it’s great news for the ecosystem: Matrix is 5 years old now, and while the project is growing faster than ever (over 300% more active users in the last year!) - it's fair to say that we haven't moved as fast as our mainstream competition - for instance, Slack is only a year older, and Discord is a year younger(!) Obviously much of this is due to Matrix being a completely different proposition: we've been creating an open spec; multiple client codebases; multiple server codebases; the bridges; a fault tolerant decentralised network - not to mention the complexities of decentralised E2E encryption. Based on comparing with our endeavours prior to Matrix, we estimate building this stuff in an open and decentralised manner takes roughly 6 times longer.

But the project is now in a position where the foundations are solid: the protocol is out of beta, reference servers and clients are production ready, and it’s more than time to make all of this mainstream. We have to redouble our focus on user experience and ensure that we compare favourably to today’s established alternatives while staying true to Matrix’s principles. Making sure there are Matrix apps out there which provide a credible alternative to with the likes of Slack and WhatsApp (until they eventually join Matrix, of course) is what will make the difference between Matrix being a cliquey FOSS curiosity versus really being the natural successor to today’s instant messaging, email and phone networks.

In the end; Matrix needs full-time contributors in order to continue to grow, and keeping New Vector funded is a very good way to achieve that (New Vector is hiring!). (That said, if any philanthropic billionaires are reading this, the Matrix.org Foundation is actively soliciting donations to improve Matrix independently of New Vector's efforts - particularly around the areas of countering online abuse and disinformation).

In the meantime, huge thanks to Jos at Notion for believing in Matrix and leading this funding round in New Vector - and huge thanks to the other investors who saw the potential! And most of all, thank you to all those supporting Matrix, whether by donating to the Foundation, promoting and using the protocol, or contributing code to the ecosystem. You are the ones keeping the dream alive :)

You can read things from the NV angle over at https://blog.vector.im/8-5m-to-accelerate-matrix/. We hope you’re as excited as we are to open a whole new chapter as Matrix picks up yet more momentum :D

-- Matthew, Amandine, and the whole Matrix team

This Week in Matrix 2019-10-04

04.10.2019 00:00 — This Week in MatrixBen Parsons

🔗Matrix Live 🎙

🔗Dept of Spec 📜

Two MSCs have reached final comment period this week:

Last week the spec core team said they'd start focusing on 3 MSCs per week. Those were MSC2290, MSC2176 and MSC1219. The first two have entered FCP, and MSC1219 will roll over into this week.

Thus, the 3 MSCs the spec core team will be focusing on this week will be MSC2199, MSC1219 and an up and coming security-related MSC. Join us in #matrix-spec:matrix.org for related spec discussions :)

There is also a new MSC, MSC2312 describing the proposed Matrix URI scheme. This is a remake of a much older proposal. The general idea is to make a standard for Matrix URIs: matrix:.

🔗Dept of Servers 🏢

🔗Synapse 1.4.0 released

Neil offered:

Synapse This week we shipped our privacy release [1.4.0](https://matrix.org/blog/2019/10/03/synapse-1-4-0-released ) which is a [huge deal in improving user data privacy](https://matrix.org/blog/2019/09/27/privacy-improvements-in-synapse-1-4-and-riot-1-4 ). Additionally we also included a significant perf improvement which will help anyone [suffering from a build up in forward extremities](https://github.com/matrix-org/synapse/issues/1760 ).

Coming up, improved room directory perf, ironing out wrinkles in the room upgrade UX as well as a major reliability boost in the sqlite -> Postgres db porting script.

🔗Dendrite dev recommences

anoa said:

Dendrite's latest hiatus has come to a close after the privacy work had taken so much of my time. Thankfully although PR review is blocked on the dendrite team, the community have continued to submit PRs and even review each other's PRs (thanks cnly !).

Fixes this week have mostly focused around the CI. We finally got Dendrite's CI unborked (t'was borked in a half-way transition between CircleCI and Buildkite), but it's now working and faster than ever. We're also working to add the Sytest test failure results to the top of the CI window such that people can see which tests failed and the associated Dendrite logs.

Additionally we had some new and merged PRs this week! A federation fix from cnly, a change to allow Dendrite to better work in kubernetes setups by aditsachde, and a codebase fix from @manasseh:matrix.org, who's also working on another fix involving updating gomatrixserverlib to support more recent spec versions.

Some things people were asking about:

  • Progress of dendrite is tracked in Dendrite's milestones. We're currently aiming for #1 (Client-Side) Bot Hosting.
  • CI not running for community PRs - Buildkite hosts currently run multiple PRs from many different projects, so we can't trust arbitrary code to run on them quite yet. There's a project in progress to run the code in a sandboxed environment (think VM or container) to lift this restriction :)

🔗Dept of Bridges 🌉

🔗matrix-appservice-slack 1.0 released

Half-Shot offered:

Aaaaaaaaaaand shipped 🥳 https://github.com/matrix-org/matrix-appservice-slack (1.0 is out!)

Check out the announcement post, big congrats to Half-Shot for getting this out.

🔗Bridging animated stickers to mautrix-telegram

tulir said:

mautrix-telegram will soon be able to bridge animated stickers to matrix as images or videos, thanks to a pull request by Eramde.

🔗Dept of Clients 📱

🔗Riot Web 1.4.0

Big release for riot-web, check out their release: https://medium.com/@RiotChat/new-privacy-controls-for-riot-dc3661888563

🔗Riot Android and RiotX

From Benoit:

Riot-Android: 2 main things have landed on develop, to be released early next week:

  • Catching up on riot-web new Privacy Controls (choose identity server, stun server, securely compare contacts)
  • [fdroid only] A new background sync mode for notifications. You can now choose between 'optimized for battery' and 'optimize for realtime'
    • build available on buildkite

RiotX:

  • Read Markers have landed on develop (jump to last read) Focus on stabilization and bug fixes

🔗Continuum

yuforia offered:

Continuum, desktop client written in Kotlin, version 0.9.26:

  • Width of columns can be adjusted by dragging, this is a screenshot showing the mouse cursor placed between the first two columns.

continuum

  • Fix occasional out-of-bounds array access errors while calculating bounds during layout passes.

🔗FluffyWeb: Matrix client with Flutter for Web

@krille:ubports.chat offered:

As a proof of concept, I have created a little Matrix client with Flutter for Web, named FluffyWeb. This client has only basic features but it shows the possibilities of Flutter for Web and it seems to work fine so far. The client has a responsive design and should work on mobile fine too. Check it out at: https://christianpauly.gitlab.io/fluffyweb/ (Not working in Internet Explorer - I recommend the AOL Messenger in this case)

🔗Dept of SDKs and Frameworks 🧰

🔗Ruby SDK 1.4.0

Ananace said:

Just released version 1.4.0 of the Ruby SDK, the main change in this release is the addition of a naïve set of methods to replace the logger implementation. This should allow the gem to slot more easily into projects where existing logging configuration is already in place.

🔗Dept of Ops 🛠

🔗1.4.0.Mania

With the release of Synapse 1.4.0, there was a rush to get packages and containers updated, the community are always fast, but we should acknowledge Anance for having his k8s images updated within a few minutes!

🔗matrix-docker-ansible-deploy

Slavi announced:

matrix-docker-ansible-deploy has also been updated to support Synapse v1.4.0 and riot-web v1.4.0.

As always, referring to the project's changelog before upgrading is a good idea.

KUDOS to this this project! I love that I can git pull, ansible-playbook -i inventory/hosts setup.yml --tags=setup-all without really checking and my toy homeserver is automagically updated.

🔗Kubernetes

Ananace offered:

Just pushed the Kubernetes-optimized images for Synapse version 1.4.0

🔗mathijs's docker images

@mathijs:matrix.vgorcum.com reported:

I set up my own docker-hub account to push images of RC's of synapse at mvgorcum/docker-matrix that aren't built for the avhost/docker-matrix repo

🔗Dept of Bots 🤖

🔗msc name linker bot

anoa said:

I made a bot that gives the link for msc names. Code is here: https://github.com/anoadragon453/msc-chatbot

🔗Dept of Welcomes 👐

Welcome to the JRuby team who announced they will be moving their official chat to Matrix.

🔗Dept of Ping 🏓

Let's reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1matrix.tetraodon.nl317
2linuxgl.ch405
3fachschaften.org473
4glowers.club489.5
5flobob.ovh501.5
6im.leptonics.com501.5
7matrix.vgorcum.com513.5
8pztrn.name515
9matrix.markusbenning.de524
10kolosowscy.pl543

🔗Final thoughts 💭

Mastodon released v3.0.0 - check it out.

🔗That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!