🔗Project Updates

🔗Clients

🔗nheko

nheko, Qt desktop client announced release v0.4.0. From their own changelog:

• Basic member list
• Basic room settings menu
• Support for displaying stickers
• Fuzzy search for rooms

https://github.com/mujx/nheko/releases/tag/v0.4.0 for more information.

mujx - https://github.com/mujx/nheko - #nheko:matrix.org

🔗Fractal

Fractal have released v0.1.28 - couple of new features plus fixes.

Fractal of course have their in-person meeting coming up soon, and are looking forward to GSoCers getting onboard.

https://gitlab.gnome.org/World/fractal - #fractal-gtk:matrix.org

🔗neo

f0x is keeping the pace up on neo. New version (alpha0.03), new website!

Rollup of changes since last week includes

• settings menu
• mentions
• unread counts
• room invite handling
• video thumbnails

and a lot more.

f0x - https://github.com/f0x52/neo/ - #neo_client:matrix.org

🔗gomuks

Ever modest, tulir has been tearing through issues and fixes over on gomuks. I'm quite excited to have a matrix-native console client getting up to speed so fast.

tulir - https://github.com/tulir/gomuks - #gomuks:maunium.net

🔗Journal

One of the riot developers, luke has a fun side-project called Journal, this being a blogging platform built on matrix.

Says Luke:

The big news this week being that I'm going to redesign the interface to focus on the personal blog use-case, optimising for easy setup and easy blog post sharing. And hopefully push a 1.0 release that I'd be happy to use as my own personal blog.

Worth noting that the linked project page (Journal) is itself a blog using journal (the url might give you a hint of this!)

luke - https://journal.ldbco.de/#/journal/journal:ldbco.de

🔗Bridges and other projects

🔗synapse-diaspora-auth

Po Shamil reports an update for synapse-diaspora-auth. New documentation on how to integrate with mxisd, plus email syncing from diaspora.

Po Shamil - https://git.fosscommunity.in/necessary129/synapse-diaspora-auth

🔗matrix-appservice-discord

matrix-appservice-discord is now at v0.2.0-rc1.

There are several changes moving this project along, but checking out the change list I can see there were a bunch of contributors to thank, (eeeeeta, Sorunome, TravisR), which is super-cool to see.

Half-Shot - https://github.com/Half-Shot/matrix-appservice-discord

🔗GTAD

This week kitsune has been focused on 'GTAD (Generate Things from API Description)', which is a code generator for C++, taking API description in Swagger/OpenAPI as it's source. Now at version 0.5, apparently GTAD

can generate correct buildable (and runnable) code to convert data structures used in CS API between JSON and C++ - for the entirety of CS API calls. That basically means that libqmatrixclient gains (so far low-level) C++ CS API for all calls in The Spec and will follow updates to it.

This is super-exciting, especially as we are going to see discussion and progress on the spec...

kitsune - https://github.com/KitsuneRal/gtad/commits/master

🔗Riot/Web

• We shipped 0.14.2 as an incremental release
• Jitsi by default on the horizon…
• Trying to work our way through the regressions which keep stacking up
• Lots of work on improved UTs for Groups and Replies; discussion about flux stuff
• Next up is E2E verification (at last).

🔗Riot/Mobile

• Replies
• Sticker sending
• Android is now Kotlin enabled!

🔗Synapse

• Handling abuse of the depth parameter; short-term fix deployed and longer term coming along shortly.
• This destroyed progress on the algorithmic perf improvements.
• Half-Shot PRs for negotiating size limits
• Amber is inbound!

🔗Dendrite

• We're behind on PRs - sorry Thibaut :(

🔗Matrix.org Ops

• Ansible stuff is being refactored based on our experiences trying to use it in the wild status.matrix.org is coming soon!

🔗Spec

• Loads of work happening to build the Spec Proposals website, tracking workflow for all the proposals in flux and putting them into a formal RFC-style process. It should help community participation in the spec process massively whilst we finalise the longer term governance model for Matrix.org
• Also looking at publishing formal roadmaps for Synapse, Dendrite and Riot (at last!) - we have them internally these days but need to just chuck them up on the web and maintain them.
• Finally, GDPR work is in full swing.

🔗New(ish) Rooms

This section is scraped manually from #newrooms:matrix.org, though there has not been much activity there this week. Meanwhile, there are a couple of rooms suggested by Creak which deserve some love:

• #music-discovery:matrix.org, a place to discuss new music, and
• #unity3d:matrix.org, a place for unity3d fans

🔗Before we go

🔗New Core team member

Amber Brown of the Twisted project will be joining the Matrix core team in a few weeks. She'll be focusing on Synapse implementation work, and will bring a lot of Python experience with her. Having someone working full time on synapse will increase others bandwidth for homeserver and spec work.

🔗Matrix Live

Matrix Live is now available, where among other things you can see this blog post being written!

Hi all,

Many people will have noticed disruption in #matrix:matrix.org and #matrix-dev:matrix.org on Sunday, when a validation bug in Synapse was exploited which allowed a malicious event to be inserted into the room with 'depth' value that made the rooms temporarily unusable. Whilst a transient workaround was found at the time (thanks to /dev/ponies, kythyria and Po Shamil for the workaround and to Half-Shot for working on a proposed fix), we're doing an urgent release of Synapse 0.28.1 to provide a temporary solution which will mitigate the attack across all rooms in upgraded servers and un-break affected ones.  Meanwhile we have a full long-term fix on the horizon (hopefully) later this week.

This vulnerability has already been exploited in the wild; please upgrade as soon as possible.

Synapse 0.28.1 is available from https://github.com/matrix-org/synapse/releases/tag/v0.28.1 as normal.

The 'depth' parameter is used primarily as a way for servers to signal the intended cosmetic order of their events within a room (particularly when the room's message graph has gaps in it due to the server being offline, or due to users backfilling old disconnected chunks of conversation). This means that affected rooms may experience message ordering problems until a full long-term fix is provided, which we're working on currently (and tentatively involves no longer trusting 'depth' information from servers).  For full details you can see the proposal documents for the temporary fix in 0.28.1 and the options for the imminent long-term fix.

We'd like to acknowledge jzk for identifying the vulnerability, and Max Dor for providing feedback on the fixes.

As a general reminder, Synapse is still beta (as is the Matrix spec) and the federation API particularly is still being debugged and refined and is pre-r0.0.0. For the benefit of the whole community, please disclose vulnerabilities and exploits responsibly by emailing security@ or DMing someone from +matrix:matrix.org. Thanks.

🔗Changes in synapse v0.28.1 (2018-05-01)

SECURITY UPDATE

• Clamp the allowed values of event depth received over federation to be [0, 2^63 - 1]. This mitigates an attack where malicious events injected with depth = 2^63 - 1 render rooms unusable. Depth is used to determine the cosmetic ordering of events within a room, and so the ordering of events in such a room will default to using stream_ordering rather than depth (topological_ordering). This is a temporary solution to mitigate abuse in the wild, whilst a long solution is being implemented to improve how the depth parameter is used.Full details at https://docs.google.com/document/d/1I3fi2S-XnpO45qrpCsowZv8P8dHcNZ4fsBsbOW7KABI
• Pin Twisted to <18.4 until we stop using the private _OpenSSLECCurve API.

Well now, today sees the release of Synapse 0.28.0!

This release is particularly exciting as it's a major bump mainly thanks to lots and lots of contributions from the wider community - including support for running Synapse on PyPy (thanks Valodim) and lots of progress towards official Python3 support (thanks notafile)!! However, almost all the changes are under the hood (and some are quite major), so this is more a performance, bugfix and synapse internals release rather than adding many new APIs or features

As always, you can get it from https://github.com/matrix-org/synapse/releases/tag/v0.28.0 and thanks to everyone who tested the release candidates.

🔗Changes in synapse v0.28.0 (2018-04-26)

Bug Fixes:

• Fix quarantine media admin API and search reindex (PR #3130)
• Fix media admin APIs (PR #3134)

🔗Changes in synapse v0.28.0-rc1 (2018-04-24)

Minor performance improvement to federation sending and bug fixes.

(Note: This release does not include state resolutions discussed in matrix live)

Features:

• Add metrics for event processing lag (PR #3090)
• Add metrics for ResponseCache (PR #3092)

Changes:

• Synapse on PyPy (PR #2760) Thanks to @Valodim!
• move handling of auto_join_rooms to RegisterHandler (PR #2996) Thanks to @krombel!
• Improve handling of SRV records for federation connections (PR #3016) Thanks to @silkeh!
• Document the behaviour of ResponseCache (PR #3059)
• Preparation for py3 (PR #3061, #3073, #3074, #3075, #3103, #3104, #3106, #3107, #3109, #3110) Thanks to @NotAFile!
• update prometheus dashboard to use new metric names (PR #3069) Thanks to @krombel!
• use python3-compatible prints (PR #3074) Thanks to @NotAFile!
• Send federation events concurrently (PR #3078)
• Limit concurrent event sends for a room (PR #3079)
• Improve R30 stat definition (PR #3086)
• Send events to ASes concurrently (PR #3088)
• Refactor ResponseCache usage (PR #3093)
• Clarify that SRV may not point to a CNAME (PR #3100) Thanks to @silkeh!
• Use str(e) instead of e.message (PR #3103) Thanks to @NotAFile!
• Use six.itervalues in some places (PR #3106) Thanks to @NotAFile!
• Refactor store.have_events (PR #3117)

Bug Fixes:

• Return 401 for invalid access_token on logout (PR #2938) Thanks to @dklug!
• Return a 404 rather than a 500 on rejoining empty rooms (PR #3080)
• fix federation_domain_whitelist (PR #3099)
• Avoid creating events with huge numbers of prev_events (PR #3113)
• Reject events which have lots of prev_events (PR #3118)

🔗Big News

🔗GSoC students

Google Summer of Code acceptees were announced, and we're really excited to have FIVE Matrix-related projects to look forward to!

Three of the students will be mentored by matrix.org folk, and two more will work on Fractal under the GNOME org.

• Valentin D. (Zil0) will work on adding E2E to matrix-python-sdk, mentored by RichVDH
• Badrul Chowdhury will develop an alternative to GCM/FCM on Android and APNS on iOS that provides the homeserver with facilities to push notifications, mentored by David Baker
• APwhitehat will work on Dendrite (Federation and Sync Server APIs), mentored by erikj
• Eisha CHEN-YEN-SU will work on a new UI for Fractal, mentored by Daniel García Moreno
• Julian Sparber will work on Fractal UI improvements and some features like a user account settings and a spell checker. Mentored by Daniel García Moreno

🔗Fractal Hackfest 2018

Fractal Hackfest 2018 will take place next month in Strasbourg, hosted by Epitech. They'll be planning a roadmap for the year, and working on their current focus areas.

https://wiki.gnome.org/Hackfests/Fractal2018

🔗Coverage of French Matrix adoption

After the blog post on matrix.org yesterday, lots of attention developed around the news. Take a look at this Hacker News thread with lots of discussion, which has stayed on the front page for nearly 24 hours, and a megathread on Reddit/r/linux: nearly 1000 upvotes and growing.

🔗Project Updates

🔗Clients

🔗libqmatrixclient

kitsune reports a new release of libqmatrixclient, v0.2.1, this is mostly a bugfix release, fully backwards-compatible with v0.2. Check out the release notes for details.

kitsune - https://github.com/QMatrixClient/libqmatrixclient - #qmatrixclient:matrix.org

🔗Quaternion

Staying in QMatrix-land, there is a pre-release of Quaternion available. Take a look at v0.0.9, improvements include "redactions, file downloading, room creation and settings editing, better timeline visualisation and much more"!

kitsune - https://github.com/QMatrixClient/Quaternion - #qmatrixclient:matrix.org

🔗neo

f0x has been beavering away on neo, and has announced that Alpha 0.01 is available now. neo is a webclient in React, it feels really light and fast. I asked f0x about the uptick in work, and he mentioned he got started with React recently, but likes it a lot.

f0x - https://github.com/f0x52/neo/ - #neo_client:matrix.org

🔗Fractal

Fractal released 0.1.27 with many new features and bugfixes. Lot's going on but I'm most excited to see Markdown support land.

https://gitlab.gnome.org/World/fractal - #fractal-gtk:matrix.org

🔗Bots

🔗matrix-trello-bot

Says TravisR: "I've whipped together a simple Trello notification bot for matrix: matrix-trello-bot. Future plans include the ability to create/update/delete cards and more granular control of what is notified about. Currently it acts very similar to how the Github notifications bot works."

TravisR - https://github.com/turt2live/matrix-trello-bot - #trellobot:t2bot.io

🔗Others

🔗Matrix DSL

MTRNord is working on a "matrix DSL", basically a config file language that generates a project Template in multiple languages. The project is just now at the discussion stage, so join #matrix_dsl:matrix.ffslfl.net to check out what's going on. (DSL = Domain Specific Language.)

MTRNord - #matrix_dsl:matrix.ffslfl.net

🔗urllib-requests-adapter

Says Coffee: "urllib-requests-adapter has been updated to work with matrix-python-sdk 0.2.0. urllib-requests-adapter is a lightweight replacement for requests and its dependencies, currently standing at 126 SLOC."

Coffee - https://github.com/Matrixcoffee/urllib-requests-adapter

🔗mxisd

mxisd, the Identity Server from kamax, saw a v1.0.2 release, following the big one-point-zero last week. Changes include de-duplicated directory search results, plus bugfixes.

Max - https://github.com/kamax-io/mxisd

🔗matrix-stfu

Missed this one last week, but matrix-stfu from xwiki is a message filtering tool released recently, it can "mass remove everything which was said by a particular user in a particular room". XWiki use matrix as their internal chat system for the ~30 people at the company.

xwiki - https://github.com/xwiki-labs/matrix-stfu - #xwiki:matrix.xwiki.com

🔗Riot/Web

• We're committing to 2-weekly releases come what may, to avoid another massive gap like the one between 0.13 and 0.14.
• RCs will get cut starting from Wednesday; actual release then happens on the Monday having given folks a chance to test the RC on /staging (and to iterate on the RCs).
• Obviously this is flexible if we need to rush out fixes sooner.
• On that note, 0.14.2-rc1 was cut on Wednesday! Please test it at riot.im/staging. It's mainly bugfixes but also the full relayering between riot-web and matrix-react-sdk.
• Dave's working on finally hooking in Jitsi as the default conferencing system
• t3chguy's been working on Replies, which continue to look awesome
• Next up: E2E cross-signing (at last!!!!)

🔗Riot/Mobile

• New releases are out! Mainly preparing for sticker viewing and trying to fix the Android push notification situation. PLEASE TELL US IF YOU ARE STILL HAVING ANDROID PUSH NOTIFICATION PROBLEMS!
• Lots of review of Android by Benoit - adding in Kotlin support as of today and establishing a formal roadmap for Android work (we'll show the blog post when we have it)
• Lots of Matrix-for-French-Government stuff.

🔗Synapse

• Synapse 28.0 was released! A major bump mainly thanks to lots and lots of contributions from the wider community.
• Massive experimental work on GDPR in progress - doing the thought experiment of pseudonymising MXIDs throughout Matrix on a per-room basis so that it's possible to redact MXIDs in the event of a “right-to-erasure” GDPR event. Rich vdH is leading the work.
• The good news is that introducing an MXID abstraction layer like this could help us enormously with some of Matrix's longest-term architectural issues - i.e. account migration and portability; improving on PERSPECTIVEs for managing server identity; future support for P2P Matrix; solving the domain name reuse problem; etc.
• The bad news is that it would obviously be a very significant change to the Matrix spec, although we'd be doing it in such a way which minimises impact to client implementers and keep the CS API looking as similar as possible.
• We're not sure whether we need to rush this through or not yet; still waiting for final GDPR clarification from lawyers, but it feels like this might be a good opportunity to force us to finally tackle some of these harder problems.
• Meanwhile, Erik's Delta State Resolution algorithm work is continuing well; we're hoping to finish & merge it asap in order to get back headroom on the server.
• Thankfully the matrix.org synapse itself has been relatively stable this week, other than being completely overloaded causing Freenode to be almost unusable.
• We've started using Ansible in production, although the playbooks need some iteration before we're fully happy to announce them & recommend folks use them as an official way of running Synapse in production.

🔗try-matrix-now

There is a update out on try-matrix-now, matrix.org's central listing of projects. Try filtering and see whether benpa mangled the metadata for your project. Submit any changes as markdown PRs on the repo.

🔗Spec

benpa is currently working on a Proposals page for the Spec to properly stack spec proposal status, at last!

🔗Articles around the web

🔗Riot: A Distributed Way of Having IRC and VOIP Client and Home Server

uhoreg pointed to some charming coverage over on https://itsfoss.com: Riot: A Distributed Way of Having IRC and VOIP Client and Home Server, by Shirish. The article covers some details about riot-web and the open source ethos of Matrix, but my favourite quote by far:

"Without Matrix, Riot would be like a body without a soul."

Shirish - https://itsfoss.com/riot-desktop/

🔗Service notifs with Matrix

Half-Shot shared an article he wrote, Service notifs with Matrix, about using Matrix and Riot to deliver automated notifications to different types of end-user at his company.

Half-Shot - https://dev.to/halfshot/service-notifs-with-matrix-3fb5

🔗New Rooms roundup

• #matrix_dsl:matrix.ffslfl.net As mentioned above, MTRNord is looking at creating a DSL for matrix. In his words: "A room to discuss about making a Matrix DSL to allow non coders to write simple as and bots. I would love to see some people discussing about syntax with me as this is my very first DSL and very first time trying JetBrains MSP. Anyone with Ideas what it should allow to do and how to have the syntax a welcome to join (and people who want to follow of course too)"
• #trellobot:t2bot.io (from above) TravisR: "a discussion/developer room for Matrix Trello Bot (https://github.com/turt2live/matrix-trello-bot). This is a bot that notifies rooms of changes to tracked Trello boards, similar to how the Github bot works. Future enhancements include being able to create, update, and delete cards from within Matrix."

🔗Lastly...

Matrix Live - Season 2, Episode 17: Apr 27th is now available!

This Week in Matrix is printed fresh every week! There will be another post before you know it, so if you'd like to be included join us in #twim:matrix.org and let us know what you've been working on. See you next week!

Hi folks,

We're incredibly excited that the Government of France has confirmed it is in the process of deploying a huge private federation of Matrix homeservers spanning the whole government, and developing a fork of Riot.im for use as their official secure communications client! The goal is to replace usage of WhatsApp or Telegram for official purposes.

It's a unbelievably wonderful situation that we're living in a world where governments genuinely care about openness, open source and open-standard based communications - and Matrix's decentralisation and end-to-end encryption is a perfect fit for intra- and inter-governmental communication.  Congratulations to France for going decentralised and supporting FOSS! We understand the whole project is going to be released entirely open source (other than the operational bits) - development is well under way and an early proof of concept is already circulating within various government entities.

I'm sure there will be more details from their side as the project progresses, but meanwhile here's the official press release, and an English translation too. We expect this will drive a lot of effort into maturing Synapse/Dendrite, E2E encryption and matrix-{'{'}react,ios,android{'}'}-sdk, which is great news for the whole Matrix ecosystem! The deployment is going to be speaking pure Matrix and should be fully compatible with other Matrix clients and projects in addition to the official client.

So: exciting times for Matrix.  Needless to say, if you work on Open Government projects in other countries, please get in touch - we're seeing that Matrix really is a sweet spot for these sort of use cases and we'd love to help get other deployments up and running.  We're also hoping it's going to help iron out many of the UX kinks we have in Riot.im today as we merge stuff back. We'd like to thank DINSIC (the Department responsible for the project) for choosing Matrix, and can't wait to see how the project progresses!

English Translation:

🔗The French State creates its own secure instant messenger

By the summer of 2018, the French State will have its own instant messenger, an alternative to WhatsApp and Telegram.

It will guarantee secure, end-to-end encrypted conversations without degradation of the user experience. It will be compatible with any mobile device or desktop, state or personal. In fact until now the installation of applications like WhatsApp or Telegram was not possible on professional mobile phones, which hindered easy sharing of information and documents.

Led by the Interministerial Department of State Digital, Information and Communication Systems (DINSIC), the project is receiving contributions from the National Agency for Information System Security (ANSSI), the IT Directorship (DSI) of the Armed Forces and the Ministry of Europe and Foreign Affairs.

The tool developed is based on open source software (Riot) that implements an open standard (Matrix). Powered by a Franco-British startup (New Vector), and benefiting from many contributions, this communication standard has already caught the attention of other states such as the Netherlands and Canada, with whom DINSIC collaborates closely.

The Matrix standard and its open source software are also used by private companies such as Thales, which has driven the teams to come together to ensure the interoperability of their tools and cooperate in the development of free and open source software.

After 3 months of development for a very limited cost, this tool is currently being tested in the State Secretary for Digital, DINSIC and in the IT departments of different ministries. It should be rolled out during the summer in administrations and cabinets.

"With this new French solution, the state is demonstrating its ability to work in an agile manner to meet concrete needs by using open source tools and very low development costs. Sharing information in a secure way is essential not only for companies but also for a more fluid dialogue within administrations." - Mounir Mahjoubi, Secretary of State to the Prime Minister, in charge of Digital.

🔗Project Updates

🔗Drupal matrix_api module

Updated the Drupal matrix_api module to pass the auth token as a header instead of as a GET param... the GET param was leaking the token if there was an http exception thrown.

(John) - https://cgit.drupalcode.org/matrix_api

🔗mxisd

v1.0.0 has been released and is now stable, making it the first community Identity server and first open federation one.

(Max) - https://github.com/kamax-io/mxisd

🔗nheko

0.3.x of nheko was released! There is progress towards e2e encryption in mujx/mtxclient. Once this encryption is working there, it can eventually be included in nheko.

(mujx) - https://github.com/mujx/nheko

🔗Fractal

Fractal 0.1.26 was released this week, which includes an animated scroll to bottom animation, fixed MacOS compilation, and bug fixes. Since then markdown support and many other commits have been spotted...

https://gitlab.gnome.org/World/fractal/

🔗libqmatrixclient

kitsune released 0.2 of libqmatrixclient a couple of weeks ago, which he says is "the first one more or less functional and stable"!

Highlights include redactions support, files transfer (both ways), rooms creation, account data (tags, direct chat flags). Also there are a few things making libqmatrixclient clients stand out compared to, e.g., Riot: one can set per-room names/avatars, and also the best estimate for the number of unread messages can be taken from the library and shown in the client application."

(kitsune) - https://github.com/QMatrixClient/libqmatrixclient

🔗matrix-monitor-bot

#monitorbot:t2bot.io is a bot to measure latency between homeservers (as perceived by users).

It's able to report metrics to Prometheus and has it's own built-in dashboard (see https://lag.t2bot.io/ for some horrendous times). In the coming weeks it'll become more production-ready, however in the meantime if people are looking to help iron out the scalability issues, please run an instance on your homeserver.

(TravisR) - https://github.com/turt2live/matrix-monitor-bot

🔗uMatriks

thrrgilag wrote on behalf of uMatriks, to say that v0.10 was published to the openstore this last week which now includes unread counts for rooms. Those not already familiar, uMatriks is a native matrix client being developed for Ubuntu Touch.

https://github.com/uMatriks/uMatriks

🔗Construct

There looks to be lots of activity on Construct (jzk's fork of the charybdis ircd which adds in a C++ homeserver implementation) based on the commit log at https://github.com/jevolk/charybdis. jzk's work there has been highlighting some of synapse's nastier problems (e.g. thundering presence bugs), and it looks like the federation support is progressing steadily, including handling EDUs (presence & typing data). (jzk) - https://github.com/jevolk/charybdis

🔗matrix-puppet-maxs

tyler shared his "horrifying" project, for people that want to use Matrix to send an SMS via an XMPP bridge to their Android phone, in order to provide SMS and call notifications inside Riot.

(tyler) - https://github.com/tfreedman/matrix-puppet-maxs

🔗matrix-register-bot

krombel built a bot to have a two step registration to synapse, check it out:

(krombel) - https://github.com/krombel/matrix-register-bot

🔗picard

Cadair reported that he and SolarDrew have written a bot to bridge a whole slack team into a matrix community. It basically handles all the room admin for you including:

• Invite the slack bot to the channel
• Creating matrix rooms
• Setting users as admins
• Setting room name and room avatar and as publicly joinable
• Setting up the link with the slack AS
• Creating and adding rooms to the community
• Inviting all members of the community to the rooms

It does these things on a 1 min cron, to pick up new channels as they are created on the slack side.

(Cadair) - https://github.com/SolarDrew/skill-picard

🔗opsdroid connector

Cadair also mentioned that picard uses the opsdroid connector for matrix, also written by himself and SolarDrew, and has had recent updates.

(Cadair) - https://github.com/opsdroid/connector-matrix

🔗matrix-python-sdk

Release v0.2.0 of the python sdk. Highlights from this release are a workaround for https://github.com/matrix-org/synapse/issues/3054 and use of connection pooling via a requests session.

(&Adam) - https://github.com/matrix-org/matrix-python-sdk

🔗matrix-twitch-bridge

MTRNord is at the early stages of working on a twitch bridge for Matrix, written in golang.

(MTRNord) - https://github.com/Nordgedanken/matrix-twitch-bridge

🔗IdleRPG

remaeus wrote in to say "we (+rpg:verse.im) just launched an initial build of #idlerpg:verse.im (now also available at #idlerpg:matrix.org) — an automated RPG that rewards you for staying online (via the presence API). It's an initial test case for prototyping out a bot framework called "doorman" with multi-service support, but focused on Matrix as we are using it as a "backbone" for our p2p app framework."

(remaeus) - https://github.com/FabricLabs/doorman

🔗URI protocol scheme

kitsune started working on a design proposal for the Matrix URI scheme, with the ultimate goal to have the scheme passed through IANA so that we're no worse than IPFS and others. The document is here (alpha quality but comments are already welcome): https://docs.google.com/document/d/18A3ZRgGR-GLlPXF_VIHxywWiX1vpMvNfAU6JCnNMVuQ/edit

There is also a spec proposal room: #uri-scheme-proposal:matrix.org.

(kitsune)

🔗Riot/Web

• The Great Relayering landed - almost all of Riot/Web's code is now gathered together in one sensible place (matrix-react-sdk). No more duplicate PRs or CSS & JS living in different repos!
• Folks with open PRs against the riot-web repository will need to reopen them against matrix-react-sdk (or more likely, merge them into their existing equivalents on matrix-react-sdk)
• Michael (t3chguy's) Replies work has almost landed and is looking fantastic!
• Next up: improving test coverage slightly; fixing regressions in 0.14; and a few other minor bits and bobs before we land 0.15.

🔗Riot/Mobile

• New releases on the horizon for both iOS & Android
• Welcomed Benoit to the Android team, acting as lead android dev!
• Finalising all new UX for configuring push notification permissions…
• Final touches for Sticker viewing (including animated stickers!)
• Fix embarrassing crash on iOS on certain shapes of Riot/Web URLs...
• Supporting French Government Matrix activity

🔗Synapse

• Operational firefighting continues: stopping malformed events from impacting federation; investigating thundering herds of presence EDUs; trying to keep Freenode joins from lagging;
• Meanwhile, Erik has been working hard on the problem of Incremental State Resolution (the big algorithmic win which could massively improve HS performance, especially for Synapse); this is looking promising and has ended up taking the form of a formal mathematical proof that the proposed algorithm works: https://matrix.org/_matrix/media/v1/download/jki.re/hSmkLkFGVUnrXjtYcjsCUsyS (aka https://github.com/matrix-org/synapse/pull/3122) is where that's happening! Warning: contains maths.
• ...and also wrote a Rust testjig for state resolution to help reason about it: https://github.com/erikjohnston/rust-matrix-state
• Official Ansible playbook repository dev is progressing on the develop branch at https://github.com/matrix-org/matrix-ansible thanks to massive work from Michael K. We'll blog properly about this when ready for use in earnest.
• Dave has been doing an interesting experiment in syncing user profile data from HSes into ISes (useful for folks running a private federation with its own IS who want to use the IS as a global user directory); almost complete - we'll see how this goes.
• GDPR work is in progress. We're seeing how far we can push the envelope on improving privacy, including a very interesting thought experiment on pseudonyimsing MXIDs to provide 'perfect' right-to-erasure. vdH is on the case. Warning: this may well require a major federation protocol bump if we go through with it.
• More Python3 port PRs are landing from NotAFile - thanks! They're getting slightly lost behind everything above.

🔗Dendrite

• Still stuck behind Synapse work.
• Barring disasters we have at least 2 new folks starting fulltime in the core team (both synapse & dendrite) to help backend progress at the end of May however!
• Lots of PRs coming in from APWhitehat though - thanks!!

🔗Spec

• uhoreg has been looking at the .well-known URI proposal again
• Spec work on the core team is currently stuck behind Riot/Web and Synapse work, sadly, on the basis that burning implementation problems are more visible and harming to the project than burning spec problems.

🔗Matrix in the News

Tom's Hardware provided a great article on about Matrix in the French Government.

Next INpact produced two articles, one from last week introducing a project from the French government to use an open messaging platform, then a follow up provided an extremely thorough story covering Riot & Matrix following the announcement of the French Government encrypted messaging project. (Article is in French, but will Google Translate nicely!)

Huge thank you to nouts who produced a great intro to Riot and Matrix! Check it out! (Note that is also in French.)

🔗New Rooms roundup

• Announcing room for all pastafarians out there... #fsm:disroot.org
• FIA Formula 1 discussion room #formula1:disroot.org
• Room for Warsaw (Poland) residents (and others) #warsaw:disroot.org
• #monitorbot:t2bot.io is the room for TravisR's matrix-monitor-bot
• #thisweekinmatrix:matrix.org so meta

🔗And Finally...

Check out Matrix Live, now available at https://youtu.be/EstVaVUWkdw

Thanks for reading the very first This Week in Matrix! There will be another post next week, so if you'd like to be included join us in #twim:matrix.org. See you next week!

Today we release Synapse 0.27.3!

Hot on the heels of 0.27.2, notable changes include API support for joining groups/communities as well as a major bug fix (#3082), that is particularly important for those upgrading for the first time in a while. Also new metrics and phone home stats. Phone home stats include better visibility of system usage so we can have a better idea of how synapse's RAM and CPU is behaving in the wild. Also, recording the number of users on the server who have been using it for at least 30 days.

Phone home stats are entirely optional and can be enabled/disabled by setting "report_ stat s" in homeserver.yaml.  Please consider enabling phone home stats if you currently have not done so - this data is really important to us in improving Matrix as a whole (and justifying future funding for Matrix.org).

As always, you can get it from https://github.com/matrix-org/synapse/releases/tag/v0.27.3 and thanks to everyone who tested the release candidates.

🔗Changes in synapse v0.27.3 (2018-04-11)

Bug fixes:

• URL quote path segments over federation (#3082)

🔗Changes in synapse v0.27.3-rc2 (2018-04-09)

v0.27.3-rc1 used a stale version of the develop branch so the changelog overstates the functionality. v0.27.3-rc2 is up to date, rc1 should be ignored.

🔗Changes in synapse v0.27.3-rc1 (2018-04-09)

Notable changes include API support for joinability of groups. Also new metrics and phone home stats. Phone home stats include better visibility of system usage so we can tweak synpase to work better for all users rather than our own experience with matrix.org. Also, recording 'r30' stat which is the measure we use to track overall growth of the Matrix ecosystem. It is defined as:-

Counts the number of native 30 day retained users, defined as:-

• Users who have created their accounts more than 30 days
• Where last seen at most 30 days ago
• Where account creation and last_seen are > 30 days"

Features:

• Add joinability for groups (PR #3045)
• Implement group join API (PR #3046)
• Add counter metrics for calculating state delta (PR #3033)
• R30 stats (PR #3041)
• Measure time it takes to calculate state group ID (PR #3043)
• Add basic performance statistics to phone home (PR #3044)
• Add response size metrics (PR #3071)
• phone home cache size configurations (PR #3063)

Changes:

• Add a blurb explaining the main synapse worker (PR #2886) Thanks to @turt2live!
• Replace old style error catching with 'as' keyword (PR #3000) Thanks to @NotAFile!
• Use .iter* to avoid copies in StateHandler (PR #3006)
• Linearize calls to _generate_user_id (PR #3029)
• Remove last usage of ujson (PR #3030)
• Use simplejson throughout (PR #3048)
• Use static JSONEncoders (PR #3049)
• Remove uses of events.content (PR #3060)
• Improve database cache performance (PR #3068)

Bug fixes:

• Add room_id to the response of rooms/{'{'}roomId{'}'}/join (PR #2986) Thanks to @jplatte!
• Fix replication after switch to simplejson (PR #3015)
• Fix replication after switch to simplejson (PR #3015)
• 404 correctly on missing paths via NoResource (PR #3022)
• Fix error when claiming e2e keys from offline servers (PR #3034)
• fix tests/storage/test_user_directory.py (PR #3042)
• use PUT instead of POST for federating groups/m.join_policy (PR #3070) Thanks to @krombel!
• postgres port script: fix state_groups_pkey error (PR #3072)

We released Synapse v0.27.2 today (the first stable release in the 0.27.x series) - it contains loads of work since Synapse v0.26 back in January.  The main highlights are:

• All the perf improvements which we've been landing as we race to keep the matrix.org homeserver in the face of ever-expanding traffic levels over the last few months
• Support for custom storage providers for media repository.
• Ability to limit the email addresses allowed to register on your HS, and ability to limit the homeservers your homeserver is allowed to federate with
• All new purge API - letting you purge history by date as well as by event (and having a nice new async way of doing it)
• Make search work again!!! (by switching from GIST to GIN indexes)

And a few release notes worth calling out:

• The common case for running Synapse is not to run separate workers, but for those that do, be aware that synctl no longer starts the main synapse when using -a option with workers. A new worker file should be added with worker_app: synapse.app.homeserver.
• This release also begins the process of renaming a number of the metrics reported to prometheus. See docs/metrics-howto.rst  Note that the v0.28.0 release will remove the deprecated metric names.

As always, you can get it from https://github.com/matrix-org/synapse/releases/tag/v0.27.2

thanks for flying Matrix!

🔗Changes in synapse v0.27.2 (2018-03-26)

Bug fixes:

• Fix bug which broke TCP replication between workers (PR #3015)

🔗Changes in synapse v0.27.1 (2018-03-26)

Meta release as v0.27.0 temporarily pointed to the wrong commit

🔗Changes in synapse v0.27.0 (2018-03-26)

No changes since v0.27.0-rc2

🔗Changes in synapse v0.27.0-rc2 (2018-03-19)

Pulls in v0.26.1

Bug fixes:

• Fix bug introduced in v0.27.0-rc1 that causes much increased memory usage in state cache (PR #3005)

🔗Changes in synapse v0.27.0-rc1 (2018-03-14)

The common case for running Synapse is not to run separate workers, but for those that do, be aware that synctl no longer starts the main synapse when using -a option with workers. A new worker file should be added with worker_app: synapse.app.homeserver.

This release also begins the process of renaming a number of the metrics reported to prometheus. See docs/metrics-howto.rst <docs/metrics-howto.rst#block-and-response-metrics-renamed-for-0-27-0>_. Note that the v0.28.0 release will remove the deprecated metric names.

Features:

• Add ability for ASes to override message send time (PR #2754)
• Add support for custom storage providers for media repository (PR #2867, #2777, #2783, #2789, #2791, #2804, #2812, #2814, #2857, #2868, #2767)
• Add purge API features, see docs/admin_api/purge_history_api.rst <docs/admin_api/purge_history_api.rst>_ for full details (PR #2858, #2867, #2882, #2946, #2962, #2943)
• Add support for whitelisting 3PIDs that users can register. (PR #2813)
• Add /room/{'{'}id{'}'}/event/{'{'}id{'}'} API (PR #2766)
• Add an admin API to get all the media in a room (PR #2818) Thanks to @turt2live!
• Add federation_domain_whitelist option (PR #2820, #2821)

Changes:

• Continue to factor out processing from main process and into worker processes. See updated docs/workers.rst <docs/metrics-howto.rst>_ (PR #2892 - #2904, #2913, #2920 - #2926, #2947, #2847, #2854, #2872, #2873, #2874, #2928, #2929, #2934, #2856, #2976 - #2984, #2987 - #2989, #2991 - #2993, #2995, #2784)
• Ensure state cache is used when persisting events (PR #2864, #2871, #2802, #2835, #2836, #2841, #2842, #2849)
• Change the default config to bind on both IPv4 and IPv6 on all platforms (PR #2435) Thanks to @silkeh!
• No longer require a specific version of saml2 (PR #2695) Thanks to @okurz!
• Remove verbosity/log_file from generated config (PR #2755)
• Add and improve metrics and logging (PR #2770, #2778, #2785, #2786, #2787, #2793, #2794, #2795, #2809, #2810, #2833, #2834, #2844, #2965, #2927, #2975, #2790, #2796, #2838)
• When using synctl with workers, don't start the main synapse automatically (PR #2774)
• Minor performance improvements (PR #2773, #2792)
• Use a connection pool for non-federation outbound connections (PR #2817)
• Make it possible to run unit tests against postgres (PR #2829)
• Update pynacl dependency to 1.2.1 or higher (PR #2888) Thanks to @bachp!
• Remove ability for AS users to call /events and /sync (PR #2948)
• Use bcrypt.checkpw (PR #2949) Thanks to @krombel!

Bug fixes:

• Fix broken ldap_config config option (PR #2683) Thanks to @seckrv!
• Fix error message when user is not allowed to unban (PR #2761) Thanks to @turt2live!
• Fix publicised groups GET API (singular) over federation (PR #2772)
• Fix user directory when using user_directory_search_all_users config option (PR #2803, #2831)
• Fix error on /publicRooms when no rooms exist (PR #2827)
• Fix bug in quarantine_media (PR #2837)
• Fix url_previews when no Content-Type is returned from URL (PR #2845)
• Fix rare race in sync API when joining room (PR #2944)
• Fix slow event search, switch back from GIST to GIN indexes (PR #2769, #2848)

Hi all,

We just rushed out an urgent hotfix release for Synapse 0.26.1, addressing a nasty bug in the ujson library which causes it to misbehave badly in the presence of JSON containing very large >64-bit integers.  Anyone whose synapses are currently filling up with "Value too big!" errors will want to upgrade immediately from https://github.com/matrix-org/synapse/releases/tag/v0.26.1.

Sorry for the inconvenience.

🔗Changes in synapse v0.26.1 (2018-03-15)

Bug fixes:

• Fix bug where an invalid event caused server to stop functioning correctly, due to parsing and serializing bugs in ujson library.

Hi all,

Heads up that we made an emergency release of Riot/Web 0.13.5 a few hours ago to fix a XSS vulnerability found and reported by walle303 - many thanks for disclosing it responsibly.

Please upgrade to Riot/Web 0.13.5 asap. If you're using riot.im/app or riot.im/develop this simply means hitting Refresh; otherwise please upgrade your Riot deployment as soon as possible. Alpine, Debian and Fedora/RPM packages are already updated - huge thanks to the maintainers for the fast turnaround.

The issue lies in the relatively obscure external_url feature, which lets bridges specify a URL for bridged events, letting Riot/Web users link through to the 'original' event (e.g. a twitter URL on a bridged tweet).  The option is hidden in a context menu and labelled "Source URL", and is only visible on events which have the external_url field set.  Unfortunately Riot/Web didn't sanitise the URL correctly, allowing a malicious URL to be injected - and this has been the case since the feature landed in Riot 0.9.0 (Nov 2016).

If you're not able to upgrade to Riot/Web 0.13.5 for some reason, then please do not click on the 'Source URL' feature on the event context menu:

Apologies for the inconvenience,

thanks,

Matthew