Synapse 0.25 is out… as is Matrix Specification 0.3(!!!)

Hi all,

Today is a crazy release day here – not only do we have Synapse 0.25, but we’ve also made a formal release of the Matrix Specification (CS API) for the first time in 16 months!

Matrix CS API 0.3

Talking first about the spec update: the workflow of the Matrix spec is that new experimental features get added to an /unstable API prefix, and then whenever we release the Matrix spec, these get moved over to being part of the /r0 prefix (or whatever version we happen to be on).  We’ve been very constrained on manpower to work on the spec over the last ~18 months, but we’ve been keeping it up-to-date on a best effort basis, with a bit of help from the wider community.   As such, this latest release does not contain all the latest APIs (and certainly not experimental ones like Groups/Communities which are still evolving), but it does release all of the unstable ones which we’ve managed to document and which are considered stable enough to become part of the ‘r0’ prefix.  Going forwards, we’re hoping that the wider community will help us fill in the remaining gaps (i.e. propose PRs against the matrix-org/matrix-doc repository to formalise the various spec drafts flying around the place) – and we’re also hoping (if/when funding crisis is abated) to locate full-time folk to work on the spec.

The full changelog for 0.3 of the spec is:

  • Breaking changes:
    • Change the rule kind of .m.rule.contains_display_name from underride to override. This works with all known clients which support push rules, but any other clients implementing the push rules API should be aware of this change. This makes it simple to mute rooms correctly in the API (#373).
    • Remove /tokenrefresh from the API (#395).
    • Remove requirement that tokens used in token-based login be macaroons (#395).
  • Changes to the API which will be backwards-compatible for clients:
    • Add filename parameter to POST /_matrix/media/r0/upload (#364).
    • Document CAS-based client login and the use of m.login.token in /login (#367).
    • Make origin_server_ts a mandatory field of room events (#379).
    • Add top-level account_data key to the responses to GET /sync and GET /initialSync (#380).
    • Add is_direct flag to POST /createRoom and invite member event. Add ‘Direct Messaging’ module (#389).
    • Add contains_url option to RoomEventFilter (#390).
    • Add filter optional query param to /messages (#390).
    • Add ‘Send-to-Device messaging’ module (#386).
    • Add ‘Device management’ module (#402).
    • Require that User-Interactive auth fallback pages call window.postMessage to notify apps of completion (#398).
    • Add pagination and filter support to /publicRooms. Change response to omit fields rather than return null. Add estimate of total number of rooms in list. (#388).
    • Allow guest accounts to use a number of endpoints which are required for end-to-end encryption. (#751).
    • Add key distribution APIs, for use with end-to-end encryption. (#894).
    • Add m.room.pinned_events state event for rooms. (#1007).
    • Add mention of ability to send Access Token via an Authorization Header.
    • New endpoints:
      • GET /joined_rooms (#999).
      • GET /rooms/{roomId}/joined_members (#999).
      • GET /account/whoami (#1063).
      • GET /media/{version}/preview_url (#1064).
  • Spec clarifications:
    • Add endpoints and logic for invites and third-party invites to the federation spec and update the JSON of the request sent by the identity server upon 3PID binding (#997)
    • Fix “membership” property on third-party invite upgrade example (#995)
    • Fix response format and 404 example for room alias lookup (#960)
    • Fix examples of m.room.member event and room state change, and added a clarification on the membership event sent upon profile update (#950).
    • Spell out the way that state is handled by POST /createRoom (#362).
    • Clarify the fields which are applicable to different types of push rule (#365).
    • A number of clarifications to authentication (#371).
    • Correct references to user_id which should have been sender (#376).
    • Correct inconsistent specification of redacted_because fields and their values (#378).
    • Mark required fields in response objects as such (#394).
    • Make m.notice description a bit harder in its phrasing to try to dissuade the same issues that occurred with IRC (#750).
    • GET /user/{userId}/filter/{filterId} requires authentication (#1003).
    • Add some clarifying notes on the behaviour of rooms with no m.room.power_levels event (#1026).
    • Clarify the relationship between username and user_id in the /register API (#1032).
    • Clarify rate limiting and security for content repository. (#1064).

…and you can read the spec itself of course over at https://matrix.org/docs/spec.  It’s worth noting that we have slightly bent the rules by including three very minor ‘breaking changes’ in 0.3, but all for features which to our knowledge nobody is depending on in the wild.  Technically this should mean bumping the major version prefix (i.e. moving to r1), but given how minor and nonimpacting these are we’re turning a blind eye this time.

Meanwhile, Synapse 0.25 is out!

This is a medium-sized release; the main thing being to support configurable room visibility within groups (so that whenever you add a room to a group, you’re not forced into sharing their existence with the general public, but can choose to just tell group members about them).  There’s also a bunch of useful bug fixes and some performance improvements, including lots of contributions from the community this release (thank you!).  Full release notes are:

Changes in synapse v0.25.0 (2017-11-15)

Bug fixes:

  • Fix port script (PR #2673)
Changes in synapse v0.25.0-rc1 (2017-11-14)

Features:

Changes:

  • Ignore tags when generating URL preview descriptions (PR #2576)
    Thanks to @maximevaillancourt!
  • Register some /unstable endpoints in /r0 as well (PR #2579) Thanks to
    @krombel!
  • Support /keys/upload on /r0 as well as /unstable (PR #2585)
  • Front-end proxy: pass through auth header (PR #2586)
  • Allow ASes to deactivate their own users (PR #2589)
  • Remove refresh tokens (PR #2613)
  • Automatically set default displayname on register (PR #2617)
  • Log login requests (PR #2618)
  • Always return is_public in the /groups/:group_id/rooms API (PR #2630)
  • Avoid no-op media deletes (PR #2637) Thanks to @spantaleev!
  • Fix various embarrassing typos around user_directory and add some doc. (PR
    #2643)
  • Return whether a user is an admin within a group (PR #2647)
  • Namespace visibility options for groups (PR #2657)
  • Downcase UserIDs on registration (PR #2662)
  • Cache failures when fetching URL previews (PR #2669)

Bug fixes:

  • Fix port script (PR #2577)
  • Fix error when running synapse with no logfile (PR #2581)
  • Fix UI auth when deleting devices (PR #2591)
  • Fix typo when checking if user is invited to group (PR #2599)
  • Fix the port script to drop NUL values in all tables (PR #2611)
  • Fix appservices being backlogged and not receiving new events due to a bug in
    notify_interested_services (PR #2631) Thanks to @xyzz!
  • Fix updating rooms avatar/display name when modified by admin (PR #2636)
    Thanks to @farialima!
  • Fix bug in state group storage (PR #2649)
  • Fix 500 on invalid utf-8 in request (PR #2663)

Finally…

If you haven’t noticed already, Riot/Web 0.13 is out today, as is Riot/iOS 0.6.2 and Riot/Android 0.7.4.  These contain massive improvements across the board – particularly mainstream Communities support at last on Riot/Web; CallKit/PushKit on Riot/iOS thanks to Denis Morozov (GSoC 2017 student for Matrix) and Share Extension on iOS thanks to Aram Sargsyan (also GSoC 2017 student!); and End-to-end Key Sharing on Riot/Android and a full rewrite of the VoIP calling subsystem on Android.

Rather than going on about it here, though, there’s a full write-up over on the Riot Blog.

 

And so there you go – new releases for eeeeeeeeveryone!  Enjoy! :)

–Matthew, Amandine & the team.

TADHack Global 2017 and THE Port 2017

TADHack Global 2017

At the end of September, TADHack Global was held where almost 150 teams spent their weekends hacking towards the $45k total prize money up for grabs. Luke spent the final day of the hack talking to teams hacking at IDEALondon in Shoreditch, meeting a few Matrix enthusiasts and long-time collaborators.

Out of 10 hacks, 2 of 4 local winners won prizes locally and went on to be global winners alongside 6 other teams using Matrix as part of their hacks. Checkout the TADHack London Wrap-up for details on all of the awesome hacks, especially Aviral Dasgupta‘s Pushtime and Polite.ai.

Well done to everyone who took part, and a special thanks to those flying Matrix :)

THE Port 2017

The following weekend was THE Port 2017, a humanitarian-themed hackathon held at CERN, Geneva in Switzerland. Among the 7 teams participating, the Matrix team consisted of a few software developers from Bity including Matrix enthusiast Alejandro Avilés (who very kindly helped us get a team into the hackathon). Luke and Dave from the Matrix London office also flew out to help the cause and by the end had a very stable, working prototype by the end of the competition.

The hack we made was a communications system backed by Matrix for use in refugee camps, an idea that hatched at the start of the hackathon (whereas the other projects were well established ideas up to 6 weeks before the event). Check out the code on GitHub if you’re interested in the client-side apps we made over the weekend.

It was another fun weekend for the Matrix team and we look forward to the next one. Stay tuned for updates on upcoming Matrix events!

Matrix & Riot for Cryptocurrency Communities

Hi folks,

Over the last few weeks there’s been a huge movement in the cryptocurrency communities over needing to find a better communication medium than Slack.  Some of the biggest communities for projects like Status, Aragon, TenX, Tezos, OmiseGo, Polkadot and many others are getting overrun by phishing attacks where malicious users have set up bots which auto-DM users joining the room in order to try to extract private keys to steal funds.  Slack has very limited support for avoiding this sort of abuse (especially at the free service tiers), so the search is on for an alternative solution.  There seems to be some confusion over what Matrix & Riot can and can’t do to help the situation, so we thought we’d write a blog post about it (especially after we had so much fun at the ETHLDN meetup last week!).

To be clear: we see Ethereum, Bitcoin, Ripple, Stellar and all the other decentralised currencies as being very closely related to Matrix.  Just as distributed ledgers disrupt the fragmented oligopoly old-school banking industry, we want Matrix to disrupt the relatively old-school communications systems of today. And so we’d really rather like that Matrix and Riot rocked when it comes to supporting cryptocurrency communities, and this is something we intend to dedicate resources to long term: we’ve got some big plans.

Things Matrix provides:

Decentralisation. Rather than each community having its own silo, with users having to juggle accounts over all of them, Matrix decentralises rooms over all the different servers. Users can have a single account and still jump into all the other communities (as well as the rest of the Matrix universe). However, each community can run its own server instance (if they want to) and have complete control over its behaviour.

Encryption. Matrix has first-class end-to-end encryption (although the UX in Riot needs refinement and is technically still beta).  This is great for encrypting rooms which need privacy – although it does come at the expense of being able to do server-side content filtering, which is desirable for fixing phishing attacks. So you probably don’t want to turn on encryption for rooms which need phish filtering (or you could use a bot to decrypt and autoremove malicious content).

A standard real-time API. One bit of feedback we’ve heard recently is that “Riot has no realtime API”.  This is spectacularly untrue; Riot is a client for the Matrix protocol, which is in and of itself an open standard realtime API for messaging, which you can use for writing whatever bots and extensions your heart desires.

Finely grained permissions per room. Likewise there seems to be some confusion over Matrix’s access control model.  In Matrix, each user in a room has a ‘power level’ – typically a number between 0 and 100.  By convention, normal users who have just joined the room have 0; the room creator and ‘admins’ have 100; and ‘moderators’ have 50.  Pretty much every access you can do in a room then has a threshold which defines how much power a user needs to perform the action.  It doesn’t get much more finely grained than this!

Ability to disable DMs and room invites. Architecturally Matrix lets you prevent users who use a given server from receiving invites (the homeserver can just autoreject the invites, based on some set of rules).  We’re currently putting together a quick demo to show this off in the Synapse server implementation, but it boils down to having an option to cancel invites here (federated) and here (local). Check out the demo below!

Ability to filter content. Similarly, Matrix architecturally lets a given server filter out messages based on content or some other pattern from being received by its users.  We’re also putting together a demo of this too in Synapse, which boils down to redacting inappropriate events here (federated) and here (local).  The demo isn’t quite ready yet but we’ll update this & yell when it is. Check out the demo below!

UPDATE – the DM/invite disabling and spam/phish filtering code has now landed on the develop branch of Synapse, and we’ve deployed an demo example of it at https://phishfree.riot.im.  Messages containing the word ‘SPAM’ are filtered, and invites are disabled (unless you are the local server admin).

Other stuff. Matrix and Riot give loads of other fun stuff too:

  • Widgets – the ability to embed arbitrary apps into your rooms (video conferences; currency tickers; DApps; wallets; monitoring dashboards; etc.).
  • 100% Native clients on iOS & Android (including Jitsi video conferencing & Widgets, as of the develop branch!)
  • Read receipts! (how can you live without them on Slack?!)
  • Internationalised to 20+ languages (thanks to the community! :)
  • Bridges through to IRC, Slack, Gitter, and more.
  • All sorts of alternative clients (e.g. nheko, quaternion) and SDKs
  • Insanely scalable and performant next-generation server (Dendrite) on the horizon
  • An open spec for the protocol.
  • 100% Apache-licensed FLOSS.  Riot/Web is particularly easy to hack on and theme & customise as needed.
  • Ability to disable federation for a room if you really want to lock it down to the users & rules of a single server.

Things we need to improve:

Groups (aka Communities):  One of the biggest missing features in Matrix is the ability to define groups of users & rooms, similar to a Slack team or Discord server, which can be used to organise together a set of discussions and generally give a feeling of community.  We’ve been working hard at this and expect to see it land in Riot/Web in the next few weeks.  In the meanwhile, you can see some of the UX we’re aiming for here!

E2E UX (and Riot UX in general):  While the underlying encryption of Matrix is solid, the UX exposed by Riot needs considerable work – specifically to improve the device verification flow and automatically share keys between trusted devices.  We’re continuing to work on this over the next few months.  Likewise there are many areas for possible improvement in Riot’s overall UX and design that we’re working through as urgently as we can.

Active Application Services: The per-server filtering described above is good if you just want to protect users on a given server (e.g. the server you point your community at).  However, if you want to filter all the messages for a given room which may be federated over multiple servers, you need a way to define a centralised chokepoint to define the filtering rules.  Architecturally this is meant to be performed by an ‘Active Application Service’ in Matrix, but we’ve not yet defined or implemented this API.  The idea for the room to define a list of services that messages are filtered through by all servers before they may be accepted for the room.  This would be the ideal solution to the phishing-filtering problem, but in practice filtering just local users (and perhaps disabling federation for particularly sensitive rooms or servers) is probably good enough for the immediate problem here.

Hope this provides some much-needed clarity to the debate! If there are other features cryptocurrency communities need to thrive please let us know, as we’d like to actively help to support decentralized communities.  #matrix-dev:matrix.org is probably the best place for further questions :)

Finally: one thing that has come up a few times in this discussion has been “Matrix’s funding crisis means they may not be here to stay”.  All I can say is that Matrix is here to stay. Even if the core team ended up just being Matthew hacking away by himself funded by Patreon/Liberapay, we have a large and passionate wider dev community who aren’t going anywhere.  But more importantly (and not wishing to jinx it), in the last few weeks we have received offers of significant funding which may hopefully resolve the funding crisis for the foreseeable.  Nothing is signed yet, but watch this space, and meanwhile I strongly suggest betting on Matrix being here to stay!

–Matthew

Thoughts on cryptocurrencies

Hi folks,

Something that has kept coming up since we ran into funding problems in July is the idea that Matrix could launch a cryptocurrency – a token for use when exchanging items of value within Matrix. This isn’t such a far-fetched idea: folks are already starting to look at how to sell content/services within Matrix, and the idea of using a Matrix-specific currency rather than credit card, PayPal, or an existing cryptocurrency could have some major advantages. Specifically:

  • It would let the value of the currency (in terms of its exchange rate relative to other currencies) grow in value directly linked to the growth and success of the Matrix ecosystem as a whole.
  • In future it could help us reward folks who run Matrix infrastructure (homeservers, identity servers, etc) by “mining” or “farming” style allocations of currency
  • It could also be a very useful tool for helping fight spam in future.  One way of proving that a user should be allowed to contact strangers (other than a vouching system) could be to spend some money.
  • An “token generation event” or “initial coin offering” where we sell initial allocations of the currency to the Matrix & cryptocurrency community could be a rather useful way to raise enough money to fully support the core Matrix team going forwards.

Meanwhile, Matrix itself is obviously already a fairly successful decentralised application ecosystem, and we believe that the above points give us a much better reason than average to actually launch a currency.  It’s important to note that we don’t have plans at this point to evolve the Matrix protocol itself into being able to support cryptocurrencies – we’d instead piggyback on top of an existing established distributed currency ledger.  (Later on, rewarding folks who run Matrix infrastructure by mining would require more interesting integration with Matrix, of course).

However (and this is the important bit), whilst we’ve been thinking about this a lot over the last few months, we have not yet announced anything concrete.  Over the last few days it’s come to our attention that there are some people advertising a “Matrix.org ICO Presale”.  This is not legitimate – we are not yet running an ICO or presale, and if/when we do the only place you will hear about it is here on the Matrix.org website.  It looks possible that this is a scam to try to steal Ethereum.  We have not yet authorised anyone to sell hypothetical Matrix currency.  If you see this rumour around please let us know so we can try to understand where it’s coming from and set the record straight.

Anyway, we thought it was worth giving an update on our thoughts about cryptocurrencies – and to publicly clarify that anyone claiming that they are running a Matrix.org ICO is lying.

We’d genuinely be very interested to hear feedback from the community on whether an ICO for Matrix would be a good idea or not – #matrix:matrix.org is probably the best place to discuss it.  It’s important to understand that our core focus will always be on Matrix itself, where we still have a lot of work to get through – and if we do an ICO it’ll be in partnership with specialist cryptocurrency experts, and hopefully minimise the impact to the core Matrix project itself.  But right now, we would be foolish not to be seriously considering the option.

thanks,

Matthew, Amandine & the team.

Synapse 0.22.0 released!

Hi Synapsefans,

Synapse 0.22.0 has just been released! This release lands a few interesting features:

  • The new User directory API which supports Matrix clients’ providing a much more intuitive and effective user search capability by exposing a list of:
    • Everybody your user shares a room with, and
    • Everybody in a public room your homeserver knows about
  • New support for server admins, including a Shutdown Room API (to remove a room from a local server) and a Media Quarrantine API (to render a media item inaccessible without its actually being deleted)

As always there are lots of bug fixes and performance improvements, including increasing the default cache factor size from 0.1 to 0.5 (should improve performance for those running their own homeservers).

You can get Synapse 0.22.0 from https://github.com/matrix-org/synapse or https://github.com/matrix-org/synapse/releases/tag/v0.22.0 as normal.

Changes in synapse v0.22.0 (2017-07-06)

No changes since v0.22.0-rc2

Changes in synapse v0.22.0-rc2 (2017-07-04)

Changes:

  • Improve performance of storing user IPs (PR #2307, #2308)
  • Slightly improve performance of verifying access tokens (PR #2320)
  • Slightly improve performance of event persistence (PR #2321)
  • Increase default cache factor size from 0.1 to 0.5 (PR #2330)

Bug fixes:

  • Fix bug with storing registration sessions that caused frequent CPU churn
    (PR #2319)

Changes in synapse v0.22.0-rc1 (2017-06-26)

Features:

  • Add a user directory API (PR #2252, and many more)
  • Add shutdown room API to remove room from local server (PR #2291)
  • Add API to quarantine media (PR #2292)
  • Add new config option to not send event contents to push servers (PR #2301)
    Thanks to @cjdelisle!

Changes:

Bug fixes:

  • Fix users not getting notifications when AS listened to that user_id (PR
    #2216) Thanks to @slipeer!
  • Fix users without push set up not getting notifications after joining rooms
    (PR #2236)
  • Fix preview url API to trim long descriptions (PR #2243)
  • Fix bug where we used cached but unpersisted state group as prev group,
    resulting in broken state of restart (PR #2263)
  • Fix removing of pushers when using workers (PR #2267)
  • Fix CORS headers to allow Authorization header (PR #2285) Thanks to @krombel!

 

Use you a Matrix for Great Good!

Hi all,

We’re currently looking into different ways that Matrix is being used in the wild, and an important question that has come up is whether anyone is using Matrix yet for decentralised communication in parts of the world where centralised communication poses a problem – due to bad connectivity or privacy concerns.  Similarly we’d love to hear from anyone who is seriously trialling Matrix’s end-to-end encryption for use in geographies where privacy is a particularly big issue for human rights.

So, if anyone has stories (anecdotal or otherwise) about how they’re using or planning to use Matrix to make the world a better place, in a location where that’s particularly critical, please can you let us know as soon as possible (@matthew:matrix.org or @Amandine:matrix.org).  This is fairly urgent because we’re currently looking at various options for how to prioritise effort and funding for Matrix, and if there are people out there who are depending on Matrix in this manner it would significantly help us support them!

thanks,

Matthew, Amandine & the team.