I should also introduce myself—I’m Nad Chishtie (@nadonomy:matrix.org) and I recently joined the Matrix core team (at New Vector) as Lead Designer, most recently focusing on end-to-end encryption.
When using encrypted messages, most existing services fall short in one or all of the following:
They don’t allow you to use multiple devices independently. For example, a web session might be locally tethered to a mobile device.
They don’t support a way to restore or temporarily access message history. For example, if you don’t have physical access to your main device because it’s broken or has been stolen.
They don’t allow you to verify that devices are controlled by their owners rather than eavesdroppers, and persist that trust across multiple devices, sessions or rooms.
Modern users, even those we talk to at security and privacy-led organisations, expect these features to ‘just work’ by default out of the box. Before enabling end-to-end encryption by default, we’ve been hard at work figuring out how we can deliver these features without compromising security or usability.
(For some users, restrictions such as limiting the number of places encryption keys reside, and not having a synchronised message history may be desirable security features. We’ll support these cases, but just not as the default behaviour.)
Let’s dive in to some of the fundamental concepts we’ll be putting forward to deliver a default end-to-end encryption experience that makes sense for most modern users. In this post we’ll look at an overview of work-in-progress wireframes, in the spirit of designing in the open and gathering feedback from the wider Matrix community. Please note that these don’t represent the actual interface design.
Cross-signing personal devices
When logging in to a new device, you’ll be able to use an existing device to verify your new one. Verification is done by scanning a QR code on whichever device has the most convenient camera to use, or by comparing a short text string. You only have to complete this process once to mutually verify both devices.
To the end user, Secure Message Recovery works a lot like setting up disk encryption or a password manager. A user can optionally secure their message history using a recovery passphrase and/or key. If logged out, or using another device, the user can use the recovery passphrase or key to access their encrypted message history.
We think that in most cases users will cross-sign personal devices, but as a safety net (for example, if a user’s devices are broken or lost) Secure Message Recovery is an invaluable tool for users to minimise the chance of them losing their encrypted message history.
People should trust people
With both cross-signing and Secure Message Recovery in place, we think that people should trust people, instead of individual devices. Now, when you verify a device, it’ll mark all of that users trusted devices as trusted.
Gone are the days of every person you talk to having to independently verify your new device upgrade. Like cross-signing, you can verify a device by scanning a QR code or comparing a short text string.
Sensible and extensible
In Riot, we’re implementing these features with a sensible default experience that strikes a balance between usability and security. We think most people would prefer to trust cross-signed devices, and that user trust shouldn’t block encryption. However, if you aren’t most people, you’ll be free to configure whatever level of security you need.
With all of the above in place, and after resolving any remaining technical issues, users will be able to:
Use end-to-end encryption by default in private rooms.
Use an existing device or Secure Message Recovery to access their encrypted message history on multiple devices, and to signal device trust to other users.
Access their encrypted message history using Secure Message Recovery, by storing encrypted message keys on their homeserver.
Mark a user as trusted by verifying one of their devices, persisting across all rooms and devices.
Keep their encrypted messages out of the hands of eavesdroppers.
Opt out, or further configure if they have more specific security requirements.
Over the coming days we’re polishing wireframes, nomenclature, iconography and microcopy as we dig deeper into development and implementation, as well as designing these features for the upcoming Riot redesign. Cryptography needn’t be intimidating, and we’re excited to iterate on these advanced features to make them work for everyone.
Back in June we blogged about the plan of action to establish a formal open governance system for the Matrix protocol: introducing both the idea of the Spec Core Team to act as the neutral technical custodian of the Matrix Spec, as well as confirming the plan to incorporate the Matrix.org Foundation to act as a neutral non-profit legal entity which can act as the legal Guardian for Matrix’s intellectual property, gather donations to fund Matrix work, and be legally responsible for maintaining and evolving the spec in a manner which benefits the whole ecosystem without privileging any individual commercial concerns. We published a draft proposal for the new governance model at MSC1318: a proposal for open governance of the Matrix.org Spec to gather feedback and to trial during the day-to-day development of the spec. Otherwise, we refocused on getting a 1.0 release of the Spec out the door, given there’s not much point in having a fancy legal governance process to safeguard the evolution of the Spec when we don’t even have a stable initial release!
We were originally aiming for end of August to publish a stable release of all Matrix APIs (and thus a so-called 1.0 of the overall standard) – and in the end we managed to publish stable releases of 4 of the 5 APIs (Client-Server, Application Service, Identity Service and Push APIs) as well as a major overhaul of the Server-Server (SS) API. However, the SS API work has run on much longer than expected, as we’ve ended up both redesigning and needing to implement many major changes to to the protocol: the new State Resolution algorithm (State Resolution Reloaded) to fix state resets; versioned rooms (in order to upgrade to the new algorithm); changing event IDs to be hashes; and fixing a myriad federation bugs in Synapse. Now, the remaining work is progressing steadily (you can see the progress over at https://github.com/orgs/matrix-org/projects/2 – although some of the cards are redacted because they refer to non-spec consulting work) – and the end is in sight!
So, in preparation for the upcoming Matrix 1.0 release, we’ve been moving ahead with the rest of the open governance plan – and we’re happy to announce that as of a few hours ago, the initial incarnation of The Matrix.org Foundation exists!Now, it’s important to understand that this process is not finished – what we’ve done is to set up a solid initial basis for the Foundation in order to finish refining MSC1318 and turning it into the full Articles of Association of the Foundation (i.e. the legal framework which governs the remit of the Foundation), which we’ll be working on over the coming weeks.
In practice, what this means is that in the first phase, today’s Foundation gives us:
Guardians, whose role is to be legally responsible for ensuring that the Foundation (and by extension the Spec Core Team) keeps on mission and neutrally protects the development of Matrix. Matrix’s Guardians form the Board of Directors of the Foundation, and will provide a ‘checks and balances’ mechanism between each other to ensure that all Guardians act in the best interests of the protocol and ecosystem.
For the purposes of initially setting up the Foundation, the initial Guardians are Matthew & Amandine – but in the coming weeks we’re expecting to appoint at least three independent Guardians in order to ensure that the current team form a minority on the board and ensure the neutrality of the Foundation relative to Matthew & Amandine’s day jobs at New Vector.The profile we’re looking for in Guardians are: folks who are independent of the commercial Matrix ecosystem (and especially independent from New Vector), and may even not be members of today’s Matrix community, but who are deeply aligned with the mission of the project, and who are respected and trusted by the wider community to uphold the guiding principles of the Foundation and keep the other Guardians honest.
An immutable asset lock, to protect the intellectual property of the Foundation and prevent it from ever being sold or transferred elsewhere.
An immutable mission lock, which defines the Foundation’s mission as a non-profit neutral guardian of the Matrix standard, with an initial formal goal of finalising the open governance process. To quote article 4 from the initial Articles of Association:
4. The objects of the Foundation are for the benefit of the community as a whole to:
4.1.1 empower users to control their communication data and have freedom over their communications infrastructure by creating, maintaining and promoting Matrix as an openly standardised secure decentralised communication protocol and network, open to all, and available to the public for no charge;
4.1.2 build and develop an appropriate governance model for Matrix through the Foundation, in order to drive the adoption of Matrix as a single global federation, an open standard unencumbered from any proprietary intellectual property and/or software patents, minimising fragmentation (whilst encouraging experimentation), maximising speed of development, and prioritising the long-term success and growth of the overall network over the commercial concerns of an individual person or persons.
You can read the initial Articles of Association here (although all the rest of it is fairly generic legal boilerplate for a non-profit company at this point which hasn’t yet been tuned; the Matrix-specific stuff is Article 4 as quoted above). You can also see the initial details of the Foundation straight from the horse’s mouth over at https://beta.companieshouse.gov.uk/company/11648710.
Then, in the next and final phase, what remains is to:
Appoint 3+ more Guardians (see above).
Finalise MSC1318 and incorporate the appropriate bits into the Articles of Associations (AoA). (We might literally edit MSC1318 directly into the final AoA, to incorporate as much input as possible from the full community)
Tune the boilerplate bits of the AoA to incorporate the conclusions of MSC1318.
Register the Foundation as a Community Interest Company, to further anchor the Foundation as being for the benefit of the wider community.
Perform an Asset Transfer of any and all Matrix.org property from New Vector to the Foundation (especially the Matrix.org domain and branding, and donations directed to Matrix.org).
So there you have it! It’s been a long time in coming, and huge thanks to everyone for their patience and support in getting to this point, but finally The Matrix.org Foundation exists. Watch this space over the coming weeks as we announce the Guardians and finish bootstrapping the Foundation into its final long-term form! Meanwhile, any questions: come ask in #matrix-spec-process:matrix.org or in the blog comments here.
Matthew, Amandine, and the forthcoming Guardians of [the] Matrix!
Today is one of those pivotal days for the Matrix ecosystem: we’re incredibly excited to announce that the world’s first ever dedicated homeserver hosting service is now fully available over at https://modular.im! This really is a massive step for Matrix towards being a mature ecosystem, and we look forward to Modular being the first of many hosting providers in the years to come :D
Modular lets anyone spin up a dedicated homeserver and Riot via a super-simple web interface, rather than having to run and admin their own server. It’s built by New Vector (the startup who makes Riot and hires many of the Matrix core team), and comes from taking the various custom homeserver deployments for people like Status and TADHack and turning them into a paid service available to everyone. You can even point your own DNS at it to get a fully branded dedicated homeserver for your own domain!
Anyway, for full details, check out the announcement over at the Riot blog. We’re particularly excited that Modular helps increase Matrix’s decentralisation, and is really forcing us to ensure that the Federation API is getting the attention it deserves. Hopefully it’ll also reduce some load from the Matrix.org homeserver! Modular will also help Matrix by directly funding Matrix development by the folks working at New Vector, which should in turn of course benefit the whole ecosystem.
Many people reading this likely already run their own servers, and obviously they aren’t the target audience for Modular. But for organisations who don’t have a sysadmin or don’t want to spend the time to run their own server, hopefully Modular gives a very cost-effective way of running your own dedicated reliable Matrix server without having to pay for a sysadmin :)
We’re looking forward to see more of these kind of services popping up in the future from everywhere in the ecosystem, and have started a Matrix Hosting page on the Matrix website so that everyone can advertise their own: don’t hesitate to get in touch if you have a service to be featured!
Guest post today from Giveth: Giveth is re-engineering charitable giving, by creating an entirely free, open-source platform, built on the Ethereum Blockchain. Our system cuts out bureaucracy and enables nonprofits to create a high level of transparency and accountability towards Givers.
Giveth’s new chatbot in action!
Online or offline, joining a new community always requires some adjustment. Even the most open, inclusive communities have shared knowledge and shared practices which new members learn as they participate.
I recently joined Giveth’s Riot community, where the majority of Giveth’s communication occurs. Immediately upon joining, I received the message pictured above from the Giveth Bot, kindly encouraging me to download Riot mobile and change my notifications to mention-only. The bot shortened my adjustment period by giving me key tidbits of information that everyone in Giveth’s community knows, but that may have taken me time to pick up on my own. This blog post will cover how the Giveth Bot came to be, what it is capable of, and where the project is headed in the future.
The Giveth Bot actually started out as an attempt to solve a completely different problem: helping Giveth efficiently distribute internal reward points. Giveth’s system for rewarding people who meaningfully contribute to the project is called RewardDAO. “If someone contributes in a meaningful way, a core contributor from each of the Giveth Campaigns can dish them points to recognize the contribution”, describes Cleo in an article explaining how RewardDAO works. At the end of each month, contributors receive Ether based on how many points they have earned.
The Giveth RewardDAO motto. Photo from https://medium.com/giveth.
However, any time that a core contributor dished points to someone, they had to record who received the points, and how many, on a spreadsheet. In search of a better way, Giveth opened up the project of automating this system to the social coding hub, a community of altruistic developers looking to tackle impactful and interesting projects, offering a 2 eth bounty for a solution.
A lot of great work was submitted, and ultimately Deam’s (@deamlabs) code was chosen to power the bot and the code for the pointsbot itself was further developed and refined by Frederik Bolding. Now, by using a command of the form “!dish [number] [type] points to [contributor] for [contribution]”, Giveth core contributors can distribute points as needed, and the bot will automatically update the spreadsheet accordingly.
The Giveth Bot dishing points like a champion!
Giveth began by using Slack, but switched to Riot to support Matrix’s decentralized, open-source model, which which aligns far more with Giveth’s own business model and values. The Giveth Bot is a great example of how Matrix enables users to build their own solutions to problems. In the future, we hope that the Giveth Bot will be able to interact directly with the Ethereum Blockchain, and that more analytics and measurement tools can be incorporated. And of course, we welcome any and all feedback on the Giveth Bot!
Giveth is an open-source platform for building decentralized altruistic communities. Anyone interested in getting involved should head to join.giveth.io
Interested in learning DApp development or helping out with cool projects like the Giveth Bot? Check out the social_coding Riot channel, tell us what you’re interested in, and help build awesome stuff!
If you’ve spent any time using Matrix public rooms, you’ve probably seen the bot DSN Traveller. This is a post-grad project from Florian Jacob, an informatics student at the Karlsruhe Institute for Technology. This week, Florian handed in his thesis on Matrix!
In summary, I could show that Matrix has few large but many small servers. Large servers reduce the overall network load, but a significant fraction of the load is concentrated in them. Introducing more small servers would further increase the load concentration. The Matrix event graph as a Conflict-Free Replicated Data Type showed to be well-suited for reliable messaging and history synchronization, and is applicable beyond Matrix.
I’m now working on a scientific paper on the results, which will boil down the more than 80 pages of the thesis to something much more digestible. 🔬 You’ll hear it in TWIM when that is finished!
As you may know, although he’s now back studying for the final year of his Computer Science degree, Half-Shot will continue to dedicate some time to bridge maintenance. He’s been working on IRC Connection Tracker, the next gen bridge for Matrix-IRC:
The IRC connection tracker has had yet more code and love applied to it. The headline changes are:
We now have a fully working IRC client that can connect to an IRCd, join channels and chat. These client’s persist over > the lifetime of the service.
There is a tool included with the service ircctl which allows you to spawn and use connections en masse. It also lets > you list the state of the currently connected clients.
Work has just begun on a client library for connecting this up to the bridge, but should be swiftly completed thanks to…
A brand new spec website in the works for describing the protocol (thanks Brendan for pointing me in the right direction)
0.33.5.1 is an interesting release. On the one hand it contains the usual bug fixes and performance improvements of a point release, but it also our first versioned release where monolith installs can be run under Python 3.5 and 3.6! Python 3 support is very much in beta, so please be cautious but if you would like to try running under a py3 environment we’d love to get your feedback.
I spent last night setting up hypertrix.ovh, a matrix server only listening on Hyperboria, a cjdns based end-to-end encrypted IPv6 overlay mesh network. I’d be glad if someone could be found to peer and federate with me there! Registration is open, but your client needs to be connected to Hyperboria to be able to talk to the server.
If you are currently using Hyperboria, go join hypertrix.ovh, or start your matrix server listening on it, and go chat to JC!
Lots of discussion about this project, specifically the question of how to efficiently render Rich Text. macOS does not make this easy, so a solution being considered is to use a WebKit for room rendering:
WebKit has the advantage of being super super fast on macOS, and also very low overheads
The current approach uses Cocoa NSTableViews and it’s horrible because Apple clearly couldn’t decide how they wanted them to work and therefore it’s not very optimised
Moving to WebKit only adds about 16mb to the RAM usage and redraws far faster than the NSTableViews can when resizing etc, and we’ll save a lot on the text formatting too which currently is a bit of a mess
September was mainly spent cleaning up loose ends on the Spec after all the releases at the end of August, and catching up on the never-ending maintenance burden of improving Synapse. However, in October the plan is to to go back again to working full out on the S2S r0 release. Wish us luck…
Lazy loading members is now on by default on riot.im/develop – reducing Riot’s RAM by 3-5x. Please give it a go and test it before we ship it in Riot 0.17 (probably next week) so we can iron out any last bugs (which will probably look like user profiles going missing)
Lazy loading also ships by default in Riot/iOS in Testflight 0.7.4 – if you want in on Testflight let us know in #riot-ios:matrix.org and we’ll share an invite link!
Lazy loading in Riot/Android coming real soon now; it’s behind a labs flag on the develop branch if you want to experiment.
Travis has started attacking the Riot/Web ‘First Impressions‘ project (starting with unbreaking onboarding in Riot/Web when GDPR consent is enabled)
Lots and lots of UX work from Nad on E2E, Communities, Onboarding and the overall redesign, complete with a redesign workshop with Jouni.
Aiming for end of Oct for first cut of redesign to be live as an experimental branch on riot.im.
Lots and lots of E2E implementation work in general; backups, cross-signing, and verification.
The end is nigh!
But only for this blog post! Check out Matrix Live below, and we’ll see you back here next week. :D
Ben’s away today, so this TWIM is brought to you mainly in association with Cadair’s TWIMbot!
Since last week’s sprint to get the new spec releases out, focus on the core team has shifted exclusively to the remaining stuff needed to cut the first stable release for the Server-Server API. In practice this means fleshing out the MSCs in flight and implementing them – work has progressed on both improving auth rules, switching event IDs to be hashes and others. Whilst implementing this in Synapse we’re also doing a complete audit and overhaul of the current federation code, hence the 0.33.3.1 security release this week.
Meanwhile, in the community, ma1uta reports:
I am working on the jeon (java matrix api) to update it to the latest stable release. Also I changed versions of api to form rX.Y.Z-N where rX.Y.Z is a API version and N is a library version whithin API. So, I have prepared Push API (r0.1.0-1), Identity API (r0.1.0-1) and Appservice API (r0.1.0-1) for the first release and current updating the C2S API to the r0.4.0 version.
Are you in the market for a Matrix-XMPP bridge? When I say “market”, I mean it because this week we have two announcements for bridging to XMPP! You can choose whether you’d prefer your bridge to be implemented as a puppet, or a bot.
It is a double-puppet bridge which can connects the Matrix and XMPP ecosystems. Just invite the @_xmpp_master:ru-matrix.org and tell him: @_xmpp_master: connect [email protected] to connect current room with the specified conference.
You can ask about this bridge in the #matrix-jabber-java-bridge:ru-matrix.org room.
Currently supports only conferences and only m.text messages. 1:1 conversations and other message types will be later.
maze appeared this week and announced MxBridge, a new Matrix-XMPP bridge:
It works as a bot, so it is non-puppeting. Rooms can be mapped dynamically by the bot administrator(s). There is no support for 1-1 chats (yet). MxBridge is written as a multi-process application in Elixir and it should scale quite well (but don’t tie me down on it ;)). https://github.com/djmaze/mxbridge
Neil powers onwards with Seaglass, with updates this week including:
Lazy-loading room history on startup to help with performance
Scrollback support (both forwards and backwards)
Support for Matthew’s Account (aka retries on initial sync for those of us with massive initial syncs, and general perf improvements to nicely support >2000 rooms)
Better avatar support & cosmetics on macOS Mojave
Encryption verification support, device blacklisting and message information
Ability to turn encryption on in rooms
Responding to encryption being turned on in rooms
Paranoid mode for encryption (only send to verified devices)
Invitation support (both in UI and /invite)
Blackhat announces that Matrique’s new design is almost done, along with GNU/Linux, MacOS and Windows nightly build!
Alexandre Franke says:
Fractal 3.30 got release alongside the rest of GNOME. It includes a bunch of new and updated translations, and redacted messages are now hidden.
Meanwhile, hidden in this screenshot, uhoreg noted that E2E plans are progressing…
Bruno has been hacking away on Riot/Web squashing the remaining Lazy Loading Members defects and various related optimisations and fixes. We also released Riot/Web 0.16.3 as a fairly minor point release (which unfortunately has a regression with DM avatars, which is fixed in 0.16.4, for which a first RC was cut a few hours ago and should be released on Monday). Meanwhile the first cut of Lazy Loading also got implemented on Android as well. Both are hidden behind labs flags, but we’re almost at a point where we can turn it on now! Otherwise, the Riot team has got sucked into working on commercial Matrix stuff, for better or worse (all shall be revealed shortly though!)
Jason has been working heavily on Construct, and has new test users. Construct is able to federate with Synapse and the rest of the Matrix ecosystem. mujx has created a docker for Construct which streamlines its deployment.
tulir has now deployed using the standalone install instructions on a very small LXC VM using ZFS. Unfortunately ZFS does not support O_DIRECT (direct disk IO) which is how Construct achieves maximum performance using concurrent reads. This is not a problem though when using an SSD or for personal deployments. I’ll be posting more about how Construct hacked RocksDB to use direct IO, which can get the most out of your hardware with multiple requests in-flight (even with an SSD).
Work was split this week into spec/security work, with the critical update for 0.33.3.1 – if you haven’t upgraded, please do so immediately.
Otherwise, Hawkowl has been on a mission to finish the Python 3 port, which is now almost merged. Testers should probably wait until it fully merges to the develop branch and we’ll yell about it then, but impatient adrenaline enthusiasts may want to check out the hawkowl/py3-3 branch (although it may explode in your face, mangle your DB and format your cat, and probably misses lots of recent important PRs like the 0.33.3.1 stuff). However, i’ve been running a variant on some servers for the last few days without problems – and it seems (placebo effect notwithstanding) incredibly snappy…
Meanwhile, the Lazy Loaded Member implementation got sped up by 2-3x, which makes /sync roughly 2-3x faster than it would be without Lazy Loading. This hasn’t merged yet, but was the main final blocker behind Lazy Loading going live!
A new bot appears! Are you a pedantic academic who likes to correct others’ misuse of Latin-derived plurals? This task can now be automated for you by means of SingularBot! Also for people who just like to have some fun. Free PongBot and SmileBot included.
kitsune on Hokkaido island
I ended up being on Hokkaido island right when a major earthquake struck it; so no activity on Matrix from me in the nearest couple of days. Also, donations to GlobalGiving for the disaster relief are welcome because people are really struggling here (abusing the communication channel, sorry).
…has got delayed again; sorry – we’re rather overloaded atm. We’ll catch up as soon as we can.