Thank you to Half-Shot for all your work on Bridges over the last months and beyond. Today is your last day, but I’m sure we’ll see you again before long. Text below is from Half-Shot.
Today marks my last day of my 3 month internship at New Vector (the startup which hires many of the core Matrix team). For those of you who haven’t been reading Ben’s fabulous blog posts, I’ve been working exclusively on bridges; in particular the IRC bridge.
Tasked with the goal of making it crash less and run faster, I hope that the evidence is visible and people are generally having a better experience on it!
Some stats pulled from the matrix-appservice-irc repo:
- 39 PRs closed (4 remain open)
- 27 issues closed, 27 issues opened.
- 334 commits, averaging 7.6 commits a PR.
Commits this year:
But aside from showing off some stats, I wanted to mention all the new features:
- Replies on Matrix translate well to IRC, or as well as IRC allows.
- People mentioning your IRC nick now ping your matrix user, finally!
- So. Many. Metrics. Everything you wanted to know about the internals of the bridge, but were too afraid to
- Not spamming homeservers with join requests on startup (it makes for a happy ops team).
- No longer are IRC users shackled to a “(IRC)” extension on their displayname, you can be who you want
with group flairs!
- Support for node 4 has been dropped, and support for 6,8 and 10 has been assured.
On the matrix-appservice-bridge side, I optimised some calls to cache locally and avoid hitting the homeserver too often, and disabling presence for homeservers that don’t support it.
There are future plans to make bridging more visible to Matrix Clients as a first class citizen. Ideas like speccing a state event (MSC1410) so that bridges can interact with each other properly and clients can create full bridge management views which are still decentralised from an integration manager.
I’d like to give a shoutout to Travis who has reviewed nearly all my changes that have made their way into the bridge, on top of all the other tasks he has on his plate. And of course a thank you to all of the Matrix team who have been very supportive during my time here.
Over the last few weeks there’s been a huge movement in the cryptocurrency communities over needing to find a better communication medium than Slack. Some of the biggest communities for projects like Status, Aragon, TenX, Tezos, OmiseGo, Polkadot and many others are getting overrun by phishing attacks where malicious users have set up bots which auto-DM users joining the room in order to try to extract private keys to steal funds. Slack has very limited support for avoiding this sort of abuse (especially at the free service tiers), so the search is on for an alternative solution. There seems to be some confusion over what Matrix & Riot can and can’t do to help the situation, so we thought we’d write a blog post about it (especially after we had so much fun at the ETHLDN meetup last week!).
To be clear: we see Ethereum, Bitcoin, Ripple, Stellar and all the other decentralised currencies as being very closely related to Matrix. Just as distributed ledgers disrupt the fragmented oligopoly old-school banking industry, we want Matrix to disrupt the relatively old-school communications systems of today. And so we’d really rather like that Matrix and Riot rocked when it comes to supporting cryptocurrency communities, and this is something we intend to dedicate resources to long term: we’ve got some big plans.
Things Matrix provides:
Decentralisation. Rather than each community having its own silo, with users having to juggle accounts over all of them, Matrix decentralises rooms over all the different servers. Users can have a single account and still jump into all the other communities (as well as the rest of the Matrix universe). However, each community can run its own server instance (if they want to) and have complete control over its behaviour.
Encryption. Matrix has first-class end-to-end encryption (although the UX in Riot needs refinement and is technically still beta). This is great for encrypting rooms which need privacy – although it does come at the expense of being able to do server-side content filtering, which is desirable for fixing phishing attacks. So you probably don’t want to turn on encryption for rooms which need phish filtering (or you could use a bot to decrypt and autoremove malicious content).
A standard real-time API. One bit of feedback we’ve heard recently is that “Riot has no realtime API”. This is spectacularly untrue; Riot is a client for the Matrix protocol, which is in and of itself an open standard realtime API for messaging, which you can use for writing whatever bots and extensions your heart desires.
Finely grained permissions per room. Likewise there seems to be some confusion over Matrix’s access control model. In Matrix, each user in a room has a ‘power level’ – typically a number between 0 and 100. By convention, normal users who have just joined the room have 0; the room creator and ‘admins’ have 100; and ‘moderators’ have 50. Pretty much every access you can do in a room then has a threshold which defines how much power a user needs to perform the action. It doesn’t get much more finely grained than this!
Ability to disable DMs and room invites. Architecturally Matrix lets you prevent users who use a given server from receiving invites (the homeserver can just autoreject the invites, based on some set of rules).
We’re currently putting together a quick demo to show this off in the Synapse server implementation, but it boils down to having an option to cancel invites here (federated) and here (local). Check out the demo below!
Ability to filter content. Similarly, Matrix architecturally lets a given server filter out messages based on content or some other pattern from being received by its users.
We’re also putting together a demo of this too in Synapse, which boils down to redacting inappropriate events here (federated) and here (local). The demo isn’t quite ready yet but we’ll update this & yell when it is. Check out the demo below!
UPDATE – the DM/invite disabling and spam/phish filtering code has now landed on the develop branch of Synapse, and we’ve deployed an demo example of it at https://phishfree.riot.im. Messages containing the word ‘SPAM’ are filtered, and invites are disabled (unless you are the local server admin).
Other stuff. Matrix and Riot give loads of other fun stuff too:
- Widgets – the ability to embed arbitrary apps into your rooms (video conferences; currency tickers; DApps; wallets; monitoring dashboards; etc.).
- 100% Native clients on iOS & Android (including Jitsi video conferencing & Widgets, as of the develop branch!)
- Read receipts! (how can you live without them on Slack?!)
- Internationalised to 20+ languages (thanks to the community! :)
- Bridges through to IRC, Slack, Gitter, and more.
- All sorts of alternative clients (e.g. nheko, quaternion) and SDKs
- Insanely scalable and performant next-generation server (Dendrite) on the horizon
- An open spec for the protocol.
- 100% Apache-licensed FLOSS. Riot/Web is particularly easy to hack on and theme & customise as needed.
- Ability to disable federation for a room if you really want to lock it down to the users & rules of a single server.
Things we need to improve:
Groups (aka Communities): One of the biggest missing features in Matrix is the ability to define groups of users & rooms, similar to a Slack team or Discord server, which can be used to organise together a set of discussions and generally give a feeling of community. We’ve been working hard at this and expect to see it land in Riot/Web in the next few weeks. In the meanwhile, you can see some of the UX we’re aiming for here!
E2E UX (and Riot UX in general): While the underlying encryption of Matrix is solid, the UX exposed by Riot needs considerable work – specifically to improve the device verification flow and automatically share keys between trusted devices. We’re continuing to work on this over the next few months. Likewise there are many areas for possible improvement in Riot’s overall UX and design that we’re working through as urgently as we can.
Active Application Services: The per-server filtering described above is good if you just want to protect users on a given server (e.g. the server you point your community at). However, if you want to filter all the messages for a given room which may be federated over multiple servers, you need a way to define a centralised chokepoint to define the filtering rules. Architecturally this is meant to be performed by an ‘Active Application Service’ in Matrix, but we’ve not yet defined or implemented this API. The idea for the room to define a list of services that messages are filtered through by all servers before they may be accepted for the room. This would be the ideal solution to the phishing-filtering problem, but in practice filtering just local users (and perhaps disabling federation for particularly sensitive rooms or servers) is probably good enough for the immediate problem here.
Hope this provides some much-needed clarity to the debate! If there are other features cryptocurrency communities need to thrive please let us know, as we’d like to actively help to support decentralized communities. #matrix-dev:matrix.org is probably the best place for further questions :)
Finally: one thing that has come up a few times in this discussion has been “Matrix’s funding crisis means they may not be here to stay”. All I can say is that Matrix is here to stay. Even if the core team ended up just being Matthew hacking away by himself funded by Patreon/Liberapay, we have a large and passionate wider dev community who aren’t going anywhere. But more importantly (and not wishing to jinx it), in the last few weeks we have received offers of significant funding which may hopefully resolve the funding crisis for the foreseeable. Nothing is signed yet, but watch this space, and meanwhile I strongly suggest betting on Matrix being here to stay!
Something that has kept coming up since we ran into funding problems in July is the idea that Matrix could launch a cryptocurrency – a token for use when exchanging items of value within Matrix. This isn’t such a far-fetched idea: folks are already starting to look at how to sell content/services within Matrix, and the idea of using a Matrix-specific currency rather than credit card, PayPal, or an existing cryptocurrency could have some major advantages. Specifically:
- It would let the value of the currency (in terms of its exchange rate relative to other currencies) grow in value directly linked to the growth and success of the Matrix ecosystem as a whole.
- In future it could help us reward folks who run Matrix infrastructure (homeservers, identity servers, etc) by “mining” or “farming” style allocations of currency
- It could also be a very useful tool for helping fight spam in future. One way of proving that a user should be allowed to contact strangers (other than a vouching system) could be to spend some money.
- An “token generation event” or “initial coin offering” where we sell initial allocations of the currency to the Matrix & cryptocurrency community could be a rather useful way to raise enough money to fully support the core Matrix team going forwards.
Meanwhile, Matrix itself is obviously already a fairly successful decentralised application ecosystem, and we believe that the above points give us a much better reason than average to actually launch a currency. It’s important to note that we don’t have plans at this point to evolve the Matrix protocol itself into being able to support cryptocurrencies – we’d instead piggyback on top of an existing established distributed currency ledger. (Later on, rewarding folks who run Matrix infrastructure by mining would require more interesting integration with Matrix, of course).
However (and this is the important bit), whilst we’ve been thinking about this a lot over the last few months, we have not yet announced anything concrete. Over the last few days it’s come to our attention that there are some people advertising a “Matrix.org ICO Presale”. This is not legitimate – we are not yet running an ICO or presale, and if/when we do the only place you will hear about it is here on the Matrix.org website. It looks possible that this is a scam to try to steal Ethereum. We have not yet authorised anyone to sell hypothetical Matrix currency. If you see this rumour around please let us know so we can try to understand where it’s coming from and set the record straight.
Anyway, we thought it was worth giving an update on our thoughts about cryptocurrencies – and to publicly clarify that anyone claiming that they are running a Matrix.org ICO is lying.
We’d genuinely be very interested to hear feedback from the community on whether an ICO for Matrix would be a good idea or not – #matrix:matrix.org is probably the best place to discuss it. It’s important to understand that our core focus will always be on Matrix itself, where we still have a lot of work to get through – and if we do an ICO it’ll be in partnership with specialist cryptocurrency experts, and hopefully minimise the impact to the core Matrix project itself. But right now, we would be foolish not to be seriously considering the option.
Matthew, Amandine & the team.
We’re currently looking into different ways that Matrix is being used in the wild, and an important question that has come up is whether anyone is using Matrix yet for decentralised communication in parts of the world where centralised communication poses a problem – due to bad connectivity or privacy concerns. Similarly we’d love to hear from anyone who is seriously trialling Matrix’s end-to-end encryption for use in geographies where privacy is a particularly big issue for human rights.
So, if anyone has stories (anecdotal or otherwise) about how they’re using or planning to use Matrix to make the world a better place, in a location where that’s particularly critical, please can you let us know as soon as possible (@matthew:matrix.org or @Amandine:matrix.org). This is fairly urgent because we’re currently looking at various options for how to prioritise effort and funding for Matrix, and if there are people out there who are depending on Matrix in this manner it would significantly help us support them!
Matthew, Amandine & the team.
As part of our semi-regular series of having Matrix core team members write about how they see the overall project, here’s Oddvar Lovaas’ view. Oddvar helps out with project management and promoting Matrix.
According to the homepage, Matrix is “a new open standard for interoperable Instant Messaging and VoIP, providing pragmatic HTTP APIs and open source reference implementations for creating and running your own real-time communication infrastructure. ”
That’s all good – but you might be asking yourself “sure, but what can it do?” And more importantly, “what can it do for me?”
The original inspiration for Matrix was to fix the problem of fragmented IP communications, by creating a standard for creating and running your own real-time communication infrastructure. This means that if you want your app or program or website to be able to communicate user to user for example, you can use Matrix. Matrix is the protocol through which your communication packets are sent and received, and we provide HTTP APIs to make it easy to make use of this protocol in your code.
The nice thing here is that the user can to talk to any other user anywhere in the Matrix ecosystem, much like email or the web. For example, let’s imagine I have an app whose goal is to keep the user updated on anything happening in the football world. Whenever any news drops in, the app is notified and thousands of users check the app for the news. This app could have a communication element where the users can talk in rooms (maybe a #general room and rooms for each football club) – or even between themselves or in groups of friends. Today, a lot of people would use an IM-client to do this, but with Matrix it wouldn’t matter if you use a dedicated IM-app or talk inside the football app – since you are using the same Matrix account, you will get the same conversations in both clients!
In fact, imagine that later on you are chatting with some (non-football) friends on your Matrix-supporting chat-application. You can then easily check the previous conversation to see if anyone’s appreciated the great joke you made earlier – without having to go back into the football app.
But Matrix’s real potential and ultimate mission is to be a generic messaging and data synchronisation system for the web – allowing people, services and devices to easily communicate with each other with full history. It’s easier to visualise the chat-application because we are used to chat-messages going back and to, but there’s nothing stopping you from putting other data instead of chat-messages. For example, you could use the Matrix protocol to exchange moves – encrypted and secured, of course – in your Chess-game. In fact, your Chess-game could use Matrix both for chatting and exchanging payloads of data.
Imagine if you open up your favourite chat-application, and your contacts there include other users of the same app and also other Matrix-users (so the app has exposed itself to Matrix). Your friend, however, much prefers a different app, but he can still talk to you over the Matrix protocol. And if he ever moves to the other app (or any other Matrix-supporting app) – he would still have all the backlog and history of the conversation!
Obviously the problem here is that we can’t instantly make the various chat-applications support Matrix. We believe if we can encourage and grow a truly open communication ecosystem, users will get used to the availability and benefits of interoperable services and they will demand it everywhere.
I’m Matthew, and I’m responsible for the techie side of what we’re up to with Matrix.
Matrix is the result of a lot of work my team’s done over the last 10 years or so (first at MX Telecom, then OpenMarket, and then Amdocs) in developing next-generation IP communication solutions. First we started with an Asterisk-based platform running basic PSTN IVR services, and then shifted to an in-house IAX-based IVR platform built in Java, and then added circuit-switched (3G-324M) video calling, then switched to SIP/RTP, C++ and a massively-distributed softswitch architecture affectionately called ‘The Next Generation’. Then the iPhone and Android came along, and we realised we didn’t have to be constrained by built-in phone capabilities and ported our whole C++ SIP/RTP VoIP stack over to iOS/Android and got writing Video/VoIP calling apps. This evolved to developing full-blown unified communication apps (e.g. Blah), using XMPP at first for messaging before switching to our own HTTP-based messaging APIs.
Somewhere along the way it became painfully obvious that VoIP and IM hasn’t really evolved as well as the rest of the internet. Back when SIP/RTP first emerged, it simply wasn’t mature enough to work on the open internet as well as HTTP or SMTP or even FTP – it needed STUN, ICE, TURN, Opus and many other refinements to be really usable in the wild. And similarly XMPP hasn’t taken over the world quite as much as we once hoped. Meanwhile, many folks went and built their own proprietary walled-garden solutions (be it Skype, FaceTime, Viber, WhatsApp, or even our own efforts) and we’ve ended up in the current horrible situation where our online communication is fragmented across hundreds of isolated apps and websites. It’s like a world where email was never unified, and half the world is still stuck on Compuserve. And it’s counter to the whole ethos of the internet as an open platform for collaboration and communication.
We decided that we want to fix this and so we have built and published a new open standard, together with open source (ASLv2) reference server (Python/Twisted) and client (JS/Angular, Python, Perl) codebases, and so provide new building blocks that can be used to build truly interoperable federated IM and VoIP functionality. We consider this effectively an investment in the industry: by creating a strictly non-profit initiative like Matrix, we both make the world a better place for end users – as well as creating new business opportunities (both opensource and commercial) for the telecoms industry as a whole.
The standard and code are all brand new and very much still in creation at this point, but we’re releasing it early to get as much feedback and input from the community as early on as we possibly can. Right now our focus is on fully decentralised federated group messaging, but VoIP and identity management is coming together well too. You can think of it as “making VoIP/IM as interoperable and flexible as email”, or perhaps “the missing signalling layer for WebRTC”, “XMPP for an HTTP world”, or “what would happen if IRC, XMPP, SIP, SMTP, IMAP and NNTP had kids?” Here are some reasons we think that you should use Matrix:
- Simple pragmatic RESTful HTTP/JSON APIs. No more XMPP or SIP stacks and wrestling XML streams or torture-testing SIP parsers.
- No single points of control for channels of communication (unless you really want it for moderation or similar). Room state for a room is synchronised with eventual consistency over all participating Matrix servers – no single server controls the room.
- No more netsplits – history re-heals itself if the matrix fractures
- All communication is group by default: 1:1 chat is just a subset of group chat.
- Multi-device aware: all state is stored and synchronised in realtime across all devices, and away-state and notifications are aware of multiple devices.
- Uses arbitrary 3rd party identifiers – doesn’t rely on JIDs or SIP URIs for identity.
- Share the same simple HTTP signalling channel for messaging and VoIP
- Support more efficient transports if you want (e.g. low-bandwidth/low-roundtrip sync on mobile)
- Built for mobile – e.g. support push notification and low-bandwidth/low-latency client-server transports if needed (in progress)
- TLS (HTTPS) by default, either with self-signed certs with published public keys or proper SSL CA signed certs (in progress)
- End-to-end PKI encryption (in progress)
- Trusted federation of public identity servers available for publishing your PKI public keys and tracking your validated 3rd party IDs
If this sounds good to you, then please take a look at the spec, or our tutorials, or jump straight into playing with the APIs, or try running your own Matrix homeserver, or sign up to our mailing lists – and whatever else, come swing by #matrix:matrix.org and say hi!