We're just back from an incredible time at FOSDEM 2020 - Europe's biggest Free & Open Source Software conference. Huge huge thanks to everyone who came to our talks (sorry if you couldn't get in :/), came to talk to us at the stand, or flagged us down to give feedback, chase PRs, file bugs, or just say thanks. Thanks also to FOSDEM to accepting all of our talks this year, and to the FOSDEM organisers for pulling together yet another amazing event :)
We'll do a proper blog write-up on enabling E2E encryption by default, cross-signing, and all the other E2E encryption work that's been going on once we ship the stable release - but as of Saturday(!) it has landed on Riot/Web Develop, RiotX/Android (0.14.2) and Riot/iOS develop TestFlight, but we're still debugging and we need a bit longer before cutting the final releases.
So, until then, please take a look at the videos if you missed the livestream or weren't at the event!
At the end of September, TADHack Global was held where almost 150 teams spent their weekends hacking towards the $45k total prize money up for grabs. Luke spent the final day of the hack talking to teams hacking at IDEALondon in Shoreditch, meeting a few Matrix enthusiasts and long-time collaborators.
Out of 10 hacks, 2 of 4 local winners won prizes locally and went on to be global winners alongside 6 other teams using Matrix as part of their hacks. Checkout the TADHack London Wrap-up for details on all of the awesome hacks, especially Aviral Dasgupta's Pushtime and Polite.ai.
The following weekend was THE Port 2017, a humanitarian-themed hackathon held at CERN, Geneva in Switzerland. Among the 7 teams participating, the Matrix team consisted of a few software developers from Bity including Matrix enthusiast Alejandro Avilés (who very kindly helped us get a team into the hackathon). Luke and Dave from the Matrix London office also flew out to help the cause and by the end had a very stable, working prototype by the end of the competition.
The hack we made was a communications system backed by Matrix for use in refugee camps, an idea that hatched at the start of the hackathon (whereas the other projects were well established ideas up to 6 weeks before the event). Check out the code on GitHub if you're interested in the client-side apps we made over the weekend.
The original promise of the Internet was to be an interoperable platform for distributing data. However, we have since increasingly seen our data fragmented and trapped in a number of proprietary silos. Matrix hopes to fix this by being a federated, open standard for data exchange that any service can use.
The Decentralized Web Summit is a meetup for anyone interested in building the Decentralized Web, which aims to make the Web open, secure and free of censorship by distributing data, processing, and hosting across millions of computers around the world, with no centralized control. It takes place at the Internet Archive, San Francisco, CA on Wednesday June 8 and Thursday June 9, 2016.
Matrix will be represented at the event, and we hope to also host a workshop or a talk about Matrix.
The meetup has a Slack room set up for pre-meetup conversations - you can also access this room via Matrix: #decentralizedweb-general:matrix.org
We are looking forward to interesting people and interesting conversations at the first Decentralized Web Summit!
TADHack-mini took place in east London over the weekend with 88 people in attendance. There were $8k in prizes, and five different services to hack on. This time, we didn't have too many people using Matrix in their hack, but we did have an excellent idea and implementation called Babelonio, who won our prize: a Phantom X Hexapod Mk3!
Babelonio adds speech-to-text and translation via Google translate to Matrix, via the Vector client and a Chrome extension. This is quite nice, because it means you don't have to run a custom client. And by using Google translate, you immediately get access to a lot of languages (although sometimes the translation doesn't quite work, as you can see in the presentation video). The project was done by Steven Bakker, Timo Uelen and Bart Uelen. You can see the presentation and demo of the hack here.
A good write-up of all the hacks and winners can be found on the TADHack blog.
Thanks to the TADHack organisers for another fine event, and also to everyone who came over to chat about Matrix - and again congratulations to the Babelonio team:
The Matrix team had a very successful trip to FOSDEM '16 last weekend. Many, many words were exchanged, and at times there was a queue of people just waiting to have a chat! We spoke to a whole lot of interested and interesting people, some of which had heard of Matrix already, and some who hadn't. The nice thing with the crowd at FOSDEM is that they very quickly "get" what we're trying to do with Matrix - and then start thinking aloud about how they might want to use it or extend it - which means we have many great conversations!
At the beginning of FOSDEM, I think we had around 2000 people "currently" in the Matrix HQ room - the next day, that number had increased by a hundred (and now it's even bigger due to a hackernews post where we mentioned Matrix)!
Our talk in the IoT dev room was very popular; unfortunately a lot of people that were queueing to get in never made it due to the limited space. However, the talk was recorded, and it's already been made available:
For now, you can look at the slides from the talk.
Another interesting talk where Matrix ended up being represented, was Daniel Pocock's Improving Telepathy talk. I'll add the video from that too, once it's available.
Finally, thanks to everyone who came to say hello and have a chat - and to everyone who has since set up their own homeserver. Please do join the Matrix HQ room (using any of these clients) and let us know how your Matrix experience is going!
We also have a stand on the 2nd floor of the K building - next to the Real Time lounge (potentially same spot as last year). If you're going to FOSDEM, please come and say hi to us at the stand or at the talks!
Our trip to FOSDEM '15 was very enjoyable; we met a lot of smart people and learnt about a lot of interesting technologies and projects. Hopefully we made a few people enthusiastic about Matrix as well!
We've been pretty bad at updating the blog over the last few months with all the progress that's been happening with Matrix. Whilst Matrix rooms like #matrix:matrix.org and #matrix-dev:matrix.org have been very active (and our twitter account too), in general we've ended up spending way too much time actually writing software and not enough time talking about it, at least here. When a blog goes quiet it normally means that either the authors have got bored, or they're too busy building cool stuff to keep it updated. I'm happy to say that option 2 is the case here!
As a result, there's a huge backlog of really cool stuff we should have talked about. Hopes of writing an Advent Calendar series of blog posts also went out the window as we set Christmas as an arbitrary deadline for loads of work on Synapse, the Matrix Spec and matrix-react-sdk.
So, to try to break the impasse, here's a slightly unorthodox whistle-stop tour of all the amazing blogposts we would have written if we'd had time. And perhaps some of them will actually expand into full write-ups when we have more time to spare in the future :)
One of the great promises of Matrix is to provide End-to-end encryption as part of the baseline standard (configurable per-room). In practice, our progress has been a little non-linear - we started writing an Axolotl ratchet implementation in C++14 (with a pure C API) named Olm back in February, and then finished it off and wired a basic 1:1 proof-of-concept implementation into matrix-react-sdk in June. We then announced Olm back at the wonderful Jardin Entropique conference in Rennes:
The main stuff remaining is basically key management (in Synapse and the matrix spec), group conversation ratchets, and UX for wiring it properly into various Matrix clients. We expect to make progress on this over the next few months :)
Meanwhile, huge kudos to Tor who was crazy enough to add the basic 1:1 Olm ratchet to Weechat before we'd even finished writing our test jig!
A few days after Jardin Entropique we made it to Lean DUS - a great tech meetup in Düsseldorf organised by Sipgate, who were kind enough to invite us to speak. This was a chance to give a full update on Matrix (as of July!) and talk some more about Olm and plans for end-to-end encryption. This one got recorded - and you can see it below. There's also an official page with full videos, slide deck and photos up at https://hello.sipgate.de/veranstaltung/lean-dus-9-java-matrix.
Somehow we've failed to blog about the amazing matrix-appservice-bridge Node framework which we've built as general purpose infrastructure for building Matrix Application Services which act as bridges between existing networks and comms solutions and Matrix. The architecture here looks something like this:
...and the goal is to end up with something like this:
the Matrix/Verto Bridge uses it to hook FreeSWITCH up to Matrix - currently used to provide multiway video and voice conferencing for Vector. It could be easily extended to do generic Matrix{'<->'}SIP or Matrix{'<->'}anything-that-FreeSWITCH-can-speak though.
a basic Matrix/Slack Bridge, which works well enough for hardcoding bridges between specific Matrix and Slack rooms.
matrix-appservice-respoke - a crazy experiment that bridges Asterisk to Matrix by implementing the Respoke API such that Asterisk can connect to Matrix using chan_respoke.
matrix-appservice-purple - another crazy experiment that hooks libpurple up to matrix-appservice-bridge such that *any* network that libpurple can talk to can be bridged into Matrix. So far we've experimented with Lync, Skype and Facebook (and AIM(!)) and it works - but it needs a lot more love to be usable other than as a toy.
As of right now our work on bridging has been on hiatus for a month or so, and we would love support from the community in advancing and extending the stuff we've built so far. Otherwise we'll get back to it ourselves in the new year.
We had a lot of fun in Orlando in October at Astricon 2015 - we put together matrix-appservice-respoke (see above) for our talk and Dangerous Demo in a desperate 24 hour hack and it even worked! The judges were kind enough to give us the "Swan Award" prize in the dangerous demo shoot-out for the glossiest demo :)
The slides for our 'Bridging Asterisk to the Matrix Ecosystem' talk are downloadable here.
We also implemented a basic libpurple plugin for Matrix - adding Matrix support to any app like Pidgin or Bitlbee that uses libpurple. (You could in theory even use it with matrix-appservice-purple to bridge from Matrix to Matrix, but that'd be silly :). It supports basic functionality and uses the new 'v2' APIs for syncing to Matrix. Adventurous libpurplers can go check it out and experiment with it from https://github.com/matrix-org/purple-matrix - feedback welcome.
We built a 3rd party Debian package repository for Synapse... and then forgot to tell anyone about it, other than buried in the Synapse readme! Well, it exists, and intrepid debianers should go check it out at http://matrix.org/packages/debian/.
In November we attended TADSummit in Lisbon - a great event for folks hacking on telco applications and the telcos themselves. Apparently we failed to do a writeup, but we had a wonderful time: highlights included sitting down with Maarten Ectors from Canonical to wrap up Synapse as an Ubuntu Snappy app such that anyone in the Ubuntu Core ecosystem can trivially run a Matrix homeserver, and demoing it as part of the Dangerous Demos track there. We also gave a 'Matrix: One Year In' talk to summarise what we got up to in 2015.
Whilst on the subject of conferences that we forgot to write up - we just got back from WebRTC Paris, where we demoed the latest & greatest Matrix clients and bridges, hung out with the OpenWebRTC guys and gave another ecosystem update. You can see the slides at https://matrix.org/~daniel/Matrix- One-year Status Report.pdf.
There have been a flurry of really interesting new clients and other projects which certainly deserve whole blog posts of their own!
There's Tensor from David A Roberts - a multiplatform native client written in QML that heavily leverages the matrix-js-sdk:
There's matrix.el from Ryan Rix - a native Matrix client for Emacs! You can read all about the whys and wherefores here.
There's also loads of cool stuff that Ryan's been doing with Matrix on his blog - including Polynomial - a decentralised webring built on Matrix (yes, webrings were and are cool, ok!?!), and his Matrix-powered Body Computing System. Also, some philosophicalposts on the benefits of Matrix which give us some hope that we're on the right track!
Then there's Power Take-Off from Torrie Fischer - an early lets-IRC-clients-connect-to-Matrix project in Rust...
...and there's Morpheus from Christine Dodrill (Xena) - a Matrix client and bot framework for Haskell; part of a more over-arching IRC{'<->'}Matrix unification project. Xena also wrote a great call to arms for Matrix :)
Very recently there's the Ruma project from Jimmy Cuadra - an ambitious mission to build Matrix components (up to and including a homeserver) in Rust!
Finally, Tor has done an amazing job on weechat-matrix-protocol-script in implementing features like V2 Sync and E2E crypto faster than we've managed to add them in the official client SDKs!
We have made some major improvements to the spec over the last few months: adding in feature profiles and spec modules to better structure the document, and most recently splitting it up explicitly into separate Client-Server, Server-Server and Application-Server APIs, each with a well-defined single global 'release' number for versioning. We started this with a 'r0.0.0' release of the Client-Server API, which consolidates the horrible mess of 'v1' and 'v2' APIs we had previously flying around into a single well-defined version of the spec. Meanwhile the spec is now entirely consolidated into a set of JSON schema and Swagger 2 API descriptors, with a bunch of ReStructured Text for the verbiage - you can find it all at https://github.com/matrix-org/matrix-doc.
We've also switched the Swagger-based API viewer over to Swagger 2.0: http://matrix.org/docs/api. We also rejigged the Matrix documentation website entirely, generating it via Jekyll and adding in a new guides section.
Also, we should have mentioned the existence of Speculator - a golang helper app (source here) which, as the name suggests, renders out copies of the spec as HTML from different branches and pull requests for ease of previewing.
Over the last few months we've also started an entirely new project, codenamed Dendron. Dendron is the project to evolve Synapse from the current single-threaded Python/Twisted monolithic homeserver into something with a lot more type-safety, horizontal scalability and high availability. We've mainly been experimenting with different ways of doing this, but the current plan is to split Synapse itself up into multiple services which can each scale independently, and then rewrite some/all of them in languages with better type safety and/or performance or profiling tools.
Some folks may remember a survey that we posted a few months ago asking for the community's thoughts on what languages they'd like their ideal homeserver to be written in, from the perspectives of someone running it as well as hacking on it. Whilst we haven't (at all) based our decisions for Dendron purely on the survey, it was still quite an interesting exercise. And here are the results (maximum 'score' is 5, not 10):
The basic feedback was that from the existing community: folks dislike running Java or Node servers; are okayish with Python, but would prefer native or near-native code (be that C, Rust or Go). Meanwhile, for contributing code, there's slightly more interest in the (relatively) new shinies of Go and Rust. And of course, everyone wanted to plug their own special snowflake language in the 'Others' section, which was mainly a mix of Erlang, Elixir, Haskell, Lisp and Perl :)
This reinforced the choices we were looking at anyway - either Rust (for its safety), or Go (for its simplicity, python-likeness, and concurrency). (We'd also consider Java, but have to concede that the FOSS community doesn't like running it.)
So we looked at the dependencies that Synapse currently has, and the Rust equivalents, and concluded that the Rust ecosystem unfortunately isn't quite mature enough yet to reliably handle the rather large set of complicated deps that we need in a homeserver. Also, nobody on the core team is really a Rust guru yet. Meanwhile, we have at least one ex-Google Go expert in the core team, and in practice it has the edge in terms of maturity. So, right now, we're looking at switching chunks of Dendron to Go where it makes sense. (This is subject to change though depending on how we get on). You should expect to hear a lot more about Dendron in 2016 :)
🔗matrix-react-sdk, Vector, and latest Matrix features
Last but not least: huge amounts of our time over the last year has gone into building matrix-react-sdk - a full set of glossy Web UI components for building super-high quality glossy apps based on Matrix, built on the matrix-js-sdk. This is basically a reaction against the original matrix-angular-sdk and Matrix Console app that we launched Matrix with back in 2014 - which had minimal attention to UI/UX and suffered from major performance problems; it was built purely as the fastest possible way we knew to prototype and demo Matrix in the first place. matrix-react-sdk however has been built for both performance and quality of UI/UX, as well as obviously using all the latest and greatest Matrix APIs. (In fact, the transition from matrix-angular-sdk to matrix-react-sdk is pretty similar to the Synapse to Dendron transition on the horizon - although the latter should be more incremental and less 'rewrite the world').
Meanwhile, as part of our commercial work at our day job (i.e. not as Matrix.org) we've been helping on a glossy FOSS app called Vector which is layered on top of matrix-react-sdk as a thin 'skin' layer of CSS and a few custom components. The intention for Vector is to ensure that Matrix has a flagship glossy client: which it clearly needs, in order to gain credibility and drive uptake of the Matrix standard, and also ensure that the Matrix standard actually does indeed fit the needs for a state-of-the-art collaboration tool.
matrix-react-sdk (and thus Vector) is still in incredibly heavy development - we're going to start a formal beta fairly soon, but as of right now it's still sprouting features and refinements on a daily basis. Meanwhile matrix-react-sdk's APIs are not remotely frozen (we entirely refactored it as recently as a few weeks ago), so not yet ready for use as a general purpose building block.
Some of the stuff going into react-sdk is incredibly cool - recent Matrix stuff that it shows off includes:
Full server-side search. We now have full-text search in the Matrix spec, and implemented on synapse both on sqlite and postgres - and now in Vector too. Having good search over all of your chat history makes Matrix *so* much more usable.
Video conferencing. We have full multi-way conferencing in Vector via matrix-appservice-verto and FreeSWITCH. The intention is to add this to the core Matrix spec (having first made it a bit more generic) - see the draft spec for details.
3rd party invites. You can now invite users into Matrix by email address as well as matrix ID, and it works. Vector implements this.
Room tagging. You can now tag rooms as favourites, low priority, or with arbitrary namespaced metadata. Vector implements this through a swanky drag & drop UI.
"V2" Sync API. Now part of the 'r0.0.0' spec, this lets Matrix support much smarter incremental and partial synchronisation patterns. Vector now implements this, meaning that browser windows sync much faster after being offline for a bit, and no longer hammer the user with stale desktop notifications.
Accessing 'historical' rooms. Matrix now lets you keep track of rooms you've left, so you can view and search the conversation logs even after you've left the conversation. Vector now implements this (as of Monday!)
Tab-complete that doesn't suck. This is a purely client-side feature which landed on Thursday!
Roll-overable animated GIFs. 'nuff said.
Markdown support. yay!
Synchronised read and notification history. This hasn't landed yet (in vector or synapse or even the spec), but finally provides a way to keep read and notification state in sync in realtime across all your clients and a meaningful favicon 'badge' telling you how many notifications you missed!
Guest access. This hasn't landed in Vector yet, but it's in the spec and Synapse. It will let folks use Matrix without having to create an account (at least for rooms which support 'guest access' from the public).
If you haven't given Vector a spin, it's well worth heading over to https://vector.im and taking a look.
There's also an Electron desktop version of Vector in progress, contributed by Steven Hammerton at https://github.com/stevenhammerton/vector-desktop (although it's currently stuck on an old release).
Okay, this has got a lot longer than it was meant to be - but hopefully makes up a bit for the lack of comms over the last few months whilst we've been drowning in work on Synapse, Vector, the Spec, Dendron, and everything else mentioned above.
2015 has been an epic year as we've taken Matrix from a very early beta to the advanced stage that it's at now. Obviously there's still a lot of stuff to do though. Right now we expect the focus in 2016 to be:
Vector - making sure the Matrix protocol has a flagship glossy FOSS client that normal (non-geek) users can use.
Dendron - making Synapse more reliable, scalable and maintainable.
Bridging - wiring as much of the rest of the world into Matrix as accurately and efficiently as possible.
Federation Spec - finishing and releasing the Server-Server API.
End-to-end crypto - finishing it off.
...and obviously continuing to refine and extend the core of Matrix itself with features like threading, editable messages, and possibly even distributed accounts.
There are very fun and exciting times ahead. We'd just like to say a profound thank you to everyone who's supported Matrix this far and helped make this mission possible - whether it's by running clients/servers/services, or writing your own, or filing bugs and feedback on our code or the spec, or telling folks about the project, or paying us to work on it(!), or just by reading this blog post. Hopefully 2016 will be the year where online communication starts to open up and interoperate once again, rather than becoming ever more fragmented and closed.
Thanks for reading - and Merry Christmas, for those who celebrate :)
Matrix will again be represented at the WebRTC Conference & Expo in Paris. Daniel and myself are catching the Eurostar tomorrow afternoon, and the conference will start early Wednesday morning with a panel about WebRTC for Mobile, where Daniel is one of the participants.
I'm sure we will have three days full of interesting talks and discussions (see the full schedule here). There will be demos as well, and Matrix is (of course!) also joining the demo competition. We hope to see many familiar faces - and hopefully meet some new ones as well!
If you are going to the conference, please come and say hello – we will have a stand at the expo (we're table #6 - see map here). And don't miss Daniel's Matrix One-year Status Report at 11.10am on Friday!
Matrix is sponsoring another TADHack, this time TADHack mini in Chicago on October 3-4. Come hack something cool together using Matrix, either in Chicago or remotely! At past TADHacks we've seen a robot controlled with Matrix, collaborative web page viewing with Matrix as the underlying protocol, and more.
As well as being lots of fun, we've teamed up with Trossen Robotics to offer a HR-OS1 Humanoid Endoskeleton for the best hack using the Matrix standard. The HR-OS1 is a hackable, modular, humanoid robot development platform designed from the ground up with customization and modification in mind. With a Raspberry Pi 2, webcam, wifi, bluetooth, and all of the limbs you expect for a human, there's a lot of scope for exciting projects to be had with it!
I for one would love to ask an open source robot to bring me snacks over chat. (Wow that's a sentence I never thought I would be able to say)
I'll be on site, (and also hanging out in on Matrix in #matrix:matrix.org) to help out with whatever creative things y'all come up with - I hope to see you there!
Also in Chicago the following week I'll be giving a tutorial on Federated Identity as well as attending and speaking about the problem of fragmented communication at the IIT Real-Time Communications Conference & Expo.
If you're going to be around, please do say hi - I only recently joined the Matrix team, and there's a lot of people I've yet to meet - I look forward to seeing you at one (or all) of these events!
The Matrix bandwagon goes on, and we have several conferences and events lined up in the next few weeks.
First up is TADHack mini in Chicago, October 3-4, where Matrix is sponsoring and handing out prizes to the best two hacks using Matrix in some way or another. The previous TADHack mini and the Global TADHack were both excellent events where we saw some really cool hacks - we are looking forward to see what people will put together this time!
Also in Chicago, a couple of days later, is the IIT Real-Time Communications Conference & Expo - October 6-8 - where we will be exhibiting and speaking about the problem of fragmented communication.
At the same time, we will also be represented at Elastix World, in Bogotá, Columbia, October 7-8, where Matthew is one of the keynote speakers! Matrix is also sponsoring this event.
Finally, we will be in Orlando, Florida for AstriCon, October 13-15. Matrix is sponsoring this conference as well, and we also have a speaking slot.
If you are going to any of these events, please do say hi - we are looking forward to talking to both old and new acquaintances! See you there!
