Many people will have noticed disruption in #matrix:matrix.org and #matrix-dev:matrix.org on Sunday, when a validation bug in Synapse was exploited which allowed a malicious event to be inserted into the room with 'depth' value that made the rooms temporarily unusable. Whilst a transient workaround was found at the time (thanks to /dev/ponies, kythyria and Po Shamil for the workaround and to Half-Shot for working on a proposed fix), we're doing an urgent release of Synapse 0.28.1 to provide a temporary solution which will mitigate the attack across all rooms in upgraded servers and un-break affected ones. Meanwhile we have a full long-term fix on the horizon (hopefully) later this week.
This vulnerability has already been exploited in the wild; please upgrade as soon as possible.
The 'depth' parameter is used primarily as a way for servers to signal the intended cosmetic order of their events within a room (particularly when the room's message graph has gaps in it due to the server being offline, or due to users backfilling old disconnected chunks of conversation). This means that affected rooms may experience message ordering problems until a full long-term fix is provided, which we're working on currently (and tentatively involves no longer trusting 'depth' information from servers). For full details you can see the proposal documents for the temporary fix in 0.28.1 and the options for the imminent long-term fix.
We'd like to acknowledge jzk for identifying the vulnerability, and Max Dor for providing feedback on the fixes.
As a general reminder, Synapse is still beta (as is the Matrix spec) and the federation API particularly is still being debugged and refined and is pre-r0.0.0. For the benefit of the whole community, please disclose vulnerabilities and exploits responsibly by emailing security@ or DMing someone from +matrix:matrix.org. Thanks.
Changes in synapse v0.28.1 (2018-05-01)
SECURITY UPDATE
Clamp the allowed values of event depth received over federation to be [0, 2^63 - 1]. This mitigates an attack where malicious events injected with depth = 2^63 - 1 render rooms unusable. Depth is used to determine the cosmetic ordering of events within a room, and so the ordering of events in such a room will default to using stream_ordering rather than depth (topological_ordering). This is a temporary solution to mitigate abuse in the wild, whilst a long solution is being implemented to improve how the depth parameter is used.Full details at https://docs.google.com/document/d/1I3fi2S-XnpO45qrpCsowZv8P8dHcNZ4fsBsbOW7KABI
Pin Twisted to <18.4 until we stop using the private _OpenSSLECCurve API.
Well now, today sees the release of Synapse 0.28.0!
This release is particularly exciting as it's a major bump mainly thanks to lots and lots of contributions from the wider community - including support for running Synapse on PyPy (thanks Valodim) and lots of progress towards official Python3 support (thanks notafile)!! However, almost all the changes are under the hood (and some are quite major), so this is more a performance, bugfix and synapse internals release rather than adding many new APIs or features
Fractal Hackfest 2018 will take place next month in Strasbourg, hosted by Epitech. They'll be planning a roadmap for the year, and working on their current focus areas.
After the blog post on matrix.org yesterday, lots of attention developed around the news. Take a look at this Hacker News thread with lots of discussion, which has stayed on the front page for nearly 24 hours, and a megathread on Reddit/r/linux: nearly 1000 upvotes and growing.
Project Updates
Clients
libqmatrixclient
kitsune reports a new release of libqmatrixclient, v0.2.1, this is mostly a bugfix release, fully backwards-compatible with v0.2. Check out the release notes for details.
Staying in QMatrix-land, there is a pre-release of Quaternion available. Take a look at v0.0.9, improvements include "redactions, file downloading, room creation and settings editing, better timeline visualisation and much more"!
f0x has been beavering away on neo, and has announced that Alpha 0.01 is available now. neo is a webclient in React, it feels really light and fast. I asked f0x about the uptick in work, and he mentioned he got started with React recently, but likes it a lot.
Says TravisR: "I've whipped together a simple Trello notification bot for matrix: matrix-trello-bot. Future plans include the ability to create/update/delete cards and more granular control of what is notified about. Currently it acts very similar to how the Github notifications bot works."
MTRNord is working on a "matrix DSL", basically a config file language that generates a project Template in multiple languages. The project is just now at the discussion stage, so join #matrix_dsl:matrix.ffslfl.net to check out what's going on. (DSL = Domain Specific Language.)
mxisd, the Identity Server from kamax, saw a v1.0.2 release, following the big one-point-zero last week. Changes include de-duplicated directory search results, plus bugfixes.
Missed this one last week, but matrix-stfu from xwiki is a message filtering tool released recently, it can "mass remove everything which was said by a particular user in a particular room". XWiki use matrix as their internal chat system for the ~30 people at the company.
We're committing to 2-weekly releases come what may, to avoid another massive gap like the one between 0.13 and 0.14.
RCs will get cut starting from Wednesday; actual release then happens on the Monday having given folks a chance to test the RC on /staging (and to iterate on the RCs).
Obviously this is flexible if we need to rush out fixes sooner.
On that note, 0.14.2-rc1 was cut on Wednesday! Please test it at riot.im/staging. It's mainly bugfixes but also the full relayering between riot-web and matrix-react-sdk.
Dave's working on finally hooking in Jitsi as the default conferencing system
t3chguy's been working on Replies, which continue to look awesome
Next up: E2E cross-signing (at last!!!!)
Riot/Mobile
New releases are out! Mainly preparing for sticker viewing and trying to fix the Android push notification situation. PLEASE TELL US IF YOU ARE STILL HAVING ANDROID PUSH NOTIFICATION PROBLEMS!
Lots of review of Android by Benoit - adding in Kotlin support as of today and establishing a formal roadmap for Android work (we'll show the blog post when we have it)
Lots of Matrix-for-French-Government stuff.
Synapse
Synapse 28.0 was released! A major bump mainly thanks to lots and lots of contributions from the wider community.
Massive experimental work on GDPR in progress - doing the thought experiment of pseudonymising MXIDs throughout Matrix on a per-room basis so that it's possible to redact MXIDs in the event of a “right-to-erasure” GDPR event. Rich vdH is leading the work.
The good news is that introducing an MXID abstraction layer like this could help us enormously with some of Matrix's longest-term architectural issues - i.e. account migration and portability; improving on PERSPECTIVEs for managing server identity; future support for P2P Matrix; solving the domain name reuse problem; etc.
The bad news is that it would obviously be a very significant change to the Matrix spec, although we'd be doing it in such a way which minimises impact to client implementers and keep the CS API looking as similar as possible.
We're not sure whether we need to rush this through or not yet; still waiting for final GDPR clarification from lawyers, but it feels like this might be a good opportunity to force us to finally tackle some of these harder problems.
Meanwhile, Erik's Delta State Resolution algorithm work is continuing well; we're hoping to finish & merge it asap in order to get back headroom on the server.
Thankfully the matrix.org synapse itself has been relatively stable this week, other than being completely overloaded causing Freenode to be almost unusable.
We've started using Ansible in production, although the playbooks need some iteration before we're fully happy to announce them & recommend folks use them as an official way of running Synapse in production.
try-matrix-now
There is a update out on try-matrix-now, matrix.org's central listing of projects. Try filtering and see whether benpa mangled the metadata for your project. Submit any changes as markdown PRs on the repo.
Spec
benpa is currently working on a Proposals page for the Spec to properly stack spec proposal status, at last!
Articles around the web
Riot: A Distributed Way of Having IRC and VOIP Client and Home Server
Half-Shot shared an article he wrote, Service notifs with Matrix, about using Matrix and Riot to deliver automated notifications to different types of end-user at his company.
#matrix_dsl:matrix.ffslfl.net As mentioned above, MTRNord is looking at creating a DSL for matrix. In his words: "A room to discuss about making a Matrix DSL to allow non coders to write simple as and bots. I would love to see some people discussing about syntax with me as this is my very first DSL and very first time trying JetBrains MSP. Anyone with Ideas what it should allow to do and how to have the syntax a welcome to join (and people who want to follow of course too)"
#trellobot:t2bot.io (from above) TravisR: "a discussion/developer room for Matrix Trello Bot (https://github.com/turt2live/matrix-trello-bot). This is a bot that notifies rooms of changes to tracked Trello boards, similar to how the Github bot works. Future enhancements include being able to create, update, and delete cards from within Matrix."
This Week in Matrix is printed fresh every week! There will be another post before you know it, so if you'd like to be included join us in #twim:matrix.org and let us know what you've been working on. See you next week!
We're incredibly excited that the Government of France has confirmed it is in the process of deploying a huge private federation of Matrix homeservers spanning the whole government, and developing a fork of Riot.im for use as their official secure communications client! The goal is to replace usage of WhatsApp or Telegram for official purposes.
It's a unbelievably wonderful situation that we're living in a world where governments genuinely care about openness, open source and open-standard based communications - and Matrix's decentralisation and end-to-end encryption is a perfect fit for intra- and inter-governmental communication. Congratulations to France for going decentralised and supporting FOSS! We understand the whole project is going to be released entirely open source (other than the operational bits) - development is well under way and an early proof of concept is already circulating within various government entities.
I'm sure there will be more details from their side as the project progresses, but meanwhile here's the official press release, and an English translation too. We expect this will drive a lot of effort into maturing Synapse/Dendrite, E2E encryption and matrix-{'{'}react,ios,android{'}'}-sdk, which is great news for the whole Matrix ecosystem! The deployment is going to be speaking pure Matrix and should be fully compatible with other Matrix clients and projects in addition to the official client.
So: exciting times for Matrix. Needless to say, if you work on Open Government projects in other countries, please get in touch - we're seeing that Matrix really is a sweet spot for these sort of use cases and we'd love to help get other deployments up and running. We're also hoping it's going to help iron out many of the UX kinks we have in Riot.im today as we merge stuff back. We'd like to thank DINSIC (the Department responsible for the project) for choosing Matrix, and can't wait to see how the project progresses!
English Translation:
The French State creates its own secure instant messenger
By the summer of 2018, the French State will have its own instant messenger, an alternative to WhatsApp and Telegram.
It will guarantee secure, end-to-end encrypted conversations without degradation of the user experience. It will be compatible with any mobile device or desktop, state or personal. In fact until now the installation of applications like WhatsApp or Telegram was not possible on professional mobile phones, which hindered easy sharing of information and documents.
Led by the Interministerial Department of State Digital, Information and Communication Systems (DINSIC), the project is receiving contributions from the National Agency for Information System Security (ANSSI), the IT Directorship (DSI) of the Armed Forces and the Ministry of Europe and Foreign Affairs.
The tool developed is based on open source software (Riot) that implements an open standard (Matrix). Powered by a Franco-British startup (New Vector), and benefiting from many contributions, this communication standard has already caught the attention of other states such as the Netherlands and Canada, with whom DINSIC collaborates closely.
The Matrix standard and its open source software are also used by private companies such as Thales, which has driven the teams to come together to ensure the interoperability of their tools and cooperate in the development of free and open source software.
After 3 months of development for a very limited cost, this tool is currently being tested in the State Secretary for Digital, DINSIC and in the IT departments of different ministries. It should be rolled out during the summer in administrations and cabinets.
"With this new French solution, the state is demonstrating its ability to work in an agile manner to meet concrete needs by using open source tools and very low development costs. Sharing information in a secure way is essential not only for companies but also for a more fluid dialogue within administrations." - Mounir Mahjoubi, Secretary of State to the Prime Minister, in charge of Digital.
Updated the Drupal matrix_api module to pass the auth token as a header instead of as a GET param... the GET param was leaking the token if there was an http exception thrown.
0.3.x of nheko was released! There is progress towards e2e encryption in mujx/mtxclient. Once this encryption is working there, it can eventually be included in nheko.
Fractal 0.1.26 was released this week, which includes an animated scroll to bottom animation, fixed MacOS compilation, and bug fixes. Since then markdown support and many other commits have been spotted...
kitsune released 0.2 of libqmatrixclient a couple of weeks ago, which he says is "the first one more or less functional and stable"!
Highlights include redactions support, files transfer (both ways), rooms creation, account data (tags, direct chat flags). Also there are a few things making libqmatrixclient clients stand out compared to, e.g., Riot: one can set per-room names/avatars, and also the best estimate for the number of unread messages can be taken from the library and shown in the client application."
#monitorbot:t2bot.io is a bot to measure latency between homeservers (as perceived by users).
It's able to report metrics to Prometheus and has it's own built-in dashboard (see https://lag.t2bot.io/ for some horrendous times). In the coming weeks it'll become more production-ready, however in the meantime if people are looking to help iron out the scalability issues, please run an instance on your homeserver.
thrrgilag wrote on behalf of uMatriks, to say that v0.10 was published to the openstore this last week which now includes unread counts for rooms. Those not already familiar, uMatriks is a native matrix client being developed for Ubuntu Touch.
There looks to be lots of activity on Construct (jzk's fork of the charybdis ircd which adds in a C++ homeserver implementation) based on the commit log at https://github.com/jevolk/charybdis. jzk's work there has been highlighting some of synapse's nastier problems (e.g. thundering presence bugs), and it looks like the federation support is progressing steadily, including handling EDUs (presence & typing data).
(jzk) - https://github.com/jevolk/charybdis
matrix-puppet-maxs
tyler shared his "horrifying" project, for people that want to use Matrix to send an SMS via an XMPP bridge to their Android phone, in order to provide SMS and call notifications inside Riot.
Cadair reported that he and SolarDrew have written a bot to bridge a whole slack team into a matrix community. It basically handles all the room admin for you including:
Invite the slack bot to the channel
Creating matrix rooms
Setting users as admins
Setting room name and room avatar and as publicly joinable
Setting up the link with the slack AS
Creating and adding rooms to the community
Inviting all members of the community to the rooms
It does these things on a 1 min cron, to pick up new channels as they are created on the slack side.
remaeus wrote in to say "we (+rpg:verse.im) just launched an initial build of #idlerpg:verse.im (now also available at #idlerpg:matrix.org) — an automated RPG that rewards you for staying online (via the presence API). It's an initial test case for prototyping out a bot framework called "doorman" with multi-service support, but focused on Matrix as we are using it as a "backbone" for our p2p app framework."
The Great Relayering landed - almost all of Riot/Web's code is now gathered together in one sensible place (matrix-react-sdk). No more duplicate PRs or CSS & JS living in different repos!
Folks with open PRs against the riot-web repository will need to reopen them against matrix-react-sdk (or more likely, merge them into their existing equivalents on matrix-react-sdk)
Michael (t3chguy's) Replies work has almost landed and is looking fantastic!
Next up: improving test coverage slightly; fixing regressions in 0.14; and a few other minor bits and bobs before we land 0.15.
Riot/Mobile
New releases on the horizon for both iOS & Android
Welcomed Benoit to the Android team, acting as lead android dev!
Finalising all new UX for configuring push notification permissions…
Final touches for Sticker viewing (including animated stickers!)
Fix embarrassing crash on iOS on certain shapes of Riot/Web URLs...
Supporting French Government Matrix activity
Synapse
Operational firefighting continues: stopping malformed events from impacting federation; investigating thundering herds of presence EDUs; trying to keep Freenode joins from lagging;
Official Ansible playbook repository dev is progressing on the develop branch at https://github.com/matrix-org/matrix-ansible thanks to massive work from Michael K. We'll blog properly about this when ready for use in earnest.
Dave has been doing an interesting experiment in syncing user profile data from HSes into ISes (useful for folks running a private federation with its own IS who want to use the IS as a global user directory); almost complete - we'll see how this goes.
GDPR work is in progress. We're seeing how far we can push the envelope on improving privacy, including a very interesting thought experiment on pseudonyimsing MXIDs to provide 'perfect' right-to-erasure. vdH is on the case. Warning: this may well require a major federation protocol bump if we go through with it.
More Python3 port PRs are landing from NotAFile - thanks! They're getting slightly lost behind everything above.
Dendrite
Still stuck behind Synapse work.
Barring disasters we have at least 2 new folks starting fulltime in the core team (both synapse & dendrite) to help backend progress at the end of May however!
Lots of PRs coming in from APWhitehat though - thanks!!
Spec
uhoreg has been looking at the .well-known URI proposal again
Spec work on the core team is currently stuck behind Riot/Web and Synapse work, sadly, on the basis that burning implementation problems are more visible and harming to the project than burning spec problems.
Matrix in the News
Tom's Hardware provided a great article on about Matrix in the French Government.
Next INpact produced two articles, one from last week introducing a project from the French government to use an open messaging platform, then a follow up provided an extremely thorough story covering Riot & Matrix following the announcement of the French Government encrypted messaging project. (Article is in French, but will Google Translate nicely!)
Huge thank you to nouts who produced a great intro to Riot and Matrix! Check it out! (Note that is also in French.)
New Rooms roundup
Announcing room for all pastafarians out there... #fsm:disroot.org
Thanks for reading the very first This Week in Matrix! There will be another post next week, so if you'd like to be included join us in #twim:matrix.org. See you next week!
Hot on the heels of 0.27.2, notable changes include API support for joining groups/communities as well as a major bug fix (#3082), that is particularly important for those upgrading for the first time in a while. Also new metrics and phone home stats. Phone home stats include better visibility of system usage so we can have a better idea of how synapse's RAM and CPU is behaving in the wild. Also, recording the number of users on the server who have been using it for at least 30 days.
Phone home stats are entirely optional and can be enabled/disabled by setting "report_stats" in homeserver.yaml.
Please consider enabling phone home stats if you currently have not done so - this data is really important to us in improving Matrix as a whole (and justifying future funding for Matrix.org).
v0.27.3-rc1 used a stale version of the develop branch so the changelog overstates
the functionality. v0.27.3-rc2 is up to date, rc1 should be ignored.
Changes in synapse v0.27.3-rc1 (2018-04-09)
Notable changes include API support for joinability of groups. Also new metrics
and phone home stats. Phone home stats include better visibility of system usage
so we can tweak synpase to work better for all users rather than our own experience
with matrix.org. Also, recording 'r30' stat which is the measure we use to track
overall growth of the Matrix ecosystem. It is defined as:-
Counts the number of native 30 day retained users, defined as:-
Users who have created their accounts more than 30 days
Where last seen at most 30 days ago
Where account creation and last_seen are > 30 days"
We released Synapse v0.27.2 today (the first stable release in the 0.27.x series) - it contains loads of work since Synapse v0.26 back in January. The main highlights are:
All the perf improvements which we've been landing as we race to keep the matrix.org homeserver in the face of ever-expanding traffic levels over the last few months
Support for custom storage providers for media repository.
Ability to limit the email addresses allowed to register on your HS, and ability to limit the homeservers your homeserver is allowed to federate with
All new purge API - letting you purge history by date as well as by event (and having a nice new async way of doing it)
Make search work again!!! (by switching from GIST to GIN indexes)
And a few release notes worth calling out:
The common case for running Synapse is not to run separate workers, but for those that do, be aware that synctl no longer starts the main synapse when using -a option with workers. A new worker file should be added with worker_app: synapse.app.homeserver.
This release also begins the process of renaming a number of the metrics reported to prometheus. See docs/metrics-howto.rst
Note that the v0.28.0 release will remove the deprecated metric names.
Fix bug which broke TCP replication between workers (PR #3015)
Changes in synapse v0.27.1 (2018-03-26)
Meta release as v0.27.0 temporarily pointed to the wrong commit
Changes in synapse v0.27.0 (2018-03-26)
No changes since v0.27.0-rc2
Changes in synapse v0.27.0-rc2 (2018-03-19)
Pulls in v0.26.1
Bug fixes:
Fix bug introduced in v0.27.0-rc1 that causes much increased memory usage in state cache (PR #3005)
Changes in synapse v0.27.0-rc1 (2018-03-14)
The common case for running Synapse is not to run separate workers, but for those that do, be aware that synctl no longer starts the main synapse when using -a option with workers. A new worker file should be added with worker_app: synapse.app.homeserver.
This release also begins the process of renaming a number of the metrics
reported to prometheus. See docs/metrics-howto.rst <docs/metrics-howto.rst#block-and-response-metrics-renamed-for-0-27-0>_.
Note that the v0.28.0 release will remove the deprecated metric names.
Features:
Add ability for ASes to override message send time (PR #2754)
Add purge API features, see docs/admin_api/purge_history_api.rst <docs/admin_api/purge_history_api.rst>_ for full details (PR #2858, #2867, #2882, #2946, #2962, #2943)
Add support for whitelisting 3PIDs that users can register. (PR #2813)
Add /room/{'{'}id{'}'}/event/{'{'}id{'}'} API (PR #2766)
Add an admin API to get all the media in a room (PR #2818) Thanks to @turt2live!
We just rushed out an urgent hotfix release for Synapse 0.26.1, addressing a nasty bug in the ujson library which causes it to misbehave badly in the presence of JSON containing very large >64-bit integers. Anyone whose synapses are currently filling up with "Value too big!" errors will want to upgrade immediately from https://github.com/matrix-org/synapse/releases/tag/v0.26.1.
Sorry for the inconvenience.
Changes in synapse v0.26.1 (2018-03-15)
Bug fixes:
Fix bug where an invalid event caused server to stop functioning correctly, due to parsing and serializing bugs in ujson library.
Heads up that we made an emergency release of Riot/Web 0.13.5 a few hours ago to fix a XSS vulnerability found and reported by walle303 - many thanks for disclosing it responsibly.
Please upgrade to Riot/Web 0.13.5 asap. If you're using riot.im/app or riot.im/develop this simply means hitting Refresh; otherwise please upgrade your Riot deployment as soon as possible. Alpine, Debian and Fedora/RPM packages are already updated - huge thanks to the maintainers for the fast turnaround.
The issue lies in the relatively obscure external_url feature, which lets bridges specify a URL for bridged events, letting Riot/Web users link through to the 'original' event (e.g. a twitter URL on a bridged tweet). The option is hidden in a context menu and labelled "Source URL", and is only visible on events which have the external_url field set. Unfortunately Riot/Web didn't sanitise the URL correctly, allowing a malicious URL to be injected - and this has been the case since the feature landed in Riot 0.9.0 (Nov 2016).
If you're not able to upgrade to Riot/Web 0.13.5 for some reason, then please do not click on the 'Source URL' feature on the event context menu:
TL;DR: We built a proof-of-concept for FOSDEM of the world's first(?) 3D video calling using Matrix and the iPhone X... and it looks like this!!
Last year we spent a few weeks putting together a proof of concept of using Matrix as an open, interoperable communication layer for VR/AR - showing how you can use it as an open signalling protocol to connect users within (and between) virtual worlds, with full-mesh E2E encrypted video conferencing in VR; WebRTC calls overlaid on 360 degree video, and other fun stuff. The reasons for building the demo were quite eclectic:
Try to highlight that Matrix is about much more than just about instant messaging or team chat
Try to encourage the community to jump in and build out more interesting use cases
Learn where the state of the art in WebVR + WebGL is
Kick off the process of encouraging folks to think about storing world geometry and physics in Matrix
Have a fun visual demo we could show to excite potential investors in New Vector (which comically backfired when the investment community spontaneously decided that VR is still too early).
In the end it succeeded on some points (highlighting exotic uses of Matrix; learning all about WebVR) and failed on others (a surge in Matrix-for-VR) - although we did have a lot of fun showing it off at the ETHLDN meetup back in October. (Eagle eyed viewers may be amused to spot team Status & Matrix sitting together in the audience ;)
However, we still believe that Matrix is the missing link for decentralised communication within VR/AR, and we were lucky enough to get a talk about Matrix+WebRTC+WebVR accepted to the Real-Time Communications Devroom at FOSDEM 2018! So, given a new chance to show the world how cool Matrix-powered comms could be in VR/AR, myself and Dave Baker went on a (very) quick detour to update the demo a little...
One of the issues of the original demo is that the video calling bits were just putting plain old video planes into the scene - floating television screens of 2D video content, if you will. This is better than nothing, but it's sort of missing the whole point of VR/AR: surely you want to see who you're talking to in 3D? Ideally they should have the same presence as if they were physically in your virtual space. This could also be a big step towards fixing one of the oldest problems of video calling: gaze correction. We've been obsessed about gaze correction since our early days (pre-Matrix) building mobile video calling stacks: gaze correction tries to fix the fact that the break in eye contact caused by staring at the screen (rather than the camera) has a terrible impact on the emotional connection of a video call. But if the person you are talking to is 3D, you can always rotate them in 3D space (or reposition yourself) to correct their line of sight - to re-align their gaze so they're actually looking (in VR) at the thing they're looking at in real life!
Back in early 2017 it would have been wildly ambitious to build an off-the-shelf 3D video calling app - but this changed overnight in late 2017 with the introduction of the iPhone X and its TrueDepth infrared-dot-projector based depth camera; effectively a mini-Kinect. Suddenly we have a mainstream high quality depth+video camera perfectly optimised for 3D video calling, with excellent API support from Apple. So we decided to see if we could be first in the world (as far as we know) to do 3D video calling using the iPhone X, using Matrix to signal the WebRTC media and using our WebVR demo as the viewing environment!
Step 1: Hack on WebRTC to add support for the iPhone X depth camera as a capture device. This is pretty easy, at least if you're just swapping WebRTC's AVFoundationVideoCapturer to request the depth camera instead of the video camera: https://github.com/matrix-org/webrtc/commit/c3044670d87c305d8f8ee72751939e281bf5223f is the starting point.
Step 2: Build a custom Riot/iOS with the right WebRTC SDK. This is relatively easy thanks to Riot/iOS using CocoaPods and Google shipping a pod for WebRTC these days - it was a matter of tweaking Google's pod so it could be referred to directly as a local project by Riot/iOS (and so that it provided debug symbols in the form CocoaPods expects). Brief notes are at https://github.com/matrix-org/webrtc/blob/matthew/depth/matrix/build_instructions.txt - many thanks to Manu for helping on this :)
Step 3: Decide how to encode the depth buffer. Now, the official WebRTC working group quite correctly insists that depth data should be treated as a first class citizen which is modelled and compressed in its own right. However, it looks like nobody has added first-class depth support to official WebRTC yet - and if we want to be able to easily display 3D calls on generic browsers capable of running WebVR+WebRTC+Matrix, we have no choice but do the ugly thing and encode the depth into a video signal which can be compressed with VP8/VP8/H.264 etc.
A quick search showed that some folks had already proposed a method for encoding depth data into a video signal, back in the days of the Kinect: https://reality.cs.ucl.ac.uk/projects/depth-streaming/depth-streaming.pdf. The paper outlines a fairly simple approach: you encode the 16-bit depth data into the three 8-bit colour channels; putting the coarse depth data into Blue, and then finer-grained depth data into Red and Green, encoding it as a periodic triangle wave:
Placing a video call through to another Matrix client then coughed up a video stream that looks like this:
As you can see, closer things (my head) are bluer than further things (the wall), and everything's covered with trippy red & green stripes to refine the fine detail. For the record, the iPhone TrueDepth camera emits 640x480 depth frames at around 24Hz.
Unfortunately, it showed that the depth encoding really wasn't working very well... you can just about make out my head, but there are dots flying around all over the place, and when you view it in profile the 3D effect was almost entirely missing.
The main problems seem to be:
Whenever there's a big jump in depth, the stripes get incredibly noisy and don't compress at all well, generating completely corrupt data at edges of objects (e.g. the sides of my head)
The complexity of the pattern as a whole isn't particularly compression-friendly
The contrast of the red/green stripes tends to dominate, causing the arguably more important blue to get overpowered during compression.
Converting from 4:4:4 RGB to 4:2:0 YUV (NV12) as required by WebRTC and then back to RGB inevitably entangles the colours - meaning that the extreme contrast of the red/green stripes is very visible on the blue channel after round-tripping due to sampling artefacts.
I probably made a mistake by bitwise casting the 16-bit half-precision floating point depth values directly onto the 16-bit unsigned int lookup index, rather than interpreting the float as a number and building a new index into the lookup table based on its numeric value. As a result, depth values being encoded ended up having a much lower range than they should.
There are probably other bugs too.
Step 5: Give up on the fancy depth encoding (for now): https://github.com/matrix-org/webrtc/commit/2f5d29352ce5d80727639991b1480f610cbdd54c. In practice, simply picking a range of the 16-bit half-precision floats to fit in the integer range [0,255] turns out to be good enough for a quick demo (i.e. 8-bit depth buffer, but over a small subset of the 16-bit depth space) - the dot cloud suddenly looked a lot more 3D and recognisable:
However, the dot cloud obviously has some limitations - especially when you zoom in like this.
Step 7: Replace the dot cloud with a displacement-mapped mesh so that it's solid. So as a final tweak for the demo, Dave switched out the dot cloud for a simple A-Frame plane with 640x480 vertices, keeping the same vertex shader. Ironically this is where we hit some nasty problems, as for some reason the video texture started being applied to the depth texture (albeit flickering a bit) - eventually we realised that the flickering was the vertex shader inexplicably flapping between using the depth and the video texture for the displacement map. At this point we compared it between laptops, and it turns out that for some reason the integrated Intel graphics on Dave's Macbook Pro was choking on the two video textures, whereas a AMD Radeon R9 M370X got it right. It's unclear if this was actually a GPU bug or an A-Frame or Three.js or WebGL or Chrome bug. Eitherway, on switching laptop to one with discrete graphics it started working perfectly! Finally, we tweaked the shader to try to reduce smearing, by discarding vertices where there are big discontinuities in depth (through looking at the partial derivatives of the depth texture). This isn't perfect yet but it's better than nothing. https://github.com/matrix-org/matrix-vr-demo/compare/bbd460e81ff1336cd63468e707d858d47261ea42...06abe34957732ba8c728b99f198d987fe48d0420
And here's the end result! (complete with trancey soundtrack as the audio we recorded at FOSDEM was unusable)
Conclusion:
Hopefully this gives a bit of a taste of what proper 3D video calling could be like in VR, and how (relatively) easy it was at the Matrix level to add it in. If anyone wants to follow along at home, the various hacky branches are:
If you'd like to get involved with hacking on Matrix in VR, please come hang out at #vr:matrix.org.
Also, New Vector (where most of the core team work) is also hiring for VoIP/VR specialists right now, so if you'd like to work on this sort of thing fulltime, please contact us at [email protected] asap!
