Over the past two weeks, we have been hunting down some more performance issues in Synapse, as well as fixing a few potential bugs in the new backfill feature that we introduced in 0.9.1. For those that were having issues, this release should really help speed up when your server joins larger remote rooms.
We have also been busy hacking on end-to-end encryption, which is very exciting. Hopefully we will have more details to share about that soon!
Changes in synapse v0.9.2 (2015-06-12)
======================================
General:
Use ultrajson for json (de)serialisation when a canonical encoding is not required. Ultrajson is significantly faster than simplejson in certain circumstances.
Use connection pools for outgoing HTTP connections.
Process thumbnails on separate threads.
Configuration:
Add option, gzip_responses, to disable HTTP response compression.
Federation:
Improve resilience of backfill by ensuring we fetch any missing auth events.
Improve performance of backfill and joining remote rooms by removing unnecessary computations. This included handling events we had previously handled as well as attempting to compute the current state for outliers.
Next weekend, June 13 and 14, the global TADHack takes place all over the world. You can participate on site or remotely, and there are a lot of different prizes to be won - in total the prize pot is worth $35k!
For the best two hacks using our technology, we will award a whole lot of Tessel modules! Tessel is a new breed of development board that runs entirely on Node.js, and they come with different modules you can plug in - for more information, see: getting started & sample projects.
Both prizes will include several tessel modules, including:
multiple core tessel boards
multiple servo modules and many servo motors
multiple ambient modules
multiple accelerometer modules
camera module
GPS module with antenna
microsd module
bluetooth module
audio module
climate module
relay module
RFID module
DIY module kit
Matrix.org will be present at the London site, Idea London in Shoreditch, where we will help both local and remote participants (via #matrix:matrix.org) using the Matrix APIs as part of their hacks.
So if you have some spare time next weekend - why not have a think about what could be a cool hack and join us for the global TADHack event! See you there!
We are back from Kamailio World, where we presented and participated in James Body's "Dangerous Demos". We were racing against the deadline, but managed to join the demos at the very last minute - and even win the award for "Most Entertaining Demo"!
It was great to catch up with old acquaintances - and meet many new ones! There were only around 150 people at Kamailio World, but given the area of expertise is very specialised, you can pretty much start talking to anyone and have a really interesting conversation.
A video from the dangerous demo event is available here:
The Parrot Drone we use in the demo has a 14 megapixel fisheye camera with advanced stabilization techniques which means that you can't actually see what happened when everybody went "ooh" - I assure you the "flip" command does exactly what you would expect!
Thanks to everybody who talked to us at Kamailio - and as always, come find us in the #matrix:matrix.org room on Matrix!
In the excitement of KamailioWorld last week we completely forgot to mention that we released Synapse 0.9.1.
This is a pretty important performance and stability update of all the new stuff that landed in 0.9.0, as well as landing a major new feature in the form of 'backfill': we finally have the ability to correctly sync in historical conversation history over federation for newly federated rooms. In other words, if you join a remote room you should be able to navigate its history as intended.
We highly recommend upgrading to 0.9.1 for all the performance improvements and backfill support - get it now from http://github.com/matrix-org/synapse.
Changes in synapse v0.9.1 (2015-05-26)
======================================
General:
Add support for backfilling when a client paginates. This allows servers to
request history for a room from remote servers when a client tries to
paginate history the server does not have - SYN-36
Fix bug where you couldn't disable non-default pushrules - SYN-378
Fix register_new_user script - SYN-359
Improve performance of fetching events from the database, this improves both
initialSync and sending of events.
Improve performance of event streams, allowing synapse to handle more
simultaneous connected clients.
Federation:
Fix bug with existing backfill implementation where it returned the wrong
selection of events in some circumstances.
Improve performance of joining remote rooms.
Configuration:
Add support for changing the bind host of the metrics listener via the metrics_bind_host option.
In our continuous journey around the world to promote Matrix, this week we have come to Kamailio World in Berlin, Germany. During the conference, there will be 5 technical workshops and 28 presentations about SIP, VoIP, WebRTC and other real time communication technologies - and Matthew will talk about Matrix at 11am on Friday.
I'm looking forward to lots of interesting talks (full schedule here), including an open discussion panel with Randy Resnick about real-time communications at 17:10 Thursday evening. Of course there will also be dangerous demos - and hopefully lots of people interested in Matrix! If you are going to the conference, please come and say hello - we will be exhibiting as well as presenting, and we will be there all day Thursday and Friday.
Amandine and I just got back from WebRTC World 2015 in Miami - the conference was a great success: a fantastic opportunity to meet up with many of the companies who are supporting Matrix and give everyone an update on what Matrix is up to with our Keynote: Defragmenting the Internet for fun and non-profit!.
We also had a little too much fun in the demo shoot-out - hooking up a Parrot Bebop quadrocopter into Matrix using a Matrix-enabled Janus WebRTC Gateway. The problem here is that the Parrot firmware and C SDK provides H.264 video, but doesn't package it up at all for use with WebRTC - let alone using interoperable signalling like Matrix. So this is a classic use of Matrix to expose a simple open consistent interface to a system which is otherwise is stuck with a proprietary non-web-friendly API. The code hasn't been tidied up yet, but our hacked Matrixified fork of Janus is up at https://github.com/matrix-org/janus-gateway/tree/ardrone3 if anyone has a drone and is crazy enough to want to experiment with it :)
Meanwhile, we also showed OpenWebRTC-powered VoIP on the latest develop iOS Matrix Console app talking hardware-accelerated H.264 through to Firefox on the desktop. My ancient 2010 MacBook Pro did its best to sabotage the demo (turns out that 1080P AirPlay + Firefox WebRTC is a bridge too far), but it gave a good idea of what's to come. Many thanks to the OpenWebRTC team for lots of help in getting the demo together in time!
It turns out that all the demo excitement was worth it in the end, as the jury seemed to like what Matrix is up to and was kind enough to award us more points than any of the other 13 demos... meaning that we won Best In Show!!. Huge thanks to the judges for believing in the Matrix vision, and congratulations to all the other demoists too :)
...and the actual video stream that the drone transmitted before I crashed it (recorded on Janus) is at...
Finally, our grand finale was meant to be combining the two demos, and showing OpenWebRTC decoding the H.264 from the Drone in hardware on an iPhone - using Matrix of course to set up the call and control the drone. Alas a TURN-related bug got in the way of this working, but we just fixed it up in the office this morning, and I'm proud to show the first ever Parrot Bebop -> Janus -> Matrix -> OpenWebRTC video stream!! (and very exciting it is too...)
Huge thanks again to Dave for doing the Matrix integration with Janus, Stefan and Rob from OpenWebRTC for all the help on the OWR side, and Manu & Giom for porting the OpenWebRTC pull request to MatrixKit and landing it in iOS Console Develop for the demo!
We have pushed out a new release of both Synapse, our reference server implementation, and matrix-angular-sdk, our reference webclient implementation!
The major new feature in Synapse is that you can now run Synapse backed by a PostgreSQL database. This increases performance and allows Synapse to scale much better! This, as well as various performance related bug fixes, should make things much snappier than before. Of course, you can still run SQLite; it's up to you what you want to use.
In the webclient you can now change or reset your password - we have had this feature requested a few times (although honestly I'm surprised it hasn't been mentioned even more - maybe people are just better than me at remembering/managing their passwords) so this should be a welcome addition! We also fixed a memory leak in Angular, so again expect better performance!
Finally, we have done some work on improving the Application Service API, making it more reliable and secure. Please see the upgrade notes as well as the full changelog below.
Changes in Synapse v0.9.0:
General:
Add support for using a PostgreSQL database instead of SQLite. See postgres.rst for details.
Add password change and reset APIs. See Registration in the spec.
Fix memory leak due to not releasing stale notifiers - SYN-339.
Fix race in caches that occasionally caused some presence updates to be dropped - SYN-369.
Check server name has not changed on restart.
Add a sample systemd unit file and a logger configuration in contrib/systemd. Contributed Ivan Shapovalov.
Federation:
Add key distribution mechanisms for fetching public keys of unavailable remote home servers. See Retrieving Server Keys in the spec.
Configuration:
Add support for multiple config files.
Add support for dictionaries in config files.
Remove support for specifying config options on the command line, except for:
--daemonize - Daemonize the home server.
--manhole - Turn on the twisted telnet manhole service on the given port.
--database-path - The path to a sqlite database to use.
--verbose - The verbosity level.
--log-file - File to log to.
--log-config - Python logging config file.
--enable-registration - Enable registration for new users.
Application services:
Reliably retry sending of events from Synapse to application services, as per Application Services spec.
Application services can no longer register via the /register API, instead their configuration should be saved to a file and listed in the synapse app_service_config_files config option. The AS configuration file has the same format as the old /register request. See application_services.rst for more information.
Changes in Matrix Angular SDK 0.6.6:
Features:
Add password change and reset feature using v2_alpha APIs.
Bug fixes:
Fix memory leak caused by not removing a watcher on the root scope.
Matrix.org is happy to be sponsoring and talking at the WebRTC Conference and Expo in Miami, Florida, 12-14 May. Both Amandine and Matthew will be there - please come have a chat by booth #22! This is one of the longest running WebRTC Events, and Matthew is delivering one of the keynotes of the conference on Wednesday 4:00-4:30pm in room K-07.
Matthew will also participate in the "Open Source Options for WebRTC Development" session in room D2-02 at 9:50am on Wednesday (full agenda here).
Finally, Matrix will also be part of the WebRTC World Demos in room X-07 sometime between 4:30 and 7:30pm on Wednesday. Expect a dangerous demo!
Just a quick note to say thanks to everyone who came to talk to us at SMR9 yesterday. SMR is a great way for developers looking for jobs and startups needing engineers to have a chat.
We had a very busy day with plenty of people interested in Matrix and eager to join the team. We received a lot of CVs and will get back to you - but in the meantime please check out our code and come say hi in the Matrix HQ room, using any of these Matrix clients!
If you missed SMR, or just generally is interested in working for Matrix.org - please feel free to send your CV to us - we need all kinds of developers, with skills ranging from backend and frontend to mobile development!
As a follow-on to the previous post, Apple approved version 0.4.0 of the iOS Matrix Console app over the weekend - grab it now from the app store!. Whilst cosmetically it looks almost identical to 0.3.x, the architecture is completely different given the app is now built using MatrixKit - and stability has improved considerably as a result. Please install and let us know how you get on!
