Matrix in Google Summer of Code!

08.03.2016 00:00 — GSOC Oddvar Lovaas
GSoC2016Logo

We are very happy to be one of the companies selected for Google Summer of Code (GSoC) 2016!

GSoC is a great, global opportunity for students to work on open source projects during their university summer break. The idea is for students to propose a project for any of the open source organisations picked by Google, and - if accepted - receive a stipend for working on it. We are very eager to see what projects students will propose - we have written up some ideas here, but students are expected to do some research and come up with projects themselves.

If you are a student wanting to participate in GSoC for Matrix, please come talk to us in #gsoc:matrix.org - we are happy to discuss project ideas and review application drafts. We have also added some general tips on what to include in the application here.

Applications can be submitted starting next Monday, so there's still plenty of time to have a play with Matrix and come up with a cool project idea.

Good luck!

Add Your Matrix Project

25.02.2016 00:00 — General Oddvar Lovaas

The try-matrix-now page is now being generated by jekyll and all the project pages have been moved to the matrix-doc project on github.

The idea is to make it very easy for anyone to add or update a project entry. All you need to do is to submit a PR with the project details; feel free to start with the template, and you can also add images (thumbnail and/or a main picture for the project page) to the images subfolder (just use the same relative URL that is in the template). Any kind of project using Matrix is welcome; if you are unsure which category to use, just use "other".

Jekyll requires a date in the project filename; we use the date to sort the various project lists (newest projects first). It might be best to submit new entries with a date like 2015-01-01.

Any questions or comments? Come talk to us in #matrix:matrix.org!

Android Matrix Console 0.5.3

16.02.2016 00:00 — General Oddvar Lovaas

We have put an updated version of the Android Matrix Console app (v0.5.3) on the Play store!

This release mainly includes performance improvements, such as using the new "V2" sync API, and other optimisations which should make your user experience a lot nicer. There's also a few new features in the SDK (e.g. tags support) - these will be added to the app hopefully soon.

For the full list of changes, look at the CHANGES files in the android console and SDK projects

Get it from the Google play store!

Enjoy! And please do let us know your feedback in #matrix:matrix.org or #android:matrix.org!

Advanced Synapse setup with Let's Encrypt

10.02.2016 00:00 — General David Baker

So, you've installed an configured synapse and started chatting from your very own Matrix home server? What's the next step? Well, right now you're probably accessing your new home server over plaintext HTTP, which is bad, particularly because you'll be sending your password over this connection when you log in. You could connect to Synapse's secure HTTP port, but your browser won't trust it by default because you'd normally need to pay for a certificate that your browser would recognise. That is, until recently!

Let's Encrypt is a new initiative that issues SSL certificates free of charge, in an effort to make SSL universal on the Internet. In this blog post, we'll be walking through an example of how we can use this service to get ourselves a securely hosted Synapse.

We're going to assume you have a Synapse installed and listening on the standard ports, 8008 and 8448. If not, follow the Synapse README and come back here when you're done. Everybody ready? Okay, good.

So, in order to get a certificate from Let's Encrypt, we need to prove that we own our domain. The simplest way to do this is to host some special files on our web server. Now, Synapse won't do this. We could use a separate web server, but then we'd have to stop Synapse and start the other web server every time we renewed our certificate, and that means downtime. Instead, let's put our Synapse behind a proper web server and let that serve the files. This has added advantages, including that we can host our Matrix home server on the standard port 443 without having to run Synapse as root.

For this example, we're going to use NGINX, so start by installing NGINX in whatever way your Linux distribution of choice recommends.

Now, you should have a webroot for your new web server somewhere. Hopefully your helpful Linux distribution has started you off with a config file - let's see:

# nano /etc/nginx/nginx.conf

We're looking for the 'server' section of that file. We need to make it look something like this:

    server {'{'}
        # Make sure this is 0.0.0.0: no use listening on 127.0.0.1 or we'll only be
        # serving to ourselves! There's no port here, which means we'll listen on
        # port 80
        listen 0.0.0.0;

        server_name example.com www.example.com;

        access_log /var/log/nginx/example.com.access_log main;
        error_log /var/log/nginx/example.com info;

        # This is where we put the files we want on our site
        root /var/www/examplecom/htdocs;

        # Here's where it gets interesting: This will send any path that starts
        # with /_matrix to our Synapse!

        location /_matrix {'{'}
            proxy_pass http://localhost:8008;
        {'}'}
    {'}'}

When you're happy with the look of that file, let's restart the server:

# nginx -s reload

Before we go any further, let's test our new configuration:

$ curl http://example.com/_matrix/key/v2/server/auto
{'{'}"old_verify_keys":{'{'}{'}'},"server_name":"example.com","signatures":{'{'}"example.com":{'{'}"ed25519:auto":"RWb+w6vHUUokoDgElwG6Cg50ezZvBrzXtJmJIH8jEwI5x0JQ7prn3FwjhbgKTH5jE7J8Ily3HEc4COn4JCCvCA"{'}'}{'}'},"tls_fingerprints":[{'{'}"sha256":"DMbzSZ5Uj7/6p/RT/UtQYJLHm5o0TwBSVYXsqpDdVDs"{'}'}],"valid_until_ts":1455203001035,"verify_keys":{'{'}"ed25519:auto":{'{'}"key":"1YiTDjmE86AlmrbIYE2lyqauV9wPo8jw2kxZAZFfl/Q"{'}'}{'}'}{'}'}

Those are your server's public keys! Now we have a web server running, we can get our SSL certificate. Let's Encrypt have their own client which will automate everything including rewriting your NGINX config file, however that means it has a large number of dependencies and needs to be run as root. For this example, we're going to use the much simpler acme_tiny.py. I'm going to assume you have a user called, 'letsencrypt', so, as root, let's set up the place for it to write its challenge files:

# mkdir /var/www/examplecom/htdocs/.well-known/acme-challenge
# chown letsencrypt:users /var/www/examplecom/htdocs/.well-known/acme-challenge

Now let's switch to our letsencrypt user:

$ ssh [email protected]

We'll start by getting ourselves a copy of acme_tiny.py:

$ git clone https://github.com/diafygi/acme-tiny.git

Now let's set up a directory structure (let's say we might want to manage more than one domain someday):

$ mkdir examplecom
$ cd examplecom
$ ln -s /var/www/examplecom/htdocs/.well-known/acme-challenge challenges

Now, we'll need to generate two keys for Let's Encrypt, and account key and a domain key. The former is what we use to identify ourselves to Let's Encrypt and the latter is the key we use to do the actual SSL.

$ openssl genrsa 4096 > letsencrypt_examplecom_account.key
Generating RSA private key, 4096 bit long modulus
..++
.................................................................................................................................................................................................................................................................................................................................................................................................................++
e is 65537 (0x10001)
$ chmod 600 letsencrypt_examplecom_account.key
$ openssl genrsa 4096 > letsencrypt_examplecom_domain.key
Generating RSA private key, 4096 bit long modulus
.............++
.............................................................................................................................................................................................++
e is 65537 (0x10001)
$ chmod 600 letsencrypt_examplecom_domain.key

Now, store those keys somewhere safe! After you've done that, let's generate a certificate request for our domain. Note that we're requesting one for both example.com and www.example.com: this isn't strictly necessary for Matrix but could be useful if we want to host a website too.

$ openssl req -new -sha256 -key letsencrypt_examplecom_domain.key -subj "/" -reqexts SAN -config <(cat /etc/ssl/openssl.cnf <(printf "[SAN]\\nsubjectAltName=DNS:example.com,DNS:www.example.com")) > examplecom.csr

Okay, we have our keys, our certificate request, and somewhere to host our challenge files, so we're ready to request a certificate! Be careful about this part and make sure you've got everything right, because Let's Encrypt enforce strict rate limits on the number of certificates you can request for one domain. Here we go:

$ python ~/acme-tiny/acme_tiny.py --account letsencrypt_examplecom_account.key --csr examplecom.csr --acme-dir challenges/ > examplecom.crt
Parsing account key...
Parsing CSR...
Registering account...
Registered!
Verifying example.com...
example.com verified!
Verifying www.example.com...
www.example.com verified!
Signing certificate...
Certificate signed!

Is that it, did it work? Well, let's see:

$ openssl x509 -in examplecom.crt -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            01:02:22:77:02:1b:eb:d5:3d:c3:14:6d:87:43:22:3d:fc:0f
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3
        Validity
            Not Before: Feb  6 21:37:00 2016 GMT
            Not After : May  6 21:37:00 2016 GMT
        Subject: CN=example.com
        Subject Public Key Info:
[etc]

Congratulations, you have an official, signed certificate for your domain! Now, before we can use it, we need to add the Let's Encrypt certificate to it, because our web server needs to send both:

$ wget https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem
--2016-02-06 23:38:55--  https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem
Resolving letsencrypt.org... 23.66.17.98, 2a02:26f0:60:489::2a1f, 2a02:26f0:60:481::2a1f
Connecting to letsencrypt.org|23.66.17.98|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1675 (1.6K) [application/x-x509-ca-cert]
Saving to: ‘lets-encrypt-x3-cross-signed.pem'
lets-encrypt-x3-cross-signed.pe 100%[======================================================>]   1.64K  --.-KB/s   in 0s
2016-02-06 23:38:55 (61.5 MB/s) - ‘lets-encrypt-x3-cross-signed.pem' saved [1675/1675]
$ cat examplecom/examplecom.crt lets-encrypt-x3-cross-signed.pem >examplecom/examplecom_cert_chain.crt

Now's let's symlink it in place, along with the domain key, so we can renew it easily later. We'll need to be root again for this:

$ ssh [email protected]
# ln -s /home/letsencrypt/examplecom/examplecom_cert_chain.crt /etc/ssl/nginx/examplecom_cert.pem
# ln -s /home/letsencrypt/examplecom/letsencrypt_examplecom_domain.key /etc/ssl/nginx/examplecom_key.pem

Now, one more crucial thing we have to do before using our SSL is to give NGINX some Diffie Hellman parameters. This is a good thing to do for any SSL configuration (it will increase your score on SSL Labs) but it's absolutely crucial for us because Synapse will only negotiate forward secret connections, so otherwise other Matrix home servers will refuse to talk to us! (Technically, Synapse also support elliptic curve Diffie Hellman, which doesn't need DH parameters, but not all Synapses will support this.) You'll already have some Diffie Hellman parameters from you existing Synapse, so you could use them:

# cp /home/synapse/synapse/matrix.example.com.tls.dh /etc/ssl/nginx/examplecom_dhparams.pem

...or you can generate your own. You'll probably want to do this on your desktop or laptop if you have OpenSSL installed, it will be much faster:

$ openssl dhparam -out examplecom_dhparams.pem 2048
Generating DH parameters, 2048 bit long safe prime, generator 2
This is going to take a long time
........................................................+................[etc, etc]
$ scp examplecom_dhparams.pem [email protected]:/etc/ssl/nginx/examplecom_dhparams.pem

Now, let's get our new certificate in action! Open up your NGINX config file again, and add another server block that look like this:

    server {'{'}
        listen 0.0.0.0:443;
        server_name example.com www.example.com;
        ssl on;
        ssl_certificate /etc/ssl/nginx/examplecom_crt.pem;
        ssl_certificate_key /etc/ssl/nginx/examplecom_key.pem;
        ssl_dhparam /etc/ssl/nginx/examplecom_dhparams.pem;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_prefer_server_ciphers on;
        # mozilla intermediate list, jan 2016
        ssl_ciphers "ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA";
        ssl_session_cache shared:SSL:50m;
        access_log /var/log/nginx/examplecom.ssl_access_log main;
        error_log /var/log/nginx/examplecom.ssl_error_log info;
        root /var/www/examplecom/htdocs;
        location /_matrix {'{'}
            proxy_pass http://localhost:8008;
        {'}'}
    {'}'}

It looks pretty similar to our previous server block, except for all that stuff about SSL in the middle. We're pointing NGINX at our certificate, key and Diffie Hellman parameter files and specifying what protocols and ciphers we want our server to talk. The long list here is taken from Mozilla's Server Side TLS guidelines and is their 'Intermediate' list. See that page for more information on what that means, and choose a different list of ciphers if you prefer: just remember we must support at least the ephemeral Diffie Hellman ciphers, or other home servers won't talk to us!

Now let's restart our NGINX and see if it works:

# nginx -s reload

...and that command again, this time with https:

$ curl https://example.com/_matrix/key/v2/server/auto
{'{'}"old_verify_keys":{'{'}{'}'},"server_name":"example.com","signatures":{'{'}"example.com":{'{'}"ed25519:auto":"RWb+w6vHUUokoDgElwG6Cg50ezZvBrzXtJmJIH8jEwI5x0JQ7prn3FwjhbgKTH5jE7J8Ily3HEc4COn4JCCvCA"{'}'}{'}'},"tls_fingerprints":[{'{'}"sha256":"DMbzSZ5Uj7/6p/RT/UtQYJLHm5o0TwBSVYXsqpDdVDs"{'}'}],"valid_until_ts":1455203001035,"verify_keys":{'{'}"ed25519:auto":{'{'}"key":"1YiTDjmE86AlmrbIYE2lyqauV9wPo8jw2kxZAZFfl/Q"{'}'}{'}'}{'}'}

Hooray! You should now be able to open a browser to https://example.com/matrix/ and log in securely over SSL!

Renewing Your Certificate

Now, there's one important step left, and that's to set up renewal for the certificate, otherwise we'll find our shiny new SSL will stop working in three months time. We can use the same acme_tiny command to do this:

$ python ~/acme-tiny/acme_tiny.py --account letsencrypt_examplecom_account.key --csr examplecom.csr --acme-dir challenges/ > examplecom.crt
Parsing account key...
Parsing CSR...
Registering account...
Already registered!
Verifying example.com...
example.com verified!
Verifying www.example.com...
www.example.com verified!
Signing certificate...
Certificate signed!
$ wget https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem
--2016-02-06 23:38:55--  https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem
Resolving letsencrypt.org... 23.66.17.98, 2a02:26f0:60:489::2a1f, 2a02:26f0:60:481::2a1f
Connecting to letsencrypt.org|23.66.17.98|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1675 (1.6K) [application/x-x509-ca-cert]
Saving to: ‘lets-encrypt-x3-cross-signed.pem'
lets-encrypt-x3-cross-signed.pe 100%[======================================================>]   1.64K  --.-KB/s   in 0s
2016-02-06 23:38:55 (61.5 MB/s) - ‘lets-encrypt-x3-cross-signed.pem' saved [1675/1675]
$ cat examplecom/examplecom.crt lets-encrypt-x3-cross-signed.pem >examplecom/examplecom_cert_chain.crt

Synapse will automatically pick up the new certificate, but we'll need to tell NGINX to reload:

# nginx -s reload

Setting up a cronjob to automate this is left as an exercise to the reader!

Federation behind the HTTP Proxy

If you like, you can stop reading now: our clients can access our home server securely but other home server are still talking to our Synapse directly on port 8448. This is fine, and if you're happy with this, you can stop reading now. But remember how we made sure other Synapses could talk to our NGINX? Well, why not put federation behind our new web server too?

Now, we need to do a couple of things to make this work: were you looking carefully at the JSON those curl commands returned? If you were, you might have noticed a key called, 'tls_fingerprints'. Our home server serves up a fingerprint of the TLS certificate its using from this API, and we've just given our web server a different certificate, so we need to give Synapse our new certificate.

How are we going to tell other home servers to talk to our NGINX instead? Well, ultimately we're going to change our DNS SRV record to point at port 443 instead of port 8448, but that change could take a while to propagate through caches, so let's test it by having our NGINX listen on port 8448 temporarily. We can do this by copying that same block from above, but with a different port:

    server {'{'}
        listen 0.0.0.0:8448;
        server_name example.com www.example.com;
    [etc]

Don't restart NGINX just yet: we need to tell our Synapse to stop listening on that port first, so lets do that and give it our new certificate:

$ nano /home/synapse/synapse/homeserver.yaml

Now we'll want to find and edit the following lines:

tls_certificate_path: "/etc/ssl/nginx/examplecom_crt.pem"
# We can comment this out, as long as we set no_tls to true below
# tls_private_key_path: "/whatever/path/synapse/generated"
# PEM dh parameters for ephemeral keys
tls_dh_params_path: "/whatever/path/synapse/generated"
# Turn off TLS everywhere (this overrides the listeners section below)
no_tls: True
  - port: 8008
    tls: false
    # We can bind to only localhost since only the local nginx needs to hit this
    bind_address: '127.0.0.1'
    type: http
    # Set this so that Synapse obeys nginx's X-Forwarded-For headers, then IP addresses will be correct in Synapse's logs
    x_forwarded: true
    resources:
      - names: [client, webclient]
        compress: true
      - names: [federation]
        compress: false

Note: if you have an old enough config file that you have 'bind_host' and 'bind_port' directives, now is the time to remove them.

Now let's restart Synapse and our web server to swap over what's listening on our port 8448:

$ synctl restart
# nginx -s reload

Now let's try that test again on port 8448:

$ curl https://example.com:8448/_matrix/key/v2/server/auto
{'{'}"old_verify_keys":{'{'}{'}'},"server_name":"example.com","signatures":{'{'}"example.com":{'{'}"ed25519:auto":"bdca31805e4209f6ff4d644251a29d0cb1dc828a4d6131c57cf8305288f337c0"{'}'}{'}'},"tls_fingerprints":[{'{'}"sha256":"1d9ec66599e229654a79f28e26675fdeb585027553af6d581926e821a6b6527c"{'}'}],"valid_until_ts":1455203001035,"verify_keys":{'{'}"ed25519:auto":{'{'}"key":"1YiTDjmE86AlmrbIYE2lyqauV9wPo8jw2kxZAZFfl/Q"{'}'}{'}'}{'}'}

Notice anything different? The tls_fingerprints part has changed because we now have a different certificate. The signatures/example.com/ed25519:auto value has changed too: that's because that part is a signature of the rest of JSON object, so changing the tls_fingerprints has caused this to change too.

And that's it! If you're happy everything is working, you can then change your DNS SRV record to point at port 443 instead of 8448, then after leaving a few days for the change to propagate through caches, remove the extra server block from your nginx.conf and restart to stop your nginx listening on port 8448.

Synapse 0.13 released!

10.02.2016 00:00 — Tech Matthew Hodgson

Hi all,

Synapse 0.13 was released this afternoon, bringing a new wave of features, bug fixes and performance fixes. The main headlines include: huge performance increases (big catchup /syncs that were taking 20s now take 0.3s!), support for server-side per-room unread message and notification badge counts, ability for guest accounts to upgrade into fully-fledged accounts, change default push rules back to notifying for group chats, and loads of bug fixes. This release incorporates what-was 0.12.1-rc1.

Please note that on first launch after upgrading a pre-0.13 server to 0.13 or later, synapse will add a large database index which may take several minutes to complete. Whilst the index is added the service will be unresponsive.

Please get the new release from https://github.com/matrix-org/synapse and have fun!

Matthew

Full release notes:

Changes in synapse v0.13.1 (2016-02-10) =======================================
  • Bump matrix-angular-sdk (matrix web console) dependency to 0.6.8 to pull in the fix for SYWEB-361 so that the default client can display HTML messages again(!)

Changes in synapse v0.13.0 (2016-02-10)

This version includes an upgrade of the schema, specifically adding an index to the events table. This may cause synapse to pause for several minutes the first time it is started after the upgrade.

Changes:

  • Improve general performance (PR #540, #543. #544, #54, #549, #567)
  • Change guest user ids to be incrementing integers (PR #550)
  • Improve performance of public room list API (PR #552)
  • Change profile API to omit keys rather than return null (PR #557)
  • Add /media/r0 endpoint prefix, which is equivalent to /media/v1/ (PR #595)

Bug fixes:

  • Fix bug with upgrading guest accounts where it would fail if you opened the registration email on a different device (PR #547)
  • Fix bug where unread count could be wrong (PR #568)

Changes in synapse v0.12.1-rc1 (2016-01-29)

Features:

  • Add unread notification counts in /sync (PR #456)
  • Add support for inviting 3pids in /createRoom (PR #460)
  • Add ability for guest accounts to upgrade (PR #462)
  • Add /versions API (PR #468)
  • Add event to /context API (PR #492)
  • Add specific error code for invalid user names in /register (PR #499)
  • Add support for push badge counts (PR #507)
  • Add support for non-guest users to peek in rooms using /events (PR #510)

Changes:

  • Change /sync so that guest users only get rooms they've joined (PR #469)
  • Change to require unbanning before other membership changes (PR #501)
  • Change default push rules to notify for all messages (PR #486)
  • Change default push rules to not notify on membership changes (PR #514)
  • Change default push rules in one to one rooms to only notify for events that are messages (PR #529)
  • Change /sync to reject requests with a from query param (PR #512)
  • Change server manhole to use SSH rather than telnet (PR #473)
  • Change server to require AS users to be registered before use (PR #487)
  • Change server not to start when ASes are invalidly configured (PR #494)
  • Change server to require ID and as_token to be unique for AS's (PR #496)
  • Change maximum pagination limit to 1000 (PR #497)

Bug fixes:

  • Fix bug where /sync didn't return when something under the leave key changed (PR #461)
  • Fix bug where we returned smaller rather than larger than requested thumbnails when method=crop (PR #464)
  • Fix thumbnails API to only return cropped thumbnails when asking for a cropped thumbnail (PR #475)
  • Fix bug where we occasionally still logged access tokens (PR #477)
  • Fix bug where /events would always return immediately for guest users (PR #480)
  • Fix bug where /sync unexpectedly returned old left rooms (PR #481)
  • Fix enabling and disabling push rules (PR #498)
  • Fix bug where /register returned 500 when given unicode username (PR #513)

Matrix in Japan!

09.02.2016 00:00 — General Oddvar Lovaas

こんにちは

Matrix is on its way to Japan where Kegan is attending the TADHack-mini (Feb 13th and 14th) and WebRTC Conference (Feb 16th and 17th).

Kegan will help hackers with their projects during the TADHack, but first, he will give a talk on Matrix and how it can be used. We are again awarding a trossen robot to the best hack using Matrix, and we are as always curious to see what kind of cool and crazy ideas people will come up with!

A couple of days later, Kegan will be giving a talk during the WebRTC Conference: "The missing signalling layer for WebRTC".

Both of the talks will be live-translated, and there will also be a translator available during the events, so please come and say hello to Kegan-san! As always, we are also available in the Matrix HQ room, via a client like Vector or any other client!

Item_A_150x100

FOSDEM '16 retrospective

03.02.2016 00:00 — Events Oddvar Lovaas
stand

The Matrix team had a very successful trip to FOSDEM '16 last weekend. Many, many words were exchanged, and at times there was a queue of people just waiting to have a chat! We spoke to a whole lot of interested and interesting people, some of which had heard of Matrix already, and some who hadn't. The nice thing with the crowd at FOSDEM is that they very quickly "get" what we're trying to do with Matrix - and then start thinking aloud about how they might want to use it or extend it - which means we have many great conversations!

At the beginning of FOSDEM, I think we had around 2000 people "currently" in the Matrix HQ room - the next day, that number had increased by a hundred (and now it's even bigger due to a hackernews post where we mentioned Matrix)!

Our talk in the IoT dev room was very popular; unfortunately a lot of people that were queueing to get in never made it due to the limited space. However, the talk was recorded, and it's already been made available:

(Click here to download the video)

You can also look at the slides from the talk. Note that due to technical problems, the first half of the talk was not recorded.

We also did a talk in the Real Time dev room, which was equally popular. Hopefully the video from this talk will also be available shortly (you can keep an eye on the video repository for this dev room) - once it is, I will add it to this post.

For now, you can look at the slides from the talk.

Another interesting talk where Matrix ended up being represented, was Daniel Pocock's Improving Telepathy talk. I'll add the video from that too, once it's available.

Finally, thanks to everyone who came to say hello and have a chat - and to everyone who has since set up their own homeserver. Please do join the Matrix HQ room (using any of these clients) and let us know how your Matrix experience is going!

FOSDEM '16

18.01.2016 00:00 — Events Oddvar Lovaas
wide

Matrix will again be represented at FOSDEM (Free and Open Source Software Developers' European Meeting) in Brussels, Belgium, January 30th and 31st. This year, we have two talks scheduled: one in the IoT devroom on Saturday at noon, and one in the Real Time devroom at 3.10pm the same day.

We also have a stand on the 2nd floor of the K building - next to the Real Time lounge (potentially same spot as last year). If you're going to FOSDEM, please come and say hi to us at the stand or at the talks!

Our trip to FOSDEM '15 was very enjoyable; we met a lot of smart people and learnt about a lot of interesting technologies and projects. Hopefully we made a few people enthusiastic about Matrix as well!

As always, for any questions or comments, please come talk to us in the official Matrix HQ room - you can join via any of these clients (or write your own!).

Synapse 0.12 released!

04.01.2016 00:00 — Tech Matthew Hodgson

Happy 2016 everyone!

To greet the new year, we bring you all new Synapse 0.12. The focus here has been a wide range of polishing, bugfixes, performance improvements and feature tweaks. The biggest news are that the 'v2' sync APIs are now production ready; the search APIs now work much better; 3rd party ID invites now work; and we now mount the whole client-server API under the /_matrix/client/r0 URI prefix, as per the r0.0.0 release of the Client Server API from a few weeks ago. The r0 release unifies what were previously the somewhat confusing mix of 'v1' and 'v2' APIs as a single set of endpoints which play nice together.

We highly recommend all homeservers upgrading to v0.12.0 as soon as possible. Get it now from https://github.com/matrix-org/synapse/ or via our shiny new Debian packages at https://matrix.org/packages/debian/.

Full changelog follows:

Changes in synapse v0.12.0 (2016-01-04)

  • Expose /login under r0 (PR #459)

Changes in synapse v0.12.0-rc3 (2015-12-23)

  • Allow guest accounts access to /sync (PR #455)
  • Allow filters to include/exclude rooms at the room level rather than just from the components of the sync for each room. (PR #454)
  • Include urls for room avatars in the response to /publicRooms (PR #453)
  • Don't set a identicon as the avatar for a user when they register (PR #450)
  • Add a display_name to third-party invites (PR #449)
  • Send more information to the identity server for third-party invites so that it can send richer messages to the invitee (PR #446)
  • Cache the responses to /initialSync for 5 minutes. If a client retries a request to /initialSync before the a response was computed to the first request then the same response is used for both requests (PR #457)
  • Fix a bug where synapse would always request the signing keys of remote servers even when the key was cached locally (PR #452)
  • Fix 500 when pagination search results (PR #447)
  • Fix a bug where synapse was leaking raw email address in third-party invites (PR #448)

Changes in synapse v0.12.0-rc2 (2015-12-14)

  • Add caches for whether rooms have been forgotten by a user (PR #434)
  • Remove instructions to use --process-dependency-link since all of the dependencies of synapse are on PyPI (PR #436)
  • Parallelise the processing of /sync requests (PR #437)
  • Fix race updating presence in /events (PR #444)
  • Fix bug back-populating search results (PR #441)
  • Fix bug calculating state in /sync requests (PR #442)

Changes in synapse v0.12.0-rc1 (2015-12-10)

  • Host the client APIs released as r0 by https://matrix.org/docs/spec/r0.0.0/client_server.html on paths prefixed by/_matrix/client/r0. (PR #430, PR #415, PR #400)
  • Updates the client APIs to match r0 of the matrix specification.
    • All APIs return events in the new event format, old APIs also include the fields needed to parse the event using the old format for compatibility. (PR #402)
    • Search results are now given as a JSON array rather than a JSON object (PR #405)
    • Miscellaneous changes to search (PR #403, PR #406, PR #412)
    • Filter JSON objects may now be passed as query parameters to /sync (PR #431)
    • Fix implementation of /admin/whois (PR #418)
    • Only include the rooms that user has left in /sync if the client requests them in the filter (PR #423)
    • Don't push for m.room.message by default (PR #411)
    • Add API for setting per account user data (PR #392)
    • Allow users to forget rooms (PR #385)
  • Performance improvements and monitoring:
    • Add per-request counters for CPU time spent on the main python thread. (PR #421, PR #420)
    • Add per-request counters for time spent in the database (PR #429)
    • Make state updates in the C+S API idempotent (PR #416)
    • Only fire user_joined_room if the user has actually joined. (PR #410)
    • Reuse a single http client, rather than creating new ones (PR #413)
  • Fixed a bug upgrading from older versions of synapse on postgresql (PR #417)

The Matrix Holiday Special!

25.12.2015 00:00 — Events Matthew Hodgson

Hi all,

We've been pretty bad at updating the blog over the last few months with all the progress that's been happening with Matrix.  Whilst Matrix rooms like #matrix:matrix.org and #matrix-dev:matrix.org have been very active (and our twitter account too), in general we've ended up spending way too much time actually writing software and not enough time talking about it, at least here. When a blog goes quiet it normally means that either the authors have got bored, or they're too busy building cool stuff to keep it updated. I'm happy to say that option 2 is the case here!

As a result, there's a huge backlog of really cool stuff we should have talked about. Hopes of writing an Advent Calendar series of blog posts also went out the window as we set Christmas as an arbitrary deadline for loads of work on Synapse, the Matrix Spec and matrix-react-sdk.

So, to try to break the impasse, here's a slightly unorthodox whistle-stop tour of all the amazing blogposts we would have written if we'd had time. And perhaps some of them will actually expand into full write-ups when we have more time to spare in the future :)

End to End Encryption Update

One of the great promises of Matrix is to provide End-to-end encryption as part of the baseline standard (configurable per-room). In practice, our progress has been a little non-linear - we started writing an Axolotl ratchet implementation in C++14 (with a pure C API) named Olm back in February, and then finished it off and wired a basic 1:1 proof-of-concept implementation into matrix-react-sdk in June. We then announced Olm back at the wonderful Jardin Entropique conference in Rennes:

Jardin Entropique

You can read the full presentation that we gave at https://matrix.org/~matthew/2015-06-26 Matrix Jardin Entropique.pdf - and you can even play with a very basic test jig at https://matrix.org/~markjh/olm/javascript/demo.html which uses an emscripten compiled version of Olm in the browser to put the ratchet through its paces.

Things then stalled for a bit, but as of this month they're moving again, and if you're interested in the progress you can read all about it at:

The main stuff remaining is basically key management (in Synapse and the matrix spec), group conversation ratchets, and UX for wiring it properly into various Matrix clients. We expect to make progress on this over the next few months :)

Meanwhile, huge kudos to Tor who was crazy enough to add the basic 1:1 Olm ratchet to Weechat before we'd even finished writing our test jig!

Lean DUS

A few days after Jardin Entropique we made it to Lean DUS - a great tech meetup in Düsseldorf organised by Sipgate, who were kind enough to invite us to speak. This was a chance to give a full update on Matrix (as of July!) and talk some more about Olm and plans for end-to-end encryption. This one got recorded - and you can see it below. There's also an official page with full videos, slide deck and photos up at https://www.leandus.de/2015/08/weil-und-hodgson/.

Lean Dus #9 - End to end encryption for decentralised communication mit Matthew Hodgson from sipgate on Vimeo.

New Matrix Bridges

Somehow we've failed to blog about the amazing matrix-appservice-bridge Node framework which we've built as general purpose infrastructure for building Matrix Application Services which act as bridges between existing networks and comms solutions and Matrix. The architecture here looks something like this:

matrix-appservice-bridge

...and the goal is to end up with something like this:

bridge2

matrix-appservice-bridge is still in development, but there are a bunch of really cool bridges already using it - and a great howto that shows how you can use it to write a Matrix{'<->'}Slack bridge in under 100 lines of code.

  • the Matrix/Verto Bridge uses it to hook FreeSWITCH up to Matrix - currently used to provide multiway video and voice conferencing for Vector. It could be easily extended to do generic Matrix{'<->'}SIP or Matrix{'<->'}anything-that-FreeSWITCH-can-speak though.
  • a basic Matrix/Slack Bridge, which works well enough for hardcoding bridges between specific Matrix and Slack rooms.
  • matrix-appservice-respoke - a crazy experiment that bridges Asterisk to Matrix by implementing the Respoke API such that Asterisk can connect to Matrix using chan_respoke.
  • matrix-appservice-purple - another crazy experiment that hooks libpurple up to matrix-appservice-bridge such that *any* network that libpurple can talk to can be bridged into Matrix. So far we've experimented with Lync, Skype and Facebook (and AIM(!)) and it works - but it needs a lot more love to be usable other than as a toy.

Meanwhile, there's also:

As of right now our work on bridging has been on hiatus for a month or so, and we would love support from the community in advancing and extending the stuff we've built so far. Otherwise we'll get back to it ourselves in the new year.

Astricon 2015

We had a lot of fun in Orlando in October at Astricon 2015 - we put together matrix-appservice-respoke (see above) for our talk and Dangerous Demo in a desperate 24 hour hack and it even worked! The judges were kind enough to give us the "Swan Award" prize in the dangerous demo shoot-out for the glossiest demo :)

2015-10-15 Matrix Astricon swan award

The slides for our 'Bridging Asterisk to the Matrix Ecosystem' talk are downloadable here.

Pidgin!

We also implemented a basic libpurple plugin for Matrix - adding Matrix support to any app like Pidgin or Bitlbee that uses libpurple. (You could in theory even use it with matrix-appservice-purple to bridge from Matrix to Matrix, but that'd be silly :). It supports basic functionality and uses the new 'v2' APIs for syncing to Matrix. Adventurous libpurplers can go check it out and experiment with it from https://github.com/matrix-org/purple-matrix - feedback welcome.

OpenWebRTC Support on iOS

We went and hooked OpenWebRTC up to matrix-ios-sdk so that iOS Matrix apps can use the OpenWebRTC stack from Ericsson Research for VoIP and Video. Apparently we haven't written it up fully yet, but you can find the code at https://github.com/matrix-org/matrix-ios-openwebrtc-wrapper for those interested in using OWR with Matrix!

Debian Repository for Synapse

We built a 3rd party Debian package repository for Synapse... and then forgot to tell anyone about it, other than buried in the Synapse readme! Well, it exists, and intrepid debianers should go check it out at http://matrix.org/packages/debian/.

TADSummit

In November we attended TADSummit in Lisbon - a great event for folks hacking on telco applications and the telcos themselves. Apparently we failed to do a writeup, but we had a wonderful time: highlights included sitting down with Maarten Ectors from Canonical to wrap up Synapse as an Ubuntu Snappy app such that anyone in the Ubuntu Core ecosystem can trivially run a Matrix homeserver, and demoing it as part of the Dangerous Demos track there. We also gave a 'Matrix: One Year In' talk to summarise what we got up to in 2015.

Matthew & Maarten 2015-11-17 Matrix TADSummit

WebRTC Paris

Whilst on the subject of conferences that we forgot to write up - we just got back from WebRTC Paris, where we demoed the latest & greatest Matrix clients and bridges, hung out with the OpenWebRTC guys and gave another ecosystem update. You can see the slides at https://matrix.org/~daniel/Matrix- One-year Status Report.pdf.

WebRTC Paris

New Clients and App Services

There have been a flurry of really interesting new clients and other projects which certainly deserve whole blog posts of their own!

There's Tensor from David A Roberts - a multiplatform native client written in QML that heavily leverages the matrix-js-sdk:

tensor

There's matrix.el from Ryan Rix - a native Matrix client for Emacs! You can read all about the whys and wherefores here.

mclient

There's also loads of cool stuff that Ryan's been doing with Matrix on his blog - including Polynomial - a decentralised webring built on Matrix (yes, webrings were and are cool, ok!?!), and his Matrix-powered Body Computing System. Also, some philosophical posts on the benefits of Matrix which give us some hope that we're on the right track!

Then there's Power Take-Off from Torrie Fischer - an early lets-IRC-clients-connect-to-Matrix project in Rust...

...and there's Morpheus from Christine Dodrill (Xena) - a Matrix client and bot framework for Haskell; part of a more over-arching IRC{'<->'}Matrix unification project. Xena also wrote a great call to arms for Matrix :)

Very recently there's the Ruma project from Jimmy Cuadra - an ambitious mission to build Matrix components (up to and including a homeserver) in Rust!

Other stuff includes a Hubot adaptor from davidar, Bender: an Elixir client and bot library from Dylan Griffith, Jon Frederickson's matrix-xmpp-bridge, rzr's guide to installing synapse on a minnowmax, and I'm sure many others we don't know about or have missed!

Finally, Tor has done an amazing job on weechat-matrix-protocol-script in implementing features like V2 Sync and E2E crypto faster than we've managed to add them in the official client SDKs!

Release Zero of the Matrix Specification

We have made some major improvements to the spec over the last few months: adding in feature profiles and spec modules to better structure the document, and most recently splitting it up explicitly into separate Client-Server, Server-Server and Application-Server APIs, each with a well-defined single global 'release' number for versioning. We started this with a 'r0.0.0' release of the Client-Server API, which consolidates the horrible mess of 'v1' and 'v2' APIs we had previously flying around into a single well-defined version of the spec. Meanwhile the spec is now entirely consolidated into a set of JSON schema and Swagger 2 API descriptors, with a bunch of ReStructured Text for the verbiage - you can find it all at https://github.com/matrix-org/matrix-doc.

The r0.0.0 changelog is exciting stuff - you can see it in its entirety at http://matrix.org/docs/spec/r0.0.0/client_server.html#changelog. Synapse itself will support the full r0 API set in 0.12, which will be released any day now.

We've also switched the Swagger-based API viewer over to Swagger 2.0: http://matrix.org/docs/api. We also rejigged the Matrix documentation website entirely, generating it via Jekyll and adding in a new guides section.

Also, we should have mentioned the existence of Speculator - a golang helper app (source here) which, as the name suggests, renders out copies of the spec as HTML from different branches and pull requests for ease of previewing.

Dendron

Over the last few months we've also started an entirely new project, codenamed Dendron. Dendron is the project to evolve Synapse from the current single-threaded Python/Twisted monolithic homeserver into something with a lot more type-safety, horizontal scalability and high availability. We've mainly been experimenting with different ways of doing this, but the current plan is to split Synapse itself up into multiple services which can each scale independently, and then rewrite some/all of them in languages with better type safety and/or performance or profiling tools.

Some folks may remember a survey that we posted a few months ago asking for the community's thoughts on what languages they'd like their ideal homeserver to be written in, from the perspectives of someone running it as well as hacking on it. Whilst we haven't (at all) based our decisions for Dendron purely on the survey, it was still quite an interesting exercise. And here are the results (maximum 'score' is 5, not 10):

language survey 1 language survey 2

The basic feedback was that from the existing community: folks dislike running Java or Node servers; are okayish with Python, but would prefer native or near-native code (be that C, Rust or Go). Meanwhile, for contributing code, there's slightly more interest in the (relatively) new shinies of Go and Rust. And of course, everyone wanted to plug their own special snowflake language in the 'Others' section, which was mainly a mix of Erlang, Elixir, Haskell, Lisp and Perl :)

This reinforced the choices we were looking at anyway - either Rust (for its safety), or Go (for its simplicity, python-likeness, and concurrency). (We'd also consider Java, but have to concede that the FOSS community doesn't like running it.)

So we looked at the dependencies that Synapse currently has, and the Rust equivalents, and concluded that the Rust ecosystem unfortunately isn't quite mature enough yet to reliably handle the rather large set of complicated deps that we need in a homeserver. Also, nobody on the core team is really a Rust guru yet. Meanwhile, we have at least one ex-Google Go expert in the core team, and in practice it has the edge in terms of maturity. So, right now, we're looking at switching chunks of Dendron to Go where it makes sense. (This is subject to change though depending on how we get on). You should expect to hear a lot more about Dendron in 2016 :)

matrix-react-sdk, Vector, and latest Matrix features

Last but not least: huge amounts of our time over the last year has gone into building matrix-react-sdk - a full set of glossy Web UI components for building super-high quality glossy apps based on Matrix, built on the matrix-js-sdk. This is basically a reaction against the original matrix-angular-sdk and Matrix Console app that we launched Matrix with back in 2014 - which had minimal attention to UI/UX and suffered from major performance problems; it was built purely as the fastest possible way we knew to prototype and demo Matrix in the first place. matrix-react-sdk however has been built for both performance and quality of UI/UX, as well as obviously using all the latest and greatest Matrix APIs. (In fact, the transition from matrix-angular-sdk to matrix-react-sdk is pretty similar to the Synapse to Dendron transition on the horizon - although the latter should be more incremental and less 'rewrite the world').

Meanwhile, as part of our commercial work at our day job (i.e. not as Matrix.org) we've been helping on a glossy FOSS app called Vector which is layered on top of matrix-react-sdk as a thin 'skin' layer of CSS and a few custom components. The intention for Vector is to ensure that Matrix has a flagship glossy client: which it clearly needs, in order to gain credibility and drive uptake of the Matrix standard, and also ensure that the Matrix standard actually does indeed fit the needs for a state-of-the-art collaboration tool.

matrix-react-sdk (and thus Vector) is still in incredibly heavy development - we're going to start a formal beta fairly soon, but as of right now it's still sprouting features and refinements on a daily basis. Meanwhile matrix-react-sdk's APIs are not remotely frozen (we entirely refactored it as recently as a few weeks ago), so not yet ready for use as a general purpose building block.

Some of the stuff going into react-sdk is incredibly cool - recent Matrix stuff that it shows off includes:

  • Animated read receipts. If you haven't seen these, you haven't lived. They are a relatively new addition to the Matrix spec.
  • Full server-side search. We now have full-text search in the Matrix spec, and implemented on synapse both on sqlite and postgres - and now in Vector too. Having good search over all of your chat history makes Matrix *so* much more usable.
  • Video conferencing. We have full multi-way conferencing in Vector via matrix-appservice-verto and FreeSWITCH. The intention is to add this to the core Matrix spec (having first made it a bit more generic) - see the draft spec for details.
  • 3rd party invites. You can now invite users into Matrix by email address as well as matrix ID, and it works. Vector implements this.
  • Room tagging. You can now tag rooms as favourites, low priority, or with arbitrary namespaced metadata. Vector implements this through a swanky drag & drop UI.
  • "V2" Sync API. Now part of the 'r0.0.0' spec, this lets Matrix support much smarter incremental and partial synchronisation patterns. Vector now implements this, meaning that browser windows sync much faster after being offline for a bit, and no longer hammer the user with stale desktop notifications.
  • Accessing 'historical' rooms. Matrix now lets you keep track of rooms you've left, so you can view and search the conversation logs even after you've left the conversation. Vector now implements this (as of Monday!)
  • Tab-complete that doesn't suck. This is a purely client-side feature which landed on Thursday!
  • Roll-overable animated GIFs. 'nuff said.
  • Markdown support. yay!
  • Synchronised read and notification history. This hasn't landed yet (in vector or synapse or even the spec), but finally provides a way to keep read and notification state in sync in realtime across all your clients and a meaningful favicon 'badge' telling you how many notifications you missed!
  • Guest access. This hasn't landed in Vector yet, but it's in the spec and Synapse. It will let folks use Matrix without having to create an account (at least for rooms which support 'guest access' from the public).

If you haven't given Vector a spin, it's well worth heading over to https://vector.im and taking a look.

There's also an Electron desktop version of Vector in progress, contributed by Steven Hammerton at https://github.com/stevenhammerton/vector-desktop (although it's currently stuck on an old release).

Epilogue

Okay, this has got a lot longer than it was meant to be - but hopefully makes up a bit for the lack of comms over the last few months whilst we've been drowning in work on Synapse, Vector, the Spec, Dendron, and everything else mentioned above.

2015 has been an epic year as we've taken Matrix from a very early beta to the advanced stage that it's at now. Obviously there's still a lot of stuff to do though. Right now we expect the focus in 2016 to be:

  • Vector - making sure the Matrix protocol has a flagship glossy FOSS client that normal (non-geek) users can use.
  • Dendron - making Synapse more reliable, scalable and maintainable.
  • Bridging - wiring as much of the rest of the world into Matrix as accurately and efficiently as possible.
  • Federation Spec - finishing and releasing the Server-Server API.
  • End-to-end crypto - finishing it off.

...and obviously continuing to refine and extend the core of Matrix itself with features like threading, editable messages, and possibly even distributed accounts.

There are very fun and exciting times ahead. We'd just like to say a profound thank you to everyone who's supported Matrix this far and helped make this mission possible - whether it's by running clients/servers/services, or writing your own, or filing bugs and feedback on our code or the spec, or telling folks about the project, or paying us to work on it(!), or just by reading this blog post. Hopefully 2016 will be the year where online communication starts to open up and interoperate once again, rather than becoming ever more fragmented and closed.

Thanks for reading - and Merry Christmas, for those who celebrate :)

  • Matthew, Amandine & the Matrix Team.