Tech

I was at IndieWebCamp Edinburgh last week and during the hack day I created a Matrix Application Service (AS) which could act as an IndieWeb Micropub client. Any Matrix message sent to the AS (@micropub:domain) would be converted to a request to a Micropub endpoint.

This required the AS to support IndieAuth - which it does by sending !indieauth http://yourdomain.dom to @micropub:domain which then returns an OAuth2 URL to login via. Currently, the AS just supports a 1:1 mapping from m.text to h:entry but in the future, it can be expanded to include categories and potentially the reverse mapping (where Micropub clients can act as Matrix users!).

Overall, it was a great weekend and I look forward to adding more support for IndieWeb protocols in the future.

Hi all,

We seem to have done that thing again where we were too busy writing new stuff (E2E crypto, all-new React-based Web SDK, long-awaited iOS Console update) to remember that we'd released a new version of Synapse - sorry!

Synapse 0.9.3 was released on July 1st (actually on June 23rd as 0.9.3-rc1, but was released without changes). It's a fairly minor release but does provide some performance improvements and bug fixes - see below for details. Get it from http://github.com/matrix-org/synapse if you haven't already.

Changes in synapse v0.9.3 (2015-07-01) ======================================

General:

• Fix a memory leak in the notifier. (SYN-412)
• Improve performance of room initial sync. (SYN-418)
• General improvements to logging.
• Remove access_token query params from INFO level logging.

Configuration:

• Add support for specifying and configuring multiple listeners. (SYN-389)

Application services:

• Fix bug where synapse failed to send user queries to application services.

We've been running an IRC gateway from Matrix to Freenode almost since day 1 - originally it was a simple perl bot written by tm604 using Net::Async::Matrix; later it was rewritten by LeoNerd to be a bit more modular, and nowadays it's a full IRC/Matrix Application Service written in Node.js by Kegan.

Up until the end of last week the bridge was limited to synchronising a fixed number of channels between Freenode and Matrix (#matrix, #matrix-dev, #openwebrtc, #vuc and #hypothes.is, to be precise), as well as any PMs. But as of Friday, with huge thanks to the admins over at Freenode, we can now bridge any channel in Freenode through to Matrix.

Doing this is trivial - you just /join #freenode_#channelname:matrix.org from a typical Matrix client - e.g. /join #freenode_#Node.js:matrix.org will give you access to all of #Node.js via Matrix, effectively using Matrix as a great big distributed IRC bouncer in the sky ;)

There are a few limitations in the current setup:

• We only incrementally synchronise the membership lists when folks speak in either IRC or Matrix. This is to avoid flooding either IRC or Matrix with lurkers when the bridge initially joins a channel. We have some plans to improve this in future - see BOTS-53 for some of the sordid details.
• We don't synchronise joins/parts currently to avoid flooding Matrix with lots of IRC join/part spam. Again, this will improve in future.
• You can't join +k channels.
• Kicks/bans/invites and other ACLs currently don't propagate between IRC & Matrix. (I.e. a Matrix user can be kicked from IRC, but it'll continue to sit in Matrix unless also kicked for there). +i chanmode supported however.
• Bridged public IRC rooms are not yet advertised in the public room list on matrix.org.
• It should be possible to change the nickname of your IRC user by messaging !nick irc.freenode.net MrFlibble to @appservice-irc:matrix.org. (This isn't actually turned on right now, but should be fixed shortly. See https://github.com/matrix-org/matrix-appservice-irc/blob/master/HOWTO.md#changing-nicks for details.

We'll keep chipping away at making the IRC<->Matrix mapping perfect, but in its current state it's still really usable. It's obviously beta still, but please give it a go and let us know in #matrix:matrix.org how you get on with it!

Matrix Console 0.4.0 for Android has been pushed to the Google Play store and shall be available shortly.

This release contains a whole lot of bugfixes, new features, and nicer UI and UX - see the changes file for details, but in summary:

• Lots of UI and UX improvements (inc. long click on images)
• Added GCM support (can be enabled/disabled)
• Added Google analytics support
• Added badges management
• Added "orientation" management for images
• Now offering image resizing before upload
• Lots of bugfixes!

Hi all,

If things seem a bit quiet here at the moment it's because most of the Matrix team is in the 'build' phase across the board on a bunch of major new projects - including the long-awaited reusable UI component library for the Web, end-to-end encryption support, stable VoIP on mobile and a few other surprises. The first wave of all this new work landed today in the form of a new major version release (0.1.0) of matrix-js-sdk: our javascript SDK library.

The history of JS client support on Matrix is that in the original rush to get a PoC webclient out the door back in September, we jumped straight to talking to the Matrix HTTP API from Angular controllers in 'syweb'. This then got split into an Angular SDK (matrix-angular-sdk) containing a bunch of services for handling the Matrix client state with the actual webapp sitting alongside as an example of the SDK. This then in turn got split into a basic matrix-js-sdk (wrapping the Matrix client-server HTTP API) and matrix-angular-sdk (handling the clientside state and exposing it as Angular services) - and this is how the current demo Angular webapp works at http://matrix.org/beta.

However, this poses a major problem if you want a richer client SDK but don't want to use Angular - e.g. if you prefer another framework (or no framework at all), or are using the SDK in an Application Service or similar. So, this has prompted a major upgrade to the matrix-js-sdk in order to extend it to act both as a simple wrapper of the Matrix Client-Server HTTP API, but also expose a full object and event model to describe the state of a Matrix client together with all the higher level functions to drive it. To quote from the README:

This SDK provides a full object model around the Matrix Client-Server API and emits events for incoming data and state changes. Aside from wrapping the HTTP API, it:

• Handles syncing (via /initialSync and /events)
• Handles the generation of "friendly" room and member names.
• Handles historical RoomMember information (e.g. display names).
• Manages room member state across multiple events (e.g. it handles typing, power levels and membership changes).
• Exposes high-level objects like Rooms, RoomState, RoomMembers and Users which can be listened to for things like name changes, new messages, membership changes, presence changes, and more.

Later versions of the SDK will:

• Automatically retry requests to send messages due to network errors.
• Automatically retry requests to send messages due to rate limiting errors.
• Mark events' sent status (e.g. 'not sent').
• Handle "local echo" of messages sent.
• Handle queueing of messages.
• Handle pagination.
• Expose a RoomSummary which would be suitable for a recents page.
• Provide different pluggable storage layers (e.g. local storage, database-backed)

It also is designed to provide an API that abstracts both version 1 of the client-server API and the upcoming version 2 (which fixes various thinkos we've discovered along the way in v1).

This is incredibly exciting stuff as it now gives a really robust yet lightweight API for client-side and application-service development in JavaScript. Predictably enough we're using it to develop our new reusable web UI components, and I suspect we'll see new Node-based application services making use of in the near future. Meanwhile, it's more than topical as it should run straight on top of the Tessel JavaScript hardware modules we're giving away this weekend as our prize for the best hack built on Matrix at TADHack - letting the little Tessel ARM system-on-a-chip speak a very rich and native Matrix dialect!

The new matrix-js-sdk comes with comprehensive jsdoc detailing the new API which you can find at http://matrix-org.github.io/matrix-js-sdk/global.html. There are also some basic examples available, including a new simple node-powered command-line client designed for testing the SDK. Finally, if you're interested in reading the design process by which we've converged on the current API, all the details may be found in JIRA on SYJS-5.

So: if you've been itching to get your hands on a much more powerful pure-JS Matrix SDK, now's your chance! It's very early days for the new API so we expect to see a few incompatible changes on the horizon, but we encourage you to have a play and tell us what you think!.

To find out more, go check out the project at http://github.com/matrix-org/matrix-js-sdk, or if you're feeling daring just grab the library with npm install matrix-js-sdk and get hacking!

In the excitement of KamailioWorld last week we completely forgot to mention that we released Synapse 0.9.1.

This is a pretty important performance and stability update of all the new stuff that landed in 0.9.0, as well as landing a major new feature in the form of 'backfill': we finally have the ability to correctly sync in historical conversation history over federation for newly federated rooms. In other words, if you join a remote room you should be able to navigate its history as intended.

We highly recommend upgrading to 0.9.1 for all the performance improvements and backfill support - get it now from http://github.com/matrix-org/synapse.

Changes in synapse v0.9.1 (2015-05-26) ======================================

General:

• Add support for backfilling when a client paginates. This allows servers to request history for a room from remote servers when a client tries to paginate history the server does not have - SYN-36
• Fix bug where you couldn't disable non-default pushrules - SYN-378
• Fix register_new_user script - SYN-359
• Improve performance of fetching events from the database, this improves both initialSync and sending of events.
• Improve performance of event streams, allowing synapse to handle more simultaneous connected clients.

Federation:

• Fix bug with existing backfill implementation where it returned the wrong selection of events in some circumstances.
• Improve performance of joining remote rooms.

Configuration:

• Add support for changing the bind host of the metrics listener via the metrics_bind_host option.

As a follow-on to the previous post, Apple approved version 0.4.0 of the iOS Matrix Console app over the weekend - grab it now from the app store!. Whilst cosmetically it looks almost identical to 0.3.x, the architecture is completely different given the app is now built using MatrixKit - and stability has improved considerably as a result. Please install and let us know how you get on!

This post has now been edited into a guide - you can find the source in github, and the formatted guide on the matrix.org website!

---

Like a lot of open source projects, we use IRC a lot. Naturally, we also use Matrix to communicate with each other. For some time now we've had an IRC bot sitting on specific channels to "bridge" together IRC and Matrix. This bot simply sent IRC messages when it received Matrix messages and vice versa. As we started to rely on it more and more though, we realised that there were things that were impossible for simple client-side bots to do by themselves. This spurred the development of Application Services which I introduced in my previous post. In this blog post, I want to outline some of the features and techniques of the IRC application service which we've been working on over the past few weeks.

Features:

• Specific channel-to-matrix room bridging : This is what the original IRC bot did. You can specify specific channels and specific room IDs, and messages will be bridged.
• Dynamic channel-to-matrix room bridging : This allows Matrix users to join any channel on an IRC network, rather than being forced to use one of the specific channels configured.
• Two-way PM support : IRC users can PM the virtual "M-" users and private Matrix rooms will be created. Likewise, Matrix users can invite the virtual "@irc_Nick:domain" user IDs to a room and a PM to the IRC nick will be made.
• IRC nick changing support : Matrix users are no longer forced to use "M-" nicks and can change them by sending "!nick" messages directly to the bridge.
• Ident support : This allows usernames to be authenticated for virtual IRC clients, which means IRC bans can be targeted at the Matrix user rather than the entire application service.

The use of the Application Services API means:

• The bot can reserve user IDs. This prevents humans from registering for @irc_... user IDs which would then clash with the operation of the bot.
• The bot can reserve room aliases. This prevents humans from register for #irc_... aliases which would then clash with the operation of the bot.
• The bot can trivially manage hundreds of users. Events are pushed to the application service directly. If you tried to do this as a client-side bot, you would need one event stream connection per virtual user.
• The bot can lazily create rooms on demand. This means Matrix users can join non-existent room aliases and have the application service quickly track an IRC channel and create a room with that alias, allowing the join request to succeed.

Implementation details:

• Written in Node.js, designed to be run using forever.
• Built on the generic matrix-appservice-node framework.
• Supports sending metrics in statsd format.
• Uses matrix-appservice-node to provide a standardised interface when writing application services, rather than an explicit web framework (though under the hood matrix-appservice-node is using Express).

At present, the IRC application service is in beta, and is being run on #matrix and #matrix-dev. If you want to give it a go, check it out on Github - it is not currently released on npm. N.B. it requires features from the develop branch of synapse; either run your own synapse off the develop branch or wait a few days for us to release Synapse 0.9.0.

Needless to say, we look forward to this being the first of many full network<->network bridges into Matrix - come chat on #matrix:matrix.org if you'd like to write or run your own! Next up is Lync and XMPP...

Hi folks,

We released a new version (0.2.3) of the Android Matrix Console application today - this is a decent upgrade over 0.2.2. Changelog below.

If you're an Android user, please install the app from the Play Store and give it a go and give us some feedback on #android:matrix.org or Google Play.

thanks!

Changes in Matrix Android SDK in 0.2.3 (2015-03-10) ===================================================

---

SDK

---

Matrix Console

Improvements:

• Avoid refreshing the home page when it is not displayed.
• Display a piechart while uploading a media.
• Refresh the display when some messages are automatically resent (after retrieving a data network connection for example).
• Update the user rename message to be compliant with the web client.
• Use the local media files instead of downloading them when they are acknowledged (messages sending).
• Create a medias management class.
• Display the offline status in the members list.
• Avoid creating new homeActivity instance when joining a room from member details sheet.
• The public rooms list are now saved in the bundle state : it should avoid having a spinner when rotated the device.
• The animated GIFs are now supported.

Features:

• Add the rate limits error management. The server could request to delay the messages sending because they were too many messages sent in a short time (to avoid spam).
• Can take a photo to send it.
• A chat room page is automatically paginated to fill. It used to get only the ten latest messages : it displayed half filled page on tablet.
• Add the sending failure reason in the message details (long tap on a message, “Message details”).
• The user is not anymore notified it the push rules are not fulfilled.
• Add some room settings (Display all events, hide unsupported events, sort members by last seen time, display left members, display public rooms in the home page).
• Add various accessibility tweaks.

Bug fixes:

• The media downloads/uploads were sometimes stuck.
• The private room creation was broken.
• SYAND-33 : number of unread messages disappears when entering another room.
• The RoomActivity creation used to crash when it was cancelled because the Room id param was not provided.
• The client used to crash when the home server was invalid but started with http.
• The account creation used to fail if the home server had a trailing slash.
• SYAND-44 In progress text entry could be saved across crashes.
• SYAND-38 Inline image viewer in Android app.

We just pushed out a major new release of Synapse 0.7.0, the python reference server for Matrix, and a minor maintenance update for matrix-angular-sdk 0.6.2, the reference web client implementation.

The emphasis here has been on federation performance and stability - with all the recent interest from FOSDEM and LWN and HackerNews we've been getting a lot of traffic on matrix.org and new servers federating up, so we've been busy profiling and fixing performance on some of the hot paths by adding in-memory caches and similar.

This release is highly recommended if you are running a federated server, as it fixes many denial-of-service failure modes in the federation implementation, as well as many performance improvements

Behind the scenes, there's also lots of work going on for v2 of the Client-Server API (which lets you filter the events your client subscribes to in a room, and combines the 'initial sync' and 'eventstream' APIs into a single simplified '/sync' API) - the alpha of this has landed in 0.7.0, but we don't recommend trying to use it yet. Meanwhile the Application Service API is effectively finished, but hasn't landed on the master branch yet. Synapse also finally has initial support for push notifications for the iOS client now via the sygnal APNS gateway, although we need to actually document how to set this up to be usable in general.

Finally, we have switched to recommending that synapse is installed in a virtualenv rather than into ~/.local, due to various problems with how setup.py's dependency management works. Instructions on setting up a virtualenv can be found in the README.

On the clientside: we've improved performance, enabled identicons for unknown users, and now fully support OpenWebRTC for VoIP calling from Safari, Bowser, and other OpenWebRTC-capable browsers!

Thanks to everyone running servers - please upgrade, tell your friends, and help us grow Matrix!

Changes in synapse v0.7.0 (2015-02-12)
======================================

* Add initial implementation of the query auth federation API, allowing
  servers to agree on whether an event should be allowed or rejected.
* Persist events we have rejected from federation, fixing the bug where
  servers would keep requesting the same events.
* Various federation performance improvements, including:
  - Add in memory caches on queries such as:
     * Computing the state of a room at a point in time, used for
       authorization on federation requests.
     * Fetching events from the database.
     * User's room membership, used for authorizing presence updates.
  - Upgraded JSON library to improve parsing and serialisation speeds.
* Add default avatars to new user accounts using pydenticon library.
* Correctly time out federation requests.
* Retry federation requests against different servers.
* Add support for push notifications and push rules.
* Add alpha versions of proposed new CSv2 APIs, including ``/sync`` API.

Changes in Matrix Angular SDK 0.6.2 (2015-02-12)
================================================
Bug fixes:
 - Fixed a bug which caused OpenWebRTC to occasionally fail.
 - Fixed a bug which caused multiple room initial syncs to occur in rapid
   succession.

Features:
 - Display a "Joining Room" dialog when joining rooms.
 - Display identicons for users with no avatar.
 - Display m.notice events with full formatting.
 - Add push notification rules to settings.

Improvements:
 - Modified the red/blue notification colours to be more noticeable on a wider
   range of displays.
 - Highlight room invitations in blue.
 - Calculate room names for rooms of 3+ members.
 - Improved page load performance.