Massive update from ma1uta about his Jeon project! This update brings Jeon into line with the most recent updates to the Client-Server, Application Service, Push and Identity APIs.
In ma1uta’s words:
Jeon is a set of the java interfaces and classes which describes the Matrix API.
client-api: r0.4.0-1 corresponds to the r0.4.0 C2S API.
application-api: r0.1.0-1 corresponse to the r0.1.0 AS API,
push-api: r0.1.0-1 corresponds to the r0.1.0 PUSH API,
identity-api: r0.1.0-1 corresponds to the r0.1.0 IS API.
All artefacts available from the Maven Central Repository.
Major changes:
Full support for the corresponding Matrix api.
Changed version for displaying the Matrix api version.
Added support to the asynchronous responses.
Also updated the swagger schemas generated from the code:
And the first hotfix: application-api r0.1.0-2 with fixed url (/transactions has been changes to the _matrix/app/v1/transactions). And this release will break all AS because synapse sends transactions to the old url.
Join #jeon:matrix.org to discuss the progress of this product more.
VoIP signalling support has landed in libQMatrixClient
After some pretty long time of being in a PR/fork, VoIP signalling support has landed in libQMatrixClient! Many thanks to mariogrip (the founder of UBports) for the initial code and to delijati (a developer behind uMatriks) for getting it to work with the most recent library.
The actual VoIP stack does not come included, client developers have to take whatever WebRTC implementation is available for their platform and glue the pieces together. However, as the example of uMatriks shows, this now becomes relatively easy if your platform is on good terms with WebRTC (like UBports). I look forward to further work with UBports community on keeping this platform a first-class Matrix citizen.
Matrix Corporal has received some updates over the past few weeks since its initial release: a couple of additional HTTP APIs for retrieving/destroying user access tokens; more consistency with the Matrix Client-Server specification when it comes to error responses; faster reconciliation for user accounts that are joined to many/large rooms.
matrix-docker-ansible-deploy
Another project from @slavi:devture.com, for those who prefer their DevOps ansible-flavoured:
Seaglass end-to-end encryption support is now complete, including device verification and blacklisting, key sharing requests, key import and export (which should be compatible with Riot) and re-requesting keys
This is really exciting news for macOS matrix users!
I’m also working on auto-updating Seaglass with Aaron Raimist‘s help in addition to finishing E2E support :-)
:-) is right!
ma1uta jmsdk
ma1uta must have been in a work-on-Matrix mode this week, because he has also updated jmsdk:
I have released a new version of the java client (https://github.com/ma1uta/jmsdk/tree/master/client-sdk). The new client works is asynchronous mode, each method doesn’t block the thread and return the CompletableFuture (promise in java). Then you can block thread to get the response or build a asynchronous promises chain.
Finally:
ma1uta is also looking forward to the release of Java 11:
with the Curve25519 key agreement (http://openjdk.java.net/jeps/324) and will try make a pure java implementation of the olm/megolm. Just for fun. :)
synapse-purge
Maze, seeing that his synapse database was already at several gigabytes, decided to produce a tool to shrink it:
The synapse-purge tool allows homeserver admins to free disk space by purging old room events from the synapse database. It is an alternative for synpurge which currently does not work correctly.
Configuration is minimalistic at the moment. Meaning it purges all remote rooms on the server with a globally configured preservation period.
synapse
andrewsh: 0.33.4 uploaded to Debian’s stretch-backports, pending approval.
0.33.5rc1 is now available, with the big news being the inclusion of support for Python 3.5 and 3.6! Hawkowl’s Py3 has merged for monoliths and is working pretty well, looking like 2-3x RAM improvement. Please help us test!
Erik’s state compressor tool is pretty much finished, we’ve been starting to run it on things and it reduces disk usage for the state group table by at least 10x.
The only catch is that it’s quite DB heavy whilst it runs, so we haven’t run it on Matrix.org yet.
And so it was – late breaking news that maubot has been used to develop a Dictionary-definition-bot! Not available for public use yet but it proves that the project is useful!
Riot Web
Lazy Loading remains the focus, we’re getting closer with more bugs solved this week! To enable Lazy Loading room members and get speed and memory benefits in Riot, use the develop branch and enable “Lazy Loading” under “Labs” in the settings.
Lots of final bug hunting for lazy loading – this is taking longer than you might expect because we’re doing end-to-end CI everywhere.
Lots of work on E2E, Dave has been working on:
UI for e2e key backup that’s waiting for some lower level bits
and hopefully our e2e core code is moving from asm.js to webassembly making it, by current estimations, significantly faster.
Redesign work continuing as well – Bruno has been working on it this week, Jouni the designer will visit next week to continue the process.
Nad has joined us to help with design bandwidth and is working on the onboarding flows for the redesign as well as fixing all the UX issues in Communities!
Riot Mobile + Mobile SDKs
Lots of work on Lazy Loading – to be released along with Riot Web.
Bridges
Half-Shot is joining us to work part-time on bridges going forwards – this is great news, especially for his connection-based IRC bridging antics as well as catching up on the PR and maintenance backlog for the IRC bridge and Slack bridge.
Modular
Modular (Hosted Homeservers) has first customers; if you want to give it a go please let us know!
Finally
Thanks for reading, take a look at Matrix Live below!
Update (this got lost in the original post; sorry Travis!): Dimension received a security update – if you run your own Dimension instance it is strongly recommended you update right away. Telegram bridge support in Dimension is underway, with more updates expected for next week in Matrix.
Clients
Fluffychat
It’s been some months since we checked in with FluffyChat. If you’re a Ubuntu Touch user, or have a device running it, you should see the progress that has been made recently on this Matrix client. Collected changelog 0.5.0 to latest (0.5.4):
Search chats
Chat avatars
Search users in chats
Security & Privacy settings:
Disable typing notifications
Auto-accept invitations
New message status:
Sending: Activity indicator
Sent: Little cloud
Received: Tick
Seen by someone: Usericon
Display stickers
Minor UI improvements
FluffyChat now automatically opens the link to the matrix.org consens
Updated translations
Seaglass
Neil has been keeping up the pace with Seaglass development:
Seaglass has had a substantial rewrite to the room cache to help improve reliability and reduce crashes, better thumbnail behaviour on inline images, various tiny visual tweaks, in-window blending, support for encryption key sharing requests for E2E rooms.
Rendering performance has been massively increased (if you ignore the occasional bug). Resizing the window shouldn’t be so slow anymore and a lot of avatar image operations are no longer repeated unnecessarily
Other than that this week has mostly featured lots and lots of bug fixes, hopefully lots of crashes fixed.
Screenshot below shows the new E2E UI:
Quaternion
When not escaping typhoons, kitsune has found some time to continue work on Quaternion:
Quaternion‘s master branch is alive again – it’s been prone to crashes in the last two weeks, now it shouldn’t. Feel free to try the new room list organised by tag!
This is a fork of Riot Android done by hrjet, f-droid release done by me. It’s removing mostly jitsi group call functionality and some other smaller stuff. In doing so it manages to require far less permissions and is also only 12 MB in size instead of riots 20 MB.
Matrique gained alpha support for multiple accounts
This is thanks to leaning on libqmatrixclient’s native multiple account support!
Riot Web 0.16.4 released
This is pretty much a maintenance release – fixing the DM avatar regression that crept in with 0.16.3, adding better support for warning users when their client hasn’t yet synced with the server, and the final bits of work needed before we can turn on membership Lazy Loading in the upcoming Riot 0.17.
Full changelogs as always are split over the three projects which make up Riot/Web:
As you may know, Matrique, led by Black Hat, and Quaternion, led by kitsune, are both projects build using libQMatrixClient, a Qt5 library from kitsune designed for writing Matrix clients. While kitsune has been working on the library for some time, Black Hat has also now
started making contributions:
libQMatrixClient now has a pkg-config file to further ease clients building on Linux systems, as well as more convenient API to track read markers if all users, not just of the local one.
This release contains support for lazy loading room members, and also some breaking changes relating to startClient().
Support for lazy loading members. This should improve performance for users who joined big rooms a lot. Pass to lazyLoadMembers = true option when calling startClient.
MatrixClient::startClient now returns a Promise. No method should be called on the client before that promise resolves. Before this method didn’t return anything.
A new CATCHUP sync state, emitted by MatrixClient#"sync" and returned by MatrixClient::getSyncState(), when doing initial sync after the ERROR state. See MatrixClient documentation for details.
RoomState::maySendEvent('m.room.message', userId) & RoomState::maySendMessage(userId) do not check the membership of the user anymore, only the power level. To check if the syncing user is allowed to write in a room, use Room::maySendMessage() as RoomState is not always aware of the syncing user’s membership anymore, in case lazy loading of members is enabled.
Synapse
Synapse 0.33.4 was released, with a whole host of bug fixes, some enhancements to resource usage management and a bunch of internal changes in readiness for room member state lazy loading and our ongoing port to Python 3.
Meanwhile, Python 3 support for monolithic (non-worker) Synapses has finally landed on the develop branch, thanks to massive work from hawkowl and notafile – if you want to help us test and flush out any remaining byte/utf8 style errors, please create a virtualenv for python 3.6 or 3.5 (twisted doesn’t support 3.7 yet) and point the develop branch of Synapse at it, tail the logs for ERRORs and report them via Github if/when you see them. In practice it seems pretty stable though, and noticeably reduces RAM and speeds things up thanks to improved GC and general performance work in Python.
We’ve also discovered that jemalloc works *very* well at improving RAM usage on Python 2 under Linux (we haven’t tried it on Python 3 yet) by providing a more fragmentation-resistent malloc implementation; if you are having problems with your Synapse RAM spiking up we recommend giving it a go. All of the Matrix.org server is using it now.
Also, lots of ops work this week; Erik has prototyped a new storage strategy for state groups which shrinks storage requirements by 10x, we’ll be applying this shortly to Matrix.org otherwise we’re going to run out of disk space. There was also a regression on Synapse develop on federation, where outbound requests would get stuck and never retry – this impacted the matrix.org server badly over the course of the week, but as of Friday night we have a workaround in place. We’re not aware of it affecting anyone other than the matrix.org deployment (and we haven’t got to the root cause yet).
Construct homeserver progress
This week:
Added notification counts which show up in Riot now, and expanded support for g++-7 and 8 instead of just g++-6. Construct repository is found at: https://github.com/matrix-construct/construct.
IRC Connection Tracker
Half-Shot is continuing to work on the project to split out IRC connection management from the IRC bridge, letting the bridge be restarted without interrupting IRC connections!
The project is going quite well, and is going to be used on matrix.org once production ready which will really speed up upgrades and give us near zero downtime indifferent to the size of the bridge.
At the moment the project has the ability to spin up and maintain connections, however the connections are not supporting IRC fully yet as there are bits to do on the parsing and maintaining state side. There is also work on a top-like tool to visualise and control the service outside of the bridge so we can quickly handle any oddities if they come up. Finally, it allows you to hot reload the configuration without dropping existing connections!
On the work done to support this on matrix-appservice-bridge, there is basic support for stating connections on the bridge but it’s in early stages at the moment.
Spec
Travis has been tidying up loose bits on the Matrix spec this week:
experimenting with splitting out the “Room” specific bits of the spec into a dedicated section rather than being split between the CS and SS APIs;
In practice, finalising the S2S API is now blocked on proving the implementation on Synapse; work on this will resume next week and then we’ll document the end result and ship the r0 at last. Timings are going to be completely determined by available manpower and what level of ops distractions we face (c.f. the Synapse section above…). Whilst we’re waiting for the final S2S details to get hashed out, Travis is going to be helping on Riot dev, to try to stop stuff like this, as there’s no point in having the platonic ideal of a perfect spec if actual users are unable to benefit from it.
#matrix-dev
#matrix-dev:matrix.org was reborn as a new room a couple of weeks ago to flush out old corrupted events, but maybe not everyone knows. Come join #matrix-dev:matrix.org, it’s a starting point for all developers looking to build on the platform. We’re also rebuilding #test:matrix.org and #riot:matrix.org, although once we ship the new state resolution
A sneak peek at Modular…
Finally, there’s been a massive amount of work on the New Vector side of things to soft-launch Modular – a paid hosting platform for Matrix servers (and, in future, paid integrations). At this point we’re looking for early adopters who want a dedicated Riot+Synapse for communities or companies of 50 or more users – but don’t want to have to run it themselves. Modular takes the homeserver hosting we’ve already been providing for Status, TADHack and others, and turns it into a mass-market product. The pricing for early adopters is over 5x cheaper than Slack, so if you’ve been dying to have a reliable, fast and expertly maintained homeserver without any of the headaches of admining one yourself, please head over to https://modular.im and give it a whirl and let us know how it goes! This is also a great way to support Matrix development in general, as money from Modular will directly keep the core Matrix team funded to work on Matrix. Once we’re happy with the soft-launch and have incorporated any feedback we’ll start yelling about it as loud as we can :)
Matrix Live
We’ve had a bit of an accidental hiatus on Matrix Live thanks to getting submerged all the different project endgames happening atm (spec releases, lazy loading, Modular, Riot redesign etc), and for the last few Fridays we’ve got to midnight and beyond with too much still on the todo list to justify recording a video. But to avoid completely falling behind, here’s a slightly exhausted Saturday morning update instead (warning: Matthew is not a morning person).
Roll up, roll up, get it while it’s hot, Synapse 0.33.4 is here.
This release brings together a whole host of bug fixes, some enhancements to resource usage management and a bunch of internal changes in readiness for room member state lazy loading and our ongoing port to Python 3 (we are hoping to ship a py3 test candidate rsn!).
Ben’s away today, so this TWIM is brought to you mainly in association with Cadair’s TWIMbot!
Spec Activity
Since last week’s sprint to get the new spec releases out, focus on the core team has shifted exclusively to the remaining stuff needed to cut the first stable release for the Server-Server API. In practice this means fleshing out the MSCs in flight and implementing them – work has progressed on both improving auth rules, switching event IDs to be hashes and others. Whilst implementing this in Synapse we’re also doing a complete audit and overhaul of the current federation code, hence the 0.33.3.1 security release this week.
Meanwhile, in the community, ma1uta reports:
I am working on the jeon (java matrix api) to update it to the latest stable release. Also I changed versions of api to form rX.Y.Z-N where rX.Y.Z is a API version and N is a library version whithin API. So, I have prepared Push API (r0.1.0-1), Identity API (r0.1.0-1) and Appservice API (r0.1.0-1) for the first release and current updating the C2S API to the r0.4.0 version.
XMPP Bridging
Are you in the market for a Matrix-XMPP bridge? When I say “market”, I mean it because this week we have two announcements for bridging to XMPP! You can choose whether you’d prefer your bridge to be implemented as a puppet, or a bot.
It is a double-puppet bridge which can connects the Matrix and XMPP ecosystems. Just invite the @_xmpp_master:ru-matrix.org and tell him: @_xmpp_master: connect [email protected] to connect current room with the specified conference.
You can ask about this bridge in the #matrix-jabber-java-bridge:ru-matrix.org room.
Currently supports only conferences and only m.text messages. 1:1 conversations and other message types will be later.
maze appeared this week and announced MxBridge, a new Matrix-XMPP bridge:
It works as a bot, so it is non-puppeting. Rooms can be mapped dynamically by the bot administrator(s). There is no support for 1-1 chats (yet). MxBridge is written as a multi-process application in Elixir and it should scale quite well (but don’t tie me down on it ;)). https://github.com/djmaze/mxbridge
Seaglass
Neil powers onwards with Seaglass, with updates this week including:
Displaying stickers
Lazy-loading room history on startup to help with performance
Scrollback support (both forwards and backwards)
Support for Matthew’s Account (aka retries on initial sync for those of us with massive initial syncs, and general perf improvements to nicely support >2000 rooms)
Better avatar support & cosmetics on macOS Mojave
Encryption verification support, device blacklisting and message information
Ability to turn encryption on in rooms
Responding to encryption being turned on in rooms
Paranoid mode for encryption (only send to verified devices)
Invitation support (both in UI and /invite)
Matrique
Blackhat announces that Matrique’s new design is almost done, along with GNU/Linux, MacOS and Windows nightly build!
Fractal
Alexandre Franke says:
Fractal 3.30 got release alongside the rest of GNOME. It includes a bunch of new and updated translations, and redacted messages are now hidden.
Meanwhile, hidden in this screenshot, uhoreg noted that E2E plans are progressing…
Riot
Bruno has been hacking away on Riot/Web squashing the remaining Lazy Loading Members defects and various related optimisations and fixes. We also released Riot/Web 0.16.3 as a fairly minor point release (which unfortunately has a regression with DM avatars, which is fixed in 0.16.4, for which a first RC was cut a few hours ago and should be released on Monday). Meanwhile the first cut of Lazy Loading also got implemented on Android as well. Both are hidden behind labs flags, but we’re almost at a point where we can turn it on now! Otherwise, the Riot team has got sucked into working on commercial Matrix stuff, for better or worse (all shall be revealed shortly though!)
Construct
Jason has been working heavily on Construct, and has new test users. Construct is able to federate with Synapse and the rest of the Matrix ecosystem. mujx has created a docker for Construct which streamlines its deployment.
tulir has now deployed using the standalone install instructions on a very small LXC VM using ZFS. Unfortunately ZFS does not support O_DIRECT (direct disk IO) which is how Construct achieves maximum performance using concurrent reads. This is not a problem though when using an SSD or for personal deployments. I’ll be posting more about how Construct hacked RocksDB to use direct IO, which can get the most out of your hardware with multiple requests in-flight (even with an SSD).
Synapse
Work was split this week into spec/security work, with the critical update for 0.33.3.1 – if you haven’t upgraded, please do so immediately.
Otherwise, Hawkowl has been on a mission to finish the Python 3 port, which is now almost merged. Testers should probably wait until it fully merges to the develop branch and we’ll yell about it then, but impatient adrenaline enthusiasts may want to check out the hawkowl/py3-3 branch (although it may explode in your face, mangle your DB and format your cat, and probably misses lots of recent important PRs like the 0.33.3.1 stuff). However, i’ve been running a variant on some servers for the last few days without problems – and it seems (placebo effect notwithstanding) incredibly snappy…
Meanwhile, the Lazy Loaded Member implementation got sped up by 2-3x, which makes /sync roughly 2-3x faster than it would be without Lazy Loading. This hasn’t merged yet, but was the main final blocker behind Lazy Loading going live!
In addition, Matrix Synapse is now more configurable from the playbook, with support for enabling stats-reporting, event cache size configurability, password peppering.
Matrix Python SDK needs a maintainer
We should say a huge Thank You to &Adam for his work leading the Python SDK over the previous months! Unfortunately due to a busy home life (best of luck for the second child!) he has decided to step down as lead maintainer. Anyone interested in this project should head to https://github.com/matrix-org/matrix-python-sdk/issues/279, and also come and chat in #matrix-python-sdk:matrix.org.
A new bot appears! Are you a pedantic academic who likes to correct others’ misuse of Latin-derived plurals? This task can now be automated for you by means of SingularBot! Also for people who just like to have some fun. Free PongBot and SmileBot included.
kitsune on Hokkaido island
I ended up being on Hokkaido island right when a major earthquake struck it; so no activity on Matrix from me in the nearest couple of days. Also, donations to GlobalGiving for the disaster relief are welcome because people are really struggling here (abusing the communication channel, sorry).
Matrix Live
…has got delayed again; sorry – we’re rather overloaded atm. We’ll catch up as soon as we can.
As referenced in yesterday’s pre-disclosure, today we are releasing Synapse 0.33.3.1 as a critical security update.
We have patched two security vulnerabilities we identified whilst working on the upcoming r0 spec release for the Server-Server API (see details below). We do not believe either have been exploited in the wild, but strongly recommend everybody running a federated Synapse upgrades immediately.
Many thanks for your patience and understanding; with fixes like this we are moving ever closer to Synapse reaching a 1.0 Thanks also to the package maintainers who have coordinated with us to ensure distro packages are available for a speedy upgrade!
Note, for anyone running Debian Jessie, we have prepared a 0.33.2.1 deb (as 0.33.3 dropped support for Jessie).
Synapse 0.33.3.1 (2018-09-06)
SECURITY FIXES
Fix an issue where event signatures were not always correctly validated (#3796)
Fix an issue where server_acls could be circumvented for incoming events (#3796)
Internal Changes
Unignore synctl in .dockerignore to fix docker builds (#3802)
Recently I’ve been working to improve some of the content on the matrix.org website.
Firstly the FAQ now has updated content and a more presentable menu.
We have a Guides Index, which includes a clarified guide list, plus links to off-site contributions from the community. It’s possible to click “recommend” on these items if you’ve had a good experience with them. If you have documentation or guides you’d like to see added to the list, contact me on Matrix or make a pull request on the site repo.
Finally, as part of a programme to improve visibility on projects in the Matrix ecosystem, we are introducing the “Matrix Clients Matrix“. This is a list of some of the most popular current Matrix clients in the ecosystem today, and should shed some light on current feature statuses! The list is not exhaustive, and if you would like to see your client project included, please contact me at the same address, or come chat in the Matrix Client Developers community room. Pretty green Features grid:
During the ongoing work to finalise a stable release of Matrix’s Server-Server federation API, we’ve been doing a full audit of Synapse’s implementation and have identified a serious vulnerability which we are going to release a security update to address (Synapse 0.33.3.1) on Thursday Sept 6th 2018 at 12:00 UTC.
We are coordinating with package maintainers to ensure that patched versions of packages will be available at that time – meanwhile, if you run your own Synapse, please be prepared to upgrade as soon as the patched versions are released. All previous versions of Synapse are affected, so everyone will want to upgrade.
Thank you for your time, patience and understanding while we resolve the issue,
Introducing Client Server API 0.4, and the first ever stable IS, AS and Push APIs spec releases!
Hi folks,
As many know, we’ve been on a massive sprint to improve the spec – both fixing omissions where features have been implemented in the reference servers but were never formalised in the spec, and fixing bugs where the spec has thinkos which stop us from being able to ratify it as stable and thus fit for purpose .
In practice, our target has been to cut stable releases of all the primary Matrix APIs by the end of August – effectively declaring Matrix out of beta, at least at the specification level. For context: historically only one API has ever been released as stable – the Client Server API, which was the result of a similar sprint back in Jan 2016. This means that the Server Server (SS) API, Identity Service (IS) API, Application Service (AS) API and Push Gateway API have never had an official stable release – which has obviously been problematic for those implementing them.
However, as of the end of Friday Aug 31, we’re proud to announce the first ever stable releases of the IS, AS and Push APIs!
To the best of our knowledge, these API specs are now complete and accurately describe all the current behaviour implemented in the reference implementations (sydent, synapse and sygnal) and are fit for purpose. Any deviation from the spec in the reference implementations should probably be considered a bug in the impl. All changes take the form of filling in spec omissions and adding clarifications to the existing behaviour in order to get things to the point that an independent party can implement these APIs without having to refer to anything other than the spec.
This is the result of a lot of work which spans the whole Spec Core Team, but has been particularly driven by TravisR, who has taken the lead on this whole mission to improve the spec. Huge thanks are due to Travis for his work here, and also massive thanks to everyone who has sufferedendured reviewed his PRs and contributed to the releases. The spec is looking unrecognisably better for it – and Matrix 1.0 is feeling closer than ever!
Alongside the work on the IS/AS/Push APIs, there has also been a massive attempt to plug all the spec omissions in the Client Server API. Historically the CS API releases have missed some of the newer APIs (and of course always miss the ones which postdate a given release), but we’ve released the APIs which /have/ been specified as stable in order to declare them stable. However, in this release we’ve tried to go through and fill in as many remaining gaps as possible.
The result is the release of Client Server API version 0.4. This is a huge update – increasing the size of the CS API by ~40%. The biggest new stuff includes fully formalising support for end-to-end encryption (thanks to Zil0!), versioning for rooms (so we can upgrade rooms to new versions of the protocol), synchronised read markers, user directories, server ACLs, MSISDN 3rd party ids, and .well-known server discovery (not that it’s widely used yet), but for the full picture, best bet is to look at the changelog (now managed by towncrier!). It’s probably fair to say that the CS API is growing alarmingly large at this point – Chrome says that it’d be 223 A4 pages if printed. Our solution to this will be to refactor it somehow (and perhaps switch to a more compact representation of the contents).
Some things got deliberately missed from the CS 0.4 release: particularly membership Lazy Loading (because we’re still testing it out and haven’t released it properly in the wild yet), the various GDPR-specific APIs (because they may evolve a bit as we refine them since the original launch), finalising ID grammars in the overall spec (because this is surprisingly hard and subtle and we don’t want to rush it) and finally Communities (aka Groups), as they are still somewhat in flux.
Meanwhile, on the Server to Server API, there has also been a massive amount of work. Since the beginning of July it’s tripled in size as we’ve filled in the gaps, over the course of >200 commits (>150 of which from Travis). If you take a look at the current snapshot it’s pretty unrecognisable from the historical draft; with the main changes being:
Adding the new State Resolution algorithm to address flaws in the original one. This has been where much of our time has gone – see MSC1442 for full details. Adopting the new algorithm requires rooms to be recreated; we’ll write more about this in the near future when we actually roll it out.
Adding room versioning so we can upgrade to the new State Resolution algorithm.
Everything is now properly expressed as Swagger (OpenAPI), just like the CS API
Adding all the details for E2E encryption (including dependencies like to-device messaging and device-list synchronisation)
Improvements in specifying how to authorize inbound events over federation
Document federation APIs such as /event_auth and /query_auth and /get_missing_events
However, we haven’t finished it all: despite our best efforts we’re running slightly past the original target of Aug 31. The current state of play for the r0 release overall (in terms of pending issues) is:…and you can see the full breakdown over at the public Github project dashboard.
The main stuff we still have remaining on the Server/Server API at this point is:
Better specifying how we validate inbound events. See MSC1646 for details & progress.
Switching event IDs to be hashes. See MSC1640 for details and progress.
Various other remaining security considerations (e.g. how to handle malicious auth events in the DAG; how to better handle DoS situations).
Merging in the changes to authoring m.room.power_levels (as per MSC1304)
Formally specifying the remaining identifiers which lack a formal grammar – MSC1597 and particularly room aliases (MSC1608)
The plan here is to continue speccing and implementing these at top priority (with Travis continuing to work fulltime on spec work), and we’ll obviously keep you up-to-date on progress. Some of the changes here (e.g. event IDs) are quite major and we definitely want to implement them before speccing them, so we’re just going to have to keep going as fast as we can. Needless to say we want to cut an r0 of the S2S API alongside the others asap and declare Matrix out of beta (at least at the spec level :)
In terms of visualising progress on this spec mission it’s interesting to look at the rate at which we’ve been closing PRs: this graph shows the total number of PRs which are in state ‘open’ or ‘closed’ on any given day:
…which clearly shows the original sprint to get the r0 of the CS API out the door at the end 2015, and then a more leisurely pace until the beginning of July 2018 since which the pace has picked up massively. Other ways of looking at include the number of open issues…
…or indeed the number of commits per week…
…or the overall Github Project activity for August. (It’s impressive to see Zil0 sneaking in there on second place on the commit count, thanks to all his GSoC work documenting E2E encryption in the spec as part of implementing it in matrix-python-sdk!)
Anyway, enough numerology. It’s worth noting that all of the dev for r0 has generally followed the proposed Open Governance Model for Matrix, with the core spec team made up of both historical core team folk (erik, richvdh, dave & matthew), new core team folk (uhoreg & travis) and community folk (kitsune, anoa & mujx) working together to review and approve the changes – and we’ve been doing MSCs (albeit with an accelerated pace) for anything which we feel requires input from the wider community. Once the Server/Server r0 release is out the door we’ll be finalising the open governance model and switching to a slightly more measured (but productive!) model of spec development as outlined there.
Meanwhile, Matrix 1.0 gets ever closer. With (almost) all this spec mission done, our plan is to focus more on improving the reference implementations – particularly performance in Synapse, UX in matrix-{react,ios,android}-sdk as used by Riot (especially for E2E encryption), and then declare a 1.0 and get back to implementing new features (particularly Editable Messages and Reactions) at last.
We’d like to thank everyone for your patience whilst we’ve been playing catch up on the spec, and hope you agree it’s been worth the effort :)
As many know, we’ve been aiming for the end of August to cut the first ever stable releases of the remaining APIs in the spec which have up to now been marked unstable – i.e. providing a snapshot of the spec which correctly matches the reference implementations (other than implementation bugs) and which can be used in isolation to develop production grade implementations of clients, servers, etc without need to reference any other implementations. There’s been a massive sprint to pull this together, and as of the time of writing there are still PRs and commits landing every few minutes. We’ll post a full update on our progress on Monday; meanwhile you can see a sneak peek over at the August 2018 r0 project board.
Spec work has completely precluded any other backend dev this week.
Half-Shot, gone but not really gone
This week we say farewell to Half-Shot, who has been working fulltime on bridges over the summer. He has managed the matrix.org bridges largely single-handedly, with a big focus on the often-volatile IRC bridge(s).
now bridges a lot more stuff, such as formatting, media and replies. I’m also almost done with desegregating users so that Matrix users join the same group chat portals rather than everyone having their own portal to the same chat
Zulip chat, bridged by Zulip
Matthew discovered there is a Matrix-Zulip bridge on the Zulip side. So if you’re running a Zulip server (for some reason), and want to bridge with Matrix check out the integration docs here.
IRC Connection Tracker
Half-Shot created a new component to enhance the reliability of IRC-Matrix bridging:
IRC Connection Tracker is a thing now. It’s a project to separate out the IRC connections from the appservice so the two can be run independently, so that restarting the appservice doesn’t affect the IRC connections. It’s hopefully going to allow bridge stuff to run faster when it’s done.
This project is still really early stage. You can take a look at the Proposal document here.
Fractal 3.29.92 got released and we are freezing strings to give GNOME translators a bit of time to complete translations for 3.30 next week. Latest developments include tweaks for the title bar, misc bug fixes, a new presentation for uploaded files (that are not images, those are still displayed inline) with buttons to download or open them.
Development builds are now parallel installable for easier testing and CI has been improved.
Seaglass now has some early support for inline images and attachments, and supports Quick Look. Also handles emotes and notices better. It also has version numbers now, various other little fixes and Aaron Raimist has been working on auto-update support.
There’s also been some work on supporting dark mode on Mojave (which looks particularly sexy) and even Touch Bar support!
Riot Android v0.8.15
Riot Android v0.8.15 is on it’s way to the Play Store.
Changes in Riot Android 0.8.15 (2018-08-30)
MatrixSdk:
Upgrade to version 0.9.9.
Improvements:
Improve intent to open document (#2544)
Avoid useless dialog for permission (#2331)
Improve wording when exporting keys (#2289)
Other changes:
Upgrade lib libphonenumber from v8.0.1 to 8.9.12
Upgrade Google firebase libs
Bugfix:
Handle \/ at the beginning of a message to send a message starting with / (#658)
Escape nicknames starting with a forward slash / in mentions (#2146)
Improve management of Push feature
MatrixError mResourceLimitExceededError is now managed in MxDataHandler (vector-im/riot-android#2547 point 2)
Changes in Riot Android 0.8.14 (2018-08-27)
MatrixSdk:
Upgrade to version 0.9.8.
Features:
Manage server quota notices (#2440)
Improvements:
Do not ask permission to write external storage at startup (#2483)
Update settings icon and transparent logo for notifications and navigation drawer (#2492)
URL previews are no longer requested from the server when displaying URL previews is disabled (PR #2514)
Fix some plural and puzzle strings, and remove other unused ones (#2444)
Manage System Alerts in a dedicated section
Other changes:
Upgrade olm-sdk.aar from version 2.2.2 to version 2.3.0
move PieFractionView from the SDK to the client (#2525)
Bugfix:
Fix media sharing (#2530)
Fix notification sound issue in settings (#2524)
Disable app icon badge for “listen for event” notification (#2104)
Riot iOS 0.7.3
Changes in 0.7.3 (2018-08-27)
Improvements:
Upgrade MatrixKit version (v0.8.3).
Bug fix:
Fix input toolbar reset in RoomViewController on MXSession state change (#2006 and #2008).
Fix user interaction disabled in master view of UISplitViewContoller when selecting a room (#2005).
Changes in 0.7.2 (2018-08-24)
Improvements:
Upgrade MatrixKit version (v0.8.2).
Server Quota Notices in Riot (#1937).
Bug fix:
User defaults: the preset application language (if any) is ignored.
Recents: Avoid to open a room twice (it crashed on room creation on quick HSes).
Riot-bot: Do not try to create a room with it if the user homeserver is not federated.
Riot Web
There’s been lots of work debugging and optimising Lazy Loading, which is edging closer to being turned on by default. We’ve also been working away at improving E2E UX – starting with finishing key backup, and then improved verification, and then finally cross-signing (at last!)
Changes to Matrix Android SDK in 0.9.9 (2018-08-30)
Improvements:
Clear unreachable Url when clearing media cache (vector-im/riot-android#2479)
“In reply to” is not clickable on Riot Android yet. Make it a plain text (vector-im/riot-android#2469)
Bugfix:
Removing room from ‘low priority’ or ‘favorite’ does not work (vector-im/riot-android#2526)
MatrixError mResourceLimitExceededError is now managed in MxDataHandler (vector-im/riot-android#2547)
API Change:
MxSession constructor is now private. Please use MxSession.Builder() to create a MxSession
Changes to Matrix Android SDK in 0.9.8 (2018-08-27)
Features:
Manage server_notices tag and server quota notices (vector-im/riot-android#2440)
Bugfix:
Room aliases including the ‘@’ and ‘=’ characters are now recognized as valid (vector-im/riot-android#2079, vector-im/riot-android#2542)
Room name and topic can be now set back to empty (vector-im/riot-android#2345)
API Change:
Remove PieFractionView class from the Matrix SDK. This class is now in Riot sources (#336)
MXMediasCache.createTmpMediaFile() methods are renamed to createTmpDecryptedMediaFile()
MXMediasCache.clearTmpCache() method is renamed to clearTmpDecryptedMediaCache()
Add MXMediasCache.moveToShareFolder() to move a tmp decrypted file to another folder to prevent deletion during sharing. New API MXMediasCache.clearShareDecryptedMediaCache() can be called when the application is resumed. (vector-im/riot-android#2530)
Matrix iOS SDK
Changes in Matrix iOS SDK in 0.11.3 (2018-08-27)
Bug fix:
MXJSONModel: Manage m.server_notice empty tag sent due to a bug server side (PR #556).
Changes in Matrix iOS SDK in 0.11.2 (2018-08-24)
Improvements:
MXSession: Add the supportedMatrixVersions method getting versions of the specification supported by the homeserver.
MXRestClient: Add testUserRegistration to check earlier if a username can be registered.
MXSession: Add MXSessionStateSyncError state and MXSession.syncError to manage homeserver resource quota on /sync requests (vector-im/riot-ios/issues/1937).
MXError: Add kMXErrCodeStringResourceLimitExceeded to manage homeserver resource quota (vector-im/riot-ios/issues/1937).
MXError: Define constant strings for keys and values that can be found in a Matrix JSON dictionary error.
Tests: MXHTTPClient_Private.h: Add method to set fake delay in HTTP requests.
Bug fix:
People tab is empty in the share extension (vector-im/riot-ios/issues/1988).
MXError: MXError lost NSError.userInfo information.
Thank you to Half-Shot for all your work on Bridges over the last months and beyond. Today is your last day, but I’m sure we’ll see you again before long. Text below is from Half-Shot.
Today marks my last day of my 3 month internship at New Vector (the startup which hires many of the core Matrix team). For those of you who haven’t been reading Ben’s fabulous blog posts, I’ve been working exclusively on bridges; in particular the IRC bridge.
Tasked with the goal of making it crash less and run faster, I hope that the evidence is visible and people are generally having a better experience on it!
Some stats pulled from the matrix-appservice-irc repo:
39 PRs closed (4 remain open)
27 issues closed, 27 issues opened.
334 commits, averaging 7.6 commits a PR.
Commits this year:
But aside from showing off some stats, I wanted to mention all the new features:
Replies on Matrix translate well to IRC, or as well as IRC allows.
People mentioning your IRC nick now ping your matrix user, finally!
So. Many. Metrics. Everything you wanted to know about the internals of the bridge, but were too afraid to --inspect.
Not spamming homeservers with join requests on startup (it makes for a happy ops team).
No longer are IRC users shackled to a “(IRC)” extension on their displayname, you can be who you want
with group flairs!
Support for node 4 has been dropped, and support for 6,8 and 10 has been assured.
On the matrix-appservice-bridge side, I optimised some calls to cache locally and avoid hitting the homeserver too often, and disabling presence for homeservers that don’t support it.
There are future plans to make bridging more visible to Matrix Clients as a first class citizen. Ideas like speccing a state event (MSC1410) so that bridges can interact with each other properly and clients can create full bridge management views which are still decentralised from an integration manager.
I’d like to give a shoutout to Travis who has reviewed nearly all my changes that have made their way into the bridge, on top of all the other tasks he has on his plate. And of course a thank you to all of the Matrix team who have been very supportive during my time here.
Recent Comments