Synapse 0.33.5.1 released!

Folks, Synapse 0.33.5.1 is here.

0.33.5.1 is an interesting release. On the one hand it contains the usual bug fixes and performance improvements of a point release, but it also our first versioned release where monolith installs can be run under Python 3.5 and 3.6! Python 3 support is very much in beta, so please be cautious but if you would like to try running under a py3 environment we’d love to get your feedback.

We’ve been running it ourselves for the past few weeks, and feel pretty good about it, not least the 2-3x improvement in RAM usage.

Currently the only way to run under python 3 is to download via github, there is no deb support as yet, though this will come as soon as we are confident to recommend python 3 as the default version.

We’ll be blogging about our porting project in more detail in the future, so watch this space – exciting times!

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, for the first time, Synapse is now available from PyPI, pick it up here.

Synapse 0.33.5.1

 

Internal Changes

  • Fix incompatibility with older Twisted version in tests. Thanks @OlegGirko! (#3940)

Synapse 0.33.5

 

Features

  • Python 3.5 and 3.6 support is now in beta. (#3576)
  • Implement event_format filter param in /sync (#3790)
  • Add synapse_admin_mau:registered_reserved_users metric to expose number of real reaserved users (#3846)

Bugfixes

  • Remove connection ID for replication prometheus metrics, as it creates a large number of new series. (#3788)
  • guest users should not be part of mau total (#3800)
  • Bump dependency on pyopenssl 16.x, to avoid incompatibility with recent Twisted. (#3804)
  • Fix existing room tags not coming down sync when joining a room (#3810)
  • Fix jwt import check (#3824)
  • fix VOIP crashes under Python 3 (#3821) (#3835)
  • Fix manhole so that it works with latest openssh clients (#3841)
  • Fix outbound requests occasionally wedging, which can result in federation breaking between servers. (#3845)
  • Show heroes if room name/canonical alias has been deleted (#3851)
  • Fix handling of redacted events from federation (#3859)
  • (#3874)
  • Mitigate outbound federation randomly becoming wedged (#3875)

Internal Changes

  • CircleCI tests now run on the potential merge of a PR. (#3704)
  • http/ is now ported to Python 3. (#3771)
  • Improve human readable error messages for threepid registration/account update (#3789)
  • Make /sync slightly faster by avoiding needless copies (#3795)
  • handlers/ is now ported to Python 3. (#3803)
  • Limit the number of PDUs/EDUs per federation transaction (#3805)
  • Only start postgres instance for postgres tests on Travis CI (#3806)
  • tests/ is now ported to Python 3. (#3808)
  • crypto/ is now ported to Python 3. (#3822)
  • rest/ is now ported to Python 3. (#3823)
  • add some logging for the keyring queue (#3826)
  • speed up lazy loading by 2-3x (#3827)
  • Improved Dockerfile to remove build requirements after building reducing the image size. (#3834)
  • Disable lazy loading for incremental syncs for now (#3840)
  • federation/ is now ported to Python 3. (#3847)
  • Log when we retry outbound requests (#3853)
  • Removed some excess logging messages. (#3855)
  • Speed up purge history for rooms that have been previously purged (#3856)
  • Refactor some HTTP timeout code. (#3857)
  • Fix running merged builds on CircleCI (#3858)
  • Fix typo in replication stream exception. (#3860)
  • Add in flight real time metrics for Measure blocks (#3871)
  • Disable buffering and automatic retrying in treq requests to prevent timeouts. (#3872)
  • mention jemalloc in the README (#3877)
  • Remove unmaintained “nuke-room-from-db.sh” script (#3888)

Synapse 0.33.4 released!

Roll up, roll up, get it while it’s hot, Synapse 0.33.4 is here.

This release brings together a whole host of bug fixes, some enhancements to resource usage management and a bunch of internal changes in readiness for room member state lazy loading and our ongoing port to Python 3 (we are hoping to ship a py3 test candidate rsn!).

As ever, you can get the new update from https://github.com/matrix-org/synapse/releases/tag/v0.33.4 or any of the sources mentioned at https://github.com/matrix-org/synapse.

Features

  • Support profile API endpoints on workers (#3659)
  • Server notices for resource limit blocking (#3680)
  • Allow guests to use /rooms/:roomId/event/:eventId (#3724)
  • Add mau_trial_days config param, so that users only get counted as MAU after N days. (#3749)
  • Require twisted 17.1 or later (fixes #3741). (#3751)

Bugfixes

  • Fix error collecting prometheus metrics when run on dedicated thread due to threading concurrency issues (#3722)
  • Fix bug where we resent “limit exceeded” server notices repeatedly (#3747)
  • Fix bug where we broke sync when using limit_usage_by_mau but hadn’t configured server notices (#3753)
  • Fix ‘federation_domain_whitelist’ such that an empty list correctly blocks all outbound federation traffic (#3754)
  • Fix tagging of server notice rooms (#3755#3756)
  • Fix ‘admin_uri’ config variable and error parameter to be ‘admin_contact’ to match the spec. (#3758)
  • Don’t return non-LL-member state in incremental sync state blocks (#3760)
  • Fix bug in sending presence over federation (#3768)
  • Fix bug where preserved threepid user comes to sign up and server is mau blocked (#3777)

Internal Changes

  • Removed the link to the unmaintained matrix-synapse-auto-deploy project from the readme. (#3378)
  • Refactor state module to support multiple room versions (#3673)
  • The synapse.storage module has been ported to Python 3. (#3725)
  • Split the state_group_cache into member and non-member state events (and so speed up LL /sync) (#3726)
  • Log failure to authenticate remote servers as warnings (without stack traces) (#3727)
  • The CONTRIBUTING guidelines have been updated to mention our use of Markdown and that .misc files have content. (#3730)
  • Reference the need for an HTTP replication port when using the federation_reader worker (#3734)
  • Fix minor spelling error in federation client documentation. (#3735)
  • Remove redundant state resolution function (#3737)
  • The test suite now passes on PostgreSQL. (#3740)
  • Fix MAU cache invalidation due to missing yield (#3746)
  • Make sure that we close db connections opened during init (#3764)
  • Unignore synctl in .dockerignore to fix docker builds (#3802)

 

Critical Security Update: Synapse 0.33.3.1

Hi All,

As referenced in yesterday’s pre-disclosure, today we are releasing Synapse 0.33.3.1 as a critical security update.

We have patched two security vulnerabilities we identified whilst working on the upcoming r0 spec release for the Server-Server API (see details below). We do not believe either have been exploited in the wild, but strongly recommend everybody running a federated Synapse upgrades immediately.

As always you can get the new update here or from any of the sources mentioned at https://github.com/matrix-org/synapse/

Many thanks for your patience and understanding; with fixes like this we are moving ever closer to Synapse reaching a 1.0 Thanks also to the package maintainers who have coordinated with us to ensure distro packages are available for a speedy upgrade!

Note, for anyone running Debian Jessie, we have prepared a 0.33.2.1 deb (as 0.33.3 dropped support for Jessie).

 

Synapse 0.33.3.1 (2018-09-06)

SECURITY FIXES

  • Fix an issue where event signatures were not always correctly validated (#3796)
  • Fix an issue where server_acls could be circumvented for incoming events (#3796)

Internal Changes

  • Unignore synctl in .dockerignore to fix docker builds (#3802)

Recent matrix.org website improvements

Recently I’ve been working to improve some of the content on the matrix.org website.

Firstly the FAQ now has updated content and a more presentable menu.

We have a Guides Index, which includes a clarified guide list, plus links to off-site contributions from the community. It’s possible to click “recommend” on these items if you’ve had a good experience with them. If you have documentation or guides you’d like to see added to the list, contact me on Matrix or make a pull request on the site repo.

Finally, as part of a programme to improve visibility on projects in the Matrix ecosystem, we are introducing the “Matrix Clients Matrix“. This is a list of some of the most popular current Matrix clients in the ecosystem today, and should shed some light on current feature statuses! The list is not exhaustive, and if you would like to see your client project included, please contact me at the same address, or come chat in the Matrix Client Developers community room. Pretty green Features grid:

Pre-disclosure: Upcoming critical security fix for Synapse

Hi all,

During the ongoing work to finalise a stable release of Matrix’s Server-Server federation API, we’ve been doing a full audit of Synapse’s implementation and have identified a serious vulnerability which we are going to release a security update to address (Synapse 0.33.3.1) on Thursday Sept 6th 2018 at 12:00 UTC.

We are coordinating with package maintainers to ensure that patched versions of packages will be available at that time – meanwhile, if you run your own Synapse, please be prepared to upgrade as soon as the patched versions are released.  All previous versions of Synapse are affected, so everyone will want to upgrade.

Thank you for your time, patience and understanding while we resolve the issue,

signed_predisclosure.txt

Matrix Spec Update August 2018

Introducing Client Server API 0.4, and the first ever stable IS, AS and Push APIs spec releases!

Hi folks,

As many know, we’ve been on a massive sprint to improve the spec – both fixing omissions where features have been implemented in the reference servers but were never formalised in the spec, and fixing bugs where the spec has thinkos which stop us from being able to ratify it as stable and thus fit for purpose .

In practice, our target has been to cut stable releases of all the primary Matrix APIs by the end of August – effectively declaring Matrix out of beta, at least at the specification level.  For context: historically only one API has ever been released as stable – the Client Server API, which was the result of a similar sprint back in Jan 2016. This means that the Server Server (SS) API, Identity Service (IS) API, Application Service (AS) API and Push Gateway API have never had an official stable release – which has obviously been problematic for those implementing them.

However, as of the end of Friday Aug 31, we’re proud to announce the first ever stable releases of the IS, AS and Push APIs!


To the best of our knowledge, these API specs are now complete and accurately describe all the current behaviour implemented in the reference implementations (sydent, synapse and sygnal) and are fit for purpose. Any deviation from the spec in the reference implementations should probably be considered a bug in the impl. All changes take the form of filling in spec omissions and adding clarifications to the existing behaviour in order to get things to the point that an independent party can implement these APIs without having to refer to anything other than the spec.

This is the result of a lot of work which spans the whole Spec Core Team, but has been particularly driven by TravisR, who has taken the lead on this whole mission to improve the spec.  Huge thanks are due to Travis for his work here, and also massive thanks to everyone who has suffered endured reviewed his PRs and contributed to the releases.  The spec is looking unrecognisably better for it – and Matrix 1.0 is feeling closer than ever!

Alongside the work on the IS/AS/Push APIs, there has also been a massive attempt to plug all the spec omissions in the Client Server API.  Historically the CS API releases have missed some of the newer APIs (and of course always miss the ones which postdate a given release), but we’ve released the APIs which /have/ been specified as stable in order to declare them stable.  However, in this release we’ve tried to go through and fill in as many remaining gaps as possible.

The result is the release of Client Server API version 0.4. This is a huge update – increasing the size of the CS API by ~40%. The biggest new stuff includes fully formalising support for end-to-end encryption (thanks to Zil0!), versioning for rooms (so we can upgrade rooms to new versions of the protocol), synchronised read markers, user directories, server ACLs, MSISDN 3rd party ids, and .well-known server discovery (not that it’s widely used yet), but for the full picture, best bet is to look at the changelog (now managed by towncrier!).  It’s probably fair to say that the CS API is growing alarmingly large at this point – Chrome says that it’d be 223 A4 pages if printed. Our solution to this will be to refactor it somehow (and perhaps switch to a more compact representation of the contents).

Some things got deliberately missed from the CS 0.4 release: particularly membership Lazy Loading (because we’re still testing it out and haven’t released it properly in the wild yet), the various GDPR-specific APIs (because they may evolve a bit as we refine them since the original launch), finalising ID grammars in the overall spec (because this is surprisingly hard and subtle and we don’t want to rush it) and finally Communities (aka Groups), as they are still somewhat in flux.

Meanwhile, on the Server to Server API, there has also been a massive amount of work.  Since the beginning of July it’s tripled in size as we’ve filled in the gaps, over the course of >200 commits (>150 of which from Travis).  If you take a look at the current snapshot it’s pretty unrecognisable from the historical draft; with the main changes being:

  • Adding the new State Resolution algorithm to address flaws in the original one.  This has been where much of our time has gone – see MSC1442 for full details.  Adopting the new algorithm requires rooms to be recreated; we’ll write more about this in the near future when we actually roll it out.
  • Adding room versioning so we can upgrade to the new State Resolution algorithm.
  • Everything is now properly expressed as Swagger (OpenAPI), just like the CS API
  • Adding all the details for E2E encryption (including dependencies like to-device messaging and device-list synchronisation)
  • Improvements in specifying how to authorize inbound events over federation
  • Document federation APIs such as /event_auth and /query_auth and /get_missing_events
  • Document 3rd party invites over federation
  • Document the /user/* federation endpoints
  • Document Server ACLs
  • Document read receipts over federation
  • Document presence over federation
  • Document typing notifications over federation
  • Document content repository over federation
  • Document room directory over federation
  • …and many many other minor bug fixes, omission fixes, and restructuring for coherency – see https://github.com/matrix-org/matrix-doc/issues/1464 for an even longer list :)

However, we haven’t finished it all: despite our best efforts we’re running slightly past the original target of Aug 31.  The current state of play for the r0 release overall (in terms of pending issues) is:…and you can see the full breakdown over at the public Github project dashboard.

The main stuff we still have remaining on the Server/Server API at this point is:

  • Better specifying how we validate inbound events. See MSC1646 for details & progress.
  • Switching event IDs to be hashes. See MSC1640 for details and progress.
  • Various other remaining security considerations (e.g. how to handle malicious auth events in the DAG; how to better handle DoS situations).
  • Merging in the changes to authoring m.room.power_levels (as per MSC1304)
  • Formally specifying the remaining identifiers which lack a formal grammar – MSC1597 and particularly room aliases (MSC1608)

The plan here is to continue speccing and implementing these at top priority (with Travis continuing to work fulltime on spec work), and we’ll obviously keep you up-to-date on progress.  Some of the changes here (e.g. event IDs) are quite major and we definitely want to implement them before speccing them, so we’re just going to have to keep going as fast as we can. Needless to say we want to cut an r0 of the S2S API alongside the others asap and declare Matrix out of beta (at least at the spec level :)

In terms of visualising progress on this spec mission it’s interesting to look at the rate at which we’ve been closing PRs: this graph shows the total number of PRs which are in state ‘open’ or ‘closed’ on any given day:

…which clearly shows the original sprint to get the r0 of the CS API out the door at the end 2015, and then a more leisurely pace until the beginning of July 2018 since which the pace has picked up massively.  Other ways of looking at include the number of open issues…


…or indeed the number of commits per week…


…or the overall Github Project activity for August.  (It’s impressive to see Zil0 sneaking in there on second place on the commit count, thanks to all his GSoC work documenting E2E encryption in the spec as part of implementing it in matrix-python-sdk!)


Anyway, enough numerology.  It’s worth noting that all of the dev for r0 has generally followed the proposed Open Governance Model for Matrix, with the core spec team made up of both historical core team folk (erik, richvdh, dave & matthew), new core team folk (uhoreg & travis) and community folk (kitsune, anoa & mujx) working together to review and approve the changes – and we’ve been doing MSCs (albeit with an accelerated pace) for anything which we feel requires input from the wider community.  Once the Server/Server r0 release is out the door we’ll be finalising the open governance model and switching to a slightly more measured (but productive!) model of spec development as outlined there.

Meanwhile, Matrix 1.0 gets ever closer.  With (almost) all this spec mission done, our plan is to focus more on improving the reference implementations – particularly performance in Synapse, UX in matrix-{react,ios,android}-sdk as used by Riot (especially for E2E encryption), and then declare a 1.0 and get back to implementing new features (particularly Editable Messages and Reactions) at last.

We’d like to thank everyone for your patience whilst we’ve been playing catch up on the spec, and hope you agree it’s been worth the effort :)

Matthew & the core spec team.

So long Half-Shot, thanks for all the bridges

Thank you to Half-Shot for all your work on Bridges over the last months and beyond. Today is your last day, but I’m sure we’ll see you again before long. Text below is from Half-Shot.


Today marks my last day of my 3 month internship at New Vector (the startup which hires many of the core Matrix team). For those of you who haven’t been reading Ben’s fabulous blog posts, I’ve been working exclusively on bridges; in particular the IRC bridge.

Tasked with the goal of making it crash less and run faster, I hope that the evidence is visible and people are generally having a better experience on it!

Some stats pulled from the matrix-appservice-irc repo:

  • 39 PRs closed (4 remain open)
  • 27 issues closed, 27 issues opened.
  • 334 commits, averaging 7.6 commits a PR.

Commits this year:

Commits this year


But aside from showing off some stats, I wanted to mention all the new features:

  • Replies on Matrix translate well to IRC, or as well as IRC allows.
  • People mentioning your IRC nick now ping your matrix user, finally!
  • So. Many. Metrics. Everything you wanted to know about the internals of the bridge, but were too afraid to --inspect.
  • Not spamming homeservers with join requests on startup (it makes for a happy ops team).
  • No longer are IRC users shackled to a “(IRC)” extension on their displayname, you can be who you want
    with group flairs!
  • Support for node 4 has been dropped, and support for 6,8 and 10 has been assured.

On the matrix-appservice-bridge side, I optimised some calls to cache locally and avoid hitting the homeserver too often, and disabling presence for homeservers that don’t support it.

There are future plans to make bridging more visible to Matrix Clients as a first class citizen. Ideas like speccing a state event (MSC1410) so that bridges can interact with each other properly and clients can create full bridge management views which are still decentralised from an integration manager.

I’d like to give a shoutout to Travis who has reviewed nearly all my changes that have made their way into the bridge, on top of all the other tasks he has on his plate. And of course a thank you to all of the Matrix team who have been very supportive during my time here.

Synapse 0.33.3 Released

All the threes, Synapse 0.33.3!

This release brings together a lot of bugfixes, and also some preparation for support for Lazy Loading and Room Versioning.

We also have, as a great contribution from @vojeroen, SNI extension support! With v0.33.3, Synapse now supports sending SNI over federation for vhosted servers, which resolves this long-standing request.

As always, you can get the new update from https://github.com/matrix-org/synapse/releases/tag/v0.33.3 or any of the sources mentioned at https://github.com/matrix-org/synapse.

 

Features

  • Add support for the SNI extension to federation TLS connections. Thanks to @vojeroen! (#3439)
  • Add /_media/r0/config (#3184)
  • speed up /members API and add at and membership params as per MSC1227 (#3568)
  • implement summary block in /sync response as per MSC688 (#3574)
  • Add lazy-loading support to /messages as per MSC1227 (#3589)
  • Add ability to limit number of monthly active users on the server (#3633)
  • Support more federation endpoints on workers (#3653)
  • Basic support for room versioning (#3654)
  • Ability to disable client/server Synapse via conf toggle (#3655)
  • Ability to whitelist specific threepids against monthly active user limiting (#3662)
  • Add some metrics for the appservice and federation event sending loops (#3664)
  • Where server is disabled, block ability for locked out users to read new messages (#3670)
  • set admin uri via config, to be used in error messages where the user should contact the administrator (#3687)
  • Synapse’s presence functionality can now be disabled with the “use_presence” configuration option. (#3694)
  • For resource limit blocked users, prevent writing into rooms (#3708)

Bugfixes

  • Fix occasional glitches in the synapse_event_persisted_position metric (#3658)
  • Fix bug on deleting 3pid when using identity servers that don’t support unbind API (#3661)
  • Make the tests pass on Twisted < 18.7.0 (#3676)
  • Don’t ship recaptcha_ajax.js, use it directly from Google (#3677)
  • Fixes test_reap_monthly_active_users so it passes under postgres (#3681)
  • Fix mau blocking calulation bug on login (#3689)
  • Fix missing yield in synapse.storage.monthly_active_users.initialise_reserved_users (#3692)
  • Improve HTTP request logging to include all requests (#3700, #3723)
  • Avoid timing out requests while we are streaming back the response (#3701)
  • Support more federation endpoints on workers (#3705, #3713)
  • Fix “Starting db txn ‘get_all_updated_receipts’ from sentinel context” warning (#3710)
  • Fix bug where state_cache cache factor ignored environment variables (#3719)

Deprecations and Removals

Internal Changes

  • The test suite now can run under PostgreSQL. (#3423)
  • Refactor HTTP replication endpoints to reduce code duplication (#3632)
  • Tests now correctly execute on Python 3. (#3647)
  • Sytests can now be run inside a Docker container. (#3660)
  • Port over enough to Python 3 to allow the sytests to start. (#3668, #3732)
  • Update docker base image from alpine 3.7 to 3.8. (#3669)
  • Rename synapse.util.async to synapse.util.async_helpers to mitigate async becoming a keyword on Python 3.7. (#3678)
  • Synapse’s tests are now formatted with the black autoformatter. (#3679)
  • Implemented a new testing base class to reduce test boilerplate. (#3684)
  • Rename MAU prometheus metrics (#3690)
  • add new error type ResourceLimit (#3707)
  • Logcontexts for replication command handlers (#3709)
  • Update admin register API documentation to reference a real user ID. (#3712)

Synapse 0.33.2 is here!

Folks, it’s release time, Synapse 0.33.2 has landed.

The release focuses on performance, notable highlights include reducing CPU consumption through speeding up state delta calculations (#3592) and reducing I/O through lazily loading state on the master process (#3579#3581#3582#3584)

Separately work continues on our python 3 port and we hope to have something concrete to trial very soon – we’re really excited about this and expect step change improvements in CPU and memory use.

Finally we have some ground work for upcoming room membership lazy loading, there is nothing to see here as yet, but rest assured we will make a lot of noise as soon as it is ready. Stay tuned.

As always, you can get the new update from https://github.com/matrix-org/synapse/releases/tag/v0.33.2 or any of the sources mentioned at https://github.com/matrix-org/synapse.

 

Synapse 0.33.2 (2018-08-09)

No significant changes.

Synapse 0.33.2rc1 (2018-08-07)

Features

  • add support for the lazy_loaded_members filter as per MSC1227 (#2970)
  • add support for the include_redundant_members filter param as per MSC1227 (#3331)
  • Add metrics to track resource usage by background processes (#3553#3556#3604#3610)
  • Add code label to synapse_http_server_response_time_seconds prometheus metric (#3554)
  • Add support for client_reader to handle more APIs (#3555#3597)
  • make the /context API filter & lazy-load aware as per MSC1227 (#3567)
  • Add ability to limit number of monthly active users on the server (#3630)
  • When we fail to join a room over federation, pass the error code back to the client. (#3639)
  • Add a new /admin/register API for non-interactively creating users. (#3415)

Bugfixes

  • Make /directory/list API return 404 for room not found instead of 400 (#2952)
  • Default inviter_display_name to mxid for email invites (#3391)
  • Don’t generate TURN credentials if no TURN config options are set (#3514)
  • Correctly announce deleted devices over federation (#3520)
  • Catch failures saving metrics captured by Measure, and instead log the faulty metrics information for further analysis. (#3548)
  • Unicode passwords are now normalised before hashing, preventing the instance where two different devices or browsers might send a different UTF-8 sequence for the password. (#3569)
  • Fix potential stack overflow and deadlock under heavy load (#3570)
  • Respond with M_NOT_FOUND when profiles are not found locally or over federation. Fixes #3585 (#3585)
  • Fix failure to persist events over federation under load (#3601)
  • Fix updating of cached remote profiles (#3605)
  • Fix ‘tuple index out of range’ error (#3607)
  • Only import secrets when available (fix for py < 3.6) (#3626)

Internal Changes

  • Remove redundant checks on who_forgot_in_room (#3350)
  • Remove unnecessary event re-signing hacks (#3367)
  • Rewrite cache list decorator (#3384)
  • Move v1-only REST APIs into their own module. (#3460)
  • Replace more instances of Python 2-only iteritems and itervalues uses. (#3562)
  • Refactor EventContext to accept state during init (#3577)
  • Improve Dockerfile and docker-compose instructions (#3543)
  • Release notes are now in the Markdown format. (#3552)
  • add config for pep8 (#3559)
  • Merge Linearizer and Limiter (#3571#3572)
  • Lazily load state on master process when using workers to reduce DB consumption (#3579#3581#3582#3584)
  • Fixes and optimisations for resolve_state_groups (#3586)
  • Improve logging for exceptions when handling PDUs (#3587)
  • Add some measure blocks to persist_events (#3590)
  • Fix some random logcontext leaks. (#3591#3606)
  • Speed up calculating state deltas in persist_event loop (#3592)
  • Attempt to reduce amount of state pulled out of DB during persist_events (#3595)
  • Fix a documentation typo in on_make_leave_request (#3609)
  • Make EventStore inherit from EventFederationStore (#3612)
  • Remove some redundant joins on event_edges.room_id (#3613)
  • Stop populating events.content (#3614)
  • Update the /send_leave path registration to use event_id rather than a transaction ID. (#3616)
  • Refactor FederationHandler to move DB writes into separate functions (#3621)
  • Remove unused field “pdu_failures” from transactions. (#3628)
  • rename replication_layer to federation_client (#3634)
  • Factor out exception handling in federation_client (#3638)
  • Refactor location of docker build script. (#3644)
  • Update CONTRIBUTING to mention newsfragments. (#3645)

 

This Week in Matrix 2018-08-03

Spec Progress

Progress on the spec has been motoring since TravisR dived (dove?) into it full time a few weeks ago – the Federation API r0 megathread bug that tracks progress on filling in the gaps on the S2S API is clearing its checkboxes at an impressive rate.

Some points of note regarding current proposals:

  • MSC1466 Erik proposes a soft_logout field to be added to the body of 401 responses, to better help handling of encryption keys. Check the proposal notes
  • MSC1452 agreement has been reached on Homeserver Warning Messages

We’re going with pinned messages (option 2) and room tags (option 5) as that seems to be where the consensus is: it re-uses existing bits of the spec and room tags also help clients that don’t know about this specific room tag to handle the room the right way

  • MSC1425 Room Versioning
    It’s likely that in the immediate future we’ll want to change the properties of rooms in a way that will not be compatible with existing servers – for example, changing the rules for event auth or state resolution, or changing the format of an event id.
  • MSC1318 Documentation describing the anticipated Open Governance of Matrix.org (aka, Matrix.org Foundation)

Python SDK -> Python 3 ?

The maintainers of the Matrix Python SDK are mulling some major changes to the library. In particular, the desire to use await / async syntax means they are considering making Python 3.5 the minimum supported version. Go chat about this change and comment on the proposal issue.

Clients

Riot/Web 0.16

Big congratulations to the Riot/Web team on the release of 0.16. You can read all about it here, but I’ll give you the headlines now:

  • Replies are now available, there is UX for them and they look great
  • Jitsi is now the default video conferencing provider across Web, iOS and Android, with new widget integrations for Riot Web
  • New composer (text box) using Slate.js rather than Draft.js, which fixes many existing bugs and improves performance

Meanwhile, Lazy Loading implementation is approaching completion, promising several factors of improved resource utilisation!

nheko 0.5.2

Also now available on flathub!

Go download nheko and check out the 0.5.2 release notes.

New features just in the last week or so:

  • Mark own read messages with a double checkmark
  • Add option to specify the scale factor
  • Add input field to specify the device name on login.
  • Add option to ignore key requests altogether.
  • Show device list in user profile & add option to create 1-1 chat.

Plus lots of improvements and bug fixes.

libQMatrixClient and Quaternion

kitsune has been working on resend functionality:

libQMatrixClient and Quaternion have gained ability to resend and discard unsent messages.
this means if Quaternion could not, after several attempts, deliver a message, a user can click “Resend” and it will try again

On the subject of libQMatrixClient, it’s exciting that Konversation, the KDE IRC client, may in future start to use libQMatrixClient for Matrix support!

Matrique

Black Hat announces a Flatpak repo for Matrique:

Matrique now has a Flatpak repo. It is the nightly build of the master branch. You can add the repo by typing flatpak remote-add matrique https://b0.gitlab.io/matrique-repo/matrique.flatpakrepo and install it by flatpak install matrique org.eu.encom.matrique
As it is still Alpha quality, bugs are expected. Feel free to open an issue if anything goes wrong!

Fractal 3.29.6

New release of Fractal to 3.29.6. Notes from the changelog:

  • Add German translation
  • Message right click menu with: view source, reply, copy text and delete
  • Styles for quotes in messages
  • Initial sync speed up

Neo

Incremental improvements to Neo from fox:

Neo now has inline youtube and image url previews, and handles room state changes such as name, avatar and topic as they occur.

Riot/Mobile

  • Android: a lot of bug fixes and small UI improvements
  • iOS: Lazy Loading is coming to life, showing huge improvements in bandwidth usage and performance in the app

Updates on IRC bridges from Half-Shot

Half-Shot has been working tirelessly on the IRC bridge lately, so I wanted to update on his recent successes:

I’ve recently been working on mitigating the effects of a netsplit on the IRC bridge, and optimising it to start and run faster. This week I trimmed down the heap usage (where the memory usually goes) to just under a gigabyte on my 10,000 matrix user test bridge. Previously it could spike to as much as 3.5GB. This optimisation is still in a testing phase but results are looking positive.

For reference here is the memory usage of the Freenode process during startup:

And here are the results of my local test bridge before and after the change:

Before:

After:

We also made some internal changes to the appservice-bridge to cache the joined state of all the bridge users and therefore avoiding joining rooms which saves us some time on startup.

Matrix for Grafana, and more from Ananace

In his regular spot, Ananace has made progress on his Matrix sysadmin/ruby suite:

Synapse

Synapse 0.33.1 is out now as a security update release. Please update if you haven’t already – it fixes two issues concerning event visibility where if you knew the event ID of an event you could read it even if you didn’t have access to it; we don’t believe these have been exploited in the wild, but you will definitely want to upgrade now.

Meanwhile the Python 3 port is progressing well (all sytests now pass in Python 3, i think!), and intrepid folks are starting to experiment with running it in production.

Decentralised Web Summit & Matrix Live

Meanwhile, Matthew & Amandine have been in San Francisco for the 2018 Decentralised Web Summit – so this week’s Matrix Live is live from SFO and gives a quick overview of the sort of things we got up to!  Some of the sessions are already online thanks to the (somewhat unreliable) live stream (e.g. here’s Muneeb (Blockstack), Amandine, Danielle (Dat), and Zooko (Zcash) talking about their respective governance models & growing pains over the last 2 years: https://youtu.be/tsz3ffrJDpw?t=12133).  The summit was a massive success, with lots of discussions about decentralised reputation, UI/UX for decentralised apps, metadata-resistance, the balance of P2P versus decentralised-servers, etc.  Hopefully some of the conversations we had will result in some major improvements to Matrix in the future!

Edit: Here are the slides for our “Diving into Decentralised Communication” workshop, for those interested in a comparison between Matrix/SSB/Mastodon/Status/Vuvuzela/Briar.  They’re pretty minimal, as they just formed a framework for discussion, but might still be of interest.