Category – Tech
54 posts tagged with "Tech" (See all categories)

Dendrite Progress Update

2018-07-30 — Tech — Ben Parsons

As you may know, for the last few months anoa (Andrew) and APWhiteHat have been working on Dendrite, the next generation Matrix homeserver, written in Go. We asked for an update on their progress, and Andrew provided the blog post below. Serious progress has been made on Dendrite this summer!


Hey everyone, my name is Andrew Morgan and I've been working full-time over the summer on Dendrite, our next-generation Matrix homeserver. Over the last two months, I've seen the project transform from a somewhat functioning toy server to a near-production-ready homeserver that is working towards complete feature support. I've appreciated the thought put into the project since day one, and enjoy the elegance of the multi-component design. Documentation is fairly decent at the moment, but comments are plentiful throughout the codebase, while the code itself tends towards simple and maintainable rather than complex and unmanageable.

Application Service Integration

The main focus of my time here has been on the implementation of application service support for Dendrite. Application services are external programs that act as privileged extensions to a homeserver, allowing such functionality as bots in rooms and bridges to third-party networks. Supporting application services requires a few different bits and pieces to be set up. Currently all planned features have a PR for them, with the bold items already merged:
  • Sending events to application services
  • Support user masquerading for events
  • Support editing event timestamps
  • Support room alias querying
  • Support user ID querying
  • Support third party lookup proxying
As you can see a decent portion of the functionality is already in master! The rest will hopefully follow after some further back and forth.

Google Summer of Code

I certainly haven't been going at this all on my own. Alongside extensive help from Erik, who's been mentoring me, our resident Google Summer of Code student, APWhiteHat, has been tackling feature after feature in Dendrite wherever he can find them. Application services received a good deal of help on client-server endpoint authentication side, however, APWhiteHat has mostly been focusing on federation and some other very useful pieces. While his GSoC period still has a week or so before its conclusion, he has so far implemented:
  • Idempotency to roomserver event processing to prevent duplication
  • Username auto generation
  • Tokens library based on macaroons
  • Lots of left-over federation stuff: state API & get missing events being the major ones
  • AS support to clientapi auth
  • Typing server: handling of PUT /typing by clientapi
  • More typing server stuff on its way
From my perspective, APWhiteHat was an excellent developer to work with. He asked good questions and was quick to answer any myself or the community had as well. His code reviews were also very comprehensive. I learned a lot from working with him and everyone else :)

OpenTracing and Prometheus Monitoring

Placing any large server into a production environment requires extensive monitoring capabilities in order to ensure operations are running smoothly. To that effect, Dendrite has been both the addition of OpenTracing and Prometheus support. Prometheus, also used heavily in Synapse, allows a homeserver operator to track a wide range of data including endpoint usage, resource management as well as user statistics over any given range of time.

In Dendrite, we are taking this one step further by introducing OpenTracing, a language and platform-agnostic framework for tracking the journey of an endpoint call from incoming request to outgoing response, with every method, hierarchy change and database call in between. It will be immensely useful in tracking down performance issues, as well as providing insight into the most critical paths throughout the codebase and where we should focus most of our optimization efforts on. It also comes with a lovely dashboard courtesy of Jaeger:

Community

We've also seen some encouraging interest and development work from the community in the past couple months. While PR review from our own side is admittedly slow due to our focus on getting the foundational work in place, that hasn't stopped both old and new developers from sending in PRs and performing code reviews. A huge thank you to everyone involved! From this we've gotten API implementations and application service fixes from @turt2live, an end-to-end encryption implementation from @fadeAce, filtering support from @CromFr, and some PRs and numerous helpful review comments from @krombel.

We've also started to see some people running Dendrite in live environments, which is incredibly exciting for us to see! While Dendrite is not considered production-ready yet (though it moves closer every day), if you are interested in giving it a go please consult the quickstart installation guide. We look forward to any feedback you may have!

Synapse 0.33.0 is here!!

2018-07-19 — Tech — Matthew Hodgson

Hi all,

We've just released Synapse 0.33.0!  This is a major performance upgrade which speeds up /sync (i.e. receiving messages) by a factor of almost 2x!  This has already made a massive difference to the CPU usage and snappiness of the matrix.org homeserver since we rolled it out a few days ago - you can see the drop in sync worker CPU just before midday on July 17th; previously we were regularly hitting the CPU ceiling (at which point everything grinds to a halt) - now we're back down hovering between 40% and 60% CPU (at the current load).  This is actually fixing a bug which crept in around Synapse 0.31, so please upgrade - especially if Synapse has been feeling slower than usual recently, and especially if you are still on Synapse 0.31.

Meanwhile we have a lot of new stuff coming on the horizon - a whole new algorithm for state resolution (watch this space for details); incremental state resolution (at last!) to massively speed up state resolution and mitigate extremities build up (and speed up the synapse master process, which is now the bottleneck again on the matrix.org homeserver); better admin tools for managing resource usage, and all the Python3 porting work (with associated speedups and RAM & GC improvements).  Fun times ahead!

The full changelog follows below; as always you can grab Synapse from https://github.com/matrix-org/synapse.   Thanks for flying Matrix!

Synapse 0.33.0 (2018-07-19)

Bugfixes

  • Disable a noisy warning about logcontexts. (#3561)

Synapse 0.33.0rc1 (2018-07-18)

Features

  • Enforce the specified API for report_event. (#3316)
  • Include CPU time from database threads in request/block metrics. (#3496#3501)
  • Add CPU metrics for _fetch_event_list. (#3497)
  • Optimisation to make handling incoming federation requests more efficient. (#3541)

Bugfixes

  • Fix a significant performance regression in /sync. (#3505#3521#3530#3544)
  • Use more portable syntax in our use of the attrs package, widening the supported versions. (#3498)
  • Fix queued federation requests being processed in the wrong order. (#3533)
  • Ensure that erasure requests are correctly honoured for publicly accessible rooms when accessed over federation. (#3546)

Misc

Security update: Synapse 0.31.2

2018-06-14 — Tech — Matthew Hodgson

Hi all,

On Monday (2018-06-11) we had an incident where #matrix:matrix.org was hijacked by a malicious user pretending to join the room immediately after its creation in 2014 and then setting an m.room.power_levels event ‘before' the correct initial power_level for the room.

Under normal circumstances this should be impossible because the initial m.room.power_levels for a room should be set before its m.room.join_rules event, meaning users who join the room are subject to its power levels. However, back before we'd even released Synapse, the first two rooms ever created in Matrix (#test:matrix.org and #matrix:matrix.org) were manually created and set the join_rules before the power_levels event, letting users join before the room's power_levels were defined, and so were vulnerable to this attack. We've since re-created #matrix:matrix.org - please re-/join the room if you haven't already!

As a defensive measure, we are releasing a security update of Synapse (0.31.2) today which changes the rules used to authenticate power_level events, such that we fail-safe rather than fail-deadly if the existing auth mechanisms fail. In practice this means changing the default power level required to set state to be 50 rather than 0 if there is no power_levels event present, thus meaning that only the room creator can set the initial power_levels event.

We are not aware of anyone abusing this (other than the old #matrix:matrix.org room) but we'd rather be safe than sorry, so would recommend that everyone upgrade as soon as possible.

This of course constitutes a change to the spec, so full technical details and ongoing discussion around the Matrix Spec Change proposal can be followed over at MSC1304.

EDIT: if you are aware of your server participating in rooms whose first power_levels event is deliberately set by a different user to their creator, please let us know asap (and don't upgrade!)

This work is all part of a general push to finalise and harden and fully specify the Server-Server API as we push towards a long-awaited stable release of Matrix!

As always, you can get the new update from https://github.com/matrix-org/synapse/releases/tag/v0.31.2 or from any of the sources mentioned at https://github.com/matrix-org/synapse.

thanks, and apologies for the inconvenience.

Changes in synapse v0.31.2 (2018-06-14)

SECURITY UPDATE: Prevent unauthorised users from setting state events in a room when there is no m.room.power_levels event in force in the room. (PR #3397)

Discussion around the Matrix Spec change proposal for this change can be followed at https://github.com/matrix-org/matrix-doc/issues/1304.

Introducing Matrix Specification Changes

2018-05-17 — Tech — Matthew Hodgson

Hi all,

We've been able to start investing more time in advancing the Matrix Specification itself over the last month or so thanks to Ben joining the core team (and should be able to accelerate even more with uhoreg joining in a few weeks!)  The first step in the new wave of work has been to provide much better infrastructure for the process of actually evolving the spec - whether that's from changes proposed by the core team or the wider Matrix Community.

So, without further ado, we'd like to introduce https://matrix.org/docs/spec/proposals - a dashboard for all the spec change proposals we've accumulated so far (ignoring most of the ones which have already been merged), as well as a clearer workflow for how everyone can help improve the Matrix spec itself.  Part of this is introducing a formal numbering system - e.g. MSC1228 stands for Matrix Spec Change 1228 (where 1228 is the ID of the Github issue on the github.com/matrix-org/matrix-doc/issues repository that tracks the proposal).

Please note that these are NOT like XEPs or RFCs - i.e. optional proposals or add-ons to the protocol; instead they are literally proposals for changes to the Matrix Spec itself.  Once merged into the spec, they are only of historical interest.

We've also created a new room: #matrix-spec:matrix.org to discuss specific spec proposal changes - please join if you want to track how proposals are evolving! (Conversation is likely to fork off into per-proposal rooms or overflow into #matrix-dev:matrix.org or #matrix-architecture:matrix.org depending on traffic levels, however).

Feedback would be much appreciated on this - so please head over to #matrix-spec:matrix.org and let us know how it feels and how it could do better.

This is also a major step towards properly formalising Matrix.org's governance model - hopefully the changes above are sufficient to improve the health of the evolution of the Spec as we work towards an initial stable release later this year, and then you should expect to see a spec proposal for formal governance once we've (at last!) exited beta :)

Huge thanks to Ben for putting this together, and thanks to everyone who's contributed so far to the spec - we're looking forward to working through the backlog of proposals and turning them all into merged spec PRs!!

Matthew

SECURITY UPDATE: Synapse 0.28.1

2018-05-01 — Tech — Matthew Hodgson

Hi all,

Many people will have noticed disruption in #matrix:matrix.org and #matrix-dev:matrix.org on Sunday, when a validation bug in Synapse was exploited which allowed a malicious event to be inserted into the room with 'depth' value that made the rooms temporarily unusable. Whilst a transient workaround was found at the time (thanks to /dev/ponies, kythyria and Po Shamil for the workaround and to Half-Shot for working on a proposed fix), we're doing an urgent release of Synapse 0.28.1 to provide a temporary solution which will mitigate the attack across all rooms in upgraded servers and un-break affected ones.  Meanwhile we have a full long-term fix on the horizon (hopefully) later this week.

This vulnerability has already been exploited in the wild; please upgrade as soon as possible.

Synapse 0.28.1 is available from https://github.com/matrix-org/synapse/releases/tag/v0.28.1 as normal.

The 'depth' parameter is used primarily as a way for servers to signal the intended cosmetic order of their events within a room (particularly when the room's message graph has gaps in it due to the server being offline, or due to users backfilling old disconnected chunks of conversation). This means that affected rooms may experience message ordering problems until a full long-term fix is provided, which we're working on currently (and tentatively involves no longer trusting 'depth' information from servers).  For full details you can see the proposal documents for the temporary fix in 0.28.1 and the options for the imminent long-term fix.

We'd like to acknowledge jzk for identifying the vulnerability, and Max Dor for providing feedback on the fixes.

As a general reminder, Synapse is still beta (as is the Matrix spec) and the federation API particularly is still being debugged and refined and is pre-r0.0.0. For the benefit of the whole community, please disclose vulnerabilities and exploits responsibly by emailing [email protected] or DMing someone from +matrix:matrix.org. Thanks.

Changes in synapse v0.28.1 (2018-05-01)

SECURITY UPDATE
  • Clamp the allowed values of event depth received over federation to be [0, 2^63 - 1]. This mitigates an attack where malicious events injected with depth = 2^63 - 1 render rooms unusable. Depth is used to determine the cosmetic ordering of events within a room, and so the ordering of events in such a room will default to using stream_ordering rather than depth (topological_ordering). This is a temporary solution to mitigate abuse in the wild, whilst a long solution is being implemented to improve how the depth parameter is used.Full details at https://docs.google.com/document/d/1I3fi2S-XnpO45qrpCsowZv8P8dHcNZ4fsBsbOW7KABI
  • Pin Twisted to <18.4 until we stop using the private _OpenSSLECCurve API.

Security update: Riot/Web 0.13.5 released - fixing XSS vulnerability

2018-02-09 — Tech — Matthew Hodgson

Hi all,

Heads up that we made an emergency release of Riot/Web 0.13.5 a few hours ago to fix a XSS vulnerability found and reported by walle303 - many thanks for disclosing it responsibly.

Please upgrade to Riot/Web 0.13.5 asap. If you're using riot.im/app or riot.im/develop this simply means hitting Refresh; otherwise please upgrade your Riot deployment as soon as possible. Alpine, Debian and Fedora/RPM packages are already updated - huge thanks to the maintainers for the fast turnaround.

The issue lies in the relatively obscure external_url feature, which lets bridges specify a URL for bridged events, letting Riot/Web users link through to the 'original' event (e.g. a twitter URL on a bridged tweet).  The option is hidden in a context menu and labelled "Source URL", and is only visible on events which have the external_url field set.  Unfortunately Riot/Web didn't sanitise the URL correctly, allowing a malicious URL to be injected - and this has been the case since the feature landed in Riot 0.9.0 (Nov 2016).

If you're not able to upgrade to Riot/Web 0.13.5 for some reason, then please do not click on the 'Source URL' feature on the event context menu:

Apologies for the inconvenience,

thanks,

Matthew

PreviousPage 1
NextPage 3