Dendrite development final report (GSoC 2019)

Overview

During GSoC 2019, I participated in and pushed forward the development of Dendrite, a high-performance, scalable Matrix homeserver implementation written in Go.

Like many others passionate about Dendrite, I've been eager to see the day when it finally joins the decentralised, federated Matrix network, so I worked on the project to see how much I could do to bridge the gap between it and the current reference homeserver implementation, Synapse, in terms of feature completeness.

Working on Dendrite in just one area would be insufficient to propel it, so my work covers various components in the project, including the Client/Server API, Sync Server, Room Server, and Federation Component. I also spent time refining the project's documentation, improving its testing/continuous integration process, as well as reviewing pull requests from Matrix.org members and the community.

Below is the major portion of my work presented as pull requests, categorised by the components they belong to; a list of links to all the pull requests I created/reviewed is also available at the end of this report.

Work in Component: Client/Server API

This component is the main handler of HTTP requests from the clients, except for /sync requests - see next part.

Work done:

• Implement client single event retrieval (#693)
• Implement profile retrieval over federation (#726)
• Add missing servers field in /directory/room/:alias response (#732)
• Implement room creation content (#754)
• ...and other fixes and improvements

Work left:

• Implement guest access (#668)
  • This was removed from the plan for GSoC so work with higher priorities according to the milestone could be done first.

Work in Component: Sync Server

The Sync Server is responsible for handling the long polling notification requests (/sync requests) and some other related requests from the clients.

Work done:

• Add EDU support and typing notifications to /sync (#718)
  • EDU stands for Ephemeral Data Units; they are events carrying short-term status in the room, for example typing notifications.
• Implement "full_state" query parameter for /sync (#751)
• ...and other fixes and improvements

Work left:

• Support for other types of EDUs

Work in Component: Room Server

The Room Server is, as its name suggests, where room events, state, etc. are handled.

Work done:

• Implement event redaction (#768)
  • This also includes a relatively large chunk of changes in the Sync Server.
• Fix permission and 404 response for alias deletion (#706)
• ...and other fixes and improvements

Work left:

• Support for alias deletion by admins
• Discussion on and implementation of event retention policies

Work in Component: Federation

The Federation Component sends requests to and receives requests from other Matrix homeservers.

Work done:

• Add joined hosts query APIs (#781)
• (In matrix-org/gomatrixserverlib) Support for more query types in Federation Client: Leave API, Public Rooms API, Profile Query API
• ...and other fixes and improvements

Work left:

• Implementation of a "Most Recent Transaction Sender" In-Memory Cache
  • There is a delay in this part of work because some dependency problems took longer than expected to resolve, but this has been planned and will be worked on after GSoC.
• Implementation of a component that handles backoff/retries for federation requests

General Fixes, Improvements, Maintenance Work

• Data race fixes
  • Fix data races reported by go test -race ./... (#748)
  • Fix data race in clientapi/routing/register.go (#787)
• Fixes for incorrect transaction ID scopes
  • Fix transaction IDs in transaction cache have global scope (#772)
  • Associate transactions with session IDs instead of device IDs (#789)
• Improvements for building/testing process
  • Refine config and docs for sytest (#714)
  • Fix build conflict between docker environment and host (#598)

Links to All Pull Requests

Here is a list of links to all the pull requests I have created/reviewed (by 26 Aug, 2019):

Appendix: On Dendrite Being a GSoC Project

I believe that Dendrite is a pretty special project to work on in the GSoC program.

At its current state, Dendrite isn't a complete homeserver yet. This has left a large portion of work on Dendrite wide open: As I worked on Dendrite, more than often I'd find myself in a place to answer questions related to topics like design or project architecture which, if not thoroughly thought through, might have consequences in the future development of the project. For this reason, I think working on Dendrite involves a set of challenges quite different from what attempting to improve an already functioning piece of software would have presented.

For the same reason, I think what Dendrite has taught me during the summer was less about how to become more fluent in Go or more familiar with various tools, but more about developing the mindset that powers software projects behind the scenes. But this mindset, to myself, seems more valuable than the former, indeed.

And at the end of this report, I'd like to thank my mentors anoa and Brendan, as well as everyone else at Matrix who has answered my questions, discussed with me, and helped me understand the code, and at the same time being super responsive. Without your help, my GSoC experience wouldn't have been this enjoyable! :)