Hi all,

Back in June we wrote about our plans to tighten up data privacy in Matrix after some areas for improvement were brought to our attention. To quickly recap: the primary concern was that the default config for Riot specifies identity servers and integration managers run by New Vector (the company which the original Matrix team set up to build Riot and fund Matrix dev) - and so folks using a standalone homeserver may end up using external services without realising it. There were some other legitimate issues raised too (e.g. contact information should be obfuscated when checking if your contacts are on Matrix; Riot defaulted to using Google for STUN (firewall detection) if no TURN server had been set up on their server; Synapse defaults to using matrix.org as a key notary server).

We’ve been working away at this fairly solidly over the last few months. Some of the simpler items shipped quickly (e.g. Riot/Web had a stupid bug where it kept incorrectly loading the integration manager; Riot/Android wasn’t clear enough about when contact discovery was happening; Riot/Web wasn’t clear enough about the fact device names are publicly visible; etc) - but other bits have turned out to be incredibly time-consuming to get right.

However, we’re in the process today of releasing Synapse 1.4.0 and Riot/Web 1.4.0 (it’s coincidence the version numbers have lined up!) which resolve the majority of the remaining issues. The main changes are as follows:

1. Riot no longer automatically uses identity servers by default. Identity servers are only useful when inviting users by email address, or when discovering whether your contacts are on Matrix. Therefore, we now wait until the user tries to perform one of these operations before explaining that they need an identity server to do so, and we prompt them to select one if they want to proceed. This makes it abundantly clear that the user is connecting to an independent service, and why.

2. Integration Managers and identity servers now have the ability to force users to accept terms of use before using them. This means they can explicitly spell out the data privacy & usage policy of the server as required by GDPR, and it should now be impossible for a user to use these services without realising it. This was particularly fun in the case of identity servers, which previously had no concept of users and so couldn’t track whether users had agreed to their terms & conditions or not… and because homeservers sometimes talk to the identity server on behalf of users rather than the user talking direct, the privacy policy flow gets even hairier. But it’s solved now, and a nice side-effect of this is that users can now explicitly select their Integration Manager in Riot, in case they want to use Dimension or similar rather than the default provided by Modular.

3. Synapse no longer uses identity servers for verifying registrations or verifying password reset. Originally, Synapse made use of the fact that the Identity Service contains email/msisdn verification logic to handle identity verification in general on behalf of the homeserver. However, in retrospect this was a mistake: why should the entity running your identity server have the right to verify password resets or registration details on your homeserver? So, we have moved this logic into Synapse. This means Synapse 1.4.0 requires new configuration for email/msisdn verification to work - please see the upgrade notes for full details.

4. Sydent now supports discovering contacts based on hashed identifiers. MSC2134 specifies entirely new IS APIs for discovering contacts using a hash of their identifier rather than directly exposing the raw identifiers being searched for. This is implemented in Riot/iOS and Riot/Android and should be in the next major release; Riot/Web 1.4.0 has it already.

5. Synapse now warns in its logs if you are using matrix.org as a default trusted key server, in case you wish to use a different server to help discover other servers’ keys.

6. Synapse now garbage collects redacted messages after N days (7 days by default). (It doesn’t yet garbage collect attachments referenced from redacted messages; we’re still working on that).

7. Synapse now deletes account access data (IP addresses and User Agent) after N days (28 days by default) of a device being deleted.

8. Riot warns before falling back to using STUN (and defaults to turn.matrix.org rather than stun.google.com) for firewall discovery (STUN) when placing VoIP calls, and makes it clear that this is an emergency fallback for misconfigured servers which are missing TURN support. (We originally deleted the fallback entirely, but this broke things for too many people, so we’ve kept it but warn instead).

All of this is implemented in Riot/Web 1.4.0 and Synapse 1.4.0. Riot/Web 1.4.0 shipped today (Fri Sept 27th) and we have a release candidate for Synapse 1.4 (1.4.0rc1) today which fully ship on Monday.

For full details please go check out the Riot 1.4.0 and Synapse 1.4.0 blog posts.

Riot/Mobile is following fast behind - most of the above has been implemented and everything should land in the next release. RiotX/Android doesn’t really have any changes to make given it hadn’t yet implemented Identity Service or Integration Manager APIs.

This has involved a surprisingly large amount of spec work; no fewer than 9 new Matrix Spec Changes (MSC) have been required as part of the project. In particular, this results in a massive update to the Identity Service API, which will be released very shortly with the new MSCs. You can see the upcoming changes on the unstable branch and compare with the previous 0.2.1 stable release, as well as checking the detailed MSCs as follows:

• MSC1961: Integration manager authentication APIs
• MSC2078: Sending Third-Party Request Tokens via the Homeserver
• MSC2134: Identity Hash Lookups
• MSC2140: Terms of Service for ISes and IMs
• MSC2229: Allowing 3PID Owners to Rebind
• MSC2230: Store Identity Server in Account Data
• MSC2263: Give homeservers the ability to handle their own 3PID registrations/password resets
• MSC2264: Add an unstable feature flag to MSC2140 for clients to detect support
• MSC2290: Separate Endpoints for Threepid Binding

This said, there is still some work remaining for us to do here. The main things which haven’t made it into this release are:

• Preferring to get server keys from the source server rather than the notary server by default (https://github.com/matrix-org/synapse/pull/6110). This almost made it in, but we need to test it more first - until then, your specified notary server will see roughly what servers your servers are trying to talk to. In future this will be mitigated properly by MSC1228 (removing mxids from events).
• Configurable data retention periods for rooms. We are tantalisingly close with this - https://github.com/matrix-org/synapse/pull/5815 is an implementation that the French Govt deployment is using; we need to port it into mainline Synapse.
• Authenticating access to the media repository - for now, we still rely on media IDs being almost impossible to guess to protect the data rather than authenticating the user.
• Deleting items from the media repository - we still need to hook up deletion APIs.
• Garbage collecting forgotten rooms. If everyone leaves & forgets a room, we should delete it from the DB.
• Communicating erasure requests over federation

We’ll continue to work on these as part of our ongoing maintenance backlog.

Separately to the data privacy concerns, we’ve had a separate wave of feedback regarding how we handle GDPR Data Subject Access Requests (DSARs). Particularly: whether DSAR responses should contain solely the info your have directly keyed by the requesting Matrix ID - or if we should provide all the data “visible” to that ID (i.e. the history of the conversations they’ve been part of). We went and got professional legal advice on this one, and the conclusion is that we should keep our responses to DSARs as tightly scoped as possible. We updated Matrix.org’s privacy policy and DSAR tools to reflect the new legal input.

Finally, it’s really worth calling out the amount of effort that went into this project. Huge huge thanks to everyone involved (given it’s cut across pretty much every project & subteam we have working on the core of Matrix) who have soldiered through the backlog. We’ve been tracking progress using our feature-dashboard tool which summarises Github issues based on labels & issue lifecycle, and for better or worse it’s ended up being the biggest project board we’ve ever had. You can see the live data here (warning, it takes tens of seconds to spider Github to gather the data) - or, for posterity and ease of reference, I’ve included the current issue list below. The issues which are completed have “done” after them; the ones still in progress say “in progress”, and ones which haven’t started yet have nothing. We split the project into 3 phases - phases 1 and 2 represent the items needed to fully solve the privacy concerns, phase 3 is right now a mix of "nice to have" polish and some more speculative items. At this point we’ve effectively finished phase 1 on Synapse & Riot/Web, and Riot/Mobile is following close behind. We're continuing to work on phase 2, and we’ll work through phase 3 (where appropriate) as part of our general maintenance backlog.

I hope this gives suitable visibility on how we’re considering privacy; after all, Matrix is useless as an open communication protocol if the openness comes at the expense of user privacy. We’ll give another update once the remaining straggling issues are closed out; and meanwhile, now the bulk of the privacy work is out of the way on Riot/Web, we can finally get back to implementing the UI E2E cross-signing verification and improving first time user experience.

Thanks for your patience and understanding while we’ve sorted this stuff out; and thanks once again for flying Matrix :)

In the absence of comments on the current blog, please feel free to discuss over at HN, or alternatively come ask stuff in our AMA over at /r/privacy (starting ~5pm GMT+1 (UK) on Friday Sept 27th).

The Privacy Project Dashboard Of Doom

• phase:1
  • vector-im/riot-web
    • 5846 Riot duplicates calls to Scalar (done)
    • 5921 Ability to disable all identity server functionality via the config file (done)
    • 6802 riot shouldn't try to load integrations from an integration manager unless the user has accepted that manager's privacy policies (done)
    • 10088 Prompt to accept integration manager polices on use (done)
    • 10090 Make sure there are no ugly edge cases running Riot without an integrations manager (done)
    • 10091 Decouple setting an email for password reset from publishing your threepid to the identity server and support choice of identity server (on registration) (done)
    • 10092 Prompt to accept Identity Server policies on registration (done)
    • 10093 Prompt to accept identity server policies before inviting them to a room (done)
    • 10094 Store identity server in Account Data and support choosing identity server integration in User Settings (done)
    • 10159 Enable toggling a threepid's association with the current IS in User Settings (done)
    • 10173 VoIP: Stop falling back to Google for STUN (done)
    • 10216 We should make it clear in the UI that device names are publicly readable (done)
    • 10386 Terms modal prompt should appear without a flash of the integration manager portal first (done)
    • 10408 Identity server v2 API authentication (done)
    • 10423 Deploy a develop environment for t's and c's flows for IM and IS (done)
    • 10424 Remove the bind true flag from 3PID calls on registration (done)
    • 10425 Ability to disconnect from the identity server by pressing buttons in user settings. (done)
    • 10452 Test IS v2 access tokens for validity (done)
    • 10497 Integration manager terms hides terms that need signing (done)
    • 10510 Remove the bind true flag from 3PID adds in settings (done)
    • 10519 Update discovery and account 3PID sections when either changes (done)
    • 10522 Handle terms agreement in the Discovery section of settings (done)
    • 10525 IS interactions proxied through the HS also need terms agreement (done)
    • 10528 don't try & show threepids in discovery section if we don't have an ID server (done)
    • 10539 Inline terms agreement on change IS and IM (done)
    • 10540 When using Riot without an IS, identity requests are sent to server hosting Riot (done)
    • 10546 Prompt for fallback VOIP should only happen when the user tries to make a call (done)
    • 10550 warn when disconnecting ID server if there are any current bindings (done)
    • 10552 Allow email registration, password reset & adding 3pids when no IS (done)
    • 10553 Remove the ability to set an IS at login/registration (done)
    • 10556 Use the hashed v2 lookup API for 3PIDs (done)
    • 10561 Scalar should serve a .well-known file (done)
    • 10566 Use nicer APIs for toggling a 3PID's shared status (done)
    • 10571 Allow email registration when no IS (done)
    • 10572 Allow password reset when no IS (done)
    • 10573 Allow adding 3pids when no IS (done)
    • 10574 Placeholder issue for MSCs that should land prior to the privacy release (done)
    • 10593 Placeholder issue for privacy migration path (done)
    • 10597 Ensure IS in account data is read / written as specced (done)
    • 10599 Send IS for adding MSISDNs only when required by HS (done)
    • 10603 Getting a terms prompt when inviting an email address clears the address picker (done)
    • 10619 Handle the case of no IS in features that require IS to lookup (done)
    • 10620 Change auth flows so that you default to no IS (done)
    • 10634 Changing to vector.im IS in Settings fails because /terms 404s (done)
    • 10635 Inline terms agreement in Discovery should still show change and do not use (done)
    • 10636 The UX is a little confusing if you haven't accepted the terms of the default identity server. (done)
    • 10637 Unable to add email address to HS account without IS set (done)
    • 10660 Sydent implementation: Unauthed v1, clone all endpoints, auth v2 (done)
    • 10666 In login/register, riot freezes if you go to Advanced, remove the IS url and click next (done)
    • 10669 Checking email invite account is unclear when no IS set (done)
    • 10674 Email help text on registration should be updated without binding (done)
    • 10677 Change text about other users being able to invite you by email on registration page (done)
    • 10744 Only set terms URLs in the account when they change (done)
    • 10749 Changing from one IS to another does not trigger bound 3PID warning (done)
    • 10750 Bound 3PIDs warning should better explain what's being left behind (done)
    • 10754 Lowercase emails during IS lookup calls (done)
    • 10755 Terms account data is meant to be additive, but currently sets only new URLs (done)
    • 10756 After changing IS to termstest.matrix.org the text field isn't cleared and the button remains active (done)
    • 10757 After changing IS back to vector.im the text field isn't cleared and the button remains active (done)
    • 10758 Revoke may be too vague for unbinding a 3PID (done)
    • 10779 When publishing a threepid to an IS, if you click 'continue' before clicking the link in your email, we delete your threepid from the HS. (done)
    • 10800 Community invite should not talk about ISes at all. (done)
    • 10823 IS with unagreed terms will block you from adding even unshared 3PIDs to your HS (done)
    • 10839 Use the new backend APIs for adding-3pid-to-homeserver and binding-3pid-on-identity-server when they exist. (done)
    • 10878 UX regression when trying to invite by email if no IS is configured (done)
    • 10883 Riot should check for r0.6.0 server support alongside feature flags (done)
    • 10923 Send validation tokens to submit_url from HS (done)
    • 10933 Do not include ID server or access token when HS supports MSC2290 (done)
    • 10939 Registration with MSISDN needs to use submit_url (done)
    • 10941 threepid_creds for registration and password reset still include id_server (done)
    • 10959 Remove id_server from email threepid_creds (done)
    • 10604 QA: Re-test ALL of the identity server interactions (in progress)
    • 10958 Terms of Service nitpicks (in progress)
    • 10704 We treat an empty string identity server as not having one
  • vector-im/riot-ios
    • 2518 Make sure there are no ugly edge cases running Riot without an integrations manager (done)
    • 2521 Privacy: Improve wording on contact upload UI (done)
    • 2532 VoIP: Stop falling back to Google for STUN (done)
    • 2600 Prompt to accept integration manager polices on use (done)
    • 2601 Decouple setting an email for password reset from publishing your threepid to the identity server and support choice of identity server (on registration) (done)
    • 2602 Prompt to accept identity server policies before inviting them to a room (done)
    • 2603 Identity server v2 API authentication (done)
    • 2606 Settings: Add a Discovery section (done)
    • 2608 Registration: Update consents flow (done)
    • 2643 Ability to disable all identity server functionality via the config file (done)
    • 2646 VoIP: Fallback to matrix.org STUN server with a confirmation dialog (done)
    • 2647 MXRestClient: Move IS API to its own (done)
    • 2648 Remove the bind true flag from 3PID calls on registration (done)
    • 2650 Remove the bind true flag from 3PID adds in settings (done)
    • 2651 Store identity server in Account Data and support choosing identity server integration in User Settings (done)
    • 2652 Use the hashed v2 lookup API for 3PIDs (done)
    • 2653 Allow email registration, password reset & adding 3pids when no IS (done)
    • 2654 Use nicer APIs for toggling a 3PID's shared status (done)
    • 2657 Allow email registration when no IS (done)
    • 2658 Allow password reset when no IS (done)
    • 2659 Allow adding 3pids when no IS (done)
    • 2660 Handle terms agreement in the Discovery section of settings (done)
    • 2661 Remove the ability to set an IS at login/registration (done)
    • 2662 We should make it clear in the UI that device names are publicly readable (done)
    • 2664 Placeholder issue for privacy migration path (done)
    • 2665 Ensure IS in account data is read / written as specced (done)
    • 2672 Handle the case of no IS in features that require IS to lookup (done)
    • 2675 Email help text on registration should be updated without binding (done)
    • 2682 Send IS for adding MSISDNs only when required by HS (done)
    • 2686 Use wellknown to discover the IS of a custom HS (done)
    • 2696 Lowercase emails during IS lookup calls (done)
    • 2704 Use id_access_token in CS API when required (done)
    • 2604 Settings: Ability to change the identity server (in progress)
    • 2649 Ability to disconnect from the identity server by pressing buttons in user settings (in progress)
    • 2713 Use the new backend APIs for adding-3pid-to-homeserver and binding-3pid-on-identity-server when they exist. (in progress)
    • 2718 Riot should check for r0.6.0 server support alongside feature flags (in progress)
    • 2683 Checking email invite account is unclear when no IS set
    • 2731 Send validation tokens to submit_url from HS
    • 2732 Do not include ID server or access token when HS supports MSC2290
    • 2736 IS terms cannot be accepted anymore
  • vector-im/riot-android
    • 3223 VoIP: Stop falling back to Google for STUN (done)
    • 3224 Make sure there are no ugly edge cases running Riot without an integrations manager (done)
    • 3225 Prompt to accept integration manager polices on use (done)
    • 3226 Decouple setting an email for password reset from publishing your threepid to the identity server and support choice of identity server (on registration) (done)
    • 3227 Prompt to accept identity server policies before inviting them to a room (done)
    • 3228 Identity server v2 API authentication (done)
    • 3229 Settings: Ability to change the identity server (done)
    • 3230 Settings: Add a Discovery section (done)
    • 3231 Registration: Update consents flow (done)
    • 3252 Remove the bind true flag from 3PID calls on registration (done)
    • 3253 Ability to disconnect from the identity server by pressing buttons in user settings (done)
    • 3254 Remove the bind true flag from 3PID adds in settings (done)
    • 3256 Store identity server in Account Data and support choosing identity server integration in User Settings (done)
    • 3257 Use the hashed v2 lookup API for 3PIDs (done)
    • 3258 Allow email registration, password reset & adding 3pids when no IS (done)
    • 3260 Allow email registration when no IS (done)
    • 3261 Allow password reset when no IS (done)
    • 3262 Allow adding 3pids when no IS (done)
    • 3263 Handle terms agreement in the Discovery section of settings (done)
    • 3264 Remove the ability to set an IS at login/registration (done)
    • 3265 We should make it clear in the UI that device names are publicly readable (done)
    • 3268 Placeholder issue for privacy migration path (done)
    • 3269 Ensure IS in account data is read / written as specced (done)
    • 3277 Handle the case of no IS in features that require IS to lookup (done)
    • 3278 Email help text on registration should be updated without binding (done)
    • 3281 Send IS for adding MSISDNs only when required by HS (done)
    • 3283 Use wellknown to discover the IS of a custom HS (done)
    • 3289 Lowercase emails during IS lookup calls (done)
    • 3295 Use id_access_token in CS API when required (done)
    • 3300 Use the new backend APIs for adding-3pid-to-homeserver and binding-3pid-on-identity-server when they exist. (done)
    • 3312 Riot should check for r0.6.0 server support alongside feature flags (done)
    • 3316 Implement MSC2290 (done)
    • 3324 id_server param sent when registering an email on server that does not requires one (done)
    • 3326 Discovery Settings / infinite loading if terms not signed on existing IS (done)
    • 3327 vector.im is seen as having no terms, but it does (done)
    • 3251 VoIP: Fallback to matrix.org STUN server with a confirmation dialog (in progress)
    • 3248 Ability to disable all identity server functionality via the config file
    • 3318 Send validation tokens to submit_url from HS
    • 3322 Do not include ID server or access token when HS supports MSC2290
  • vector-im/riotX-android
    • 490 Allow email registration, password reset & adding 3pids when no IS (done)
    • 432 Prompt to accept integration manager polices on use (in progress)
    • 431 Make sure there are no ugly edge cases running Riot without an integrations manager
    • 433 Decouple setting an email for password reset from publishing your threepid to the identity server and support choice of identity server (on registration)
    • 434 Prompt to accept identity server policies before inviting them to a room
    • 435 Identity server v2 API authentication
    • 436 Settings: Ability to change the identity server
    • 438 Settings: Add a Discovery section
    • 439 Registration: Update consents flow
    • 489 Use the hashed v2 lookup API for 3PIDs
  • matrix-org/matrix-doc
    • 1961 MSC1961: Integration manager authentication APIs (done)
    • 2078 MSC2078: Sending Third-Party Request Tokens via the Homeserver (done)
    • 2130 Identity service should do lookups based on hashed 3PIDs, not plaintext ones. (done)
    • 2134 MSC2134: Identity Hash Lookups (done)
    • 2139 Method for ISes and Integration managers to provide terms of service (done)
    • 2140 MSC2140: Terms of Service for ISes and IMs (done)
    • 2229 MSC2229: Allowing 3PID Owners to Rebind (done)
    • 2230 MSC2230: Store Identity Server in Account Data (done)
    • 2263 MSC2263: Give homeservers the ability to handle their own 3PID registrations/password resets (done)
    • 2264 MSC2264: Add an unstable feature flag to MSC2140 for clients to detect support (done)
    • 2290 MSC2290: Separate Endpoints for Threepid Binding (in progress)
  • matrix-org/sydent
    • 178 Support for MSC2140 (done)
    • 179 Support testing with matrix-is-tester (done)
    • 180 Support for MSC2140 (done)
    • 181 Deny bindings to anything other than the mxid you're authed as (done)
    • 184 Hash 3PID lookups (done)
    • 186 Don't fail the unbind request if the binding doesn't exist (done)
    • 187 Add more tweaks to terms branch (done)
    • 189 Redact looked-up threepids from the application logs. (done)
    • 191 Add OpenID auth to 3PID hash lookup (done)
    • 197 Have 'mappings' contain medium as well as address (done)
    • 198 Throw exception on bad args & catch in wrapper (done)
    • 199 Return algorithm, lookup_pepper in M_INVALID_PEPPER response (done)
    • 200 Support MSC2140 register/terms endpoints (done)
    • 201 Port v1 endpoints to v2 (done)
    • 202 Deny bindings to anything other than the mxid you're authed as (done)
    • 204 Fix the signing servlet (done)
    • 205 Correct API v1 path check in get_args (done)
    • 206 Fix terms POST (done)
    • 207 Deploy policy file (done)
    • 208 Remove PII logging from log-level INFO (done)
    • 213 Delete threepid assocs on unbind and remove existing with migration (done)
    • 217 Allow non-matrix.org MXIDs via v2 only once terms are enabled (done)
    • 218 Unbind fails to remove the binding on matrix.org IS (done)
    • 220 Unbind fails to work for bindings where it was attempted with #213 (done)
    • 192 Sydent doesn't delete 3PIDs from DB on unbind (in progress)
  • matrix-org/synapse
    • 1287 We need to actually censor redactions from the DB (SYN-284) (done)
    • 5751 Make homeservers able to handle registration-with-email without depending on an Identity Server (done)
    • 5786 Support IS v2 API with authentication for requests proxied to the IS (done)
    • 5827 Add 3PID unbind API for MSC2140 (done)
    • 5835 [1/2] Allow homeservers to send registration emails | Sending the email (done)
    • 5861 Use the v2 lookup API for 3PID invites (done)
    • 5862 Allow account owners to rebind 3PIDs as in MSC2229 (done)
    • 5868 Add flag in /versions for whether clients should send id_server params (done)
    • 5874 Placeholder issue for privacy migration path (done)
    • 5875 Remove trusted_third_party_id_servers functionality (done)
    • 5876 Replace trust_identity_server_for_password_resets with account_threepid_delegate (done)
    • 5892 Switch to using v2 Identity Service APIs other than lookup (MSC 2140) (done)
    • 5897 Use the v2 lookup API for 3PID invites (done)
    • 5927 Add a m.id_access_token flag to unstable_features (done)
    • 5928 Split account_threepid_handler into a msisdn and email versions (done)
    • 5929 Use account_threepid_delegate for sending SMS (done)
    • 5930 Add m.id_access_token flag (done)
    • 5934 Censor redactions in DB after a month (done)
    • 5935 Use federation blacklist for requests to identity servers (done)
    • 5937 Revert "Use the v2 lookup API for 3PID invites" (done)
    • 5940 [2/2] Allow homeservers to send registration emails | Accepting the verification (done)
    • 5945 Revert "Add m.id_access_token flag" (done)
    • 5948 Support v2 Identity Server APIs (done)
    • 5964 Remove bind_email and bind_msisdn (done)
    • 5968 Set m.require_identity_server to always be False (done)
    • 5969 Change account_threepid_delegate to a dictionary (done)
    • 5972 Add m.require_identity_server to /versions unstable_flags (done)
    • 5974 Add m.id_access_token to /versions unstable_features (MSC2264) (done)
    • 5976 Use the v2 Identity Service API for lookups (MSC2134 + MSC2140) (done)
    • 5979 v2 3PID Invites (part of MSC2140) (done)
    • 5980 Add POST /_matrix/client/r0/account/3pid/unbind (MSC2140) (done)
    • 5987 Allow Synapse to send registration emails + choose Synapse or an external server to handle 3pid validation (done)
    • 5996 Allow users to rebind 3pids they own (MSC2229) (done)
    • 6000 Use the federation blacklist for requests to untrusted Identity Servers (done)
    • 6011 Use account_threepid_delegate for 3pid validation (done)
    • 6013 Fix existing v2 identity server calls (MSC2140) (done)
    • 6042 Allow HS to send emails when adding an email to the HS (done)
    • 6043 Implement MSC2290 (done)
    • 6044 Add an unstable feature flag for separate add/bind 3pid APIs (done)
    • 6062 Use unstable prefix for 3PID unbind API (done)
    • 6067 Drop support for bind param on POST /account/3pid (MSC2290) (done)
    • 6076 Synapse doesn't give clients a submit_url on requestToken breaking msisdn adding (done)
    • 6078 Add POST submit_token endpoint for MSISDN (done)
    • 6079 Add submit_url response parameter to msisdn /requestToken (done)
    • 6087 Remove hardcoded defaults of matrix.org and vector.im in configuration (done)
    • 6090 Explicitly log when a homeserver does not have a trusted key server configured (done)
    • 6096 Riot Web expects the email validation next_link to have the sid appended (done)
    • 6100 Remove email from registration flows if it’s unsupported (done)
    • 6103 _check_threepid in auth.py incorrect for MSISDN (done)
    • 6045 Support Client-Server API r0.6.0 (in progress)
    • 1263 When we redact events, any mxc content they refer to should be redacted too (SYN-216)
    • 4565 Metadata resistance.
    • 6086 Fetch signing-keys directly from servers before falling back to the trusted_key_servers
• phase:2
  • vector-im/riot-web
    • 4913 Option for homeservers to set a default integration manager (done)
    • 10161 Store Integration Manager preferences in account data and allow user to change them somewhere sensible (done)
    • 10967 Disconnect from identity server fails (done)
    • 10443 Remove v1 IS API fallbacks once servers are updated
    • 10557 Prompt users each time before sending data to an Identity Server that doesn't have a terms of service (unless you have actively set that IS in your account data).
    • 10696 Allow users to disconnect from an integration manager entirely in the same way that we support doing this for identity servers
    • 10909 Unable to disconnect from IS if the server is unavailable
    • 10936 Refine submit_url handling to only activate with separate add and bind
  • vector-im/riot-ios
    • 2711 Provide a better UX for users considering sharing their contact list with an IS to discover people they know already using Matrix
  • vector-im/riot-android
    • 3282 Checking email invite account is unclear when no IS set
    • 3323 Post MSC2290 bug / Clicking on the validation mail link fails registration
  • matrix-org/matrix-doc
    • 1957 MSC1957: Integration manager discovery (done)
  • matrix-org/sydent
    • 196 Remove the lookup_hash field from local_threepid_associations (in progress)
  • matrix-org/synapse
    • 5881 Remove trust_identity_server_for_password_resets and account_threepid_delegate options
• phase:3
  • vector-im/riot-web
    • 10563 Add better domain for turn.matrix.org use as STUN server (done)
    • 10087 Prompt to accept integration manager policies on registration
    • 10167 Present an aggregated terms of service dialogue at registration if possible
    • 10168 If a user has disabled the identity server integration on their account, we should make invite user handle this nicely
    • 10401 Invite new users to publish their threepids to the identity server
    • 10422 Use more contextual prompt for integration manager polices on use
    • 10453 Log out from IM on Riot log out and IM removal
    • 10455 Log out from IS on Riot log out and IS removal
    • 10487 Store the date of users having read and accepted the IM policies in the IM db
    • 10488 Store the date of users having read and accepted the IS policies in the IS db
    • 10498 Terms test scalar requires the legacy ?v query param on the new account route
    • 10575 We should show the 'must configure TURN' warning when a call fails, even after using the fallback turn.matrix.org
    • 10615 Change all IS access token APIs to use getIdentityAccessToken only
    • 10671 riot submits sign-ed25519 requests as POST requests with params in query string and empty body
    • 10746 /invite doesn't force terms with older homeservers
    • 10830 Show IS policy links in user settings somewhere.
    • 10950 Unhelpful 400 error when adding MSISDN and server doesn’t delegate
  • vector-im/riot-ios
    • 2710 Show IS policy links in user settings somewhere.
  • matrix-org/matrix-doc
    • 447 We need a way to be able to expire data from a HS. (SPEC-141)
  • matrix-org/synapse
    • 5830 pushers table contains user device names, which may include user real names

Dept of Status of Matrix 🌡

Matrix AMA on /r/privacy happening this weekend

Go check out https://www.reddit.com/r/privacy/comments/da219t/im_project_lead_for_matrixorg_the_open_protocol/ and join in asking questions!

First Librem 5 Smartphones are Shipping

Might be interesting to readers: the phone with Matrix at its core is starting to ship: https://puri.sm/posts/first-librem-5-smartphones-are-shipping/.

Fork Awesome includes Matrix icon

Fork Awesome now include the [ m ] as an icon! View it at https://forkaweso.me/Fork-Awesome/icon/matrix-org/

Dept of Spec 📜

Following on from last week, the 3 MSCs the spec core team have chosen to focus on this week are: 2290 (3pid binding endpoints), 2176 (redaction rules), and 1219 (key backups), spurred on by finishing off the spec work for the privacy sprint, and cross signing.

MSC Updates:

News from 2019-09-20 09:00:00 until 2019-09-27 20:07:55.

Merged MSCs

No MSCs have been merged this week.

Final Comment Period

No MSCs have entered FCP this week.

New MSCs

• [WIP] MSC2306: Removing MSISDN password resets
• MSC2299: Proposal to add m.textfile msgtype
• MSC2300: /ping endpoint
• MSC2301: server info endpoint

Dept of Servers 🏢

Synapse v1.4.0rc1

Neil told us:

This week we put out a release candidate for 1.4.0 which support a whole host of privacy features including greater control over interactions with identity servers, cleaning up redacted events and user meta data (like IPs and user agents) and warning users when they are using the default trusted key server (matrix.org).

Aside from privacy, the thing that is most exciting is switching on our solution for mitigating forward extremities build up by default. It should make a big difference for the CPU of servers in fragmented rooms.

matrix-corporal v1.6.0

Slavi reported:

matrix-corporal v1.6.0 was recently released to address an issue when used in conjunction with Synapse Single-Sign-On login flows (CAS or SAML).

Until now, matrix-corporal used to interfere with /login requests and demand that users always authenticate with a username/password. Since v1.6.0, you can use matrix-corporal for automatic management of users/rooms/communities, while letting authentication happen through SSO (as provided by Synapse).

Dept of Bridges 🌉

famedly-email-bridge

sorunome offered:

famedly-email-bridge should be fully working now! Not only is there a readme now, but also a share of other changes:

• whitelist/blacklist who may use the bridge
• threaded conversation: rooms in which you just send messages as if to chat and the bridge bundles them into emails and sends them off
• invite ghosts to a room to start a threaded conversation
• incoming emails in email rooms have a link to reply to, starting a threaded conversation
• email templates for sending emails: add a header or footer, if you want
• sanitize incoming HTML to make sure it is only matrix' subset

New Twilio bridge: mautrix-twilio

Tulir told us:

https://github.com/tulir/mautrix-twilio / #twilio:maunium.net is now a thing. Message (+formatting) and media bridging works. Unlike all my other bridges, this one is a relaybot-style bridge.

I'm also working on a maubot that accepts invites, announces them in a control room and then invites people from that control room when requested. That bot should be ready NWIM.

matrix-appservice-slack 1.0.0-rc3

Half-Shot reported:

The slack bridge got an RC3 today and it is now LIVE on matrix.org!! This brings in a whole host of new changes like speedups for message processing, typing notifications and more reliable edits/reactions/replies. Since the 1.0 release required a migration of data files, we have made every effort to migrate integrations across. If you find that your slack bridge (hosted on matrix.org) is no longer working, please contact me

matrix-appservice-irc 0.13.0

Half-Shot told us:

Hi folks, the IRC bridge has finally gotten its 0.13.0 release this morning. https://github.com/matrix-org/matrix-appservice-irc/releases contains all the juicy details

Bifrost resumed

Half-Shot offered:

Work on Bifrost has resumed! We're doing our best to refactor and replace bits that were hacked together at the start of the year and really improve on reliability and documentation. The official matrix.org bridge awaiting our work on PostgreSQL before we can move it further, but the project is accelerating :) https://github.com/matrix-org/matrix-bifrost/tree/develop

Dept of Clients 📱

"riot-vim"

maze reported:

Hello. I made a thing for using the Vim text editor for sending messages in Riot, and a friend suggested I share it here. Here it is: https://gitlab.com/MRAAGH/riot-vim#riot-vim

This thing is bananas - take a look at the gif.

New client library for Elixir: polyjuice_client

uhoreg said:

I have extracted Igor's guts, and have distilled them into a new client library for Elixir: polyjuice_client. I've also started working on a library of Matrix utility functions for Elixir that would be relevant to multiple components (clients, application services, homeservers, identity servers): polyjuice_util.

Continuum 0.9.25

yuforia told us:

Continuum, desktop client written in Kotlin, version 0.9.25:

• Change text color of selected item in room list for higher contrast.

https://matrix.org/_matrix/media/r0/download/matrix.org/ubsrqTgTUzbzklPGnTbRfqQr

koma, the library behind Continuum:

• Force new TCP connection when a SocketTimeoutException occurs in a pooled connection to fix recurrent timeout errors caused by connection reuse.

Fractal 4.2.1

Alexandre Franke reported:

Fractal 4.2.1 got released, with a bunch of updated translations, a crasher fix and a couple of other bug fixes.

Riot Android

benoit told us:

the privacy work is in review. The release will be done soon

RiotX

benoit announced:

RiotX: A big work to stabilize the application and to implement little missing feature has been done. Also Two releases has been done this week. Please refer the the changelog for a (rather) complete list of what has been done (https://github.com/vector-im/riotX-android/blob/develop/CHANGES.md) Also the read marker feature is in review. There are still remaining bugs which will be fixed before the merge.

RiotX is really coming along, please try it out.

Dept of Ops 🛠

New Synapse Docker Image

Black Hat reported:

I made a docker image for Synapse (again). However, this time it uses PyPy3. It is a drop-in replacement for matrixdotorg/synapse. Anyone is welcomed to test the impact of it on CPU utilization.

OpenSAPS

Stanislav told us:

OpenSAPS (Open Slack APi Server, https://gitlab.com/pztrn/opensaps) is now provides Docker container for ease of use. Just mount /app/opensaps.yaml and you're set. Registry is reachable at https://gitlab.com/pztrn/opensaps/container_registry

Dept of Bots 🤖

matrix-nio project template

anoa offered:

I made a template repository for creating Matrix bots with poljar's matrix-nio: https://github.com/anoadragon453/nio-template! It also has a room: #nio-template:amorgan.xyz.

If you've ever wanted to make a Matrix bot with python, this repository can help you get started.

matrix-fly-paper is now "matrix-fly-swatter", has new scope

serra-allgood offered:

The matrix-fly-paper bot has been renamed to matrix-fly-swatter. On reflection, I realized the original goals for the bot were too ambitious and the project has become an exercise in becoming familiar with the client<->server api. The planned features have been cut back to simply automating sending m.room.server_acl events to several rooms at once. At a later date, planned features may be expanded, but there are other projects I'd have more fun working on in the meantime.

Dept of Ping 🏓

It's the section where we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Overview

*libQMatrixClient was renamed to libQuotient

The project aimed at adding end-to-end encryption to libQuotient for future support in Qt/libQuotient-based clients like Quaternion and Spectal, or even Telepathy IM.

The work done

Regarding my initial proposal, I’ve completed tasks enough for message receiving, and not finished tasks related to message sending.

Focused on libQuotient, I've also added changes to Quaternion client to support the proposed API and test the results. I've reused libQtOlm, which is a Qt wrapper to the matrix olm library and contributed to provide better compatibility during its building and deployment. This also required to dive into the olm library itself and provide minor patch for the olm CMake files too.

So, the basic structure of the project changed a bit, libQtOlm was added as a dependency to support libolm:

                              +--------------------------------------------+
                              |   Quaternion/Spectral/Telepathy|Tank/etc   |
+------------------+          +--------------------------------------------+
|      CS API      |  <---->  |                  libQuotient               |
+------------------+          +-------------------------------+------------+
|Synapse homeserver|          |               Qt              |  libQtOlm  |
+------------------+          +--------------------------------------------+
                                                              |   libolm   |
                                                              +------------+

During the coding period, I've resorted to the specification, the guide and the last year GSoC python implementation actively. And added minor fix for the documentation about names constants at the documentation examples.

Demonstration

Future work

Talking about future work, the status is going to be captured at libQuotient project board. Next steps are:

• Managing devices list for users in the room
• Sending encrypted messages

While olm account management architecture and device_one_time_keys_count sync data handling is here, such tasks as session management after a restart and device verification still requires additional efforts. After that encrypted attachments support and key backups could also be added.

I'm going to take care about that, but it definitely will be harder as just a side project. I still have enthusiasm though :)

Conclusion

To take part in the Google Summer of Code project was my dream. I'm very happy I've managed to took part this year, since it was the last year of my study, while I had doubts about my personal results until I've received final confirmation and the review.

From the beginning of the project, I've realized I'm very lucky since I've got all the chances to provide perfect results. @Kaffeine helped me a lot with my TelepathyIM contribution. He helped me to evolve my CMake, git and Qt/C++ skills and that's how I've started contributing to the libQuotient before GSoC. After that, it turns out that I've accepted and received even two mentors. @uhoreg from the Matrix team, who helped me with End-to-End Encryption logic and olm/megolm understanding. And @kitsune from the Quotient project, who helped me a lot with the code review, with the architecture decisions and the library logic, and even with the time management (he was the one who watched carefully about my results). The last year GSoC python implementation and guide from @Zil0 were also here, and it turns out that Spectral developer @BHat provided QtOlm wrapper right before GSoC stated.

However, planning and updating the plan were my weak points, where I've made key mistakes, such as poor combination with my regular part-time job. And I definitely should have reserved a small vacation at least for the final period of the project to handle tasks better.

In the end, I've managed to debug the mistakes and provided encrypted messages receiving that could already be used at the clients. Also, I evolved my skills and dived into the megolm E2EE subject. I'm willing to continue my contribution to develop libQuotient as full-featured Qt-based Matrix library.

In general, I am not disappointed. I'm wishing luck to all the future students who are reading this. I'm happy to receive support and contribute to an international open project not only for myself, but also for the other developers and users.

Appendix: Additional links

GSoC project page: link

Mentors: Alexey Rusakov, Hubert Chathi

GSoC Matrix room: link

Contribution: libQuotient GSoC E2EE PRs list, full final report at the project wiki

Matrix Live 🎙

I spoke to Half-Shot about LOTS of topics in bridging, well worth a watch if you're interested in using Slack, IRC and other platforms with Matrix.

Dept of Spec 📜

Approved MSCs

No MSCs have cleared FCP this week.

Final Comment Period

No MSCs have entered FCP this week

In Progress MSCs

• MSC2291: Configuration to Control Crawling

The Spec Core Team is going to try and select 3 MSCs to prioritise per week to work on, in order to not stretch resources too thin and to communicate clearly what it is we're actually up to. More details to follow.

Otherwise this week has been continuing to iterate on existing MSCs, as well as updating the MSC template to rename the "Tradeoffs" section to "Alternatives", and to get rid of "Conclusion" which was always just an unnecessary repeat of the introduction.

Dept of Servers 🏢

Construct

Notes from Jason:

Construct did a round of performance work and pursued some of the easier opportunities with the most effective returns. At present, the biggest performance bottleneck Construct experiences is with I/O, specifically latency from accessing disks in a random pattern. This week a few open opportunities for prefetch pipelines were taken, allowing the server to stream data directly from the disk to the client with minimal buffering. This was implemented in several places, mostly for the /media/ system, and also for the /members lists, improving the performance of requesting the members of a room like #whalepool:ericmartindale.com .

There is one area though that requires a special mention for being an order of magnitude more impactful than any other area where a pipelined stream was placed. We have analyzed the way Riot conducts a room change, and by the simple placement of a prefetch operation when the server receives an im.vector.breadcrumb_rooms, which comes early during the room change, Construct is able to populate the room's timeline significantly faster. That has a compounding effect, as the browser initiates several requests based on (and after) the received timeline data- meaning those responses come significantly faster.

Construct is the community driven Matrix server written in C++ for maximum performance. Swing by #test:zemos.net or https://github.com/matrix-construct/construct/ for more.

Synapse

Neil said:

Synapse This week, lots of hacking but not much in the way of news. We’ve been focusing on more privacy improvements as well as polishing up the SAML integration. Next week, expect a wodge of privacy related features to land.

Dept of Bridges 🌉

Matrix Slack bridge 1.0 RC2

Half-Shot said:

The Matrix Slack bridge has had another release candidate this week. 1.0 RC2 is out now. We've rewritten the codebase in Typescript and made many changes, and need your feedback before release so please do not hold back on testing it! The 1.0 release is slated to arrive sometime next week :)

famedly-email-bridge - new email bridge!!

sorunome told us:

Presenting famedly-email-bridge, a new email bridge! It is still early in development but can already send and receive emails.

Features so far:

• Receive emails by a prefix (e.g. [email protected], [email protected])
• Send emails with said prefix
• Receiving emails parses plain body, html body and attachments
• Sending emails collects all messages for a given amount of time. After that it creates an email based on that! Supports edits, redactions and files (image, video, audio, file)!

Things planned for the future:

• Route emails to channels based on threading (reply-chain)
• Send emails by making a new channel and inviting an email ghost and just sending messages!

IRC bridge progress: postgres

Half-Shot told us:

My other news that I forgot to mention is that I have a branch of the IRC bridge that runs off postgres, rather than flat file storage. It's much less memory/cpu intensive and might even offer a bit of a speedup :)

For ref: The current room/user store files for the freenode bridge take up 250MB when stored to disk. The existing storage system works on the principle of loading the whole datastore into memory and periodically saving to disk. Hopefully by not requiring the bridge to load all the state into memory, there are savings to be made.

mautrix-telegram v0.6.1

Tulir told us:

mautrix-telegram v0.6.1 has been released, no changes since the release candidates two weeks ago. mautrix-telegram v0.7.0 is scheduled to be released within 6 weeks.

Also, there's a WhatsApp business API bridge using Twilio in development, though I'm guessing most people won't have any use for that.

Dept of Clients 📱

Continuum progress

yuforia said:

Continuum desktop client written in pure Kotlin, version 0.9.24:

• Non-square avatar images are scaled up and cropped to fit a square, instead of being scaled down and put into a square.
• Scale avatar images when font size is increased/decreased with Ctrl and +/-. Below is a screenshot of Continuum with UI scaled to be 67% larger.
• After signing-in, open the messaging view each time it's launched instead of showing the login view and requiring one extra click.
• Shutdown background threads and close database so it won't be busy if Continuum is closed and immediately re-launched.

https://matrix.org/_matrix/media/r0/download/matrix.org/yLOmalevIOxrKATqZVjsmfvr

Riot Web

Ryan offered:

Riot Web worked on the last remaining bits of the current privacy sprint. We are planning to do a release soon that includes this privacy work from the last few months. We are also working on a few improvements for first time users around the create room dialog. We also shipped several patch releases (1.3.5 and 1.3.6) to fix high priority bugs.

Riot-iOS

Manu said:

Riot-iOS has made some progress on the privacy sprint. We fixed some major issues with VoiceOver. Those fixes will be released soon in a hot fix release: 0.9.5.

Riot Android

Benoit:

Riot Android: Still working on privacy

RiotX

Benoit reported:

A release has been done (0.5.0) with login with SSO support. There is also a new "no network" indicator, which is a bit bugged, we will fix that. Stabilization sprint is still ongoing. We already have fixed quite a lot of issues. François is still working on the read marker and is also fixing issues related to the permalinks.

krombel reported:

It might be interesting to some of you that the F-Droid repo for RiotX is finally working again. There was an issue with buildkite which in the end led to the situation that the built apk had a lower versionCode as the apps already in the repo. The repo was then "intelligent enough" to no offer it as update. benoit was able to resolve that so the repos are updating again.

Dept of Ping 🏓

New!! A new section where we will reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

Final thoughts 💭

Slack have invented bridging, very creative of them.

Nico has reappeared, on the tropical island Reunion and is looking again at his Elixir Homeserver project: Plasma.

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Matrix Live 🎙

Dept of Spec 📜

Approved MSCs

• MSC1802: Remove the '200' value from some federation responses

Final Comment Period

No MSCs have entered FCP

In Progress MSCs

• MSC2290: Separate Endpoints for Threepid Binding
• MSC2260: Update the auth rules for m.room.aliases events

Dept of Status of Matrix 🌡

Mozilla trialling Matrix as an IRC replacement

In fact they're having a standoff between Matrix, Mattermost, Rocket.chat and Slack.

We're eager to see the results of this trial, the feedback will be very valuable for everyone regardless of the final outcome. (Though of course, we are optimistic..!)

Keep up with the discussion in various places.

Dept of Servers 🏢

Synapse

This week sees us complete the last items on the privacy sprint and we expect a release to land early next week. Aside from that OpenTracing is now live, the configuration tool is ready to go (modulo the new Synapse release) and we’ve been trialling out room directory improvements.

Next week expect Synapse 1.4 to land which will contain privacy sprint support as well as a host of bug fixes and perf improvements.

Dept of Bridges 🌉

matrix-appservice-slack 1.0.0-rc1

MAJOR milestone for Half-Shot this week!

Hello Twimians! After many weeks of creativity, pain, and slack, the bridge team are proud to present our first release candidate for the slack bridge. This release brings in a ton of new changes that should massively improve the slack<->matrix experience. The headline features are Postgres support, Puppeting, RTM (websocket) support as well as an entirely refactored codebase written in Typescript.

This release has made heavy changes to the schema of the slack bridge and as such we are not recommending that anyone upgrade their existing slack bridge instances to 1.0.0-rc1. A migration script is currently being worked on, but for the time being existing installs should not be upgraded to 1.0.0 release candidates.

matrix-appservice-irc 0.13.0-rc2

Half-Shot announced:

HELLO IRC FANS. Or possibly not irc fans, if you are running a Matrix bridge.. After a long period of not-being-ready the IRC bridge has a new RC, https://github.com/matrix-org/matrix-appservice-irc/releases/tag/0.13.0-rc2. Hopefully the last, but time will tell.

Keybase bridge, mega-pre-alpha

Half-Shot reported:

There is also now a keybase bridge

This is ready for testing, room here: #keybase:half-shot.uk

Dept of Clients 📱

Scylla

@daniel:riot.danilafe.com reported:

Hi all! I haven't been around much because our homeserver chugs a little when federation is enabled, but I've got a few updates to share about Scylla, the Elm-based web client for Matrix:

1. There have been a few optimization changes. The naive Elm approach leads to recomputing room names and usernames on every keystroke, which causes huge performance issues in rooms with many users (like #freenode_#haskell!). The new versions of Scylla avoid recomputing room names and message HTML, leading to much better performance in large rooms and rooms where a lot of history is loaded.
2. The recommended fixes in the spec regarding flickering have been implemented, and messages should no longer flicker.
3. "m.notice" and "m.emote" events are now supported
4. There has been a minor visual update to make the interface look a little more consistent and professional.

FluffyChat 11.16

@krille:ubports.chat offered:

We have released FluffyChat 11.16 with minor bug fixes and updated translations. A nice new feature: Messages which contains emojis only are now displayed with a much bigger font. 💕

Continuum 0.9.23

yuforia reported:

Continuum desktop client written in pure Kotlin, version 0.9.23:

• Include Elliptic Curve module to be able to connect to HTTPS servers using ECDSA

• Removed dependency on kotlin-reflect and tornadofx completely and make packaged size about 5 MB smaller, which is a 10% to 20% reduction, depending on the target platform and package format

• In addition to automatic scaling when HiDPI is detected, font size scaling is added. Press Ctrl and + or - to increase or decrease the font size. Font size scaling is not limited to texts, components with their size defined relative to the font size also gets scaled.

• When a image message is clicked, a viewer is opened to provide a closer look. Now the updated viewer is opened in new windows, multiple images can be viewed at once, and interaction with the main window is not blocked. Images also gets scaled when the viewer window is resized.

• Removed borders and backgrounds from 3 buttons and give them a flat appearance.

Fractal

Alexandre Franke told us:

Since last time, Fractal got many translation updates. Current coverage can be seen here. We’re also now disabling the message entry if the user is not allowed to send messages to the room and fixed a crashed when logging out or logging in with the wrong password.

Brawl, new web client from Bruno the Riot dev

Bruno told us:

I've been working occasionally for the past months on a basic matrix web client called Brawl, focusing on performance, offline usage and working on my old phone. I recently got it in still limited, but somewhat useful state. Unsure how much time I'll have to keep working on this regularly, but wanted to share it here anyways. Check it out on https://github.com/bwindels/brawl-chat, there's a GIF of it in action 🙂

Riot Web

From the team:

We’ve made good progress on our first FTUE (First Time User Experience) project: “Improve add & create room”, with only the makeover of the create room dialog remaining. Next week more FTUE projects, and we hope to get some progress on cross-signing, barring any other distractions.

Riot iOS

From the team:

• Finalising privacy
• Release 0.9.4 with support of the new Riot configuration link to quickly customise servers on the authentication screen.

Riot Android

From the team:

• Two minor releases have been made (0.9.5 and 0.9.6), to allow auto configuration of the homeserver in the login screen with a configuration link, and to fix an issue with SSO using Google account.
• Still working on privacy

RiotX (Android)

From the team:

• Working on stabilization. We already have fixed some issues.
• François is still working on ReadMarker
• Benoit is working on saving draft (storing unsent messages)
• Also implementing SSO login and M_CONSENT_NOT_GIVEN error

ALSO: "bashtrix"

Matthew reported:

i wrote a toy client in ~8 lines of bash (thanks to anoa for fixing it up): https://github.com/ara4n/random/blob/master/bashtrix.sh; context: https://news.ycombinator.com/item?id=20948530

Dept of Ops 🛠

synapse-simple-antispam now available in matrix-docker-ansible-deploy

Slavi offered:

due to the large number of invite-spam attacks and increased interest in the synapse-simple-antispam spam checker module, matrix-docker-ansible-deploy now has support for installing and configuring it.

Dept of Bots 🤖

Release Tracking Bot from Ananace

Ananace reported:

Had some free time and ideas this week, so I've improved my release tracking bot to now correctly handle repos mixing full releases as well as tags (lightweight and/or annotated). GitHub's API really sucks in the tags department, so this required a rework to GraphQL queries instead, and it's looking good on that so far. The bot currently still only handles tracking all starred repos of a given account on GitHub, but I'm working on GitLab support, as well as support for tracking arbitrary repos. And a provisioning API is also somewhere on the roadmap,

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Overview

During GSoC 2019, I participated in and pushed forward the development of Dendrite, a high-performance, scalable Matrix homeserver implementation written in Go.

Like many others passionate about Dendrite, I've been eager to see the day when it finally joins the decentralised, federated Matrix network, so I worked on the project to see how much I could do to bridge the gap between it and the current reference homeserver implementation, Synapse, in terms of feature completeness.

Working on Dendrite in just one area would be insufficient to propel it, so my work covers various components in the project, including the Client/Server API, Sync Server, Room Server, and Federation Component. I also spent time refining the project's documentation, improving its testing/continuous integration process, as well as reviewing pull requests from Matrix.org members and the community.

Below is the major portion of my work presented as pull requests, categorised by the components they belong to; a list of links to all the pull requests I created/reviewed is also available at the end of this report.

Work in Component: Client/Server API

This component is the main handler of HTTP requests from the clients, except for /sync requests - see next part.

Work done:

• Implement client single event retrieval (#693)
• Implement profile retrieval over federation (#726)
• Add missing servers field in /directory/room/:alias response (#732)
• Implement room creation content (#754)
• ...and other fixes and improvements

Work left:

• Implement guest access (#668)
  • This was removed from the plan for GSoC so work with higher priorities according to the milestone could be done first.

Work in Component: Sync Server

The Sync Server is responsible for handling the long polling notification requests (/sync requests) and some other related requests from the clients.

Work done:

• Add EDU support and typing notifications to /sync (#718)
  • EDU stands for Ephemeral Data Units; they are events carrying short-term status in the room, for example typing notifications.
• Implement "full_state" query parameter for /sync (#751)
• ...and other fixes and improvements

Work left:

• Support for other types of EDUs

Work in Component: Room Server

The Room Server is, as its name suggests, where room events, state, etc. are handled.

Work done:

• Implement event redaction (#768)
  • This also includes a relatively large chunk of changes in the Sync Server.
• Fix permission and 404 response for alias deletion (#706)
• ...and other fixes and improvements

Work left:

• Support for alias deletion by admins
• Discussion on and implementation of event retention policies

Work in Component: Federation

The Federation Component sends requests to and receives requests from other Matrix homeservers.

Work done:

• Add joined hosts query APIs (#781)
• (In matrix-org/gomatrixserverlib) Support for more query types in Federation Client: Leave API, Public Rooms API, Profile Query API
• ...and other fixes and improvements

Work left:

• Implementation of a "Most Recent Transaction Sender" In-Memory Cache
  • There is a delay in this part of work because some dependency problems took longer than expected to resolve, but this has been planned and will be worked on after GSoC.
• Implementation of a component that handles backoff/retries for federation requests

General Fixes, Improvements, Maintenance Work

• Data race fixes
  • Fix data races reported by go test -race ./... (#748)
  • Fix data race in clientapi/routing/register.go (#787)
• Fixes for incorrect transaction ID scopes
  • Fix transaction IDs in transaction cache have global scope (#772)
  • Associate transactions with session IDs instead of device IDs (#789)
• Improvements for building/testing process
  • Refine config and docs for sytest (#714)
  • Fix build conflict between docker environment and host (#598)

Links to All Pull Requests

Here is a list of links to all the pull requests I have created/reviewed (by 26 Aug, 2019):

Appendix: On Dendrite Being a GSoC Project

I believe that Dendrite is a pretty special project to work on in the GSoC program.

At its current state, Dendrite isn't a complete homeserver yet. This has left a large portion of work on Dendrite wide open: As I worked on Dendrite, more than often I'd find myself in a place to answer questions related to topics like design or project architecture which, if not thoroughly thought through, might have consequences in the future development of the project. For this reason, I think working on Dendrite involves a set of challenges quite different from what attempting to improve an already functioning piece of software would have presented.

For the same reason, I think what Dendrite has taught me during the summer was less about how to become more fluent in Go or more familiar with various tools, but more about developing the mindset that powers software projects behind the scenes. But this mindset, to myself, seems more valuable than the former, indeed.

And at the end of this report, I'd like to thank my mentors anoa and Brendan, as well as everyone else at Matrix who has answered my questions, discussed with me, and helped me understand the code, and at the same time being super responsive. Without your help, my GSoC experience wouldn't have been this enjoyable! :)

The Google Summer of Code 2019 is coming to an end for me, so it means that it’s time for the final report.

The work done

I’ve been taking care of the project “Matrix Visualisations” during these past months. This project aimed at initiating the development of a tool which allows the real-time visualisation of the events DAG of a given Matrix room, as seen from the perspective of one or more homeservers (HS’s).

Regarding my initial proposal, I’ve completed every task proposed and even implemented some additional functionalities. The application is not finished yet and there could be a lot of improvements added to it (especially regarding the design of the UI) but the core functionalities have been implemented.
I am going to precise what has been accomplished and then give some ideas of features to improve.

During GSoC, I have used two separate repositories for the frontend and the backend. I will keep both of them because I’m referencing PRs from them (as PRs are easier to link than lengthy lists of commits).
However, this is the repository regrouping these two parts and this one will be moved to matrix-org for the continuation of this project.

Complete the implementation of the CS API backend

During the application period, I wrote a prototype for this project. This prototype implemented some requests to the CS API (like /login, /logout or /sync) but there were more requests to implement in order to be able to fully use this API:

• In order for the application to automatically join (and leave at the end) a room if the provided account hasn’t already joined it, I implemented the requests to /joined_rooms, /join and /leave.
• The request /messages has been implemented to allow the application to fetch previous messages.
• For retrieving in real-time new events, I used the field from in the /sync request.

I also did a lot of clean up in the source code from the prototype during this task. You can have more details in this PR.

Implement the first UI to interact with the DAG

First of all, a lot of work had to be made in order to properly update the displayed DAG when adding new events to it. At this point, I previously used the setData method of the Network object of the visjs library (which is used for displaying the graph and interacting with it) each time a node was added, but it was resetting the display each time it was called.
The proper solution was to progressively add nodes and edges to the DataSet object passed to the constructor of the network (see the documentation of DataSet and Network for more details).

The DAG has been set to be displayed vertically, the events with the same depth are at the same level, the deepest events are at the bottom. The events which origin is the HS on which we are making the observation are in green, those which are coming from other servers are in orange. Here is a picture of what it looks like:

In the graph, there is a special node just after the earliest event which allows us to ask the application to load more events. Here is what it looks like:

When you double click on the node of an event, there is a text area on the right which displays its complete JSON body, like this:

I’ve also implemented the possibility to choose which particular fields of the event can be directly displayed in the labels of the nodes of the displayed graph:

Synapse PostgreSQL backend

Next, I implemented a backend for retrieving events from the PostgreSQL database of a Synapse HS. It is a small HTTP server which receives requests from the frontend application and then makes queries to the Synapse database to get the requested events.
You can find details about the API for using this backend in this readme, in the “HTTP REST API” section.
You can find more details about this initial implementation: here, here, here (my mentor helped me on this one, thanks to him), and here.
I’ve added the support of this backend in the frontend application, as well as a way to choose which backend to use (between this one and the CS API) in this PR.

Multiple views

I implemented the ability to observe the same DAG of a room from multiple HS’s. This is done with “views”.

In the picture above, you can see that there is a drop down menu from which you can select the view. The fields under this line are used to control the selected view: indicate where it will be connecting, for which room (you could as well observe a different room in a different view), etc…
By default, there is only one view but you can add as many as you want by clicking “Add a view”.

All the DAGs from the different views are displayed side-by-side within the same canvas, like this:

You can have a look at the details of the implementation in this PR.

Add Federation API support

The next major task I had to do was the implementation of a backend for retrieving events via the Federation API. I thought a lot about the possible options for the location of this backend and I decided to extend the web server created for the PostgreSQL Synapse backend, so that we could launch it in a “postgres mode” or “federation mode”. But the backend would offer the same HTTP API in both modes.

The backend uses the Federation API in the following way:

• Before being able to retrieve events from a certain room, the backend must join it with /make_join and /send_join requests, it creates an “imaginary user” in this room. The join event created during this process will be the one returned by the endpoint /deepest.
• To get earlier events, the backend uses /backfill requests.
• In order to get new events, the backend listens for pushed events from other HS’s with the /send request.
• When the observation is done, the backend makes the “imaginary user” leave the room by sending /make_leave and /send_leave requests.

The full details of the implementation are in this PR. My mentor also helped me get the usage of the Futures right thanks to this PR.
There has been a small modification in the frontend too, because of the addition of the /stop endpoint in the backend’s HTTP API, these modifications are in this PR.

Display the state of the room for a given event

For each event, there is a state of the room associated with it, which describes what was the state of the room at the moment this event was accepted (the name of the room, its topic, its members and parameters, etc…).
So I’ve added a way to display this: when you have selected and displayed the JSON body of a given event, you can also request the associated room’s state. I have made it possible to use this feature with every backends: the CS API, the PostgreSQL database and the Federation API. You can have the full details of the implementation here (for the backend) and here (for the frontend).

You can see the result of this feature in the picture below (there is a button “Room state at the selected event”, which allows to ask the application to fetch the state, and the text area under this button where the state is displayed):

Additional fixes

Lastly, I’ve applied small fixes to the code of the backend, you can see them in this PR.

Possible improvements

The objective of this project was to develop the core functionalities of this application, however there are a lot of improvements to bring to it, like:

• Adding the possibility to start the observation of a room from any events (provided that we have the ID of this events) instead of the latest one.
• These hasn’t been any UI/UX work design, the CSS style sheet is minimal and the overall look isn’t beautiful or correctly organised. So there would be a lot of work to be done in this area by people with better knowledge in this field than me.
• The timestamps of the events are not displayed in a human readable format and would be written as dates and hours to greatly improve the readability.
• The application has been tested a lot, especially in situations of misusage. I’ve fixed some bugs which occurred when I was testing it but it was far from being an exhaustive testing, so there could be many improvements regarding the overall robustness of this software.
• The backend supports HTTPS connections but has no mechanism for controlling the access of the data behind it (in particular, it means that if you should not run on a production database as it would basically allow anyone to access any data on it). So more work would be needed to make it secure.

Conclusion

This experience has been really rewarding for me. I could discover more about the Matrix community and how the Matrix ecosystem works (on a technical point of view). I want to thank my mentor, Erik Johnston, for his guidance during these past months, and the people in this community who gave me advice.

GSoC has also allowed me to further improve my programming skills in general and discover many various things: the WASM technology, how to use Rust in this context thanks to the various existing libraries/frameworks available, the practical usage of SQL requests as well as TLS certificates and how to apply cryptographic signatures.
It was sometimes challenging to use such experimental technologies (due to the lack of clear documentation) but also very exiting!

Mid-September, I will start my class for my second and final year of my master degree (software engineering, specialised in distributed systems and applications) at Sorbonne Université so I will definitely have less free time. So I don’t think I’ll be able to actively continue to contribute but I will do my best to help other people to continue the work I’ve initiated.

Matrix Live 🎙

Dept of Servers 🏢

Synapse

Neil from Synapseland:

This week privacy items have dominated, but we’ve also been edging towards getting our room directory improvements out, shipping a new configuration creator and making further progress on the federation side bus project to improve performance of smaller instances.

We also deployed open tracing on matrix.org, which will help a lot in tracking down all sorts of strange behaviour in the wild and fixing some bugs in the sqlite -> postgres migrator.

Next week we’ll land the room directory improvements, much of the privacy work, and have the configuration creator ready for testing.

Construct

Jason:

Construct added user room listing support so that you can find out what public rooms a user is a member of should they choose to not be invisible. To find out the list of rooms for a user, simply open the room directory in Riot and type a user mxid starting with @...

Construct is implementing the concept of invisibility for users:

The exact mechanism for invisibility is not completely finished but there are obviously some limitations for this feature:

• Only users who are local to the server can set any kind of invisibility. Remote users are always fully listed. One day when the spec has "user's rooms" we would be glad to adhere to an m.user.invisible-etc configuration state event in such a room
• Since there is no federation support for user's rooms list publication, the rooms list for a user is limited to what the server knows (rooms it has seen the user in).

Construct is eager to implement any MSC's regarding user room and group membership listing over the federation.

Construct is written for maximum performance in C++, and is developing rapidly - join #construct:maunium.net for more info.

matrix-media-repo updates for quarantining and purging media

TravisR announced:

matrix-media-repo has received a bunch of updates regarding quarantining and purging media - see the admin docs for more information. There's also an s3 upload bug fix if you're using that.

Dept of Bridges 🌉

GitLab bot to be re-written as a maubot

Tulir announced:

lorico has been rewriting my old GitLab bot to python/maubot: https://github.com/maubot/gitlab/pull/23. Progress on that is looking very good and it'll probably be ready within a week or two.

Check the maubot GH page for more.

mautrix-telegram, small release

Tulir again:

I decided to make a minor mautrix-telegram release with some small fixes, since v0.7.0 will take a while still. v0.6.1-rc2 is out now, the full release will probably happen next week if nobody finds any problems. Main changes:

• Invalid reply fallbacks sent by Riot web in edits are now ignored
• User and portal sync errors are now caught so that the sync command wouldn't fail if one telegram chat disappears.

Dept of Clients 📱

Riot Web

Tidy update from Bruno this week:

This week we were doing the last bits of the privacy work, and fixing bugs and adding a format bar in the new composer (still behind labs flag).

We’ve also been planning the First Time User Experience work, which will start at full speed next week. This is an effort to address some issues that new users hit in the first days of using Riot. See https://github.com/vector-im/riot-web/projects/16#card-25137120 and Matthew talking over the details at https://youtu.be/JYsiHSL0lEA?t=710. Letterboxes in the left panel will likely only be removed as part of the new community project though, as we need parts of the new community UX to keep things usable for power users.

Also, if you're using https://riot.im/develop please enable the new composer in Labs, it works well and deserves testing.

Continuum desktop client, many updates

yuforia reported:

Continuum desktop client written in pure Kotlin, version 0.9.20:

• Updated login page appearance
• Added validation to user ID and server address fields
• The server address gets automatically filled with https:// followed by server name when user ID is being typed
• After signing in, the token and server address are automatically saved
• Added progress indicator when being signed in. It doesn't block user from controlling Continuum, pressing Esc will cancel the signing-in process.
• Speed up application start-up by drawing the window and opening the database in parallel. Now the window typically appears within half a second with its size and position on screen restored to be the same as the last time it was opened. The most recently used accounts get loaded and automatically filled within a second.
• Added drop shadow effect to scroll-to-latest-messages button to increase contrast
• Fix: Always retry synchronization after getting a network error. Previous versions didn't retrieve new messages in a timely manner when the network is unreliable. A delay is added if the error is not a time-out to avoid stressing the homeserver.

koma the Kotlin library behind Continuum:

• Fix suspend/hibernate detection. Synchronization is restarted automatically after the computer resumes from sleeping. The reason is that any previously opened connection is unlikely to be still alive, opening new connections reduces delay.

• Fix incorrect interface declaration that caused login to fail

Pattle 0.13.0 released

Wilko said:

A new version has been made available and should soon be on F-Droid, Google Play and TestFlight!

Expect the new version to be on most platforms in 2 days.

It's not the biggest release, but a release nonetheless.

Changes

• Separate public and personal chats
• Use a bullhorn icon for public chats without an avatar
• Fix sync being started multiple times on startup
• Fix the room.member_count error (might require clearing of data)
• Many SDK changes, for features in future releases

Get Pattle

• Android
  • F-Droid
  • Google Play
• iOS (TestFlight)

Please let me know what you think of the separation of personal and public chats! Currently it's implemented using a bottom navigation bar, but tabs could also be an option (which is closer to what WhatsApp does)

Riot Mobile versions

All teams when I asked responded:

Privacy, privacy and privacy.

Dept of Services 🚀

t2bot.io has 300k MAU

TravisR told us:

t2bot.io has surpassed 300k monthly active users (about 35k daily) 🎉

roughly 250k are (bridged) from Telegram, the rest is from Discord. (there's some very small numbers which include the bots, webhooks, and other tiny bridges, but together they form less than 100 users)

room list viewer tool

joepie91 offered:

Inspired by a question in the Matrix HQ channel, I've built a (rudimentary) room list viewer at https://matrix-rooms.cryto.net/ - you can just enter a homeserver (hostname) and see all the public rooms on it, without needing to log into anything. This might be useful for showing other people what kinds of rooms exist on Matrix!

Dept of Blockchain 🤷‍

OI Chat, Matrix based chat uses blockchain for authentication

@friedger:matrix.org said:

OI Chat has been updated and now connects blockchain users on Blockstack and EOS (and the rest of the matrix universe): https://www.producthunt.com/posts/oi-chat-2 The sever has no passwords or requires permissions by other social account providers, it verifies the authenticity of the user using properties of the blockchain IDs, it names registered on the blockchain.

Dept of Ops 🛠

Matrixmon

Matrixmon new release 0.4.0, featuring Docker image hosted on Docker Hub and an option to specify custom config file location.

Dept of Bots 🤖

matrix-fly-paper: new project for room moderation

@serra:lost-angles.im reported:

matrix-fly-paper is a new bot project intended to help automate server and room moderation. It's currently not ready for any kind of use, as I'm trying out serious TDD and attempting a functional approach to break from my OO training. Feedback is appreciated, stop by #matrix-fly-paper:lost-angles.im to talk shop

If you're interested in such things, there is an existing tutorial on how to use matrix-bot-sdk for exactly this at https://matrix.org/docs/guides/matrix-bot-sdk-room-admin-features.

Dept of Merch 👕

About a month ago we started selling merch at shop.matrix.org, and WOWEE ZOMG we sold a lot of stuff! If you are happy with your purchase I'd love to hear from you - share photos of where you put the stickers or you happily wearing the t-shirt in #twim:matrix.org and I'll make a little collage.

If you are not happy with your purchase then please message me at @benpa:matrix.org and I will try to make it right.

Final thoughts 💭

You should complete this EU-related FOSS survey, and if you wanted to say nice things about Matrix that would be fine by us.

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Matrix Live 🎙

Dept of Spec 📜

Approved MSCs

• MSC1957: Integration manager discovery

Final Comment Period

• MSC2263: Give homeservers the ability to handle their own 3PID registrations/password resets
• MSC2264: Add an unstable feature flag to MSC2140 for clients to detect support

In Progress MSCs

• MSC2246: Asynchronous media uploads
• [WIP] MSC2249: Require users to have visibility on an event when submitting reports
• [WIP] MSC2260: Update the auth rules for m.room.aliases events
• MSC2261: Allow m.room.aliases events to be redacted by room admins
• MSC2244: Mass redactions
• MSC2265: Proposal for mandating lowercasing when processing e-mail address localparts

Notes on the Spec from tulir

I made a few spec proposals: MSC2244: Mass redactions and MSC2246: Asynchronous media uploads. Mass redactions are useful for fighting spam (and bridging telegram deletions) more efficiently. Asynchronous media uploads would make it possible for bridges (like telegram) to handle messages synchronously and thus keep message ordering without blocking all messages behind some media upload.

Dept of Servers 🏢

Synapse

Erik reported:

• Busy landing the remaining items for the privacy project.
• Reviewing and landing the room stats work Oliver has been busy with before he finishes his internship this week (thanks Oliver!).
• Jorik has been doing the finishing touches to the Synapse installer (which looks awesome fyi).
• We’ve also been looking at a basic message retention implementation in Synapse.
• Finally, Hawkowl had a lot of fun at Kiwi Pycon last week (woo keynote!), and is now back and looking at improving performance of outbound requests.

Construct

Jason from Construct:

Construct added support for server notice rooms. As the server administrator I've placed quick access to the server control room for myself in this section.

Construct has also added support for the server-side commands \\read <room-tag>, for example \\read m.low_priority will mark all low-priority tagged rooms as read. The command \\read * also will mark all rooms as read.

Also, we now allow users to configure whether they transmit typing events by setting state events in their home-room. If a user creates an event with the type of ircd.typing.disable.send and the state_key of the room_id, typing events will not be sent to that room for them. Further variations exist, such as ircd.typing.disable.sync to prevent the client from receiving typing events for a particular room or all rooms.

Learn more in #construct:zemos.net

Matrix Visualisations GSOC project 🎓

Eisha announced:

I've finished my GSoC project "Matrix Visualisations", you can view my final report at this address https://gist.github.com/Kagamihime/d0e7ac434b0d80eb201260d213dea0b7

Dept of Bridges 🌉

mx-puppet-twitter NEW

sorunome reported:

mx-puppet-twitter....do they ever stop?!

Now there is a twitter DM bridge based on mx-puppet-bridge! It uses twitters webhook API to receive direct messages instantly.

So far implemented:

• basic message sending in both directions
• Image, Gif and Video sending (twitter special-cases those) in both directions
• twitter oauth to authenticate

Support room: #mx-puppet-bridge:sorunome.de Donate on liberapay: https://liberapay.com/Sorunome/donate

Jojii email bridge NEW

Jojii announced:

For all those who want a new emailbridge, I have great news for you! The new bridge supports sending and writing emails, using multiple rooms for multiple email-addresses. In addition you can use markup to improve the style of your emailbody, which gets translated to HTML. Here you can find the repository: Matrix-EmailBridge

Go join #jojiiMail:matrix.jojii.de for more.

Big news for the Slack bridges

Half-Shot offered:

The Slack bridge has had a whole week of care applied to it. We've switched the bridge entirely over to Typescript, and overhauled large parts of the codebase to make future work easier (for ex, using the Slack node library rather than request calls). If that's not exciting enough, we've also added support for the RTM API, which means the bridge can use websockets when talking to slack. This gives us nice things like typing updates, and the ability to deploy without needing a webserver to listen for events. Release 1.0 is imminent, but before then we are hoping to land Postgresql support so that the bridge can scale up to meet the high demands placed upon it.

mautrix-whatsapp

Tulir told us:

mautrix-whatsapp got a bunch of small fixes and a dbms migration tool.

Dept of Clients 📱

Riot Web

Ironed the bugs out of the new composer, which is available on https://riot.im/develop behind a labs flag. Please test!

Riot iOS

Privacy, privacy and privacy.

Riot Android

A security release has been done on thursday. More details here: https://medium.com/@RiotChat/matrix-android-sdk-security-issue-affecting-riot-android-36b4792ea0d6

In parallel we continued working on privacy.

RiotX (Android)

A release has been done today (v0.4.0). The main new feature is Read Receipts!

Neo Renaming

f0x reported:

I'm considering renaming Neo, to help with getting a nice domain, and better findability in general. Any suggestions welcome in #neo:pixie.town.

Pattle 0.12.0

Wilko told us:

A new version has been pushed to F-Droid, Google Play and TestFlight!

⚠️ This update requires a reinstall or clearing of data ⚠️: Because of some database changes which were not that easy to migrate, you'll have to reinstall the app or clear the data.

Expect the new version to be on most platforms in 2 days, and on Google Play in a few hours.

Changes

• Add a dark theme!
• Add basic settings page
• Add ability to change your own display name
• Allow multiple lines while typing a message
• Render line breaks in messages
• Fix issue where homeserver setting was overridden by the user domain
• Mark new messages as read while chat is opened
• Fix chats being sorted by your own display name change
• Fix multiple issues regarding the member count of room being null
• Use Flutter beta, which means that you can paste in textfields again!

Get Pattle

• Android

  • F-Droid
  • Google Play
  • APK

• iOS (TestFlight)

Issues or suggestions

If you stumble upon any issues or would like to suggest features, please do so here! You can login via GitHub and Gitlab.com!

Follow development

Follow development in #app:pattle.im!

Continuum 0.9.19

yuforia reported:

Continuum the pure Kotlin desktop client

• Provide builds of self-contained executables that have no external dependency. The Linux version has been done. AppImage is used, producing a binary of about 50 MB.
• Adjust appearance. Remove alternating row high-lighting in list views. Remove extra padding between messages.

Quaternion has many visual improvements

kitsune reported:

Thanks to Roland Pallai, Quaternion master branch is now able to use customised fonts and sports a faster timeline, better copying and a few other improvements. Settings dialog window is coming.

Dept of SDKs and Frameworks 🧰

igor: a Matrix bot framework for Elixir

uhoreg offered:

for fans of strange programming languages or of monster movie clichés, I've created a Matrix bot framework for Elixir: https://gitlab.com/uhoreg/igor

Dept of Encryption 🔐

poljar:

Initial code dump of Seshat, a Matrix event indexer/database that should enable search for encrypted rooms in riot-desktop. The dump contains a rust library as well as node bindings. The repo can be found here.

Dept of Ops 🛠

ARM docker images for synapse

Black Hat told us:

Apparently there are no ARM docker images for synapse, so I built a multi-arch synapse docker image. Supported CPU architectures include armv7, aarch64 and amd64.

Dept of Bots 🤖

tulir:

There's a useless new maubot plugin that lets you run python and soon shell scripts from Matrix. Not intended as a public bot as it doesn't have any sandboxing, but it does have a "userbot" mode to run it under your own account so that you can use it in any room.

Slightly mean to call your own bot useless :(

Final thoughts 💭

If you've suffered from invite spam lately, you can solve the problem by following these instructions:

There's 3 methods to block the room:

1. Use the synapse admin API for it:

curl -s -X POST -H "Authorization: Bearer YOUR_ACCESS_TOKEN" -H "Content-Type: application/json" -d '{"new_room_user_id":"@abuse:YOUR_SERVER.org"}' 'https://YOUR_SERVER/_matrix/client/unstable/admin/shutdown_room/!BAD_ROOM:example.org'

2. Block the room with an antispam module ( https://github.com/t2bot/synapse-simple-antispam )
3. Ignore the user with /ignore

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Dept of Status of Matrix 🌡

Installing Matrix @ CCCamp 2019

It's CCCamp 2019 this weekend. Tonight, anoa will be leading an "Install Synapse" workshop in the Librehosters Village- keep decentralisation growing!

Dept of Servers 🏢

Synapse

Neil:

This week we’ve been working on getting our brand new room directory implementation ready, creating a new Synapse installer, and creating a TURN server testing tool. We’ve also been pushing on with the privacy project and fixing up some bugs and perf issues.
Next week it’s all about getting the room directory out of the door, shipping the installer and continuing on with the privacy project. We are in the early stages of a project designed to reduce the load caused by outbound SSL connections, it is looking very promising. We’re also going to give the sqlite->postgres porting script some love as well as working on some message retention ideas, and also ways to reduce the meta data we store.

Dept of Bridges 🌉

mautrix-facebook updated

Tulir announced:

the adding portal rooms to user-specific communities feature that appeared in mautrix-whatsapp and mautrix-hangouts last week is now also available in mautrix-facebook

Gitter bridge is back

Half-Shot offered:

The gitter bridge is back to working order again today! We've made changes so that Gitter side admins can now issue a !matrix kick|ban name command to evict users who are causing issues, rather than relying on reports or matrix side admins.

matrix-appservice-bridge library 1.10.0

Half-Shot reported:

The matrix-appservice-bridge library also saw its 1.10.0 release this week, which has experimental support for bridge errors and also has a bugfix related to room upgrades.

Discord bridge will soon use matrix-bot-sdk

Half-Shot said:

Finally, the discord bridge has very nearly switched to Travis's bot SDK stuff. We are very close to a release, so please stand by :)

Dept of Clients 📱

Riot Web

From the team:

• Work on privacy.
• We’ve also been continuing the work on the new message composer implementation. Features are still missing, but this has now landed behind a labs flag on /develop. We’d be especially interested in feedback whether typing with an IME (Chinese, Japanese, …) works better than before.

Riot iOS && Riot Android

From the team:

• Working on privacy:
• Working on identity server v2 API

RiotX (Android)

From the team:

• Read receipt work has been merged to develop. We will release next week, after a small UI iteration
• François is working on Read Markers

Riot with Math support

Tim offered:

I've hosted a version of riot with the changes from https://github.com/matrix-org/matrix-react-sdk/pull/3251 (for math support) built in so that people can test/use it if they want. It can be found at https://www.pigeon.digital — please do let me know about any bugs you may find!
there is now a room specifically for the LaTeX fork hosted on www.pigeon.digital where you can leave any feedback, report any bugs, or ask for any help: #pigeon.digital:matrix.org

Pattle 0.11.0 released

Wilko reported:

A new version has been pushed to F-Droid, Google Play and TestFlight!

Pattle is now available on Google Play and the official F-Droid repository!

⚠️ F-Droid users ⚠️: The old repository (fdroid.pattle.im) is gone! Please use the official F-Droid repository.

Expect the new version to be on most platforms in 2 days (however, Google Play in a few hours).

Changes

• Fix UI not updating on sync!

• Add ability to send images from your gallery!

• Use Material Design on iOS (although with slight changes to fit iOS)

• Reporting errors to Sentry is now optional, you can choose so at login time

  I do urge everyone to enable it to help me develop Pattle!

• Show a dialog with the option to choose between register and login if the server does not have the ability to check for username availability

• Show notification count in chat overview

  Default notifications will be grey, highlighted notifications will be red.

  Chats with unread notifications will always show on top.

• Send read markers and receipts when a chat is opened for 2 seconds

• Add ability to show password while logging in

• Pattle will now show up as 'Pattle Android' or 'Pattle iOS' in the device list

• Decrease spacing between items

• Fix spacing of messages when sending a new message

• Numerous other fixes

Get Pattle

• Android
  • F-Droid
  • Google Play
  • APK
• iOS (TestFlight)

Issues or suggestions

If you stumble upon any issues or would like to suggest features, please do so here! You can login via GitHub and Gitlab.com!

Follow development

Follow development in #app:pattle.im!

Support me

If you would like to support me, you can do so via Liberapay and Patreon!

'matrixcli', new client enables Matrix as a unix util

Very interesting project: matrixcli is a versatile command line client using the matrix-python-sdk. It's not a conventional IM client, rather it exposes access to Matrix rooms so you can pipe to them as you would other Unix utils.

Continuum progress

yuforia offered:

Continuum

• Handle key mappings and mouse mappings in a more idiomatic way
• Updated appearance, spacing and alignments. https://matrix.org/_matrix/media/r0/download/matrix.org/glNXYasGpmkWJBNYBtUCqgYO

Fractal Progress

Fractal, the GNOME client, continues to progress:

• fractal-gtk: Migrate to GtkSourceView4
• autocompletion-popover: Fix misplacement on small window widths
• Backend-API: Split request building and parsing from singleton
• directory: Remove duplicate entries for rooms

Neo refactoring progress

f0x told us:

Neo's refactoring is almost done, with the latest changes being css-modulesifying the old stylesheet
We're also looking for nice domain name suggestions, to house the soon to be redesigned project page

Dept of Encryption 🔐

libQuotient E2E Progress (GSOC 🎓)

aa13q reported:

The GSoC is soon coming to the end this year and I'm happy no announce that receiving of the encrypted messages PR is here! https://github.com/quotient-im/libQuotient/pull/346 A significant amount of work still needs to be done (session storage, sending messages, device handling, attachments). And I'm interested to implement it after the summer period since I've already dived into the subject :) Here's the small demo of the receiving messages process: https://youtu.be/gGykLh4mVDg

Dept of SDKs and Frameworks 🧰

Matrix-ClientServer-API-java

Jojii created a new Java-based implementation of the Matrix CS API: Matrix-ClientServer-API-java, they call it "A small and simple java API for the Matrix ClientServer Protocol".

They created Matrix-Grep-Bot using the API.

Dept of Ops 🛠

flexible ansible role just for synapse

JCG offered:

For people looking for a rather flexible ansible role just for synapse, https://gitlab.com/famedly/ansible/synapse got your back. You can use it to deploy synapse with pip or docker, depending on what you prefer for your host. It's an actively maintained fork of https://github.com/matrix-org/matrix-ansible-synapse, which has been archived recently and was unmaintained for a while before that. It's just been updated to v1.3.1 too.

New Bridges for matrix-docker-ansible-deploy

Slavi offered:

The matrix-docker-ansible-deploy Ansible playbook has gained new bridging capabilities:

• thanks to Eduardo Beltrame (Munfred) and Robbie D (microchipster), support has been added for bridging to Google Hangouts via the mautrix-hangouts bridge. Learn more in: Setting up Mautrix Hangouts bridging.

• thanks to kingoftheconnors and Stuart Mumford (Cadair), support has been added for bridging to Slack via the appservice-slack bridge. Learn more in: Setting up Appservice Slack bridging.

This increases the number of services/protocols that the playbook can setup bridging for to 8: Telegram, Whatsapp, Facebook, Google Hangouts, IRC, Discord, Slack and Email. Learn more in Bridging other networks.

matrix-docker-ansible-deploy gets support for all Synapse config options

Slavi offered:

matrix-docker-ansible-deploy can now harness the full power of Synapse, thanks to our newly-added support for Extensible Synapse configuration.

Dept of Bots 🤖

New /health maubot

Ananace announced:

Just built a quick maubot plugin to add a /health endpoint to maubot, so that database failures and other brokenness can be tracked and dealt with appropriately.

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!