Synapse 0.26 released!

Hi folks,

Synapse 0.26 is here (with no changes since RC1 which we released just before Christmas).  It’s a general maintenance release, albeit with a few new features but mainly lots of bugfixes and general refinements.  Enjoy!

As always, you can get it from https://github.com/matrix-org/synapse/releases/tag/v0.26.0.

Changes in synapse v0.26.0 (2018-01-05)

No changes since v0.26.0-rc1

Changes in synapse v0.26.0-rc1 (2017-12-13)

Features:

  • Add ability for ASes to publicise groups for their users (PR #2686)
  • Add all local users to the user_directory and optionally search them (PR
    #2723)
  • Add support for custom login types for validating users (PR #2729)

Changes:

  • Update example Prometheus config to new format (PR #2648) Thanks to
    @krombel!
  • Rename redact_content option to include_content in Push API (PR #2650)
  • Declare support for r0.3.0 (PR #2677)
  • Improve upserts (PR #2684#2688#2689#2713)
  • Improve documentation of workers (PR #2700)
  • Improve tracebacks on exceptions (PR #2705)
  • Allow guest access to group APIs for reading (PR #2715)
  • Support for posting content in federation_client script (PR #2716)
  • Delete devices and pushers on logouts etc (PR #2722)

Bug fixes:

  • Fix database port script (PR #2673)
  • Fix internal server error on login with ldap_auth_provider (PR #2678) Thanks
    to @jkolo!
  • Fix error on sqlite 3.7 (PR #2697)
  • Fix OPTIONS on preview_url (PR #2707)
  • Fix error handling on dns lookup (PR #2711)
  • Fix wrong avatars when inviting multiple users when creating room (PR #2717)
  • Fix 500 when joining matrix-dev (PR #2719)

Synapse 0.25 is out… as is Matrix Specification 0.3(!!!)

Hi all,

Today is a crazy release day here – not only do we have Synapse 0.25, but we’ve also made a formal release of the Matrix Specification (CS API) for the first time in 16 months!

Matrix CS API 0.3

Talking first about the spec update: the workflow of the Matrix spec is that new experimental features get added to an /unstable API prefix, and then whenever we release the Matrix spec, these get moved over to being part of the /r0 prefix (or whatever version we happen to be on).  We’ve been very constrained on manpower to work on the spec over the last ~18 months, but we’ve been keeping it up-to-date on a best effort basis, with a bit of help from the wider community.   As such, this latest release does not contain all the latest APIs (and certainly not experimental ones like Groups/Communities which are still evolving), but it does release all of the unstable ones which we’ve managed to document and which are considered stable enough to become part of the ‘r0’ prefix.  Going forwards, we’re hoping that the wider community will help us fill in the remaining gaps (i.e. propose PRs against the matrix-org/matrix-doc repository to formalise the various spec drafts flying around the place) – and we’re also hoping (if/when funding crisis is abated) to locate full-time folk to work on the spec.

The full changelog for 0.3 of the spec is:

  • Breaking changes:
    • Change the rule kind of .m.rule.contains_display_name from underride to override. This works with all known clients which support push rules, but any other clients implementing the push rules API should be aware of this change. This makes it simple to mute rooms correctly in the API (#373).
    • Remove /tokenrefresh from the API (#395).
    • Remove requirement that tokens used in token-based login be macaroons (#395).
  • Changes to the API which will be backwards-compatible for clients:
    • Add filename parameter to POST /_matrix/media/r0/upload (#364).
    • Document CAS-based client login and the use of m.login.token in /login (#367).
    • Make origin_server_ts a mandatory field of room events (#379).
    • Add top-level account_data key to the responses to GET /sync and GET /initialSync (#380).
    • Add is_direct flag to POST /createRoom and invite member event. Add ‘Direct Messaging’ module (#389).
    • Add contains_url option to RoomEventFilter (#390).
    • Add filter optional query param to /messages (#390).
    • Add ‘Send-to-Device messaging’ module (#386).
    • Add ‘Device management’ module (#402).
    • Require that User-Interactive auth fallback pages call window.postMessage to notify apps of completion (#398).
    • Add pagination and filter support to /publicRooms. Change response to omit fields rather than return null. Add estimate of total number of rooms in list. (#388).
    • Allow guest accounts to use a number of endpoints which are required for end-to-end encryption. (#751).
    • Add key distribution APIs, for use with end-to-end encryption. (#894).
    • Add m.room.pinned_events state event for rooms. (#1007).
    • Add mention of ability to send Access Token via an Authorization Header.
    • New endpoints:
      • GET /joined_rooms (#999).
      • GET /rooms/{roomId}/joined_members (#999).
      • GET /account/whoami (#1063).
      • GET /media/{version}/preview_url (#1064).
  • Spec clarifications:
    • Add endpoints and logic for invites and third-party invites to the federation spec and update the JSON of the request sent by the identity server upon 3PID binding (#997)
    • Fix “membership” property on third-party invite upgrade example (#995)
    • Fix response format and 404 example for room alias lookup (#960)
    • Fix examples of m.room.member event and room state change, and added a clarification on the membership event sent upon profile update (#950).
    • Spell out the way that state is handled by POST /createRoom (#362).
    • Clarify the fields which are applicable to different types of push rule (#365).
    • A number of clarifications to authentication (#371).
    • Correct references to user_id which should have been sender (#376).
    • Correct inconsistent specification of redacted_because fields and their values (#378).
    • Mark required fields in response objects as such (#394).
    • Make m.notice description a bit harder in its phrasing to try to dissuade the same issues that occurred with IRC (#750).
    • GET /user/{userId}/filter/{filterId} requires authentication (#1003).
    • Add some clarifying notes on the behaviour of rooms with no m.room.power_levels event (#1026).
    • Clarify the relationship between username and user_id in the /register API (#1032).
    • Clarify rate limiting and security for content repository. (#1064).

…and you can read the spec itself of course over at https://matrix.org/docs/spec.  It’s worth noting that we have slightly bent the rules by including three very minor ‘breaking changes’ in 0.3, but all for features which to our knowledge nobody is depending on in the wild.  Technically this should mean bumping the major version prefix (i.e. moving to r1), but given how minor and nonimpacting these are we’re turning a blind eye this time.

Meanwhile, Synapse 0.25 is out!

This is a medium-sized release; the main thing being to support configurable room visibility within groups (so that whenever you add a room to a group, you’re not forced into sharing their existence with the general public, but can choose to just tell group members about them).  There’s also a bunch of useful bug fixes and some performance improvements, including lots of contributions from the community this release (thank you!).  Full release notes are:

Changes in synapse v0.25.0 (2017-11-15)

Bug fixes:

  • Fix port script (PR #2673)
Changes in synapse v0.25.0-rc1 (2017-11-14)

Features:

Changes:

  • Ignore tags when generating URL preview descriptions (PR #2576)
    Thanks to @maximevaillancourt!
  • Register some /unstable endpoints in /r0 as well (PR #2579) Thanks to
    @krombel!
  • Support /keys/upload on /r0 as well as /unstable (PR #2585)
  • Front-end proxy: pass through auth header (PR #2586)
  • Allow ASes to deactivate their own users (PR #2589)
  • Remove refresh tokens (PR #2613)
  • Automatically set default displayname on register (PR #2617)
  • Log login requests (PR #2618)
  • Always return is_public in the /groups/:group_id/rooms API (PR #2630)
  • Avoid no-op media deletes (PR #2637) Thanks to @spantaleev!
  • Fix various embarrassing typos around user_directory and add some doc. (PR
    #2643)
  • Return whether a user is an admin within a group (PR #2647)
  • Namespace visibility options for groups (PR #2657)
  • Downcase UserIDs on registration (PR #2662)
  • Cache failures when fetching URL previews (PR #2669)

Bug fixes:

  • Fix port script (PR #2577)
  • Fix error when running synapse with no logfile (PR #2581)
  • Fix UI auth when deleting devices (PR #2591)
  • Fix typo when checking if user is invited to group (PR #2599)
  • Fix the port script to drop NUL values in all tables (PR #2611)
  • Fix appservices being backlogged and not receiving new events due to a bug in
    notify_interested_services (PR #2631) Thanks to @xyzz!
  • Fix updating rooms avatar/display name when modified by admin (PR #2636)
    Thanks to @farialima!
  • Fix bug in state group storage (PR #2649)
  • Fix 500 on invalid utf-8 in request (PR #2663)

Finally…

If you haven’t noticed already, Riot/Web 0.13 is out today, as is Riot/iOS 0.6.2 and Riot/Android 0.7.4.  These contain massive improvements across the board – particularly mainstream Communities support at last on Riot/Web; CallKit/PushKit on Riot/iOS thanks to Denis Morozov (GSoC 2017 student for Matrix) and Share Extension on iOS thanks to Aram Sargsyan (also GSoC 2017 student!); and End-to-end Key Sharing on Riot/Android and a full rewrite of the VoIP calling subsystem on Android.

Rather than going on about it here, though, there’s a full write-up over on the Riot Blog.

 

And so there you go – new releases for eeeeeeeeveryone!  Enjoy! :)

–Matthew, Amandine & the team.

Synapse 0.24 is here!

Hi folks,

Synapse 0.24 is out (currently at 0.24.1)! This is a pretty big release as it includes initial support for Groups, also known as Communities (UI for which is landing currently on Riot/Web and later Riot/Mobile). Groups let you associate together a set of users and rooms, letting you define a community – e.g. +matrix:matrix.org is the community of the core Matrix project itself (whose users are the core Matrix.org team, and whose public rooms are the rooms we officially manage/moderate as Matrix.org).  We’ll yell more about Groups once the UI is ready for action in the near future, but the good news is that Synapse should be ready to go (although the API is still fairly experimental and very much evolving).

Other stuff worth calling out in this release includes: massive performance improvements on receiving federation traffic (we now process federation traffic for different rooms in parallel); fixing a major cause of performance issues (caused when processing spurious events for rooms you’ve actually left); modularising and improving the the spamchecker; @room notification support; backup media repository support; and finally the ability to autojoin new users to a set of rooms on the server!

You can get the latest release from Github as usual; have fun – and thanks for flying Matrix :)

Changes in synapse v0.24.1 (2017-10-24)

Bug fixes:

  • Fix updating group profiles over federation (PR #2567)

Changes in synapse v0.24.0 (2017-10-23)

No changes since v0.24.0-rc1

Changes in synapse v0.24.0-rc1 (2017-10-19)

Features:

Changes:

  • Make the spam checker a module (PR #2474)
  • Delete expired url cache data (PR #2478)
  • Ignore incoming events for rooms that we have left (PR #2490)
  • Allow spam checker to reject invites too (PR #2492)
  • Add room creation checks to spam checker (PR #2495)
  • Spam checking: add the invitee to user_may_invite (PR #2502)
  • Process events from federation for different rooms in parallel (PR #2520)
  • Allow error strings from spam checker (PR #2531)
  • Improve error handling for missing files in config (PR #2551)

Bug fixes:

  • Fix handling SERVFAILs when doing AAAA lookups for federation (PR #2477)
  • Fix incompatibility with newer versions of ujson (PR #2483) Thanks to
    @jeremycline!
  • Fix notification keywords that start/end with non-word chars (PR #2500)
  • Fix stack overflow and logcontexts from linearizer (PR #2532)
  • Fix 500 error when fields missing from power_levels event (PR #2552)
  • Fix 500 error when we get an error handling a PDU (PR #2553)

Synapse 0.23 is out!

We’ve just released Synapse 0.23 – which contains a bunch of significant performance improvements, bug and stability fixes – as well as a few new features: basic spam checking (the ability to configure your homeserver to reject events which match arbitrary rules, both from users and other servers) – and long-awaited support for privacy-preserving (‘event_id_only’) push notifications.  This means that apps can choose to register themselves to receive push notifications which do *not* contain any information about the actual push, but instead act as a simple “wake up!” event, which triggers the app to then sync via the client-server API in order to display the actual push notification details.  This is particularly useful for push notifications for E2E encrypted rooms, as it means the client has a chance of decrypting the message in order to display the push notification details in the UI (if the user wants that).  matrix-ios-sdk and matrix-android-sdk are in the process of being moved over to use the new ‘event_id_only’ push format.

Long-awaited Communities/Groups will land in Synapse 0.24, which should come quite soon (we’re almost ready to merge it to develop, but it’s a major update so we wanted to get 0.23 out the door first).

As always, you can get your latest Synapse from https://github.com/matrix-org/synapse or a OS repository of your choice (we’ve just released the official Debian packages).

Full details of Synapse 0.23:

Features:

  • Add a frontend proxy worker (PR #2344)
  • Add support for event_id_only push format (PR #2450)
  • Add a PoC for filtering spammy events (PR #2456)
  • Add a config option to block all room invites (PR #2457)

Changes:

  • Use bcrypt module instead of py-bcrypt (PR #2288) Thanks to @kyrias!
  • Improve performance of generating push notifications (PR #2343#2357#2365,
    #2366#2371)
  • Improve DB performance for device list handling in sync (PR #2362)
  • Include a sample prometheus config (PR #2416)
  • Document known to work postgres version (PR #2433) Thanks to @ptman!

Bug fixes:

  • Fix caching error in the push evaluator (PR #2332)
  • Fix bug where pusherpool didn’t start and broke some rooms (PR #2342)
  • Fix port script for user directory tables (PR #2375)
  • Fix device lists notifications when user rejoins a room (PR #2443#2449)
  • Fix sync to always send down current state events in timeline (PR #2451)
  • Fix bug where guest users were incorrectly kicked (PR #2453)
  • Fix bug talking to IPv6 only servers using SRV records (PR #2462)
  • Fix regression in performance of syncs (PR #2470)

Synapse 0.22.0 released!

Hi Synapsefans,

Synapse 0.22.0 has just been released! This release lands a few interesting features:

  • The new User directory API which supports Matrix clients’ providing a much more intuitive and effective user search capability by exposing a list of:
    • Everybody your user shares a room with, and
    • Everybody in a public room your homeserver knows about
  • New support for server admins, including a Shutdown Room API (to remove a room from a local server) and a Media Quarrantine API (to render a media item inaccessible without its actually being deleted)

As always there are lots of bug fixes and performance improvements, including increasing the default cache factor size from 0.1 to 0.5 (should improve performance for those running their own homeservers).

You can get Synapse 0.22.0 from https://github.com/matrix-org/synapse or https://github.com/matrix-org/synapse/releases/tag/v0.22.0 as normal.

Changes in synapse v0.22.0 (2017-07-06)

No changes since v0.22.0-rc2

Changes in synapse v0.22.0-rc2 (2017-07-04)

Changes:

  • Improve performance of storing user IPs (PR #2307, #2308)
  • Slightly improve performance of verifying access tokens (PR #2320)
  • Slightly improve performance of event persistence (PR #2321)
  • Increase default cache factor size from 0.1 to 0.5 (PR #2330)

Bug fixes:

  • Fix bug with storing registration sessions that caused frequent CPU churn
    (PR #2319)

Changes in synapse v0.22.0-rc1 (2017-06-26)

Features:

  • Add a user directory API (PR #2252, and many more)
  • Add shutdown room API to remove room from local server (PR #2291)
  • Add API to quarantine media (PR #2292)
  • Add new config option to not send event contents to push servers (PR #2301)
    Thanks to @cjdelisle!

Changes:

Bug fixes:

  • Fix users not getting notifications when AS listened to that user_id (PR
    #2216) Thanks to @slipeer!
  • Fix users without push set up not getting notifications after joining rooms
    (PR #2236)
  • Fix preview url API to trim long descriptions (PR #2243)
  • Fix bug where we used cached but unpersisted state group as prev group,
    resulting in broken state of restart (PR #2263)
  • Fix removing of pushers when using workers (PR #2267)
  • Fix CORS headers to allow Authorization header (PR #2285) Thanks to @krombel!

 

Synapse 0.21.1 released!

Hi folks – we forgot to mention that Synapse 0.21.1 was released last week.  This contains a important fix to the report-stats option, which was otherwise failing to report any usage stats for folks who had the option turned on.

This is a good opportunity to note that the report-stats option is really really important for the ongoing health of the Matrix ecosystem: when raising funding to continue working on Matrix we have to be able to demonstrate how the ecosystem is growing and why it’s a good idea to support us to work on it. In practice, the data we collect is: hostname, synapse version & uptime, total_users, total_nonbridged users, total_room_count, daily_active_users, daily_active_rooms, daily_messages and daily_sent_messages.

Folks: if you have turned off report-stats for whatever reason, please consider upgrading to 0.21.1 and turning it back on.

In return, the plan is that we’ll start to publish an official Grafana of the anonymised aggregate stats, probably embedded into the frontpage of Matrix.org, and then you and everyone else can have a view of the state of the Matrix universe. And critically, it’ll really help us continue to justify $ to spend on growing the project!

You can get Synapse 0.21.1 from https://github.com/matrix-org/synapse or https://github.com/matrix-org/synapse/releases/tag/v0.21.1 as normal.

Synapse 0.21.0 is released!

Hi all,

Synapse 0.21.0 was released a moment ago. This release lands a number of performance improvements and stability fixes, plus a couple of small features.

For those of you upgrading https://github.com/matrix-org/synapse has the details as usual.  Full changelog follows:

Changes in synapse v0.21.0 (2017-05-17)

Features:

  • Add per user rate-limiting overrides (PR #2208)
  • Add config option to limit maximum number of events requested by /sync and /messages (PR #2221) Thanks to @psaavedra!

Changes:

  • Various small performance fixes (PR #2201, #2202, #2224, #2226, #2227, #2228, #2229)
  • Update username availability checker API (PR #2209, #2213)
  • When purging, don’t de-delta state groups we’re about to delete (PR #2214)
  • Documentation to check synapse version (PR #2215) Thanks to @hamber-dick!
  • Add an index to event_search to speed up purge history API (PR #2218)

Bug fixes:

  • Fix API to allow clients to upload one-time-keys with new sigs (PR #2206)

Changes in synapse v0.21.0-rc2 (2017-05-08)

Changes:

  • Always mark remotes as up if we receive a signed request from them (PR #2190)

Bug fixes:

  • Fix bug where users got pushed for rooms they had muted (PR #2200)

Changes in synapse v0.21.0-rc1 (2017-05-08)

Features:

  • Add username availability checker API (PR #2183)
  • Add read marker API (PR #2120)

Changes:

Bug fixes:

  • Fix nuke-room script to work with current schema (PR #1927) Thanks @zuckschwerdt!
  • Fix db port script to not assume postgres tables are in the public schema (PR #2024) Thanks @jerrykan!
  • Fix getting latest device IP for user with no devices (PR #2118)
  • Fix rejection of invites to unreachable servers (PR #2145)
  • Fix code for reporting old verify keys in synapse (PR #2156)
  • Fix invite state to always include all events (PR #2163)
  • Fix bug where synapse would always fetch state for any missing event (PR #2170)
  • Fix a leak with timed out HTTP connections (PR #2180)
  • Fix bug where we didn’t time out HTTP requests to ASes (PR #2192)

Docs:

  • Clarify doc for SQLite to PostgreSQL port (PR #1961) Thanks @benhylau!
  • Fix typo in synctl help (PR #2107) Thanks @HarHarLinks!
  • web_client_location documentation fix (PR #2131) Thanks @matthewjwolff!
  • Update README.rst with FreeBSD changes (PR #2132) Thanks @feld!
  • Clarify setting up metrics (PR #2149) Thanks @encks!

Synapse 0.20.0 is released!

Hi folks,

Synapse 0.20.0 was released a few hours ago – this is a major new release with loads of stability and performance fixes and some new features too. The main headlines are:

  • Support for using phone numbers as 3rd party identifiers as well as email addresses!  This is huge: letting you discover other users on Matrix based on whether they’ve linked their phone number to their matrix account, and letting you log in using your phone number as your identifier if you so desire.  Users of systems like WhatsApp should find this both familiar and useful ;)
  • Fixes some very nasty failure modes where the state of a room could be reset if a homeserver received an event it couldn’t verify.  Folks who have suffered rooms suddenly losing their name/icon/topic should particularly upgrade – this won’t fix the rooms retrospectively (your server will need to rejoin the room), but it should fix the problem going forwards.
  • Improves the retry schedule over federation significantly – previously there were scenarios where synapse could try to retry aggressively on servers which were offline.  This fixes that.
  • Significant performance improvements to /publicRooms, /sync, and other endpoints.
  • Lots of juicy bug fixes.

We highly recommend upgrading (or installing!) asap – https://github.com/matrix-org/synapse has the details as usual.  Full changelog follows:

Changes in synapse v0.20.0 (2017-04-11)

Bug fixes:

  • Fix joining rooms over federation where not all servers in the room saw the
    new server had joined (PR #2094)

Changes in synapse v0.20.0-rc1 (2017-03-30)

Features:

  • Add delete_devices API (PR #1993)
  • Add phone number registration/login support (PR #1994, #2055)

Changes:

  • Use JSONSchema for validation of filters. Thanks @pik! (PR #1783)
  • Reread log config on SIGHUP (PR #1982)
  • Speed up public room list (PR #1989)
  • Add helpful texts to logger config options (PR #1990)
  • Minor /sync performance improvements. (PR #2002, #2013, #2022)
  • Add some debug to help diagnose weird federation issue (PR #2035)
  • Correctly limit retries for all federation requests (PR #2050, #2061)
  • Don’t lock table when persisting new one time keys (PR #2053)
  • Reduce some CPU work on DB threads (PR #2054)
  • Cache hosts in room (PR #2060)
  • Batch sending of device list pokes (PR #2063)
  • Speed up persist event path in certain edge cases (PR #2070)

Bug fixes:

  • Fix bug where current_state_events renamed to current_state_ids (PR #1849)
  • Fix routing loop when fetching remote media (PR #1992)
  • Fix current_state_events table to not lie (PR #1996)
  • Fix CAS login to handle PartialDownloadError (PR #1997)
  • Fix assertion to stop transaction queue getting wedged (PR #2010)
  • Fix presence to fallback to last_active_ts if it beats the last sync time.
    Thanks @Half-Shot! (PR #2014)
  • Fix bug when federation received a PDU while a room join is in progress (PR
    #2016)
  • Fix resetting state on rejected events (PR #2025)
  • Fix installation issues in readme. Thanks @ricco386 (PR #2037)
  • Fix caching of remote servers’ signature keys (PR #2042)
  • Fix some leaking log context (PR #2048, #2049, #2057, #2058)
  • Fix rejection of invites not reaching sync (PR #2056)

Synapse 0.19.3 released

Hi all,

We’ve released Synapse 0.19.3-rc2 as 0.19.3 with no changes. This is a slightly unusual release, as 0.19.3-rc2 dates from March 13th and a lot of stuff has landed on the develop branch since then – however, we’ll be releasing that as 0.20.0 once it’s ready. Instead, 0.19.3 has a set of intermediary performance and bug fixes; the only new feature is a set of admin APIs kindly contributed by @morteza-araby.

The changelog follows – please upgrade from https://github.com/matrix-org/synapse or your OS packages as normal :)

Changes in synapse v0.19.3 (2017-03-20)

No changes since v0.19.3-rc2

Changes in synapse v0.19.3-rc2 (2017-03-13)

Bug fixes:

  • Fix bug in handling of incoming device list updates over federation.

Changes in synapse v0.19.3-rc1 (2017-03-08)

Features:

Changes:

Bug fixes:

  • Fix synapse_port_db failure. Thanks to @Pneumaticat! (PR #1904)
  • Fix caching to not cache error responses (PR #1913)
  • Fix APIs to make kick & ban reasons work (PR #1917)
  • Fix bugs in the /keys/changes api (PR #1921)
  • Fix bug where users couldn’t forget rooms they were banned from (PR #1922)
  • Fix issue with long language values in pushers API (PR #1925)
  • Fix a race in transaction queue (PR #1930)
  • Fix dynamic thumbnailing to preserve aspect ratio. Thanks to @jkolo! (PR #1945)
  • Fix device list update to not constantly resync (PR #1964)
  • Fix potential for huge memory usage when getting device that have changed (PR #1969)