Category – General
143 posts tagged with "General" (See all categories)

Synapse 1.3.1 released

2019-08-17 — General — Neil Johnson

Some of you will have been bitten by a bug that prevented Synapse 1.3.0 from starting up correctly. If this is you, please upgrade.

Additionally we have taken the opportunity to fix a dependency bug for our intrepid packagers.

Apologies for the inconvenience.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Also, check out our Synapse installation guide page

The changelog since 1.3.0 follows:

Synapse 1.3.1 (2019-08-17)

Features

  • Drop hard dependency on sdnotify python package. (#5871)

Bugfixes

  • Fix startup issue (hang on ACME provisioning) due to ordering of Twisted reactor startup. Thanks to @chrismoos for supplying the fix. (#5867)

Synapse 1.3.0 released

2019-08-15 — General — Neil Johnson

Well now, Synapse 1.3.0 is here.

The main thing to know about 1.3.0 is that is contains performance improvements to reduce disk I/O and reduce RAM usage. We’ve been running it on matrix.org for a week or so and are really pleased with the results.

Check out our message send heat map.

Message Send

Other than that there are a bunch of bug fixes and tweaks to generally make things run more smoothly.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Also, check out our Synapse installation guide page

The changelog since 1.2.1 follows:

Synapse 1.3.0 (2019-08-15)

Bugfixes

  • Fix 500 Internal Server Error on publicRooms when the public room list was cached. (#5851)

Synapse 1.3.0rc1 (2019-08-13)

Features

  • Use M_USER_DEACTIVATED instead of M_UNKNOWN for errcode when a deactivated user attempts to login. (#5686)
  • Add sd_notify hooks to ease systemd integration and allows usage of Type=Notify. (#5732)
  • Synapse will no longer serve any media repo admin endpoints when enable_media_repo is set to False in the configuration. If a media repo worker is used, the admin APIs relating to the media repo will be served from it instead. (#5754, #5848)
  • Synapse can now be configured to not join remote rooms of a given "complexity" (currently, state events) over federation. This option can be used to prevent adverse performance on resource-constrained homeservers. (#5783)
  • Allow defining HTML templates to serve the user on account renewal attempt when using the account validity feature. (#5807)

Bugfixes

  • Fix UISIs during homeserver outage. (#5693, #5789)
  • Fix stack overflow in server key lookup code. (#5724)
  • start.sh no longer uses deprecated cli option. (#5725)
  • Log when we receive an event receipt from an unexpected origin. (#5743)
  • Fix debian packaging scripts to correctly build sid packages. (#5775)
  • Correctly handle redactions of redactions. (#5788)
  • Return 404 instead of 403 when accessing /rooms/{roomId}/event/{eventId} for an event without the appropriate permissions. (#5798)
  • Fix check that tombstone is a state event in push rules. (#5804)
  • Fix error when trying to login as a deactivated user when using a worker to handle login. (#5806)
  • Fix bug where user /sync stream could get wedged in rare circumstances. (#5825)
  • The purge_remote_media.sh script was fixed. (#5839)

Deprecations and Removals

  • Synapse now no longer accepts the -v/--verbose, -f/--log-file, or --log-config command line flags, and removes the deprecated verbose and log_file configuration file options. Users of these options should migrate their options into the dedicated log configuration. (#5678, #5729)
  • Remove non-functional 'expire_access_token' setting. (#5782)

Internal Changes

  • Make Jaeger fully configurable. (#5694)
  • Add precautionary measures to prevent future abuse of window.opener in default welcome page. (#5695)
  • Reduce database IO usage by optimising queries for current membership. (#5706, #5738, #5746, #5752, #5770, #5774, #5792, #5793)
  • Improve caching when fetching get_filtered_current_state_ids. (#5713)
  • Don't accept opentracing data from clients. (#5715)
  • Speed up PostgreSQL unit tests in CI. (#5717)
  • Update the coding style document. (#5719)
  • Improve database query performance when recording retry intervals for remote hosts. (#5720)
  • Add a set of opentracing utils. (#5722)
  • Cache result of get_version_string to reduce overhead of /version federation requests. (#5730)
  • Return 'user_type' in admin API user endpoints results. (#5731)
  • Don't package the sytest test blacklist file. (#5733)
  • Replace uses of returnValue with plain return, as returnValue is not needed on Python 3. (#5736)
  • Blacklist some flakey tests in worker mode. (#5740)
  • Fix some error cases in the caching layer. (#5749)
  • Add a prometheus metric for pending cache lookups. (#5750)
  • Stop trying to fetch events with event_id=None. (#5753)
  • Convert RedactionTestCase to modern test style. (#5768)
  • Allow looping calls to be given arguments. (#5780)
  • Set the logs emitted when checking typing and presence timeouts to DEBUG level, not INFO. (#5785)
  • Remove DelayedCall debugging from the test suite, as it is no longer required in the vast majority of Synapse's tests. (#5787)
  • Remove some spurious exceptions from the logs where we failed to talk to a remote server. (#5790)
  • Improve performance when making .well-known requests by sharing the SSL options between requests. (#5794)
  • Disable codecov GitHub comments on PRs. (#5796)
  • Don't allow clients to send tombstone events that reference the room it's sent in. (#5801)
  • Deny redactions of events sent in a different room. (#5802)
  • Deny sending well known state types as non-state events. (#5805)
  • Handle incorrectly encoded query params correctly by returning a 400. (#5808)
  • Handle pusher being deleted during processing rather than logging an exception. (#5809)
  • Return 502 not 500 when failing to reach any remote server. (#5810)
  • Reduce global pauses in the events stream caused by expensive state resolution during persistence. (#5826)
  • Add a lower bound to well-known lookup cache time to avoid repeated lookups. (#5836)
  • Whitelist history visbility sytests in worker mode tests. (#5843)

Critical Security Update - Synapse 1.2.1 released

2019-07-26 — General — Neil Johnson

Today we release Synapse 1.2.1 as a critical security update. It contains patches relating to redactions and event federation. The patches address long standing bugs, and are not regressions specific to the previous version (1.2). All admins, regardless of current version, should upgrade asap.

This release includes four security fixes:

  • Prevent an attack where a federated server could send redactions for arbitrary events in v1 and v2 rooms. (#5767)
  • Prevent a denial-of-service attack where cycles of redaction events would make Synapse spin infinitely. Thanks to @lrizika:matrix.org for identifying and responsibly disclosing this issue. (0f2ecb961)
  • Prevent an attack where users could be joined or parted from public rooms without their consent. Thanks to Dylanger for identifying and responsibly disclosing this issue. (#5744)
  • Fix a vulnerability where a federated server could spoof read-receipts from users on other servers. Thanks to Dylanger for identifying this issue too. (#5743)

Additionally, the following fix was in Synapse 1.2.0, but was not correctly identified during the original release:

  • It was possible for a room moderator to send a redaction for an m.room.create event, which would downgrade the room to version 1. Thanks to @/dev/ponies:ponies.im for identifying and responsibly disclosing this issue! (#5701)

You can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Alternatively check out our Synapse installation guide page

Thanks for bearing with us.

Synapse 1.2.0 released

2019-07-25 — General — Neil Johnson

Hey hey, Synapse 1.2.0 is here. It contains aggregations support, better error handling for deactivated accounts and some important bug fixes for redacting messages. Special thanks to community members skalarproduktraum and Lrizika for submissions to improve our documentation.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our Synapse installation guide page

The changelog since 1.1.0 follows:

Synapse 1.2.0 (2019-07-25)

No significant changes.

Synapse 1.2.0rc2 (2019-07-24)

Bugfixes

  • Fix a regression introduced in v1.2.0rc1 which led to incorrect labels on some prometheus metrics. (#5734)

Synapse 1.2.0rc1 (2019-07-22)

Features

  • Add support for opentracing. (#5544, #5712)
  • Add ability to pull all locally stored events out of synapse that a particular user can see. (#5589)
  • Add a basic admin command app to allow server operators to run Synapse admin commands separately from the main production instance. (#5597)
  • Add sender and origin_server_ts fields to m.replace. (#5613)
  • Add default push rule to ignore reactions. (#5623)
  • Include the original event when asking for its relations. (#5626)
  • Implement session_lifetime configuration option, after which access tokens will expire. (#5660)
  • Return "This account has been deactivated" when a deactivated user tries to login. (#5674)
  • Enable aggregations support by default (#5714)

Bugfixes

  • Fix 'utime went backwards' errors on daemonization. (#5609)
  • Various minor fixes to the federation request rate limiter. (#5621)
  • Forbid viewing relations on an event once it has been redacted. (#5629)
  • Fix requests to the /store_invite endpoint of identity servers being sent in the wrong format. (#5638)
  • Fix newly-registered users not being able to lookup their own profile without joining a room. (#5644)
  • Fix bug in #5626 that prevented the original_event field from actually having the contents of the original event in a call to /relations. (#5654)
  • Fix 3PID bind requests being sent to identity servers as application/x-form-www-urlencoded data, which is deprecated. (#5658)
  • Fix some problems with authenticating redactions in recent room versions. (#5699, #5700, #5707)
  • Ignore redactions of m.room.create events. (#5701)

Updates to the Docker image

  • Base Docker image on a newer Alpine Linux version (3.8 -> 3.10). (#5619)
  • Add missing space in default logging file format generated by the Docker image. (#5620)

Improved Documentation

  • Add information about nginx normalisation to reverse_proxy.rst. Contributed by @skalarproduktraum - thanks! (#5397)
  • --no-pep517 should be --no-use-pep517 in the documentation to setup the development environment. (#5651)
  • Improvements to Postgres setup instructions. Contributed by @Lrizika - thanks! (#5661)
  • Minor tweaks to postgres documentation. (#5675)

Deprecations and Removals

  • Remove support for the invite_3pid_guest configuration setting. (#5625)

Internal Changes

  • Move logging code out of synapse.util and into synapse.logging. (#5606, #5617)
  • Add a blacklist file to the repo to blacklist certain sytests from failing CI. (#5611)
  • Make runtime errors surrounding password reset emails much clearer. (#5616)
  • Remove dead code for persiting outgoing federation transactions. (#5622)
  • Add lint.sh to the scripts-dev folder which will run all linting steps required by CI. (#5627)
  • Move RegistrationHandler.get_or_create_user to test code. (#5628)
  • Add some more common python virtual-environment paths to the black exclusion list. (#5630)
  • Some counter metrics exposed over Prometheus have been renamed, with the old names preserved for backwards compatibility and deprecated. See docs/metrics-howto.rst for details. (#5636)
  • Unblacklist some user_directory sytests. (#5637)
  • Factor out some redundant code in the login implementation. (#5639)
  • Update ModuleApi to avoid register(generate_token=True). (#5640)
  • Remove access-token support from RegistrationHandler.register, and rename it. (#5641)
  • Remove access-token support from RegistrationStore.register, and rename it. (#5642)
  • Improve logging for auto-join when a new user is created. (#5643)
  • Remove unused and unnecessary check for FederationDeniedError in _exception_to_failure. (#5645)
  • Fix a small typo in a code comment. (#5655)
  • Clean up exception handling around client access tokens. (#5656)
  • Add a mechanism for per-test homeserver configuration in the unit tests. (#5657)
  • Inline issue_access_token. (#5659)
  • Update the sytest BuildKite configuration to checkout Synapse in /src. (#5664)
  • Add a docker type to the towncrier configuration. (#5673)
  • Convert synapse.federation.transport.server to async. Might improve some stack traces. (#5689)
  • Documentation for opentracing. (#5703)

5-user Matrix homeserver hosting now available from Modular

2019-07-17 — General — Modular.im

Hi all,

If you’ve been looking for a way to have you own Matrix homeserver without having to run it yourself, you may be interested to hear that Modular (the Matrix hosting provider run by New Vector, the startup which hires many of the Matrix core team) is now offering a personal-sized small homeserver hosting service, supporting a minimum size of 5 user servers.

A lot of recent performance work on Synapse has been driven by the need to make smaller dedicated servers more efficient to run - and so if you run your own homeserver you’ll be benefiting from all this work too :) Meanwhile, if you choose to outsource your server hosting to Modular, you’ll be indirectly supporting core Matrix and Synapse development, given most of the core Matrix team work for New Vector - it’s through buying services like this which lets us keep folks able to hack on Matrix as their day job.

See more details over at the Modular blog post!

Synapse 1.1.0 released

2019-07-04 — General — Neil Johnson

Right folks, this is our first post 1.0 release, which means that we have now officially dropped support for Python 2 and Postgres 9.4. This means that we can start making use of Python 3 specific features and you should expect lots of associated performance wins over the coming months. See the upgrade notes for more.

Synapse 1.1.0 also contains a reworked approach to the Docker image, as well lots of performance improvements with special focus on DB IO - expect more to come in this area.

Special thanks to community member Alexander Trost for rounding out our SAML support and also to Daniel Hoffend for contributing the ability to disable local password authentication.

As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Note, Synapse is now available from PyPI, pick it up here. Also, check out our Synapse installation guide page

The changelog since 1.0.0 follows:

Synapse 1.1.0 (2019-07-04)

As of v1.1.0, Synapse no longer supports Python 2, nor Postgres version 9.4. See the upgrade notes for more details.

This release also deprecates the use of environment variables to configure the docker image. See the docker README for more details.

No changes since 1.1.0rc2.

Synapse 1.1.0rc2 (2019-07-03)

Bugfixes

  • Fix regression in 1.1rc1 where OPTIONS requests to the media repo would fail. (#5593)
  • Removed the SYNAPSE_SMTP_* docker container environment variables. Using these environment variables prevented the docker container from starting in Synapse v1.0, even though they didn't actually allow any functionality anyway. (#5596)
  • Fix a number of "Starting txn from sentinel context" warnings. (#5605)

Internal Changes

  • Update github templates. (#5552)

Synapse 1.1.0rc1 (2019-07-02)

As of v1.1.0, Synapse no longer supports Python 2, nor Postgres version 9.4. See the upgrade notes for more details.

Features

  • Added possibilty to disable local password authentication. Contributed by Daniel Hoffend. (#5092)
  • Add monthly active users to phonehome stats. (#5252)
  • Allow expired user to trigger renewal email sending manually. (#5363)
  • Statistics on forward extremities per room are now exposed via Prometheus. (#5384, #5458, #5461)
  • Add --no-daemonize option to run synapse in the foreground, per issue #4130. Contributed by Soham Gumaste. (#5412, #5587)
  • Fully support SAML2 authentication. Contributed by Alexander Trost - thank you! (#5422)
  • Allow server admins to define implementations of extra rules for allowing or denying incoming events. (#5440, #5474, #5477)
  • Add support for handling pagination APIs on client reader worker. (#5505, #5513, #5531)
  • Improve help and cmdline option names for --generate-config options. (#5512)
  • Allow configuration of the path used for ACME account keys. (#5516, #5521, #5522)
  • Add --data-dir and --open-private-ports options. (#5524)
  • Split public rooms directory auth config in two settings, in order to manage client auth independently from the federation part of it. Obsoletes the "restrict_public_rooms_to_local_users" configuration setting. If "restrict_public_rooms_to_local_users" is set in the config, Synapse will act as if both new options are enabled, i.e. require authentication through the client API and deny federation requests. (#5534)
  • The minimum TLS version used for outgoing federation requests can now be set with federation_client_minimum_tls_version. (#5550)
  • Optimise devices changed query to not pull unnecessary rows from the database, reducing database load. (#5559)
  • Add new metrics for number of forward extremities being persisted and number of state groups involved in resolution. (#5476)

Bugfixes

  • Fix bug processing incoming events over federation if call to /get_missing_events fails. (#5042)
  • Prevent more than one room upgrade happening simultaneously on the same room. (#5051)
  • Fix a bug where running synapse_port_db would cause the account validity feature to fail because it didn't set the type of the email_sent column to boolean. (#5325)
  • Warn about disabling email-based password resets when a reset occurs, and remove warning when someone attempts a phone-based reset. (#5387)
  • Fix email notifications for unnamed rooms with multiple people. (#5388)
  • Fix exceptions in federation reader worker caused by attempting to renew attestations, which should only happen on master worker. (#5389)
  • Fix handling of failures fetching remote content to not log failures as exceptions. (#5390)
  • Fix a bug where deactivated users could receive renewal emails if the account validity feature is on. (#5394)
  • Fix missing invite state after exchanging 3PID invites over federaton. (#5464)
  • Fix intermittent exceptions on Apple hardware. Also fix bug that caused database activity times to be under-reported in log lines. (#5498)
  • Fix logging error when a tampered event is detected. (#5500)
  • Fix bug where clients could tight loop calling /sync for a period. (#5507)
  • Fix bug with jinja2 preventing Synapse from starting. Users who had this problem should now simply need to run pip install matrix-synapse. (#5514)
  • Fix a regression where homeservers on private IP addresses were incorrectly blacklisted. (#5523)
  • Fixed m.login.jwt using unregistred user_id and added pyjwt>=1.6.4 as jwt conditional dependencies. Contributed by Pau Rodriguez-Estivill. (#5555, #5586)
  • Fix a bug that would cause invited users to receive several emails for a single 3PID invite in case the inviter is rate limited. (#5576)

Updates to the Docker image

  • Add ability to change Docker containers timezone with the TZ variable. (#5383)
  • Update docker image to use Python 3.7. (#5546)
  • Deprecate the use of environment variables for configuration, and make the use of a static configuration the default. (#5561, #5562, #5566, #5567)
  • Increase default log level for docker image to INFO. It can still be changed by editing the generated log.config file. (#5547)
  • Send synapse logs to the docker logging system, by default. (#5565)
  • Open the non-TLS port by default. (#5568)
  • Fix failure to start under docker with SAML support enabled. (#5490)
  • Use a sensible location for data files when generating a config file. (#5563)

Deprecations and Removals

  • Python 2.7 is no longer a supported platform. Synapse now requires Python 3.5+ to run. (#5425)
  • PostgreSQL 9.4 is no longer supported. Synapse requires Postgres 9.5+ or above for Postgres support. (#5448)
  • Remove support for cpu_affinity setting. (#5525)

Improved Documentation

  • Improve README section on performance troubleshooting. (#4276)
  • Add information about how to install and run black on the codebase to code_style.rst. (#5537)
  • Improve install docs on choosing server_name. (#5558)

Internal Changes

  • Add logging to 3pid invite signature verification. (#5015)
  • Update example haproxy config to a more compatible setup. (#5313)
  • Track deactivated accounts in the database. (#5378, #5465, #5493)
  • Clean up code for sending federation EDUs. (#5381)
  • Add a sponsor button to the repo. (#5382, #5386)
  • Don't log non-200 responses from federation queries as exceptions. (#5383)
  • Update Python syntax in contrib/ to Python 3. (#5446)
  • Update federation_client dev script to support .well-known and work with python3. (#5447)
  • SyTest has been moved to Buildkite. (#5459)
  • Demo script now uses python3. (#5460)
  • Synapse can now handle RestServlets that return coroutines. (#5475, #5585)
  • The demo servers talk to each other again. (#5478)
  • Add an EXPERIMENTAL config option to try and periodically clean up extremities by sending dummy events. (#5480)
  • Synapse's codebase is now formatted by black. (#5482)
  • Some cleanups and sanity-checking in the CPU and database metrics. (#5499)
  • Improve email notification logging. (#5502)
  • Fix "Unexpected entry in 'full_schemas'" log warning. (#5509)
  • Improve logging when generating config files. (#5510)
  • Refactor and clean up Config parser for maintainability. (#5511)
  • Make the config clearer in that email.template_dir is relative to the Synapse's root directory, not the synapse/ folder within it. (#5543)
  • Update v1.0.0 release changelog to include more information about changes to password resets. (#5545)
  • Remove non-functioning check_event_hash.py dev script. (#5548)
  • Synapse will now only allow TLS v1.2 connections when serving federation, if it terminates TLS. As Synapse's allowed ciphers were only able to be used in TLSv1.2 before, this does not change behaviour. (#5550)
  • Logging when running GC collection on generation 0 is now at the DEBUG level, not INFO. (#5557)
  • Reduce the amount of stuff we send in the docker context. (#5564)
  • Point the reverse links in the Purge History contrib scripts at the intended location. (#5570)
NextPage 2