Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Fix 500 error with Postgres when looking backwards with the MSC3030 /timestamp_to_event endpoint #12024

Conversation

MadLittleMods
Copy link
Contributor

@MadLittleMods MadLittleMods commented Feb 18, 2022

Fix 500 error with Postgres when looking backwards with the MSC3030 /timestamp_to_event endpoint.

== is a valid comparison operator in SQLite but not in Postgres.

This wasn't caught in the MSC3030 Complement tests because they use SQLite, see https://github.com/matrix-org/complement/blob/18aa82f2635cb5f69d720adb4b6bb722debb7c47/dockerfiles/synapse/homeserver.yaml#L34-L35

Sources

SQLite says:

Equals can be either = or ==.

-- https://sqlite.org/lang_expr.html

But for Postgres, https://www.postgresql.org/docs/8.2/functions-comparison.html, only lists =

Example 500 error

Example Synapse 500 error when using /_matrix/client/unstable/org.matrix.msc3030/rooms/!HBehERstyQBxyJDLfR:my.synapse.server/timestamp_to_event?ts=1644278400000&dir=b:

2022-02-17 22:05:00,116 - synapse.http.server - 100 - ERROR - GET-14 - Failed handle request via 'TimestampLookupRestServlet': <XForwardedForRequest at 0x10e498eb0 method='GET' uri='/_matrix/client/unstable/org.matrix.msc3030/rooms/!HBehERstyQBxyJDLfR:my.synapse.server/timestamp_to_event?ts=1644451200000&dir=b' clientproto='HTTP/1.1' site='8008'>
Traceback (most recent call last):
  File "/Users/eric/Documents/github/element/synapse/synapse/http/server.py", line 269, in _async_render_wrapper
    callback_return = await self._async_render(request)
  File "/Users/eric/Documents/github/element/synapse/synapse/http/server.py", line 471, in _async_render
    callback_return = await raw_callback_return
  File "/Users/eric/Documents/github/element/synapse/synapse/rest/client/room.py", line 1134, in on_GET
    ) = await self.timestamp_lookup_handler.get_event_for_timestamp(
  File "/Users/eric/Documents/github/element/synapse/synapse/handlers/room.py", line 1308, in get_event_for_timestamp
    await self.store.is_event_next_to_forward_gap(local_event)
  File "/Users/eric/Documents/github/element/synapse/synapse/storage/databases/main/events_worker.py", line 1891, in is_event_next_to_forward_gap
    return await self.db_pool.runInteraction(
  File "/Users/eric/Documents/github/element/synapse/synapse/storage/database.py", line 743, in runInteraction
    result = await self.runWithConnection(
  File "/Users/eric/Documents/github/element/synapse/synapse/storage/database.py", line 857, in runWithConnection
    return await make_deferred_yieldable(
  File "/Users/eric/Documents/github/element/synapse/env/lib/python3.9/site-packages/twisted/python/threadpool.py", line 238, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/Users/eric/Documents/github/element/synapse/env/lib/python3.9/site-packages/twisted/python/threadpool.py", line 254, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/Users/eric/Documents/github/element/synapse/env/lib/python3.9/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/Users/eric/Documents/github/element/synapse/env/lib/python3.9/site-packages/twisted/python/context.py", line 83, in callWithContext
    return func(*args, **kw)
  File "/Users/eric/Documents/github/element/synapse/env/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection
    compat.reraise(excValue, excTraceback)
  File "/Users/eric/Documents/github/element/synapse/env/lib/python3.9/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction
    return function(*args, **kwargs)
  File "/Users/eric/Documents/github/element/synapse/env/lib/python3.9/site-packages/twisted/python/compat.py", line 404, in reraise
    raise exception.with_traceback(traceback)
  File "/Users/eric/Documents/github/element/synapse/env/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/Users/eric/Documents/github/element/synapse/synapse/storage/database.py", line 850, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/Users/eric/Documents/github/element/synapse/synapse/storage/database.py", line 609, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/Users/eric/Documents/github/element/synapse/synapse/storage/databases/main/events_worker.py", line 1884, in is_event_next_to_gap_txn
    txn.execute(forward_edge_query, (event.room_id, event.event_id))
  File "/Users/eric/Documents/github/element/synapse/synapse/storage/database.py", line 310, in execute
    self._do_execute(self.txn.execute, sql, *args)
  File "/Users/eric/Documents/github/element/synapse/synapse/storage/database.py", line 343, in _do_execute
    return func(sql, *args)
psycopg2.errors.UndefinedFunction: operator does not exist: text == text
LINE 1: ...d */ LEFT JOIN rejections ON event_edges.event_id == rejecti...
                                                             ^
HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.

Pull Request Checklist

  • Pull request is based on the develop branch
  • Pull request includes a changelog file. The entry should:
    • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
    • Use markdown where necessary, mostly for code blocks.
    • End with either a period (.) or an exclamation mark (!).
    • Start with a capital letter.
    • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
  • Pull request includes a sign off
  • Code style is correct
    (run the linters)

…timestamp_to_event endpoint

`==` is a valid comparison operator in SQLite but not in Postgres.

Sources:

SQLite says:

> Equals can be either = or ==.
>
> -- https://sqlite.org/lang_expr.html

But for Postgres, https://www.postgresql.org/docs/8.2/functions-comparison.html, only lists `=`

Example Synapse 500 error when using `/_matrix/client/unstable/org.matrix.msc3030/rooms/!HBehERstyQBxyJDLfR:my.synapse.server/timestamp_to_event?ts=1644278400000&dir=b`:

```
2022-02-17 22:05:00,116 - synapse.http.server - 100 - ERROR - GET-14 - Failed handle request via 'TimestampLookupRestServlet': <XForwardedForRequest at 0x10e498eb0 method='GET' uri='/_matrix/client/unstable/org.matrix.msc3030/rooms/!HBehERstyQBxyJDLfR:my.synapse.server/timestamp_to_event?ts=1644451200000&dir=b' clientproto='HTTP/1.1' site='8008'>
Traceback (most recent call last):
  File "/Users/eric/Documents/github/element/synapse/synapse/http/server.py", line 269, in _async_render_wrapper
    callback_return = await self._async_render(request)
  File "/Users/eric/Documents/github/element/synapse/synapse/http/server.py", line 471, in _async_render
    callback_return = await raw_callback_return
  File "/Users/eric/Documents/github/element/synapse/synapse/rest/client/room.py", line 1134, in on_GET
    ) = await self.timestamp_lookup_handler.get_event_for_timestamp(
  File "/Users/eric/Documents/github/element/synapse/synapse/handlers/room.py", line 1308, in get_event_for_timestamp
    await self.store.is_event_next_to_forward_gap(local_event)
  File "/Users/eric/Documents/github/element/synapse/synapse/storage/databases/main/events_worker.py", line 1891, in is_event_next_to_forward_gap
    return await self.db_pool.runInteraction(
  File "/Users/eric/Documents/github/element/synapse/synapse/storage/database.py", line 743, in runInteraction
    result = await self.runWithConnection(
  File "/Users/eric/Documents/github/element/synapse/synapse/storage/database.py", line 857, in runWithConnection
    return await make_deferred_yieldable(
  File "/Users/eric/Documents/github/element/synapse/env/lib/python3.9/site-packages/twisted/python/threadpool.py", line 238, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/Users/eric/Documents/github/element/synapse/env/lib/python3.9/site-packages/twisted/python/threadpool.py", line 254, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/Users/eric/Documents/github/element/synapse/env/lib/python3.9/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/Users/eric/Documents/github/element/synapse/env/lib/python3.9/site-packages/twisted/python/context.py", line 83, in callWithContext
    return func(*args, **kw)
  File "/Users/eric/Documents/github/element/synapse/env/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection
    compat.reraise(excValue, excTraceback)
  File "/Users/eric/Documents/github/element/synapse/env/lib/python3.9/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction
    return function(*args, **kwargs)
  File "/Users/eric/Documents/github/element/synapse/env/lib/python3.9/site-packages/twisted/python/compat.py", line 404, in reraise
    raise exception.with_traceback(traceback)
  File "/Users/eric/Documents/github/element/synapse/env/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/Users/eric/Documents/github/element/synapse/synapse/storage/database.py", line 850, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/Users/eric/Documents/github/element/synapse/synapse/storage/database.py", line 609, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/Users/eric/Documents/github/element/synapse/synapse/storage/databases/main/events_worker.py", line 1884, in is_event_next_to_gap_txn
    txn.execute(forward_edge_query, (event.room_id, event.event_id))
  File "/Users/eric/Documents/github/element/synapse/synapse/storage/database.py", line 310, in execute
    self._do_execute(self.txn.execute, sql, *args)
  File "/Users/eric/Documents/github/element/synapse/synapse/storage/database.py", line 343, in _do_execute
    return func(sql, *args)
psycopg2.errors.UndefinedFunction: operator does not exist: text == text
LINE 1: ...d */ LEFT JOIN rejections ON event_edges.event_id == rejecti...
                                                             ^
HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
```
@MadLittleMods MadLittleMods requested a review from a team as a code owner February 18, 2022 09:17
@MadLittleMods MadLittleMods removed the request for review from a team February 18, 2022 09:17
@MadLittleMods MadLittleMods added the T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. label Feb 18, 2022
@@ -1854,7 +1854,7 @@ def is_event_next_to_gap_txn(txn: LoggingTransaction) -> bool:
forward_edge_query = """
SELECT 1 FROM event_edges
/* Check to make sure the event referencing our event in question is not rejected */
LEFT JOIN rejections ON event_edges.event_id == rejections.event_id
LEFT JOIN rejections ON event_edges.event_id = rejections.event_id
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

== is a valid comparison operator in SQLite but not in Postgres.

@MadLittleMods MadLittleMods requested a review from a team February 18, 2022 09:25
Copy link
Contributor

@babolivier babolivier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quite worrying that Complement isn't able to catch this kind of things. Not something for this PR to solve though.

Edit: I've opened matrix-org/complement#323 to track progress on this.

@babolivier
Copy link
Contributor

CI failure is due to pallets/jinja#1587

@babolivier
Copy link
Contributor

I've merged develop into this PR's branch to pull #12025 which fixes the olddeps check.

@babolivier babolivier merged commit 5a69115 into develop Feb 18, 2022
@babolivier babolivier deleted the madlittlemods/fix-forward-gap-db-query-comparison-in-postgres branch February 18, 2022 12:11
@MadLittleMods
Copy link
Contributor Author

@babolivier Thanks for the review, merge, and raising that issue 🦈

reivilibre added a commit that referenced this pull request Mar 2, 2022
Synapse 1.54.0rc1 (2022-03-02)
==============================

Please note that this will be the last release of Synapse that is compatible with Mjolnir 1.3.1 and earlier.
Administrators of servers which have the Mjolnir module installed are advised to upgrade Mjolnir to version 1.3.2 or later.

Features
--------

- Add support for [MSC3202](matrix-org/matrix-spec-proposals#3202): sending one-time key counts and fallback key usage states to Application Services. ([\#11617](#11617))
- Improve the generated URL previews for some web pages. Contributed by @AndrewRyanChama. ([\#11985](#11985))
- Track cache invalidations in Prometheus metrics, as already happens for cache eviction based on size or time. ([\#12000](#12000))
- Implement experimental support for [MSC3720](matrix-org/matrix-spec-proposals#3720) (account status endpoints). ([\#12001](#12001), [\#12067](#12067))
- Enable modules to set a custom display name when registering a user. ([\#12009](#12009))
- Advertise Matrix 1.1 and 1.2 support on `/_matrix/client/versions`. ([\#12020](#12020), ([\#12022](#12022))
- Support only the stable identifier for [MSC3069](matrix-org/matrix-spec-proposals#3069 `is_guest` on `/_matrix/client/v3/account/whoami`. ([\#12021](#12021))
- Use room version 9 as the default room version (per [MSC3589](matrix-org/matrix-spec-proposals#3589)). ([\#12058](#12058))
- Add module callbacks to react to user deactivation status changes (i.e. deactivations and reactivations) and profile updates. ([\#12062](#12062))

Bugfixes
--------

- Fix a bug introduced in Synapse 1.48.0 where an edit of the latest event in a thread would not be properly applied to the thread summary. ([\#11992](#11992))
- Fix long-standing bug where the `get_rooms_for_user` cache was not correctly invalidated for remote users when the server left a room. ([\#11999](#11999))
- Fix a 500 error with Postgres when looking backwards with the [MSC3030](matrix-org/matrix-spec-proposals#3030) `/timestamp_to_event?dir=b` endpoint. ([\#12024](#12024))
- Properly fix a long-standing bug where wrong data could be inserted into the `event_search` table when using SQLite. This could block running `synapse_port_db` with an `argument of type 'int' is not iterable` error. This bug was partially fixed by a change in Synapse 1.44.0. ([\#12037](#12037))
- Fix slow performance of `/logout` in some cases where refresh tokens are in use. The slowness existed since the initial implementation of refresh tokens in version 1.38.0. ([\#12056](#12056))
- Fix a long-standing bug where Synapse would make additional failing requests over federation for missing data. ([\#12077](#12077))
- Fix occasional `Unhandled error in Deferred` error message. ([\#12089](#12089))
- Fix a bug introduced in Synapse 1.51.0 where incoming federation transactions containing at least one EDU would be dropped if debug logging was enabled for `synapse.8631_debug`. ([\#12098](#12098))
- Fix a long-standing bug which could cause push notifications to malfunction if `use_frozen_dicts` was set in the configuration. ([\#12100](#12100))
- Fix an extremely rare, long-standing bug in `ReadWriteLock` that would cause an error when a newly unblocked writer completes instantly. ([\#12105](#12105))
- Make a `POST` to `/rooms/<room_id>/receipt/m.read/<event_id>` only trigger a push notification if the count of unread messages is different to the one in the last successfully sent push. This reduces server load and load on the receiving device. ([\#11835](#11835))

Updates to the Docker image
---------------------------

- The Docker image no longer automatically creates a temporary volume at `/data`. This is not expected to affect normal usage. ([\#11997](#11997))
- Use Python 3.9 in Docker images by default. ([\#12112](#12112))

Improved Documentation
----------------------

- Document support for the `to_device`, `account_data`, `receipts`, and `presence` stream writers for workers. ([\#11599](#11599))
- Explain the meaning of spam checker callbacks' return values. ([\#12003](#12003))
- Clarify information about external Identity Provider IDs. ([\#12004](#12004))

Deprecations and Removals
-------------------------

- Deprecate using `synctl` with the config option `synctl_cache_factor` and print a warning if a user still uses this option. ([\#11865](#11865))
- Remove support for the legacy structured logging configuration (please see the the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#legacy-structured-logging-configuration-removal) if you are using `structured: true` in the Synapse configuration). ([\#12008](#12008))
- Drop support for [MSC3283](matrix-org/matrix-spec-proposals#3283) unstable flags now that the stable flags are supported. ([\#12018](#12018))
- Remove the unstable `/spaces` endpoint from [MSC2946](matrix-org/matrix-spec-proposals#2946). ([\#12073](#12073))

Internal Changes
----------------

- Make the `get_room_version` method use `get_room_version_id` to benefit from caching. ([\#11808](#11808))
- Remove unnecessary condition on knock -> leave auth rule check. ([\#11900](#11900))
- Add tests for device list changes between local users. ([\#11972](#11972))
- Optimise calculating `device_list` changes in `/sync`. ([\#11974](#11974))
- Add missing type hints to storage classes. ([\#11984](#11984))
- Refactor the search code for improved readability. ([\#11991](#11991))
- Move common deduplication code down into `_auth_and_persist_outliers`. ([\#11994](#11994))
- Limit concurrent joins from applications services. ([\#11996](#11996))
- Preparation for faster-room-join work: when parsing the `send_join` response, get the `m.room.create` event from `state`, not `auth_chain`. ([\#12005](#12005), [\#12039](#12039))
- Preparation for faster-room-join work: parse MSC3706 fields in send_join response. ([\#12011](#12011))
- Preparation for faster-room-join work: persist information on which events and rooms have partial state to the database. ([\#12012](#12012))
- Preparation for faster-room-join work: Support for calling `/federation/v1/state` on a remote server. ([\#12013](#12013))
- Configure `tox` to use `venv` rather than `virtualenv`. ([\#12015](#12015))
- Fix bug in `StateFilter.return_expanded()` and add some tests. ([\#12016](#12016))
- Use Matrix v1.1 endpoints (`/_matrix/client/v3/auth/...`) in fallback auth HTML forms. ([\#12019](#12019))
- Update the `olddeps` CI job to use an old version of `markupsafe`. ([\#12025](#12025))
- Upgrade Mypy to version 0.931. ([\#12030](#12030))
- Remove legacy `HomeServer.get_datastore()`. ([\#12031](#12031), [\#12070](#12070))
- Minor typing fixes. ([\#12034](#12034), [\#12069](#12069))
- After joining a room, create a dedicated logcontext to process the queued events. ([\#12041](#12041))
- Tidy up GitHub Actions config which builds distributions for PyPI. ([\#12051](#12051))
- Move configuration out of `setup.cfg`. ([\#12052](#12052), [\#12059](#12059))
- Fix error message when a worker process fails to talk to another worker process. ([\#12060](#12060))
- Fix using the `complement.sh` script without specifying a directory or a branch. Contributed by Nico on behalf of Famedly. ([\#12063](#12063))
- Add type hints to `tests/rest/client`. ([\#12066](#12066), [\#12072](#12072), [\#12084](#12084), [\#12094](#12094))
- Add some logging to `/sync` to try and track down #11916. ([\#12068](#12068))
- Inspect application dependencies using `importlib.metadata` or its backport. ([\#12088](#12088))
- Use `assertEqual` instead of the deprecated `assertEquals` in test code. ([\#12092](#12092))
- Move experimental support for [MSC3440](matrix-org/matrix-spec-proposals#3440) to `/versions`. ([\#12099](#12099))
- Add `stop_cancellation` utility function to stop `Deferred`s from being cancelled. ([\#12106](#12106))
- Improve exception handling for concurrent execution. ([\#12109](#12109))
- Advertise support for Python 3.10 in packaging files. ([\#12111](#12111))
- Move CI checks out of tox, to facilitate a move to using poetry. ([\#12119](#12119))
Fizzadar added a commit to Fizzadar/synapse that referenced this pull request Apr 25, 2022
Synapse 1.54.0 (2022-03-08)
===========================

Please note that this will be the last release of Synapse that is compatible with Mjolnir 1.3.1 and earlier.
Administrators of servers which have the Mjolnir module installed are advised to upgrade Mjolnir to version 1.3.2 or later.

Bugfixes
--------

- Fix a bug introduced in Synapse 1.54.0rc1 preventing the new module callbacks introduced in this release from being registered by modules. ([\matrix-org#12141](matrix-org#12141))
- Fix a bug introduced in Synapse 1.54.0rc1 where runtime dependency version checks would mistakenly check development dependencies if they were present and would not accept pre-release versions of dependencies. ([\matrix-org#12129](matrix-org#12129), [\matrix-org#12177](matrix-org#12177))

Internal Changes
----------------

- Update release script to insert the previous version when writing "No significant changes" line in the changelog. ([\matrix-org#12127](matrix-org#12127))
- Relax the version guard for "packaging" added in [\matrix-org#12088](matrix-org#12088). ([\matrix-org#12166](matrix-org#12166))

Synapse 1.54.0rc1 (2022-03-02)
==============================

Features
--------

- Add support for [MSC3202](matrix-org/matrix-spec-proposals#3202): sending one-time key counts and fallback key usage states to Application Services. ([\matrix-org#11617](matrix-org#11617))
- Improve the generated URL previews for some web pages. Contributed by @AndrewRyanChama. ([\matrix-org#11985](matrix-org#11985))
- Track cache invalidations in Prometheus metrics, as already happens for cache eviction based on size or time. ([\matrix-org#12000](matrix-org#12000))
- Implement experimental support for [MSC3720](matrix-org/matrix-spec-proposals#3720) (account status endpoints). ([\matrix-org#12001](matrix-org#12001), [\matrix-org#12067](matrix-org#12067))
- Enable modules to set a custom display name when registering a user. ([\matrix-org#12009](matrix-org#12009))
- Advertise Matrix 1.1 and 1.2 support on `/_matrix/client/versions`. ([\matrix-org#12020](matrix-org#12020), ([\matrix-org#12022](matrix-org#12022))
- Support only the stable identifier for [MSC3069](matrix-org/matrix-spec-proposals#3069 `is_guest` on `/_matrix/client/v3/account/whoami`. ([\matrix-org#12021](matrix-org#12021))
- Use room version 9 as the default room version (per [MSC3589](matrix-org/matrix-spec-proposals#3589)). ([\matrix-org#12058](matrix-org#12058))
- Add module callbacks to react to user deactivation status changes (i.e. deactivations and reactivations) and profile updates. ([\matrix-org#12062](matrix-org#12062))

Bugfixes
--------

- Fix a bug introduced in Synapse 1.48.0 where an edit of the latest event in a thread would not be properly applied to the thread summary. ([\matrix-org#11992](matrix-org#11992))
- Fix long-standing bug where the `get_rooms_for_user` cache was not correctly invalidated for remote users when the server left a room. ([\matrix-org#11999](matrix-org#11999))
- Fix a 500 error with Postgres when looking backwards with the [MSC3030](matrix-org/matrix-spec-proposals#3030) `/timestamp_to_event?dir=b` endpoint. ([\matrix-org#12024](matrix-org#12024))
- Properly fix a long-standing bug where wrong data could be inserted into the `event_search` table when using SQLite. This could block running `synapse_port_db` with an `argument of type 'int' is not iterable` error. This bug was partially fixed by a change in Synapse 1.44.0. ([\matrix-org#12037](matrix-org#12037))
- Fix slow performance of `/logout` in some cases where refresh tokens are in use. The slowness existed since the initial implementation of refresh tokens in version 1.38.0. ([\matrix-org#12056](matrix-org#12056))
- Fix a long-standing bug where Synapse would make additional failing requests over federation for missing data. ([\matrix-org#12077](matrix-org#12077))
- Fix occasional `Unhandled error in Deferred` error message. ([\matrix-org#12089](matrix-org#12089))
- Fix a bug introduced in Synapse 1.51.0 where incoming federation transactions containing at least one EDU would be dropped if debug logging was enabled for `synapse.8631_debug`. ([\matrix-org#12098](matrix-org#12098))
- Fix a long-standing bug which could cause push notifications to malfunction if `use_frozen_dicts` was set in the configuration. ([\matrix-org#12100](matrix-org#12100))
- Fix an extremely rare, long-standing bug in `ReadWriteLock` that would cause an error when a newly unblocked writer completes instantly. ([\matrix-org#12105](matrix-org#12105))
- Make a `POST` to `/rooms/<room_id>/receipt/m.read/<event_id>` only trigger a push notification if the count of unread messages is different to the one in the last successfully sent push. This reduces server load and load on the receiving device. ([\matrix-org#11835](matrix-org#11835))

Updates to the Docker image
---------------------------

- The Docker image no longer automatically creates a temporary volume at `/data`. This is not expected to affect normal usage. ([\matrix-org#11997](matrix-org#11997))
- Use Python 3.9 in Docker images by default. ([\matrix-org#12112](matrix-org#12112))

Improved Documentation
----------------------

- Document support for the `to_device`, `account_data`, `receipts`, and `presence` stream writers for workers. ([\matrix-org#11599](matrix-org#11599))
- Explain the meaning of spam checker callbacks' return values. ([\matrix-org#12003](matrix-org#12003))
- Clarify information about external Identity Provider IDs. ([\matrix-org#12004](matrix-org#12004))

Deprecations and Removals
-------------------------

- Deprecate using `synctl` with the config option `synctl_cache_factor` and print a warning if a user still uses this option. ([\matrix-org#11865](matrix-org#11865))
- Remove support for the legacy structured logging configuration (please see the the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#legacy-structured-logging-configuration-removal) if you are using `structured: true` in the Synapse configuration). ([\matrix-org#12008](matrix-org#12008))
- Drop support for [MSC3283](matrix-org/matrix-spec-proposals#3283) unstable flags now that the stable flags are supported. ([\matrix-org#12018](matrix-org#12018))
- Remove the unstable `/spaces` endpoint from [MSC2946](matrix-org/matrix-spec-proposals#2946). ([\matrix-org#12073](matrix-org#12073))

Internal Changes
----------------

- Make the `get_room_version` method use `get_room_version_id` to benefit from caching. ([\matrix-org#11808](matrix-org#11808))
- Remove unnecessary condition on knock -> leave auth rule check. ([\matrix-org#11900](matrix-org#11900))
- Add tests for device list changes between local users. ([\matrix-org#11972](matrix-org#11972))
- Optimise calculating `device_list` changes in `/sync`. ([\matrix-org#11974](matrix-org#11974))
- Add missing type hints to storage classes. ([\matrix-org#11984](matrix-org#11984))
- Refactor the search code for improved readability. ([\matrix-org#11991](matrix-org#11991))
- Move common deduplication code down into `_auth_and_persist_outliers`. ([\matrix-org#11994](matrix-org#11994))
- Limit concurrent joins from applications services. ([\matrix-org#11996](matrix-org#11996))
- Preparation for faster-room-join work: when parsing the `send_join` response, get the `m.room.create` event from `state`, not `auth_chain`. ([\matrix-org#12005](matrix-org#12005), [\matrix-org#12039](matrix-org#12039))
- Preparation for faster-room-join work: parse MSC3706 fields in send_join response. ([\matrix-org#12011](matrix-org#12011))
- Preparation for faster-room-join work: persist information on which events and rooms have partial state to the database. ([\matrix-org#12012](matrix-org#12012))
- Preparation for faster-room-join work: Support for calling `/federation/v1/state` on a remote server. ([\matrix-org#12013](matrix-org#12013))
- Configure `tox` to use `venv` rather than `virtualenv`. ([\matrix-org#12015](matrix-org#12015))
- Fix bug in `StateFilter.return_expanded()` and add some tests. ([\matrix-org#12016](matrix-org#12016))
- Use Matrix v1.1 endpoints (`/_matrix/client/v3/auth/...`) in fallback auth HTML forms. ([\matrix-org#12019](matrix-org#12019))
- Update the `olddeps` CI job to use an old version of `markupsafe`. ([\matrix-org#12025](matrix-org#12025))
- Upgrade Mypy to version 0.931. ([\matrix-org#12030](matrix-org#12030))
- Remove legacy `HomeServer.get_datastore()`. ([\matrix-org#12031](matrix-org#12031), [\matrix-org#12070](matrix-org#12070))
- Minor typing fixes. ([\matrix-org#12034](matrix-org#12034), [\matrix-org#12069](matrix-org#12069))
- After joining a room, create a dedicated logcontext to process the queued events. ([\matrix-org#12041](matrix-org#12041))
- Tidy up GitHub Actions config which builds distributions for PyPI. ([\matrix-org#12051](matrix-org#12051))
- Move configuration out of `setup.cfg`. ([\matrix-org#12052](matrix-org#12052), [\matrix-org#12059](matrix-org#12059))
- Fix error message when a worker process fails to talk to another worker process. ([\matrix-org#12060](matrix-org#12060))
- Fix using the `complement.sh` script without specifying a directory or a branch. Contributed by Nico on behalf of Famedly. ([\matrix-org#12063](matrix-org#12063))
- Add type hints to `tests/rest/client`. ([\matrix-org#12066](matrix-org#12066), [\matrix-org#12072](matrix-org#12072), [\matrix-org#12084](matrix-org#12084), [\matrix-org#12094](matrix-org#12094))
- Add some logging to `/sync` to try and track down matrix-org#11916. ([\matrix-org#12068](matrix-org#12068))
- Inspect application dependencies using `importlib.metadata` or its backport. ([\matrix-org#12088](matrix-org#12088))
- Use `assertEqual` instead of the deprecated `assertEquals` in test code. ([\matrix-org#12092](matrix-org#12092))
- Move experimental support for [MSC3440](matrix-org/matrix-spec-proposals#3440) to `/versions`. ([\matrix-org#12099](matrix-org#12099))
- Add `stop_cancellation` utility function to stop `Deferred`s from being cancelled. ([\matrix-org#12106](matrix-org#12106))
- Improve exception handling for concurrent execution. ([\matrix-org#12109](matrix-org#12109))
- Advertise support for Python 3.10 in packaging files. ([\matrix-org#12111](matrix-org#12111))
- Move CI checks out of tox, to facilitate a move to using poetry. ([\matrix-org#12119](matrix-org#12119))
babolivier added a commit to matrix-org/synapse-dinsic that referenced this pull request Apr 28, 2022
Synapse 1.54.0 (2022-03-08)
===========================

Please note that this will be the last release of Synapse that is compatible with Mjolnir 1.3.1 and earlier.
Administrators of servers which have the Mjolnir module installed are advised to upgrade Mjolnir to version 1.3.2 or later.

Bugfixes
--------

- Fix a bug introduced in Synapse 1.54.0rc1 preventing the new module callbacks introduced in this release from being registered by modules. ([\#12141](matrix-org/synapse#12141))
- Fix a bug introduced in Synapse 1.54.0rc1 where runtime dependency version checks would mistakenly check development dependencies if they were present and would not accept pre-release versions of dependencies. ([\#12129](matrix-org/synapse#12129), [\#12177](matrix-org/synapse#12177))

Internal Changes
----------------

- Update release script to insert the previous version when writing "No significant changes" line in the changelog. ([\#12127](matrix-org/synapse#12127))
- Relax the version guard for "packaging" added in [\#12088](matrix-org/synapse#12088). ([\#12166](matrix-org/synapse#12166))

Synapse 1.54.0rc1 (2022-03-02)
==============================

Features
--------

- Add support for [MSC3202](matrix-org/matrix-spec-proposals#3202): sending one-time key counts and fallback key usage states to Application Services. ([\#11617](matrix-org/synapse#11617))
- Improve the generated URL previews for some web pages. Contributed by @AndrewRyanChama. ([\#11985](matrix-org/synapse#11985))
- Track cache invalidations in Prometheus metrics, as already happens for cache eviction based on size or time. ([\#12000](matrix-org/synapse#12000))
- Implement experimental support for [MSC3720](matrix-org/matrix-spec-proposals#3720) (account status endpoints). ([\#12001](matrix-org/synapse#12001), [\#12067](matrix-org/synapse#12067))
- Enable modules to set a custom display name when registering a user. ([\#12009](matrix-org/synapse#12009))
- Advertise Matrix 1.1 and 1.2 support on `/_matrix/client/versions`. ([\#12020](matrix-org/synapse#12020), ([\#12022](matrix-org/synapse#12022))
- Support only the stable identifier for [MSC3069](matrix-org/matrix-spec-proposals#3069 `is_guest` on `/_matrix/client/v3/account/whoami`. ([\#12021](matrix-org/synapse#12021))
- Use room version 9 as the default room version (per [MSC3589](matrix-org/matrix-spec-proposals#3589)). ([\#12058](matrix-org/synapse#12058))
- Add module callbacks to react to user deactivation status changes (i.e. deactivations and reactivations) and profile updates. ([\#12062](matrix-org/synapse#12062))

Bugfixes
--------

- Fix a bug introduced in Synapse 1.48.0 where an edit of the latest event in a thread would not be properly applied to the thread summary. ([\#11992](matrix-org/synapse#11992))
- Fix long-standing bug where the `get_rooms_for_user` cache was not correctly invalidated for remote users when the server left a room. ([\#11999](matrix-org/synapse#11999))
- Fix a 500 error with Postgres when looking backwards with the [MSC3030](matrix-org/matrix-spec-proposals#3030) `/timestamp_to_event?dir=b` endpoint. ([\#12024](matrix-org/synapse#12024))
- Properly fix a long-standing bug where wrong data could be inserted into the `event_search` table when using SQLite. This could block running `synapse_port_db` with an `argument of type 'int' is not iterable` error. This bug was partially fixed by a change in Synapse 1.44.0. ([\#12037](matrix-org/synapse#12037))
- Fix slow performance of `/logout` in some cases where refresh tokens are in use. The slowness existed since the initial implementation of refresh tokens in version 1.38.0. ([\#12056](matrix-org/synapse#12056))
- Fix a long-standing bug where Synapse would make additional failing requests over federation for missing data. ([\#12077](matrix-org/synapse#12077))
- Fix occasional `Unhandled error in Deferred` error message. ([\#12089](matrix-org/synapse#12089))
- Fix a bug introduced in Synapse 1.51.0 where incoming federation transactions containing at least one EDU would be dropped if debug logging was enabled for `synapse.8631_debug`. ([\#12098](matrix-org/synapse#12098))
- Fix a long-standing bug which could cause push notifications to malfunction if `use_frozen_dicts` was set in the configuration. ([\#12100](matrix-org/synapse#12100))
- Fix an extremely rare, long-standing bug in `ReadWriteLock` that would cause an error when a newly unblocked writer completes instantly. ([\#12105](matrix-org/synapse#12105))
- Make a `POST` to `/rooms/<room_id>/receipt/m.read/<event_id>` only trigger a push notification if the count of unread messages is different to the one in the last successfully sent push. This reduces server load and load on the receiving device. ([\#11835](matrix-org/synapse#11835))

Updates to the Docker image
---------------------------

- The Docker image no longer automatically creates a temporary volume at `/data`. This is not expected to affect normal usage. ([\#11997](matrix-org/synapse#11997))
- Use Python 3.9 in Docker images by default. ([\#12112](matrix-org/synapse#12112))

Improved Documentation
----------------------

- Document support for the `to_device`, `account_data`, `receipts`, and `presence` stream writers for workers. ([\#11599](matrix-org/synapse#11599))
- Explain the meaning of spam checker callbacks' return values. ([\#12003](matrix-org/synapse#12003))
- Clarify information about external Identity Provider IDs. ([\#12004](matrix-org/synapse#12004))

Deprecations and Removals
-------------------------

- Deprecate using `synctl` with the config option `synctl_cache_factor` and print a warning if a user still uses this option. ([\#11865](matrix-org/synapse#11865))
- Remove support for the legacy structured logging configuration (please see the the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#legacy-structured-logging-configuration-removal) if you are using `structured: true` in the Synapse configuration). ([\#12008](matrix-org/synapse#12008))
- Drop support for [MSC3283](matrix-org/matrix-spec-proposals#3283) unstable flags now that the stable flags are supported. ([\#12018](matrix-org/synapse#12018))
- Remove the unstable `/spaces` endpoint from [MSC2946](matrix-org/matrix-spec-proposals#2946). ([\#12073](matrix-org/synapse#12073))

Internal Changes
----------------

- Make the `get_room_version` method use `get_room_version_id` to benefit from caching. ([\#11808](matrix-org/synapse#11808))
- Remove unnecessary condition on knock -> leave auth rule check. ([\#11900](matrix-org/synapse#11900))
- Add tests for device list changes between local users. ([\#11972](matrix-org/synapse#11972))
- Optimise calculating `device_list` changes in `/sync`. ([\#11974](matrix-org/synapse#11974))
- Add missing type hints to storage classes. ([\#11984](matrix-org/synapse#11984))
- Refactor the search code for improved readability. ([\#11991](matrix-org/synapse#11991))
- Move common deduplication code down into `_auth_and_persist_outliers`. ([\#11994](matrix-org/synapse#11994))
- Limit concurrent joins from applications services. ([\#11996](matrix-org/synapse#11996))
- Preparation for faster-room-join work: when parsing the `send_join` response, get the `m.room.create` event from `state`, not `auth_chain`. ([\#12005](matrix-org/synapse#12005), [\#12039](matrix-org/synapse#12039))
- Preparation for faster-room-join work: parse MSC3706 fields in send_join response. ([\#12011](matrix-org/synapse#12011))
- Preparation for faster-room-join work: persist information on which events and rooms have partial state to the database. ([\#12012](matrix-org/synapse#12012))
- Preparation for faster-room-join work: Support for calling `/federation/v1/state` on a remote server. ([\#12013](matrix-org/synapse#12013))
- Configure `tox` to use `venv` rather than `virtualenv`. ([\#12015](matrix-org/synapse#12015))
- Fix bug in `StateFilter.return_expanded()` and add some tests. ([\#12016](matrix-org/synapse#12016))
- Use Matrix v1.1 endpoints (`/_matrix/client/v3/auth/...`) in fallback auth HTML forms. ([\#12019](matrix-org/synapse#12019))
- Update the `olddeps` CI job to use an old version of `markupsafe`. ([\#12025](matrix-org/synapse#12025))
- Upgrade Mypy to version 0.931. ([\#12030](matrix-org/synapse#12030))
- Remove legacy `HomeServer.get_datastore()`. ([\#12031](matrix-org/synapse#12031), [\#12070](matrix-org/synapse#12070))
- Minor typing fixes. ([\#12034](matrix-org/synapse#12034), [\#12069](matrix-org/synapse#12069))
- After joining a room, create a dedicated logcontext to process the queued events. ([\#12041](matrix-org/synapse#12041))
- Tidy up GitHub Actions config which builds distributions for PyPI. ([\#12051](matrix-org/synapse#12051))
- Move configuration out of `setup.cfg`. ([\#12052](matrix-org/synapse#12052), [\#12059](matrix-org/synapse#12059))
- Fix error message when a worker process fails to talk to another worker process. ([\#12060](matrix-org/synapse#12060))
- Fix using the `complement.sh` script without specifying a directory or a branch. Contributed by Nico on behalf of Famedly. ([\#12063](matrix-org/synapse#12063))
- Add type hints to `tests/rest/client`. ([\#12066](matrix-org/synapse#12066), [\#12072](matrix-org/synapse#12072), [\#12084](matrix-org/synapse#12084), [\#12094](matrix-org/synapse#12094))
- Add some logging to `/sync` to try and track down #11916. ([\#12068](matrix-org/synapse#12068))
- Inspect application dependencies using `importlib.metadata` or its backport. ([\#12088](matrix-org/synapse#12088))
- Use `assertEqual` instead of the deprecated `assertEquals` in test code. ([\#12092](matrix-org/synapse#12092))
- Move experimental support for [MSC3440](matrix-org/matrix-spec-proposals#3440) to `/versions`. ([\#12099](matrix-org/synapse#12099))
- Add `stop_cancellation` utility function to stop `Deferred`s from being cancelled. ([\#12106](matrix-org/synapse#12106))
- Improve exception handling for concurrent execution. ([\#12109](matrix-org/synapse#12109))
- Advertise support for Python 3.10 in packaging files. ([\#12111](matrix-org/synapse#12111))
- Move CI checks out of tox, to facilitate a move to using poetry. ([\#12119](matrix-org/synapse#12119))
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants