Merge remote-tracking branch 'origin/develop' into clokep/db-upgrades

matrix-org · Oct 16, 2023 · c1878cd · c1878cd
2 parents b22a14c + 7291c68
commit c1878cd
Show file tree

Hide file tree

Showing 168 changed files with 2,413 additions and 1,786 deletions.
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
@@ -56,6 +56,7 @@ jobs:
             - 'pyproject.toml'
             - 'poetry.lock'
             - 'docker/**'
+            - 'scripts-dev/complement.sh'
 
           linting:
             - 'synapse/**'
@@ -280,7 +281,6 @@ jobs:
       - check-lockfile
       - lint-clippy
       - lint-rustfmt
-      - check-signoff
     runs-on: ubuntu-latest
     steps:
       - run: "true"

diff --git a/CHANGES.md b/CHANGES.md
@@ -1,3 +1,73 @@
+# Synapse 1.94.0 (2023-10-10)
+
+No significant changes since 1.94.0rc1.
+However, please take note of the security advisory that follows.
+
+## Security advisory
+
+The following issue is fixed in 1.94.0 (and RC).
+
+- [GHSA-5chr-wjw5-3gq4](https://github.com/matrix-org/synapse/security/advisories/GHSA-5chr-wjw5-3gq4) / [CVE-2023-45129](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-45129) — Moderate Severity
+
+  A malicious server ACL event can impact performance temporarily or permanently leading to a persistent denial of service.
+
+  Homeservers running on a closed federation (which presumably do not need to use server ACLs) are not affected.
+
+See the advisory for more details. If you have any questions, email [email protected].
+
+
+# Synapse 1.94.0rc1 (2023-10-03)
+
+### Features
+
+- Render plain, CSS, CSV, JSON and common image formats in the browser (inline) when requested through the /download endpoint. ([\#15988](https://github.com/matrix-org/synapse/issues/15988))
+- Add experimental support for [MSC4028](https://github.com/matrix-org/matrix-spec-proposals/pull/4028) to push all encrypted events to clients. ([\#16361](https://github.com/matrix-org/synapse/issues/16361))
+- Minor performance improvement when sending presence to federated servers. ([\#16385](https://github.com/matrix-org/synapse/issues/16385))
+- Minor performance improvement by caching server ACL checking. ([\#16360](https://github.com/matrix-org/synapse/issues/16360))
+
+### Improved Documentation
+
+- Add developer documentation concerning gradual schema migrations with column alterations. ([\#15691](https://github.com/matrix-org/synapse/issues/15691))
+- Improve documentation of the user directory search algorithm. ([\#16320](https://github.com/matrix-org/synapse/issues/16320))
+- Fix rendering of user admin API documentation around deactivation. This was broken in Synapse 1.91.0. ([\#16355](https://github.com/matrix-org/synapse/issues/16355))
+- Update documentation around message retention policies. ([\#16382](https://github.com/matrix-org/synapse/issues/16382))
+- Add note to `federation_domain_whitelist` config option to clarify its usage. ([\#16416](https://github.com/matrix-org/synapse/issues/16416))
+- Improve legacy release notes. ([\#16418](https://github.com/matrix-org/synapse/issues/16418))
+
+### Deprecations and Removals
+
+- Remove Python version from `/_synapse/admin/v1/server_version`. ([\#16380](https://github.com/matrix-org/synapse/issues/16380))
+
+### Internal Changes
+
+- Avoid running CI steps when the files they check have not been changed. ([\#14745](https://github.com/matrix-org/synapse/issues/14745), [\#16387](https://github.com/matrix-org/synapse/issues/16387))
+- Improve type hints. ([\#14911](https://github.com/matrix-org/synapse/issues/14911), [\#16350](https://github.com/matrix-org/synapse/issues/16350), [\#16356](https://github.com/matrix-org/synapse/issues/16356), [\#16395](https://github.com/matrix-org/synapse/issues/16395))
+- Added support for pydantic v2 in addition to pydantic v1. Contributed by Maxwell G (@gotmax23). ([\#16332](https://github.com/matrix-org/synapse/issues/16332))
+- Get CI to check PRs have been signed-off. ([\#16348](https://github.com/matrix-org/synapse/issues/16348))
+- Add missing licence header. ([\#16359](https://github.com/matrix-org/synapse/issues/16359))
+- Improve type hints, and bump types-psycopg2 from 2.9.21.11 to 2.9.21.14. ([\#16381](https://github.com/matrix-org/synapse/issues/16381))
+- Improve comments in `StateGroupBackgroundUpdateStore`. ([\#16383](https://github.com/matrix-org/synapse/issues/16383))
+- Update maturin configuration. ([\#16394](https://github.com/matrix-org/synapse/issues/16394))
+- Downgrade replication stream time out error log lines to warning. ([\#16401](https://github.com/matrix-org/synapse/issues/16401))
+
+### Updates to locked dependencies
+
+* Bump actions/checkout from 3 to 4. ([\#16250](https://github.com/matrix-org/synapse/issues/16250))
+* Bump cryptography from 41.0.3 to 41.0.4. ([\#16362](https://github.com/matrix-org/synapse/issues/16362))
+* Bump dawidd6/action-download-artifact from 2.27.0 to 2.28.0. ([\#16374](https://github.com/matrix-org/synapse/issues/16374))
+* Bump docker/setup-buildx-action from 2 to 3. ([\#16375](https://github.com/matrix-org/synapse/issues/16375))
+* Bump gitpython from 3.1.35 to 3.1.37. ([\#16376](https://github.com/matrix-org/synapse/issues/16376))
+* Bump msgpack from 1.0.5 to 1.0.6. ([\#16377](https://github.com/matrix-org/synapse/issues/16377))
+* Bump msgpack from 1.0.6 to 1.0.7. ([\#16412](https://github.com/matrix-org/synapse/issues/16412))
+* Bump phonenumbers from 8.13.19 to 8.13.22. ([\#16413](https://github.com/matrix-org/synapse/issues/16413))
+* Bump psycopg2 from 2.9.7 to 2.9.8. ([\#16409](https://github.com/matrix-org/synapse/issues/16409))
+* Bump pydantic from 2.3.0 to 2.4.2. ([\#16410](https://github.com/matrix-org/synapse/issues/16410))
+* Bump regex from 1.9.5 to 1.9.6. ([\#16408](https://github.com/matrix-org/synapse/issues/16408))
+* Bump sentry-sdk from 1.30.0 to 1.31.0. ([\#16378](https://github.com/matrix-org/synapse/issues/16378))
+* Bump types-netaddr from 0.8.0.9 to 0.9.0.1. ([\#16411](https://github.com/matrix-org/synapse/issues/16411))
+* Bump types-psycopg2 from 2.9.21.11 to 2.9.21.14. ([\#16381](https://github.com/matrix-org/synapse/issues/16381))
+* Bump urllib3 from 1.26.15 to 1.26.17. ([\#16422](https://github.com/matrix-org/synapse/issues/16422))
+
 # Synapse 1.93.0 (2023-09-26)
 
 No significant changes since 1.93.0rc1.

diff --git a/Cargo.lock b/Cargo.lock
diff --git a/changelog.d/14745.misc b/changelog.d/14745.misc
diff --git a/changelog.d/15691.doc b/changelog.d/15691.doc
diff --git a/changelog.d/15988.feature b/changelog.d/15988.feature
diff --git a/changelog.d/16162.misc b/changelog.d/16162.misc
@@ -0,0 +1 @@
+Bump pyo3 from 0.17.1 to 0.19.2.
diff --git a/changelog.d/16320.doc b/changelog.d/16320.doc
diff --git a/changelog.d/16332.misc b/changelog.d/16332.misc
diff --git a/changelog.d/16348.misc b/changelog.d/16348.misc
diff --git a/changelog.d/16355.doc b/changelog.d/16355.doc
diff --git a/changelog.d/16359.misc b/changelog.d/16359.misc
diff --git a/changelog.d/16360.misc b/changelog.d/16360.misc
diff --git a/changelog.d/16361.feature b/changelog.d/16361.feature
diff --git a/changelog.d/16380.removal b/changelog.d/16380.removal
diff --git a/changelog.d/16381.misc b/changelog.d/16381.misc
diff --git a/changelog.d/16382.doc b/changelog.d/16382.doc
diff --git a/changelog.d/16383.misc b/changelog.d/16383.misc
diff --git a/changelog.d/16385.misc b/changelog.d/16385.misc
diff --git a/changelog.d/16387.misc b/changelog.d/16387.misc
diff --git a/changelog.d/16394.misc b/changelog.d/16394.misc
diff --git a/changelog.d/16395.misc b/changelog.d/16395.misc
diff --git a/changelog.d/16401.misc b/changelog.d/16401.misc
diff --git a/changelog.d/16403.bugfix b/changelog.d/16403.bugfix
@@ -0,0 +1 @@
+Remove legacy unspecced `knock_state_events` field returned in some responses.
diff --git a/changelog.d/16404.bugfix b/changelog.d/16404.bugfix
@@ -0,0 +1 @@
+Fixes possbile `AttributeError` when `_matrix/client/v3/account/whoami` is called over a unix socket. Contributed by @Sir-Photch.
diff --git a/changelog.d/16418.doc b/changelog.d/16418.doc
diff --git a/changelog.d/16419.misc b/changelog.d/16419.misc
@@ -0,0 +1 @@
+Update registration of media repository URLs.
diff --git a/changelog.d/16420.doc b/changelog.d/16420.doc
@@ -0,0 +1 @@
+Document internal background update mechanism.
diff --git a/changelog.d/14911.misc → changelog.d/16421.misc b/changelog.d/14911.misc → changelog.d/16421.misc
diff --git a/changelog.d/16426.misc b/changelog.d/16426.misc
@@ -0,0 +1 @@
+Refactor some code to simplify and better type receipts stream adjacent code.
diff --git a/changelog.d/16427.misc b/changelog.d/16427.misc
@@ -0,0 +1 @@
+Factor out `MultiWriter` token from `RoomStreamToken`.
diff --git a/changelog.d/16428.misc b/changelog.d/16428.misc
@@ -0,0 +1 @@
+Improve code comments.
diff --git a/changelog.d/16429.misc b/changelog.d/16429.misc
@@ -0,0 +1 @@
+Reduce memory allocations.
diff --git a/changelog.d/16431.misc b/changelog.d/16431.misc
@@ -0,0 +1 @@
+Reduce memory allocations.
diff --git a/changelog.d/16433.misc b/changelog.d/16433.misc
@@ -0,0 +1 @@
+Reduce memory allocations.
diff --git a/changelog.d/16434.misc b/changelog.d/16434.misc
@@ -0,0 +1 @@
+Reduce memory allocations.
diff --git a/changelog.d/16435.misc b/changelog.d/16435.misc
@@ -0,0 +1 @@
+Remove unused method.
diff --git a/changelog.d/16438.misc b/changelog.d/16438.misc
@@ -0,0 +1 @@
+Reduce memory allocations.
diff --git a/changelog.d/16440.bugfix b/changelog.d/16440.bugfix
@@ -0,0 +1 @@
+Properly return inline media when content types have parameters.
diff --git a/changelog.d/16441.misc b/changelog.d/16441.misc
@@ -0,0 +1 @@
+Improve rate limiting logic.
diff --git a/changelog.d/16444.misc b/changelog.d/16444.misc
@@ -0,0 +1 @@
+Reduce memory allocations.
diff --git a/changelog.d/16454.misc b/changelog.d/16454.misc
@@ -0,0 +1 @@
+Do not block running of CI behind the check for sign-off on PRs.
diff --git a/changelog.d/16455.bugfix b/changelog.d/16455.bugfix
@@ -0,0 +1 @@
+Prevent the purging of large rooms from timing out when Postgres is in use. The timeout which causes this issue was introduced in Synapse 1.88.0.
diff --git a/changelog.d/16457.bugfix b/changelog.d/16457.bugfix
@@ -0,0 +1 @@
+Improve the performance of purging rooms, particularly encrypted rooms.
diff --git a/changelog.d/16461.misc b/changelog.d/16461.misc
@@ -0,0 +1 @@
+Update the release script to remind releaser to check for special release notes.
diff --git a/changelog.d/16466.misc b/changelog.d/16466.misc
@@ -0,0 +1 @@
+Update complement.sh to match new public API shape.
diff --git a/changelog.d/16350.misc → changelog.d/16468.misc b/changelog.d/16350.misc → changelog.d/16468.misc
diff --git a/changelog.d/16356.misc → changelog.d/16469.misc b/changelog.d/16356.misc → changelog.d/16469.misc
diff --git a/changelog.d/16477.doc b/changelog.d/16477.doc
@@ -0,0 +1 @@
+Fix a typo in the sql for [useful SQL for admins document](https://matrix-org.github.io/synapse/latest/usage/administration/useful_sql_for_admins.html).
diff --git a/changelog.d/16488.misc b/changelog.d/16488.misc
@@ -0,0 +1 @@
+Clean up logging on event persister endpoints.
diff --git a/changelog.d/16491.misc b/changelog.d/16491.misc
@@ -0,0 +1 @@
+Remove useless async job to delete device messages on sync, since we only deliver (and hence delete) up to 100 device messages at a time.
diff --git a/debian/changelog b/debian/changelog
@@ -1,3 +1,15 @@
+matrix-synapse-py3 (1.94.0) stable; urgency=medium
+
+  * New Synapse release 1.94.0.
+
+ -- Synapse Packaging team <[email protected]>  Tue, 10 Oct 2023 10:57:41 +0100
+
+matrix-synapse-py3 (1.94.0~rc1) stable; urgency=medium
+
+  * New Synapse release 1.94.0rc1.
+
+ -- Synapse Packaging team <[email protected]>  Tue, 03 Oct 2023 11:48:18 +0100
+
 matrix-synapse-py3 (1.93.0) stable; urgency=medium
 
   * New Synapse release 1.93.0.

diff --git a/docs/development/database_schema.md b/docs/development/database_schema.md
@@ -150,6 +150,67 @@ def run_upgrade(
     ...
 ```
 
+## Background updates
+
+It is sometimes appropriate to perform database migrations as part of a background
+process (instead of blocking Synapse until the migration is done). In particular,
+this is useful for migrating data when adding new columns or tables.
+
+Pending background updates stored in the `background_updates` table and are denoted
+by a unique name, the current status (stored in JSON), and some dependency information:
+
+* Whether the update requires a previous update to be complete.
+* A rough ordering for which to complete updates.
+
+A new background updates needs to be added to the `background_updates` table:
+
+```sql
+INSERT INTO background_updates (ordering, update_name, depends_on, progress_json) VALUES
+  (7706, 'my_background_update', 'a_previous_background_update' '{}');
+```
+
+And then needs an associated handler in the appropriate datastore:
+
+```python
+self.db_pool.updates.register_background_update_handler(
+    "my_background_update",
+    update_handler=self._my_background_update,
+)
+```
+
+There are a few types of updates that can be performed, see the `BackgroundUpdater`:
+
+* `register_background_update_handler`: A generic handler for custom SQL
+* `register_background_index_update`: Create an index in the background
+* `register_background_validate_constraint`: Validate a constraint in the background
+  (PostgreSQL-only)
+* `register_background_validate_constraint_and_delete_rows`: Similar to
+  `register_background_validate_constraint`, but deletes rows which don't fit
+  the constraint.
+
+For `register_background_update_handler`, the generic handler must track progress
+and then finalize the background update:
+
+```python
+async def _my_background_update(self, progress: JsonDict, batch_size: int) -> int:
+    def _do_something(txn: LoggingTransaction) -> int:
+        ...
+        self.db_pool.updates._background_update_progress_txn(
+            txn, "my_background_update", {"last_processed": last_processed}
+        )
+        return last_processed - prev_last_processed
+
+    num_processed = await self.db_pool.runInteraction("_do_something", _do_something)
+    await self.db_pool.updates._end_background_update("my_background_update")
+
+    return num_processed
+```
+
+Synapse will attempt to rate-limit how often background updates are run via the
+given batch-size and the returned number of processed entries (and how long the
+function took to run). See
+[background update controller callbacks](../modules/background_update_controller_callbacks.md).
+
 ## Boolean columns
 
 Boolean columns require special treatment, since SQLite treats booleans the

diff --git a/docs/usage/administration/useful_sql_for_admins.md b/docs/usage/administration/useful_sql_for_admins.md
@@ -193,7 +193,7 @@ SELECT rss.room_id, rss.name, rss.canonical_alias, rss.topic, rss.encryption,
     rsc.joined_members, rsc.local_users_in_room, rss.join_rules
   FROM room_stats_state rss
   LEFT JOIN room_stats_current rsc USING (room_id)
-  WHERE room_id IN ( WHERE room_id IN (
+  WHERE room_id IN (
     '!OGEhHVWSdvArJzumhm:matrix.org',
     '!YTvKGNlinIzlkMTVRl:matrix.org' 
   );
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Remove legacy unspecced `knock_state_events` field returned in some responses.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Fixes possbile `AttributeError` when `_matrix/client/v3/account/whoami` is called over a unix socket. Contributed by @Sir-Photch.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Refactor some code to simplify and better type receipts stream adjacent code.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Factor out `MultiWriter` token from `RoomStreamToken`.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Properly return inline media when content types have parameters.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Do not block running of CI behind the check for sign-off on PRs.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Prevent the purging of large rooms from timing out when Postgres is in use. The timeout which causes this issue was introduced in Synapse 1.88.0.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Improve the performance of purging rooms, particularly encrypted rooms.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Update the release script to remind releaser to check for special release notes.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Update complement.sh to match new public API shape.