Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Update postgres docs #9989

Merged
merged 3 commits into from
May 14, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion changelog.d/9988.doc
Original file line number Diff line number Diff line change
@@ -1 +1 @@
Fix outdated minimum PostgreSQL version in postgres.md.
Updates to the PostgreSQL documentation (`postgres.md`).
1 change: 1 addition & 0 deletions changelog.d/9989.doc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Updates to the PostgreSQL documentation (`postgres.md`).
198 changes: 96 additions & 102 deletions docs/postgres.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,28 +33,15 @@ Assuming your PostgreSQL database user is called `postgres`, first authenticate
# Or, if your system uses sudo to get administrative rights
sudo -u postgres bash

Then, create a user ``synapse_user`` with:
Then, create a postgres user and a database with:

# this will prompt for a password for the new user
createuser --pwprompt synapse_user

Before you can authenticate with the `synapse_user`, you must create a
database that it can access. To create a database, first connect to the
database with your database user:
createdb --encoding=UTF8 --locale=C --template=template0 --owner=synapse_user synapse
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we're using createuser, we may as well use createdb which is somewhat simpler to use on the commandline.


su - postgres # Or: sudo -u postgres bash
psql

and then run:

CREATE DATABASE synapse
ENCODING 'UTF8'
LC_COLLATE='C'
LC_CTYPE='C'
template=template0
OWNER synapse_user;

This would create an appropriate database named `synapse` owned by the
`synapse_user` user (which must already have been created as above).
The above will create a user called `synapse_user`, and a database called
`synapse`.

Note that the PostgreSQL database *must* have the correct encoding set
(as shown above), otherwise it will not be able to store UTF8 strings.
Expand All @@ -63,79 +50,6 @@ You may need to enable password authentication so `synapse_user` can
connect to the database. See
<https://www.postgresql.org/docs/current/auth-pg-hba-conf.html>.

If you get an error along the lines of `FATAL: Ident authentication failed for
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to move this stuff out of the happy path, so it's now in "troubleshooting" below.

user "synapse_user"`, you may need to use an authentication method other than
`ident`:

* If the `synapse_user` user has a password, add the password to the `database:`
section of `homeserver.yaml`. Then add the following to `pg_hba.conf`:

```
host synapse synapse_user ::1/128 md5 # or `scram-sha-256` instead of `md5` if you use that
```

* If the `synapse_user` user does not have a password, then a password doesn't
have to be added to `homeserver.yaml`. But the following does need to be added
to `pg_hba.conf`:

```
host synapse synapse_user ::1/128 trust
```

Note that line order matters in `pg_hba.conf`, so make sure that if you do add a
new line, it is inserted before:

```
host all all ::1/128 ident
```

### Fixing incorrect `COLLATE` or `CTYPE`

Synapse will refuse to set up a new database if it has the wrong values of
`COLLATE` and `CTYPE` set, and will log warnings on existing databases. Using
different locales can cause issues if the locale library is updated from
underneath the database, or if a different version of the locale is used on any
replicas.

The safest way to fix the issue is to take a dump and recreate the database with
the correct `COLLATE` and `CTYPE` parameters (as shown above). It is also possible to change the
parameters on a live database and run a `REINDEX` on the entire database,
however extreme care must be taken to avoid database corruption.

Note that the above may fail with an error about duplicate rows if corruption
has already occurred, and such duplicate rows will need to be manually removed.


## Fixing inconsistent sequences error

Synapse uses Postgres sequences to generate IDs for various tables. A sequence
and associated table can get out of sync if, for example, Synapse has been
downgraded and then upgraded again.

To fix the issue shut down Synapse (including any and all workers) and run the
SQL command included in the error message. Once done Synapse should start
successfully.


## Tuning Postgres
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

likewise I've moved this down since it apparently it only affects "larger scale deployments".


The default settings should be fine for most deployments. For larger
scale deployments tuning some of the settings is recommended, details of
which can be found at
<https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server>.

In particular, we've found tuning the following values helpful for
performance:

- `shared_buffers`
- `effective_cache_size`
- `work_mem`
- `maintenance_work_mem`
- `autovacuum_work_mem`

Note that the appropriate values for those fields depend on the amount
of free memory the database host has available.

## Synapse config

When you are ready to start using PostgreSQL, edit the `database`
Expand Down Expand Up @@ -165,18 +79,42 @@ may block for an extended period while it waits for a response from the
database server. Example values might be:

```yaml
# seconds of inactivity after which TCP should send a keepalive message to the server
keepalives_idle: 10
database:
args:
# ... as above

# seconds of inactivity after which TCP should send a keepalive message to the server
keepalives_idle: 10

# the number of seconds after which a TCP keepalive message that is not
# acknowledged by the server should be retransmitted
keepalives_interval: 10
# the number of seconds after which a TCP keepalive message that is not
# acknowledged by the server should be retransmitted
keepalives_interval: 10

# the number of TCP keepalives that can be lost before the client's connection
# to the server is considered dead
keepalives_count: 3
# the number of TCP keepalives that can be lost before the client's connection
# to the server is considered dead
keepalives_count: 3
```

## Tuning Postgres

The default settings should be fine for most deployments. For larger
scale deployments tuning some of the settings is recommended, details of
which can be found at
<https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server>.

In particular, we've found tuning the following values helpful for
performance:

- `shared_buffers`
- `effective_cache_size`
- `work_mem`
- `maintenance_work_mem`
- `autovacuum_work_mem`

Note that the appropriate values for those fields depend on the amount
of free memory the database host has available.


## Porting from SQLite

### Overview
Expand All @@ -185,9 +123,8 @@ The script `synapse_port_db` allows porting an existing synapse server
backed by SQLite to using PostgreSQL. This is done in as a two phase
process:

1. Copy the existing SQLite database to a separate location (while the
server is down) and running the port script against that offline
database.
1. Copy the existing SQLite database to a separate location and run
the port script against that offline database.
Comment on lines +126 to +127
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

given this is just the overview, and the detailed steps below explain the process more clearly, I think we can do without the "while the server is down" bit here.

2. Shut down the server. Rerun the port script to port any data that
has come in since taking the first snapshot. Restart server against
the PostgreSQL database.
Expand Down Expand Up @@ -245,3 +182,60 @@ PostgreSQL database configuration file `homeserver-postgres.yaml`:
./synctl start

Synapse should now be running against PostgreSQL.


## Troubleshooting

### Alternative auth methods

If you get an error along the lines of `FATAL: Ident authentication failed for
user "synapse_user"`, you may need to use an authentication method other than
`ident`:

* If the `synapse_user` user has a password, add the password to the `database:`
section of `homeserver.yaml`. Then add the following to `pg_hba.conf`:

```
host synapse synapse_user ::1/128 md5 # or `scram-sha-256` instead of `md5` if you use that
```

* If the `synapse_user` user does not have a password, then a password doesn't
have to be added to `homeserver.yaml`. But the following does need to be added
to `pg_hba.conf`:

```
host synapse synapse_user ::1/128 trust
```

Note that line order matters in `pg_hba.conf`, so make sure that if you do add a
new line, it is inserted before:

```
host all all ::1/128 ident
```

### Fixing incorrect `COLLATE` or `CTYPE`

Synapse will refuse to set up a new database if it has the wrong values of
`COLLATE` and `CTYPE` set, and will log warnings on existing databases. Using
different locales can cause issues if the locale library is updated from
underneath the database, or if a different version of the locale is used on any
replicas.

The safest way to fix the issue is to dump the database and recreate it with
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this now includes the change from @babolivier in #9990

the correct locale parameter (as shown above). It is also possible to change the
parameters on a live database and run a `REINDEX` on the entire database,
however extreme care must be taken to avoid database corruption.

Note that the above may fail with an error about duplicate rows if corruption
has already occurred, and such duplicate rows will need to be manually removed.

### Fixing inconsistent sequences error

Synapse uses Postgres sequences to generate IDs for various tables. A sequence
and associated table can get out of sync if, for example, Synapse has been
downgraded and then upgraded again.

To fix the issue shut down Synapse (including any and all workers) and run the
SQL command included in the error message. Once done Synapse should start
successfully.