Add config options for media retention

[anoadragon453]

Closes #6832.

Adds configuration options for automatically removing media if it hasn't been accessed (either locally or remotely) after a certain period of time. This is managed by an hourly background job that searches the database for old media according to the configured lifetimes, removing the files from disk if any is found.

TODO:

• unit tests
  • Test purging media with an old last_accessed date, and that recently accessed media is not affected
  • Test purging media with no last_accessed date, but an old creation date, and that recently created media is not affected

Questions:

• Do we apply any jitter to our background jobs? Otherwise I feel that a lot is going to happen 1hr after your start your homeserver. I don't think we do. However, the period is now configurable with the default being every 24 hours. This is also not really a problem to consider in this PR.

[dklimpel]

IMHO to run it every hour is very often. Is it enough to run every 6, 12 or 24 hours? Then it need a notice that the retention policy shorter then this period is not very sensible.

[anoadragon453]

@dklimpel Perhaps it is worth making it configurable, yes. There's really no reason not to, and the implementation work to add that is minimal.

[anoadragon453]

Yes, I too find this fiendishly messy.

I've another PR in the works to create a MXCUri object which can be much more easily passed around, and used to convert between str and object representations throughout the codebase.

[anoadragon453]

This PR has been updated to address review feedback and add unit tests.

[erikjohnston]

IMHO to run it every hour is very often. Is it enough to run every 6, 12 or 24 hours? Then it need a notice that the retention policy shorter then this period is not very sensible.

@dklimpel @anoadragon453: The choice here is between a) run it often with each run being cheap, or b) run it less frequently and have each run be potentially expensive. To illustrate this consider a busy server that is receiving GBs of new media a day, if we delete it every hour then we're talking about deleting ~50MB of data every run, but running once a day means deleting GBs at once. This may or may not have a noticeable impact on the server.

In general, assuming each run has relatively low overhead we prefer running more frequently rather than doing lots of work infrequently.

[dklimpel]

@dklimpel @anoadragon453: The choice here is between a) run it often with each run being cheap, or b) run it less frequently and have each run be potentially expensive. To illustrate this consider a busy server that is receiving GBs of new media a day, if we delete it every hour then we're talking about deleting ~50MB of data every run, but running once a day means deleting GBs at once. This may or may not have a noticeable impact on the server.

In general, assuming each run has relatively low overhead we prefer running more frequently rather than doing lots of work infrequently.

Should this information added to docs/usage/configuration/config_documentation.md?
There is a config switch now that solve the problem.

[anoadragon453]

We had a discussion in #synapse-dev:matrix.org regarding the benefit of the purge_period config option.

The conclusion is that it's probably not worth adding a config option until it becomes apparent that one is needed. Instead, we're going to aim for some reasonable default (say, 6 hours) and then see how that works in testing. If it ends up that a different period is needed for each system, then we can easily re-introduce the config option. But options are hard to remove once they're added - so let's see if we need one first.