Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Add basic function to get all data for a user out of synapse #5589

Merged
merged 9 commits into from
Jul 15, 2019

Conversation

erikjohnston
Copy link
Member

@erikjohnston erikjohnston commented Jul 2, 2019

This is currently not hooked up to anything. The plan is to write a command app type thing that lets you run such functionality manually without having to hit production instances.

Note: This only fetches events that a user can see, rather than all data we store for a user.

(This is a port of a hacky script we manually run to do this currently)

@erikjohnston erikjohnston requested a review from a team July 2, 2019 11:18
@codecov
Copy link

codecov bot commented Jul 2, 2019

Codecov Report

Merging #5589 into develop will increase coverage by 0.05%.
The diff coverage is 81.25%.

@@             Coverage Diff             @@
##           develop    #5589      +/-   ##
===========================================
+ Coverage     63.2%   63.26%   +0.05%     
===========================================
  Files          328      328              
  Lines        35931    35995      +64     
  Branches      5918     5932      +14     
===========================================
+ Hits         22710    22772      +62     
- Misses       11595    11598       +3     
+ Partials      1626     1625       -1

Copy link
Member

@richvdh richvdh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A lot of quibbling about comments and names, but generally looks sane to me.

I'm raising a bit of an eyebrow at the mix of synchronous I/O with asynchronous database ops. I guess it's fine as long as the functions in question are only called from an external process, but it may be worth adding a health-warning to the functions? Better, could they even be moved to a separate python module?

synapse/handlers/admin.py Outdated Show resolved Hide resolved
synapse/handlers/admin.py Outdated Show resolved Hide resolved
synapse/handlers/admin.py Outdated Show resolved Hide resolved
synapse/handlers/admin.py Outdated Show resolved Hide resolved
synapse/handlers/admin.py Outdated Show resolved Hide resolved
synapse/handlers/admin.py Outdated Show resolved Hide resolved
synapse/handlers/admin.py Outdated Show resolved Hide resolved
synapse/handlers/admin.py Show resolved Hide resolved
synapse/handlers/admin.py Outdated Show resolved Hide resolved
synapse/handlers/admin.py Outdated Show resolved Hide resolved
@erikjohnston erikjohnston requested a review from richvdh July 5, 2019 15:31
synapse/storage/stream.py Outdated Show resolved Hide resolved
synapse/storage/stream.py Outdated Show resolved Hide resolved
synapse/storage/stream.py Outdated Show resolved Hide resolved
Copy link
Member

@richvdh richvdh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@erikjohnston erikjohnston merged commit d863213 into develop Jul 15, 2019
anoadragon453 added a commit that referenced this pull request Jul 22, 2019
v1.2.0rc1

Features
--------

- Add support for opentracing. ([\#5544](#5544), [\#5712](#5712))
- Add ability to pull all locally stored events out of synapse that a particular user can see. ([\#5589](#5589))
- Add a basic admin command app to allow server operators to run Synapse admin commands separately from the main production instance. ([\#5597](#5597))
- Add `sender` and `origin_server_ts` fields to `m.replace`. ([\#5613](#5613))
- Add default push rule to ignore reactions. ([\#5623](#5623))
- Include the original event when asking for its relations. ([\#5626](#5626))
- Implement `session_lifetime` configuration option, after which access tokens will expire. ([\#5660](#5660))
- Return "This account has been deactivated" when a deactivated user tries to login. ([\#5674](#5674))
- Enable aggregations support by default ([\#5714](#5714))

Bugfixes
--------

- Fix 'utime went backwards' errors on daemonization. ([\#5609](#5609))
- Various minor fixes to the federation request rate limiter. ([\#5621](#5621))
- Forbid viewing relations on an event once it has been redacted. ([\#5629](#5629))
- Fix requests to the `/store_invite` endpoint of identity servers being sent in the wrong format. ([\#5638](#5638))
- Fix newly-registered users not being able to lookup their own profile without joining a room. ([\#5644](#5644))
- Fix bug in #5626 that prevented the original_event field from actually having the contents of the original event in a call to `/relations`. ([\#5654](#5654))
- Fix 3PID bind requests being sent to identity servers as `application/x-form-www-urlencoded` data, which is deprecated. ([\#5658](#5658))
- Fix some problems with authenticating redactions in recent room versions. ([\#5699](#5699), [\#5700](#5700), [\#5707](#5707))
- Ignore redactions of m.room.create events. ([\#5701](#5701))

Updates to the Docker image
---------------------------

- Base Docker image on a newer Alpine Linux version (3.8 -> 3.10). ([\#5619](#5619))
- Add missing space in default logging file format generated by the Docker image. ([\#5620](#5620))

Improved Documentation
----------------------

- Add information about nginx normalisation to reverse_proxy.rst. Contributed by @skalarproduktraum - thanks! ([\#5397](#5397))
- --no-pep517 should be --no-use-pep517 in the documentation to setup the development environment. ([\#5651](#5651))
- Improvements to Postgres setup instructions. Contributed by @Lrizika - thanks! ([\#5661](#5661))
- Minor tweaks to postgres documentation. ([\#5675](#5675))

Deprecations and Removals
-------------------------

- Remove support for the `invite_3pid_guest` configuration setting. ([\#5625](#5625))

Internal Changes
----------------

- Move logging code out of `synapse.util` and into `synapse.logging`. ([\#5606](#5606), [\#5617](#5617))
- Add a blacklist file to the repo to blacklist certain sytests from failing CI. ([\#5611](#5611))
- Make runtime errors surrounding password reset emails much clearer. ([\#5616](#5616))
- Remove dead code for persiting outgoing federation transactions. ([\#5622](#5622))
- Add `lint.sh` to the scripts-dev folder which will run all linting steps required by CI. ([\#5627](#5627))
- Move RegistrationHandler.get_or_create_user to test code. ([\#5628](#5628))
- Add some more common python virtual-environment paths to the black exclusion list. ([\#5630](#5630))
- Some counter metrics exposed over Prometheus have been renamed, with the old names preserved for backwards compatibility and deprecated. See `docs/metrics-howto.rst` for details. ([\#5636](#5636))
- Unblacklist some user_directory sytests. ([\#5637](#5637))
- Factor out some redundant code in the login implementation. ([\#5639](#5639))
- Update ModuleApi to avoid register(generate_token=True). ([\#5640](#5640))
- Remove access-token support from `RegistrationHandler.register`, and rename it. ([\#5641](#5641))
- Remove access-token support from `RegistrationStore.register`, and rename it. ([\#5642](#5642))
- Improve logging for auto-join when a new user is created. ([\#5643](#5643))
- Remove unused and unnecessary check for FederationDeniedError in _exception_to_failure. ([\#5645](#5645))
- Fix a small typo in a code comment. ([\#5655](#5655))
- Clean up exception handling around client access tokens. ([\#5656](#5656))
- Add a mechanism for per-test homeserver configuration in the unit tests. ([\#5657](#5657))
- Inline issue_access_token. ([\#5659](#5659))
- Update the sytest BuildKite configuration to checkout Synapse in `/src`. ([\#5664](#5664))
- Add a `docker` type to the towncrier configuration. ([\#5673](#5673))
- Convert `synapse.federation.transport.server` to `async`. Might improve some stack traces. ([\#5689](#5689))
- Documentation for opentracing. ([\#5703](#5703))
@erikjohnston erikjohnston deleted the erikj/admin_exfiltrate_data branch January 9, 2020 15:50
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants