EWPP-2210: Optimize query for sanitizing of user data. #155
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
OPENEUROPA-EWPP-2210
Description
On one of the sites (Epale – from EACEA) the sanitization led to memory issues and following further investigation the problem was pinpointed to the new sanitization in oe_authentication (see e3ca1f5). The problem is that the code implemented is loading in memory all users in a single go as objects and therefore the memory needed depends linearly on the number of users. For a large site such as EPALE (~130K users) this leads to more than 5GB RAM needed for the sanitization. Furthermore, the sanitization is done one by one which requires a very long time.
Related Github issue: #152
Change log
Commands