Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resolving imported policies in Ditto's "things-search" does not use a cache to load same policy #1869

Closed
thjaeckle opened this issue Jan 22, 2024 · 1 comment
Assignees
Labels
Milestone

Comments

@thjaeckle
Copy link
Member

During updating a single policy which is imported in ~5.000 other Policies, I noticed many "ask timeout exceptions" being thrown in Ditto's "things-search" service.

It is intended that - when a imported policy is changed - all things using policies importing from that single policy are updated in the search index (to reflect the changed permissions).

It is however quite unexpected that e.g. the same policy org.eclipse.ditto:my-root-policy is loaded for all policies which use that exact same policy. But this seems to be the case:

return policy.withResolvedImports(
importedPolicyId -> policyCacheLoader.asyncLoad(importedPolicyId, executor)

This will of course cause a huge amount of SudoRetrievePolicy commands issued at the exact same time against the policies shard region, leading to many timeouts.
This place should make use of the Policy cache, so that for each cluster instance of "things-search" a heavily "imported" policy is only loaded once.

@thjaeckle thjaeckle added the bug label Jan 22, 2024
@thjaeckle thjaeckle self-assigned this Jan 22, 2024
thjaeckle added a commit that referenced this issue Jan 23, 2024
…loaded after invalidation in search

* cache is used in ResolvedPolicyCacheLoader
* it is now added that an invalidation caused by a policy contains the "causingPolicyTag" - which is then also invalidated
* however, the "causingPolicyTag" is only invalidated once (per search cluster node) to not overwhelm the cluster again due to too many invalidations

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>
@thjaeckle thjaeckle added this to the 3.5.0 milestone Jan 23, 2024
thjaeckle added a commit that referenced this issue Jan 24, 2024
…loaded after invalidation in search

* cache is used in ResolvedPolicyCacheLoader
* it is now added that an invalidation caused by a policy contains the "causingPolicyTag" - which is then also invalidated
* however, the "causingPolicyTag" is only invalidated once (per search cluster node) to not overwhelm the cluster again due to too many invalidations

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>
thjaeckle added a commit that referenced this issue Jan 24, 2024
…loaded after invalidation in search

* cache is used in ResolvedPolicyCacheLoader
* it is now added that an invalidation caused by a policy contains the "causingPolicyTag" - which is then also invalidated
* however, the "causingPolicyTag" is only invalidated once (per search cluster node) to not overwhelm the cluster again due to too many invalidations

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>
thjaeckle added a commit that referenced this issue Jan 24, 2024
Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>
thjaeckle added a commit that referenced this issue Jan 25, 2024
…orted-policies-in-ditto-search

#1869 use cache in order to load imported policies when policies are …
@thjaeckle
Copy link
Member Author

Resolved by: #1872

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Done
Development

No branches or pull requests

1 participant