Fixing that search ThingUpdater could be blocked when waiting for bulk write completions #1340
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* send where acks are issued * change "ReceiveTimeout" in ThingUpdater to a separate timer fixing that any consumed message prolonges the timeout of 2 minutes again Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>
…ull`) Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>
…e same thing * by creating SubSources using "groupBy(bulkWriteSize)" in EnforcementFlow in order to perform updates regarding a certain thingId always from the same "BulkWriter" * removing behavior changes in ThingUpdater in order to not block it unnecessarily Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>
…axBulkSize" for that Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>
Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>
… substreams based on the thingId hash should be started for bulk updates Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>
Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>
Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>
Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>
Signed-off-by: Yufei Cai <yufei.cai@bosch.io>
…reordering. Signed-off-by: Yufei Cai <yufei.cai@bosch.io>
The config is not needed. Signed-off-by: Yufei Cai <yufei.cai@bosch.io>
…e diff computation. Signed-off-by: Yufei Cai <yufei.cai@bosch.io>
…ed method. Signed-off-by: Yufei Cai <yufei.cai@bosch.io>
Signed-off-by: Yufei Cai <yufei.cai@bosch.io>
Signed-off-by: Yufei Cai <yufei.cai@bosch.io>
Signed-off-by: Yufei Cai <yufei.cai@bosch.io>
Signed-off-by: Yufei Cai <yufei.cai@bosch.io>
Signed-off-by: Yufei Cai <yufei.cai@bosch.io>
thjaeckle
changed the title
thjaeckle
reviewed
Mar 17, 2022
Signed-off-by: Yufei Cai <yufei.cai@bosch.io>
Signed-off-by: Yufei Cai <yufei.cai@bosch.io>
Signed-off-by: Cai Yufei <yufei.cai@bosch.io>
The test is failing only on Github. Signed-off-by: Yufei Cai <yufei.cai@bosch.io>
Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>
Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>
Signed-off-by: Yufei Cai <yufei.cai@bosch.io>
Signed-off-by: Yufei Cai <yufei.cai@bosch.io>
Signed-off-by: Yufei Cai <yufei.cai@bosch.io>
The test is failing only on Github. Signed-off-by: Yufei Cai <yufei.cai@bosch.io>
…f Map<ThingId, Metadata> Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>
Signed-off-by: Yufei Cai <yufei.cai@bosch.io>
…arch-index would not process this event Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>
Signed-off-by: Yufei Cai <yufei.cai@bosch.io>
Signed-off-by: Yufei Cai <yufei.cai@bosch.io>
Signed-off-by: Yufei Cai <yufei.cai@bosch.io>
yufei-cai
force-pushed
the
bugfix/search-updater-bulk-write-completion
branch
2 times, most recently
from
d6a66a2
to
4a98d57
Compare
…oduced in EnforcementFlow Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>
Signed-off-by: Yufei Cai <yufei.cai@bosch.io>
Signed-off-by: Yufei Cai <yufei.cai@bosch.io>
Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>
Signed-off-by: Yufei Cai <yufei.cai@bosch.io>
This pull request was closed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
@thjaeckle wrote:
The behavior change added in PR #1320 caused problems when the "bulk write complete" was not received by the ThingUpdater.
In that case it was blocked for 2 minutes, not processing further updates.
The fix removes this "behavior change" and blocking of new WriteModels in ThingUpdater again.
Instead, it applies sharding (based on the thingId hash) creating substreams in the EnforcementFlow.create() with a configured "bulk-shard-count" ensuring that the same things are updated by the same bulkWriter, or at least reducing probability of concurrent modifications in async bulk writes.
Reopened #1335