Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ChangeFeedIterator gets stuck on multihashed containers for prefix partitions keys that span multiple physical partitions. #3865

Closed
NaluTripician opened this issue May 19, 2023 · 5 comments · Fixed by #3879
Assignees
Labels
bug Something isn't working HierarchicalPartitioning Tag to track issues related to Hierarchical Partitioned containers Hotfix A hotfix is required for the issue

Comments

@NaluTripician
Copy link
Contributor

NaluTripician commented May 19, 2023

Describe the bug
When reading the ChangeFeed for a prefix partition key that spans multiple physical partitions the ChangeFeed will not be able to handle the partition split and will get stuck in an infinite loop.

Expected behavior
Results should be returned.

Actual behavior
ReadNextAsync becomes stuck in an infinite loop.

// Create Container Hierarchical PartitionKeys
this.containerProperties = new ContainerProperties("mycoll", new List<string> { "/State", "/City", "/ZipCode" });
this.container = await database.CreateContainerAsync(this.containerProperties);

// ...
//Container is then filled with enough items span multiple physical partitions 
// ...

//Create a Prefix PartitionKey
Cosmos.PartitionKey prefixPK = new PartitionKeyBuilder().Add("Redmound").Build();

//Create ChangeFeedIterator
FeedRange feedRange = FeedRange.FromPartitionKey(shardPartitionKey);
FeedIterator<dynamic> feedIterator = container.GetChangeFeedIterator<dynamic>(ChangeFeedStartFrom.Beginning(shardFeedRange), ChangeFeedMode.Incremental, new ChangeFeedRequestOptions() { PageSizeHint = 10 });

while (feedIterator .HasMoreResults)
{
    FeedResponse<dynamic> response = await feedIterator .ReadNextAsync(); // BUG: Code gets stuck here!
}
@NaluTripician NaluTripician added bug Something isn't working Hotfix A hotfix is required for the issue labels May 19, 2023
@NaluTripician NaluTripician self-assigned this May 19, 2023
@ealsur
Copy link
Member

ealsur commented May 19, 2023

Related logical investigation steps that yielded the bug:

  1. User sends request with PK with less components
  2. RequestInvokerHandler converts the FeedRangePK to FeedRangeEPK https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos/src/Handler/RequestInvokerHandler.cs#L166-L169
  3. RequestInvokerHandler finds that the FeedRangeEPK spans multiple partitions: https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos/src/Handler/RequestInvokerHandler.cs#L239-L251
  4. Error bubbles to https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos/src/Pagination/CrossPartitionRangePageAsyncEnumerator.cs#L111
  5. Then it calls into the FeedRangeProvider.GetChildRanges https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos/src/Pagination/CrossPartitionRangePageAsyncEnumerator.cs#L113-L118
  6. Which calls into FeedRangePartitionKey and always returns a point range, even if the PK spans multiple ranges: https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos/src/FeedRange/FeedRanges/FeedRangePartitionKey.cs#L25-L37 because that method is simply designed for a normal PK that never goes outside a physical partition.

@NaluTripician NaluTripician changed the title ChangeFeedIterator gets stuck on multihased containers for prefix partitions keys that span multiple physical partitions. ChangeFeedIterator gets stuck on multihashed containers for prefix partitions keys that span multiple physical partitions. May 19, 2023
@amitdhar
Copy link

Hi, any updates on this? Thanks!

@NaluTripician
Copy link
Contributor Author

Hi, any updates on this? Thanks!

Working on a hotfix now, should be out next week.

@amitdhar
Copy link

amitdhar commented Jun 6, 2023

Any status updates?

@ghost ghost closed this as completed in #3879 Jun 12, 2023
@NaluTripician
Copy link
Contributor Author

Issue is fixed with latest release of the SDK (3.35.0)

@SrinikhilReddy SrinikhilReddy added the HierarchicalPartitioning Tag to track issues related to Hierarchical Partitioned containers label Sep 21, 2023
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working HierarchicalPartitioning Tag to track issues related to Hierarchical Partitioned containers Hotfix A hotfix is required for the issue
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

4 participants