Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: determine autoshard for a given content topic #1712

Closed
Tracked by #65
adklempner opened this issue Nov 8, 2023 · 0 comments · Fixed by #1718
Closed
Tracked by #65

feat: determine autoshard for a given content topic #1712

adklempner opened this issue Nov 8, 2023 · 0 comments · Fixed by #1718
Assignees
Labels
E:1.2: Autosharding for autoscaling See https://github.com/waku-org/pm/issues/65 for details

Comments

@adklempner
Copy link
Member

This is a feature request

Problem

In order to determine the pubsub topic to use for a content topic in the context of autosharding, we need to map the content topic to a shard number using the following algorithm: https://rfc.vac.dev/spec/51//#algorithm

Proposed Solutions

Add a function to utils, which, given a string and number of shards:

  • validates that the string is a valid content topic
  • parses application and version fields from the string
  • SHA-256 hash the concatenation of the UTF-8 string bytes of the application and version fields
  • perform hash mod numShards to determine which shard index to use

Notes

only use the last 64 bits of the hash when performing modulo step

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
E:1.2: Autosharding for autoscaling See https://github.com/waku-org/pm/issues/65 for details
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

1 participant