Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Remote Store] Make repository name configurable #3286

Closed
sachinpkale opened this issue May 11, 2022 · 1 comment
Closed

[Remote Store] Make repository name configurable #3286

sachinpkale opened this issue May 11, 2022 · 1 comment
Labels
distributed framework enhancement Enhancement or improvement to existing feature or request

Comments

@sachinpkale
Copy link
Member

Describe the solution you'd like
As remote store depends on repository plugin to store/restore segments to/from the remote store, before creating the index with remote_store=true, it is required to create the repository first. As part of the change introduced in #3285, the repository name is hardcoded. But it takes away the flexibility and would create issues if the repository with the same name exists. The repository name should be configurable.

@satyajitg28
Copy link

satyajitg28 commented Jun 10, 2022

Ideas and concerns around this:

  1. Any hard-coded or predetermined name of the repository can conflict with an already existing repository that the customer might have. [Note: @sachinpkale is currently using clusterId as the repository name]
  2. For a configurable name or a name that the customer wants to choose, we might have to expose it as a cluster setting, preferably a non dynamic one (point 3 explains why), and hence, the repository name has to be initialised in the settings file (yml) of all the nodes for it to be picked up. The problem here might be that the repository name is now confined to the scope of the node and not at a global cluster level.
  3. If the customer goes ahead and updates the bucket endpoint of the same repository via the PUT API, it is the same scenario as updating the remote store repository name altogether. In both cases, i.e. new remote store repository being created or the already created repository's bucket endpoints being updated, we will have to do the heavy lifting of pushing all the data of every remote store index again, to this new bucket.

Considering all the above points, can we allow this remote store repository name to be a cluster level dynamic setting (giving the option to the customer to update the name) and calling out the repercussions of this change (re-pushing the entire data in case of the endpoint or the repository name being changed)

Please feel free to add any other thoughts/concerns regarding this.

CC: @nknize @Bukhtawar @sachinpkale

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
distributed framework enhancement Enhancement or improvement to existing feature or request
Projects
None yet
Development

No branches or pull requests

3 participants