Distributor ring not removing it's key from kvstore during shutdown.

[kavirajk]

Describe the bug
We found in our clusters, during rollout, the distributors that are leaving the ring failed to remove i'ts key from the underlying kvstore (consul in our case)
"unhealthy" on /distributor/ring status page.

To Reproduce
Steps to reproduce the behavior:
Happens on tip of Loki's main branch

Expected behavior
When distributor that is leaving the ring (during rollout for example) should also cleanup it's key from kvstore properly.

Environment:

• Kubernetes

Related:

• distributor ring introduced in this PR

[trevorwhitney]

@kavirajk I'm not sure the ring was added in #4938. I know that PR first exposed the page, but wasn't there always a distributor ring used for when limit enforcement strategy was set to global?

[kavirajk]

@trevorwhitney yes you are absolutely right. Ring was there even before that PR. We found the issue is actually because of the subtle behavior of CAS operation on KV store the ring uses (consul in our case).

Explained in more detail in this issue on dskit repo.