Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimise memberlist kv store access by storing data unencoded. #4345

Merged
merged 3 commits into from
Jul 9, 2021
Merged

Optimise memberlist kv store access by storing data unencoded. #4345

merged 3 commits into from
Jul 9, 2021

Commits on Jul 9, 2021

  1. Optimise memberlist kv store access by storing data unencoded. 

    The following profile data was taken from running 50 idle ingesters with
memberlist, with almost everything at default values (5s heartbeats):

```
52.16% mergeBytesValueForKey
+- 52.16% mergeValueForKey
   +- 47.84% computeNewValue
      +- 27.24% codec Proto Decode
      +- 26.25% mergeWithTime
```

It is apparent from the this that a lot of time is spent on the memberlist
receive path, as might be expected, specifically, the merging of the update
into the current state. The cost however is not in decoding the incoming
states (occurs in `mergeBytesValueForKey` before `mergeValueForKey`), but
in fact decoding _current state_ of the value in the store (as it is stored
encoded). The ring state was measured at 123K (50 ingesters), so it makes
sense that decoding could be costly.

This can be avoided by storing the value in it's decoded `Mergeable` form.
When doing this, care has to be taken to deep copy the value when
accessed, as it is modified in place before being updated in the store,
and accessed outside the store mutex.

Note a side effect of this change is that is no longer straightforward
to expose the `memberlist_kv_store_value_bytes` metric, as this reported
the size of the encoded data, therefore it has been removed.

Signed-off-by: Steve Simpson <steve.simpson@grafana.com>
    @stevesg
    stevesg committed Jul 9, 2021
    Configuration menu
    Copy the full SHA
    d8e37b0 View commit details
    Browse the repository at this point in the history

  2. Typo. 

    Signed-off-by: Steve Simpson <steve.simpson@grafana.com>
    @stevesg
    stevesg committed Jul 9, 2021
    Configuration menu
    Copy the full SHA
    375e813 View commit details
    Browse the repository at this point in the history

  3. Review comments. 

    Signed-off-by: Steve Simpson <steve.simpson@grafana.com>
    @stevesg
    stevesg committed Jul 9, 2021
    Configuration menu
    Copy the full SHA
    30af11b View commit details
    Browse the repository at this point in the history