Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[release-3.4] fix the potential data loss for clusters with only one member #14423

Merged
merged 1 commit into from
Sep 5, 2022

Conversation

serathius
Copy link
Member

@serathius serathius commented Sep 5, 2022

Backport #14400 to v3.4 fixing #14370

Benchmarking showed 4.5% performance degradation.

Command

./bin/etcd --quota-backend-bytes=4300000000
bin/tools/benchmark txn-put --endpoints="http://127.0.0.1:2379" --clients=200 --conns=200 --key-space-size=4000000000 --key-size=128 --val-size=10240  --total=200000 --rate=40000

Before

Summary:
  Total:	23.7596 secs.
  Slowest:	0.0911 secs.
  Fastest:	0.0020 secs.
  Average:	0.0236 secs.
  Stddev:	0.0181 secs.
  Requests/sec:	8417.6492

Response time histogram:
  0.0020 [1]	|
  0.0109 [9664]	|∎∎
  0.0198 [130352]	|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.0287 [20841]	|∎∎∎∎∎∎
  0.0376 [5570]	|∎
  0.0465 [1158]	|
  0.0554 [6105]	|∎
  0.0643 [14826]	|∎∎∎∎
  0.0732 [8162]	|∎∎
  0.0821 [2434]	|
  0.0911 [887]	|

Latency distribution:
  10% in 0.0115 secs.
  25% in 0.0129 secs.
  50% in 0.0155 secs.
  75% in 0.0222 secs.
  90% in 0.0593 secs.
  95% in 0.0654 secs.
  99% in 0.0771 secs.
  99.9% in 0.0883 secs.

After

Summary:
  Total:	24.8914 secs.
  Slowest:	0.1210 secs.
  Fastest:	0.0073 secs.
  Average:	0.0248 secs.
  Stddev:	0.0178 secs.
  Requests/sec:	8034.9071

Response time histogram:
  0.0073 [1]	|
  0.0186 [123685]	|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.0300 [35753]	|∎∎∎∎∎∎∎∎∎∎∎
  0.0414 [5384]	|∎
  0.0528 [5270]	|∎
  0.0642 [19048]	|∎∎∎∎∎∎
  0.0755 [8874]	|∎∎
  0.0869 [1528]	|
  0.0983 [253]	|
  0.1097 [4]	|
  0.1210 [200]	|

Latency distribution:
  10% in 0.0131 secs.
  25% in 0.0141 secs.
  50% in 0.0167 secs.
  75% in 0.0235 secs.
  90% in 0.0592 secs.
  95% in 0.0649 secs.
  99% in 0.0754 secs.
  99.9% in 0.1118 secs.

For a cluster with only one member, the raft always send identical
unstable entries and committed entries to etcdserver, and etcd
responds to the client once it finishes (actually partially) the
applying workflow.

When the client receives the response, it doesn't mean etcd has already
successfully saved the data, including BoltDB and WAL, because:
   1. etcd commits the boltDB transaction periodically instead of on each request;
   2. etcd saves WAL entries in parallel with applying the committed entries.
Accordingly, it may run into a situation of data loss when the etcd crashes
immediately after responding to the client and before the boltDB and WAL
successfully save the data to disk.
Note that this issue can only happen for clusters with only one member.

For clusters with multiple members, it isn't an issue, because etcd will
not commit & apply the data before it being replicated to majority members.
When the client receives the response, it means the data must have been applied.
It further means the data must have been committed.
Note: for clusters with multiple members, the raft will never send identical
unstable entries and committed entries to etcdserver.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
@serathius
Copy link
Member Author

cc @ptabor @ahrtr @tbg

Copy link
Member

@spzala spzala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

@ahrtr ahrtr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Thanks @serathius

I guess it's the same test environment as the PR for 3.5.

@ahrtr ahrtr merged commit b2b7b9d into etcd-io:release-3.4 Sep 5, 2022
@serathius serathius deleted the one_member_data_loss_raft_3_4 branch June 15, 2023 20:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

4 participants