Question: promote learning member on a 1 size cluster

[ereslibre]

etcd version: 3.4.3

Given a cluster setup with a single member,

# etcdctl member list
ab75c79a2c3090aa, started, controlplane, http://172.17.0.2:30002, http://172.17.0.2:30003, false

I added a new learner member, and started it.

Member f9d1e8f6c55eb887 added to cluster a9c95578e770c6db

ETCD_NAME="controlplane2"
ETCD_INITIAL_CLUSTER="controlplane=http://172.17.0.2:30002,controlplane2=http://172.17.0.2:30004"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.17.0.2:30004"
ETCD_INITIAL_CLUSTER_STATE="existing"

The member list afterwards adding the learner member, and after having started it.

# etcdctl member list
ab75c79a2c3090aa, started, controlplane, http://172.17.0.2:30002, http://172.17.0.2:30003, false
f9d1e8f6c55eb887, started, controlplane2, http://172.17.0.2:30004, http://172.17.0.2:30005, true

I see that if I now try to promote it to a voting member, I get the following error:

# etcdctl member promote f9d1e8f6c55eb887 
{"level":"warn","ts":"2020-02-16T18:41:05.652Z","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"endpoint://client-0d90d825-371b-4a77-b1e1-bcf99ab3fab2/127.0.0.1:2379","attempt":0,"error":"rpc error: code = Unknown desc = etcdserver: re-configuration failed due to not enough started members"}
Error: etcdserver: re-configuration failed due to not enough started members

This error looks like it's coming from IsReadyToPromoteMember, called when the strict reconfig check is enabled.

This is the error present on the only etcd instance with voting rights during that time:

2020-02-16 18:41:05.651445 W | etcdserver/membership: Reject promote member request: the number of started member (1) will be less than the quorum number of the cluster (2)
2020-02-16 18:41:05.651501 W | etcdserver: not enough started members, rejecting promote member f9d1e8f6c55eb887

I can disable the strict reconfig check, but I'm wondering if even with strict reconfiguration this should not be possible at all by design, or if it's a bug. I'm following the recommendation of only joining one learner node at a time, but seems impossible to succeed if you start with a single instance cluster.

Thank you!

[ereslibre]

At first sight looks like this case would be solved by changing

etcd/etcdserver/api/membership/cluster.go

Line 660 in 3cf2f69

nmembers := 1

to 0, because right now IsReadyToPromoteMember thinks it has two voting members, when there is only one in reality.

I might be missing many important nuances here, though.

[jingyih]

Looks like a bug to me. A cluster with 1 started voting member and 1 started learner member should allow promoting of that learner member.

[ereslibre]

I'll open a PR.

[jingyih]

cc @WIZARD-CXY

[jingyih]

I'll open a PR.

@ereslibre Sounds great! I am thinking it will be helpful if we could have a unit test for that function, something similar to

etcd/etcdserver/api/membership/cluster_test.go

Line 627 in d6a3c99

func TestIsReadyToAddVotingMember(t *testing.T) {

[ereslibre]

@jingyih I created #11640 to fix this issue, but as I comment on the PR I would be eager to increase coverage for the promotion logic on other parts of the code as well.