[3.4] Allow new server to join higher cluster version if NextClusterVersionCompatible is true

[siyuanfoundation]

Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.

Part of #15878 (comment).

#17330 allows existing server to be downgraded to 3.4, with haveWAL=true.
This PR would allow a brand new 3.4 server to join an existing 3.5 cluster.
This would enable the downgrade method of stopping a 3.5 server and starting a new 3.4 server one by one.

Tested with TestMixVersionsSnapshotByAddingMember e2e test in #17531

[siyuanfoundation]

cc @serathius @ahrtr @lavacat

[ahrtr]

/lgtm

Please at least manually verify: add a 3.4 member (with NextClusterVersionCompatible enabled) into a 3.5 cluster for now.

[fuweid]

LGTM

[jmhbnz]

LGTM - Thanks @siyuanfoundation

Note: We will need to update https://github.com/etcd-io/website/blob/main/content/en/docs/v3.4/op-guide/configuration.md also. We can make it a good first issue for a community member perhaps.

[siyuanfoundation]

LGTM - Thanks @siyuanfoundation

Note: We will need to update https://github.com/etcd-io/website/blob/main/content/en/docs/v3.4/op-guide/configuration.md also. We can make it a good first issue for a community member perhaps.

Sounds good. Created good first issue #17671

[siyuanfoundation]

verified manually

1. start 2 node 3.5 cluster

bin/etcd --name infra1 --listen-client-urls http://127.0.0.1:2379 --advertise-client-urls http://127.0.0.1:2379 --listen-peer-urls http://127.0.0.1:12380 --initial-advertise-peer-urls http://127.0.0.1:12380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380' --initial-cluster-state new

bin/etcd --name infra2 --listen-client-urls http://127.0.0.1:22379 --advertise-client-urls http://127.0.0.1:22379 --listen-peer-urls http://127.0.0.1:22380 --initial-advertise-peer-urls http://127.0.0.1:22380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380' --initial-cluster-state new

go run tools/benchmark/main.go put --total=10000000 --key-size=64 --val-size=512

bin/etcdctl --endpoints=http://127.0.0.1:2379,http://127.0.0.1:22379 endpoint status -w table

2. add a new member

bin/etcdctl member add infra3 --peer-urls=http://127.0.0.1:32380

../etcd-3.4/bin/etcd --name infra3 --listen-client-urls http://127.0.0.1:32379 --advertise-client-urls http://127.0.0.1:32379 --listen-peer-urls http://127.0.0.1:32380 --initial-advertise-peer-urls http://127.0.0.1:32380 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state existing --next-cluster-version-compatible

3. verify cluster status

bin/etcdctl --endpoints=http://127.0.0.1:2379,http://127.0.0.1:22379,http://127.0.0.1:32379 endpoint status -w table

# output
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|        ENDPOINT        |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|  http://127.0.0.1:2379 | 8211f1d0f64f3269 |  3.5.12 |  121 MB |      true |      false |         2 |     145815 |             145815 |        |
| http://127.0.0.1:22379 | 91bc3c398fb3c146 |  3.5.12 |  121 MB |     false |      false |         2 |     145815 |             145815 |        |
| http://127.0.0.1:32379 | 7a64c9167531e5c1 |  3.4.31 |  121 MB |     false |      false |         2 |     145817 |             145817 |        |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

[ahrtr]

Thanks for the verification.