embed: Can't restart etcd

[purpleidea]

Etcd now has an embed API. Sweet :)

I believe there is still a bind problem when starting a server, then stopping it, and then starting it again. On the second start, you'll get a bind: address already in use error because the listen isn't closed properly.

Opening this issue at the request of @heyitsanthony

I think this may be casually related to #2920

Thanks!

[gyuho]

Can you provide code snippet to reproduce? Thanks.

[purpleidea]

@gyuho Fair enough, I don't have a trivial snippet at the moment, I'll reopen when I get one. Thanks!

[purpleidea]

I forget if this is related to: golang/go#4674 or not.

[purpleidea]

Apologies for the delay, but I can reliably reproduce this issue, so I will re-open this. I have an easy POC inside of https://github.com/purpleidea/mgmt/

[purpleidea]

Here is the reproducer:

start up three members...

./mgmt run --hostname h1 --tmp-prefix --no-pgp empty
./mgmt run --hostname h2 --tmp-prefix --no-pgp --seeds http://127.0.0.1:2379 --client-urls http://127.0.0.1:2381 --server-urls http://127.0.0.1:2382 empty
./mgmt run --hostname h3 --tmp-prefix --no-pgp --seeds http://127.0.0.1:2379 --client-urls http://127.0.0.1:2383 --server-urls http://127.0.0.1:2384 empty

tell the ideal cluster size to be three...

ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 put /_mgmt/chooser/dynamicsize/idealclustersize 3

check that it is...

ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 member list

add two more clients...

./mgmt run --hostname h4 --tmp-prefix --no-pgp --seeds http://127.0.0.1:2379 --client-urls http://127.0.0.1:2385 --server-urls http://127.0.0.1:2386 empty
./mgmt run --hostname h5 --tmp-prefix --no-pgp --seeds http://127.0.0.1:2379 --client-urls http://127.0.0.1:2387 --server-urls http://127.0.0.1:2388 empty

tell the cluster size to be 4...

ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 put /_mgmt/chooser/dynamicsize/idealclustersize 4

one more member will be started now...

ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 member list

set it back to three...

ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 put /_mgmt/chooser/dynamicsize/idealclustersize 3

make note of who shutdown...

ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 member list

bring it back to 4...

ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 put /_mgmt/chooser/dynamicsize/idealclustersize 4

you'll most likely have one that previously started, try to start again... repeat the above 3->4->3 if not.

ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 member list

in the logs of that member:

main.go:385: etcd: runtime error: listen tcp 127.0.0.1:2384: bind: address already in use
server start failed

I believe I am using the embed API correctly, but if there is some additional shutdown/unbind step that I should be performing that I am not, then please let me know. Thanks!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[purpleidea]

hi bot! please stop pinging here

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[cretz]

Bump, I don't believe this is resolved (IIRC in my integration tests cannot stop and start up embedded etcd again due to this issue). At the least deserves confirmation that's fixed before closing.

[purpleidea]

@cretz It's not fixed last I checked, but I'm giving up fighting with the bots, it's very end-user hostile I think. One bump should be enough for the lifetime of the bug. And the bot bugged about eight others today. :/

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[purpleidea]

Stop closing this bot. This is important.

[ptabor]

I believe it might be about (etcd not using) 'SO_REUSEADDR'. Without this setting port is 'locked' for additional 60-120s,
to avoid getting left-over communication from previous customers.

That's pretty good article about this:
https://stackoverflow.com/questions/3229860/what-is-the-meaning-of-so-reuseaddr-setsockopt-option-linux/3233022#3233022
and https://hea-www.harvard.edu/~fine/Tech/addrinuse.html

[purpleidea]

Think you can send a patch to fix this?

…

On Mon, Oct 5, 2020 at 4:34 AM Piotr Tabor ***@***.***> wrote: I believe it might be about (etcd not using) 'SO_REUSEADDR'. Without this setting port is 'locked' for additional 60-120s, to avoid getting left-over communication from previous customers. That's pretty good article about this: https://stackoverflow.com/questions/3229860/what-is-the-meaning-of-so-reuseaddr-setsockopt-option-linux/3233022#3233022 and https://hea-www.harvard.edu/~fine/Tech/addrinuse.html — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or unsubscribe.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[purpleidea]

bot

[ptabor]

I think / hope it got fixed in:

#12702

Please test and reopen.

[purpleidea]

@ptabor Fantastic news, thanks! I'll test in 3.5