'StopEtcd()' cleanly?

[unknownzerx]

I am trying to embed etcd server in our program, making them live in one process. I would like to StopEtcd() cleanly without killing the process. Is it possible?

Here is a case

1. Start a etcd server
2. From a client, GET a key with ?wait=true
3. Stop etcd without killing the process.
   • I've tried to close some resource unknownzerx@2606aa4fa08c161dacea6e34f69a7e43cf13362c
   • The tcp connection is still open and a goroutine gets blocked at client.go:handleKeyWatch
4. The previous GET on client will never return or timeout

Pass a stopped channel to the select {...} in handleKeyWatch() could solve the problem. But there seems to be other places where a handler blocks and the connection will not get closed. Could you help to figure them out completely?

[yichengq]

We are doing integration tests using embedded etcd, so we are interested in StopEtcd cleanly too.

This goal is totally reachable AFAIK. But it is not high priority today.

Would appreciate that if you could jump in and help on this one.

[iankronquist]

@yichengq in IRC I was told to ask you about this issue. What needs to be done to fix this issue? What files should I start looking at, and what moving parts are missing?

[yichengq]

@iankronquist We have done embedded etcd in integration testing(https://github.com/coreos/etcd/blob/master/integration/cluster_test.go#L799), and it works well in testing.

But there is still some resources left open. One example is The tcp connection is still open and a goroutine gets blocked at client.go:handleKeyWatch (the related code is here: https://github.com/coreos/etcd/blob/master/etcdserver/etcdhttp/client.go#L569) This could be solved by exit the for loop when the server is stopped.

As @unknownzerx mentioned, there may be other blocked handlers.

[unknownzerx]

@yichengq @iankronquist
What is about peer connection?

[yichengq]

@unknownzerx @iankronquist
all peer connection is closed before stopping AFAIK.

Moreover, it may have idle connections maintained by net/http library.

[purpleidea]

@unknownzerx I have a patch to help embedding etcd since I am working on this as well. Please have a look here: #5584

@yichengq and others, I have noticed that in my programs embedding etcd, if I start the embedded server, then stop it, and then start it again, I get a bind: address already in use error because the listen isn't closed properly. I think this probably is part of the same issue, but if not please let me know and I will make a new ticket. I think this would be very much a good thing to have in v3.0.0.

Thank you for your consideration!

[gyuho]

This should be covered by GracefulStop in embed package (works for gRPC requests, not for v2).

[purpleidea]

@gyuho Awesome... However, I don't see any public methods called GracefulStop in that package in git master, I only see: https://github.com/coreos/etcd/blob/master/embed/etcd.go#L264

Am I looking in the wrong place?

Thanks!

[gyuho]

This should do it https://github.com/coreos/etcd/blob/master/embed/etcd.go#L228-L287

Close() method call does graceful shutdown on gRPC-server.

[purpleidea]

@gyuho Okay... To clarify, what you're saying is that the Close method which previously existed now also shuts down the RPC server so that starting and stopping embed repeatedly in a single process will now work, correct?

Secondly should I still use embed.Server.Stop() or not?

Thanks!

PS: Any chance you've got the commit sha1 where this was changed?

[gyuho]

@purpleidea embed.*Etcd.Close() should be enough (which calls embed.Server.Stop(), so no need additional embed.Server.Stop() call).

shuts down the RPC server so that starting and stopping embed repeatedly in a single process will now work, correct?

Yes, it will stop all incoming gRPC requests and gracefully shut down gRPC server. Should be safe to restart.

It was added via c407e09#diff-269605d45c104f12cbcdc8fb71434acc.

[purpleidea]

Thank you so much for the info!

…

On Thu, Oct 5, 2017 at 5:03 PM, Gyu-Ho Lee ***@***.***> wrote: @purpleidea <https://github.com/purpleidea> embed.*Etcd.Close() should be enough (which calls embed.Server.Stop(), so no need additional embed.Server.Stop() call). shuts down the RPC server so that starting and stopping embed repeatedly in a single process will now work, correct? Yes, it will stop all incoming gRPC requests and gracefully shut down gRPC server. Should be safe to restart. It was added via c407e09#diff-269605d45c104f12cbcdc8fb71434acc <c407e09#diff-269605d45c104f12cbcdc8fb71434acc> . — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2920 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIPs4RWobWhkbtiGT9l22K5UQxxY_lFks5spUQigaJpZM4E5jRC> .