From b5a8e61d00eec69abb808fe55d1ee89993835773 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jean-S=C3=A9bastien=20P=C3=A9dron?= Date: Mon, 9 Aug 2021 16:28:39 +0200 Subject: [PATCH] systemd: Do not report "stopping" state to systemd MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The problem is we only know about the state of the `rabbit` Erlang application — when it is started and stopped. But we can't know the fate of the Erlang VM, except if `rabbit:stop_and_halt()` is called. This function is not called if `init:stop()` or a SIGTERM are used for instance. systemd is interested in the state of the system process (the Erlang VM), not what's happening inside. But inside, we have multiple situations where the Erlang application is stopped, but not the Erlang VM. For instance: * When clustering, the Erlang application is stopped before the cluster is created or expanded. The application is restarted once done. This is controled either manually or using the peer discovery plugins. * The `pause_minority` or `pause_if_all_down` partition strategies both stop the Erlang application for an indefinite period of time, but RabbitMQ as a service is still up (even though it is managing its own degraded mode and no connections are accepted). In both cases, the service is still running from the system's service manager's point of view. As said above, we can never tell "the VM is being terminated" with confidence. We can only know about the Erlang application itself. Therefore, it is best to report the latter as a systemd state description, but not reporting the "STOPPING=1" state at all. systemd will figure out itself that the Erlang VM exited anyway. Before this change, we were reporting the "STOPPING=1" state to systemd every time the Elang application was stopped. The problem was that systemd expected the system process (the Erlang VM) to exit within a configured period of time (90 seconds by default) or report that's it's ready again ("READY=1"). This issue remained unnoticed when the cluster was created/expanded because it probably happened within that time frame. However, it was reported with the partition healing strategies because the partition might last longer than 90 seconds. When this happened, the Erlang VM was killed (SIGKILL) and the service restarted. References #3262. Fixes #3289. (cherry picked from commit 23c71b254f3041a5871443853b57d9ea86c7ad15) --- .../rabbitmq_prelaunch/src/rabbit_boot_state_systemd.erl | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/deps/rabbit/apps/rabbitmq_prelaunch/src/rabbit_boot_state_systemd.erl b/deps/rabbit/apps/rabbitmq_prelaunch/src/rabbit_boot_state_systemd.erl index 5627a65daa68..4a031ceacc13 100644 --- a/deps/rabbit/apps/rabbitmq_prelaunch/src/rabbit_boot_state_systemd.erl +++ b/deps/rabbit/apps/rabbitmq_prelaunch/src/rabbit_boot_state_systemd.erl @@ -45,8 +45,7 @@ code_change(_OldVsn, State, _Extra) -> %% Private -notify_boot_state(BootState) - when BootState =:= ready orelse BootState =:= stopping -> +notify_boot_state(ready = BootState) -> Status = boot_state_to_desc(BootState), ?LOG_DEBUG( ?LOG_PREFIX "notifying of state `~s`", @@ -62,7 +61,7 @@ notify_boot_state(BootState) -> systemd:notify({status, Status}). boot_state_to_desc(stopped) -> - ""; + "Standing by"; boot_state_to_desc(booting) -> "Startup in progress"; boot_state_to_desc(core_started) ->