[v21.11.x] redpanda: shutdown quota manager after kafka server is stopped

[mmaslankaprv]

Quota manager refrence is a part of request context. We need to make
sure that quota manager is available as long as the kafka server is
running. Moved quota manager initialization before waiting for Kafka
server shutdown to make sure that its reference is available for all
requests that are being processed.

(cherry picked from commit 3253995)

[andrewhsu]

which milestone should this PR belong to?

[andrewhsu]

adding to the v21.11.17 milestone so we don't lose track of this for future release

[graphcareful]

Just curious why we want to place this cherry-picked commit into a point release. Is it intended to fix something? I remember also observing a problem that necessitated this fix and coming to the realization that its only an issue if the portion of work that is done by connection_context in the background invokes quota_manager which I see nothing currently does.

Also a quota_manager ref cannot be obtained by request_context, so does this change intend to fix something else? Maybe we can change the cover letter then?

[mmaslankaprv]

Just curious why we want to place this cherry-picked commit into a point release. Is it intended to fix something? I remember also observing a problem that necessitated this fix and coming to the realization that its only an issue if the portion of work that is done by connection_context in the background invokes quota_manager which I see nothing currently does.

Also a quota_manager ref cannot be obtained by request_context, so does this change intend to fix something else? Maybe we can change the cover letter then?

This fixes shutdown sequence, it may lead to segmentation fault during redpanda shutdown.

[graphcareful]

Just curious why we want to place this cherry-picked commit into a point release. Is it intended to fix something? I remember also observing a problem that necessitated this fix and coming to the realization that its only an issue if the portion of work that is done by connection_context in the background invokes quota_manager which I see nothing currently does.
Also a quota_manager ref cannot be obtained by request_context, so does this change intend to fix something else? Maybe we can change the cover letter then?

This fixes shutdown sequence, it may lead to segmentation fault during redpanda shutdown.

Ok, how though? I see that background work is protected by the connections gate as well

[mmaslankaprv]

Just curious why we want to place this cherry-picked commit into a point release. Is it intended to fix something? I remember also observing a problem that necessitated this fix and coming to the realization that its only an issue if the portion of work that is done by connection_context in the background invokes quota_manager which I see nothing currently does.
Also a quota_manager ref cannot be obtained by request_context, so does this change intend to fix something else? Maybe we can change the cover letter then?

This fixes shutdown sequence, it may lead to segmentation fault during redpanda shutdown.

Ok, how though? I see that background work is protected by the connections gate as well

Before this patch quota_manager was stopped before Kafka RPC server was stopped. Connection context hold a reference to ss::sharded<quota_manager> but the sharded object is already stopped i.e. all underlying thread local instances are destroyed.

[dotnwat]

Needs CI failure linked to issue.

[mmaslankaprv]

Needs CI failure linked to issue.

I am not sure if that was a CI failure we spotted that in one of the consumer clusters. It was related with the fix for #4051

[dotnwat]

@graphcareful what's left for approval on this PR?

[graphcareful]

@graphcareful what's left for approval on this PR?

All good , LGTM

[mmaslankaprv]

@graphcareful can you click Approve button