Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[v23.1.x] Put consumers in their own scheduling group #10410

Merged
merged 4 commits into from
Apr 27, 2023
Merged

[v23.1.x] Put consumers in their own scheduling group #10410

merged 4 commits into from
Apr 27, 2023

Commits on Apr 27, 2023

  1. Make _scheduling_groups public in application 

    By analogy with smp_groups which is already public we need to
access this outside of application.cc to pass the right scheduling
groups through to instantiations of services in our redpanda fixture.

(cherry picked from commit d505693)
    @travisdowns @vbotbuildovich
    travisdowns authored and vbotbuildovich committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    230a6f4 View commit details
    Browse the repository at this point in the history

  2. Add fetch scheduling group 

    This scheduling group is used for consumer fetch processing. We assign
it the same priority as the default group (where most other kafka
handling takes place), but by putting it into its own group we prevent
non-fetch requests from being significantly delayed when fetch requests
use all the CPU.

Issue redpanda-data/core-internal#435

(cherry picked from commit d133843)
    @travisdowns @vbotbuildovich
    travisdowns authored and vbotbuildovich committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    6b1a160 View commit details
    Browse the repository at this point in the history

  3. Use fetch scheduling group to handle fetches 

    Add the with_scheduling_group call to switch to the kafka scheduling
group in the fetch handling flow. See prior change for context.

Issue redpanda-data/core-internal/redpanda-data#435.

(cherry picked from commit 9a93a9c)
    @travisdowns @vbotbuildovich
    travisdowns authored and vbotbuildovich committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    6ddc9d1 View commit details
    Browse the repository at this point in the history

  4. Make the fetch scheduling group configurable 

    Introduce a config tunable which would let users set the scheduling
group to use for kafka fetch request back to the default group.

Also useful for quick A/B testing of the perf effect of the change.

(cherry picked from commit bc72f5e)
    @travisdowns @vbotbuildovich
    travisdowns authored and vbotbuildovich committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    60fc4ab View commit details
    Browse the repository at this point in the history