Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ingester/Submitter: Remove sleeps #154

Merged
merged 9 commits into from
Aug 2, 2022
Merged

Ingester/Submitter: Remove sleeps #154

merged 9 commits into from
Aug 2, 2022

Conversation

bartelink
Copy link
Collaborator

@bartelink bartelink commented Aug 1, 2022

Improves the baseline impl of two key components in the Propulsion pipeline:

  • Ingester: per Reader (Partition, Tranche etc) loop that manages:
    • accepting data from a single partition's Reader
    • managing the checkpointing as batches are confirmed complete
  • Submitter: single loop that manages:
    • ingesting items supplied by the Ingester instances
    • propagating items into the Scheduler promptly and fairly

The key improvement is that the Pump methods now sleep until a trigger wakes them, rather than using a short sleep time and proactively checking whether some propagation activity has become possible. This provides the following benefits:

  • there's no need to consider/tune the sleep interval
  • under low load, there's dramatically less activity (CPU usage)
  • under high load, there's never an overly long sleep (which might waste potential processing time for a downstream activity)

Also:

  • Consolidated logging to have a single line of output
  • de-emphasizing cycle counts and/or other information that was previously an input to troubleshooting and/or tuning sleep intervals etc
  • Replaced defaulting of ingesterStatsInterval to 5 minutes with an explicit argument (which defaults to the statsInterval in some cases)

@bartelink bartelink changed the title Tune ingester & Submitter ingester/Submitter: Remove sleeps Aug 1, 2022
@bartelink bartelink marked this pull request as ready for review August 1, 2022 21:35
@bartelink bartelink changed the title ingester/Submitter: Remove sleeps Ingester/Submitter: Remove sleeps Aug 2, 2022
@bartelink bartelink merged commit 2c5791b into master Aug 2, 2022
@bartelink bartelink deleted the tune-ingester branch August 2, 2022 22:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant