ScaledJob: introduce `MultipleScalersCalculation` #2016

TsuyoshiUshio · 2021-08-06T18:38:55Z

This PR fix issue of multi scalers on Scaled Job. And introduce a new option.
The current login assume only one scaler is active and available.
The most of the cases, it is ok for that. That is why we don't get any issues about it.

However @zroubalik pointed out this issue, I come up with the fix and do some refactoring to make it clear.

I'm not writing documentation yet, before that, I'd like to confirm this logic if fine for you guys.
Once It looks, ok, I'll go ahead full testing and write the documentation and make this PR as not draft.

Spec

Introduce a new option MultipleScalersCalculation on ScaledJob.

apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: {scaled-job-name}
spec:
  jobTargetRef:
         :
  scalingStrategy:
    strategy: "custom"                        # Optional. Default: default. Which Scaling Strategy to use. 
    customScalingQueueLengthDeduction: 1      # Optional. A parameter to optimize custom ScalingStrategy.
    customScalingRunningJobPercentage: "0.5"  # Optional. A parameter to optimize custom ScalingStrategy.
    pendingPodConditions:                     # Optional. A parameter to calculate pending job count per the specified pod conditions
      - "Ready"
      - "PodScheduled"
      - "AnyOtherCustomPodCondition"
    MultipleScalersCalculation: "max" # Optional. Default: max. If multiple scaler is exists, how to calculate the queueLength and maxValue (that is the expected number of pods that will be created)
  triggers:

Available Options

The option is how keda calculate queueLength and maxValue that is the number of pod will be created.

max : default, pick a scaler that has the max number of queueLength.
min: pick a scaler that has the min number of queueLength.
avg: sum up all the active scalers metrics and devided by the number of the active scalers.
sum: sum up all the active scalers metrics

Checklist

Commits are signed with Developer Certificate of Origin (DCO - learn more)
Tests have been added
A PR is opened to update the documentation on (repo) (if applicable)
Changelog has been updated

Fixes #

Document #
kedacore/keda-docs#505

Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

TsuyoshiUshio · 2021-08-09T03:58:37Z

I tested, and I add test case and improve logging for clarifying on the multiple scalers scenario.
One question is, the change of controller-gen.kubebuilder.io/version: v0.5.0 from v0.3.0 Should I keep it to v0.3.0 ? It is auto generated, so that I keep it for now.

config/crd/bases/keda.sh_clustertriggerauthentications.yaml

config/crd/bases/keda.sh_scaledjobs.yaml

pkg/scaling/scale_handler.go

Co-authored-by: Zbynek Roubalik <726523+zroubalik@users.noreply.github.com> Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

…lation Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

TsuyoshiUshio · 2021-09-04T20:03:49Z

Hi @zroubalik
I update this PR as you requested. Could you have a look?
I also update the documentation according to the name suggestion.
kedacore/keda-docs#505

CHANGELOG.md

pkg/metrics/scaledjob_prometheus_metrics.go

pkg/scaling/scaledjob/scale_metrics.go

Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

TsuyoshiUshio · 2021-09-08T04:36:39Z

Hi @zroubalik
I updated the PR. Could you have a look?

zroubalik

Looking good! I have a few very minor nits. Thanks @TsuyoshiUshio !

pkg/scaling/scaledjob/scale_metrics.go

pkg/scaling/scaledjob/scale_metrics_test.go

pkg/scaling/scaledjob/scale_metrics.go

CHANGELOG.md

Co-authored-by: Zbynek Roubalik <726523+zroubalik@users.noreply.github.com> Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

TsuyoshiUshio · 2021-09-11T19:11:34Z

Hi @zroubalik I accept all your suggestion, however, CI / Validate PR looks start to fail. I looked the code change again, however, I just changed ScaledJob related part. The CI fails with the ScaledObjectController. Do you know this issue?

ScaledObjectController
/__w/keda/keda/controllers/keda/scaledobject_controller_test.go:50
  functional tests
  /__w/keda/keda/controllers/keda/scaledobject_controller_test.go:204
    deploys ScaledObject and creates HPA, when IdleReplicaCount, MinReplicaCount and MaxReplicaCount is defined [It]
    /__w/keda/keda/controllers/keda/scaledobject_controller_test.go:269

    Timed out after 20.001s.
    Expected
        <v1.ConditionStatus>: Unknown
    to equal
        <v1.ConditionStatus>: True

zroubalik · 2021-09-13T07:43:53Z

Hi @zroubalik I accept all your suggestion, however, CI / Validate PR looks start to fail. I looked the code change again, however, I just changed ScaledJob related part. The CI fails with the ScaledObjectController. Do you know this issue?
ScaledObjectController
/__w/keda/keda/controllers/keda/scaledobject_controller_test.go:50
  functional tests
  /__w/keda/keda/controllers/keda/scaledobject_controller_test.go:204
    deploys ScaledObject and creates HPA, when IdleReplicaCount, MinReplicaCount and MaxReplicaCount is defined [It]
    /__w/keda/keda/controllers/keda/scaledobject_controller_test.go:269

    Timed out after 20.001s.
    Expected
        <v1.ConditionStatus>: Unknown
    to equal
        <v1.ConditionStatus>: True

That is just a flaky test :( I rerun the job and it fine now.

zroubalik

Looking good, great job @TsuyoshiUshio !

* Fix bug of ScaledJob multiple triggers Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com> * fix pre-commit issue Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com> * Update logs on scale handler and add crds Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com> * update change log Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com> * Add test case for exceeding MaxReplicaCount in sum Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com> * fix pre-commit Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com> * Update pkg/scaling/scale_handler.go Co-authored-by: Zbynek Roubalik <726523+zroubalik@users.noreply.github.com> Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com> * Update pkg/scaling/scale_handler.go Co-authored-by: Zbynek Roubalik <726523+zroubalik@users.noreply.github.com> Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com> * revert version and refactor scaling logic Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com> * Change option name from MultipleScalersOption to MultipleScalersCalculation Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com> * update changelog for multiple triggers section Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com> * Error Handling Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com> * Refactor logging Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com> * remove scaledjob prometheus Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com> * Update pkg/scaling/scaledjob/scale_metrics.go Co-authored-by: Zbynek Roubalik <726523+zroubalik@users.noreply.github.com> Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com> * Update pkg/scaling/scaledjob/scale_metrics.go Co-authored-by: Zbynek Roubalik <726523+zroubalik@users.noreply.github.com> Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com> * Update pkg/scaling/scaledjob/scale_metrics.go Co-authored-by: Zbynek Roubalik <726523+zroubalik@users.noreply.github.com> Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com> * Update pkg/scaling/scaledjob/scale_metrics_test.go Co-authored-by: Zbynek Roubalik <726523+zroubalik@users.noreply.github.com> Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com> * Update pkg/scaling/scaledjob/scale_metrics.go Co-authored-by: Zbynek Roubalik <726523+zroubalik@users.noreply.github.com> Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com> * Update CHANGELOG.md Co-authored-by: Zbynek Roubalik <726523+zroubalik@users.noreply.github.com> Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com> * Move section to new section Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com> * fix pre-commit white space issue Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com> Co-authored-by: Zbynek Roubalik <726523+zroubalik@users.noreply.github.com> Signed-off-by: nilayasiktoprak <nilayasiktoprak@gmail.com>

taylorchu · 2021-12-04T08:49:16Z

is there a reason why we use queuelength instead of maxscale in max/min/avg calculation?

For example consider 2 scalers:

queuelength=10, target=4 will produce maxscale=3
queuelength=5, target=1 will produce maxscale=5

Right now max in this implementation, will use maxscale=3. To me it makes more sense to use maxscale=5. Similarly min seems more intuitive to pick the one that will produce maxscale=3.

taylorchu · 2021-12-04T08:49:41Z

@TsuyoshiUshio @zroubalik thanks!

Fix bug of ScaledJob multiple triggers

c0c4814

Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

TsuyoshiUshio requested a review from zroubalik August 6, 2021 18:40

TsuyoshiUshio added 4 commits August 6, 2021 11:44

fix pre-commit issue

16671c8

Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

Update logs on scale handler and add crds

12dc683

Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

update change log

2416bfa

Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

Add test case for exceeding MaxReplicaCount in sum

96b9907

Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

TsuyoshiUshio mentioned this pull request Aug 9, 2021

Add MultipleScalersCalculation explanation kedacore/keda-docs#505

Merged

1 task

fix pre-commit

d273f13

Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

TsuyoshiUshio marked this pull request as ready for review August 9, 2021 03:56

TsuyoshiUshio requested a review from ahmelsayed as a code owner August 9, 2021 03:56

zroubalik reviewed Aug 17, 2021

View reviewed changes

zroubalik mentioned this pull request Aug 18, 2021

Add support for the "Value" metric type in addition to "AverageValue" #2030

Closed

TsuyoshiUshio and others added 5 commits September 4, 2021 09:50

Update pkg/scaling/scale_handler.go

12a06ae

Co-authored-by: Zbynek Roubalik <726523+zroubalik@users.noreply.github.com> Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

Update pkg/scaling/scale_handler.go

a9d9476

Co-authored-by: Zbynek Roubalik <726523+zroubalik@users.noreply.github.com> Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

revert version and refactor scaling logic

c5e4cd0

Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

Merge branch 'main' into tsushi/fixscaledjobmultitrigger

e604a68

Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

Change option name from MultipleScalersOption to MultipleScalersCalcu…

784bc7c

…lation Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

TsuyoshiUshio force-pushed the tsushi/fixscaledjobmultitrigger branch from 141bd85 to 784bc7c Compare September 4, 2021 19:52

zroubalik reviewed Sep 7, 2021

View reviewed changes

zroubalik changed the title ~~Fix issue of ScaledJob scaling and introduce MultiScalersOption~~ ScaledJob: introduce MultipleScalersCalculation Sep 7, 2021

TsuyoshiUshio force-pushed the tsushi/fixscaledjobmultitrigger branch from e5b9da9 to cddf760 Compare September 8, 2021 03:35

TsuyoshiUshio added 4 commits September 7, 2021 20:44

update changelog for multiple triggers section

15e39d9

Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

Error Handling

72147d2

Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

Refactor logging

93568a9

Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

remove scaledjob prometheus

96116f8

Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

TsuyoshiUshio force-pushed the tsushi/fixscaledjobmultitrigger branch from cddf760 to 96116f8 Compare September 8, 2021 03:45

zroubalik requested changes Sep 10, 2021

View reviewed changes

zroubalik added this to the v2.5.0 milestone Sep 10, 2021

TsuyoshiUshio and others added 7 commits September 11, 2021 10:43

Update pkg/scaling/scaledjob/scale_metrics.go

d5b519c

Co-authored-by: Zbynek Roubalik <726523+zroubalik@users.noreply.github.com> Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

Update pkg/scaling/scaledjob/scale_metrics.go

49c6eb4

Co-authored-by: Zbynek Roubalik <726523+zroubalik@users.noreply.github.com> Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

Update pkg/scaling/scaledjob/scale_metrics.go

3821105

Co-authored-by: Zbynek Roubalik <726523+zroubalik@users.noreply.github.com> Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

Update pkg/scaling/scaledjob/scale_metrics_test.go

faa30df

Co-authored-by: Zbynek Roubalik <726523+zroubalik@users.noreply.github.com> Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

Update pkg/scaling/scaledjob/scale_metrics.go

4c3bf05

Co-authored-by: Zbynek Roubalik <726523+zroubalik@users.noreply.github.com> Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

Update CHANGELOG.md

87740cf

Co-authored-by: Zbynek Roubalik <726523+zroubalik@users.noreply.github.com> Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

Move section to new section

6b7e6da

Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

TsuyoshiUshio force-pushed the tsushi/fixscaledjobmultitrigger branch from 41b2c15 to 6b7e6da Compare September 11, 2021 17:49

fix pre-commit white space issue

6dda3c9

Signed-off-by: Tsuyoshi Ushio <ushio@simplearchitect.com>

zroubalik approved these changes Sep 13, 2021

View reviewed changes

TsuyoshiUshio merged commit af5981a into main Sep 13, 2021

TsuyoshiUshio deleted the tsushi/fixscaledjobmultitrigger branch September 13, 2021 17:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ScaledJob: introduce `MultipleScalersCalculation` #2016

ScaledJob: introduce `MultipleScalersCalculation` #2016

TsuyoshiUshio commented Aug 6, 2021 •

edited by zroubalik

Loading

TsuyoshiUshio commented Aug 9, 2021

TsuyoshiUshio commented Sep 4, 2021

TsuyoshiUshio commented Sep 8, 2021

zroubalik left a comment

TsuyoshiUshio commented Sep 11, 2021

zroubalik commented Sep 13, 2021

zroubalik left a comment

taylorchu commented Dec 4, 2021

taylorchu commented Dec 4, 2021

ScaledJob: introduce MultipleScalersCalculation #2016

ScaledJob: introduce MultipleScalersCalculation #2016

Conversation

TsuyoshiUshio commented Aug 6, 2021 • edited by zroubalik Loading

Spec

Available Options

Checklist

TsuyoshiUshio commented Aug 9, 2021

TsuyoshiUshio commented Sep 4, 2021

TsuyoshiUshio commented Sep 8, 2021

zroubalik left a comment

Choose a reason for hiding this comment

TsuyoshiUshio commented Sep 11, 2021

zroubalik commented Sep 13, 2021

zroubalik left a comment

Choose a reason for hiding this comment

taylorchu commented Dec 4, 2021

taylorchu commented Dec 4, 2021

ScaledJob: introduce `MultipleScalersCalculation` #2016

ScaledJob: introduce `MultipleScalersCalculation` #2016

TsuyoshiUshio commented Aug 6, 2021 •

edited by zroubalik

Loading