Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HPA targeting a Rollout scales down AnalysisRun Job generated pods #3828

Open
2 tasks done
davcd opened this issue Sep 10, 2024 · 0 comments
Open
2 tasks done

HPA targeting a Rollout scales down AnalysisRun Job generated pods #3828

davcd opened this issue Sep 10, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@davcd
Copy link

davcd commented Sep 10, 2024

Checklist:

  • I've included steps to reproduce the bug.
  • I've included the version of argo rollouts.

Describe the bug

Having a Rollout that runs an analysis, and a HPA targeting to that Rollout. If an scale down event occurs during the execution of the analysis, the pod generated by the AnalysisRun Job is terminated. Also the Rollout proceeds as passed even the analysis did not finish.

To Reproduce

Having the following resources

  • Rollout with an AnalysisRun
apiVersion: argoproj.io/v1alpha1
kind: Rollout
...
spec:
  ...
  strategy:
    blueGreen:
      activeService: my-service
      previewService: my-service-preview
      prePromotionAnalysis:
        templates:
          - templateName: some-analysis-maybe-e2e
  • HPA referencing a Rollout:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
...
spec:
  maxReplicas: Y
  minReplicas: X
  ...
  scaleTargetRef:
    apiVersion: argoproj.io/v1alpha1
    kind: Rollout
    name: rollout-name

And the following events occurrs:

  • A new Rollout starts:

    • Preview pods are booted
    • Pre promotion analysis run job starts, creating analysis pod
  • For some reason (ie burst traffic end) an scale down event is generated:

    • Both old and preview replicaset are scaled down
    • Analysis pod is terminated too

Expected behavior

AnalysisRun Job generated pods are not affected by HPA

Version
v1.7.2+59e5bd3


Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

@davcd davcd added the bug Something isn't working label Sep 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant