Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kubernetes CI Policy: create and enforce policy of removing continuously unhealthy jobs #18601

Open
spiffxp opened this issue Aug 1, 2020 · 16 comments
Open

Kubernetes CI Policy: create and enforce policy of removing continuously unhealthy jobs #18601

spiffxp opened this issue Aug 1, 2020 · 16 comments
Assignees
@dims @hasheddan
Labels
area/jobs help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/documentation Categorizes issue or PR as related to documentation. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. sig/testing Categorizes an issue or PR as relevant to SIG Testing.
Milestone
v1.25

Comments

@spiffxp
Copy link
Member

spiffxp commented Aug 1, 2020

Part of #18551

Once we have:

It's time to craft a policy to maintain a reasonable state of job health going forward!

A rough suggestion is: "continuously failing for O(weeks) means removal"

Perhaps there is more room for nuance here:

  • maybe infrequently scheduled jobs are permissible for longer?
  • do we care about continuously failing jobs not run on prow.k8s.io?
  • what are the logistics of outreach, deadlines, "hang on we can't fix it now" etc.
@spiffxp spiffxp added sig/testing Categorizes an issue or PR as relevant to SIG Testing. kind/documentation Categorizes issue or PR as related to documentation. labels Aug 1, 2020
@spiffxp spiffxp mentioned this issue Aug 1, 2020
Open
8 tasks
@hasheddan
Copy link
Contributor

/assign

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 9, 2020
@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jan 8, 2021
@spiffxp
Copy link
Member Author

spiffxp commented Jan 8, 2021

/remove-lifecycle rotten

@k8s-ci-robot k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Jan 8, 2021
@hasheddan
Copy link
Contributor

/milestone v1.21

@k8s-ci-robot k8s-ci-robot added this to the v1.21 milestone Feb 4, 2021
@liggitt
Copy link
Member

liggitt commented Mar 6, 2021

jobs that fail 100% of Up tests are a good candidate set - https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=%5EUp%24

@dims
Copy link
Member

dims commented Apr 4, 2021

💯 agree with the criteria above from @liggitt

@spiffxp
Copy link
Member Author

spiffxp commented Apr 5, 2021

If someone wants to take a crack at implementing @liggitt's suggestion and assign to me for review I will happily approve

@spiffxp
Copy link
Member Author

spiffxp commented Apr 5, 2021

Don't need to block on contact info. I'll update description when back at keyboard

/help

@k8s-ci-robot
Copy link
Contributor

@spiffxp:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

Don't need to block on contact info. I'll update description when back at keyboard

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. label Apr 5, 2021
@BenTheElder BenTheElder modified the milestones: v1.21, v1.23 Jun 25, 2021
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 23, 2021
@spiffxp
Copy link
Member Author

spiffxp commented Oct 7, 2021

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 7, 2021
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 5, 2022
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Feb 4, 2022
@ameukam
Copy link
Member

ameukam commented Feb 4, 2022

/remove-lifecycle rotten
/lifecycle frozen

@k8s-ci-robot k8s-ci-robot added lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. and removed lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. labels Feb 4, 2022
@dims dims mentioned this issue Apr 18, 2022
Closed
@dims
Copy link
Member

dims commented Apr 18, 2022

/assign

@BenTheElder BenTheElder modified the milestones: v1.23, v1.25 Apr 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dims dims

@hasheddan hasheddan

Labels
area/jobs help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/documentation Categorizes issue or PR as related to documentation. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. sig/testing Categorizes an issue or PR as relevant to SIG Testing.
Projects
None yet
Milestone
v1.25
Development

No branches or pull requests
9 participants
@dims @spiffxp @BenTheElder @liggitt @ameukam @k8s-ci-robot @fejta-bot @hasheddan @k8s-triage-robot