Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DNM] kubelet: debug phase logic for daemonset pods #705

Closed
wants to merge 1 commit into from

Conversation

ehashman
Copy link

Trying to see if this addresses Bug 1933760. Adds some debug logging to assist.

@openshift-ci-robot openshift-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. backports/unvalidated-commits Indicates that not all commits come to merged upstream PRs. labels Apr 22, 2021
@openshift-ci-robot
Copy link

@ehashman: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

@openshift-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ehashman
To complete the pull request process, please assign sjenning after the PR has been reviewed.
You can assign the PR to them by writing /assign @sjenning in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot
Copy link

@ehashman: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

@ehashman ehashman marked this pull request as draft April 28, 2021 19:58
@ehashman ehashman changed the title [WIP] DNM: kubelet: add case for missing phase [DNM] kubelet: debug phase logic for daemonset pods Apr 28, 2021
@openshift-ci-robot
Copy link

@ehashman: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

@ehashman
Copy link
Author

/test e2e-gcp-upgrade

1 similar comment
@ehashman
Copy link
Author

/test e2e-gcp-upgrade

@ehashman
Copy link
Author

/test e2e-aws-upgrade

@openshift-ci-robot
Copy link

@ehashman: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

@ehashman
Copy link
Author

/test e2e-gcp-upgrade
/test e2e-aws-upgrade

@openshift-ci-robot
Copy link

@ehashman: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

@ehashman
Copy link
Author

ehashman commented May 7, 2021

/test e2e-gcp-upgrade
/test e2e-aws-upgrade

@openshift-ci
Copy link

openshift-ci bot commented May 7, 2021

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ehashman
To complete the pull request process, please assign sjenning after the PR has been reviewed.
You can assign the PR to them by writing /assign @sjenning in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

pkg/kubelet/kubelet_pods.go Outdated Show resolved Hide resolved
@openshift-ci-robot
Copy link

@ehashman: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

@ehashman
Copy link
Author

ehashman commented May 7, 2021

/test e2e-gcp-upgrade
/test e2e-aws-upgrade

@ehashman ehashman marked this pull request as ready for review May 7, 2021 21:11
@ehashman
Copy link
Author

ehashman commented May 7, 2021

/test all

@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 7, 2021
@ehashman ehashman marked this pull request as draft May 12, 2021 19:40
@ehashman
Copy link
Author

ehashman commented Jun 4, 2021

/test unit

This causes a regression in the kubelet_test.go unit tests. Trying to decide if how I've implemented the fix is incorrect or if there's some other cleaning up that needs to happen elsewhere.

@openshift-ci-robot
Copy link

@ehashman: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

@ehashman
Copy link
Author

ehashman commented Jun 5, 2021

/test e2e-gcp-upgrade
/test unit

@ehashman
Copy link
Author

ehashman commented Jun 9, 2021

/test e2e-gcp-upgrade
/test unit

@openshift-ci-robot
Copy link

@ehashman: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

@ehashman
Copy link
Author

ehashman commented Jun 9, 2021

/test e2e-gcp-upgrade
/test e2e-aws-upgrade

@openshift-ci-robot
Copy link

@ehashman: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

Comment on lines -1788 to -1790
if pod.DeletionTimestamp == nil {
continue
}
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I finally got it. This section here was preventing pods that didn't have a DeletionTimestamp set for some reason from getting an unknown termination set.

So, remove this check, and increment restart count by one if this is not set below.

@openshift-ci-robot
Copy link

@ehashman: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

status := statuses[container.Name]
// if the status we're about to write indicates the default, the Waiting status will force this pod back into Pending.
// That isn't true, we know the pod is going away.
// That isn't true, we know the pod was restarted.
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is sort of the crux of the issue -- previously, this logic only toggled on termination, but it's possible for pods to need this logic on restart (e.g. daemonsets, node crash).

@ehashman
Copy link
Author

/test e2e-gcp-upgrade
/test unit

@openshift-ci
Copy link

openshift-ci bot commented Jun 12, 2021

@ehashman: The following tests failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/prow/verify ce18f1365df0313f6e19a53fe968d412761b768e link /test verify
ci/prow/e2e-aws-csi ce18f1365df0313f6e19a53fe968d412761b768e link /test e2e-aws-csi
ci/prow/e2e-aws-fips ce18f1365df0313f6e19a53fe968d412761b768e link /test e2e-aws-fips
ci/prow/e2e-aws-upgrade 26967780c7e1c22cc356ed4ee825269660a1d896 link /test e2e-aws-upgrade
ci/prow/e2e-gcp-upgrade 34062e1 link /test e2e-gcp-upgrade

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ehashman
Copy link
Author

Only 22 pods moved back to pending and the unit tests pass 🎉 🎉

@ehashman ehashman mentioned this pull request Jun 14, 2021
@openshift-bot
Copy link

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

@openshift-ci openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 12, 2021
@openshift-ci
Copy link

openshift-ci bot commented Sep 12, 2021

@ehashman: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 12, 2021
@openshift-bot
Copy link

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

@openshift-ci openshift-ci bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Oct 12, 2021
@openshift-bot
Copy link

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

@openshift-ci openshift-ci bot closed this Nov 11, 2021
@openshift-ci
Copy link

openshift-ci bot commented Nov 11, 2021

@openshift-bot: Closed this PR.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backports/unvalidated-commits Indicates that not all commits come to merged upstream PRs. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants