Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use UnmountMountPoint util to clean up subpaths #71804

Merged
merged 7 commits into from
Jan 4, 2019

Conversation

msau42
Copy link
Member

@msau42 msau42 commented Dec 6, 2018

What type of PR is this?
/kind bug

What this PR does / why we need it:
UnmountMountPoint is a wrapper util that handles corrupt and stale mounts. Cleaning subpaths mounts should use the util instead of directly calling Unmount.

Which issue(s) this PR fixes:

Fixes #71584

Special notes for your reviewer:
To make this small for backporting, I've tried to do the minimum changes required and will look into refactoring/cleanup afterwards.

Does this PR introduce a user-facing change?:

Fixes issue with cleaning up stale NFS subpath mounts

@k8s-ci-robot k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Dec 6, 2018
@msau42
Copy link
Member Author

msau42 commented Dec 6, 2018

/assign @jsafrane

pkg/util/mount/mount_linux.go Outdated Show resolved Hide resolved
} else if IsCorruptedMnt(err) {
return true, err
} else {
// TODO: MSAU: should unknown err return pathExists=true for safety?
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The original function returned false. I think it should actually return true otherwise we may return cleanup success for unknown errors.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed it back to original for now. I see util/file/FileExists also returns false for all errors.

// PathExists returns true if the specified path exists.
// TODO: MSAU this is not a good name
func PathExists(path string, mounter Interface) (bool, error) {
_, err := mounter.ExistsPath(path)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may not work for containerized kubelet because it would cause the path to be checked based on the host, not inside the kubelet container.

I will change it back to os.Stat(). I will need to figure out a different way to unit test this.

@k8s-ci-robot k8s-ci-robot added sig/storage Categorizes an issue or PR as relevant to SIG Storage. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Dec 27, 2018
@msau42
Copy link
Member Author

msau42 commented Dec 27, 2018

Still todo:

  • Figure out a way to unit test this
  • Write an e2e test
  • Open up issues for refactoring/cleanup
  • Test containerized kubelet

@k8s-ci-robot k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. sig/testing Categorizes an issue or PR as relevant to SIG Testing. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 27, 2018
@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Dec 28, 2018
@msau42 msau42 changed the title [WIP] Use UnmountMountPoint util to clean up subpaths Use UnmountMountPoint util to clean up subpaths Dec 28, 2018
@k8s-ci-robot k8s-ci-robot removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 28, 2018
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 4, 2019
@jingxu97
Copy link
Contributor

jingxu97 commented Jan 4, 2019

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 4, 2019
@jingxu97
Copy link
Contributor

jingxu97 commented Jan 4, 2019

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jingxu97, msau42

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 4, 2019
@msau42
Copy link
Member Author

msau42 commented Jan 4, 2019

/test pull-kubernetes-integration

@msau42
Copy link
Member Author

msau42 commented Jan 4, 2019

/retest

klog.V(4).Infof("%q is unmounted, deleting the directory", mountPath)
return os.Remove(mountPath)
}
return fmt.Errorf("Failed to unmount path %v", mountPath)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The message here seems inaccurate : the path is likely mount point

k8s-ci-robot added a commit that referenced this pull request Jan 7, 2019
…-upstream-release-1.13

Automated cherry pick of #71804: Move unmount volume util from pkg/volume/util to
k8s-ci-robot added a commit that referenced this pull request Jan 8, 2019
…-upstream-release-1.12

Automated cherry pick of #71804: Move unmount volume util from pkg/volume/util to
k8s-ci-robot added a commit that referenced this pull request Jan 12, 2019
…-upstream-release-1.11

Automated cherry pick of #71804: Move unmount volume util from pkg/volume/util to
@msau42 msau42 deleted the fix-subpath-nfs branch March 4, 2019 04:14
return fmt.Errorf("Failed to unmount path %v", mountPath)
}

// TODO: clean this up to use pkg/util/file/FileExists
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are more than one version of FileExists:

pkg/kubelet/kubeletconfig/util/files/files.go
vendor/golang.org/x/tools/go/buildutil/util.go

I wonder which one we should use.

This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/storage Categorizes an issue or PR as relevant to SIG Storage. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Pod using subpath with nfs is stuck in terminating
6 participants